We present the Rubin Observatory system for data storage/retrieval and pipelined code execution. The layer for data storage and retrieval is named the Butler. It consists of a relational database, known as the registry, to keep track of metadata and relations, and a system to manage where the data is located, named the datastore. Together these systems create an abstraction layer that science algorithms can be written against. This abstraction layer manages the complexities of the large data volumes expected and allows algorithms to be written independently, yet be tied together automatically into a coherent processing pipeline. This system consists of tools which execute these pipelines by transforming them into execution graphs which contain concrete data stored in the Butler. The pipeline infrastructure is designed to be scalable in nature, allowing execution on environments ranging from a laptop all the way up to multi-facility data centers. This presentation will focus on the data management aspects as well as an overview on the creation of pipelines and the corresponding execution graphs.

The Rubin Observatory's Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a laptop using a local file system. The Butler and pipeline system are now in daily use during Rubin construction and early operations.

The INT Galactic Plane Survey (IGAPS) is the merger of the optical photometric surveys, IPHAS and UVEX, based on data from the Isaac Newton Telescope (INT) obtained between 2003 and 2018. Here, we present the IGAPS point source catalogue. It contains 295.4 million rows providing photometry in the filters, i , r , narrow-band H α , g , and U RGO . The IGAPS footprint fills the Galactic coordinate range, | b | < 5° and 30° < ℓ < 215°. A uniform calibration, referred to as the Pan-STARRS system, is applied to g , r , and i , while the H α calibration is linked to r and then is reconciled via field overlaps. The astrometry in all five bands has been recalculated in the reference frame of Gaia Data Release 2. Down to i ∼ 20 mag (Vega system), most stars are also detected in g , r , and H α . As exposures in the r band were obtained in both the IPHAS and UVEX surveys, typically a few years apart, the catalogue includes two distinct r measures, r I and r U . The r 10 σ limiting magnitude is approximately 21, with median seeing of 1.1 arcsec. Between approximately 13th and 19th mag in all bands, the photometry is internally reproducible to within 0.02 mag. Stars brighter than r = 19.5 mag are tested for narrow-band H α excess signalling line emission, and for variation exceeding | r I − r U | = 0.2 mag. We find and flag 8292 candidate emission line stars and over 53 000 variables (both at > 5 σ confidence).

3C 279 is an archetypal blazar with a prominent radio jet that show broadband flux density variability across the entire electromagnetic spectrum. We use an ultra-high angular resolution technique – global Very Long Baseline Interferometry (VLBI) at 1.3 mm (230 GHz) – to resolve the innermost jet of 3C 279 in order to study its fine-scale morphology close to the jet base where highly variableγ-ray emission is thought to originate, according to various models. The source was observed during four days in April 2017 with the Event Horizon Telescope at 230 GHz, including the phased Atacama Large Millimeter/submillimeter Array (ALMA), at an angular resolution of ∼20 μas (at a redshift ofz = 0.536 this corresponds to ∼0.13 pc ∼ 1700 Schwarzschild radii with a black hole massMBH = 8 × 108 M⊙). Imaging and model-fitting techniques were applied to the data to parameterize the fine-scale source structure and its variation. We find a multicomponent inner jet morphology with the northernmost component elongated perpendicular to the direction of the jet, as imaged at longer wavelengths. The elongated nuclear structure is consistent on all four observing days and across different imaging methods and model-fitting techniques, and therefore appears robust. Owing to its compactness and brightness, we associate the northern nuclear structure as the VLBI “core”. This morphology can be interpreted as either a broad resolved jet base or a spatially bent jet. We also find significant day-to-day variations in the closure phases, which appear most pronounced on the triangles with the longest baselines. Our analysis shows that this variation is related to a systematic change of the source structure. Two inner jet components move non-radially at apparent speeds of ∼15 cand ∼20 c(∼1.3 and ∼1.7 μas day−1, respectively), which more strongly supports the scenario of traveling shocks or instabilities in a bent, possibly rotating jet. The observed apparent speeds are also coincident with the 3C 279 large-scale jet kinematics observed at longer (cm) wavelengths, suggesting no significant jet acceleration between the 1.3 mm core and the outer jet. The intrinsic brightness temperature of the jet components are ≲1010K, a magnitude or more lower than typical values seen at ≥7 mm wavelengths. The low brightness temperature and morphological complexity suggest that the core region of 3C 279 becomes optically thin at short (mm) wavelengths.

Cosmic particles hitting Earth's moon produce radio emission via the Askaryan effect. If the resulting radio ns-pulse can be detected by radio telescopes, this technique potentially increases the available collective area for ZeV scale particles by several orders of magnitude compared to current experiments. The LOw Frequency ARray (LOFAR) is the largest radio telescope operating in the optimum frequency regime for this technique. In this contribution, we report on the status of the implementation of the lunar detection mode at LOFAR.

The LOw-Frequency ARray (LOFAR) telescope measures radio emission from air showers. In order to interpret the data, an absolute, frequency dependent calibration is required. Due to a growing need for a better understanding of the measured frequency spectrum, we revisit the calibration of the LOFAR antennas in the range of 30—80 MHz. Using the galactic radio emission and a detailed model of the LOFAR signal chain, we find a calibration that provides an absolute energy scale and allows us to study frequency dependent features in measured air shower signals.

Lightning is a dangerous yet poorly understood natural phenomenon. Lightning forms a network of plasma channels propagating away from the initiation point with both positively and negatively charged ends-called positive and negative leaders1. Negative leaders propagate in discrete steps, emitting copious radio pulses in the 30-300-megahertz frequency band2-8 that can be remotely sensed and imaged with high spatial and temporal resolution9-11. Positive leaders propagate more continuously and thus emit very little high-frequency radiation12. Radio emission from positive leaders has nevertheless been mapped13-15, and exhibits a pattern that is different from that of negative leaders11-13,16,17. Furthermore, it has been inferred that positive leaders can become transiently disconnected from negative leaders9,12,16,18-20, which may lead to current pulses that both reconnect positive leaders to negative leaders11,16,17,20-22 and cause multiple cloud-to-ground lightning events1. The disconnection process is thought to be due to negative differential resistance18, but this does not explain why the disconnections form primarily on positive leaders22, or why the current in cloud-to-ground lightning never goes to zero23. Indeed, it is still not understood how positive leaders emit radio-frequency radiation or why they behave differently from negative leaders. Here we report three-dimensional radio interferometric observations of lightning over the Netherlands with unprecedented spatiotemporal resolution. We find small plasma structures-which we call 'needles'-that are the dominant source of radio emission from the positive leaders. These structures appear to drain charge from the leader, and are probably the reason why positive leaders disconnect from negative ones, and why cloud-to-ground lightning connects to the ground multiple times.

The radio signals emitted by Extensive Air Showers have been successfully used for the last decade by LOFAR to reconstruct the properties of the primary cosmic rays. Since an effective real-time recognition system for the very short radio pulses is lacking, cosmic-ray acquisition is currently triggered by an external array of particle detector, called LORA, limiting the LOFAR collecting area to the area covered by LORA. A new algorithm for the real-time cosmic-ray detection has been developed for the LOFAR Low Band Antenna, which are sensitive between 10 and 90 MHz, and is here presented together with the latest results.

The LOFAR radio telescope is able to measure the radio emission from cosmic ray induced air showers with hundreds of individual antennas. This allows for precision testing of the emission mechanisms for the radio signal as well as determination of the depth of shower maximum Xmax, the shower observable most sensitive to the mass of the primary cosmic ray, to better than 20 g cm−2. With a densely instrumented circular area of roughly 320 m2, LOFAR is targeting for cosmic ray astrophysics in the energy range 1016–1018 eV. In this contribution we give an overview of the status, recent results, and future plans of cosmic ray detection with the LOFAR radio telescope.

The LOw-Frequency ARray (LOFAR) is used to make precise measurements of radio emission from extensive air showers, yielding information about the primary cosmic ray. Interpreting the measured data requires an absolute and frequency-dependent calibration of the LOFAR system response. This is particularly important for spectral analyses, because the shape of the detected signal holds information about the shower development. We revisit the calibration of the LOFAR antennas in the range of 30-80 MHz. Using the Galactic emission and a detailed model of the LOFAR signal chain, we find an improved calibration that provides an absolute energy scale and allows for the study of frequency dependent features in measured signals. With the new calibration, systematic uncertainties of 13% are reached, and comparisons of the spectral shape of calibrated data with simulations show promising agreement. (C) 2019 Elsevier B.V. All rights reserved.

The Large Synoptic Survey Telescope (LSST) is an ambitious astronomical survey with a similarly ambitious Data Management component. Data Management for LSST includes processing on both nightly and yearly cadences to generate transient alerts, deep catalogs of the static sky, and forced photometry light-curves for billions of objects at hundreds of epochs, spanning at least a decade. The algorithms running in these pipelines are individually sophisticated and interact in subtle ways. This paper provides an overview of those pipelines, focusing more on those interactions than the details of any individual algorithm.

The LOw Frequency ARay (LOFAR) is a multi-purpose radio antenna array aimed to detect radio signals in the frequency range 10 - 240 MHz, covering a large surface in Northern Europe with a higher density in the Netherlands. Analytical calculations and simulation studies performed in the 2000s indicate a dependence of the radio frequency spectrum on cosmic-ray characteristics. The high number density of radio antennas at the LOFAR core allows to characterise the observed cascade in a detailed way. The radio signal emitted by air showers in the atmosphere has been studied accurately in the 30 - 80 MHz frequency range. The analysis has been conducted on simulated events and on real data detected by LOFAR since 2011. The final aim of this study is to find an independent method to infer information of primary cosmic rays for improving the reconstruction of primary particle parameters. Results show a strong dependence of the frequency spectrum on the distance to the shower axis for both real data and simulations. Furthermore, results show that this method is very sensitive to the precision in reconstructing the position of the shower axis at ground, and to different antenna calibration procedures. A correlation between the frequency spectrum and geometrical distance to the shower maximum development X-max has also been investigated.

Writing generic data processing pipelines requires that the algorithmic code does not ever have to know about data formats of files, or the locations of those files. At LSST we have a software system known as "the Data Butler," that abstracts these details from the software developer. Scientists can specify the dataset they want in terms they understand, such as filter, observation identifier, date of observation, and instrument name, and the Butler translates that to one or more files which are read and returned to them as a single Python object. Conversely, once they have created a new dataset they can give it back to the Butler, with a label describing its new status, and the Butler can write it in whatever format it has been configured to use. All configuration is in YAML and supports standard defaults whilst allowing overrides.

We describe here the most ambitious survey currently planned in the optical, the Large Synoptic Survey Telescope (LSST). The LSST design is driven by four main science themes: probing dark energy and dark matter, taking an inventory of the solar system, exploring the transient optical sky, and mapping the Milky Way. LSST will be a large, wide-field ground-based system designed to obtain repeated images covering the sky visible from Cerro Pachón in northern Chile. The telescope will have an 8.4 m (6.5 m effective) primary mirror, a 9.6 deg 2 field of view, a 3.2-gigapixel camera, and six filters ( ugrizy ) covering the wavelength range 320–1050 nm. The project is in the construction phase and will begin regular survey operations by 2022. About 90% of the observing time will be devoted to a deep-wide-fast survey mode that will uniformly observe a 18,000 deg 2 region about 800 times (summed over all six bands) during the anticipated 10 yr of operations and will yield a co-added map to r ∼ 27.5. These data will result in databases including about 32 trillion observations of 20 billion galaxies and a similar number of stars, and they will serve the majority of the primary science programs. The remaining 10% of the observing time will be allocated to special projects such as Very Deep and Very Fast time domain surveys, whose details are currently under discussion. We illustrate how the LSST science drivers led to these choices of system parameters, and we describe the expected data products and their characteristics.

The Event Horizon Telescope (EHT) is a very long baseline interferometry (VLBI) array that comprises millimeter- and submillimeter-wavelength telescopes separated by distances comparable to the diameter of the Earth. At a nominal operating wavelength of 1.3 mm, EHT angular resolution (lambda/D) is 25 micro-as, which is sufficient to resolve nearby supermassive black hole candidates on spatial and temporal scales that correspond to their event horizons. With this capability, the EHT scientific goals are to probe general relativistic effects in the strong-field regime and to study accretion and relativistic jet formation near the black hole boundary. In this Letter we describe the system design of the EHT, detail the technology and instrumentation that enable observations, and provide measures of its performance. Meeting the EHT science objectives has required several key developments that have facilitated the robust extension of the VLBI technique to EHT observing wavelengths and the production of instrumentation that can be deployed on a heterogeneous array of existing telescopes and facilities. To meet sensitivity requirements, high-bandwidth digital systems were developed that process data at rates of 64 gigabit/s, exceeding those of currently operating cm-wavelength VLBI arrays by more than an order of magnitude. Associated improvements include the development of phasing systems at array facilities, new receiver installation at several sites, and the deployment of hydrogen maser frequency standards to ensure coherent data capture across the array. These efforts led to the coordination and execution of the first Global EHT observations in 2017 April, and to event-horizon-scale imaging of the supermassive black hole candidate in M87.

The steep decrease of the flux of ultra-high energy cosmic rays (UHECR) provides a challenge to answer the long standing question about their origin and nature. A significant increase in detector volume may be achieved byemploying Earth’s moon as a detector that is read out using existing Earth-bound radio telescopes by searching for the radio pulses emitted by the particle shower in the lunar rock. In this contribution we will report on the properties of a corresponding detection mode currently under development for the LOFAR Radio telescope.

Context . In the previous decade, two new classes of fast radio transients were detected: the Galactic, rotating radio transients (RRATs) and the extragalactic fast radio bursts (FRBs). If the detectable emission of these objects extends to lower radio frequencies, the LOw Frequency ARray (LOFAR) is ideally suited to seek and localize these transients at frequencies of 10–250 MHz. This is due to LOFAR’s sensitivity, diverse beamform capabilities, and transient buffers for the individual elements that allow post-event imaging of events, potentially at arcsecond resolution. Aims . Our aim is to identify and localize pulses at frequencies below 250 MHz and, in the case of nondetections, derive upper limits on the sky and volume rates of FRBs. Methods . A real-time search program for fast radio transients is installed on the LOFAR systems which runs commensally with other observations, and uses the wide incoherent LOFAR beam (11.25 deg 2 at 150 MHz). Buffered data from hundreds of dipoles are used to reconstruct the direction and polarization information of the event, and to distinguish between celestial, terrestrial, and instrumental origins. Results . Observations were taken covering either the frequency range 119–151 MHz or in four frequency bands, each of 2 MHz in width, centered at 124, 149, 156, and 185 MHz. A first pilot survey covered a range of dispersion measures (DM) below 120 pc cm −3 , focusing on Galactic sources, and resulted in an upper limit on the transient rate at LOFAR frequencies of less than 1500 events per sky per day above a fluency of 1.6 kJy ms for an 8-ms pulse. A second pilot survey covered a range of DMs below 500 pc cm −3 , focusing on extragalactic sources to about 1 Gpc, and resulted in an upper limit of 1400 events per sky per day above a fluency of 6.0 kJy ms for an 8-ms pulse. Using a model for the distance-DM relationship, this equates to an upper limit of 134 events per Gpc 3 per day.

The energy and mass composition of cosmic rays influence how the energy density of the radio emission of air showers is distributed on the ground. A precise description of the radio profiles can, therefore, be used to reconstruct the properties of the primary cosmic rays. Here, such a description is presented, using a separate treatment of the two radio-emission mechanisms, the geomagnetic effect and the charge excess effect. The model is parametrized as a function that depends only on the shower parameters, allowing for a precise reconstruction of the properties of the primary cosmic rays. This model is applied to cosmic-ray events measured with LOFAR and it is capable of reconstructing the properties of air showers correctly.