Astronomers require efficient automated detection and classification pipelines when conducting large-scale surveys of the (optical) sky for variable and transient sources. Such pipelines are fundamentally important, as they permit rapid follow-up and analysis of those detections most likely to be of scientific value. We therefore present a deep learning pipeline based on the convolutional neural network architecture called MeerCRAB. It is designed to filter out the so called "bogus" detections from true astrophysical sources in the transient detection pipeline of the MeerLICHT telescope. Optical candidates are described using a variety of 2D images and numerical features extracted from those images. The relationship between the input images and the target classes is unclear, since the ground truth is poorly defined and often the subject of debate. This makes it difficult to determine which source of information should be used to train a classification algorithm. We therefore used two methods for labelling our data (i) thresholding and (ii) latent class model approaches. We deployed variants of MeerCRAB that employed different network architectures trained using different combinations of input images and training set choices, based on classification labels provided by volunteers. The deepest network worked best with an accuracy of 99.5% and Matthews correlation coefficient (MCC) value of 0.989. The best model was integrated to the MeerLICHT transient vetting pipeline, enabling the accurate and efficient classification of detected transients that allows researchers to select the most promising candidates for their research goals.

With the arrival of a number of wide-field snapshot image-plane radio transient surveys, there will be a huge influx of images in the coming years making it impossible to manually analyse the datasets. Automated pipelines to process the information stored in the images are being developed, such as the LOFAR Transients Pipeline, outputting light curves and various transient parameters. These pipelines have a number of tuneable parameters that require training to meet the survey requirements. This paper utilises both observed and simulated datasets to demonstrate different machine learning strategies that can be used to train these parameters. We use a simple anomaly detection algorithm and a penalised logistic regression algorithm. The datasets used are from LOFAR observations and we process the data using the LOFAR Transients Pipeline; however the strategies developed are applicable to any light curve datasets at different frequencies and can be adapted to different automated pipelines. These machine learning strategies are publicly available as Python tools that can be downloaded and adapted to different datasets (https://github.com/AntoniaR/TraP_ML_tools).

Coming high-cadence wide-field optical telescopes will image hundreds of thousands of sources per minute. Besides inspecting the near real-time data streams for transient and variability events, the accumulated data archive is a wealthy laboratory for making complementary scientific discoveries. The goal of this work is to optimise column-oriented database techniques to enable the construction of a full-source and light-curve database for large-scale surveys, that is accessible by the astronomical community. We adopted LOFAR's Transients Pipeline as the baseline and modified it to enable the processing of optical images that have much higher source densities. The pipeline adds new source lists to the archive database, while cross-matching them with the known catalogued sources in order to build a full light-curve archive. We investigated several techniques of indexing and partitioning the largest tables, allowing for faster positional source look-ups in the cross matching algorithms. We monitored all query run times in long-term pipeline runs where we processed a subset of IPHAS data that have image source density peaks over 170,000 per field of view (500,000 deg(-2)). Our analysis demonstrates that horizontal table partitions of declination widths of one-degree control the query run times. Usage of an index strategy where the partitions are densely sorted according to source declination yields another improvement. Most queries run in sublinear time and a few (< 20%) run in linear time, because of dependencies on input source-list and result-set size. We observed that for this logical database partitioning schema the limiting cadence the pipeline achieved with processing IPHAS data is 25 s. (C) 2018 Elsevier B.V. All rights reserved.

The Radboud University Nijmegen in collaboration with the NOVA Optical Infrared Instrumentation group at ASTRON is currently leading the development and realization of the BlackGEM observing facility. The BlackGEM science team aims to be the first to catch the optical counterpart of a gravitational wave event. The BlackGEM project will put an array of three medium-sized optical telescopes at the La Silla site of the European Southern Observatory in Chile. It is uniquely equipped to achieve a combination of wide-field and high sensitivity through its array-like approach. Each BlackGEM unit telescope is a modified Dall-Kirkham-type telescope consisting of a 65cm primary mirror, a 21cm spherical secondary mirror and a triplet corrector lens. The spatial resolution on the sky will be 0.56 asec/pixel and the total field-of-view per telescope is 2.7 square degrees. The main requirement is to achieve a 5-sigma sensitivity of 23rd magnitude within a 5-minute exposure under 15 m/s wind gust conditions. This demands a very stable optical system with tight control of all the error contributions. This has been realized with a spreadsheet based integrated instrument model. The model contains all relevant telescope instrument parameters and environmental conditions. The spreadsheet is partly used for performance calculations and partly used to combine and integrate the output from several other sources. The spreadsheet model calculates the overall performance based on an Exposure Time Calculator using the Noise Equivalent Area metric (NEA). The NEA is further budgeted over 7 main High Level Requirements. The spreadsheet model is coupled to 1) a ZEMAX telescope optical model 2) a telescope FEM analysis to predict the optomechanical response under various gravity, temperature and wind load conditions, 3) a Matlab Simulink thermal model to predict the transient temperature behaviour of the most important telescope elements and 4) a Matlab Simulink control model to predict the performance of the active M2 mirror. All outputs are collected in a system performance budget that readily shows the compliance with the main High Level Requirements.

The ground-based wide-angle camera array (GWAC), a part of the SVOM space mission, will search for various types of optical transients by continuously imaging a field of view (FOV) of 5000 degrees2 every 15 s. Each exposure consists of 36 × 4k × 4k pixels, typically resulting in 36 × ∼175,600 extracted sources. For a modern time-domain astronomy project like GWAC, which produces massive amounts of data with a high cadence, it is challenging to search for short timescale transients in both real-time and archived data, and to build long-term light curves for variable sources. Here, we develop a high-cadence, high-density light curve pipeline (HCHDLP) to process the GWAC data in real-time, and design a distributed shared-nothing database to manage the massive amount of archived data which will be used to generate a source catalog with more than 100 billion records during 10 years of operation. First, we develop HCHDLP based on the column-store DBMS of MonetDB, taking advantage of MonetDB's high performance when applied to massive data processing. To realize the real-time functionality of HCHDLP, we optimize the pipeline in its source association function, including both time and space complexity from outside the database (SQL semantic) and inside (RANGE-JOIN implementation), as well as in its strategy of building complex light curves. The optimized source association function is accelerated by three orders of magnitude. Second, we build a distributed database using a two-level time partitioning strategy via the MERGE TABLE and REMOTE TABLE technology of MonetDB. Intensive tests validate that our database architecture is able to achieve both linear scalability in response time and concurrent access by multiple users. In summary, our studies provide guidance for a solution to GWAC in real-time data processing and management of massive data.

We present the MeerLICHT and BlackGEM telescopes, which are wide-field optical telescopes that are currently being built to study transient phenomena, gravitational wave counterparts and variable stars. The telescopes have 65 cm primary mirrors and a 2.7 square degree field-of-view. The MeerLICHT and BlackGEM projects have different science goals, but will use identical telescopes. The first telescope, MeerLICHT, will be commissioned at Sutherland (South Africa) in the first quarter of 2017. It will co-point with MeerKAT to collect optical data commensurate with the radio observations. After careful analysis of MeerLICHT's performance, three telescopes of the same type will be commissioned in La Silla (Chile) in 2018 to form phase I of the BlackGEM array. BlackGEM aims at detecting and characterizing optical counterparts of gravitational wave events detected by Advanced LIGO and Virgo. In this contribution we present an overview of the science goals, the design and the status of the two projects.

We present the results of a four-month campaign searching for low-frequency radio transients near the North Celestial Pole with the Low-Frequency Array (LOFAR), as part of the Multifrequency Snapshot Sky Survey (MSSS). The data were recorded between 2011 December and 2012 April and comprised 2149 11-minute snapshots, each covering 175 deg^2. We have found one convincing candidate astrophysical transient, with a duration of a few minutes and a flux density at 60 MHz of 15-25 Jy. The transient does not repeat and has no obvious optical or high-energy counterpart, as a result of which its nature is unclear. The detection of this event implies a transient rate at 60 MHz of 3.9 (+14.7, -3.7) x 10^-4 day^-1 deg^-2, and a transient surface density of 1.5 x 10^-5 deg^-2, at a 7.9-Jy limiting flux density and ~10-minute time-scale. The campaign data were also searched for transients at a range of other time-scales, from 0.5 to 297 min, which allowed us to place a range of limits on transient rates at 60 MHz as a function of observation duration.

We present the highest-quality polarisation profiles to date of 16 non-recycled pulsars and four millisecond pulsars, observed below 200 MHz with the LOFAR high-band antennas. Based on the observed profiles, we perform an initial investigation of expected observational effects resulting from the propagation of polarised emission in the pulsar magnetosphere and the interstellar medium. The predictions of magnetospheric birefringence in pulsars have been tested using spectra of the pulse width and fractional polarisation from multifrequency data. The derived spectra offer only partial support for the expected effects of birefringence on the polarisation properties, with only about half of our sample being consistent with the model's predictions. It is noted that for some pulsars these measurements are contaminated by the effects of interstellar scattering. For a number of pulsars in our sample, we have observed significant variations in the amount of Faraday rotation as a function of pulse phase, which is possibly an artefact of scattering. These variations are typically two orders of magnitude smaller than that observed at 1400 MHz by Noutsos et al. (2009), for a different sample of southern pulsars. In this paper we present a possible explanation for the difference in magnitude of this effect between the two frequencies, based on scattering. Finally, we have estimated the magnetospheric emission heights of low-frequency radiation from four pulsars, based on the phase lags between the flux-density and the PA profiles, and the theoretical framework of Blaskiewicz, Cordes & Wasserman (1991). These estimates yielded heights of a few hundred km; at least for PSR B1133+16, this is consistent with emission heights derived based on radius-to-frequency mapping, but is up to a few times larger than the recent upper limit based on pulsar timing.

Cassiopeia A was observed using the low-band antennas of the LOw Frequency ARray (LOFAR) with high spectral resolution. This allowed a search for radio recombination lines (RRLs) along the line-of-sight to this source. Five carbon alpha RRLs were detected in absorption between 40 and 50 MHz with a signal-to-noise ratio of >5 from two independent LOFAR data sets. The derived line velocities (v(LSR) similar to -50 km s(-1)) and integrated optical depths (similar to 13 s(-1)) of the RRLs in our spectra, extracted over the whole supernova remnant, are consistent within each LOFAR data set and with those previously reported. For the first time, we are able to extract spectra against the brightest hotspot of the remnant at frequencies below 330 MHz. These spectra show significantly higher (15-80 percent) integrated optical depths, indicating that there is small-scale angular structure of the order of similar to 1 pc in the absorbing gas distribution over the face of the remnant. We also place an upper limit of 3 x 10(-4) on the peak optical depths of hydrogen and helium RRLs. These results demonstrate that LOFAR has the desired spectral stability and sensitivity to study faint recombination lines in the decameter band.

Cassiopeia A was observed using the low-band antennas of the LOw Frequency ARray (LOFAR) with high spectral resolution. This allowed a search for radio recombination lines (RRLs) along the line-of-sight to this source. Five carbon α RRLs were detected in absorption between 40 and 50 MHz with a signal-to-noise ratio of > 5 from two independent LOFAR data sets. The derived line velocities ( v LSR ∼ − 50 km s − 1 ) and integrated optical depths ( ∼ 13 s − 1 ) of the RRLs in our spectra, extracted over the whole supernova remnant, are consistent within each LOFAR data set and with those previously reported. For the first time, we are able to extract spectra against the brightest hotspot of the remnant at frequencies below 330 MHz. These spectra show significantly higher (15–80 percent) integrated optical depths, indicating that there is small-scale angular structure of the order of ∼ 1 pc in the absorbing gas distribution over the face of the remnant. We also place an upper limit of 3 × 10 − 4 on the peak optical depths of hydrogen and helium RRLs. These results demonstrate that LOFAR has the desired spectral stability and sensitivity to study faint recombination lines in the decameter band.

SciQL (pronounced as 'cycle') is a novel SQL-based array query language for scientific applications with both tables and arrays as first class citizens. SciQL lowers the entrance fee of adopting relational DBMS (RDBMS) in scientific domains, because it includes functionality often only found in mathematics software packages. In this paper, we demonstrate the usefulness of SciQL for astronomical data processing using examples from the Transient Key Project of the LOFAR radio telescope. In particular, how the LOFAR light-curve database of all detected sources can be constructed, by correlating sources across the spatial, frequency, time and polarisation domains.

Dispersion in the interstellar medium is a well known phenomenon that follows a simple relationship, which has been used to predict the time delay of dispersed radio pulses since the late 1960s. We performed wide-band simultaneous observations of four pulsars with LOFAR (at 40-190 MHz), the 76-m Lovell Telescope (at 1400 MHz) and the Effelsberg 100-m Telescope (at 8000 MHz) to test the accuracy of the dispersion law over a broad frequency range. In this paper we present the results of these observations which show that the dispersion law is accurate to better than 1 part in 100000 across our observing band. We use this fact to constrain some of the properties of the ISM along the line-of-sight and use the lack of any aberration or retardation effects to determine upper limits on emission heights in the pulsar magnetosphere. We also discuss the effect of pulse profile evolution on our observations, and the implications that it could have for precision pulsar timing projects such as the detection of gravitational waves with pulsar timing arrays.

The International LOFAR Telescope is designed to carry out unique science in the spatial, spectral, polarisation and temporal domains. The Transients Key Science Project aims to study all transient and variable sources detected by LOFAR. One of its products will be an up-to-date catalogue of all sources detected by LOFAR, i.e. a spectral light-curve database, with real-time capabilities, and expected to grow gradually with 50-100 TB/yr. The response time to transient and variable events depends strongly on the query execution plans of the algorithms that search the LOFAR light-curve database for previous (non-)detections in the spatial, spectral, polarisation and temporal domains.Here we show how the Transients Key Science Project of LOFAR approaches these challenges by using column-stores, sharded databases and implementing the new array query language SciQL (pronounced as 'cycle').

Time-domain astronomy is becoming a fundamental aspect of the next generation of astronomical instruments. The timing properties will revolutionise the studies of all kinds of astronomical objects. Consequetially, the huge complex data volumes and high cadences of these facilities will force us to overhaul and extend current software solutions. LOFAR, laying the groundwork for this, will produce a continuously updated spectral light-curve catalogue of all detected sources, with real-time capabilities to cope with the growth of 50 - 100TB/yr, making it the largest dynamic astronomical catalogue. Automated pipelines use the column-store MonetDB as their key component. We exploit SciLens, a 300+ node, 4-tier locally distributed cluster focussed on massive I/O. Introduction of the new array-based query language, SciQL, simplifies data exploration and mining. I will demonstrate how MonetDB/SQL & SciQL on its SciLens platform manages the millions of lightcurves for LOFAR. Initial benchmark results confirm the linear scale-up performance over tens of TBs using tens of nodes.

Low frequency radio waves, while challenging to observe, are a rich source of information about pulsars. The LOw Frequency ARray (LOFAR) is a new radio interferometer operating in the lowest 4 octaves of the ionospheric "radio window": 10-240 MHz, that will greatly facilitate observing pulsars at low radio frequencies. Through the huge collecting area, long baselines, and flexible digital hardware, it is expected that LOFAR will revolutionize radio astronomy at the lowest frequencies visible from Earth. LOFAR is a next-generation radio telescope and a pathfinder to the Square Kilometre Array (SKA), in that it incorporates advanced multi-beaming techniques between thousands of individual elements. We discuss the motivation for low-frequency pulsar observations in general and the potential of LOFAR in addressing these science goals. We present LOFAR as it is designed to perform high-time-resolution observations of pulsars and other fast transients, and outline the various relevant observing modes and data reduction pipelines that are already or will soon be implemented to facilitate these observations. A number of results obtained from commissioning observations are presented to demonstrate the exciting potential of the telescope. This paper outlines the case for low frequency pulsar observations and is also intended to serve as a reference for upcoming pulsar/fast transient science papers with LOFAR.

In this paper we present the results of a survey for radio transients using data obtained from the Very Large Array archive. We have reduced, using a pipeline procedure, 5037 observations of the most common pointings - i.e. the calibrator fields. These fields typically contain a relatively bright point source and are used to calibrate `target' observations: they are therefore rarely imaged themselves. The observations used span a time range ~ 1984 - 2008 and consist of eight different pointings, three different frequencies (8.4, 4.8 and 1.4 GHz) and have a total observing time of 435 hours. We have searched for transient and variable radio sources within these observations using components from the prototype LOFAR transient detection system. In this paper we present the methodology for reducing large volumes of Very Large Array data; and we also present a brief overview of the prototype LOFAR transient detection algorithms. No radio transients were detected in this survey, therefore we place an upper limit on the snapshot rate of GHz frequency transients > 8.0 mJy to rho less than or equal to 0.032 deg^-2 that have typical timescales 4.3 to 45.3 days. We compare and contrast our upper limit with the snapshot rates - derived from either detections or non-detections of transient and variable radio sources - reported in the literature. When compared with the current Log N - Log S distribution formed from previous surveys, we show that our upper limit is consistent with the observed population. Current and future radio transient surveys will hopefully further constrain these statistics, and potentially discover dominant transient source populations. In this paper we also briefly explore the current transient commissioning observations with LOFAR, and the impact they will make on the field.

The 2004 December 27 giant flare from SGR 1806-20 produced a radio nebula that was detectable for weeks. It was observed at a wide range of radio frequencies. We made a total of 19 WSRT observations. Most of these were performed quasi simultaneously at either two or three frequencies, starting 2005 January 4 and ending 2005 January 29. We reobserved the field in 2005 April/May, which facilitated an accurate subtraction of background sources. At 350 MHz, we find that the total intensity of the source is lower than expected from the GMRT 240 MHz and 610 MHz measurements and inconsistent with spectral indices published previously. Our 850 MHz flux densities, however, are consistent with earlier results. There is no compelling evidence for significant depolarization at any frequency. We do, however, find that polarization angles differ substantially from those at higher frequencies. Low frequency polarimetry and total intensity measurements provide a number of clues with regard to substructure in the radio nebula associated with SGR 1806-20. In general, for a more complete understanding of similar events, low frequency observations can provide new insights into the physics of the radio source.

Context. The International LOFAR Telescope will produce vast amounts of data on time scales as short as one second. The Transients Key Project aims to study transient and variable sources detected by the LOFAR telescope. An automated software pipeline that processes the data to identify, classify, monitor and respond to astronomical transients in nearly real-time is under development. Aims. LOFAR calibration, observations, and future surveys will depend on reliable local and global sky models. Building and im- proving such models is asserted by an implemented internal catalog database that keeps track of the sources detected by LOFAR and is preloaded with existing external catalogs. Source association and transient/variability detection algorithms are needed in order to produce the source lists for such models. Methods. Algorithms concerning source identification, association, and variability detection are developed inside the database engine as an integrated part of the software pipeline. To quantify the reliability of a source association pair and the variability of a source, we use the likelihood ratio method, and two variability indices, respectively. First, we will investigate simulated data, after which we will apply it to the intensively monitored field of GRB 030329. Results. Criteria for a reliable source association and source variability characteristics along with source lists are produced by the algorithms. A list of sources of interest in the GRB 030329 field is presented.