As current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational challenges, and their long-term storage and archiving is scientifically essential for generating reliable, reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer. The RTP system consists of analysis routines run on raw data shortly after they are acquired, such as calibration and detection of radio-frequency interference (RFI) events. RTP works closely with the Librarian, the HERA data storage and transfer manager which automatically ingests data and transfers copies to other clusters for post-processing analysis. Both the RTP system and the Librarian are public and open source software, which allows for them to be modified for use in other scientific collaborations. When fully constructed, HERA is projected to generate over 50 terabytes (TB) of data each night, and the RTP system enables the successful scientific analysis of these data.

We present new radio continuum observations of NGC 253 from the Murchison Widefield Array at frequencies between 76 and 227 MHz. We model the broadband radio spectral energy distribution for the total flux density of NGC 253 between 76 MHz and 11 GHz. The spectrum is best described as a sum of a central starburst and extended emission. The central component, corresponding to the inner 500 pc of the starburst region of the galaxy, is best modeled as an internally free–free absorbed synchrotron plasma, with a turnover frequency around 230 MHz. The extended emission component of the spectrum of NGC 253 is best described as a synchrotron emission flattening at low radio frequencies. We find that 34% of the extended emission (outside the central starburst region) at 1 GHz becomes partially absorbed at low radio frequencies. Most of this flattening occurs in the western region of the southeast halo, and may be indicative of synchrotron self-absorption of shock-reaccelerated electrons or an intrinsic low-energy cutoff of the electron distribution. Furthermore, we detect the large-scale synchrotron radio halo of NGC 253 in our radio images. At 154–231 MHz the halo displays the well known X-shaped/horn-like structure, and extends out to ∼8 kpc in the z -direction (from the major axis).

We present a new method for interferometric imaging that is ideal for the large fields of view and compact arrays common in 21 cm cosmology. We first demonstrate the method with simulations for two very different low frequency interferometers, the Murchison Widefield Array (MWA) and the MIT Epoch of Reionization (MITEoR) Experiment. We then apply the method to the MITEoR data set collected in July 2013 to obtain the first northern sky map from 128 MHz to 175 MHz at about 2 degree resolution, and find an overall spectral index of -2.73+/-0.11. The success of this imaging method bodes well for upcoming compact redundant low-frequency arrays such as HERA. Both the MITEoR interferometric data and the 150 MHz sky map are publicly available at http://space.mit.edu/home/tegmark/omniscope.html.

We present an improved Global Sky Model (GSM) of diffuse Galactic radio emission from 10 MHz to 5 THz, whose uses include foreground modelling for cosmic microwave background (CMB) and 21 cm cosmology. Our model improves on past work both algorithmically and by adding new data sets such as the Planck maps and the enhanced Haslam map. Our method generalizes the principal component analysis approach to handle non-overlapping regions, enabling the inclusion of 29 sky maps with no region of the sky common to all. We also perform a blind separation of our GSM into physical components with a method that makes no assumptions about physical emission mechanisms (synchrotron, free-free, dust, etc). Remarkably, this blind method automatically finds five components that have previously only been found 'by hand', which we identify with synchrotron, free-free, cold dust, warm dust, and the CMB anisotropy. Computing the cross-power spectrum between these blindly extracted components and Planck component maps, we find a strong correlation at all angular scales. The improved GSM is available online at http://github.com/jeffzhen/gsm2016.

Detection of the Epoch of Reionization HI signal requires a precise understanding of the intervening galaxies and AGN, both for instrumental calibration and foreground removal. We present a catalogue of 7394 extragalactic sources at 182 MHz detected in the RA=0 field of the Murchison Widefield Array Epoch of Reionization observation programme. Motivated by unprecedented requirements for precision and reliability we develop new methods for source finding and selection. We apply machine learning methods to self-consistently classify the relative reliability of 9490 source candidates. A subset of 7466 are selected based on reliability class and signal-to-noise ratio criteria. These are statistically cross-matched to four other radio surveys using both position and flux density information. We find 7369 sources to have confident matches, including 90 partially resolved sources that split into a total of 192 sub-components. An additional 25 unmatched sources are included as new radio detections. The catalogue sources have a median spectral index of -0.85. Spectral flattening is seen toward lower frequencies with a median of -0.71 predicted at 182 MHz. The astrometric error is 7 arcsec. compared to a 2.3 arcmin. beam FWHM. The resulting catalogue covers approximately 1400 sq. deg. and is complete to approximately 80 mJy within half beam power. This provides the most reliable discrete source sky model available to date in the MWA EoR0 field for precision foreground subtraction.

We present deep polarimetric observations at 154 MHz with the Murchison Widefield Array (MWA), covering 625 deg^2 centered on RA=0 h, Dec=-27 deg. The sensitivity available in our deep observations allows an in-band, frequency-dependent analysis of polarized structure for the first time at long wavelengths. Our analysis suggests that the polarized structures are dominated by intrinsic emission but may also have a foreground Faraday screen component. At these wavelengths, the compactness of the MWA baseline distribution provides excellent snapshot sensitivity to large-scale structure. The observations are sensitive to diffuse polarized emission at 54' resolution with a sensitivity of 5.9 mJy beam^-1 and compact polarized sources at 2.4' resolution with a sensitivity of 2.3 mJy beam^-1 for a subset (400 deg^2) of this field. The sensitivity allows the effect of ionospheric Faraday rotation to be spatially and temporally measured directly from the diffuse polarized background. Our observations reveal large-scale structures ( 1 deg - 8 deg in extent) in linear polarization clearly detectable in 2 minute snapshots, which would remain undetectable by interferometers with minimum baseline lengths >110 m at 154 MHz. The brightness temperature of these structures is on average 4 K in polarized intensity, peaking at 11 K. Rotation measure synthesis reveals that the structures have Faraday depths ranging from -2 rad m^-2 to 10 rad m^-2 with a large fraction peaking at +1 rad m^-2. We estimate a distance of 51+/-20 pc to the polarized emission based on measurements of the in-field pulsar J2330-2005. We detect four extragalactic linearly polarized point sources within the field in our compact source survey. Based on the known polarized source population at 1.4 GHz and non-detections at 154 MHz, we estimate an upper limit on the depolarization ratio of 0.08 from 1.4 GHz to 154 MHz.

In this paper we present observations, simulations, and analysis demonstrating the direct connection between the location of foreground emission on the sky and its location in cosmological power spectra from interferometric redshifted 21 cm experiments. We begin with a heuristic formalism for understanding the mapping of sky coordinates into the cylindrically averaged power spectra measurements used by 21 cm experiments, with a focus on the effects of the instrument beam response and the associated sidelobes. We then demonstrate this mapping by analyzing power spectra with both simulated and observed data from the Murchison Widefield Array. We find that removing a foreground model which includes sources in both the main field-of-view and the first sidelobes reduces the contamination in high k_parallel modes by several percent relative to a model which only includes sources in the main field-of-view, with the completeness of the foreground model setting the principal limitation on the amount of power removed. While small, a percent-level amount of foreground power is in itself more than enough to prevent recovery of any EoR signal from these modes. This result demonstrates that foreground subtraction for redshifted 21 cm experiments is truly a wide-field problem, and algorithms and simulations must extend beyond the main instrument field-of-view to potentially recover the full 21 cm power spectrum.

We present first results from radio observations with the Murchison Widefield Array seeking to constrain the power spectrum of 21 cm brightness temperature fluctuations between the redshifts of 11.6 and 17.9 (113 and 75 MHz). 3 h of observations were conducted over two nights with significantly different levels of ionospheric activity. We use these data to assess the impact of systematic errors at low frequency, including the ionosphere and radio-frequency interference, on a power spectrum measurement. We find that after the 1-3 h of integration presented here, our measurements at the Murchison Radio Observatory are not limited by RFI, even within the FM band, and that the ionosphere does not appear to affect the level of power in the modes that we expect to be sensitive to cosmology. Power spectrum detections, inconsistent with noise, due to fine spectral structure imprinted on the foregrounds by reflections in the signal-chain, occupy the spatial Fourier modes where we would otherwise be most sensitive to the cosmological signal. We are able to reduce this contamination using calibration solutions derived from autocorrelations so that we achieve an sensitivity of 10(4) mK on comoving scales k <= 0.5 h Mpc(-1). This represents the first upper limits on the 21 cm power spectrum fluctuations at redshifts 12 <= z a <= 18 but is still limited by calibration systematics. While calibration improvements may allow us to further remove this contamination, our results emphasize that future experiments should consider carefully the existence of and their ability to calibrate out any spectral structure within the EoR window.

Experiments that pursue detection of signals from the Epoch of Reionization (EoR) are relying on spectral smoothness of source spectra at low frequencies. This article empirically explores the effect of foreground spectra on EoR experiments by measuring high-resolution full-polarization spectra for the 586 brightest unresolved sources in one of the MWA EoR fields using 45 h of observation. A novel peeling scheme is used to subtract 2500 sources from the visibilities with ionospheric and beam corrections, resulting in the deepest, confusion-limited MWA image so far. The resulting spectra are found to be affected by instrumental effects, which limit the constraints that can be set on source-intrinsic spectral structure. The sensitivity and power-spectrum of the spectra are analysed, and it is found that the spectra of residuals are dominated by PSF sidelobes from nearby undeconvolved sources. We release a catalogue describing the spectral parameters for each measured source.

Detection of the cosmological neutral hydrogen signal from the Epoch of Reionization (EoR) and estimation of its basic physical parameters are principal scientific aims of many current low-frequency radio telescopes. Here we describe the Cosmological H i Power Spectrum Estimator (CHIPS), an algorithm developed and implemented with data from the Murchison Widefield Array, to compute the two-dimensional and spherically-averaged power spectrum of brightness temperature fluctuations. The principal motivations for CHIPS are the application of realistic instrumental and foreground models to form the optimal estimator, thereby maximizing the likelihood of unbiased signal estimation, and allowing a full covariant understanding of the outputs. CHIPS employs an inverse-covariance weighting of the data through the maximum likelihood estimator, thereby allowing use of the full parameter space for signal estimation (“foreground suppression”). We describe the motivation for the algorithm, implementation, application to real and simulated data, and early outputs. Upon application to a set of 3 hr of data, we set a 2σ upper limit on the EoR dimensionless power at k = 0.05 h ?> Mpc−1 of Δ k 2 < 7.6 × 10 4 ?> mK2 in the redshift range z = [6.2–6.6], consistent with previous estimates.

We present the 21 cm power spectrum analysis approach of the Murchison Widefield Array Epoch of Reionization project. In this paper, we compare the outputs of multiple pipelines for the purpose of validating statistical limits cosmological hydrogen at redshifts between 6 and 12. Multiple independent data calibration and reduction pipelines are used to make power spectrum limits on a fiducial night of data. Comparing the outputs of imaging and power spectrum stages highlights differences in calibration, foreground subtraction, and power spectrum calculation. The power spectra found using these different methods span a space defined by the various tradeoffs between speed, accuracy, and systematic control. Lessons learned from comparing the pipelines range from the algorithmic to the prosaically mundane; all demonstrate the many pitfalls of neglecting reproducibility. We briefly discuss the way these different methods attempt to handle the question of evaluating a significant detection in the presence of foregrounds.

The Murchison Widefield Array (MWA) has collected hundreds of hours of Epoch of Reionization (EoR) data and now faces the challenge of overcoming foreground and systematic contamination to reduce the data to a cosmological measurement. We introduce several novel analysis techniques, such as cable reflection calibration, hyper-resolution gridding kernels, diffuse foreground model subtraction, and quality control methods. Each change to the analysis pipeline is tested against a two-dimensional power spectrum figure of merit to demonstrate improvement. We incorporate the new techniques into a deep integration of 32 hours. of MWA data. This data set is used to place a systematic-limited upper limit on the cosmological power spectrum of Delta(2) <= 2.7 x 10(4) mK(2) at k = 0.27 h Mpc(-1) and z = 7.1, consistent with other published limits, and a modest improvement (factor of 1.4) over previous MWA results. From this deep analysis, we have identified a list of improvements to be made to our EoR data analysis strategies. These improvements will be implemented in the future and detailed in upcoming publications.

The Detection of redshifted 21 cm emission from the epoch of reionization (EoR) is a challenging task owing to strong foregrounds that dominate the signal. In this paper, we propose a general method, based on the delay spectrum approach, to extract HI power spectra that is applicable to tracking observations using an imaging radio interferometer (Delay Spectrum with Imaging Arrays (DSIA)). Our method is based on modelling the HI signal taking into account the impact of wide field effects such as the $w$-term which are then used as appropriate weights in cross-correlating the measured visibilities. Our method is applicable to any radio interferometer that tracks a phase center and could be utilized for arrays such as MWA, LOFAR, GMRT, PAPER and HERA. In the literature the delay spectrum approach has been implemented for near-redundant baselines using drift scan observations. In this paper we explore the scheme for non-redundant tracking arrays, and this is the first application of delay spectrum methodology to such data to extract the HI signal. We analyze 3 hours of MWA tracking data on the EoR1 field. We present both 2-dimensional ($k_\parallel,k_\perp$) and 1-dimensional (k) power spectra from the analysis. Our results are in agreement with the findings of other pipelines developed to analyse the MWA EoR data.

AbstractThe Murchison Widefield Array is a new low-frequency interferometric radio telescope built in Western Australia at one of the locations of the future Square Kilometre Array. We describe the automated radio-frequency interference detection strategy implemented for the Murchison Widefield Array, which is based on the aoflagger platform, and present 72–231 MHz radio-frequency interference statistics from 10 observing nights. Radio-frequency interference detection removes 1.1% of the data. Radio-frequency interference from digital TV is observed 3% of the time due to occasional ionospheric or atmospheric propagation. After radio-frequency interference detection and excision, almost all data can be calibrated and imaged without further radio-frequency interference mitigation efforts, including observations within the FM and digital TV bands. The results are compared to a previously published Low-Frequency Array radio-frequency interference survey. The remote location of the Murchison Widefield Array results in a substantially cleaner radio-frequency interference environment compared to Low-Frequency Array’s radio environment, but adequate detection of radio-frequency interference is still required before data can be analysed. We include specific recommendations designed to make the Square Kilometre Array more robust to radio-frequency interference, including: the availability of sufficient computing power for radio-frequency interference detection; accounting for radio-frequency interference in the receiver design; a smooth band-pass response; and the capability of radio-frequency interference detection at high time and frequency resolution (second and kHz-scale respectively).

Detection of 21~cm emission of HI from the epoch of reionization, at redshifts z>6, is limited primarily by foreground emission. We investigate the signatures of wide-field measurements and an all-sky foreground model using the delay spectrum technique that maps the measurements to foreground object locations through signal delays between antenna pairs. We demonstrate interferometric measurements are inherently sensitive to all scales, including the largest angular scales, owing to the nature of wide-field measurements. These wide-field effects are generic to all observations but antenna shapes impact their amplitudes substantially. A dish-shaped antenna yields the most desirable features from a foreground contamination viewpoint, relative to a dipole or a phased array. Comparing data from recent Murchison Widefield Array observations, we demonstrate that the foreground signatures that have the largest impact on the HI signal arise from power received far away from the primary field of view. We identify diffuse emission near the horizon as a significant contributing factor, even on wide antenna spacings that usually represent structures on small scales. For signals entering through the primary field of view, compact emission dominates the foreground contamination. These two mechanisms imprint a characteristic "pitchfork" signature on the "foreground wedge" in Fourier delay space. Based on these results, we propose that selective down-weighting of data based on antenna spacing and time can mitigate foreground contamination substantially by a factor ~100 with negligible loss of sensitivity.

We confirm our recent prediction of the "pitchfork" foreground signature in power spectra of high-redshift 21 cm measurements where the interferometer is sensitive to large-scale structure on all baselines. This is due to the inherent response of a wide-field instrument and is characterized by enhanced power from foreground emission in Fourier modes adjacent to those considered to be the most sensitive to the cosmological H I signal. In our recent paper, many signatures from the simulation that predicted this feature were validated against Murchison Widefield Array (MWA) data, but this key pitchfork signature was close to the noise level. In this paper, we improve the data sensitivity through the coherent averaging of 12 independent snapshots with identical instrument settings and provide the first confirmation of the prediction with a signal-to-noise ratio > 10. This wide-field effect can be mitigated by careful antenna designs that suppress sensitivity near the horizon. Simple models for antenna apertures that have been proposed for future instruments such as the Hydrogen Epoch of Reionization Array and the Square Kilometre Array indicate they should suppress foreground leakage from the pitchfork by 40 dB relative to the MWA and significantly increase the likelihood of cosmological signal detection in these critical Fourier modes in the three-dimensional power spectrum.

Detection of the fluctuations in a 21 cm line emission from neutral hydrogen during the Epoch of Reionization in thousand hour integrations poses stringent requirements on calibration and image quality, both of which necessitate accurate primary beam models. The Murchison Widefield Array (MWA) uses phased-array antenna elements which maximize collecting area at the cost of complexity. To quantify their performance, we have developed a novel beam measurement system using the 137 MHz ORBCOMM satellite constellation and a reference dipole antenna. Using power ratio measurements, we measure the in situ beampattern of the MWA antenna tile relative to that of the reference antenna, canceling the variation of satellite flux or polarization with time. We employ angular averaging to mitigate multipath effects (ground scattering) and assess environmental systematics with a null experiment in which the MWA tile is replaced with a second-reference dipole. We achieve beam measurements over 30 dB dynamic range in beam sensitivity over a large field of view (65% of the visible sky), far wider and deeper than drift scans through astronomical sources allow. We verify an analytic model of the MWA tile at this frequency within a few percent statistical scatter within the full width at half maximum. Toward the edges of the main lobe and in the sidelobes, we measure tens of percent systematic deviations. We compare these errors with those expected from known beamforming errors.

Astronomical widefield imaging of interferometric radio data is computationally expensive, especially for the large data volumes created by modern non-coplanar many-element arrays. We present a new widefield interferometric imager that uses the w-stacking algorithm and can make use of the w-snapshot algorithm. The performance dependencies of CASA's w-projection and our new imager are analysed and analytical functions are derived that describe the required computing cost for both imagers. On data from the Murchison Widefield Array, we find our new method to be an order of magnitude faster than w-projection, as well as being capable of full-sky imaging at full resolution and with correct polarisation correction. We predict the computing costs for several other arrays and estimate that our imager is a factor of 2-12 faster, depending on the array configuration. We estimate the computing cost for imaging the low-frequency Square-Kilometre Array observations to be 60 PetaFLOPS with current techniques. We find that combining w-stacking with the w-snapshot algorithm does not significantly improve computing requirements over pure w-stacking. The source code of our new imager is publicly released.

We report on the MIT Epoch of Reionization (MITEoR) experiment, a pathfinder low-frequency radio interferometer whose goal is to test technologies that improve the calibration precision and reduce the cost of the high-sensitivity 3D mapping required for 21 cm cosmology. MITEoR accomplishes this by using massive baseline redundancy, which enables both automated precision calibration and correlator cost reduction. We demonstrate and quantify the power and robustness of redundancy for scalability and precision. We find that the calibration parameters precisely describe the effect of the instrument upon our measurements, allowing us to form a model that is consistent with $\chi^2$ per degree of freedom < 1.2 for as much as 80% of the observations. We use these results to develop an optimal estimator of calibration parameters using Wiener filtering, and explore the question of how often and how finely in frequency visibilities must be reliably measured to solve for calibration coefficients. The success of MITEoR with its 64 dual-polarization elements bodes well for the more ambitious Hydrogen Epoch of Reionization Array (HERA) project and other next-generation instruments, which would incorporate many identical or similar technologies.