In this paper we demonstrate improved techniques to extend coherent processing intervals for passive radar processing, with the Murchison Widefield Array. Specifically, we apply a two stage linear range and Doppler migration compensation by utilising Keystone Formatting and a recent dechirping method. These methods are used to further demonstrate the potential for the surveillance of space with the Murchison Widefield Array using passive radar, by detecting objects orders of magnitude smaller than previous work. This paper also demonstrates how the linear Doppler migration methods can be extended to higher order compensation to further increase potential processing intervals.

Just one of the Square Kilometre Array (SKA) Phase I science projects will produce data of the order of tens of terabytes per second. The SKA Phase I project will have stringent power constraints in order to limit the operational costs (Dewdney et al. 2013); it is a considerable challenge to manage, process and store such large datasets within these constraints. The current state-of-the-art astronomy data processing systems are designed to handle data approximately two to three orders of magnitude smaller than the SKA Phase I. To tackle this challenge, we have developed the Data-Activated Flow Graph Engine (DALiuGE), as part of the prototyping effort for the Science Data Processor Consortium of the SKA Phase I design. DALiuGE aims to provide a distributed data management platform and a scalable pipeline execution environment to support continuous, time and power bounded, data-intensive processing for producing SKA science-ready products. In this paper, we provide a brief overview of DALiuGE.

The correlator output of the SKA arrays will be of the order of 1 TB/s. That data rate will have to be processed by the Science Data Processor using dedicated HPC infrastructure in both Australia and South Africa. Radio astronomical processing in principle is thought to be highly data parallel, with little to no communication required between individual tasks. Together with the ever increasing number of cores (CPUs) and stream processors (GPUs) this led us to step back and think about the traditional pipeline and task driven approach on a more fundamental level. We have thus started to look into dataflow representations (Dennis & Misunas 1974) and data flow programming models (Davis 1978) as well as data flow languages (Johnston et al. 2004) and scheduling (Benoit et al. 2014). We have investigated a number of existing systems and prototyped some implementations using simplified, but real radio astronomy workflows. Despite the fact that many of these approaches are already focussing on data and dataflow as the most critical component, we still missed a rigorously data driven approach, where the data itself is essentially driving the whole process. In this talk we will present the new concept of DROP Computing (condensed data cloud), which is an integral part of the current SKA Data Layer architecture. In short a DROP is an abstract class, instances of which represent data (DataDrop), collections of DROPs (Container Drop), but also applications (ApplicationDrop, e.g. pipeline components). The rest are just details, which will be presented in the talk.

The Murchison Widefield Array (MWA) is a next-generation radio telescope, generating visibility data products continuously at about 400 MB/s. The entire MWA archive consists of dataflow and storage sub-systems distributed across three tiers.

The Murchison Wide Field Array (MWA) is being upgraded from 32 tiles to 128 tiles of 16 dual-polarization dipole antennas. In the course of this project the software and the data infrastructure are also undergoing a major overhaul in order to cope with a more continuous and remote operational model; and the substantial increase in data rate (400 MB/s from the correlator plus 160 MB/s from the real time imaging pipeline). During the course of 2012/13 the data collected by the MWA will be transported via a dedicated 40 Gbit WAN network link between the Murchison Radio Observatory (MRO) and Perth (700 km). However, this network will not be available for some time; and until then, the data will be transported using disk arrays instead. Once in Perth, the data will be ingested into a tape library. The archiving process itself consists of various steps executed either at the MRO site or in Perth, and makes use of a modied version of the archiving system from the Atacama Large Millimeter Array (ALMA). This includes the extraction of meta-data from the original raw data and the ingestion and generation of the appropriate data links for the MWA archive. The MWA correlator generates a collection of small les all belonging to the same observation. In order to optimise the network transfer and the storage of those les they are transparently packed into larger containers at the MRO site and subsequently handled as one big le. This paper describes the setup of the MWA archiving infrastructure.