We describe an extension of the Enzo code to enable fully coupled radiation hydrodynamical simulation of inhomogeneous reionization in large ∼(100 Mpc)3 cosmological volumes with thousands to millions of point sources. We solve all dynamical, radiative transfer, thermal, and ionization processes self-consistently on the same mesh, as opposed to a postprocessing approach which coarse-grains the radiative transfer. We do, however, employ a simple subgrid model for star formation which we calibrate to observations. The numerical method presented is a modification of an earlier method presented in Reynolds et al. differing principally in the operator splitting algorithm we use to advance the system of equations. Radiation transport is done in the gray flux-limited diffusion (FLD) approximation, which is solved by implicit time integration split off from the gas energy and ionization equations, which are solved separately. This results in a faster and more robust scheme for cosmological applications compared to the earlier method. The FLD equation is solved using the hypre optimally scalable geometric multigrid solver from LLNL. By treating the ionizing radiation as a grid field as opposed to rays, our method is scalable with respect to the number of ionizing sources, limited only by the parallel scaling properties of the radiation solver. We test the speed and accuracy of our approach on a number of standard verification and validation tests. We show by direct comparison with Enzo's adaptive ray tracing method Moray that the well-known inability of FLD to cast a shadow behind opaque clouds has a minor effect on the evolution of ionized volume and mass fractions in a reionization simulation validation test. We illustrate an application of our method to the problem of inhomogeneous reionization in a 80 Mpc comoving box resolved with 32003 Eulerian grid cells and dark matter particles.

This paper describes the open-source code Enzo, which uses block-structured adaptive mesh refinement to provide high spatial and temporal resolution for modeling astrophysical fluid flows. The code is Cartesian, can be run in 1, 2, and 3 dimensions, and supports a wide variety of physics including hydrodynamics, ideal and non-ideal magnetohydrodynamics, N-body dynamics (and, more broadly, self-gravity of fluids and particles), primordial gas chemistry, optically-thin radiative cooling of primordial and metal-enriched plasmas (as well as some optically-thick cooling models), radiation transport, cosmological expansion, and models for star formation and feedback in a cosmological context. In addition to explaining the algorithms implemented, we present solutions for a wide range of test problems, demonstrate the code's parallel performance, and discuss the Enzo collaboration's code development methodology.

The light from early galaxies had a dramatic impact on the gasses filling the universe. This video highlights the spatial structure of the light's effect, by comparing two simulations: one with a self-consistent radiation field (radiative), and one without (non-radiative), each with a very high dynamic range. Looking at the simulations side-by-side it's hard to see any difference. However, because the simulations have the same initial conditions, we can directly compare them, by looking at the relative difference of the density. The corallike blobs are regions where light has radiated out, heating the gas, and raising the pressure. The red regions show where the density is much higher in the radiative simulation, while the yellow regions are where the non-radiative has more density, showing where gravity was able to pull the filaments into tighter cylinders, without having to work against pressure from stellar heating. This is the first known visualization of this process, known as Jeans smoothing.

The submitted visualization represents work performed by the Enzo PRAC team lead by Brian O'Shea on the Blue Waters Early Science system. A relatively small test calculation was performed followed by several much larger AMR cosmological runs. The overall scientific goal was to understand how galaxies in the early Universe (the first billion years or so after the Big Bang) grow and evolve, in several statistically-dissimilar environments. Specifically, we looked at a region that is statistically over dense (substantially more galaxies than the average per volume), under dense (the opposite), of average mean density, and, finally, a region that will make a Milky Way-like galaxy at the present day. For each calculation, we used two separate formulations for our sub grid models of star formation and feedback. The simulation visualized in these movies represents the "average" calculations, which are the most statistically comparable to galaxies that have been observed with the Hubble Space Telescope and large ground-based telescopes. The visualization were done using the VisIt volume renderer. The analysis was completely performed on the Blue Waters Early science system. Volume renderings of density and temperature are presented. Each of these simulations was substantially larger than any previous Enzo AMR calculation ever run (as well as larger than any other AMR cosmological simulation ever done). By the end of the run, the calculations have several billion unique resolution elements on six levels of adaptive mesh.

This simulation uses a flux-limited diffusion solver to explore the radiation hydrodynamics of early galaxies, in particular, the ionizing radiation created by Population III stars. At the time of this rendering, the simulation has evolved to a redshift of 3.5. The simulation volume is 11.2 comoving megaparsecs, and has a uniform grid of 10243 cells, with over 1 billion dark matter and star particles. This animation shows a combined view of the baryon density, dark matter density, radiation energy and emissivity from this simulation. The multi-variate rendering is particularly useful because is shows both the baryonic matter ("normal") and dark matter, and the pressure and temperature variables are properties of only the baryonic matter. Visible in the gas density are "bubbles", or shells, created by the radiation feedback from young stars. Seeing the bubbles from feedback provides confirmation of the physics model implemented. Features such as these are difficult to identify algorithmically, but easily found when viewing the visualization

We use hydrodynamic cosmological simulations in a (600 Mpc)^3 volume to study the observability of baryon acoustic oscillations (BAO) in the intergalactic medium as probed by Lyman alpha forest (LAF) absorption. The large scale separation between the wavelength of the BAO mode (~150 Mpc) and the size of LAF absorbers (~100 kpc) makes this a numerically challenging problem. We report on several 2048^3 simulations of the LAF using the ENZO code. We adopt WMAP5 concordance cosmological parameters and power spectrum including BAO perturbations. 5000 synthetic HI absorption line spectra are generated randomly piercing the box face. We calculate the cross-correlation function between widely separated pairs. We detect the BAO signal at z=3 where theory predicts to moderate statistical significance.

In this paper we describe our massively parallel version of Enzo, a multiphysics, parallel, AMR application for simulating cosmological structure formation developed at UCSD and Columbia. We describe its physics, numerical algorithms, implementation, and performance on current terascale platforms. We also discuss our future plans and some of the challenges we face as we move to the petascale.

We present the first results from a new generation of simulated large sky coverage (~100 deg2) Sunyaev-Zel'dovich effect (SZE) cluster surveys using the cosmological adaptive mesh refinement N-body/hydro code Enzo. We have simulated a very large (5123 h-3 Mpc3) volume with unprecedented dynamic range. We have generated simulated light cones to match the resolution and sensitivity of current and future SZE instruments. Unlike many previous studies of this type, our simulation includes unbound gas, where an appreciable fraction of the baryons in the universe reside. We have found that cluster line-of-sight overlap may be a significant issue in upcoming single-dish SZE surveys. Smaller beam surveys (~1') have more than one massive cluster within a beam diameter 5%-10% of the time, and a larger beam experiment like Planck has multiple clusters per beam 60% of the time. We explore the contribution of unresolved halos and unbound gas to the SZE signature at the maximum decrement. We find that there is a contribution from gas outside clusters of ~16% per object, on average, for upcoming surveys. This adds both bias and scatter to the deduced value of the integrated SZE, increasing difficulty in accurately calibrating a cluster Y-M relationship. Finally, we find that in images where objects with M > 5 × 1013 M☉ have had their SZE signatures removed, roughly a third of the total SZE flux remains. This gas exists at least partially in the warm-hot intergalactic medium (WHIM) and will possibly be detectable with the upcoming generation of SZE surveys.

We study the ionization and thermal evolution of the intergalactic medium during the epoch of \heii reionization by means of radiation hydrodynamical cosmological simulations. We post-process baryonic density fields from a standard optically-thin IGM simulation with a homogeneous galaxy-dominated UV background (UVB) which reionizes \hi and \hei at z=6.5 but does not have any contribution to the ionization of \heii. Quasars with luminosities proportional to the mass of the host halos are then introduced as point sources throughout the 100 Mpc simulation volume consistent with the Pei luminosity function. We evolve the spatial distribution of the \heii ionizing radiation field using a time-implicit variable tensor Eddington factor radiative transfer scheme. Simultaneously, we also solve for the local ionization of \heii to \heii and the associated photoheating of the gas. We find that the percolation of the \heiii regions is essentially complete by z=2.5. When comparing to a self-consistent optically thin simulation we find that in optically thick calculation the gas temperature is higher by a factor of approximately 1.7 at the mean gas density level. We use 300 random lines of sight to compute at $\bar{z} = 2.5 \pm 0.1$ a mean \heii \lya line transmission of $\bar{F} = 0.304 \pm 0.002$. We compare the broadening width of the \hi and \heii \lya lines to the results from the self-consistent optically thin simulation and find a shift by approximately 1.25 km/s of the b-parameter distribution. Estimating the relative broadening width between the two forests shows that the \heii median b-parameter is about 0.8 times the median \hi broadening width. This implies that the \heii absorbers are physically extended consistent with conclusions from observed lines of sight.

The BlueGene/L (BG/L) supercomputer is designed to deliver new levels of application performance by providing a combination of good single-node computational performance and high scalability. To achieve good single-node performance, the BG/L design includes a special dual floating-point unit on each processor and the ability to use two processors per node. BG/L also includes both a torus and a tree network to achieve high scalability. We demonstrate how benchmarks and applications can take advantage of these architectural features to get the most out of BG/L.Achieving high sustained application performance has been one of the chief goals of the BG/L project [1]. The BG/L system was designed to provide a very high density of compute nodes with a modest power requirement, using a low-frequency embedded system-on-a-chip technology. The BG/L compute node is targeted to operate at 700 MHz. To obtain good performance at this relatively low frequency, each node needs to process multiple instructions per clock cycle. This can be achieved through two main strategies. First, one can make use of both processors in each BG/L node. Second, each processor has a dual floating-point unit with fused multiply-add instructions, which can perform four operations per cycle using special SIMD-like instructions. We investigate two strategies for leveraging the two processors in each node: coprocessor mode and virtual node mode.

A supercomputing hyper-grid spanning two continents was created to move a step towards interoperability of leading grids. A dedicated network connection was established between DEISA, the leading European supercomputing grid, and TeraGrid, the leading American supercomputing grid. Both grids have adopted the approach of establishing a common, high performance global file system, the wide-area version of IBM's GPFS. Teragrid's approach is based on a single site server solution under Linux, hosted by San Diego Supercomputer Centre, DEISA's approach is a multi-site server solution, with currently servers in France, Germany and Italy. These two Grid-internal global file systems were interconnected over a dedicated, trusted network connection. During the Supercomputing Conference 2005, Grand Challenge applications were carried out both within DEISA and within Teragrid, and results were written transparently to the combined global file system with physically distributed locations of the involved disk systems. Simulations were carried out in Europe and in America, and results were directly written to the respective remote continent, accessible for all participating scientists in both continents, and were then directly further processed for visualization in a third location, the SC05 exhibition hall in Seattle. Grand Challenge applications used for the demo included a Protein Structure Prediction and a Cosmological Simulation carried out at San Diego Supercomputer Center (SDSC), US (www.sdsc.edu) and a Gyrokinetic Turbulence Simulation and also a Cosmological Simulation carried out at Garching Computing Centre of the Max Planck Society (RZG), Germany (www.rzg.mpg.de).

In this paper we introduce Enzo, a 3D MPI-parallel Eulerian block-structured adaptive mesh refinement cosmology code. Enzo is designed to simulate cosmological structure formation, but can also be used to simulate a wide range of astrophysical situations. Enzo solves dark matter N-body dynamics using the particle-mesh technique. The Poisson equation is solved using a combination of fast fourier transform (on a periodic root grid) and multigrid techniques (on non-periodic subgrids). Euler’s equations of hydrodynamics are solved using a modified version of the piecewise parabolic method. Several additional physics packages are implemented in the code, including several varieties of radiative cooling, a metagalactic ultraviolet background, and prescriptions for star formation and feedback. We also show results illustrating properties of the adaptive mesh portion of the code. Information on profiling and optimizing the performance of the code can be found in the contribution by James Bordner in this volume.

We describe an extension of the Enzo code to enable the direct numerical simulation of inhomogeneous reionization in large cosmological volumes. By direct we mean all dynamical, radiative, and chemical properties are solved self-consistently on the same mesh, as opposed to a postprocessing approach which coarse-grains the radiative transfer. We do, however, employ a simple subgrid model for star formation, which we calibrate to observations. The numerical method presented is a modication of an earlier method presented in Reynolds et al. Radiation transport is done in the grey ux-limited diusion (FLD) approximation, which is solved by implicit time integration split o from the gas energy and ionization equations, which are solved separately. This results in a faster and more robust scheme for cosmological applications compared to the earlier method. The FLD equation is solved using the hypre optimally scalable geometric multigrid solver from LLNL. By treating the ionizing radiation as a grid eld as opposed to rays, our method is scalable with respect to the number of ionizing sources, limited only by the parallel scaling properties of the radiation solver. We test the speed and accuracy of our approach on a number of standard verication and validation tests. We show that the well-known inability of FLD to cast a shadow behind opaque clouds has little eect on the photoevaporation timescale of the cloud, or the evolution of ionized volume fraction in a reionization simulation validation test. We illustrate its application to the problem of inhomogeneous reionization in a 20 Mpc comoving box resolved with 800 3 Eulerian grid cells and dark matter particles.