Urban areas are likely locations for release of toxic material into the atmosphere, whether by accident or terrorist act. Both the Department of Energy, through the Chemical and Biological National Security Program, and the Department of Defense, through the Defense Threat Reduction Agency, are supporting simulation and experimental efforts to develop urban modeling capabilities. These developed tools would be used in response to the release of toxic material in populated urban centers. We are developing a capability to predict the detailed flow and dispersion in and around population centers, to address issues of response to release of toxic agents into the atmosphere. Due to the complexity of these problems and their great demand on computing power, the scientific community has not had the ability to address the urban problem previously. LLNL's unique combination of modeling capability and access to terascale computing resources allows us to address such problems. However, in regions with such small scale features and with heterogeneous building configurations and complex terrain, classical approaches with simplifying assumptions are no longer valid. Turbulence closure approximations that are employed in models with 5km resolution are inappropriate when the resolution is 3 orders of magnitude finer. Also, closure assumptions based onmore » vertical transfer are likely not appropriate when the nearest surface is a building face rather than the ground. The goal of the work described here is to advance the theoretical and empirical basis for transport and dispersion at urban and local scales. This would allow us to extend NARAC's capability for realistic dispersion prediction for smaller scales such as in urban areas. Our objectives within this goal and as part of the mid-year call for proposals are to: (1) test sub-grid scale (SGS) turbulence approximations in the high-resolution building-scale model FEM3MP, and determine sensitivity of dispersion predictions to these approximations; and (2) design procedures for information transfer from a regional scale model COAMPS to the FEM3MP model for prediction purposes, including parallel implementation. Circulation in the atmosphere is characterized by high Reynolds number, implying that many scales of motion exist; thus direct numerical simulation is not feasible and a Large Eddy Simulation (LES) approach is necessary. In the LES approach, the largest eddies are directly simulated but sub-grid scale motions are modeled. The size of the resolved eddies changes as the resolution of a host model changes, but often the SGS model remains unchanged. The resolved flow features and the turbulence intensity depend on that host model resolution. Empirical Monin-Obukhov similarity profiles for neutral conditions provide a good test for the sensitivity to host model resolution. We investigated resolution issues in the sub-grid scale modeling of turbulence and in the coupling and nesting of models from larger- to smaller-scale domains. These different but related issues are critical in accurately simulating circulation at small scales such as around buildings, street canyons and urban areas. Two numerical models were used in this study: a finite element model, FEM3MP, using a Smagorinsky eddy viscosity formulation for the SGS, and a mesoscale numerical weather prediction model, COAMPS. The coupling of COAMPS to FEM3MP is a one-way nesting, i.e. information passes only from COAMPS to FEM3MP. Time dependent profiles of velocity and turbulence kinetic energy from COAMPS are used to generate, and interpolate if necessary, to provide initial conditions and boundary conditions for FEM3MP. During the course of this project, we (1) developed protocols and procedures for coupling COAMPS to FEM3MP, (2) conducted mesh refinement studies, (3) tested the COAMPS-FEM3MP coupling for fidelity of the information transfer, and (4) compared the FEM3MP LES simulations with data from the Prairie Grass Field Experiment. Comparisons to that dataset included mean horizontal wind profiles, turbulence intensity, and concentration of released tracer as it disperses downwind from the source, as presented in the next section.« less

Three-dimensional high-resolution simulations (up to 8 billion zones) have been performed for a Richtmyer–Meshkov instability produced by passing a shock through a contact discontinuity with a two-scale initial perturbation. The setup approximates shock-tube experiments with a membrane pushed through a wire mesh. The simulation produces mixing-layer widths similar to those observed experimentally. Comparison of runs at various resolutions suggests a mixing transition from unstable to turbulent flow as the numerical Reynolds number is increased. At the highest resolutions, the spectrum exhibits a region of power-law decay, in which the spectral flux is approximately constant, suggestive of an inertial range, but with weaker wave number dependence than Kolmogorov scaling, about k−6/5. Analysis of structure functions at the end of the simulation indicates the persistence of structures with velocities largest in the stream-wise direction. Comparison of three-dimensional and two-dimensional runs illustrates the tendency toward forward cascade in three dimensions, versus inverse cascade in two dimensions. Comparison of the full simulation with a simulation of a single-scale perturbation indicates that the coupling of the disparate scales leads to destruction of the small-scale bubbles and spikes except near the spike growing from the large-scale perturbation. Finally, an analysis of the sub-grid-scale stresses in filtered data indicates significant correlation of the resultant forward and back transfer of energy with the determinant of the rate-of-strain tensor of the resolved scale flow. A possible relation between this trend and alignment of vorticity on small scales with the principal directions of strain on large scales is discussed. The observed correlation lends support to the use of sub-grid-scale models proportional to the determinant of the rate-of-strain tensor for large-eddy simulation.

The steadily increasing power of supercomputing systems is enabling very high resolution simulations of compressible, turbulent flows in the high Reynolds number limit, which is of interest in astrophysics as well as in several other fluid dynamical applications. This paper discusses two such simulations, using grids of up to 8 billion cells. In each type of flow, convergence in a statistical sense is observed as the mesh is refined. The behavior of the convergent sequences indicates how a subgrid-scale model of turbulence could improve the treatment of these flows by high-resolution Euler schemes like PPM. The best resolved case, a simulation of a Richtmyer-Meshkov mixing layer in a shock tube experiment, also points the way toward such a subgrid-scale model. Analysis of the results of that simulation indicates a proportionality relationship between the energy transfer rate from large to small motions and the determinant of the deviatoric symmetric strain as well as the divergence of the velocity for the large-scale field.

Understanding turbulence and mix in compressible flows is of fundamental importance to real-world applications such as chemical combustion and supernova evolution. The ability to run in three dimensions and at very high resolution is required for the simulation to accurately represent the interaction of the various length scales, and consequently, the reactivity of the intermixin species. Toward this end, we have carried out a very high resolution (over 8 billion zones) 3-D simulation of the Richtmyer-Meshkov instability and turbulent mixing on the IBM Sustained Stewardship TeraOp (SST) system, developed under the auspices of the Department of Energy (DOE) Accelerated Strategic Computing Initiative (ASCI) and located at Lawrence Livermore National Laboratory. We have also undertaken an even higher resolution proof-of-principle calculation (over 24 billion zones) on 5832 processors of the IBM system, which executed for over an hour at a sustained rate of 1.05 Tflop/s, as well as a short calculation with a modified algorithm that achieved a sustained rate of 1.18Tflop/s. The full production scientific simulation, using a further modified algorithm, ran for 27,000 timesteps in slightly over a week of wall time using 3840 processors of the IBM system, clockin a sustained throughput of roughly 0.6 teraflop per second (32-bit arithmetic). Nearly 300,000 graphics files comprising over three terabytes of data were produced and post-processed. The capability of running in 3-D at high resolution enabled us to get a more accurate and detailed picture of the fluid-flow structure - in particular, to simulate the development of fine scale structures from the interactions of long-and short-wavelength phenomena, to elucidate differences between two-dimensional and three-dimensional turbulence, to explore a conjecture regarding the transition from unstable flow to fully developed turbulence with increasing Reynolds number, and to ascertain convergence of the computed solution with respect to mesh resolution.

In 1993, members of our team collaborated with Silicon Graphics to perform the first full-scale demonstration of the computational power of the SMP cluster supercomputer architecture. That demonstration involved the simulation of homogeneous, compressible turbulence on a uniform grid of a billion cells, using our PPM gas dynamics code. This computation was embarrassingly parallel, the ideal test case, and it achieved only 4.9 Gflop/s performance, slightly over half that achievable by this application on the most expensive supercomputers of that day. After four to five solid days of computation, when the prototype machine had to be dismantled, the simulation was only about 20% completed. Nevertheless, this computation gave us important new insights into compressible turbulence and also into a powerful new mode of cost-effective, commercially sustainable supercomputing [S]. In the intervening 6 years, the SMP cluster architecture has become a fundamental strategy for several large supercomputer centers in the US, including the DOE's ASCI centers at Los Alamos National Laboratory and at the Lawrence Livermore National Laboratory and the NSF's center NCSA at the University of Illinois. This SMP cluster architecture now underlies product offerings at the high-end of performance from SGI, IBM, and HP, among others. Nevertheless, despite many successes, it is our opinion that the computational science community is only now beginning to exploit the full promise of these new computing platforms. In this paper, we will briefly discuss two key architectural issues, vector computing and the flat multiprocessor architecture, which continue to drive spirited discussions among computational scientists, and then we will describe the hierarchical shared memory programming paradigm that we feel is best suited to the creative use of SMP cluster systems. Finally, we will give examples of recent large-scale simulations carried out by our team on these kinds of systems and point toward the still more challenging work which we foresee in the near future.

The new generation of powerful DSM and SMP cluster computers enables simulations of fluid dynamics at sufficient resolution to compute the complex nonlinear interactions of small-scale turbulent motions with a large-scale driving flow. With a new programming model of hierarchical shared memory multitasking, it is possible to exploit these new systems without disrupting the flow of small and medium-sized jobs that makes their existence possible.

Three-dimensional high-resolution simulations are performed of the Richtmyer-Meshkov (RM) instability for a Mach 6 shock, and of the passage of a second shock from the same side through a developed RM instability. The second shock is found to rapidly smear fine structure and strongly enhance mixing. Studies of the interaction of moderately strong shocks with a pre-existing turbulent field indicate amplification of transverse vorticity and reduction of stream-wise vorticity, as well as the...

A comprehensive climate system model is under development at Lawrence Livermore National Laboratory. The basis for this model is a consistent coupling of multiple complex subsystem models, each describing a major component of the Earth's climate. Among these are general circulation models of the atmosphere and ocean, a dynamic and thermodynamic sea ice model, and models of the chemical processes occurring in the air, sea water, and near-surface land. The computational resources necessary to carry out simulations at adequate spatial resolutions for durations of climatic time scales exceed those currently available. Distributed memory massively parallel processing (MPP) computers promise to affordably scale to the computational rates required by directing large numbers of relatively inexpensive processors onto a single problem. We have developed a suite of routines designed to exploit current generation MPP architectures via domain and functional decomposition strategies. These message-passing techniques have been implemented in each of the component models and in their coupling interfaces. Production runs of the atmospheric and oceanic components performed on the National Environmental Supercomputing Center (NESC) Gray T3D are described.

Preliminary results of three-dimensional simulations of compressible Rayleigh-Taylor instability and turbulent mixing in an ideal gas using the piecewise-parabolic method (PPM) (with and without molecular dissipation terms) are presented. Simulations with spatial resolutions up to 512{sup 3} were performed. Two types of convergence studies are presented. Statistical analyses of the data are discussed, include: 1: spectra, and; 2) horizontally-averaged terms in the kinetic energy and onstrophy density evolution equations. The application of this statistical data to the development and testing of subgrid-scale models appropriate for compressible Rayleigh-Taylor instability-induced turbulent mixing is discussed.

Global warming, acid rain, ozone depletion, and biodiversity loss are some of the major climate-related issues presently being addressed by climate and environmental scientists. Because unexpected changes in the climate could have significant effect on our economy, it is vitally important to improve the scientific basis for understanding and predicting the earth`s climate. The impracticality of modeling the earth experimentally in the laboratory together with the fact that the model equations are highly nonlinear has created a unique and vital role for computer-based climate experiments. However, today`s computer models, when run at desired spatial and temporal resolution and physical complexity, severely overtax the capabilities of our most powerful computers. Parallel processing offers significant potential for attaining increased performance and making tractable simulations that cannot be performed today. The principal goals of this project have been to develop and demonstrate the capability to perform large-scale climate simulations on high-performance computing systems (using methodology that scales to the systems of tomorrow), and to carry out leading-edge scientific calculations using parallelized models. The demonstration platform for these studies has been the 256-processor Cray-T3D located at Lawrence Livermore National Laboratory. Our plan was to undertake an ambitious program in optimization, proof-of-principle and scientific study. These goals have been met. We are now regularly using massively parallel processors for scientific study of the ocean and atmosphere, and preliminary parallel coupled ocean/atmosphere calculations are being carried out as well. Furthermore, our work suggests that it should be possible to develop an advanced comprehensive climate system model with performance scalable to the teraflops range. 9 refs., 3 figs.

A new version of the UCLA atmospheric general circulation model suitable for massively parallel computer architectures has been developed. This paper presents the principles for the code's design and examines performance on a variety of distributed memory computers. A two dimensional domain decomposition strategy is used to achieve parallelism and is implemented by message passing. This parallel algorithm is shown to scale favorably as the number of processors is increased. In the fastest configuration, performance roughly equivalent to that of multitasking vector supercomputers is achieved.

The output of simulations (e.g., global circulation models) can run into terabytes. The computational cost as well as the cost of storing and retrieving model data can be quite high. Recently there have been some e orts to develop on-line visualization capabilities that can be used, for example, to monitor whether the model is behaving properly. There are however, many other uses for on-line data analysis including feature extraction, computational steering of the model, and controlled saving of model output (e.g., more frequent samples of state information under certain conditions). Each of these applications is a potential client of model output data. We present a software architecture which stresses modularity and exibility and supports a variety of clients. Some preliminary performance numbers are given from a prototype implementation. Real Time Data Mining, Management, and Visualization of GCM Output Edmond Mesrobian, Richard Muntz, and Eddie Shek Data Mining Laboratory Computer Science Department University of California, Los Angeles Los Angeles, CA 90024 Email: fedmond,muntz,shekg@cs.ucla.edu Carlos R. Mechoso, John D. Farrara, and Joesph A. Spahr Department of Atmospheric Sciences University of California, Los Angeles Los Angeles, CA 90024 Email: fmechoso,farrara,spahrg@atmos.ucla.edu Paul Stolorz Jet Propulsion Lab California Institute of Technology Pasadena, CA 91109 Email: stolorz@telerobotics.jpl.nasa.gov

We have developed a Climate System Modeling Framework (CSMF) for high-performance computing systems, designed to schedule and couple multiple physics simulation packages in a flexible and transportable manner. Some of the major packages in the CSMF include models of atmospheric and oceanic circulation and chemistry, land surface and sea ice processes, and trace gas biogeochemistry. Parallelism is achieved through both domain decomposition and process-level concurrency, with data transfer and synchronization accomplished through message-passing. Both machine transportability and architecture-dependent optimization are handled through libraries and conditional compile directives. Preliminary experiments with the CSMF have been executed on a number of high-performance platforms, including the Intel Paragon, the TMC CM-5 and the Meiko CS-2, and we are in the very early stages of optimization. Progress to date is presented.

We have developed a climate system modeling framework (CSMF) for high-performance systems, designed to schedule and couple multiple physics simulation packages in a flexible and transportable manner. Some of the major packages in the CSMF include models of atmospheric and oceanic circulation and chemistry, land surface and sea ice processes, and trace gas biogeochemistry. Parallelism is achieved through both domain decomposition and process-level concurrency, with data transfer and synchronization accomplished through message-passing. Both machine transportability and architecture-dependent optimization are handled through libraries and conditional compile directives. Preliminary experiments with the CSMF have been executed on a number of high-performance platforms, including the Intel Paragon, the TMC CM-5 and the Meiko CS-2, and we are in the very early stages of optimization. Progress to date (1994) is presented