SciDAC Review - SCIDAC-2: SciDAC-2: Advancing Science through Large-Scale Modeling and Simulation

SCIDAC-2

SciDAC-2: Advancing Science through Large-Scale Modeling and Simulation

The U.S. Department of Energy (DOE) Scientific Discovery through Advanced Computing (SciDAC) initiative—now in its seventh year—has become one of the nation's leading interdisciplinary research programs.

The SciDAC-2 program, begun in 2006, seeks to bring scientific computing to the petascale. Now in its third year, it boasts numerous successes in computational science—including beam dynamics, computational statistical mechanics, multidimensional simulation of core-collapse supernovae, petascale atmospheric models, simulation of clean and efficient combustion of alternative fuels, metagenomic sequence data analysis, first-principles molecular dynamics, and quantum chromodynamics. Essential to these successes have been dramatic advances in applied mathematics and computer science: new eigensolvers, adaptive mesh tools, embedded boundary grid generation, high-performance indexing and search mechanisms, and new visualization techniques capable of operating at extreme scale.

Many of the recent SciDAC-2 accomplishments were presented at the SciDAC 2008 Conference, held in Seattle, Washington, in July. And the message was clear: long-term partnerships have been forged, exciting science is being done, and high-performance simulation on leadership-class computers is keeping the United States at the forefront of science and technology (sidebar "Ten Most Significant Science Accomplishments" p10).

Science Applications and Science Application Partnerships
A major source of acceleration in simulation-based science under the SciDAC-2 initiative has been multidisciplinary studies among application domains, computer science, and applied mathematics. Investigations are under way in SciDAC Science Application projects in six areas—physics, climate, groundwater, fusion, life sciences, and materials science and chemistry—and already significant accomplishments have been achieved. Highlighted here are a few of the many achievements that were presented recently at the SciDAC 2008 Conference.

Physics
One of the most challenging areas of physics is the study of fundamental forces and elementary particles. Astrophysicist Dr. Stan Woosley and his team have been conducting numerical simulations of the explosion of Type Ia supernovae. Such explosions occur after the star has expanded significantly and a transition from subsonic nuclear burning to a more rapid form of burning, or detonation, occurs. Using both a three-dimensional low Mach number model and a one-dimensional linear eddy model, the team has been exploring the structure of turbulent flames deep within a white dwarf. Their results suggest that as the flame moves outward from the center of the star, the density decreases. Figure 1 shows instantaneous vertical slices of fuel consumption; as the flame burns in increasingly larger integral length scales, the relative size of the flame structure decreases, and burning occurs in distributed pockets. "But just decreasing the density or increasing the turbulence doesn't do the trick," says Dr. Woosley. "To get supersonic burning requires not only that turbulent speeds be no slower than about 10% sonic, but also that the region that burns faster than sound must be large enough to initiate a self-sustaining detonation." Dr. Woosley adds "Our calculations show that both these conditions may be satisfied at 10⁷g cm^-3."

Long-term partnerships have been forged, exciting science is being done, and high-performance simulation on leadership-class computers is keeping the United States at the forefront of science and technology.

ANL

Figure 1. Instantaneous vertical slices through a flame in the distributed burning regime. The left-hand panel is fuel consumption rate (red denotes intense burning), and the right-hand panel is temperature (red is hot).

Another team of scientists, led by Dr. Sanjiva Lele, is developing improved computational methods for modeling complex shock/turbulence interaction problems. Such problems are of prime importance in applications ranging from supernovae to the next generation of jet propulsion. The researchers have developed a solution-adaptive code, called Hybrid, that treats shock waves and turbulence with numerics suited to each phenomenon, which leads to dramatic improvements in the fidelity of the results. The code is routinely used for production runs on up to 4,096 processors on the Franklin machine at the National Energy Research Scientific Computing (NERSC) Center. The interaction with the shock changes the turbulence (figure 2, p10). The simulations show that the smallest turbulent eddies return to small-scale isotropy some distance behind the shock, whereas the largest eddies carry the shock-induced anisotropy for a long time; whether they eventually return to isotropy is an open question that requires simulations at even larger scale to resolve. Furthermore, the simulations also show that the turbulence forces the shock to exhibit an increasingly broad range of instantaneous structures as the turbulent Mach number is increased. Whether there exists a well-defined threshold for turbulence-induced structural change in the shock, and whether it can be predicted or explained by theory, is another open question in need of further research.

ANL

Figure 2. Snapshot of shock/turbulence interaction. Left: turbulence is amplified and modified as it flows through a shock wave. Turbulent eddies flowing from left to right are colored by vorticity magnitude; both the number and strength of eddies are increased behind the shock. Right: turbulent pockets of high (red regions) and low (blue regions) velocity induce structural changes to the shock wave.

Climate
With the focus of the world on climate change, researchers have been developing models with increasingly higher resolution and ever-greater integration of critical factors such as sea ice and clouds. Key to these modeling efforts has been the dramatic advance in computer power. This power has also enabled much longer simulations—from months to centuries. And over the next few decades, computers are expected to speed up by a factor of a million. One might well ask, as Dr. David Randall has, "What are we going to do with that next million?" His answer: "Run global cloud-resolving models," or GCRMs, that can be used for both numerical weather prediction and climate simulation. To this end, Dr. Randall and his team are developing a new global nonhydrostatic dynamical core based on a geodesic grid and suitable for use with grid spacings ranging from meters to hundreds of kilometers (figure 3). The geodesic grid has the advantage that all grid cells on the sphere are very nearly the same size, with the largest cells only about 5% larger in area than the smallest, thereby avoiding the problem of computational stability for advection. Moreover, contrary to popular belief, the geodesic grid does permit researchers to construct schemes of arbitrary higher-order accuracy; indeed, Dr. Randall's group has already used third-order-accurate finite-difference schemes. Components of the GCRM are being tested on the Cray XT4 at NERSC, and a high-resolution kernel of the model has exhibited good scaling performance on 10,000 processors.

ANL

Figure 3. An "exploded" picture of a low-resolution geodesic grid. The grid can be divided into ten logically rectangular panels, as shown here. Memory addressing can be organized using these panels.

Continued improvement of climate models also requires evaluation of their parameterizations, for instance for processes driven by subgrid features such as soil characteristics, vegetation, and land use. Dr. Rao Kotamarthi and his team of researchers at Argonne National Laboratory (ANL) and the University of Chicago are exploring the use of "data ensembles" generated through multiple runs to interpolate sparsely located surface measurements into a uniform spatial grid, thereby providing estimates of mean values over the entire domain containing the measurement sites. The method has been used successfully for interpolating surface sensible heat flux data from 14 sites within the DOE Atmospheric Radiation Measurement (ARM) program. The researchers next plan to address other measured parameters, many of which require at least a 10 year time series—some at one-minute intervals. The computational challenge is enormous: for example, at the 1 km resolution, there are approximately 100,000 interpolated locations and 99 simulations of the full time series at each location. To meet this challenge, Kotamarthi and his colleagues plan to use the Common Component Architecture—a component-based approach, supported by the DOE, in which units of software are encapsulated as components that interact with other components through well-defined interfaces.

Groundwater Modeling
One of the most challenging problems in environmental remediation involves hazardous materials that have leached into the subsurface and may be more widely dispersed by groundwater to sensitive water resource areas such as rivers and lakes. As part of the SciDAC groundwater science application area, researchers are developing new techniques to simulate radionuclide transport, in particular uranium, at the DOE Hanford 300 Area in the state of Washington. The task is complicated by the fact that the Hanford Unit is highly permeable and the groundwater has the potential of flowing rapidly with very small pressure gradients in the aquifer. This situation is further aggravated by the rapid fluctuations in the Columbia River, which produce changes, not only in magnitude but also in the flow direction. Uranium is leaching very slowly from the Hanford sediment governed by diffusive mass transfer, at levels that exceed the EPA maximum permissible concentration, prolonging its presence. Dr. Peter C. Lichtner is leading a multi-institutional SciDAC team whose members are developing a multiphase, multicomponent code called PFLOTRAN to simulate the variably saturated groundwater flow and reactive transport of uranium at the site. PFLOTRAN is based on a domain decomposition approach in which the computational problem is divided into subdomains, with one domain assigned to each processor. The Argonne-developed toolkit PETSc is used as a parallel framework for solvers and message passing within PFLOTRAN. According to Dr. Lichtner, "PETSc hides the communication from the user, thus allowing the application scientist to focus on the science (physics and chemistry in this case) rather than worry about the solvers and preconditioners needed." PFLOTRAN has already been run on a one-billion-node problem—an important proof of concept for petascale computing—and has been demonstrated to scale to 27,580 processor cores on the Jaguar XT3 Cray at Oak Ridge National Laboratory (ORNL).

Fusion Energy
Fusion has the potential to provide a long-term, environmentally acceptable source of energy for the future. To make this potential a reality, scientists are developing advanced methods for simulating fusion systems on terascale and petascale computers.

One approach being explored by the SciDAC project FACETS—Framework Application for Core-Edge Transport Simulations—has the goal of providing whole-tokamak modeling through coupling separate components for each of the core region, edge region, and wall. To achieve this goal, FACETS has organized interdisciplinary teams for each of the core, edge, and wall components; domain scientists provide the model and basic component, applied mathematicians and computer scientists provide algorithmic improvements, interlanguage tools, and performance measurement techniques. All groups work together to define the necessary interfaces that can represent the physics while being sufficiently abstract for flexible composition.

Midway through its second year, FACETS has already achieved core-edge coupled simulations through development of a multicomponent framework that allows separate components to run on different processors. The matching grids from the separate components are shown in figure 4 along with a pseudocolor plot of the ion temperature from both the core and edge. A significant accomplishment was the design of a new parallel core solver, which enables parallelization of the flux calculations. The solver achieves a factor of 30 speedup on 128 processors with a 128 cell grid, a step on the way to enabling researchers to fully exploit leadership computers. By improving the parallelism and algorithms, the researchers hope to simulate "full 1,000 second experiments on the International Thermonuclear Experimental Reactor (ITER) in less than an hour," says Dr. J. R. Cary. Other accomplishments include parallelization and componentization of the existing edge-plasma modeling application, UEDGE, and the implementation of a wall model, a 1D multiscale, multispecies code for particle and heat transport inside plasma-facing components.

ANL

Figure 4. Grids from the core and edge components in the coupled core-edge simulation are seen to match as desired, with the nodes from the edge (2D) mesh in the center of the last flux surface zone of the core, which is effectively running on a 1D mesh. The colors on the plot represent the temperature of the ions, which ranges from 0.8 eV at the wall to 2,100 eV in the central plasma core.

FACETS also involves interactions with a number of other SciDAC projects. For example, it includes an embedded Scientific Application Partnership whose goal is to develop a strategy to use local and global turbulence simulations in steady-state transport calculations. The researchers to date have developed a prototype transport driver that can run multiple instances of GYRO, a fundamental physics gyrokinetic code. Initial results of a 12 hour simulation on 1,284 cores of the Cray XT4 at ORNL were promising, indicating that the simulation can easily scale to very high numbers of cores. FACETS also is collaborating with the Simulation of Wave Interactions with MHD (SWIM) SciDAC project, through exchange of turbulent transport flux computational modules, which were developed under the Framework for Modernization and Componentization of Fusion Modules SBIR project.

Fusion has the potential to provide a long-term, environmentally acceptable source of energy for the future. To make this potential a reality, scientists are developing advanced methods for simulating fusion systems on terascale and petascale computers.

Life Sciences
Life sciences efforts in DOE often involve developing new methods for modeling complex biological systems, ranging from metabolic pathways to individual cells and, ultimately, interacting organisms and ecosystems. This wide range means scientists must deal with time scales from microseconds to thousands of years.

An exciting new SciDAC application is bringing together experimental high-throughput methods, large-scale systems biology models, and high-performance simulation tools to address the challenge of an in silico cell. Researchers have begun developing a high-performance software toolkit, called HiPer SBTK, to simulate, fit to quantitative high-throughput data, and optimize metabolite concentrations and fluxes within flexible, user-defined constraints. Their goal is a detailed enzymological kinetic modeling of the metabolism that captures all reliable experimental data for the green alga called Chlamydomonas reinhardtii.

Dr. Christopher Chang, an investigator on the project (which includes the National Renewable Energy Laboratory in Golden, Colorado, the Colorado School of Mines, and Stanford University under the direction of Dr. Michael Seibert), emphasizes the complexity of the problem. "In moving from desktop-scale simulations of a small set of biochemical reactions to genome-scale simulations in the high-performance computing arena, a paradigm shift must occur in the way we think of biological models," he says. "A complete representation of metabolism for a single organism implies detailed model networks with thousands of nodes and edges. Capturing quantitative responses from a thermodynamic and chemical kinetic basis, and exploring this extremely high dimensional space for purposes of discovery and engineering, entails moving from single-investigator desktop simulations to multi-investigator efforts, with persistent large reference models and datasets mined and analyzed by the computational biochemistry community."

Illustration: A. Tovey Source: ANL

Figure 5. The software stack, dependencies, and workflow for the SciDAC-sponsored software package HiPer SBTK, the High-Performance Systems Biology Toolkit. The software includes SBML-to-C model translation, creation of a model-specific library for evaluating various quantities, and a set of driver programs which when linked to the model-specific library create codes for HPC systems biology simulations.

To meet this challenge, Dr. Chang and his colleagues are using standardized languages such as the Systems Biology Markup Language (SBML), to enhance the utility of the simulation software for the general biological community. The researchers have developed a stack of software that includes translation of the SBML model and job parameters to a compiled language, C, for performance; simulation of the resulting ordinary differential equation system; characterization and visualization through parameter sampling; fitting arbitrary sets of kinetic parameters to experimental data; and optimization by a gradient-based local optimizer based on TAO and a scatter search-based global optimizer (figure 5).

Materials Science and Chemistry
The analysis and optimization of many complex processes, such as corrosion, require expansion of current modeling capabilities to molecules interacting with extended structures such as clusters or surfaces. In partnership with the National Nuclear Security Administration, DOE is supporting research to determine complex material properties. Efforts include computational nanophotonics, molecular computing devices, quantum simulations of materials and nanostructures, and multiscale simulations of strongly correlated materials.

One new effort under the SciDAC-2 project focuses on cracking under stress. While the subject title may strike one as humorous, stress corrosion cracking (SCC) in fact presents a serious danger to the safe operation of structural systems exposed to chemically aggressive environments. To prevent SCC requires understanding the atomistic mechanisms that influence the initiation, dynamics, and growth rates of such corrosive cracking. To this end, researchers led by Dr. Priya Vashishta and colleagues Rajiv Kalia and Aiichiro Nakano at the University of Southern California and partners at Los Alamos and Lawrence Livermore national laboratories are developing a scalable parallel and distributed computation framework of methods, algorithms, and software tools for petascale simulations of SCC. The researchers have demonstrated multimillion- and billion-atom molecular dynamics simulations on 196,608 processors of the Blue Gene/L with parallel efficiency as high as 0.99.

The researchers have demonstrated multimillion- and billion-atom molecular dynamics simulations on 196,608 processors of the Blue Gene/L with parallel efficiency as high as 0.99.

A recent achievement has been the discovery of a novel mechanism involving strain-enhanced defect transport in amorphous silica (figure 6). Molecular dynamics simulations reveal that shear-induced void deformation, damage, and flow in silica glass all have the same underlying mechanism involving Si-O bond breaking and the migration of threefold coordinated silicon and nonbridging oxygen defects on -Si-O-Si-O- rings. A startling observation is that—despite enormous differences in spatio-temporal scales—the shape changes and fragmentation of voids in silica glass are remarkably similar to the deformation and breakup of macroscopic inviscid drops at high shear rates.

ANL

Figure 6. Defects responsible for shear deformation and flow in silica glass. In the unconstrained case, each silicon atom is connected to four oxygen atoms in the form of a SiO4 tetrahedron. (a) (5% strain) The magenta atom is a bridging atom, and the blue atom is a threefold coordinated silicon atom in a nine-member ring. (b) (8% strain) The magenta oxygen atom becomes nonbridging and the blue silicon atom remains uncoordinated. (c) (10% strain) The green silicon and magenta oxygen atoms become fully coordinated by bonding with each other.

Centers for Enabling Technologies
Coordinating with the SciDAC Scientific Applications are the SciDAC Centers for Enabling Technologies (CET). The purpose of these CETs is to address the mathematical and computing systems software issues involved in creating a high-performance, scalable scientific computing infrastructure that will enable the effective use of terascale and petascale systems by SciDAC applications. CET projects include development of new algorithms that scale to parallel systems having hundreds of thousands of processors; methodology for achieving portability and interoperability of complex high-performance scientific software packages; operating systems and runtime tools and support for application execution performance and system management; and data management and visualization tools for ultrascale scientific datasets. Here we highlight recent accomplishments of a few of the nine CETs. See also the sidebar, "SciDAC Scientific Data Management Project Receives R&D 100 Award."

Applied Partial Differential Equations Center
Researchers in the Applied Partial Differential Equations Center (APDEC) are developing simulation tools for solving multiscale, multiphysics problems. Structured grid methods based on the finite-volume approach offer significant advantages for problems with regular geometries, including predictable memory access, high accuracy, and multiresolution features. To provide similar capabilities to problems with complex geometries, APDEC computer scientists are investigating a new approach—the embedded boundary (EB) method—in which domains with irregular boundaries are represented on a rectangular grid by specifying the intersection of each grid cell with the region on one or the other side of the boundary. The result is a natural conservative discretization of the partial differential equation on either side of the boundary. The researchers have also developed robust hyperbolic and linear elliptic/parabolic solvers to be used with the EB method.

The purpose of the CETs is to address the mathematical and computing systems software issues involved in creating a high-performance, scalable scientific computing infrastructure that will enable the effective use of terascale and petascale systems by SciDAC applications.

Figure 7 shows an inviscid gas dynamics calculation solved by using the explicit EB method. This application was motivated by the problem of gas jet formation in novel laser wakefield accelerator designs—a problem of particular interest to the DOE.

ANL

Figure 7. Compressible gas dynamics flow from a high-pressure reservoir into a vacuum through a nozzle. The image was made with the VisIT visualization package.

Interoperable Technologies for Advanced Petascale Simulations
The Center for Interoperable Technologies for Advanced Petascale Simulations (ITAPS) is developing interoperable and interchangeable mesh, geometry, and field manipulation tools for science applications ranging from accelerator design and fusion energy science to groundwater reactive transport modeling and simulation. One exciting project during the past year has involved simulation of pellet ablation for tokamak fueling. Since the injection of small frozen deuterium-tritium pellets is considered the most likely refueling technique for ITER, scientists are interested in evaluating the pellet ablation rate of this technique. Conventional analytical and numerical ablation models lack important details of physics processes in the vicinity of the pellet and in the ablation channel and suffer from low accuracy as a result of the extreme change of thermodynamics states on short length scales. To overcome these limitations, ITAPS researchers have developed numerical algorithms and parallel software based on a front tracking method. In recent simulations, the pellet ablation rate and lifetime in magnetic fields were systematically studied for the first time and compared with theory and experimental databases. These simulations revealed several new features of the pellet ablation. For all but one example, the ablation rate appears to be strongly dependent on plasma edge (pedestal) properties and on the rise time of the heat flux seen by the pellet—an important consideration in future reactor designs because lower ablation rates can mean higher fueling efficiency.

ITAPS researchers have also used a new model for the potential distribution in the ablation channel. Pellet injection experiments in the late 1990s recorded striation instabilities. The ITAPS simulations suggest that these instabilities may be caused by the supersonic rotation of the ablated gas cloud, which widens the ablation channel and redistributes the ablated gas (figure 8).

ANL

Figure 8. Isosurfaces of rotational Mach number in the steady-state ablation cloud.

Center for Enabling Distributed Petascale Science
Researchers in the Center for Enabling Distributed Petascale Science (CEDPS) work with DOE application science communities to provide services required to move datasets where they are needed, to enable analysis near their data, and to detect and recover from failures in the distributed environment. One component of the CEDPS project involves data placement, where efforts focus on GridFTP enhancements. A surprising challenge here has been improving the transfer of large groups of small files. In advanced high-speed networks, datasets less than 100 MB are too small for underlying protocols such as TCP to use the maximum capacity of the network. To enhance the performance, CEDPS researchers added a feature called "concurrency" to the GridFTP client. This feature has already helped DOE users of the Advanced Photon Facility at ANL to transfer more than a terabyte of data to Australia at a rate 30 times faster than traditional data transfer mechanisms (figure 9).

Illustration: A. Tovey Source: ANL

Figure 9. DOE users of the Advanced Photon Facility at Argonne National Laboratory used GridFTP enhanced with "concurrency" to transfer more than a terabyte of data to Australia at a rate 30 times faster than traditional data transfer mechanisms.

Another component of the CEDPS project focuses on scalable services. A major accomplishment here has been the development of a Java-based tool called Grid Remote Application Virtualization Interfaces (gRAVI, pronounced "gravy"). The tool addresses service support features such as discovery, invocation, and security. Moreover, it allows researchers to wrap applications as Grid services without writing a single line of code. Participants in the caBIG initiative, funded by the National Cancer Institute, have used gRAVI to create, register, discover, and invoke analytical routines as Grid services. The toolset has also been used by researchers at the Advanced Photon Source at ANL to shorten development time and provide security for computational tasks. According to the developer, Ravi Madduri, "authorized researchers can invoke these services and compose multiple services into workflows for individual applications."

Earth System Grid Center for Enabling Technologies
Increased computing power and increasingly comprehensive climate models have resulted in a dramatic increase in data output describing the Earth system model. Indeed, over the past decade, model output has increased from megabytes to terabytes, and climatologists expect to generate hundreds of petabytes of simulation data within the next five years. Using these data presents enormous challenges. For example, the data will probably need to be stored in a few specific archival sites; data users will therefore need assistance in locating data of interest and then in performing data reduction, analysis, and visualization, presumably on the server side to reduce the volume of data sent over the network and stored at the user's location. Users will also need to have confidence in the origins and integrity of the data and the associated metadata. The Earth System Grid Center for Enabling Technologies (ESG-CET) was established to address such challenges.

Over the past decade, model output has increased from megabytes to terabytes, and climatologists expect to generate hundreds of petabytes of simulation data within the next five years. Using these data presents enormous challenges.

While the ESG-CET builds on the first and highly successful Earth System Grid SciDAC project, that effort resulted in architecture designed to support a small set of well-known U.S. archive sites operating ESG nodes. For the current project, an entirely new architecture is needed that provides for a larger number of distributed sites throughout the world, requiring multiple portals and data delivery mechanisms, a variety of means for user access, and reliable mechanisms to handle system and network failures. The new framework is based on a three-tiered approach that includes metadata services for search and discovery, data gateways that act as brokers handling data requests to serve specific user communities, and ESG nodes with actual data holdings and metadata accessing services. A distributed testbed based on this new architecture is scheduled to be in place by late 2008 to early 2009.

The Center is also working closely with the CEDPS SciDAC project to develop a basic analysis service framework. The work requires development of techniques to apply existing analysis procedures, written in scripting language systems, to large (many-terabyte) datasets; tools to manage the movement of data between archival storage and disk, since the datasets in storage may be too large to move onto disk in their entirety at one time; and mechanisms to enable users to define and upload their own analysis procedures, without compromising the security or reliability of the server on which those procedures run. Figure 10 shows an example of how these capabilities can be used.

Illustration: A. Tovey Source: ANL

Figure 10. Example of a user accessing and analyzing data with the new ESG-CET architecture.

SciDAC Institutes
The SciDAC Institutes were established as a new component of the SciDAC program. Patterned after other institutes such as the Aspen Institute of Physics, four SciDAC Institutes have been started: petascale data storage (Carnegie Mellon University), performance engineering research (University of Southern California), ultrascale visualization (University of California-Davis), and combinatorial scientific computing and petascale simulations (Old Dominion University). One of the goals of the SciDAC Institutes is to foster the next generation of computational scientists through hands-on workshops, tutorials, and summer schools. The Institutes also assist the SciDAC Scientific Applications teams, helping them to use petascale computing more effectively.

SciDAC Institute for Ultrascale Visualization
How are scientists going to extract meaning from massive datasets with hundreds of terabytes or more? Parallel visualization is an obvious answer, but it is still in its infancy.

The goal of the SciDAC Institute for Ultrascale Visualization is to produce an open-source suite of visualization tools that is portable across platforms to enable scientific discovery at this scale. To achieve this goal, the Institute sponsors a series of outreach activities designed to bring together experts from visualization, high-performance computing, and science application areas on specific topics. The Institute also works with industry, recommending ways in which hardware and software can better support large-scale visualization. SciDAC science applications teams serve as early adopters of the new visualization technology produced by the Institute.

One of the goals of the SciDAC Institutes is to foster the next generation of computational scientists through hands-on workshops, tutorials, and summer schools. The Institutes also assist the SciDAC Scientific Applications teams, helping them to use petascale computing more effectively.

One example of the Institute's research is parallel volume rendering on leadership-class machines such as the Blue Gene/P at ANL. Dr. Robert Ross, an Institute PI, says, "At first glance, using a leadership computing resource for analysis might not make much sense." After all, modern GPUs enable peak rendering rates up to 100 times faster than the software rendering rate of a low-power general-purpose processor like BG/P's PowerPC. But today's datasets—gigabytes or larger per time step—demand parallel processing for visualization. "In this scenario," says Dr. Ross, "the bottleneck is not rendering time but data movement between storage and processors."

Institute members are exploring various approaches to optimizing access to storage. In one application, they manipulate the color and transparency of rendered data in their volume rendering code in order to help astrophysicists understand the origin of static accretion shock instability in supernova core collapse (figure 11). In another application, instead of directly reading the simulation data, the Institute members have used aggressive preprocessing prior to analysis. In this approach, data blocks are clustered based on visibility feature vectors. By reorganizing these data blocks according to calculated access patterns, the number of necessary noncontiguous I/O operations can be greatly reduced. This approach has proved successful on a 512 x 512 x 1728 dataset called the "Visible Woman." Results show up to 50% reduction in I/O time; moreover, the variances of I/O time under different view directions were far less.

Image: ANL Data: J. Blondin, North Carolina State University

Figure 11. Image resulting from volume rendering of astrophysics dataset on IBM Blue Gene/P. By manipulating the transparency of rendered data with a volume rendering code, the scientists can quickly see different combinations of variables and isolate features of interest.

CSCAPES
The Institute for Combinatorial Scientific Computing and Petascale Simulations (CSCAPES) focuses on four key research areas: load balancing and parallelization toolkits for petascale computers, automatic differentiation capabilities for complex SciDAC applications, basic ingredients in linear solver technology, and runtime data and iteration reordering to improve performance in irregular computation. The unifying element for these four areas is the formulation and solution of the underlying combinatorial problems, often expressed through graphs or hypergraphs. Recent accomplishments include the following:

Dynamic Hypergraph-Based Repartitioning: CSCAPES members have developed a new hypergraph-based repartitioning method and applied it to several applifcation areas, including a shock-physics simulation with adaptive mesh refinement (figure 12). The new method produces lower total communication volume (communication associated with both the application and data migration) than commonly used methods.
Parallel Coloring: CSCAPES researchers have designed novel sequential algorithms for various coloring problems and incorporated them in the automatic differentiation software, ADOL-C. Experiments using ADOL-C on Jacobian computation in a simulated moving bed process—a method used in chromatographic separation in chemical engineering—have shown that the use of the coloring techniques in large-scale problems reduces overall runtime by several orders of magnitude. Parallel versions of several of the coloring algorithms have also been developed using a common framework. The parallel implementations are being deployed through the Zoltan software toolkit.
Automatic Differentiation: CSCAPE researchers have identified a way to use automatic differentiation techniques to reduce the cost of computing the gradient of a function arising in computational chemistry (from Dr. Ron Shepard at Argonne) from O(n5) to O(n4), where n is the number of electrons. This reduction will enable larger sizes of the problem to be solved.

ANL

Figure 12. Shock physics simulation performed at Sandia National Laboratories using adaptive mesh refinement: the initial mesh (left) and the mesh after 108 time steps (right). Zoltan was used for dynamic load balancing.

The Next Step: Computing at the Exascale
The SciDAC-2 initiative is scheduled to continue until 2011. And we expect the number of SciDAC successes to increase during that time. But what comes after SciDAC-2?

"We envision that there will be a SciDAC-X, working at the extreme scale. SciDAC teams will carry out science in the areas that will have a great societal impact, such as alternative fuels and transportation, climate, fusion science, high-energy physics, advanced fuel cycles, carbon management, and groundwater."

Dr. Michael Strayer
DOE Office of Science

According to Dr. Michael Strayer, Associate Director of Science for Advanced Scientific Computing Research at DOE, the Office of Science is putting together an ambitious plan called the E3 Initiative. "We envision that there will be a SciDAC-X, working at the extreme scale," says Dr. Strayer. "SciDAC teams will carry out science in the areas that will have a great societal impact, such as alternative fuels and transportation, climate, fusion science, high-energy physics, advanced fuel cycles, carbon management, and groundwater."

It will certainly be challenging, but judging from the way computer scientists, applied mathematicians, and computational scientists are forging ahead to petascale, we can anticipate an exciting new era in advanced computing.

Contributors: Dr. Gail Pieper (ANL) prepared this article based on the presentations at the 2008 SciDAC Conference and the following people: Dr. J. R. Cary (Tech-X Corporation), Dr. Christopher Chang (National Renewable Energy Laboratory), Ravi Kettinuthu (ANL), Dr. Rao Kotamarthi (ANL), Dr. SanJiva Lele (Stanford University), Dr. Peter Lichtner (LANL), Dr. Alex Pothen (Purdue University), Dr. David Randall (Colorado State University), Dr. Robert Ross (ANL), Dr. Roman Samulyak (Stony Brook University and Brookhaven National Laboratory), Dr. Brian Van Straalen (LBNL), Dr. Priya Vashista (University of Southern California), Dr. D. N. Williams (LLNL), and Dr. Stan Woosley (University of California-Santa Cruz).

Further Reading: http://www.scidac.gov