DOESciDAC ReviewOffice of Science
HARDWARE
Science-Based PREDICTION at LANL
Figure 1. The Roadrunner installed base system at LANL.
Scientific computing has been an essential part of Los Alamos National Laboratory (LANL) since the days of the Manhattan Project. LANL scientists have taken on the challenge of building computational tools to examine complex physical processeses by using innovations that extend beyond the classic scientific method. LANL has a rich legacy of high-performance computing resources—from the MANIAC, the Cray-1, and Thinking Machines’ CM-5 to the Q machine, the Lightning and Coyote Linux clusters, and now the Roadrunner system (figure 1). Computational science and engineering worldwide has benefited from DOE investments and innovations. Raising the bar, the Advanced Simulation and Computing (ASC) Program began in 1996 as a collaborative program to ensure the safety and reliability of the nation’s nuclear weapons stockpile. Today’s scientists combine observations, data, and theory with complex modeling, extremely fast computers, and visualization capabilities to understand the physical world on all scales, from the cosmically large to the infinitesimally small.

Behavior of Materials
Study of the dynamic behavior of materials is part of LANL’s rich scientific tradition. Figure 2 (p34) represents the 3D evolution of a titanium jet running into an obstacle as simulated with the Los Alamos ASC RAGE code. The several-millimeter-size jet was created on the OMEGA laser at the University of Rochester for a collaboration involving multiple laboratories (LANL, LLNL, Laboratory for Laser Energetics, General Atomics, and the Atomic Weapons Establishment) and universities (Rice and University of Rochester).
Figure 2. Model of a titanium jet running into an obstacle.

Biomolecular Modeling
Less traditional uses of supercomputing at LANL include drug interactions at the molecular level. Researchers have simulated the process of drug binding in a calculation that is at least 30 times longer than any previous drug binding simulation. Using the Coyote supercomputer, the team simulated the interaction of a critical antibiotic drug, gentamicin, used to treat plague and anthrax (figure 3). In July 2006 Dr. Kevin Sanbonmatsu, lead researcher for this project and other biomolecular modeling, received a Presidential Early Career Award for Scientists and Engineers.
Today’s scientists combine observations, data, and theory with complex modeling, extremely fast computers, and visualization capabilities to understand the physical world on all scales, from the cosmically large to the infinitesimally small.
Figure 3. Drug interaction at the molecular level, shown at the top as a computer simulation, and at the bottom as a traditional chemical diagram.

Climate Modeling
The Climate, Ocean, and Sea Ice Modeling (COSIM) project at LANL is responsible for providing state-of-the-art ocean, sea ice, and ice sheet models to the community for use in climate change simulations (figure 2, p9). The Parallel Ocean Program (POP) and CICE models are used as ocean and sea ice components of the DOE/NSF Community Climate System Model (CCSM). COSIM is supported by the DOE Office of Science Climate Change Prediction Program and the SciDAC project “Scalable and Extensible Earth System Model for Climate Change Science.”
Figure 4. FIRETEC simulation (upper) in comparison with an actual fire (lower), which was part of the International Crown Fire Modeling Experiment.

Wildfire Prediction
FIRETEC is a physics-based, 3D computer code designed to simulate the constantly changing, interactive relationship between fire and its environment. Figure 4 is a comparison between a FIRETEC simulation and a Crown Fire Modeling Experiment conducted in Canada. FIRETEC was the winner of an R&D 100 award in 2003.

Cosmology
An astonishing 99.6% of our Universe is “dark,” meaning it is not accessible to direct observation with light. Perhaps 70% is a mysterious dark energy now responsible for an accelerated expansion, and 25% is an as-yet-unidentified dark matter component. Simulations carried out at LANL have led to new insights into the distribution of dark matter clumps in the Universe and their masses (figure 5). These and other simulations are being used to interpret measurements from the Sloan Digital Sky Survey.
Figure 5. Results of a cosmological dark matter N-body simulation.

Roadrunner Project and Hybrid Computing
LANL and IBM have formed a partnership to develop and deploy the Roadrunner advanced supercomputer system. Roadrunner is an ensemble of integrated hybrid nodes: AMD Opterons and Cell Broadband Engine™ (BE) blades. This system will support a variety of science, engineering, and certification applications for the nuclear weapons program. Scheduled for final delivery in mid-2008, this system will have a peak speed of approximately 1.3 petaflop/s and is targeted to achieve a sustained petaflop/s on LINPACK. Effective use of this technology, as well as other future systems, will require new programming paradigms as well as some modifications to algorithms and data structures.
The Roadrunner project has three phases. In 2006, LANL took delivery of an Opteron-only configured base system, consisting of 16 connected units and an InfiniBand (IB) interconnect fabric with a peak speed of over 80 teraflop/s. This system was scheduled to move into classified operation in April 2007. The second phase is an R&D partnership. LANL leads the examination of applications suitable for initial implementation on the final hybrid system, and IBM leads the development of the programming framework for Roadrunner. This framework will also form the basis for effectively programming a broad class of future many-core and hybrid systems. Delivery, acceptance, and initial science runs on the final hybrid system is the third phase of the project (figure 6). The Roadrunner final system will then become an LANL/ASC production resource.
The Climate, Ocean, and Sea Ice Modeling (COSIM) project at LANL is responsible for providing state-of-the-art ocean, sea ice, and ice sheet models to the community for use in climate change simulations.
Figure 6. Cell BE Architecture™ Technology Competitive Roadmap (Version 5.0, 7-24-2006; all future dates are estimations only and subject to change without notice).
The final Roadrunner system will consist of more than 3,000 nodes interconnected via InfiniBand 4x DDR. Each node will consist of dual-core AMD Opterons connected to Cell BE blades via multiple PCIe x8 links, a very significant improvement over the initially proposed IB-connected Cell blades. Each Cell blade will contain two of IBM’s enhanced double-precision (eDP) Cell chips. Each integrated hybrid node will have a peak speed in excess of 400 gigaflop/s in double precision and will approach a teraflop/s in single precision. The eDP Cell BE leverages a substantial investment by Sony, Toshiba, and IBM to develop a power efficient, cost-effective system-on-a-chip for game consoles, consumer electronics, and high-performance computing. The original Cell BE is now the heart of the Sony PlayStation 3™. Public reports put this investment at roughly $400 million.
Contributors: Dr. Andrew White, Director, and Manuel Vigil, Manager, Roadrunner Project, LANL (LALP-07-027)
Futher Reading:
http://www.lanl.gov/roadrunner