DOESciDAC ReviewOffice of Science
The World's Most Powerful Computers for Science
High-performance computers at Department of Energy facilities are providing powerful tools unprecedented in the history of science, allowing researchers to attack scientific problems of global importance, many of which were previously unapproachable.
What mysteries about the makeup of the Universe and the origins of life could you unravel if you could create large-scale, highly-detailed, three-dimensional simulations of stars exploding in core-collapse supernovae (sidebar "Core-Collapse Supernovae" p56)?
What if you could find a way to eliminate biomass recalcitrance—the resistance of cellulosic biomass to enzymatic breakdown into sugars, which is the single-greatest barrier to the cost-effective production of biofuels (sidebar "Developing Ethanol" p56)?
What if you could build detailed regional models depicting the impact of greenhouse gas emissions—models that take into account the complex biophysical interactions between land, sea, and sky (sidebar "Modeling Greenhouse Gas Emissions" p57)?
Suppose you could reach a far greater understanding of the direct and indirect impact of atmospheric aerosols on the world's climate (sidebar "Atmospheric Aerosols" p60)?
What if you could explore and validate the science necessary to transition from today's studies in plasma physics to the electricity producing fusion power plants of tomorrow (sidebar "Fusion Power" p61)?
What if you could improve the power and energy densities of energy storage systems to the point that 500-mile rechargeable batteries could be designed and built for your automobile?
What if the margins of safety in next-generation nuclear reactors could be so confidently predicted that they could be built more much quickly and cheaply? What if nanoparticles could be designed to act as electrical switches, hard drives, or superconductors?
What if nanoparticles could be designed to act as electrical switches, hard drives, or superconductors?
Finding the Answers
The solutions to these and many other high-priority scientific applications that have a direct bearing on the health of the planet and its inhabitants, such as climate change and sustainable energy, are now within reach. The latest generation of supercomputers is making this possible.
For example, in the United States, the Department of Energy's (DOE) computing facilities are equipped with massively parallel systems and computational resources that are unprecedented in the history of science.
DOE Office of Science computing facilities include: Oak Ridge National Laboratory (ORNL) in Tennessee; Argonne National Laboratory in Illinois (sidebar "Argonne National Laboratory" p58); and the National Energy Research Scientific Computing (NERSC) Center (sidebar "National Energy Research Scientific Computing Center" p62) at Lawrence Berkeley National Laboratory (LBNL) in California. Another major DOE facility is the Molecular Science Computing Facility (MSCF) at Pacific Northwest National Laboratory (PNNL) in Washington state (sidebar "Pacific Northwest National Laboratory" p64).
The solutions to many high-priority scientific applications that have a direct bearing on the health of the planet and its inhabitants, such as climate change and sustainable energy, are now within reach. The latest generation of supercomputers is making this possible.
The big news is that petascale computing—the processing of more than thousand trillion floating-point operations per second (petaflops)—is here. The National Center for Computation Sciences (NCCS) at ORNL has completed the installation of a Cray XT5 petaflop supercomputer known as Jaguar—the world's first petaflop machine available for open scientific research (figure 1).
Charles Brooks Photography
Figure 1. The 1.64-petaflop Jaguar at the NCCS features more than 180,000 processing cores, each with two gigabytes of local memory. The resources of the ORNL computing complex provide scientists with a total performance of 2.5 petaflops.
NCCS (figure 2) was established in 1992. Its mission is to advance the state of the art in high-performance computing and make available the resources of powerful parallel supercomputers to scientists investigating a wide variety of compute-intensive projects. NCCS works with industry, laboratories, government agencies, and academia to deploy a computational environment that will enable the scientific community to exploit this extraordinary capability, achieving substantially higher effective performance than is possible today. They are developing an ambitious approach to achieving a high level of scientific productivity to address challenges in such diverse areas as climate, fusion, astrophysics, materials science, nanoscience, chemistry, biology, combustion, accelerator physics, engineering, and other disciplines relevant to maintaining U.S. science leadership.
Photo: C. Boles, Courtesy of ORNL/U.S. DOE
Figure 2. ORNL's $300 million modernization program, funded by a federal, state, and private partnership, included two new facilities for Computing and Computational Sciences.
NCCS leadership computing efforts initially centered on a Cray X1 computer named Phoenix that operated at 6.4 teraflops, and a Cray XT3, Jaguar, running at 26 teraflops. A series of upgrades culminated in the Center's Cray XT4 Jaguar system that in 2007 was ranked as the second fastest supercomputer in the world and the fastest for open scientific research. The system achieved 101.7 teraflops running the LINPACK benchmark, the basis for the rankings on the Top500 list, the semiannual tally of the world's fastest computers. Peak performance for the Cray XT4 was clocked at more than 119 teraflops. But equally important for the successful operation of the supercomputer was the development of a support structure consisting of a high-speed fiber optic network to expedite data movement, a scientific visualization center for data analysis, and high-performance data archiving and retrieval systems. This infrastructure will support the advanced capabilities of the Cray XT5 Jaguar.
The big news is that petascale computing—the processing of more than thousand trillion floating-point operations per second (petaflops)—is here.
As Buddy Bland, director of ORNL LCF projects notes, "Jaguar lets us solve problems that were not even approachable before."
Although Jaguar's latest incarnation is making a major contribution to advancing the frontiers of science, ORNL and the other three DOE leadership facilities also have an impressive array of other supercomputers, high-performance memory, file systems, and networking capabilities that, along with Jaguar, are helping to solve some of today's knottiest scientific problems.
Simulation: E. Endeve Visualization: R. Toedte, ORNL National Renewable Energy Laboratory
Figure 3. A magnetohydrodynamic simulation performed with the GenASiS code explores what effects the instability of a supernova shock wave has on the magnetic fields in stellar cores during core-collapse supernova explosions. Shown is a volumetric representation of magnetic field strength (the semitransparent part) as well as a sampling of magnetic field strength and orientation at selected nodes (the vectors). The purpose of this was to help understand the effect of magnetic fields on the evolution of the 3D shock front. Figure 4. Pseudocolor SEM micrograph of lignin redeposited on cellulose (10,000x).

Focus on Jaguar
Jaguar is the culmination of a close four-year partnership between ORNL and Cray that has pushed computing capability relentlessly upward. The XT system grew in strength through a series of upgrades. In 2008 a 263-teraflop Cray XT4 was upgraded with the addition of a 1.4-petaflop Cray XT5. The combined system uses an InfiniBand network, the Spider file system, and approximately 182,000 processing cores to form the DOE Office of Science's 1.64-petaflop system. Occupying 284 cabinets, Jaguar uses the latest quad-core Opteron processors from AMD and features 362 terabytes of memory and a 10-petabyte file system. It has 578 terabytes per second of memory bandwidth and unprecedented input/output (I/O) bandwidth of 284 gigabytes per second to tackle the biggest bottleneck in monster systems—moving data into and out of processors. Physically this is a big system. The machine occupies 5,700 square feet, a space that is a little larger than a college basketball court.
"When I look at Jaguar, I see the most balanced, powerful computer in the world," says Bland. "But even though it's taking scientific computing to a whole new level, the system relies on a proven microprocessor—the AMD Opteron—that has been a supercomputing workhorse for years. Because people understand how to program the Opteron chips, you don't need an exotic programming model to figure out how to make applications run well on the Cray XT5. Another major plus is that we can port and run the considerable software investments that we have made over the years in applications running on X86 software and the Linux operating system. We've seen researchers port their code to the upgraded Jaguar quickly and seamlessly, and they realize immediate leaps in performance."
"Jaguar lets us solve problems that were not even approachable before."

Buddy Bland
F. Hoffman and J. Daniel, ORNL
Figure 5. As the Sun rises over   Eastern Europe, the instantaneous net ecosystem exchange (NEE) of CO2 is shown in the Eastern Hemisphere. Strong uptake is shown in green-to-white colors and is strongest in the tropics. A net release of CO2 to the atmosphere is shown in red-to-white colors and is strongest over the Congo, where the Sun is not shining. This image was produced from a C-LAMP simulation performed as part of a SciDAC-2 project using NCCS supercomputers.

Scalable Architecture
Cray achieved this level of large-scale, massively parallel computing by using a highly scalable architecture that incorporates dedicated compute and service nodes (figure 6).
L. Hoelzeman, Cray Inc
Figure 6. The Cray XT5 architecture directly connects all the AMD processors using the Cray SeaStar interconnection chips in a three-dimensional torus architecture.
Service nodes provide system and I/O connectivity and also act as login nodes where jobs can be compiled and launched. But it is the compute nodes that allow researchers to take their investigations to a previously unobtainable level. Each compute node is made up of quad-core AMD Opteron microprocessors (2.3 GHz) and high bandwidth direct attached memory, as well as dedicated communications resources (figure 7).
L. Hoelzeman, Cray Inc
Figure 7. Jaguar's compute node.
Bland points out that the system has more than three times the onboard memory as the next-largest supercomputer in the world. "The size of the memory lets us tackle problems that can't be handled by any other system," he comments. "Think of this memory as a scratch pad, a working space on the computer that allows you to solve larger, more complex problems more effectively. By being able to keep much larger scientific problems in memory, you can handle complex applications like modeling supernova or the Earth's climate without having to offload data. You can also make existing applications more realistic and predictable by incorporating higher-fidelity models. This is a far more productive approach than working portions of the application offline, and then laboriously bringing all the bits and pieces back together to get a coherent simulation of the problem you're trying to solve."
The system features a high-bandwidth, low-latency interconnect based on Cray's SeaStar2+TM chip. Each communications chip contains communications processing and high-speed routing on a single device including a HyperTransportTM link and a direct memory access engine (DMA). SeaStar directly connects all the computer's nodes in a 3D torus (a 3D mesh with the ends connected) topology—this allows the system to scale to tens of thousands of nodes.
Images courtesy of Paul Dave, ANL
Figure 8. Examples of INCITE research projects at ACLF. (Top left) Research findings offer insight into Parkinson's disease. A research team from the San Diego Supercomputer Center at University of California—San Diego recently proposed and elucidated the molecular mechanism of Parkinson's disease progression. Processor-hour allocations from the INCITE program enabled the team to perform complex calculations for the research on high-performance Blue Gene/L computers at the ALCF and the San Diego Supercomputer Center. (Top right) Blue Gene/L helps improve quality of protein structure predictions. Aided by the Blue Gene/L supercomputer, University of Washington researchers are able to sample a substantially larger pool of conformations—about two to three orders of magnitude greater than was previously possible. They have sampled closer to near-native conformations, resulting in an overall improvement in the quality of their predictions. (Bottom left) Improving aircraft engine combustor simulations. A Pratt & Whitney team is developing improved aircraft engine combustor simulations to enable reduced emissions and improved operability. (Bottom right) Exploring the molecular mechanisms of bubble formation. Procter & Gamble is investigating the molecular mechanisms of surfactant-assisted bubble formation.
Figure 9. Ionic map of atmospheric particles enriched in sulfate (green) and methane sulfonate (blue) typical for a marine environment.

High-Performance File System
One of the other major features supporting the new Jaguar system is a Lustre-based file system called Spider. "Jaguar also has an enormous amount of I/O bandwidth—all that memory is useless if you can't load data into the computer and get results back out," Bland adds. "Jaguar has a disc bandwidth of 288 gigabytes per second—larger than any other supercomputer, but very balanced for a system of this size."
Lustre, originally developed by Cluster File Systems, is an open-source, scalable, secure, robust, highly available cluster file system designed, developed, and maintained by Sun Microsystems. Spider will provide the Jaguar system with 10 petabytes of storage space. It replaces multiple file collections scattered around the NCSS network with a single scalable system. File transfers between other computers and file systems will be eliminated, resulting in higher performance, greater convenience, and significant cost savings. The file system connects all the computers in the NCCS data center, its high-performance storage system (HPSS) (figure 10), data analysis systems, and EVEREST (figure 11), the Center's visualization system.
Larry Hamill Photography
Figure 10. The NCCS massively parallel high-performance storage system (HPSS) data archive allows users to store vast amounts of long-term data.
Spider is absolutely essential for the efficient operation of the supercomputer. For example, without this high-bandwidth file system, transferring petascale datasets between Jaguar and the visualization system could take hours, tying up the supercomputer's bandwidth and slowing other simulations in progress. This high file transfer rate is critical for handling the torrent of data associated with climate studies or leading edge scientific experiments such as output from the Large Hadron Collider (LHC) at CERN, the ITER project designed to demonstrate the scientific and technical feasibility of fusion power, and other major scientific applications.
Keeping the Cray Cool
Jaguar includes 284 cabinets that house 45,376 quad-core AMD Opteron processors and 362 terabytes of memory. The unique packaging of Jaguar squeezes all of these parts into very dense cabinets of about seven teraflops. This translates to a lot of heat that must be removed to keep the system running.
"The XT5 portion of Jaguar consumes up to seven megawatts in a footprint of less than 4,200 square feet," says Jim Rogers, LCF director of operations. "This equates to a power density of over 1,500 watts per square foot."
To cool the system, Jaguar deploys a liquid cooling technology known as ECOphlex(TM), which relies on R134a refrigerant piped through an evaporator on the top of each cabinet. Fans flush heat into the evaporator, where it boils the R134a, and the vaporized refrigerant absorbs the heat. The coolant is then condensed in a chilled-water heat exchange system that absorbs the heat from the R134a vapor and exhausts it externally.
This cooling system makes the NCCS one of the most efficient computer centers in terms of the power used for cooling versus total power consumption, Rogers noted. At the NCCS, less than 25% of the total power consumption is used for cooling. A more typical average is closer to 45%. Jaguar runs directly on 480-volt AC power, a crucial innovation Cray and ORNL have worked on for several years, Rogers said. "Directly using 480-volt power and installing the transformers close to the computer saved nearly $1 million in installation costs. Using 480-volt power also eliminates losses resulting from a step-down to 208 volts and reduces the electrical losses from the power transmission, further reducing overall power consumption."
This cooling system makes the NCCS one of the most efficient computer centers in terms of the power used for cooling versus total power consumption.
High-Performance Storage System
One of the systems is a major contributor to NCCS's extraordinary computational capabilities. HPSS is a very large (petabytes of data), high-bandwidth centralized repository that has been developed over the past five years in a collaboration between five of the DOE's national laboratories and IBM.
HPSS (figure 10, p60) currently consists of tape and disk storage components, Linux servers, and HPSS software. As of October 2008 the system is storing over 3.5 petabytes of data. Tape storage is provided by robotic tape libraries that can each hold up to 30,000 cartridges, each of which can store up to a terabyte of data.
HPSS will continue to grow in importance as computing becomes increasingly data intensive. The facility also acts as a storehouse for community experimental data that can be shared by various labs and researchers.
Analyzing the Flood of Data
Petascale computing means an outpouring of data that could inundate researchers if they did not have the tools in place to cope with computational capabilities that can produce hundreds of terabytes of data in a single run.
One of the mainstays of data analysis and visualization at NCCS is EVEREST (Exploratory Visualization Environment for Research in Science and Technology).
S. Ahern, ORNL
Figure 11. EVEREST (Exploratory Visualization Environment for REsearch in Science and Technology) is a large-scale venue for data exploration and analysis.
D. Batchelor, F. Jaeger, and S. Ahern, ORNL
Figure 12. Three-dimensional simulations of radiofrequency heating with the AORSA code in plasmas representative of the ITER fusion reactor as well as in present tokamaks such as the NSTX (shown here) shed new light on the behavior of superheated ionic gas. Image courtesy of Don Batchelor, Fred Jaeger, and Sean Ahern, ORNL.
ORNL's EVEREST (figure 11, p61) is a large-scale immersive venue for data exploration and analysis. This facility houses a screen that is 30 feet wide by 8 feet high and has a resolution of over 11,000 x 3,000 pixels, creating a total space of 35 million pixels. It is integrated with NCSS, creating a high-bandwidth data path between large-scale high-performance computing and large-scale data visualization. The immersive qualities of this environment create a powerful discovery tool for research groups and collaborations. It is an essential tool for scientists working with the massive amounts of data that the Jaguar system is expected to generate.
Making the Connection
NCCS has direct connection to ESnet (Energy Sciences Network), providing a high-bandwidth pipe that links the Center with more than 40 other DOE sites, as well as fast interconnections to more than 100 additional networks. Funded by DOE's Office of Science, ESnet services allow scientists to make effective use of unique DOE research facilities and computing resources, independent of time and geographic location.
NCCS is also connected to the Internet2 network and TeraGrid. Internet2 provides the U.S. research and education community with a network that meets their bandwidth-intensive requirements. The network is a dynamic, robust, and cost-effective hybrid optical and packet network. It furnishes a high-speed network backbone that can handle full motion video and 3D animations to more than 200 U.S. educational institutions, corporations, and non-profit and government agencies.
The TeraGrid uses high-performance network connections, to integrate high-performance computers, data resources and tools, and high-end experimental facilities around the country. Currently, TeraGrid resources include more than a petaflop of computing capability and more than 30 petabytes of online and archival data storage, with rapid access and retrieval over high-performance networks. Researchers can also access more than 100 discipline-specific databases.
Data: C. Kerr, GFDL/NOAA Visualization: Prabhat, LBNL/NERSC
Figure 13. This project at NERSC couples a high-resolution ocean general circulation model with a high-resolution dynamic-thermodynamic sea ice model in a global context. Shown here are hurricanes forming over the Atlantic Ocean.

A Little Help from Your Friends
Users of the Jaguar system—whether they are from ORNL, other DOE laboratories, or from universities, businesses, and other institutions that have been awarded time on the supercomputer through the INCITE program ("Unparalleled Computing Allocations Guarantee Revolutionary Science" p20)—can count on support from a team more than 30 knowledgeable computational scientists within the NCCS.
R. Kaltschmidt and J. Bashor, NERSC
Figure 14. The Franklin supercomputer at NERSC.
The Scientific Computing Group's (SCG) role is to collaborate with and accelerate scientific progress for users of the Center's computational capabilities. They work directly with users as liaisons between the science teams and the NCCS to achieve the science goals of their specific projects. The SCG has extensive experience in model formulation, parallel algorithm development and implementation, and developing, porting, tuning, and utilizing software on NCCS resources.
On the road to exascale systems, the scientific computing community will be making the most of supercomputers running at tens, hundreds, and eventually thousands of petaflops of computing power.
It also serves the data-movement, workflow, analysis, and visualization needs of the NCCS user community. The members of the SCG team are all respected scientists and active researchers with backgrounds in a variety of key scientific fields—from climate change to fusion. This allows them to not only assist users with the Center's resources, but also add considerable value by providing insights into the projects based on their own experience.
DOE's Other Advanced Computing Facilities
Although ORNL's NCCS is currently in the spotlight with the Jaguar implementation, the Oak Ridge lab is part of the DOE's Office of Science computing facilities that also include Argonne (sidebar "Argonne National Laboratory" p58), NERSC (sidebar "National Energy Research Scientific Computing Center" p62), and the Molecular Science Computing Facility at PNNL (sidebar "Pacific Northwest National Laboratory"). Equipped with their own extensive supercomputer and high-performance computing clusters, these labs are tackling some of today's most important and strategic scientific problems in physics and chemistry.
Illustration: A. Tovey Source: L. L. Pan, J. Li, and L. S. Wang. 2008. J. Chem. Phys. 129, 024302
Figure 15. Two new isomers of negatively charged nine-atom boron clusters, which could be vital in designing storage devices for the hydrogen economy, were discovered through the use of the supercomputer at EMSL.

From Petaflops to Exaflops—The Road Ahead
"The evolution of high-performance computing is a clear example of accelerating technology," notes ORNL's Bland. "In the last four years, we have gone from just over one-teraflop machines to Jaguar, which runs at 1.6 petaflops. That's a factor of a thousand. Over the next ten years we expect to see another technological leap of the same order of magnitude. In parallel we have to keep developing increasingly sophisticated visualization and data analysis tools so scientists are not overwhelmed by all the data pouring out of these machines. To discover the knowledge often hidden in the data, they will need facilities like EVEREST to uncover the interesting tidbits that will lead to new scientific breakthroughs."
To design these post-petaflop systems, ORNL is working in collaboration with the Office of Science and the Department of Defense in the High Productivity Computer Systems (HPCS) program. Cray and IBM have been selected to work on building machines capable of around 20 petaflops. ORNL will work closely with both companies to help them understand the strengths and weaknesses of their designs and the computational needs of large-scale scientific applications. By 2011-2012, the Office of Science plans to install a 20-petaflop machine designed by the vendor whose design is selected. Looking even further into the future, the goal is to install a 100-250-petaflop machine in the 2015 time-frame, and an exaflop machine by 2018.
S. Kerisit, N.A. Deskins, K.M. Rosso, and M Dupuis. 2008. J of Phys Chem C, 112:7678
Figure 16. Computational capabilities at EMSL offer scientists insight into the catalytic mechanism of titania, a material with potential alternative energy and environmental remediation applications.
On the road to exascale systems, the scientific computing community will be making the most of supercomputers running at tens, hundreds, and eventually thousands of petaflops of computing power. Petascale computing allows them to explore some of today's most important and pressing scientific challenges, such as climate change and ensuring adequate, sustainable energy supplies. Climate scientists will use the systems to give planners and leaders the tools to anticipate the future changes. Petascale simulations will clarify the role the oceans play in regulating the carbon cycle and map the complex interactions of elements as diverse as plants, land-use patterns, the atmosphere, and ice sheets.
On the energy front, petascale systems will enable fusion researchers to conduct more detailed simulations of phenomena such as plasma turbulence. Materials scientists will use the results of atomic-scale simulations to design improved catalysts that will make fuel cells practical, develop materials that convert waste heat directly into electricity, and revolutionize energy storage technologies. Biologists will gain insight into efficiently converting cellulose into ethanol. And nuclear engineers will use computation to design safer, efficient, and cost-effective fission reactors.
"The Office of Science is deploying systems with what it takes to support dramatic advances in our understanding the world and the Universe," Bland concludes. "The combination of top scientists, talented staff, leading-edge computational capabilities, and partnerships with government, industry, and other research institutions also sets the stage for revolutionary science—we will discover and explore new and often unexpected alternatives to some of our most cherished, long-standing scientific assumptions. There is no question that we are on the threshold of a new era in science, and supercomputers like Jaguar are making it possible."
Contributor: John Kirkley