DOESciDAC ReviewOffice of Science
HARDWARE
Opening New Possibilities in Science and Engineering
Cutting-edge science demands cutting-edge computational resources. To meet that need, the Argonne Leadership Computing Facility has recently installed an IBM Blue Gene/P system. With this new system and its smaller predecessor, the Blue Gene/L, the facility is accelerating the coming era of petascale computation in support of breakthrough science and engineering.
As a designated Department of Energy (DOE) leadership-class facility, the Argonne Leadership Computing Facility (ALCF) is transforming research in a wide range of fields, from finding better catalysts to modeling nanoscience particles for novel devices. Established in 2004, the ALCF initially deployed a 5.7 teraflop (TF) IBM Blue Gene/L (BG/L) computer with 2,048 processors. Since then, the ALCF staff have been working closely with IBM to meet the demand for ever-increasing compute and data capabilities. The recent installation of the 111 TF Blue Gene/P (BG/P; figures 1 and 2) is the first step of a phased project that includes the installation of a 445 TF BG/P later this year. Science and engineering computations will be able to use up to 163,840 processors at a time to explore how complex physical, chemical, and biological systems behave.
Figure 1. The Blue Gene/P is the newest acquisition in Argonne's ALCF, shown here during installation. This "green machine" with low wattage and unprecedented performance is being tested on applications at the forefront of science.

At the Forefront of Science
The ALCF provides an open science environment for researchers to attack cutting-edge problems in science and engineering at unprecedented scale and speed. Beginning in 2006, the ALCF commenced production support of DOE's Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, which encourages and supports compute-intensive research from industry, academia, and scientific laboratories (sidebar "DOE's INCITE Program Enables High-Impact Scientific Advances," p48). A timely partnership with IBM's T.J. Watson Research Center also provided several INCITE projects with access to the Watson 114 TF BG/L system, typically used for IBM corporate research. In 2007, the number of ALCF INCITE projects increased to nine, spanning a wide range of areas from materials science to climate change to molecular biology. Following are just a few highlights of recent successes under this program.
Nanophotonics. An open question in fabricating metal nanoparticle arrays has been how to size, shape, and arrange the particles to yield a desired optical property or make a desired photonic device. Researchers from Northwestern University have devised a numerical approach that optimizes the coherence properties of laser radiation and the structural parameters of the material. Using this approach on the BG/L at the ALCF, the researchers have been able to conduct precise numerical analyses of the optical properties of silver L-shaped nanoparticles. Their results show a clear association between high-energy resonances and volume plasmon oscillations caused by the particle's geometrical asymmetry (figure 3, p44). This simulation was the first observation of volume plasmons.
The ALCF provides an open science environment for researchers to attack cutting-edge problems in science and engineering at unprecedented scale and speed.
Figure 2. Installation of the IBM BG/P in the ALCF began at the end of 2007. By January 2008, the first system was already being used by scientists and installation of the balance of the 556 TF system was far along.

Molecular modeling. Using molecular modeling and molecular dynamics simulations in combination with biochemical and ultrastructural analysis, researchers from the University of California-San Diego have shown that increased aggregation of a protein known as alpha-synuclein (aS) can lead to the formation of harmful pore-like structures in human membranes ("Modeling the Molecular Basis of Parkinson's Disease," SciDAC Review, Winter 2007, p52). Familial studies of Parkinson's disease suggest that its progression is associated with such defects. Thanks to an INCITE award on the BG/L, the researchers were able to conduct large simulations, with the number of atoms in the system reaching as high as 800,000. Their results show encouraging correlations between the molecular dynamics predictions and laboratory experimental results (figure 4). The team hopes to develop cues for the design of effective drugs for Parkinson's and other diseases.
Protein structure. Proteins are the workhorse molecules of all biological systems. A deep and predictive understanding of life thus requires a detailed picture of their structure. Researchers from the University of Washington have devised and are testing a new approach for determining protein structures computationally. The approach has been tested on proteins of human structure up to 189 amino acids in length—with, in many cases, a remarkable accuracy within one angstrom of the experimentally solved high-resolution crystal structure. Since the protein conformation search space is so large, such a high level of accuracy would not have been possible without access to the high-performance BG/L computers at the ALCF and the IBM T.J. Watson Research Center (figure 5).
Figure 3. Studies of plasmon resonance of silver nanoparticles by Northwestern University researchers provide insights into the origin and content of a high-energy band and its dependence on the external field polarization.

Jet-engine science. Industry also has made use of the ALCF resources. For example, high-performance computing is now essential to jet engine development. Pratt & Whitney researchers have used 750,000 hours of computer time to simulate conditions inside a jet engine combustor to gain deeper insight for the next generation of fuel-efficient, low-emission jet engines. In addition, Pratt & Whitney researchers have benefited from the applications support and tools provided by the ALCF.
Figure 4. Shown here is a membrane embedded with residues of alpha-synuclein pentamer. San Diego Supercomputer Center scientists are using such simulations to study the molecular mechanism for Parkinson's disease progression.
Nuclear physics. A small portion of the BG/L also has been devoted to non-INCITE projects. One example is the work of Argonne physicists, in collaboration with researchers from the University of Illinois at Urbana-Champaign, Los Alamos National Laboratory, and SISSA in Italy, to describe the properties of nuclei and neutron stars. In 2000, when computers were less powerful, the precision of the researchers' Monte Carlo calculations were limited. With the availability of the BG/L, however, the researchers could make far more extensive runs with many different parameter values. Frequently, these calculations use 1,000 processors of the BG/L and range from ten minutes to whole-day runs. According to the researchers, having more cycles enables them to do the extensive parameter searching needed for calculating the new Illinois 3-nucleon potential. Encouraged by this success, they are collaborating with computer scientists at Argonne as part of a SciDAC project, preparing to run precise calculations of carbon-12 on the BG/P.

A Science Partnership
The ALCF systems' impact is due in large part to Argonne's successful partnership with IBM (sidebar "DOE-IBM Partnership Delivers Leadership Computing," p47). Originally intended to meet life sciences requirements, the Blue Gene system evolved—with input from Argonne and other DOE researchers—into a balanced, highly scalable architecture that could be used to address a wide range of DOE modeling and simulation problems. This interaction helped in optimizing system performance for DOE missions, reducing risk, and shortening the interval between system concept and first science. In 2005, Argonne received what was only the third BG/L system in the field.
The BG/L system offers researchers powerful processors, multiple internal networks, and outstanding balance in its performance of computation and communication. Moreover, the BG/L system is highly scalable—to 64,000 nodes and beyond. Hence, it is an ideal platform for evaluation of highly scalable massively parallel processor architectures; implementation and testing of applications for next-generation computers; and development of systems software, tools, and algorithms for such systems. ALCF researchers immediately began carrying out extensive evaluations with the BG/L. Although IBM supports the BG/L hardware and a complement of systems software, many important systems tools, libraries, and capabilities had yet to be ported. Therefore, ALCF researchers focused on developing new BG/L operational capabilities. Much of this work involved close interaction with IBM as well as with the SciDAC Scalable Software Enabling Technology Center.
In particular, optimized versions of the Cobalt job manager and the Parallel Virtual File System (PVFS) were developed and subsequently deployed not only in the ALCF but also at multiple other BG/L sites. These advances in system software are crucial to ensuring that researchers can make full use of DOE's investment in leadership-class computers for scientific discovery. Moreover, new benchmarking tools created by ALCF computer scientists dramatically improved the software development environment, making these systems more useful for our nation's research scientists and engineers.
The ALCF systems' impact is due in large part to Argonne's successful partnership with IBM.
The IBM/Argonne interaction has continued through the BG/P design process, with an expanded partnership between the DOE Office of Science led by Argonne, the National Nuclear Security Administration (NNSA) Advanced Simulation and Computing (ASC) program led by Lawrence Livermore National Laboratory, and groups at the IBM T.J. Watson Research Center and IBM Rochester, Minnesota facilities. Topical working groups met regularly to discuss requirements, options, and tradeoffs in areas such as functionality, software architecture, application performance, and system management. These frequent and detailed interchanges improved many aspects of the BG/P. Furthermore, several dozen science applications were ported to the BG/P and tested before the first system was shipped.
"The ALCF has been a valuable contributor in the development of Blue Gene/P," said Leo Suarez, head of deep computing at IBM. The result of this collaboration is a much more powerful system with even broader utility for DOE science and engineering. Suarez noted that the close working relationship between the two institutions has helped deliver a machine that will "propel scientific discovery in the most profound way since Galileo's telescope."
Extraordinary Capabilities for Extraordinary Science
The BG/L installed at the ALCF in 2005 was a one-rack evaluation system with 2,048 processors. The ALCF chose the system because it offered scientists several unique features. The dual processors per node and system-on-a-chip design provided good compute-communications balance. The dual floating-point unit architecture with parallel loads and stores provided a fast way to transfer data between processors and memory; moreover, swap order was available, a feature useful for scientists performing matrix transpose. The network configuration was designed for low-latency barriers and interrupts; the three-dimensional torus with active messages allowed for fast transposes as required by science users for fast Fourier transforms.
Just two years later, the ALCF has dramatically expanded its system capabilities—by a factor of more than 20—with the installation of two BG/P systems (figure 6). The main system, known as Endeavour, is an 8-rack production system with 32,768 processors and 800 terabytes of high-performance disk storage; the smaller system, known as Surveyor, is a 13 TF (single-rack) 4,096-processor test and development system. Raw processor performance improvements were complemented by increases in memory bandwidth, large caches, and faster interprocessor communications. Full symmetric multiprocessor (SMP) support was incorporated within the four-way PowerPC 450-based processors and Direct Memory Access (DMA) engines were added to streamline communications. Together, the SMP and DMA capabilities open up new programming models that support a wider range of applications. The performance per watt improved by 60% as well.
Figure 5. Researchers at the University of Washington use their code ROSETTA to determine protein structures. Electron density (black) and structure (red) are generated from a low-resolution model (left panel) and predicted by refined phases from ROSETTA (right panel).
Endeavour will be available for a new and expanded set of INCITE projects in spring 2008. Already, this BG/P computer has an array of early science users, who have been testing the waters in areas ranging from astrophysics to nuclear reactor simulation.

Scientists, however, are rarely satisfied even with state-of-the-art machines. Pushing toward ever more powerful systems for ever more complex problem solving, the ALCF is already planning an expanded installation later in 2008, comprising 32 more racks of BG/P with an additional 8 petabytes of next-generation disk storage. Initially this 445 TF production system, known as the Intrepid, will be operated as a separate system, in order to minimize interruptions to scientific projects already under way on Endeavour. When both systems attain the required levels of operational robustness and reliability, the resources will be merged. The result will be capabilities for scientific runs up to 557 TF using 163,840 processors.

Just two years later, the ALCF has dramatically expanded its system capabilities--by a factor of more than 20--with the installation of two BG/P systems.
And the amazing feature is that all this equipment, including disk storage, needs only 4,000 square feet of computer room floor. Equally important, particularly in view of today's critical concerns about power consumption and heat rejection and the nation's growing interest in green leadership, the Blue Gene was designed to have extremely low power consumption per floating-point operation—less than two megawatts in this case! In fact, BG/P's power consumption per teraflop is much lower than that of the next-best leadership system and typically a fraction of many systems built with commodity microprocessors.

High-Performance Data Management for Leadership Science
Leadership systems such as Blue Gene have challenging input/output requirements. The extraordinary speed of petascale systems must be matched to disk storage systems that can keep up with the requirements for reading and writing data files and checkpoint files. ALCF's production BG/P systems will have a total of 640 10 Gigabit per second Ethernet data ports that ultimately interface to 9,760 disk drives (in RAID 6+2 configuration). The heart of this connection is a very large full-Clos Ethernet switch that connects the 640 Blue Gene ports to 88 file servers. At the other end, the disk drives are packaged into 21 high-performance storage area networks, each connecting to four file servers. This architecture ensures DOE scientists of both the excellent performance they demand and the fault tolerance they require. Both IBM's GPFS and Argonne's PVFS2 parallel file systems will be deployed on this hardware.
Figure 6. Installation of the new 445-teraflops IBM BG/P system at Argonne National Laboratory.
The ALCF tertiary storage will be provided by a multi-petabyte tape archive system, managed by the High Performance Storage System (HPSS) developed by DOE laboratories and IBM. An array of dedicated data mover nodes will enable high-speed Grid FTP transport between disk and tape and between ALCF and external locations worldwide. Argonne's high-speed fiber infrastructure and proximity to Chicago's global communications interchange provide rapid transport on ESnet, Internet-2, and other peer networks.
Creating an Ecosystem for Scientific Discovery
Of course, it takes more than state-of-the-art computers and data systems to make a successful leadership computing facility. ALCF's service offerings draw on a wide range of experience at Argonne in operating and using advanced facilities. The User Services and Outreach Group provides Tier-1 technical support, manages the ALCF User Group and educational programs, and leads development of online content, user guides, media releases, and publications. The Catalyst Team's computational scientists provide expert-to-expert liaisons with INCITE projects assisting with computational campaign planning, and providing points of coordination for partnerships and problem solving with the ALCF. In addition, the ALCF has a dedicated Performance Engineering and Data Analysis Group, with deep expertise in porting, debugging, and tuning applications at scale. This group also assists with post-processing and analysis approaches for massive datasets. The computer and data resources are managed and operated by the Facility Operations and Networking Group and the Extreme Data Systems Group.
To help the high-performance computing community learn about and evaluate Blue Gene, and then share expertise and tools, Argonne and IBM also organized the Blue Gene Consortium in 2004 (sidebar "The Blue Gene Consortium"). Early on, the ALCF staff reached out to scientists worldwide, offering applications workshops in the United States, Europe, and Japan: holding tutorials, hosting software development and benchmarking efforts, and nurturing working groups to share software and systems expertise. In just three years Consortium membership has spread worldwide, topping 70 institutions and hundreds of participants by 2007. It rapidly built a diverse ecosystem of university, laboratory, and industry personnel who ported hundreds of tools and applications and shared experiences on dozens of Blue Gene systems. The Blue Gene Consortium continues into the BG/P era, including access to the BG/P test and development system in the ALCF.

More to Come
It has been an exciting few years, from the birth of the ALCF to the installation of the BG/L and then the BG/P. But the ALCF staff are certainly not resting on their laurels. The next step is a move to a new supercomputing Theory and Computing Sciences (TCS) building, with groundbreaking scheduled to begin in early spring. Located on the boundary of Argonne's secure perimeter, the approximately 200,000 ft2 facility will be home to more than 600 laboratory employees. In addition to housing the new BG/P systems, the TCS will provide office space and meeting rooms for visiting scientists and research groups using the computer to collaborate on new science.
Scientists and engineers are already hard at work designing Blue Gene/Q-- the successor to BG/P.
"The new building will provide a fitting environment for the ALCF, the new IBM Blue Gene/P and its successors, and the scientific research under way at Argonne," said Dr. Rick Stevens, Argonne's Associate Laboratory Director for computing and life sciences. "The research enabled by the new BG/P systems will touch many areas of science and society—from research in astrophysics and nuclear reactor design, to searching for cures for Parkinson's disease and better drugs to fight antibiotic-resistant staph infections such as MRSA, to long-term climate and ecosystem simulations, and to a better understanding of the global carbon cycle which underpins global climate," he noted.
Moreover, scientists and engineers are already hard at work designing Blue Gene/Q—the successor to BG/P—drawing on technological advances, science program requirements, and a wealth of experience with high-performance, low-power system designs. In the sprint from petascale to exascale, the next generation of systems and applications has crucial groundwork to lay. A critical concern will be a balance between power consumption and multicore architecture. Along the way, science and engineering applications will be using the BG/P to assist in the retooling and algorithm redesign needed to leap from a few cores per chip to many. Without question, the scientific computing community anticipates an extraordinary decade of opportunity and discovery for leadership computing.

Contributors: Dr. Ray Bair, Senior Computational Scientist in the Computing, Environmental, and Life Sciences Directorate at Argonne; Dr. Gail W. Pieper, Senior Coordinator of Technical Editing and Writing in the Mathematics and Computer Science Division at Argonne; Cheryl Drugan, Coordinating Writer and Editor in the Leadership Computing Division at Argonne
Further Reading
http://www.alcf.anl.gov
 
Published by IOP Publishing in association with Oak Ridge National Laboratory, for the US Department of Energy, Office of Science. Copyright © 2008 by IOP.