SciDAC Review - BLUEGENE / L

BLUEGENE / L

High Performance Hardware

Figure 1. Up and running, the BlueGene/L (BGL) computer at work at LLNL's Terascale Simulation Facility (TSF).

With its innovative design, BlueGene/L (BGL), housed in the Terascale Simulation Facility (TSF) at Lawrence Livermore National Laboratory (LLNL) is helping researchers usher in a new era in high performance scientific computing – predictive science. The IBM machine takes a radically different design approach from preceding supercomputing systems. Using a highly integrated compute node design, BGL is a scalable architecture that expands the computational power of the system by adding more building blocks without introducing bottlenecks as the machine scales up.

BGL (figures 1 and 2) serves scientists and engineers at the National Nuclear Security Administration's (NNSA) Advanced Simulation and Computing (ASC) Program. BGL is running the most complex materials science simulations ever created, in support of NNSA's Stockpile Stewardship Program. This program seeks to ensure the safety, security, and reliability of the nation's nuclear arsenal without empirical testing. The simulation capabilities on ASC supercomputing systems are a cornerstone of that effort.

Figure 2. A look inside BlueGene/L hardware and at the installation process at LLNL.

The machine uses system-on-a-chip design technology, and low-cost, low-power embedded microprocessors. Low-power processors are a critical design element that allows BGL to achieve remarkable compute densities, providing 1,024 nodes and more than five teraflops (five trillion floating operations per second) in a single aircooled cabinet. BGL has thirty-two terabytes of memory, but only uses 1.5 megawatts of power. With a footprint of 2,500 square feet, it is a fraction of the size of previous NNSA and Department of Energy (DOE) supercomputer systems.

The unique design of BGL allows problems with different time and length scales to overlap. This overlap of problem spaces will permit researchers, for the first time, to directly compare different techniques for modeling materials – comparisons that will allow scientists to improve the detail and accuracy of models over larger physical domains, leading to a better understanding of the relevant physics. BGL was originally developed as a science research device for exploring avenues to petaflop computing (assimilating 1,000 trillion floating operations per second). However, BGL is proving to be versatile in running a variety of molecular and dislocation dynamics and turbulence codes that are crucial for stewarding the nation's nuclear stockpile.

Qbox, a first-principles molecular dynamics (MD) code, recently set a new world mark for a scientific application with a sustained performance of 207.3 teraflops on BGL. Qbox is designed to predict the properties of metals under extreme conditions of temperature and pressure, a longstanding goal for researchers in materials science and high energy-density physics. The three-dimensional code run, which studies how the transition metal molybdenum behaves under pressure, represents one of only a handful of "predictive science" first-principles simulations achieving the noteworthy size of 1,000 molybdenum atoms. Predictive simulations such as this allow researchers to understand how complex physical, chemical, and biological systems behave over time. Previously, it was only possible to obtain brief snapshots of system behavior on a small scale. The capability to conduct predictive science is important to NNSA researchers trying to understand how the materials in nuclear weapons age, particularly for those warheads that have aged beyond their intended life.

First-principles molecular dynamics codes are used for complex simulations at the atomistic and molecular levels in a number of scientific areas, including metallurgy, solid-state physics, chemistry, biology, and nanotechnology.

Figure 3. Cross sectional images taken from atomistic simulations of tantalum solidification: solid (red), liquid (blue), and grain boundaries (yellow). The solidifying sample on the left (64,000 atoms) has developed an artificial structure due to the small size of this simulation, while the 16,384,000-atom simulation is free from such finite-size effects.

Running ddcMD, a three-dimensional MD code on BGL, LLNL scientists investigated solidification in tantalum (figure 3) and uranium with simulations including up to 524 million atoms. The team achieved a performance rate of up to 107 teraflops sustained over a seven-hour run on BGL's 131,072 processors, earning them the Gordon Bell Prize at the 2005 Supercomputing Conference in Seattle, Washington. These ddcMD simulations allowed researchers to examine the process of solid formation at high temperature and pressure from the atomistic level for the first time. Scientists were able to watch, atom by atom, as macroscopic grains grew out of the liquid and formed structures. The scale and detail provided by this process allow researchers to better understand the properties of metals, which has important implications for developing stronger metals for use in aircraft or automobile components, as well as other applications.

BGL, first developed through the ASC program, is now being used for scientific research at centers throughout the U.S., Europe, and other areas of the world. The Blue Brain project is the first collaboration under a joint research agreement between IBM and Ecole Polytechnique Federale de Lausanne (EPFL). Leveraging the unique computational capabilities of BGL, EPFL, and Watson Research, scientists will develop a detailed model of the Neo Cortical Column – a set of 10,000 neurons that represents the fundamental building block of the human brain.

The NNSA continues to build on the DOE's highperformance computing foundation. Together, the NNSA and the DOE Office of Science are advancing the boundaries of scientific computing dramatically. These changes will improve U.S. energy and economic security measures by accelerating new energy technology developments, simulating nuclear weapons behavior, and leading to new scientific knowledge and discoveries.