DOESciDAC ReviewOffice of Science
SCIDAC - 2
EXCERPTS from the SciDAC-2 Strategic PLAN
As SciDAC moves forward, it is poised to meet the challenges of petascale computing. Plans for scientific infrastructure - including software, centers, partnerships, and institutes - aim to facilitate breakthrough science.
Over the past five years, numerous reports and scientific meeting discourses have underscored the need for yet greater simulation fidelity and new theoretical models. Indeed, the recent and expected growth in computational capabilities (along with the methodological advances achieved by SciDAC) has elevated the currently available scientific opportunities associated with advanced computing to a level that far exceeds that available five years ago, when SciDAC began. Such endeavors will be further complicated by increasingly complex computer architectures and the need to scale scientific computing codes up to the petascale regime.
The scientific software infrastructure being developed under SciDAC will enable researchers to exploit the ever-increasing capabilities of highend information technology to solve some of today's most challenging scientific problems. These problems encompass an expansive range, from looking back to the origins of our universe to predicting global climate change. Research efforts range from basic physics and chemistry, such as investigating the behavior of the smallest subatomic particles and examining how to combine atoms to create new nanotechnologies, to applied environmental sciences, such as developing technologies to burn fuels more efficiently and cleanly and developing environmentally and economically sustainable energy sources. Clearly, computational science advanced under SciDAC affects human populations on many levels. "The 21st century should pave the way to a millennium that excels in science, technology, and the way in which these disciplines interface with society. Advanced computing and computational science will be indispensable parts of the new ethos, and SciDAC will help lead the way," ( SciDAC Review, Spring 2006, p62).
SciDAC Program Elements
The expectation for new scientific discoveries drives the priorities of the Department of Energy (DOE). High-performance computing is one of the highest priorities in the DOE's 20-facilities plan. Moreover, the American Competitiveness Initiative has identified supercomputing as a priority among high-impact areas of research and development necessary to protect America's competitive edge for innovation in an increasingly competitive global marketplace. The DOE's effort in leadership-class computing infrastructure is an initial step towards this goal. Computational science at the petascale with hundreds of thousands of processors will face enormous challenges, but holds immense promise.
American Competitiveness Initiative
In the State of the Union Address on February 2, 2006, President Bush introduced the American Competitiveness Initiative to encourage American innovation and strengthen America's ability to compete in the global economy. A central component of this initiative is the doubling of the federal financial support to the most critical basic research programs in physical sciences and engineering over the next ten years. This initiative will focus on nanotechnologies, supercomputing, and alternative-energy technologies. The SciDAC program could contribute significantly to this initiative by providing technological advancements for simulation and modeling that can be brought to innovative science applications, to leadership computing facilities, and to large-scale and multiscale data processing, analysis, and visualization capabilities.
To take advantage of the opportunity offered by petascale computers, the SciDAC program is structured as a set of broad, coordinated technology investments to achieve breakthrough scientific advances through advanced computing technology, computer simulation, and knowledge discovery from large datasets from experiments and simulations. SciDAC intends to create a Scientific Computing Software Infrastructure that will bridge the gap between the advanced computing technologies being developed by the computer industry and scientific applications. SciDAC has several fundamental aims as follows. Firstly, SciDAC intends to create a new generation of Scientific Simulation Codes that take full advantage of the extraordinary computing capabilities of petascale computers. Secondly, SciDAC wants to make available Mathematical and Computing Systems Software that will enable the Scientific Simulation Codes to effectively and efficiently use terascale computers. Finally SciDAC is working to develop a Distributed Science Software Environment that will enable management, dissemination, and analysis of large datasets generated by simulation-intensive and experimental/observational- intensive science. The SciDAC Program consists of four major interrelated elements: (1) Science Application Software, (2) Centers for Enabling Technologies, (3) Scientific Application Partnerships, and (4) SciDAC Institutes.
Future Opportunities
There are many outstanding scientific challenges and opportunities that advanced computing technologies can help address. This list is not exhausexhaustive, but rather it is meant to be representative of the opportunities that can be exploited within the SciDAC program.

Figure 1. A simulation of the metabolic process that converts inorganic carbon into organic sugar. This snapshot from a ChemCell model of carboxysome organelles in the cyanobacterium Synechococcus illustrates the spatial and chemical complexity such a particle model can encode.		 Biology
The 21st century has been called the "biological century"–an era when advances in biology, spurred by achievements in genomic research, including the sequencing of the human genome, will bring revolutionary and unconventional solutions to some of our most pressing and expensive challenges in energy, the environment, and medicine. Current research programs are elucidating how living organisms interact with and respond to their environments. Such findings will enable us to use biological processes to produce clean energy, remove excess carbon dioxide from the atmosphere, and help clean up the environment.

There are multiple areas in which SciDAC may aid biological research. For example, SciDAC simulation technology may be applied to the systems-level of cells and to microbial communities. Predictive modeling of microbial behavior is a critical capability, as it will be required to harness microbial communities for energy production, carbon sequestration, and environmental remediation. In addition, SciDAC computing technology may help make sense of the plethora of biological data being generated by streamlining data analysis and model building. Analysis of experimental data and model building are key to reaping the full benefits of high-throughput experimental methods in biology. Indeed the avalanche of data being produced at the DOE's Genomes-to-Life facility represents a tremendous challenge for information technology, and SciDAC-developed technology can provide a means for efficiently analyzing immense genomic datasets. Furthermore, SciDAC advances can be applied to structural modeling and simulation for biological systems. Molecular modeling is a unique tool for obtaining new insights into the relationships between structure, dynamics, and function, at a level of detail that is difficult or expensive to obtain experimentally.

Figure 2. Molecular orbital of the benzene dimer with the adaptive grid and an isosurface. This visualization was generated by Multiresolution ADaptive NumErical Scientific Simulation (MADNESS), advanced quantum chemistry software developed through extensive collaborative efforts including SciDAC-funded projects.
Chemistry and Materials Science
Chemistry and materials science encompass a large and diverse set of research goals including fundamental chemical reaction dynamics, nanoscale science and technology, catalysis, nanomaterials synthesis, magnetism, and superconductivity. The computational issues in this field revolve around the ability to predict atomic geometry, electron charge density or magnetization of small atomic clusters or crystals, as well as more exotic properties such as superconductivity. Understanding chemical reaction dynamics is key to reducing emissions and improving combustion, refining petroleum, and producing fertilizer. Predictive models for nanomaterials synthesis are necessary to enable the broad application of new technologies based on the remarkable properties of zero-dimensional and one-dimensional nanostructures. There is an increasing interest in biomimetic materials and the quantitative study of biochemical processes, such as in protein structures and enzyme kinetics.
Elucidation of these issues will provide the insight needed to improve photovoltaics for solar energy, solid state lighting, hydrogen production and storage, combustion, and the conversion of biomass into usable fuels. Atomic scale processes are critical for climate modeling and environmental remediation, including heavy element (actinide) transport in soils. The computational challenges that arise within both theoretical science and DOE user facilities are enormous. Nonetheless, the opportunities to overcome those challenges in ways that will enable fundamental advances in science and more efficient operation of unique facilities are at least as immense.
The following applications represent just a few examples of the ways in which SciDAC may enhance research and development in chemistry and materials science. Nanoscience, widely recognized as a key for enabling technologies of the 21st century, will make possible the placement of individual atoms in clusters and on solid surfaces. This technology will enable scientists to control the synthesis and fabrication of novel materials and devices on a fine scale. SciDAC aims to provide integrated information and workflow management for the DOE's five Nanoscience Research Centers that will spur new discoveries by facilitating experiment-toexperiment and experiment-to-simulation comparisons. In addition, realization of the full scientific promise of the Linac Coherent Light Source (LCLS) will require the timely development and application of advanced computing tools for experimental design, data handling, and analysis– tools that LCLS does not have the expertise to develop alone. Furthermore, SciDAC can aide catalysis chemistry by providing leadership-scale computing that can be combined with continued improvements in theory and algorithm; the resultant computational simulation can enable the design of catalysts from first principles. SciDAC applications may also help reveal the electronic structure of molecules, clusters, thin films, and crystals. Specifically, there is a need to expand the current algorithms and theoretical models by at least one order of magnitude: from hundreds of atoms to thousands for high accuracy methods, and from thousands to tens of thousands of atoms to improve approximate methods.
Climate
Climate models must continue to be improved to meet the increasing demands of climate science and, to an even greater extent, of climate policy communities. More accurate climate models are needed on the regional scale so that the impacts of potential climate change can be ascertained with sufficient confidence that response strategies can be devised. Increased resolution not only provides better regional precision, but often results in more accurate simulation of the smallerscale processes, such as storms and clouds, that are critical for correct climate simulation.
There are three major areas in which climate science may benefit from SciDAC's efforts: carbon cycle modeling, coupled model integration and evaluation, and meeting the next-generation data requirements. The inclusion of carbon cycle processes into climate models, particularly those relating to ocean ecology and dynamic vegetation on land, is one of the most active areas of research today. Model integration, testing, and an evaluation facility are needed to rapidly incorporate new ideas so that researchers will be able to evaluate whether any proposed improvement in integrated model process representations, resolution, or numerical algorithms achieves the desired result. At present such evaluations are difficult and time-consuming, and they therefore impose high barriers to change. Finally, high-speed network bandwidth linking distributed archives of experimental, observational, and simulation data are required for scientific discoveries by the large national and international climate research communities. These next-generation data requirements can be met by advanced computing technologies developed by SciDAC.
Science Application Software
The chief source of acceleration in simulation-based science has been the strength and depth of partnerships among application domains, computer science, and applied mathematics. The success of such interdisciplinary teams and partnerships is the hallmark of the SciDAC program. Hence SciDAC will continue to invest in interdisciplinary research teams focused on scientific discovery and engineering innovation.
The scientific landscape of interest in the SciDAC program spans the breadth of scientific communities important to the DOE's Office of Science, the National Nuclear Security Agency, and includes areas of cooperation with the National Science Foundation. Of particular interest are research efforts focused on leading-edge problems in the following fields: accelerator science and simulation, astrophysics, climate modeling and simulation, computational biology, fusion science, subsurface reactive transport modeling and simulation, high-energy physics, nuclear physics, materials science, chemistry, quantum chromodynamics, radiation transport, and turbulence.
Key elements of successful SciDAC applications include both significant insight into, or solution to, a challenging problem of national scientific or engineering significance clearly related to DOE missions through computational science, as well as the creation of scientific simulation codes that achieve high single-node performance, scale up to thousands of nodes and tens of thousands of processors, and can be readily ported to other computer architectures.
Combustion
Approximately 85% of U.S. energy needs are currently met by combustion. Whatever mix of energy sources we utilize in the future, we will almost certainly continue to rely heavily on combustion. Given this major importance to energy production and use, we need to answer fundamental scientific questions that can be adequately addressed only with the application of advanced computing. Specifically these include the factors that control the production of particulate matter (soot), and of polluting gases, as well as the development of optimally efficient designs.
New combustion systems based on ultra-lean premixed gaseous burners have the potential to dramatically reduce pollutant emissions in combustion processes. Ultra-lean flames, however, are highly susceptible to fluid-dynamical combustion instabilities, making robust and reliable systems difficult to design. The range of possible fuels and fuel mixtures also adds complexity to the development of ultra-lean burners. Likewise, a more detailed understanding of mixing and reaction dynamics in non-premixed turbulent combustion is required and this can be achieved only from numerical simulation; the level of detail required is not accessible experimentally with current techniques and thus will benefit greatly from SciDAC technological developments. In addition, as the quality of high-fidelity simulations of laboratory-scale experiments in highthroughput laser imaging for combustion science increases, there will be an increasing emphasis on improving the accuracy of analysis algorithms and obtaining tools to facilitate the fusion of simulation and experimental data.
Figure 3. Gyrokinetic toroidal code (GTC) mesh for a single magnetic surface, showing equilibrium magnetic field lines (black), and values of perturbed electrostatic potential in the linear regime (colors).
Fusion Energy
Fusion as a power source is potentially inexhaustible, produces no troublesome emissions, is considered safe, and has few, if any, proliferation concerns. It creates no long-lived waste and runs on fuel that is readily available to all nations. Computational simulation techniques and distributed computing technologies are fundamental to the goal of developing environmentally and economically sustainable fusion reactors. For example, sustained fusion power generation requires that we understand the detailed physics of the fluid of ions and electrons, or plasma, in a fusion reactor. Moreover, a tight engagement of U.S. fusion scientists in the International Thermonuclear Experimental Reactor (ITER), to be located in France, will be essential both for it to succeed and for the U.S. to profit from its operation.
The decision to participate in ITER and the advances achieved in the first five years of SciDAC create both challenges and opportunities that the SciDAC program will be well positioned to address. The following are two representative examples: firstly, coupling computation will aid the experimental goals at ITER. Building on the successes of the SciDAC Fusion Collaboratory, SciDAC researchers will be able to perform quasireal-time analysis of experimental data from long-running experiments such as those planned for ITER; secondly, SciDAC technology can be applied in the building of ultra-high-fidelity predictive models. Advanced mathematical algorithms, leadership-class computing capabilities, and support for data-intensive applications will lead to a new ability to understand plasma evolution from a statistical perspective.
Subsurface Science
The DOE currently faces wide-ranging subsurface contamination problems and other challenges at diverse sites around the country. At such locations not only continued energy production, but also human health and environmental wellbeing, will require accurate assessment and prediction of subsurface flow and contaminant behavior. A next generation of mathematical models of subsurface flow and process simulation would provide the DOE with new discovery-class tools to handle the task of managing the legacy wastes from the nuclear research and weapons programs, and to address the challenge of carbon sequestration and the environmental impacts from energy production. The challenge is to develop predictive, scientifically defensible models for multiphysics-multiphase-multicomponent-multidimensional subsurface reactive flow and transport with suitably high resolution so as to account for naturally occurring subsurface heterogeneities. By leveraging advances in our understanding of physical, chemical, and biological processes, mathematical algorithms, and computational performance, an opportunity exists to build a discovery-class suite of subsurface simulation models for realistic site characterization and simulation.		  
Figure 4. The control rooms of NSTX (upper left), DIII-D (upper right), and C-Mod (lower) with shared display walls being used to enhance collocated collaboration. On the DIII-D shared display is video from remote collaborators in Europe who were participating in that days experiment.
Development of the next-generation suite of models will require the following four new initiatives. Firstly, new conceptual models that fully couple key physical, chemical, and biological processes need to be developed in order to address the problems of environmental contamination, carbon sequestration, and energy production. Secondly, new computational mathematics, computer science, and algorithms are needed to enable effective high-performance computing. These advances will need to include enhanced resolution, time adaptability, spatially accurate threedimensional models, new multiscale mathematics for bridging physical scales, new algorithms and mathematics for inverse modeling, and new data fusion techniques using multiple data sources to extend understanding. Thirdly, experimental facilities in the laboratory and the field that are dedicated to the validation of model results are needed. And finally, a robust Problem Solving Environment (PSE) is needed to enable more efficient generation of simulation input, management of large, diverse datasets sets, and visualization to effectively communicate concepts between scientists, engineers, policy makers and the public.
Opportunities to Broaden the Link to Experimental Sciences
In addition to opportunities in the applications mentioned above, each of the Office of Science programs is making extensive efforts to support the experimental sciences and the establishment of new facilities as outlined in the DOE Office of Science's "Facilities for the Future of Science" report. Several of these new experimental facilities, either under construction or in the planning stages, involve significant investments in both experiments and simulation. This environment will lead to new opportunities for the integration of experimentation with theory, modeling and simulation for the advancement of science. In many cases, the result is new informatics science, which requires new data management and analytic approaches to support the discovery process. In SciDAC, new program areas will be added to address this important new area of computational science.
Centers for Enabling Technologies
Centers for Enabling Technologies (CETs) address the mathematical and computing systems software environment element of the SciDAC scientific computing software infrastructure. CETs address the following needs: new algorithms that scale to parallel systems that have hundreds of thousands of processors; methodology that can achieve portability and interoperability of complex high-performance scientific software packages and libraries; operating systems and runtime tools and support for application execution performance and system management; and effective tools for feature identification, data management, and visualization of petabyte-scale scientific datasets. CETs also address the opportunity for advanced computing technologies to improve experimental sciences and user facilities.
CETs provide the essential computing and communications infrastructure for support of SciDAC applications through the development and application of a new generation of data management and knowledge discovery tools for the large datasets obtained from large experimental facilities and from high-end simulations. A successful CET program emphasizes collaboration with, and impact on, multiple SciDAC application teams, creates horizontally integrated, multi-institutional CET teams, and also provides stable, long-term expertise for core technology development. It is the aim of SciDAC that the assembly of CETs together span a breadth of interdependent, relevant topics and that they maintain sufficient flexibility to respond to near-term opportunities
Common areas that can be exploited, either through collaborative application projects or through crosscutting technology projects, include distributed data management and computing, information management and knowledge discovery, and cyber security for open facilities. Climate science, physics, biology, and nanoscience (among others) face similar needs for the distributed management and analysis of large volumes of heterogeneous data produced at different facilities, analyzed by means of often complex workflows, and consumed by a large and distributed user community. In addition to large data volumes, DOE experimental science disciplines face challenges associated with increasingly complex data due to different experimental modalities and analysis techniques. Again, there are opportunities to achieve technological and methodological advances that benefit multiple disciplines. DOE user facilities also face the difficult task of operating open facilities in an increasingly hostile security environment while maintaining utility of the facilities and preserving the free flow of information.
Looking to the future, there is a timely and significant opportunity to increase both the utility of DOE experimental facilities and the effectiveness of DOE experimental science through judicious and targeted application of advanced computing and information technology. There are several driving factors propelling the attainment of these aims. Firstly, DOE experimental facilities and science are of profound importance to the nation. Secondly, experimental scientists currently face urgent challenges as a result of rapidly growing data volumes and new experimental techniques. And finally, there has been significant progress in advanced computation, distributed computing, and interdisciplinary and interlaboratory collaboration achieved within the SciDAC program. A broader SciDAC program can empower experimentalists by automating important steps in the experimental process, from the operation of experimental facilities to the informed design of experiments through simulation techniques, to the collection, comparison, and analysis of experimental data and model results.
Software Opportunities for High-Productivity Computing Systems
Sustained, year-after-year exponential growth in computing, storage, and networking capabilities has resulted in an ongoing revolution in information technology. This trend, along with the clear embrace of massive parallelism, drives the advance of ultrascale computing and information processing capabilities. Moreover, the computational science community has been through a decade of stable architectural and programming models for highend computing, namely the Message Passing Interface (MPI) on distributed-memory, cache-based machines. A significant share of the credit for the success achieved can be attributed to this stable growth. The size and type of systems that will become available over the next five to ten years will differ dramatically from these established norms for high-end computing. With the Office of Science's emphasis on leadership computing, systems at the facilities will grow beyond 100 Tflop/s in the near term and will be approaching petascale performance by 2008–2009, with systems consisting of 20,000 to 100,000 processors. Scientific codes currently running on thousands of processors need to scale to tens of thousands or even hundreds of thousands of processors. Many additional codes will need to make the jump from hundreds of processors to thousands of processors. In short, the next ten years promise to be more tumultuous, with multi-core and vector processing entering (or reentering) the high-end computing landscape. Although this could potentially be disruptive to programming models and represents a challenge for the community, it is an opportunity for the SciDAC program to advance the field through this transition.
SciDAC Institutes
Science Institutes have performed extremely useful roles in a number of science communities. Examples include the Kavli Institute for Theoretical Physics, the Aspen Institute for Physics, and the Mathematical Sciences Research Institute. To date, no such community-based institution exists for the computational sciences community.
The SciDAC Program will establish SciDAC Institutes to provide for the sustained infusion of new ideas and community focus to enhance the SciDAC program. The SciDAC Institutes are conceived as university-led centers of excellence intended to complement the efforts of the other three SciDAC program elements. As such they will provide a unique forum for discussion of fundamental issues affecting scientific discovery. SciDAC Institutes will be loosely patterned on successful institute models in other scientific communities. Although it will not be necessary for SciDAC Institutes to have a single location, the facilities used must be conducive to research and collaboration. The Institutes will sponsor a variety of activities, which will differ in format to satisfy different purposes and needs. SciDAC Institute activities will be diverse including the following: topical programs, proposed by the SciDAC community, to address a particular problem of critical interest; short courses and summer education programs to give researchers at all levels training in new highperformance computing concepts, technology, techniques, and software; collaborative workshops to facilitate the interaction of leading researchers working on related problems; pre-doctoral and post-doctoral fellowships for young computational scientists; opportunities for visiting scientists and university faculty sabbaticals; and "coding camps" to facilitate rapid progress in software development.
Many areas of science are at a critical juncture where the infusion of novel ideas will enable future scientific discoveries whose impact can only be imagined today. As computational technology and computing power continue to advance, computation clearly will play an even more important role in these new discoveries. Programs that train future generations of computational scientists and that provide forums for bringing together leading scientists, computer scientists, and applied mathematicians to tackle critical scientific problems will be essential in enabling such advances. The SciDAC Institutes will be established to accomplish these objectives.
MSciDAC Institutes will benefit the DOE and the scientific community at large by building a broader community of researchers who understand the challenges of providing and using high-performance modeling and simulation capabilities and are willing to address these problems collaboratively. SciDAC's most significant accomplishment to date has been to enable breakthrough scientific discoveries by engaging applications scientists, computer scientists, and mathematicians within the DOE Office of Science community in new ways. Involving the broader community of American and international scientists in the activities of scientific discovery through advanced computation will increase the impact of the SciDAC model for collaborative science, and will benefit science in the U.S. as a whole.
In order to ensure that computing platforms are well suited for the complex applications anticipated over the next decade, SciDAC will develop partnerships with computer vendors. Such partnerships will go beyond mere evaluation of planned platforms and will involve the following developing and implementing strategies for scalable application software to enable the efficient utilization of petaflop computers within scientific communities of national priority. We intend that these partnerships will develop scalable, fault tolerant system software and runtime technologies to enable the efficient utilization of petaflop computers and enable the tools to more fully automate the performance tuning of applications for a given computer system and thus to increase the productivity of the application development environment for petascale systems. The partnerships will also abstract the requirements of scientific applications so that they can be addressed in hardware design and develop the software environments that will allow scientists to extract the maximum performance and capability from that hardware.
Scientific Computing Hardware Infrastructure
Leadership-Class Computing Facilities
For the past 40 years, the DOE has been a world leader in using supercomputers to advance scientific research. In 2004, the DOE established a Leadership Computing Facility (LCF) as a partnership between Oak Ridge National Laboratory (ORNL), Argonne National Laboratory (ANL), and Pacific Northwest National Laboratory (PNNL) for open science. The LCF is the realization of the Ultra-Scale Scientific Computing Capability, which was identified as a priority in the DOE's Office of Science's "Facilities for the Future of Science" report. According to this plan, this facility will result in a 100-fold increase in the computing capability available for open, unclassified scientific research. This shift in landscape toward leadership-class computer architectures will have a profound impact on the kind of research that will be possible in SciDAC
Today, the computing landscape in the DOE Office of Science includes a comprehensive range of different classes of facilities for meeting the diverse computing needs of the research community. These facilities include the LCF, which provides computing resources for priority scientific applications and serves as a focal point for an application community to ensure the leadership of the United States in critical areas of science, and the National Energy Research Scientific Computing Center (NERSC) facility, which provides highend computing resources for all of the Office of Science with a mixture of capacity and capability computing. The DOE Office of Science also recognizes the importance of specialized application computing facilities offering specialized hardware to be exploited by unique features of the application. An example of such a system is the QCDOC machine (BNL). A key metric in the deployment of these systems is cost-effectiveness. At least one facility that can assess the promise of experimental and new computing technologies for scientific applications is also needed. It is expected that the function of experimental computing facilities supported in the past several years within the Advanced Computing Research Testbed (ACRT) at ORNL will be realized within DOE's Advanced Scientific Computing Research (ASCR) Research and Evaluation (R&E) program in fiscal year (FY) 20060.

High-Speed Communications Network
As part of its mission, the DOE has taken a leadership role in providing advanced networking. For 20 years, ESnet has been widely used by application scientists. ESnet traffic flow indicates that networking bandwidth usage has increased by a factor of ten every 46 months since 1990, and all indications are that this exponential growth will continue through the next five-year period of SciDAC. The establishment of the specialized and ultra-high-scale computing platforms discussed above is expected to place increasing demands on wide-area networks (WAN). Future networking requirements are discussed in the report "DOE Science Networking Challenges: Roadmap to 2008."
Figure 5. Side (upper) and from above (lower) views of the structure of benzenedithiol molecules adsorbed on the surface of gold. This study uses grand canonical ensemble Monte Carlo methods.
However, making wide-area network "pipes" bigger will solve only part of the problem. To achieve science and technology goals, the end to end problem must also be solved; that is, the data must make it all the way from the end source to the end sink at the desired speeds and with the desired levels of quality of service. The source drivers are diverse, including a variety of high-end computational systems, experimental raw data sources, and massive data stores. Examples of data sinks include high-end computational systems, massive data stores, visualization systems, and a nearly infinite sea of low-end computational systems.
The challenge is to integrate the science requirements, end sources and sinks, transport protocols, jitter control, packet switching, circuit switching, routing, WAN components, and other aspects into a system that meets the end-to-end requirements and thereby the overall science goals. The SciDAC approach is ideal for meeting this challenge because it will optimally enhance scientific discovery potential by vertically integrating networking capabilities from science applications from the top down through fundamental network technologies. SciDAC will need to address significant technical issues in order to meet the expected quality of service.
Science Application Partnerships
Science Application Partnerships (SAPs) are formed as a result of targeted efforts to integrate advanced applied mathematics and computer science technologies into specific SciDAC applications projects. Although the technical focus of SAPs is similar to that of the CETs, the two program elements are complementary. First, the work of mathematicians and computer scientists funded by SAPs is focused on the needs of a specific science application team. CET efforts, on the other hand, are larger and focus on interaction with multiple application teams. Second, SAPs provide the mechanism to supply application teams with the computer science and mathematics expertise that are not covered by CET teams, and thereby fill important technical gaps. To ensure continued success of SAPs, SciDAC will implement the strategies that have proven successful in the past, and furthermore, augment this program element to better address the needs of application scientists. SAPs will focus on specific, targeted application science needs of a specific area; they fill topical gaps not covered by CETs; and they will work to maintain a cohesive interdisciplinary project with a shared goal. SAPs will collaborate with science communities not yet familiar with the SciDAC model of interdisciplinary teams and highperformance computing and therefore prepare the nextgeneration of SciDAC applications.
Pathway to Sustained Petaflops Computing
With the priorities of the American Competitiveness Initiative, the DOE's 20-year facilities plan, and numerous other reports and recommendations from the science, technology, and business community, it is clear that leadership in supercomputing is a significant national priority. Given the now common expectation of exponential growth in computing power expressed in "Moore's Law," achieving sustained petaflop computational performance is an ambitious but reasonable goal. Moreover it is an achievement that may motivate the nation's science and technology communities.
The DOE/ASCR has established a program pathway to petaflop computing and beyond that will be executed in parallel with SciDAC over the next five years. This plan includes upgrading the ORNL LCF to provide more than 250 teraflop peak capability by the end of FY 2007, and 1,000 teraflops ("peak petaflop") by the end of FY 2008. The plan also includes acquiring a 100 teraflop IBM BlueGene/P high-performance computer system at ANL in FY 2007, which will create the Argonne LCF, and increase capability to the range of 250–500 teraflops by the end of FY 2008. Meanwhile NERSC will be upgraded to a peak capacity in the range of 150 teraflops by the end of FY 2007, and to a 500 teraflop peak capacity by the end of the decade. Furthermore the ESnet will evolve over the upcoming 5-year period to include dual backbone rings at 40 Gb/s with fault tolerant 10 Gb/s connections to most major Office of Science laboratories, and even higher bandwidth connections to NERSC, the LCFs, and other sites with exceptional data requirements.
Execution of this ambitious plan will return the DOE to the position of being a world leader in computational science, consistent with the 20-year facilities plan. Research into high-performance computer architectures will involve a coordinated partnership with the National Nuclear Security Administration (NNSA) and will be focused on the Defense Advanced Research Projects Administration (DARPA) High Productivity Computing Systems (HPCS) program partnership. This is in recognition of common features of the computational science needs and workload at DARPA, NNSA, and DOE/SC. The central aim of the HPCS program is to have computing systems that achieve "sustained petaflop" performance by 2010–2011.
Priorities for SciDAC's Future
The next five years will see significant change in computational science. The first petascale systems will become available enabling petabyte datasets to be generated by experimental facilities; these enhanced capabilities will enable new scientific breakthroughs. Centers for Enabling Technologies (CETs) and Science Application Partnerships (SAPs) must build on their current successes in both application partnering and technology development. In particular, the next set of CETs and SAPs must continue to meet the current needs of the scientific applications teams, as well as address the technology gaps that will emerge with petascale computing systems and data-intensive experimental facilities.
The elements of the SciDAC program play an important integrating role beyond the specific goal they support. They enable vertical integration within ASCR and horizontal integration across the Office of Science. Within ASCR, CETs and SAPs allow new ideas generated in the base research program to be deployed on the Office's high-end computing facilities and networks and to be propagated to the larger national and international computational science community. Conversely, the close coupling of CETs and SAPs with their applications teams can help them identify future directions for ASCR base research programs.
The SciDAC program complements the experimental programs supported by the DOE. SciDAC's modeling and simulation effort will draw heavily on the DOE's experimental programs to provide basic insights into and data on complex systems as well as the fundamental processes that govern their behavior. The experimental programs also provide a means of validating computational models and simulations; and validation is essential to advancing the state of the art in scientific simulation. In turn, computational modeling and simulation advances will provide insights into fundamental physical, chemical, and biological processes that would otherwise be unattainable. These advances will maximize the return on investments in experimental facilities by more quickly and efficiently harvesting scientific results, and will also provide the technical capability to design and construct innovative new experimental facilities.
Computational modeling and simulation provide a strong link between the Office of Science and the other DOE offices. By demonstrating how realworld devices and systems can be simulated from a knowledge of the fundamental physical, chemical, and biological processes involved, the investment in scientific computing connects the basic research programs in the Office of Science with the energy and environmental research and development programs in the Offices of Fossil Energy, Energy Efficiency, and Environmental Management.
Contributor: Dr. Jack Wells, Oak Ridge National Laboratory, on behalf of The Working Group

 
Published by IOP Publishing in association with Oak Ridge National Laboratory, for the US Department of Energy, Office of Science. Copyright © 2006 by IOP.