DOESciDAC ReviewOffice of Science
Science for Problems Under the SURFACE
Humans have entered into and explored a wide range of environments, ranging from the deep ocean to the upper atmosphere and even outer space. But for the most part, the subsurface environment—the realm hidden beneath the surface of our planet—remains enigmatic and directly observable only through limited points of access. Nevertheless, physical, chemical, and biological processes in Earth's subsurface are central players in several interrelated energy and environmental issues critical to the world's security and economy. Leadership-class computing will soon be brought to bear on understanding and predicting these processes across a wide range of time and space scales.

The United States is faced with severe geopolitical, economic, and environmental challenges associated with energy production and use. Several of these challenges are fundamentally driven by the nature of complex processes occurring in the Earth's subsurface environment. High-profile examples include carbon sequestration for mitigation of climate change, production of fossil fuels, and long-term geological solutions for the disposal of wastes generated by nuclear energy production.
In 1950 the U.S. was the world's leading oil producer, and imports were negligible. By the time of the 1973-1974 Arab oil embargo, the production of oil in the U.S. had passed its peak, and approximately one-third of the oil consumed nationally was imported. Today, the U.S. imports well over half of the oil that it consumes, much of it from countries that are unstable and/or unfriendly to the U.S. The rapid rise in the cost of transportation fuels (primarily liquid hydrocarbon) is having a substantial negative impact on the American economy and balance of payments with other countries, and the increased reliance on foreign oil producers threatens our security.
Several challenges are fundamentally driven by the nature of complex processes occurring in the Earth's subsurface environment.
Figure 1. A schematic diagram of options for geological storage of carbon dioxide.
Despite these adverse secular trends, the U.S. is an energy-rich nation. Its reserves of unconventional hydrocarbon—including oil shale, heavy oil, coal bed methane, and natural gas from "tight" formations—far exceed the oil reserves in the Middle East. In addition, the economically recoverable oil reserves in the Alberta oil sands are comparable to the proven oil reserves in Saudi Arabia, and technological advances may considerably increase the amount of oil that can be extracted from this source. The U.S. also has abundant coal reserves, which can be used directly to generate electricity or be converted to liquids. In addition, the energy content of methane hydrates far exceeds that of all other fossil fuel sources combined, but technological challenges that must be overcome to economically recover methane from this widely distributed potential energy source have not yet been overcome.

The large-scale production and consumption of fossil fuel has substantial potential environmental impacts that must be mitigated. In recent years, concern about the effects of carbon dioxide generated as a result of fossil fuel combustion and production (in situ coal gasification and partial in situ oil shale combustion, for example) has increased to the point at which it seems almost certain that large-scale carbon capture and sequestration (figure 1) will be required in the future. This complicates decisions the U.S. must make regarding its energy future since the amount of carbon dioxide generated from the combustion of imported oil, liquefied natural gas, or liquids synthesized from natural gas is substantially less than the amount of carbon dioxide generated by the production and combustion of oil sands and some of the more plentiful indigenous fossil fuels. The U.S. produces about one billion tonnes of coal per year. The amount of carbon dioxide produced per tonne of coal depends on the type of coal, but combustion of one billion tonnes of coal will generate about two billion tonnes of carbon dioxide. The current U.S. oil consumption is also about 1 billion tonnes per year, and this produces about three billion tonnes of carbon dioxide. Since 1996, Statoil has sequestered about 1 million tonnes per year of carbon dioxide near the Sleipner gas field in the North Sea. This is the largest carbon dioxide sequestration project. The Department of Energy (DOE) is currently exploring a number of scientific avenues related to carbon sequestration, including injection into deep saline formations (sidebar "Simulating Carbon Sequestration Strategies," p34).

There is also substantial concern about the impact of fuel recovery processes—in situ coal gasification, the future large-scale production of liquid fuel from oil shale and oil sands, and others—on soil and water quality. Public awareness of the value of abundant clean water has increased substantially in recent years, and concerns about water quality and complex legal issues related to water rights could constrain the development of some of the more abundant energy resources in the U.S.
The economic extraction of fuels from the abundant sources in the United States, as well as the mitigation of environmental impacts and concerns about potential environmental impacts, will depend on our ability to understand and control subsurface processes.
The total world geothermal electric generating capacity is only on the order of 10 gigawatts (compared with a total U.S. generating capacity that now exceeds one terawatt), and many of the highest-grade sites have been developed. However, the potential geothermal electric generating capacity is much larger if the costs of generating electricity from less favorable, lower-grade, sites can be reduced. Relative to fossil power plants, geothermal plants are environmentally benign. Rare geological systems such as geysers have been destroyed in the past, but this is not likely to be an important issue in the future if the technology needed to produce electricity economically from less unique sites is developed.
Figure 3. Extraction of subsurface energy reserves is a complex and expensive undertaking. Better understanding and prediction of subsurface processes will reduce risks and enhance productivity. This photograph shows the Troll A platform being towed 174 miles to the Troll gas field in the North Sea. With a height of 472 meters and a mass of 656,000 tons, this is the largest man-made structure that has ever been transported. The tow, which required seven days, employed ten of the most powerful and advanced tugs.
The economic extraction of fuels from the abundant sources in the U.S., as well as the mitigation of environmental impacts and concerns about potential environmental impacts, will depend on our ability to understand and control subsurface processes (figure 3). The experiences of the oil industry provide a relevant paradigm. Technological advances including better acquisition of geophysical data and their analysis using high-performance computing, directional drilling, and better reservoir simulations have resulted in a steady improvement in oil recovery factors and increased recoverable reserves. Nevertheless, most of the oil—typically about 55% in large reservoirs and 65% in smaller reservoirs (but sometimes much more)—remains in place when the reservoir is abandoned for economic reasons. Given the scale of the enterprise, the economic value of the technological advances has been enormous, and the opportunities for additional gains are equally impressive. To illustrate, assuming that four trillion barrels of oil recoverable with current technology remain out of an initial complement of five trillion barrels, an increase in the average recovery factor from 40.00% to 40.01% would produce an additional billion barrels of oil worth more than $60 billion at today's prices.
The potential climate impacts of fossil fuel energy utilization have recently created new interest in nuclear energy as a significant component of the world's energy production. In addition to public concerns regarding environmental and health risks associated with nuclear plant operation, the safe disposal of waste products remains a major hurdle. Although the DOE originally agreed to begin accepting nuclear waste from civilian reactors in 1998, the U.S.'s primary high-level nuclear waste repository remains in the planning stage because of legal and regulatory constraints, and it is currently scheduled to open in 2017. The proposed Yucca Mountain site in Nevada relies heavily on subsurface isolation of wastes in an arid environment, and the long-term (106 years) prediction of water migration pathways (figure 4, p36) is a central element of decisions regarding the safety of the proposed system.
Figure 4. A schematic diagram of water flow pathways in the proposed high-level waste repository at Yucca Mountain, NV.
There are also important challenges that are not as directly related to energy production. For example, the DOE is responsible for the hazardous radiological and chemical "legacy waste" that was buried in or injected into the subsurface during the development and manufacture of nuclear weapons and nuclear reactors. More cost-effective ways of isolating subsurface contaminants are needed. If the uncertainties associated with computer models used to predict the behavior of subsurface contaminants could be reduced, less conservative and less expensive remediation could be used, and in some cases stakeholders could be convinced that no action is needed. Contamination of the subsurface due to chemical spills, leaky fuel tanks, and the improper disposal of hazardous materials is also an important concern, and the effective treatment of the many sites that have been contaminated in this manner will be very costly.
The intrusion of salt water into coastal aquifers is an important problem, and the potential contamination of aquifers by agricultural chemicals is of concern. Desalination is economical for drinking water and some industrial applications, but the large-scale desalination of water for agricultural use is not practical. Rising sea levels associated with global warming is expected to exacerbate the intrusion of salt water into coastal aquifers. The build up of salt and minerals in irrigated soil is also a major problem related to the fluid flow and reactive transport in the subsurface.
Critical aspects of contaminant movement and fate can be controlled by very small microfractures.
Figure 5. A schematic diagram of the range of length scales involved in subsurface processes.

The Role of Leadership-Class Computing
The application of high-performance computing to subsurface science and technology will play an important role in the energy future of the U.S. The three most important opportunities are:
  • More economical extraction of fossil fuel from the subsurface and increased recoverable reserves
  • Protection of our water resources
  • Reduction of the impact of fossil fuel extraction and combustion on global climate change through carbon dioxide sequestration
The social, economic, and security impacts of these applications would be difficult to exaggerate. In addition, the renaissance of nuclear energy is also expected to play an important role in the energy future of the U.S. There are two main challenges related to subsurface science and technology:
  • Mining of nuclear fuel sources with an acceptable environmental impact
  • Safe disposal of spent nuclear fuel in geological repositories—a much more important challenge than the first

From a fundamental science and computational point of view, the subsurface is extremely challenging. One of the most important challenges arises from the fact that the subsurface is hidden from direct observation. It is possible to examine very small samples taken from a small number of locations (usually at great cost, on the order of $10 million for an offshore well in deep water), but on the scale of practical applications, characterization of the subsurface is based primarily on low-resolution, indirect physical measurements such as seismic and electrical resistivity tomography and ground-penetrating radar. High-performance computing plays an important role in the interpretation of the large quantities of data obtained from these measurements, but the uncertainties associated with the characterization of subsurface systems are large.
Another challenge is the inherent complexity of the subsurface. Subsurface systems are formed over a very broad range of time scales by a large variety of geophysical and geochemical processes acting on a wide range of length scales. In addition, subsurface environments often support a rich community of microorganisms. Microorganisms can be used to treat organic contaminants (liquid fuel and chlorocarbons, for example), and this process can be accelerated by introducing nutrients in a process known as accelerated bioremediation. The subsurface may also be contaminated with viruses, microorganisms, and microorganism spores, which may be hazardous to humans, farm animals, or plants. The range of length scales that must be taken into account is very large, from atomistic processes taking place on the 10-10 m scale, to contaminant plumes, oil reservoirs, and aquifers on the 102-105 m scale (figure 5). The range of relevant time scales is even larger. The time step in a typical molecular dynamics simulation is on the order of 10-15 seconds, but Environmental Protection Agency (EPA) regulations require the Yucca Mountain high-level radioactive waste repository to protect the public for 106 years (greater than 1015 seconds), and some geological processes relevant to the subsurface entrapment or formation of fossil fuels take place on time scales of the order of 1017-1018 seconds.
Figure 7. Backscattered electron images of granitic rock fragments obtained from beneath a spill at the Hanford site. Uranium contamination shows up as white in microfractures within mineral grains. P = plagioclase; K = potassium feldspar; M = mica. Remarkably, these rock fragments account for less than 4% of the total sediment mass, yet contain greater than 90% of the uranium.
Because of the inherent complexity of the subsurface there is substantial risk that predictions essential to the assessment of environmental impacts will be compromised by inadequate understanding and/or information. For this reason, extreme computing—in the very near term, computing at the petascale—should be applied judiciously to subsurface application. Unless models are properly validated, based on the correct physics and information, better algorithms and faster computers will not lead to more accurate and reliable results. This is a particularly serious issue in subsurface applications because of the complexity and incomplete characterization inherent to this set of problems.
For the same reasons, verification of computer code developed for subsurface applications is also extremely difficult or impossible, depending on the degree of rigor required. This problem is exacerbated by the perturbation to subsurface systems due to the emplacement of measuring devices, the difficulty of placing detectors in critical flow paths, and phenomena such as intermittent flow and/or path switching, which may be difficult to distinguish from detector malfunction. In shallow subsurface systems, for example, fluid flow may be dominated by flow in fractures that occupy only a very small fraction of the system. A very dense array of detectors would be required to adequately sample the fractures, and this would lead to a large perturbation. The outcome of a field-scale experiment may be critically dependent on difficult-to-detect subsurface features such as thin impermeable layers (which would very likely be punctured by the emplacement of detectors) and fractures.
Extreme computing can play a critical role by enabling a "first principles" approach to the modeling and simulation of subsurface processes to improve understanding and investigation of a broad range of relevant scenarios.
Figure 8. A schematic illustration of molecular-scale complexity at mineral-water interfaces, which includes variable topography and distribution of reactive surface sites, various kinds of adsorption, diffusion, incorporation, and precipitation reactions that define solute-surface and solute-cell interactions.
Some processes are very sensitive to seemingly minor variations. For example, the presence of clay smears on fracture surfaces can convert what would otherwise be a preferred flow path into a barrier to flow. Critical aspects of contaminant movement and fate can be controlled by very small microfractures (figure 7). Quantitative comparison between the results of computer simulations and laboratory experiments can help, but experimental systems usually lack the complexity of real subsurface systems, and natural length scales associated with important subsurface processes may exceed the scale of the experiment. Large-scale laboratory experiments, sometime referred to as "mesoscale experiments," can help. However, they are expensive, time-consuming, and lack the complexity of real subsurface systems. Extreme computing can play a critical role by enabling a "first principles" approach to the modeling and simulation of subsurface processes to improve understanding and investigation of a broad range of relevant scenarios.
Figure 9. A schematic diagram showing the potential integration of genomics-based models of cellular function (in silico models) with models of flow and reactive transport in the subsurface environment.
Figure 10. Simulated particle paths (multi-colored spheres) within a computed velocity field based on finite volume simulation of fluid flow partial differential equations at the pore scale. Individual soil grains are visualized as sets of planar surfaces in a triangular mesh, shaded to show three-dimensional structure.

Simulation Approaches and Research Directions
Because of the complexity of subsurface systems, a variety of modeling and simulation approaches must be used to develop a better understanding of subsurface processes and address important practical problems. For convenience, these can be subdivided into five scales.
Atomistic scales. The most commonly used methods used to address atomistic processes are ab initio quantum mechanics, molecular dynamics, and Monte Carlo simulations. Important challenges that require petascale and greater computing capabilities include accurate prediction of the kinetics of reactions at mineral-aqueous phase interfaces (figure 8, p39), the interactions between microorganisms and mineral surfaces, the behavior of complex molecules such as biopolymers siderophores (molecules, secreted by microorganisms, that complex strongly with iron and promote dissolution) in the vicinity of interfaces, and the adsorption/desorption of heavy atom f-electron species.
Microscopic processes. Methods such as dissipative particle dynamics, Brownian dynamics, and Monte Carlo simulations are widely used to simulate microscopic processes. Applications that require high-end computing include the simulation of biopolymers and colloids, including biocolloids. Integration of genomics-based models of cell function into geochemical reaction systems (figure 9) is another area with potential for significant improvement of models but may require extensive computational resources.

Even relatively simple subsurface processes span a wide range of length scales and time scales and require a combination of computational methods.
Pore scale. Pore network models with simplified pore geometries and simplified physics and chemistry have been used quite extensively to simulate pore scale processes (figure 10). More recently, methods such as lattice Boltzmann simulations, smoothed particle hydrodynamics, and computational fluid dynamics, which are based more firmly on first principles (but require more capable computing systems) have been used to simulate pore scale physical and chemical processes. Petascale computing will be required to simulate multiphase fluid flow under the full range of conditions applicable to subsurface and with the full complexity revealed by experimental investigations.
Laboratory scale. Simulation on the scale of typical laboratory experiments and mesoscale—scales up to O(10 m)—are challenging. It is not possible to fully characterize the pore geometry (and/or fracture network geometry) on this scale. Consequently, statistical methods must be used to construct a model for the pore space geometry and physics models, such as invasion percolation-based models, can be used to simulate processes such as fluid-fluid displacement.
Reservoir scale/field scale. To simulate subsurface systems on the large scales required for most practical applications (figure 11), the subsurface system can be represented by spatially varying property fields (porosity, permeability, partition coefficient, reactivity, composition, and so on) and/or sub-domains with individual properties. In many cases creation of an adequate model for the subsurface system can be more challenging than numerically solving the partial differential equations that describe fluid flow, reactive transport, and biological processes. This can be accomplished using some combination of information from geophysical measurements, geological interpretation and insight, process-based modeling, statistical modeling, and information from exposed analog sites (outcrops, road cuttings, excavations, and other data sources).
Figure 11. A seven-component, three-phase simulation of the alternating injection of water and gas to improved oil recovery. This simulation, performed by Dr. Sunil G. Thomas and Dr. Mary Wheeler at the University of Texas, illustrated how computer simulations can be used to explore various production scenarios with the objective of evaluating oil production strategies. The reservoir dimensions are 100 feet (depth) × 3,500 feet × 3,500 feet, and the simulation required 24 hours on 64 processors. The upper panels show the permeability field (left) and water saturation (right) at 800.0 days. The lower panels show oil saturation (left) and gas saturation (right) at 800.0 days.
Even relatively simple subsurface processes span a wide range of length scales and time scales and require a combination of computational methods. For example the motion of a fluid-fluid-solid contact line couples atomistic, microscopic, and continuum processes covering length scales ranging from the order of 10-10 m to 10-6 m or greater, and a much longer range of time scales. And reactive transport often couples processes occurring on length scales ranging from the order of 10-10 m with processes involving scales that can vary from the microscopic to the order of 103 m or greater, depending on the circumstances.
Very often, the outcome of subsurface processes depends on the coupling between many processes. For example, the in situ production of liquid hydrocarbon from oil shale involves many coupled organic and inorganic chemical reactions, multiphase fluid flow, and reactive transport. Bioremediation also involves the coupling between a large number of processes, including the growth of biofilm under nutrient-rich conditions, deformation of the soft rheomechanically complex biofilm, fluid flow, and complex intracellular and extracellular chemical processes. Two new SciDAC-2 science applications were funded to address issues of scaling and coupled multicomponent reactions in the next generation of subsurface flow and transport models (sidebar "Simulating Carbon Sequestration Strategies," p34; sidebar "Multiscale Simulations of Subsurface Biogeochemical Processes," p37). The accurate and reliable simulation of complex coupled multiscale processes based on first principles will require new information and understanding, better algorithms, and computing capabilities that far exceed those available today.
Contributors: Dr. Tim Schiebe, PNNL; Dr. Paul Meakin, Idaho National Laboratory; Dr. Peter Lichtner, LANL; Dr. Dave Zachmann, Colorado State University