SciDAC Review - Simulating STAR POWER on Earth

THE GYROKINETIC PARTICLE SIMULATION CENTER PROJECT

Simulating STAR POWER on Earth

BY KIM KRIEGER

The Gyrokinetic Particle Simulation Center (GPSC) is one of several SciDAC projects that use high performance computational techniques to research harnessing the energy source of the stars. One of the major achievements of the GPSC has been improving understanding of particle and energy transport and heat diffusion for realistic fusion plasma systems.

Fig. 1. A 3D visualization of electrostatic potential in a global, self-consistent GTC simulation of plasma microturbulence in a magnetic fusion device.

Our nearest star, the Sun, may be considered our parent star. Without this thermonuclear power source, life would certainly not exist on our planet. Almost every living being on Earth uses, directly or indirectly, the energy that is produced by nuclear fusion in the Sun and transmitted to us in the form of light and radiation.

Harnessing fusion energy for use in controlled reactors on Earth has been a goal for decades. Fusion fuel - or "heavy hydrogen" - is readily available in sea-water, and fusion reactions offer the benefits of being both clean (i.e. they generate no dangerous waste material) and relatively well controlled. But these attractive aspects of the potential of fusion power are offset by the scientific complexity and technical challenges of obtaining it in practice. For one thing, the temperatures required to ignite the fusion of hydrogen nuclei exceed 10 million° C. At such temperatures hydrogen atoms are stripped of their electrons, and the resulting gas, consisting of hydrogen nuclei and electrons, is called a plasma (see sidebar "Fusion and how it works," p47). Figure 1 shows a plasma simulation by the GTC team. High temperatures translate into higher energies of motion for plasma nuclei (also called positively charged ions), and sometimes two heavy hydrogen nuclei will collide with sufficient energy to fuse together into a new element: helium. This fusion reaction results in the conversion of a small amount of nuclear mass into energy in accordance with Einstein's famous E=mc2 equation, with c, the speed of light, accounting in part for the great amount of energy released in a nuclear fusion reaction. Fusing a single kilogram of hydrogen can produce as much energy as burning 10 million kg of coal.

A plasma with abundant fusion reactions burns at more than 10 million °C and cannot be contained by any material vessel. Confinement of such a hot plasma is enabled by the presence of a force field. The solar plasma, for example, is confined by its own strong gravitational field (see figure 2). The Sun is simply so massive that its gravity balances the radiation and thermal pressure of the explosive fusion reactions, and makes it a stable star.

Returning to Earth, any plasma produced for fusion reactors would be much too low in mass for gravity to act as a containing force field, and finding an effective containment technique is one of the most daunting challenges for practical fusion energy (see sidebar "Magnetic containment," p42). A common method of containment entails using magnetic fields to confine the plasma. The most common type of experimental fusion system with a toroidal geometry ("torus") is called a tokamak.

Fig. 2.The burning plasma of the Sun is contained by its own gravitational field. This photograph is a composite image taken by NASA's Transition Region and Coronal Explorer (TRACE).

Magnetic containment

No known material is capable of holding fusion plasmas burning at more than 100 million °C. Plasma containment in tokamaks is achieved by the electromagnetic forces acting on the charged particles. Charged particles moving through magnetic fields experience forces that can influence their motion and cause them to spiral (or gyrate) along the field lines. The charges spiral in a clockwise or counter-clockwise direction, depending on the alignment of the field and on whether the charged particles are positive or negative.

This principle is often used to separate charged particles or to detect them (see figure 3), and explains why the Earth's magnetic field is able to trap cosmic ray particles in the Van Allen belt.

Figure 4 shows how charged particles would move randomly in a container without a magnetic field, and how they undergo spiraling motions in the presence of a magnetic field. Figure 5 demonstrates how the ions (carrying positive charge) and electrons (with negative charge) spiral in opposite directions along the magnetic field line. This response of charged particles to magnetic fields is utilized to insulate the burning plasma from the walls of its container in fusion tokamaks. Cleverly designed magnetic fields can effectively confine the plasma and make fusion plasmas amenable to experimental research.

It is most advantageous to design confinement systems with closed magnetic field lines with the "donut-shaped" torus being the signature shape of tokamaks.

Fig. 3. The spirals show how charged particles move in spiraling paths in the magnetic field of a bubble chamber that detects their tracks.

Fig. 4. Charged particles normally move randomly, but undergo spiraling motions in the presence of a magnetic field.

Fig. 5. Ions and electrons spiral in opposite directions in a magnetic field.

This can be described in simple terms as a donutshaped device that can host the appropriate magnetic field to effectively contain the plasma. A plasma containing hydrogen ions moves around the torus with sufficiently energetic ions fusing to release nuclear energy carried away by neutrons. Just as in an oil- or coal-fueled electric generator, heat is the key product of a fusion device, and just like in a coal-burning engine, the central engineering challenge is to produce and use that heat as efficiently as possible. In possible future fusion reactors, this heat would boil water into steam that would then spin turbines to create electricity. One of the differences between a coal engine and a fusion reactor is that the amount of heat produced in the latter is many times greater than in the former. The more fundamental difference lies in the fact that coal fires produce chemical energy, whereas fusion produces nuclear energy.

Nuclear fusion has the potential to satisfy the power demands of the entire world for millennia. However, the science and engineering challenges of magnetic confinement systems such as tokamaks are extremely complicated, and the associated technology is still in its infancy. No existing tokamak produces enough energy to be commercially practical at present. It is not only the length of time over which sufficiently high temperatures and densities must be sustained that makes tokamak research difficult; the physical phenomena involved span a vast range of spatial and temporal scales (see sidebar "The challenges of scales and data," p44). In addition, fusion plasmas need to sustain high densities, effective heating mechanisms and efficient heat retention. Nonlinear effects add to the complexity of the science.

The best way to understand what goes on inside a tokamak is to simulate its dynamics on advanced computers and then verify and validate the results against experimental measurement. A coalition of countries, including the US, is planning to build the next-generation "burning plasma" tokamak device in France by 2016 (see sidebar "ITER: the International Thermonuclear Experimental Reactor," p48). In order to best benefit from this multi-billion-dollar investment, improved physics understanding relevant to ITER is clearly needed. The most effective way to tackle this research challenge is by utilizing high-performance computing techniques that can simulate and model burning plasmas. Such models must be properly validated against analytic theories as well as available experimental results. This approach represents a practical as well as a costeffective necessity. As Dr Stephane Ethier, a computational plasma physicist at the Princeton Plasma Physics Laboratory (PPPL), explains: "Since working with an actual plasma is expensive and often very difficult, we can carry out properly validated simulations to acquire the information needed to accelerate progress in understanding complex plasma behavior under realistic conditions for burning plasmas."

Cyber tokamak simulations
Several SciDAC projects probe different aspects of fusion plasma research (see the SciDAC home page at www.scidac.org for details). One of these projects is based at the Gyrokinetic Particle Simulation Center (GPSC), which has brought together a team of fusion physicists, applied mathematicians and computer scientists to study energy transport including particle motion. Using the Gyrokinetic Toroidal Code (GTC), the team's fusion simulations are designed to replicate the complex turbulent transport processes which dominate the confinement properties in a tokamak. This could be viewed as a key part of a "cyber" tokamak simulation of magnetic containment (see sidebar "Magnetic containment," p42).

The GTC code was originally developed by Zhihong Lin and colleagues, and its main features are described later in this article. The simulation models the behavior of particles and electromagnetic waves in a toroidal plasma in which the ions and electrons are confined by intense magnetic fields. The average tokamak plasma has about 1020 particles per cubic meter, and a typical transit time for each of these to move through the torus of plasma is about 0.0001 s. Every particle exerts an electric force on every other particle in the plasma, and this force changes as the particles gyrate around the torus. A great simplification in dealing with this simulation challenge is the ability to use "finite-sized" particles. Basically, short-range forces (within a Debye sphere) can be ignored because there are equal numbers of electrons and ions there. Accordingly, point particles can be replaced with uniformly charged spheres of Debye-length radius.

To picture the plasma's flow, the computer code calculates the individual position of each "finite-sized" particle. It then determines the collective force-field generated by the particles' charges and maps it over the torus. In the next step, it moves each particle minutely in response to the force the particle experiences. This series of processes is then repeated all over again (see "The particle-in-cell method" section, p46). Tracking such a large number of particles is computationally intensive and such simulations, involving over a billion particles, are now made possible by the modern high-performance computers that are available to researchers through the SciDAC program. One of the critical goals of this project is to gain improved understanding of heat losses for realistic plasma scenarios. The GTC team expects to introduce models for simulating turbulent plasma transport with unprecedented physics fidelity as computing power moves forward to the petascale regime.

The vast range of temporal and spatial scales involved adds to the difficulties imposed by the sheer number of particles in the code. The tiny scale of the particles' gyrations through the plasma requires the computations to include extensive detail at the microscopic level. At the smallest scale, the particles are gyrating rapidly around magnetic field lines in complex helices. The gyrating or spiraling motion of the particles can be interpreted in terms of "gyrokinetic" orbits indicating the time for each rotation. Typically, the time for each gyration in a gyrokinetic orbit is 10^-8 s. (This means that a single particle undergoes about 10⁸ gyrations per second.) On a slightly larger scale, the particle flow resembles a liquid that allows eddies and vortices to form. On an even larger scale, these eddies and vortices cause turbulence and diffusion that disrupts the smooth flow of plasma.

The challenges of scales and data

The power of high-performance computers is best utilized for the most challenging and complex problems. Fusion plasmas represent science across a vast range of spatial and temporal scales. This makes it difficult to find a common equation or analysis tool to study the multiple-component physics they involve. In the context of natural phenomena, nuclear fusion occurs at the femtometer (10^-15 m) distances corresponding to the range of nuclear forces. Acting collectively, trillions of nuclei can influence stellar dynamics and power stars like our Sun that are gravitationally confined at distances over 100 times the size of the Earth. This vast range of length scales, from the distances at which fusion occurs to the astronomical nuclear furnaces it can ignite, is indicated in figure 6.

Fig. 6. The GTC has to manage scales running from the electron gyro radius at one-hundredth of a millimeter to the plasma confinement scales of the order of 100 m. (Illustration not to scale.)

A fusion tokamak on Earth likewise houses a burning plasma that is magnetically contained at length scales of the order of 100 m. Electrons in its interior move in spiraling and circular paths at the microscopic length scales of 10^-5 m - that is, about seven orders of magnitude smaller. One of the daunting tasks of simulating cyber tokamaks is dealing with these very different spatial dimensions and the related physics.

In addition to the spatial dimension challenges, fusion researchers must also deal with temporal scales that are even more diverse, ranging from confinement times of 100 s to the plasma-wave periods of 10^-11 s. It is also important to note that while significant advances have been enabled by fluid-type simulations of large-scale phenomena, more comprehensive kinetic particle models are required to enable simulations of key fine-scale phenomena for dealing with turbulent transport and the associated formation of eddies. The particle-in-cell (PIC) method (see p46), which is used in the GTC code, holds significant promise of being able to incorporate the microscopic kinetics of the plasma particles while at the same time capturing the macroscopic description of thermonuclear fusion plasmas. Moving on from distance and time scales, the number of "finite-sized" particles currently simulated in a cyber tokamak already exceeds a billion. Although is is understood that the more particles that can be included in a simulation, the more realistic the predicted results will be, it is also the case that the computational difficulties correspondingly increase with the associated demands in data analysis and management.

As might be expected, computations involving very large numbers of particles across multiscales and multi-rates demand high processor power on the order of gigaflops and teraflops per second. They involve thousands of processors working in parallel in modern supercomputer machines. In particular, GTC has run successfully on the NERSC's IBM-SP3 Seaborg, the Oak Ridge National Laboratory's CRAY X1E and XT3, the IBM-Blue-Gene-L, and the Earth Simulator in Japan.

These massive data outputs in the giga and tera-scale range have associated challenges in developing the necessary data analysis tools, including feature tracking in the usual three dimensions in real space as well as in velocity-space. Research scientists in this field also need to develop efficient ways to manage, transport, and analyze the data to produce scientific results in a timely way. Scientific visualization is a key data analysis/management tool that is being increasingly deployed. For further details on how scientists working in the SciDAC program successfully manage the challenges of scales and data, see feature "Perfecting the languages and tools of science," p50.

The tools that probe the science

To understand and advance science, we must describe the behavior of physical or biological phenomena in terms of mathematical equations that represent their key observables and inter-relationships. Sometimes, these are sets of partial differential equations such as those discussed in this issue in the article on the TOPS project (see p50). The relevant equations for plasma physics are the Boltzmann/Vlasov equations and Maxwell's equations governing electrodynamics, as shown in figure 7. These equations collectively represent both macroscopic and microscopic physics, but cannot be solved exactly for all the time and length scales involved.

Fig. 7. Equations relating the electric and magnetic fields and the particle variables.

Two developments have greatly facilitated higher-quality solutions of plasma behavior: the PIC method, which represents individual "finite-sized" particle interactions with electromagnetic fields via a representative grid; and the gyrokinetic algorithm, which greatly enhances the efficiency of PIC simulations by representing the gyrating particles as moving charged rings.

As technology and computational hardware evolve to more powerful levels, the algorithms, software and calculation techniques available need to be able to use these efficiently. Modern supercomputer architectures utilize parallel computing modes so that computational tasks can be distributed to different processors. The GTC is attractively amenable to use on parallel computing platforms, as it responds well to the adding of more processors as the number of particles being tracked increases.

One of the most effective tools for understanding the data and analyzing it is visualization. The extensive data generated by the GTC as it tracks billions of particles is obtained as an irregular grid that makes its visualization particularly daunting. SciDAC scientists have developed special techniques to enable 3D visualization.

GTC researchers have found that the code performs well on four of the largest highperformance computers in use today. These include the CRAY X1E and XT-3, the IBM Blue Gene-L, the IBM SP-3 Seaborg and the Earth Simulator in Japan. The team was able to include more than 13 billion particles on 4096 processors on the Earth Simulator running at 7.2 Tflop in late October 2005. The HPC resources provided by the SciDAC program has enabled the GTC to produce important new insights into turbulent heattransport dynamics in tokamaks, including the prediction of possible favorable confinement trends in future ITER-size burning plasma experiments.

Ideally, in the absence of turbulence plasma particles in a tokamak would exhibit excellent confinement as they circled around in the torus. In addition, the magnetic surfaces must be free of openings - any "holes in these magnetic walls" would offer rapid particle escape routes. In actual tokamak experiments, eddies, or pools, perpendicular to the main flow form due to nonlinear effects. Particles trapped in these eddies drift outward towards the walls of the tokamak with much of the heat energy that they carry being lost as they escape the hot plasma core.

To understand the impact of the effects described with the aim of improving the confinement efficiency in tokamaks, plasma scientists have effectively utilized advanced codes such as the GTC on powerful computational platforms to investigate the break-up of these nonlineardriven eddies. A key aspect of the associated dynamics involves wave-particle interactions. "Plasma particles can be compared to surfers," says Dr Wei-li Lee, Principal Investigator of the GPSC project at PPPL. "The particles have to catch the wave and stay in the wave. Just as a surfer needs to be going at the same speed as a passing wave to catch it and stay with it, a plasma particle needs to be going the same speed and direction as a plasma wave to join the flow."

Theory, codes and technology
When plasma fusion research began in the 1950s, computers were bulky and slow. Calculating the interactions between billions of particles was not even an option. Plasma modeling generally used magnetohydrodynamics, a set of equations that describe the plasma as an electrically conductive fluid moving through a magnetic field. While these equations describe macroscopic stellar astrophysics well, fusion plasma simulations built on magnetohydrodynamic models alone did not reproduce laboratory results with respect to kinetic effects.

Fig. 8. The PIC method uses a grid to communicate effective forces between particles.

Fig. 9. The gyratingforward motion of the particles is approximated by forward-moving rings of charge

As computer technology improved in power and sophistication in the 1960s, microscopic calculations tracking positions and velocities of individual particles became possible. In addition to the usual three dimensions of space, three additional virtual dimensions of velocity needed to be taken into account for realistic simulations. Simulating particles realistically requires that their motion be followed in six instead of just the familiar three spatial dimensions.

The particle-in-cell method
The particle-in-cell (PIC) method was developed to deal more efficiently with the formidable challenge of calculating the dynamics of the aforementioned "finite-sized" particles while taking into account the collective forces from the electromagnetic fields in the plasma. In reality, each moving particle has a charge that exerts an electromagnetic force on every other particle in the plasma. Even for finite-sized particles, tracking the variations in electric forces for billions of particles would make calculations prohibitive in terms of computer time or data handling.

The PIC method suggested a way around this problem: instead of calculating the force exerted by each particle on every other particle, it uses the long-range nature of the electric forces to approximate them via a grid (see figure 8). Called the "scatter" phase of the PIC simulation, the first step estimates the charge density at each point on the grid arising from the particles in its immediate neighborhood. This is used to calculate the electromagnetic potential using the laws of physics (the "solve" step). The forces are then applied back to the particles during the "gather" phase of the simulation. The motion of the particles is governed by the laws of motion and the forces acting on them. These laws are used to appropriately move or advance the particle in cyberspace during the "push" stage of the simulation. The chain of "scatter-solve-gather-push" steps is repeated in a recurring way during the PIC simulation run.

The PIC method significantly reduced the number of calculations needed for a fusion simulation, since the problem now scaled with the number of particles rather than its square. The physics is important at scales that are not sensitive to the smaller-scale fluctuations in the field, since turbulence processes depend on larger-scale fluctuations. However, replicating the motions of realistic numbers of particles in a tokamak while including their gyrating motion still remained computationally prohibitive.

The gyrokinetic algorithm
A major advance in PIC applications for magnetically confined toroidal plasmas arrived with the introduction of the gyrokinetic algorithm in the early 1980s by Dr Wei-li Lee at PPPL (see Further Reading on p49 for details). Gyrokinetics made a simple, clever approximation: instead of treating the ion as a point particle gyrating about the magnetic field lines, it smeared the ion into a charged ring moving along a straight path. This technique is justified because the stability and energy-transfer processes in a tokamak are insensitive to the timescales of the ionic gyrations. The overall orbital motions of the ions that influence the physics are thus retained and the helical (or screw-like) motion of the ion is replaced by a charged, moving ring as shown in figure 9. On the other hand, electrons are approximated as point particles.

This technique enables the retention of the microscopic kinematics of the system that introduce the important non-local interactions, while avoiding the rapidly fluctuating gyration details. The implementation of the gyrokinetic algorithm within the PIC method has enabled modern codes such as the GTC to carry out simulations on supercomputing platforms that can more realistically investigate microturbulence in plasmas than ever before.

Fusion and how it works

Nuclear fusion is the process in which two atomic nuclei (for example, deuterium and tritium) "fuse together" to produce an alpha particle (isotope of helium), accompanied by a large amount of energy release with the expulsion of a neutron. Because these nuclei are positively charged, they repel each other electrically with associated electric forces normally keeping them apart. It is only when the nuclei come within the range of the stronger nuclear force that nuclear reactions, including nuclear fusion, can happen.

Fig. 10. The Coulomb barrier normally keeps positively charged.

Fig. 11. Tritium and deuterium fuse together.

The electric repulsive force between two positively charged particles is a potential barrier (known as a Coulomb barrier) that increases as the particles approach each other, according to what is called Coulomb's law. Although the electrical forces are effective in keeping nuclei apart because of their mutual positive charges, stronger nuclear forces take over if the nuclei can approach within the range of their action.

Nuclear forces are the strongest known forces and are many orders of magnitude stronger than electromagnetic effects. They are also short-range, cutting off at distances of the order of one fermi (or 10^-13 cm - less than one-millionth of one-millionth of a centimeter). This is why the sizes of the nuclei of atoms are themselves of the order of fermi.

Colliding nuclei have to have enough energy to overcome the electrical potential barrier and approach separation distances of the order of fermis for fusion reactions to happen. Fusion plasmas must exceed temperatures of 10 million °C for individual nuclei to have enough energy to initiate fusion reactions.

We can get a feel for the difficulty of achieving sustained fusion reactions by recalling the physics of the birth of a star. It takes forming stars some 50 million years to collect the hydrogen that will fuel them and gravitationally contract to temperatures high enough to ignite the plasma to produce the nuclear fusion reaction. In stars, gravitational energy is converted to heat as a forming star contracts. Fusion reactions start in its interior, and a "protostar" is born. An average star such as our Sun has enough fuel to burn for 10 billion years, and its plasma is contained by its gravitational field. On Earth, fusion devices such as tokamaks are of course much smaller plasmas which nevertheless need to be heated to ignition temperatures (containment problems have been discussed separately.) This heating challenge can be addressed by introducing energetic electromagnetic waves and/or the injection of neutrally charged energetic beams into the plasma.

Once attained, the high temperatures must be sustained for a sufficient period of time to produce the conditions needed for fusion reactions. This includes reducing heat loss, which is one of the problems that research using GTC aims to solve.

Most tokamaks such as ITER are designed to produce deuterium-tritium fusion reactions. Deuterium and tritium are two isotopes of hydrogen. All hydrogen atoms contain exactly one proton, but different isotopes have different numbers of neutrons; deuterium has one neutron, and tritium has two. Deuterium and tritium are often called "heavy hydrogen," and are abundantly found in sea-water. Inside a tokamak, deuterium and tritium atoms are heated to over 100 million °C.

At temperatures this high, the atoms form a plasma - a hot gas comprised of charged particles with the electrons and ions spiraling round in the confining magnetic fields. The nuclei are energetic enough to overcome their mutual electrical repulsion and fuse into a helium nucleus (alpha particle) and a neutron (see figure 11). When they fuse, the resulting helium nucleus is more stable than the two heavy hydrogen nuclei alone. Because it is more stable, it needs less energy to bind itself together, and it gives off this available energy, which amounts to about 17 MeV. The process also releases a free neutron which leaves the plasma carrying most of the excess energy. The energy multiplication is about 450 to 1 (PPPL website), making fusion an attractive energy source.

The GTC and computer architecture Computer architectures implementing the concept of "computing in parallel" have added greatly to the total computing power available for complex problems. To take advantage of this technique, code algorithms need to be amenable to sharing tasks between multiple processors. A tremendous advantage of the GTC is its "scalability" on parallel computers. This means that it is possible to do a bigger simulation by dividing it into discrete tasks amongst multiple processors. To track more particles, one simply needs to connect to more processor units. Not all simulation codes can do this easily, however - many can only be used on a limited number of processors because so much communication is required between the processors.

ITER: the International Thermonuclear Experimental Reactor

"Plentiful, reliable energy is critical to worldwide economic development. Fusion technologies have the potential to transform how energy is produced and provide significant amounts of safe, environmentally friendly power in the future. The ITER project will make this vision a reality."

Energy secretary Samuel Bodman
June 28, 2005

The International Thermonuclear Experimental Reactor (ITER) began its life in 1985 as a Geneva Summit initiative, and has evolved over the years into a formal collaboration between the US, the European Union, Russia, China, Japan, and the Republic of Korea to construct the first fusion science facility capable of producing a self-sustaining fusion reaction, called a "burning plasma." At present the ITER is projected to go online in 2016 in Cadarache, France.

In the ITER project, researchers plan to coil superconducting magnets around a toroidal vessel to produce a long-pulsed burning plasma medium. SciDAC-empowered simulations using US Department of Energy (DOE) leadership-class computers have already contributed significantly towards the development of refueling ITER by pellet injection, as well as towards the improved understanding of plasma turbulence needed to help guide the research program that is being developed for this burning plasma experiment.

Given the promise of high-energy output coupled with fusion's low environmental impact, the ITER project might seem like a dream that is too good to be true. But recent advances in both fusion science and computer modeling have positioned the ITER as the DOE's number-one facility priority, as detailed in the report, "Facilities for the Future of Science: A Twenty-Year Outlook." The DOE's PPPL and ORNL have been designated as the partnership responsible for overseeing US ITER activities with respect to the requisite staffing and facilities.

This past summer, the six ITER member nations gathered in Moscow to announce the Cadarache site decision. As they celebrated reaching this major milestone after almost 20 years of negotiations, Dr Raymond Orbach, director of the DOE Office of Science, reminded the group that many years of work, collaboration and cooperation remained ahead before the promise of "plentiful, safe and environmentally benign" fusion energy becomes a reality. At the same time, he made it quite clear that "the United States remains committed to this promise."

Fig. 12. A schematic of the International Thermonuclear Experimental Reactor's design.

Fig. 13. The horizontal axis expresses the tokamak size, and the point at 1000 represents the ITER size. The vertical axis represents the thermal diffusion, or heat loss.

Fig. 14. The granular structures represent the scales of the turbulences in a typical plasma which need to be included in realistic plasma simulations.

The bigger and faster the computer, the more particles it can track, and the better the simulation with respect to both the accuracy and physics fidelity of the results. However, the demand for time on those computers that can best run the GTC's largest simulations is very high. Nevertheless, the GPSC has very productively utilized its supercomputer time allocations. A single simulation can generate several terabytes of data. Just processing the data to see what happened in the simulation can take months, as can checking for correctness. A large GTC run will produce hexabytes of data (10¹⁸ bytes - six orders of magnitude more than 1 Tbyte, the largest amount of data easily stored on a hard drive).

Data management and visualization
Even 10 years ago, when the amount of data the simulation was producing was much smaller, data-transfer was difficult. It remains a significant challenge, despite advances in computer technology, because the amount of data generated by the GTC is so enormous. "While FTP is faster for transferring data, the amount of data is now so much larger [that] it still takes too much time," says Dr Scott Klasky, a member of the GPSC team in charge of visualization.

Fig. 15. Real-time interactive visualization images of the plasma.

Fig. 16. The torus is divided into domains. Multiple processors can work on a domain, allowing good scaling on parallel platforms as the number of data samples increases.

Dr Klasky, currently at ORNL, along with other members of the group, has developed faster data-transfer techniques, improved data storage, and analyze-as-you-go methods to understand the results of the simulation as they come out. The team also adapted computer graphics cards, originally invented for graphicsintensive games, to visualize the flow of plasma within the cyber tokamak.

Advanced simulations of kinetic turbulence dynamics is particularly data-intensive. Because of the critical nature of the phenomena occurring in the turbulent regions of the plasma, every piece of data counts. (In less complex regions, superfluous data can be ignored and dropped from the analysis.) The GPSC group is working with other SciDAC groups, in particular the supernova and combustion researchers, to figure out better ways to visualize and understand turbulence.

Advanced visualization allows extraction of key science from the extensive data generated in the simulation of many billions of particles over multiple timesteps in the GTC code. The associated data is obtained in an irregular grid form that makes its visualization particularly challenging. Physical parameters such as density and temperature are computed at each mesh point-time co-ordinate. For example, a billion particles run over 125 million mesh points generates 4 Tbyte of data. If all data were stored, the requirements would increase to a formidable 115 Tbyte. New graphics hardware with 3D texture support makes real-time volume representation possible. A mixed co-ordinate system and other techniques allow interactive visualization of the type shown in figure 16.

Making a difference
The GTC calculates in real space, and this key feature distinguishes it from other micro-turbulence codes. Real space sampling permits better scaling for parallel calculations. Scientists strive not only to maximize efficiency in data layout and access, but also to retain flexibility in data porting between computers as needed. The GTC has been very successful in providing an improved understanding of turbulent transport dynamics impacting on the vitally important issue of efficient confinement in fusion plasmas. In particular, by efficiently using more than 1000 processors on the IBM SP Power3, it has demonstrated that a more favorable scaling of confinement can result in future ITER-sized plasmas.

Since there are no existing fusion experiments approaching this large size and in the absence of accurate nonlinear theoretical models capable of dealing with such scenarios, these advanced simulations can indeed be characterized as a "tool for scientific discovery." With its ability to run efficiently on the most advanced high-performance computing platforms (including the Earth Simulator in Japan, the CRAY X1E and XT3 at ORNL, and the IBM-Blue-Gene-L), combined with its progress in implementing increasingly realistic physics and geometric features, the GTC is poised to continue its successes in scientific discovery using advanced computing.

Further reading

The GPSC Center http://w3.pppl.gov/theory/GPSC.html.

S. Ethier, W. M. Tang and Z. Lin 2005 Gyrokinetic particle-in-cell simulations of plasma microturbulence on advanced computing platforms J. Phys.: Conf. Ser. 16 1-15.

W. W. Lee 1983 Gyrokinetic approach in particle simulation Physics of Fluids 26(2) 556-562.

W. W. Lee 1987 Gyrokinetic particle simulation-model Journal of Computational Physics 72(1) 243-269.

Z. Lin, T. S. Hahm, W. W. Lee, W. M. Tang and R. B. White 1998 Turbulent transport reduction by zonal flow: massively parallel simulations Science 281 1835.