CUDA Spotlight: Yifeng Cui




Yifeng Cui

GPU-Accelerated Earthquake Simulations

This week's Spotlight is on Yifeng Cui, director of the High Performance GeoComputing Lab (HPGeoC) at the San Diego Supercomputer Center (and adjunct professor in the Dept. of Geological Sciences at San Diego State University).

HPGeoC was recently named a winner of the HPC Innovation Excellence Award by IDC for developing a highly-scalable computer code that promises to dramatically cut research times and energy costs in simulating seismic hazards.

Also contributing to the code’s development were Jun Zhou and Efecan Poyraz, graduate students of Electrical and Computer Engineering (ECE) with the Jacobs School of Engineering at UC San Diego; and Dong Ju Choi, a computational scientist at SDSC.

This interview is part of the CUDA Spotlight Series.


Q & A with Yifeng Cui

NVIDIA: Yifeng, what are the key technology challenges in your field?
Yifeng: In Southern California, we know it’s only a matter of time before a massive earthquake along the San Andreas fault strikes. To help predict the potential destructive power of this quake, sometimes referred to as “The Big One”, supercomputers are being called on to model how waves are generated by and travel outward from the fault, and then interact with complex 3D geological structures, such as sedimentary basins in the Los Angeles region.

Producing realistic seismograms at high frequencies relevant to building engineering design, up to 10 hertz, will require a major increase in supercomputing power with exascale hardware capabilities. Moreover, to realistically calculate seismic hazards across huge expanses of California and other tectonically active regions requires more than just a few earthquake simulations.

Toward that end, the Southern California Earthquake Center (SCEC), a consortium of earthquake researchers from more than 60 universities and other institutions funded by NSF and USGS, has embarked on an ambitious effort called CyberShake with a goal of performing physics-based, probabilistic seismic hazard analyses to better inform engineers and planners about constructing and retrofitting buildings to reduce seismic risk, increase community resilience, and save lives.

To put this problem into perspective, a California state-wide seismic hazard map at the maximum frequency of 1 hertz would require intensive ground shaking calculations at a few thousand sites. Thus, to advance and apply earthquake system science research within the practical limits of currently available HPC resources, we must aggressively improve the computational performance of our physics-based ground motion simulation software.

NVIDIA: How are GPUs helping you solve these challenges?
Yifeng: Our team developed a scalable GPU code based on AWP-ODC, an Anelastic Wave Propagation code originally developed by Prof. Kim Olsen of SDSU. This code has two versions that provide equivalent results. The first version can efficiently calculate extreme scale ground motions at many sites. The second can efficiently calculate ground motions from many single-site ruptures as capacity computing. The optimization of the code results in around a 110x speedup over the CPU in key strain tensor calculations critical to the probabilistic seismic hazard analysis.

NVIDIA: What specific approaches do you use to apply GPU computing to your work?
Yifeng: This code is a memory-bounded stencil that is limited in compute performance by its low computational intensity and poor data locality. We re-designed the Fortran code to C to maximize throughput and memory locality. Good scalability was achieved through a two-layer 2D decomposition, and an algorithm-level communication reduction scheme, which eliminates stress data exchange otherwise needed per iteration.

CUDA asynchronous memory copy operations help effective overlap of CPU/PCIe data transfer with GPU computation. A two-layer scalable IO technique was developed to efficiently handle many terabytes of dynamic source and media inputs, as well as 3D volume velocity outputs. We are also tuning co-scheduling to allow full utilization of both CPUs and GPUs in the hybrid heterogeneous systems. We are grateful for NVIDIA’s support during our implementation process.


Magnitude 8 earthquake simulation on southern San Andreas fault

NVIDIA: Tell us about your use of the Titan system at Oak Ridge National Lab.
Yifeng: We simulated realistic 0-10 hertz ground motions on a mesh comprising 443 billion elements in a calculation that includes both small-scale fault geometry and media complexity at a model size far beyond what has been done previously. This was done in collaboration with Profs. Olsen and Steve Day of SDSU; Prof. Thomas Jordan of USC, the Director of SCEC; and others at SCEC. The validation simulation on Titan demonstrated ideal scalability up to 8K Titan nodes, and sustained 2.3 petaflop/s on 16K Titan nodes.

NVIDIA: How did you get interested in this field?
Yifeng: It was the NPACI program (the National Partnership for Advanced Computational Infrastructure, housed at SDSC until 2005) that attracted me to move to HPC more than a decade ago. Working together with many of my talented colleagues from diverse science backgrounds at an environment encouraging innovation through freedom inspired me to pursue new ideas related to computational issues. In 2004, I helped enable the San Andreas fault scenario called TeraShake, the first terascale earthquake simulation that discovered how the rupture directivity of the southern San Andreas fault, a source effect, could couple to the excitation of sedimentary basins, a site effect, to substantially increase the seismic hazard in Los Angeles. Since then, I have been very fortunate to work with some of the nation’s finest seismologists and support their earthquake research.

NVIDIA: When did you first think of applying GPUs to earthquake simulations?
Yifeng: AWP-ODC was transformed from a ‘personal’ researcher code into a SCEC community code for large-scale earthquake simulations. Although highly scalable, the CPU version of the code would still take hundreds of millions of computing core-hours to generate the California state-wide seismic hazard map at the maximum frequency of 1 hertz. Using GPUs provided an alternative solution.

NVIDIA: What advice would you offer other seismologists looking at using CUDA?
Yifeng: Know your code and what you are looking for and develop a good long-term plan before jumping into any new programming models. Time-to-solution is the single most important metric for scientific applications, I would say. Computing is changing more rapidly than ever before, which is particularly challenging for earthquake applications; however, these changes also offer an unprecedented opportunity to re-design your code. Keep an open mind and enjoy the fun with emerging many-core architectures.


Bio for Dr. Yifeng Cui

Yifeng Cui is a computational scientist at SDSC and an adjunct professor at SDSU, a lead of the Computational Science Committee of the Southern California Earthquake Center. He holds a Ph.D. in hydrology from University of Freiburg, Germany. Dr. Cui’s earthquake research efforts were honored with the 2009/2011 SciDAC OASCR awards and a 2010 ACM Gordon Bell finalist. In 2013, his HPGeoC Lab was named a winner of IDC's HPC Innovation Excellence Award. He has lived in four countries and speaks three languages. In his spare time, he enjoys hiking and biking.

Relevant Links
http://hpgeoc.sdsc.edu
http://www.sdsc.edu
http://scec.usc.edu/
http://www.sdsu.edu

Contact Info
San Diego Supercomputer Center MC0505
University of California, San Diego
9500 Gilman Drive, La Jolla, CA, 92093-0505, USA
Tel: +1-858-822-0916
Email: yfcui@sdsc [dot] edu