Get started today with this GPU Ready Apps Guide.
SPECFEM3D Globe simulates global and continental scale regional seismic wave propagation. This application enables researchers to analyze the effects of lateral variations in compressional-wave speed, shear-wave speed, density, rotation, and self-gravitation on a 3D crustal model.
The latest version 7.0 offers GPU graphics card support for both OpenCL and CUDA hardware accelerators, based on an automatic source-to-source transformation library.
SPECFEM3D GLOBE runs over 25X faster on a single NVIDIA® Tesla® P100 GPU, helping scientists run their seismic simulations in hours instead of weeks.
SPECFEM3D_GLOBE should work on any platform with MPI support. For the purpose of this guide, we suggest the following configuration:
1. Download the SPECFEM3D_Globe 7.0.0 source code
2. Extract the Package
tar xf SPECFEM3D_GLOBE_V7.0.0.tar.gz
3. Required Compliers
Make sure that nvcc, gcc and gfortran are available on your system and in your path. If they are not available please contact the system administrator.
4. Configure the Package
./configure --with-cuda=cuda5
Note here we are configuring to use the more modern CUDA bindings in CUDA 5.0 and later.
5. Build the Program
make
Running SPECFEM3D Globe is a two stage process. First, the mesh used to solve the final problem must be created. Then the solver can be run to generate the final solution.
Note that as some arrays are statically sized if any changes are made to the DATA/Par_file (which sets some parameters) then the program must be recompiled.
1. Update the DATA/Par_file and remaining data files.
It is crucial that to use GPUs, the GPU_MODE parameter in DATA/Par_file must be set to .true
2. Rebuild the program. Note, because the program uses static array sizes then any changes in the Par_file require the program to be rebuilt.
make clean
make
3. Run the mesher (in this case we are running with 4 MPI tasks and on 4 GPUs)
mpirun -np 4 bin/xmeshfem3D
4. Run the main program
mpirun -np 4 bin/xspecfem3D
This section walks through the process of testing the performance on your system and shows performance on a typical configuration.
1. Download and install the program as described in the Installation section
2. Download the tar file with the required input files
3. Change to the root SPECFEM directory
4. Extract the various Par_files, from the tar file.
tar -xf parfiles.tar
This will extract the data files into the DATA directory in your SPECFEM installation, and update the STATIONS and CMTSOLUTION files. We include files for 1 GPU, 2 GPUs and 4 GPUs and for 36 CPUs. The example we are using is a modified version of the “global_s362ani_shakemovie” example that is distributed with SPECFEM3D Globe. The problem size has been reduced so that it fits more easily on one node. Note also that there are restrictions in the number of MPI tasks that can be used. The NPROC variables will change depending on whether the run uses 1 GPU (1x1), 2 GPUs (2x1), 4 GPUs (2x2) or 36 CPU cores (6x6).
Key changes that are made to the original Par_file under the are as follows :
This section provides expected performance benchmarks for different across single and multi-node systems.