Advancing Multi-GPU Processing
Systems with multiple GPUs and CPUs are becoming common in a variety of industries as developers rely on more parallelism in applications like AI computing. These include 4-GPU and 8-GPU system configurations using PCIe system interconnect to solve very large, complex problems. But PCIe bandwidth is increasingly becoming the bottleneck at the multi-GPU system level, driving the need for a faster and more scalable multiprocessor interconnect.
NVIDIA® NVLink™ technology addresses this interconnect issue by providing higher bandwidth, more links, and improved scalability for multi-GPU and multi-GPU/CPU system configurations. A single NVIDIA Tesla® V100 GPU supports up to six NVLink connections and total bandwidth of 300 GB/sec—10X the bandwidth of PCIe Gen 3. Servers like the new NVIDIA DGX-1™ take advantage of these technologies to give you greater scalability for ultrafast deep learning training.
First introduced with the NVIDIA Pascal™ architecture, NVLink on Tesla V100 has increased the signaling rate from 20 to 25 GB/second in each direction. It can be used for GPU-to-CPU or GPU-to-GPU communication, as in the DGX-1 with Tesla V100.
Tesla V100 with NVLink GPU-to-GPU and GPU-to-CPU connections
NVLink connecting eight Tesla V100 accelerators in a hybrid cube mesh topology as used in the DGX-1V server
NVLink: GPU Servers: Dual Xeon Gold 6140@2.30GHz or E5-2698 v4@3.6GHz for PyTorch with 8xV100 PCIe vs 8xV100 NVLink. SW benchmarks: MILC (APEX medium). HOOMD-Blue (microsphere), LAMMPS (LJ 2.5).
NVIDIA NVLink can bring up to 70% more performance to an otherwise identically configured server. Its dramatically higher bandwidth and reduced latency will enable even larger deep learning workloads to scale in performance as they continue to grow.
The rapid growth in deep learning workloads has driven the need for a faster and more scalable interconnect, as PCIe bandwidth increasingly becomes the bottleneck at the multi-GPU system level.
NVLink is a great advance to enable eight GPUs in a single server, and accelerate performance beyond PCIe. But taking deep learning performance to the next level will require a GPU fabric that enables more GPUs in a single server, and full-bandwidth connectivity between them.
NVIDIA NVSwitch is the first on-node switch architecture to support 16 fully-connected GPUs in a single server node and drive simultaneous communication between all eight GPU pairs at an incredible 300 GB/s each. These 16 GPUs can be used as a single large-scale accelerator with 0.5 Terabytes of unified memory space and 2 petaFLOPS of deep learning compute power. A single HGX-2/DGX-2 system with NVSwitch delivers up to 2.7x more application performance than 2 HGX-1/DGX-1 systems connected via infinband.
2 HGX-1V servers have dual socket Xeon E5 2698v4 Processor, 8X V100 GPUs. Servers connected via 4X 100Gb IB ports (run on DGX-1) . HGX-2 server has dual-socket Xeon Platinum 8168 Processor, 16X V100 GPUs (run on DGX-2).
* ECWMF’s IFS: The Integrated Forecasting System (IFS) is a global numerical weather prediction model developed by the European Centre for Medium-Range Weather Forecasts (ECMWF) based in Reading, United Kingdom. ECMWF is an independent intergovernmental organization supported by most of the nations of Europe, and operates one of the largest supercomputer centers in Europe for frequent updates of global weather forecasts. The IFS mini-app benchmark focuses its work on a spherical harmonics transformation that represents a significant computational load of the full model. The benchmark speedups shown here are better than those for the full IFS model, since the benchmark amplifies the transform stages of the algorithm (by design). However, this benchmark demonstrates that ECMWF’s extremely effective and proven methods for providing world-leading predictions remain valid on NVSwitch-equipped servers such as NVIDIA’s DGX-2, since they are such a good match to the problem.
* Mixture of Experts (MoE): Based on a network published by Google at the Tensor2 Tensor github, using the Transformer model with MoE layers. The MoE layers each consist of 128 experts, each of which is a smaller feed-forward deep neural network (DNN). Each expert specializes in a different domain of knowledge, and the experts are distributed to different GPUs, creating significant all-to-all traffic due to communications between the Transformer network layers and the MoE layers. The training dataset used is the “1 billion word benchmark for language modeling” according to Google. Training operations use Volta Tensor Core and runs for 45,000 steps to reach perplexity equal to 34. This workload uses a batch size of 8,192 per GPU.