This site requires Javascript in order to view all its content. Please enable Javascript in order to access all the functionality of this web site. Here are the instructions how to enable JavaScript in your web browser.

Instructor-Led Workshop
Data Parallelism: How to Train Deep Learning Models on Multiple GPUs

Request a workshop for your organization

Register for public workshops

Modern deep learning challenges leverage increasingly larger datasets and more complex models. As a result, significant computational power is required to train models effectively and efficiently. Learning to distribute data across multiple GPUs during deep learning model training makes possible an incredible wealth of new applications utilizing deep learning.

Additionally, the effective use of systems with multiple GPUs reduces training time, allowing for faster application development and much faster iteration cycles. Teams who are able to perform training using multiple GPUs will have an edge, building models trained on more data in shorter periods of time and with greater engineer productivity.

This workshop teaches you techniques for data-parallel deep learning training on multiple GPUs to shorten the training time required for data-intensive applications. Working with deep learning tools, frameworks, and workflows to perform neural network training, you’ll learn how to decrease model training time by distributing data to multiple GPUs, while retaining the accuracy of training on a single GPU.

Learning Objectives

By participating in this workshop, you’ll:

Understand how data parallel deep learning training is performed using multiple GPUs
Achieve maximum throughput when training, for the best use of multiple GPUs
Distribute training to multiple GPUs using Pytorch Distributed Data Parallel
Understand and utilize algorithmic considerations specific to multi-GPU training performance and accuracy

Download workshop datasheet (PDF 47 KB)

Workshop Outline

Introduction (15 mins)	Meet the instructor. Create an account at courses.nvidia.com/join
Stochastic Gradient Descent and the Effects of Batch Size (120 mins)	Learn the significance of stochastic gradient descent when training on multiple GPUs Understand the issues with sequential single-thread data processing and the theory behind speeding up applications with parallel processing. Understand loss function, gradient descent, and stochastic gradient descent (SGD). Understand the effect of batch size on accuracy and training time with an eye towards its use on multi-GPU systems.
Break (60 mins)
Training on Multiple GPUs with PyTorch Distributed Data Parallel (DDP) (120 mins)	Learn to convert single GPU training to multiple GPUs using PyTorch Distributed Data Parallel Understand how DDP coordinates training among multiple GPUs. Refactor single-GPU training programs to run on multiple GPUs with DDP.
Break (15 mins)
Maintaining Model Accuracy when Scaling to Multiple GPUs (90 mins)	Understand and apply key algorithmic considerations to retain accuracy when training on multiple GPUs Understand what might cause accuracy to decrease when parallelizing training on multiple GPUs. Learn and understand techniques for maintaining accuracy when scaling training to multiple GPUs.
Workshop Assessment (30 mins)	Use what you have learned during the workshop: complete the workshop assessment to earn a certificate of competency
Final Review (15 mins)	Review key learnings and wrap up questions. Take the workshop survey.
Next Steps	Continue learning with these DLI trainings: Model Parallelism: Building and Deploying Large Neural Networks

Workshop Details

Duration: 8 hours

Price: Contact us for pricing.

Prerequisites:

Experience with deep learning training using Python

Technologies: PyTorch, PyTorch Distributed Data Parallel, NCCL

Assessment Type: Skills-based coding assessments evaluate learners' ability to train deep learning models on multiple GPUs.

Certificate: Upon successful completion of the assessment, participants will receive an NVIDIA DLI certificate to recognize their subject matter competency and support professional career growth.

Hardware Requirements: Desktop or laptop computer capable of running the latest version of Chrome or Firefox. Each participant will be provided with dedicated access to a fully configured, GPU-accelerated workstation in the cloud.

Languages: English

Upcoming Public Workshops

Fundamentals of Accelerated Computing with CUDA C/C++

Wednesday, Jan 11, 2023 at 9:00 a.m.-5:00 p.m. PST, UTC-8

Computer Vision for Industrial Inspection
Wednesday, Jan 18, 2023 at 9:00 a.m.-5:00 p.m. CET, UTC+1

If your organization is interested in boosting and developing key skills in AI, accelerated data science, or accelerated computing, you can request instructor-led training from the NVIDIA DLI.

Request a Workshop

Continue Your Learning with These DLI Trainings

FUNDAMENTALS

NEW

Model Parallelism: Building and Deploying Large Neural Networks

8 hours | $500 | PyTorch, Megatron-LM, DeepSpeed, Slurm, Triton Inference Server, NVIDIA Nsight™

Certificate Available

FUNDAMENTALS

Deploying a Model for Inference at Production Scale

4 hours | $30 | NVIDIA Triton

FUNDAMENTALS

NEW

Introduction to Physics-Informed Machine Learning with Modulus

4 hours | $30 | NVIDIA Modulus

Resources

Contact Us for Questions on
Deep Learning Training

Section

Section

First Name

Last Name

Business Email Address

Organization / University Name

Industry

Job Title

Job Role

Location

City

State

Reason for requesting contact

本人知悉且同意 NVIDIA Corporation 基于调研、活动组织的目的对本人的上述信息的收集和处理，并已经阅读并同意 <a href="https://www.nvidia.cn/about-nvidia/privacy-policy/" target="_blank">NVIDIA 隐私政策</a>。

本人知悉且同意，因调研、活动组织的必须，以及相对应的 NVIDIA 内部管理和系统操作的需要，上述信息会被传输到位于美国的 NVIDIA Corporation 按照符合 <a href="https://www.nvidia.cn/about-nvidia/privacy-policy/" target="_blank">NVIDIA 隐私政策</a>的方式进行存储，您可以通过发送邮件至 <a href="mailto:privacy@nvidia.com">privacy@nvidia.com</a> 进行联系以解决相关问题，实现可适用的数据保护法所规定的权利。

Instructor-Led Workshop
Data Parallelism: How to Train Deep Learning Models on Multiple GPUs

Learning Objectives

Workshop Outline

Workshop Details

Upcoming Public Workshops

Continue Your Learning with These DLI Trainings

Model Parallelism: Building and Deploying Large Neural Networks

Deploying a Model for Inference at Production Scale

Introduction to Physics-Informed Machine Learning with Modulus

Resources

Contact Us for Questions on
Deep Learning Training

Read our FAQs.

Inquire about NVIDIA Deep Learning Institute services.

For technical questions, check out the NVIDIA Developer Forums.

NVIDIA DLI Certificate

Instructor-Led Workshop Data Parallelism: How to Train Deep Learning Models on Multiple GPUs

Learning Objectives

Workshop Outline

Workshop Details

Upcoming Public Workshops

Continue Your Learning with These DLI Trainings

Model Parallelism: Building and Deploying Large Neural Networks

Deploying a Model for Inference at Production Scale

Introduction to Physics-Informed Machine Learning with Modulus

Resources

Contact Us for Questions on Deep Learning Training

Read our FAQs.

Inquire about NVIDIA Deep Learning Institute services.

For technical questions, check out the NVIDIA Developer Forums.

NVIDIA DLI Certificate

Instructor-Led Workshop
Data Parallelism: How to Train Deep Learning Models on Multiple GPUs

Contact Us for Questions on
Deep Learning Training