NVIDIA Home NVIDIA Home Menu Menu icon Menu Menu icon Close Close icon Close Close icon Close Close icon Caret down icon Accordion is closed, click to open. Caret down icon Accordion is closed, click to open. Caret up icon Accordion is open, click to close. Caret right icon Click to expand Caret right icon Click to expand Caret right icon Click to expand menu. Caret left icon Click to collapse menu. Caret left icon Click to collapse menu. Caret left icon Click to collapse menu. Shopping Cart Click to see cart items Search icon Click to search
Skip to main content
Artificial Intelligence Computing Leadership from NVIDIA
    • Gaming and Entertainment
      • GeForce Graphics Cards
      • Gaming Laptops
      • G-SYNC Monitors
      • SHIELD
    • Laptops and Workstations
      • Gaming Laptops
      • NVIDIA RTX Desktop Workstations
      • NVIDIA RTX in Professional Laptops
      • DGX Station
      • NVIDIA RTX Data Science Workstation
      • Studio products
    • Cloud and Data Center
      • Overview
      • Grace CPU
      • DGX Systems
      • NVIDIA OVX
      • EGX Platform
      • IGX Platform
      • HGX Platform
      • DRIVE Constellation
    • Networking
      • Overview
      • DPU
      • Ethernet
      • InfiniBand
    • GPUs
      • GeForce
      • NVIDIA RTX / Quadro
      • Data Center
      • Titan RTX
    • Embedded Systems
      • Jetson
      • DRIVE AGX
      • Clara AGX
    • Application Frameworks
      • Metaverse Applications - Omniverse
      • Automotive - DRIVE
      • Cloud-AI Video Streaming - Maxine
      • Speech AI - Riva
      • Data Analytics - RAPIDS
      • Healthcare - Clara
      • High-Performance Computing
      • Intelligent Video Analytics - Metropolis
      • Recommender Systems - Merlin
      • Robotics - Isaac
      • Telecommunications - Aerial
    • Apps and Tools
      • Application Catalog
      • NGC Catalog
      • NVIDIA NGC
      • 3D Workflows - Omniverse
      • Data Center
      • GPU Monitoring
      • NVIDIA RTX Experience
      • NVIDIA RTX Desktop Manager
      • RTX Accelerated Creative Apps
      • Video Conferencing
      • NVIDIA Workbench
    • Gaming and Creating
      • GeForce NOW Cloud Gaming
      • GeForce Experience
      • NVIDIA Broadcast App
      • Animation - Machinima
      • Modding - RTX Remix
    • Infrastructure
      • AI Enterprise Suite
      • Cloud Native Support
      • Cluster Management
      • Edge Deployment Management
      • Inference Serving
      • IO Acceleration
      • Networking
      • Virtual GPU
    • Cloud Services
      • Metaverse - Omniverse
      • Cloud Gaming
    • AI and Data Science
      • Overview
      • Data Analytics
      • Machine Learning
      • Deep Learning Training
      • Deep Learning Inference
      • Conversational AI
      • Prediction and Forecasting
      • Speech AI
      • Large Language Models
    • Data Center and Cloud Computing
      • Overview
      • Accelerated Computing for Enterprise IT
      • Cloud Computing
      • Colocation
      • Edge Computing
      • Networking
      • Virtualization
      • MLOps
    • Design and Simulation
      • Overview
      • Augmented and Virtual Reality
      • Multi-Display
      • Rendering
      • Metaverse
      • Graphics Virtualization
      • Engineering Simulation
      • Broadcast
    • Robotics and Edge Computing
      • Overview
      • AI-on-5G
      • Intelligent Video Analytics
      • Industrial
      • Robotics
      • Edge Deployment Management
      • Edge Solutions
    • High-Performance Computing
      • Overview
      • HPC and AI
      • Simulation and Modeling
      • Scientific Visualization
    • Self-Driving Vehicles
      • Overview
      • Chauffeur
      • Concierge
      • Training
      • HD Mapping
      • Simulation
      • Robotaxis
      • Trucking
      • ADAS
    • Industries
      • Overview
      • Architecture, Engineering, Construction & Operations
      • Consumer Internet
      • Cybersecurity
      • Energy
      • Financial Services
      • Healthcare and Life Sciences
      • Higher Education
      • Game Development
      • Manufacturing
      • Media and Entertainment
      • US Public Sector
      • Restaurants
      • Retail and CPG
      • Robotics
      • Smart Cities
      • Supercomputing
      • Telecommunications
      • Transportation
    • Startups
    • NVIDIA Studio
      • Overview
      • Accelerated Apps
      • Products
      • Compare
      • Shop
    • Industries
      • Media and Entertainment
      • Manufacturing
      • Architecture, Engineering, and Construction
      • All Industries >
    • Solutions
      • Data Center/Cloud
      • Laptops/Desktops
      • Augmented and Virtual Reality
      • Multi-Display
      • Rendering
      • Metaverse - Omniverse
      • Graphics Virtualization
      • Engineering Simulation
    • Industries
      • Financial Services
      • Consumer Internet
      • Healthcare
      • Higher Education
      • Retail
      • Public Sector
      • All Industries >
    • Solutions
      • Data Analytics
      • Machine Learning
      • Deep Learning Training
      • Deep Learning Inference
      • Conversational AI
      • Prediction and Forecasting
      • Large Language Models
    • Software
      • NGC Catalog
      • NVIDIA NGC
      • RAPIDS
      • Apache Spark
      • Inference Serving
      • Recommender Systems - Merlin
      • Open Source Portal
      • AI Enterprise Suite
      • NVIDIA Workbench
    • Products
      • PC
      • Laptops & Workstations
      • Data Center
      • Cloud
    • Resources
      • Professional Services
      • Technical Training
      • Startups
      • AI Accelerator Program
      • Content Library
      • NVIDIA Research
      • Developer Blog
      • Kaggle Grandmasters
    • Developer Resources
      • Join the Developer Program
      • NGC Catalog
      • NVIDIA NGC
      • Technical Training
      • News
      • Blog
      • Forums
      • Open Source Portal
      • NVIDIA GTC
      • Startups
      • Developer Home >
    • Application Frameworks
      • Metaverse - Omniverse
      • Automotive - DRIVE
      • Cloud-AI Video Streaming - Maxine
      • Speech AI - Riva
      • Data Analytics - RAPIDS
      • Healthcare - Clara
      • High-Performance Computing
      • Intelligent Video Analytics - Metropolis
      • Recommender Systems - Merlin
      • Robotics - Isaac
      • Telecommunications - Aerial
    • Top SDKs and Libraries
      • Parallel Programming - CUDA Toolkit
      • Edge AI applications - Jetpack
      • BlueField data processing - DOCA
      • Accelerated Libraries - CUDA-X Libraries
      • Deep Learning Inference - TensorRT
      • Deep Learning Training - cuDNN
      • Deep Learning Frameworks
      • Conversational AI - NeMo
      • Intelligent Video Analytics - DeepStream
      • NVIDIA Unreal Engine 4
      • Ray Tracing - RTX
      • Video Decode/Encode
    • GeForce
      • Overview
      • GeForce Graphics Cards
      • Gaming Laptops
      • G-SYNC Monitors
      • RTX Games
      • GeForce Experience
      • GeForce Drivers
      • Forums
      • Support
      • Shop
    • GeForce NOW
      • Overview
      • Download
      • Games
      • Pricing
      • FAQs
      • Forums
      • Support
    • SHIELD
      • Overview
      • Compare
      • Shop
      • FAQs
      • Knowledge Base
    • Unwrap an Upgrade
    • Solutions
      • Data Center (On-Premises)
      • Edge Computing
      • Cloud Computing
      • Networking
      • Virtualization
      • Enterprise IT Solutions
    • Software
      • AI Enterprise Suite
      • Cloud Native Support
      • Cluster Management
      • Edge Deployment Management
      • Inference Serving
      • IO Acceleration
      • Networking
      • Virtual GPU
    • Apps and Tools
      • Data Center
      • GPU Monitoring
      • NVIDIA RTX Experience
      • NVIDIA RTX Desktop Manager
    • Resources
      • Data Center & IT Resources
      • Technical Training and Certification
      • Enterprise Support
      • Drivers
      • Security
      • Product Documentation
      • Forums
      • NVIDIA Research Home
      • Research Areas
      • AI Playground
      • Video Highlights
      • COVID-19
      • NGC Catalog
      • Technical Training
      • Startups
      • News
      • Developer Blog
      • Open Source Portal
      • Cambridge-1 Supercomputer
      • 3D Deep Learning Research
    • Products
      • AI Training - DGX
      • Edge Computing - EGX
      • Embedded Computing - Jetson
    • Software
      • Robotics - Isaac SDK
      • Simulation - Isaac Sim
      • TAO Toolkit
      • Vision AI - Deepstream SDK
      • Edge Deployment Management
      • Synthetic Data Generation - Replicator
    • Use Cases
      • Healthcare and Life Sciences
      • Manufacturing
      • Public Sector
      • Retail
      • Robotics
      • More >
    • Resources
      • NVIDIA Blog
      • Robotics Research
      • Developer Blog
      • Technical Training
      • Startups
  • NVIDIA GTC
  • Shop
  • Drivers
  • Support
  • Login LogOut
Skip to main content
  • 0
    Cart
  • Login LogOut
NVIDIA NVIDIA logo
  • Products
    • Hardware
      • Gaming and Entertainment
        • GeForce Graphics Cards
        • Gaming Laptops
        • G-SYNC Monitors
        • SHIELD
      • Laptops and Workstations
        • Gaming Laptops
        • NVIDIA RTX Desktop Workstations
        • NVIDIA RTX in Professional Laptops
        • DGX Station
        • NVIDIA RTX Data Science Workstation
        • Studio products
      • Cloud and Data Center
        • Overview
        • Grace CPU
        • DGX Systems
        • NVIDIA OVX
        • EGX Platform
        • IGX Platform
        • HGX Platform
        • DRIVE Constellation
      • Networking
        • Overview
        • DPU
        • Ethernet
        • InfiniBand
      • GPUs
        • GeForce
        • NVIDIA RTX / Quadro
        • Data Center
        • Titan RTX
      • Embedded Systems
        • Jetson
        • DRIVE AGX
        • Clara AGX
    • Software
      • Application Frameworks
        • Metaverse Applications - Omniverse
        • Automotive - DRIVE
        • Cloud-AI Video Streaming - Maxine
        • Speech AI - Riva
        • Data Analytics - RAPIDS
        • Healthcare - Clara
        • High-Performance Computing
        • Intelligent Video Analytics - Metropolis
        • Recommender Systems - Merlin
        • Robotics - Isaac
        • Telecommunications - Aerial
      • Apps and Tools
        • Application Catalog
        • NGC Catalog
        • NVIDIA NGC
        • 3D Workflows - Omniverse
        • Data Center
        • GPU Monitoring
        • NVIDIA RTX Experience
        • NVIDIA RTX Desktop Manager
        • RTX Accelerated Creative Apps
        • Video Conferencing
        • NVIDIA Workbench
      • Gaming and Creating
        • GeForce NOW Cloud Gaming
        • GeForce Experience
        • NVIDIA Broadcast App
        • Animation - Machinima
        • Modding - RTX Remix
      • Infrastructure
        • AI Enterprise Suite
        • Cloud Native Support
        • Cluster Management
        • Edge Deployment Management
        • Inference Serving
        • IO Acceleration
        • Networking
        • Virtual GPU
      • Cloud Services
        • Metaverse - Omniverse
        • Cloud Gaming
  • Solutions
    • AI and Data Science
      • Overview
      • Data Analytics
      • Machine Learning
      • Deep Learning Training
      • Deep Learning Inference
      • Conversational AI
      • Prediction and Forecasting
      • Speech AI
      • Large Language Models
    • Data Center and Cloud Computing
      • Overview
      • Accelerated Computing for Enterprise IT
      • Cloud Computing
      • Colocation
      • Edge Computing
      • Networking
      • Virtualization
      • MLOps
    • Design and Simulation
      • Overview
      • Augmented and Virtual Reality
      • Multi-Display
      • Rendering
      • Metaverse
      • Graphics Virtualization
      • Engineering Simulation
      • Broadcast
    • Robotics and Edge Computing
      • Overview
      • AI-on-5G
      • Intelligent Video Analytics
      • Industrial
      • Robotics
      • Edge Deployment Management
      • Edge Solutions
    • High-Performance Computing
      • Overview
      • HPC and AI
      • Simulation and Modeling
      • Scientific Visualization
    • Self-Driving Vehicles
      • Overview
      • Chauffeur
      • Concierge
      • Training
      • HD Mapping
      • Simulation
      • Robotaxis
      • Trucking
      • ADAS
  • Industries
    • Industries
      • Overview
      • Architecture, Engineering, Construction & Operations
      • Consumer Internet
      • Cybersecurity
      • Energy
      • Financial Services
      • Healthcare and Life Sciences
      • Higher Education
      • Game Development
      • Manufacturing
      • Media and Entertainment
      • US Public Sector
      • Restaurants
      • Retail and CPG
      • Robotics
      • Smart Cities
      • Supercomputing
      • Telecommunications
      • Transportation
  • For You
    • Creatives/Designers
      • NVIDIA Studio
        • Overview
        • Accelerated Apps
        • Products
        • Compare
        • Shop
      • Industries
        • Media and Entertainment
        • Manufacturing
        • Architecture, Engineering, and Construction
        • All Industries >
      • Solutions
        • Data Center/Cloud
        • Laptops/Desktops
        • Augmented and Virtual Reality
        • Multi-Display
        • Rendering
        • Metaverse - Omniverse
        • Graphics Virtualization
        • Engineering Simulation
    • Data Scientists
      • Industries
        • Financial Services
        • Consumer Internet
        • Healthcare
        • Higher Education
        • Retail
        • Public Sector
        • All Industries >
      • Solutions
        • Data Analytics
        • Machine Learning
        • Deep Learning Training
        • Deep Learning Inference
        • Conversational AI
        • Prediction and Forecasting
        • Large Language Models
      • Software
        • NGC Catalog
        • NVIDIA NGC
        • RAPIDS
        • Apache Spark
        • Inference Serving
        • Recommender Systems - Merlin
        • Open Source Portal
        • AI Enterprise Suite
        • NVIDIA Workbench
      • Products
        • PC
        • Laptops & Workstations
        • Data Center
        • Cloud
      • Resources
        • Professional Services
        • Technical Training
        • Startups
        • AI Accelerator Program
        • Content Library
        • NVIDIA Research
        • Developer Blog
        • Kaggle Grandmasters
    • Developers
      • Developer Resources
        • Join the Developer Program
        • NGC Catalog
        • NVIDIA NGC
        • Technical Training
        • News
        • Blog
        • Forums
        • Open Source Portal
        • NVIDIA GTC
        • Startups
        • Developer Home >
      • Application Frameworks
        • Metaverse - Omniverse
        • Automotive - DRIVE
        • Cloud-AI Video Streaming - Maxine
        • Speech AI - Riva
        • Data Analytics - RAPIDS
        • Healthcare - Clara
        • High-Performance Computing
        • Intelligent Video Analytics - Metropolis
        • Recommender Systems - Merlin
        • Robotics - Isaac
        • Telecommunications - Aerial
      • Top SDKs and Libraries
        • Parallel Programming - CUDA Toolkit
        • Edge AI applications - Jetpack
        • BlueField data processing - DOCA
        • Accelerated Libraries - CUDA-X Libraries
        • Deep Learning Inference - TensorRT
        • Deep Learning Training - cuDNN
        • Deep Learning Frameworks
        • Conversational AI - NeMo
        • Intelligent Video Analytics - DeepStream
        • NVIDIA Unreal Engine 4
        • Ray Tracing - RTX
        • Video Decode/Encode
    • Gamers
      • GeForce
        • Overview
        • GeForce Graphics Cards
        • Gaming Laptops
        • G-SYNC Monitors
        • RTX Games
        • GeForce Experience
        • GeForce Drivers
        • Forums
        • Support
        • Shop
      • GeForce NOW
        • Overview
        • Download
        • Games
        • Pricing
        • FAQs
        • Forums
        • Support
      • SHIELD
        • Overview
        • Compare
        • Shop
        • FAQs
        • Knowledge Base
    • IT Professionals
      • Solutions
        • Data Center (On-Premises)
        • Edge Computing
        • Cloud Computing
        • Networking
        • Virtualization
        • Enterprise IT Solutions
      • Software
        • AI Enterprise Suite
        • Cloud Native Support
        • Cluster Management
        • Edge Deployment Management
        • Inference Serving
        • IO Acceleration
        • Networking
        • Virtual GPU
      • Apps and Tools
        • Data Center
        • GPU Monitoring
        • NVIDIA RTX Experience
        • NVIDIA RTX Desktop Manager
      • Resources
        • Data Center & IT Resources
        • Technical Training and Certification
        • Enterprise Support
        • Drivers
        • Security
        • Product Documentation
        • Forums
    • Researchers
        • NVIDIA Research Home
        • Research Areas
        • AI Playground
        • Video Highlights
        • COVID-19
        • NGC Catalog
        • Technical Training
        • Startups
        • News
        • Developer Blog
        • Open Source Portal
        • Cambridge-1 Supercomputer
        • 3D Deep Learning Research
    • Roboticists
      • Products
        • AI Training - DGX
        • Edge Computing - EGX
        • Embedded Computing - Jetson
      • Software
        • Robotics - Isaac SDK
        • Simulation - Isaac Sim
        • TAO Toolkit
        • Vision AI - Deepstream SDK
        • Edge Deployment Management
        • Synthetic Data Generation - Replicator
      • Use Cases
        • Healthcare and Life Sciences
        • Manufacturing
        • Public Sector
        • Retail
        • Robotics
        • More >
      • Resources
        • NVIDIA Blog
        • Robotics Research
        • Developer Blog
        • Technical Training
        • Startups
    • Startups
    • NVIDIA GTC
    • Shop
    • Drivers
    • Support
Deep Learning Institute
Self Paced Courses
Instructor-Led Workshops
Educator Programs
Enterprise Solutions
Resources
  • Self Paced Courses
  • Instructor-Led Workshops
  • Educator Programs
  • Enterprise Solutions
  • Resources
  • Self Paced Courses
  • Instructor-Led Workshops
  • Educator Programs
  • Enterprise Solutions
  • Resources

 Instructor-Led Workshop
Fundamentals of Deep Learning for Multi-GPUs

Request a workshop for your organization
Notify me when public workshops are available
View public workshops

This workshop teaches you techniques for training deep neural networks on multi-GPU technology to shorten the training time required for data-intensive applications. Working with deep learning tools, frameworks, and workflows to perform neural network training, you’ll learn concepts for implementing Horovod multi-GPUs to reduce the complexity of writing efficient distributed software and to maintain accuracy when training a model across many GPUs.

 

Learning Objectives


At the conclusion of the workshop, you’ll have an understanding of:
  • Stochastic gradient descent (SGD), a crucial tool in parallelized training
  • Batch size and its effect on training time and accuracy
  • Transforming a single-GPU implementation to a Horovod multi-GPU implementation
  • Techniques for maintaining high accuracy when training across multiple GPUs

Download workshop datasheet (PDF 73.3 KB)

Workshop Outline

Introduction
(15 mins)
  • Meet the instructor.
  • Create an account at courses.nvidia.com/join
Stochastic Gradient Descent and the Effects of Batch Size
(120 mins)
  • Understand the issues with sequential single-thread data processing and the theory behind speeding up applications with parallel processing.
  • Explore loss function, gradient descent, and SGD.
  • Learn the effect of batch size on accuracy and training time.
Break (60 mins)
Training on Multiple GPUs with Horovod
(120 mins)
  • Discover the benefits of training on multiple GPUs with Horovod.
  • Learn to transform single-GPU training on the Fashion-MNIST dataset to Horovod multi-GPU implementation.
Break (15 mins)
Maintaining Model Accuracy when Scaling to Multiple GPUs
(120 mins)
  • Understand why accuracy can decrease when parallelizing training on multiple GPUs.
  • Explore tools for maintaining accuracy when scaling training to multiple GPUs.
Final Review
(15 mins)
  • Review key learnings and answer questions.
  • Complete the assessment and earn a certificate.
  • Complete the workshop survey.
  • Learn how to set up your own AI application development environment.
 

Workshop Details

Duration: 8 hours

Price: Contact us for pricing.

Prerequisites: Experience with gradient descent model training

Technologies: TensorFlow, Keras, Horovod

Assessment Type: Code-based

Certificate: Upon successful completion of the assessment, participants will receive an NVIDIA DLI certificate to recognize their subject matter competency and support professional career growth.

Hardware Requirements: Desktop or laptop computer capable of running the latest version of Chrome or Firefox. Each participant will be provided with dedicated access to a fully configured, GPU-accelerated server in the cloud.

Languages: English, Korean, Simplified Chinese, Traditional Chinese

Upcoming Workshops

Pacific Time
GTC
 

Thu, Apr 15, 2021

9:00 a.m.–5:00 p.m.

Register now

If your organization is interested in boosting and developing key skills in AI, accelerated data science, or accelerated computing, you can request instructor-led training from the NVIDIA DLI.

Request a Workshop

Questions?

Contact Us for Questions on
Deep Learning Training

  1. Section
  • Section

Optional

Read our FAQs

Read our FAQs.

NVIDIA Deep Learning institute services

Inquire about NVIDIA Deep Learning Institute services.

NVIDIA Developer Forums

For technical questions, check out the NVIDIA Developer Forums.

DLI Solutions
  • Self-Paced, Online Courses
  • Live Instructor-Led Workshops
  • Educator Programs and Teaching Kits
  • Enterprise Solutions
Products
  • DGX Systems
  • DGX A100
  • DGX Station
  • EGX Platform
  • Data Center GPUs
  • Virtual GPU
  • NVIDIA DRIVE
  • NVIDIA Isaac
  • Jetson
  • GeForce RTX
Technologies
  • CUDA-X AI
  • NGC Catalog
  • Data Analytics
  • Deep Learning SDKs
  • Deep Learning Frameworks
  • Conversational AI
  • Recommender Systems
  • Industry Frameworks
Resources
  • Technical Resources
  • NVIDIA Developer
  • NVIDIA Research
  • GPU Technology Conference
  • Careers
  • Newsroom
  • Deep Learning Blogs
  • NVIDIA On-Demand
Follow NVIDIA
Facebook Twitter LinkedIn Instagram YouTube
NVIDIA
United Kingdom
  • Privacy Policy
  • Manage My Privacy
  • Legal
  • Accessibility
  • Corporate Policies
  • Product Security
  • Contact
Copyright © 2023 NVIDIA Corporation