NVIDIA DCGM


Manage and Monitor GPUs in Cluster Environments

NVIDIA Data Center GPU Manager (DCGM) is a suite of tools for managing and monitoring NVIDIA datacenter GPUs in cluster environments. It includes active health monitoring, comprehensive diagnostics, system alerts and governance policies including power and clock management. It can be used standalone by infrastructure teams and easily integrates into cluster management tools, resource scheduling and monitoring products from NVIDIA partners.

DCGM simplifies GPU administration in the data center, improves resource reliability and uptime, automates administrative tasks, and helps drive overall infrastructure efficiency. DCGM supports Linux operating systems on x86_64, Arm and POWER (ppc64le) platforms. The installer packages include libraries, binaries, NVIDIA Validation Suite (NVVS) and source examples for using the API (C, Python and Go).

DCGM also integrates into the Kubernetes ecosystem using DCGM-Exporter to provide rich GPU telemetry in containerized environments.

DCGM is now open-source! Check us out on GitHub!


Benefits



GPU Diagnostics and System Validation

Effectively identify failures, performance degradations, power inefficiencies and their root causes.



GPU Telemetry

Gather rich set of GPU telemetry to explain job behavior, identifying opportunities to drive utilization and efficiencies, and determining root causes of potential application performance issues.



Active GPU Health Monitoring

Use low-overhead, non-invasive health monitoring while jobs are running without impact to application behavior and performance.


Integration with Management Ecosystem

Easily deploy a DCGM based monitoring solution in a Kubernetes cluster environment. Out of the box integration with various ISV solutions such as Bright Cluster Manager, IBM Spectrum LSF and open-source tools such as Prometheus, collectd.

Learn More

Installing the Latest DCGM

By downloading and using the software, you agree to fully comply with the terms and conditions of the NVIDIA DCGM License.

Note that it is recommended to use the latest R450+ NVIDIA datacenter driver that can be downloaded from NVIDIA Driver Downloads page.

As the recommended method, install DCGM directly from the CUDA network repos. Older DCGM releases are also available from the repos.

Quickstart Instructions:

Ubuntu LTS

Set up the CUDA network repository meta-data, GPG key. The example shown below is for Ubuntu 20.04 on x86_64:
$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
$ sudo dpkg -i cuda-keyring_1.0-1_all.deb
$ sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
Install DCGM
$ sudo apt-get update
&& sudo apt-get install -y datacenter-gpu-manager

Red Hat

Set up the CUDA network repository meta-data, GPG key. The example shown below is for RHEL 8 on x86_64:
$ sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
Install DCGM
$ sudo dnf clean expire-cache \
&& sudo dnf install -y datacenter-gpu-manager

Set up the DCGM service

$ sudo systemctl --now enable nvidia-dcgm

Review the release notes and the documentation for install instructions on supported distributions and platforms.

Archived Releases

DCGM 2.1.4 Downloads (January 2021)

By downloading the using the software, you agree to fully comply with the terms and conditions of the NVIDIA DCGM License.

Note that it is recommended to use the latest R450+ NVIDIA datacenter driver that can be downloaded from NVIDIA Driver Downloads page.

You can either DCGM install directly from the CUDA network repos or download the installer packages below.

Quickstart Instructions:

Ubuntu LTS

Set up the CUDA network repository meta-data, GPG key
$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
$ sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
$ sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
Install DCGM
$ sudo apt-get update \
&& sudo apt-get install -y datacenter-gpu-manager

Red Hat

Set up the CUDA network repository meta-data, GPG key
$ sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
Install DCGM
$ sudo dnf clean expire-cache \
&& sudo dnf install -y datacenter-gpu-manager

Review the release notes and the documentation for install instructions on supported distributions and platforms.

If you would like to download the DCGM installer packages, please register for the NVIDIA developer program using the "Join Now" button below. The program is free to join and everyone is accepted. If you are signed up and logged in, you can directly proceed to download the packages.

[nvidia:application_button]

Downloads
DCGM RPM Package (x86_64) RPM
DCGM RPM Packages (POWER) RPM
DCGM RPM Package (Arm64) RPM
DCGM DEB Packages (x86_64) DEB
DCGM DEB Packages (Arm64) DEB

DCGM 2.0.15 Downloads (January 2021)

Please download the DCGM installer packages for your distribution below. Note that it is recommended to use the latest R450 Tesla driver that can be downloaded from NVIDIA Driver Downloads page.

For what's new in DCGM 2.0.15, review the release notes.

By downloading the using the software, you agree to fully comply with the terms and conditions of the EULA.

If you would like to download the DCGM installer packages, please register for the NVIDIA developer program using the "Join Now" button below. The program is free to join and everyone is accepted. If you are signed up and logged in, you can directly proceed to download the packages.

[nvidia:application_button]

Downloads
DCGM RPM Package (x86_64) RPM
DCGM RPM Packages (POWER) RPM
DCGM RPM Package (Arm64) RPM
DCGM DEB Packages (x86_64) DEB
DCGM DEB Packages (POWER) DEB
DCGM DEB Packages (Arm64) DEB

DCGM 2.0.13 Downloads (October 2020)

Please download the DCGM installer packages for your distribution below. Note that it is recommended to use the latest R450 Tesla driver that can be downloaded from NVIDIA Driver Downloads page.

For what's new in DCGM 2.0.13, review the release notes.

By downloading the using the software, you agree to fully comply with the terms and conditions of the EULA.

If you would like to download the DCGM installer packages, please register for the NVIDIA developer program using the "Join Now" button below. The program is free to join and everyone is accepted. If you are signed up and logged in, you can directly proceed to download the packages.

[nvidia:application_button]

Downloads
DCGM RPM Package (x86_64) RPM
DCGM RPM Packages (POWER) RPM
DCGM RPM Package (Arm64) RPM
DCGM DEB Packages (x86_64) DEB
DCGM DEB Packages (POWER) DEB
DCGM DEB Packages (Arm64) DEB

DCGM 2.0.10 Downloads (July 2020)

Please download the DCGM installer packages for your distribution below. Note that it is recommended to use the latest R450 Tesla driver that can be downloaded from NVIDIA Driver Downloads page.

For what's new in DCGM 2.0.10, review the release notes.

If you would like to download the DCGM installer packages, please register for the NVIDIA developer program using the "Join Now" button below. The program is free to join and everyone is accepted. If you are signed up and logged in, you can directly proceed to download the packages. By downloading the using the software, you agree to fully comply with the terms and conditions of the EULA.

[nvidia:application_button]

Downloads
DCGM RPM Package (x86_64) RPM
DCGM RPM Packages (POWER) RPM
DCGM DEB Packages (x86_64) DEB
DCGM DEB Packages (POWER) DEB
EULA PDF

DCGM 1.7.2 Downloads (December 2019)

Please download the DCGM installer packages for your distribution below. Note that this version of DCGM requires at least an R418 Tesla driver that can be downloaded from NVIDIA Driver Downloads page.

For what's new in DCGM 1.7.2, review the release notes.

Downloads
DCGM RPM Package (x86_64) RPM
DCGM Fabric Manager RPM Package (x86_64) RPM
DCGM RPM Packages (POWER) RPM
DCGM DEB Packages (x86_64) DEB
DCGM Fabric Manager DEB Packages (x86_64) DEB
DCGM DEB Packages (POWER) DEB
EULA PDF

DCGM 1.7.1 Downloads (September 2019)

Please download the DCGM installer packages for your distribution below. Note that this version of DCGM requires at least an R384 Tesla driver that can be downloaded from NVIDIA Driver Downloads page.

For what's new in DCGM 1.7.1, review the release notes.

Downloads
DCGM RPM Package (x86_64) RPM
DCGM Fabric Manager RPM Package (x86_64) RPM
DCGM RPM Packages (POWER) RPM
DCGM DEB Packages (x86_64) DEB
DCGM Fabric Manager DEB Packages (x86_64) DEB
DCGM DEB Packages (POWER) DEB
EULA PDF

DCGM 1.6.3 Downloads (April 2019)

Please download the DCGM installer packages for your distribution below. Note that this version of DCGM requires at least an R384 Tesla driver that can be downloaded from NVIDIA Driver Downloads page.

For what's new in DCGM 1.6.3, review the release notes.

Downloads
DCGM RPM Package (x86_64) RPM
DCGM Fabric Manager RPM Package (x86_64) RPM
DCGM RPM Packages (POWER) RPM
DCGM DEB Packages (x86_64) DEB
DCGM Fabric Manager DEB Packages (x86_64) DEB
DCGM DEB Packages (POWER) DEB
EULA PDF

DCGM 1.5.6 Downloads (Dec 2018)

Please download the DCGM 1.5.6 Package for your distribution below. Note that this version of DCGM requires at least an R384 Tesla driver that can be downloaded from NVIDIA Driver Downloads page

Downloads
User Guide & Install Instructions PDF
DCGM API Reference Guide PDF
DCGM Release Notes PDF
DCGM RPM Package (x86_64) RPM
DCGM Fabric Manager RPM Package (x86_64) RPM
DCGM DEB Packages (x86_64) DEB
DCGM Fabric Manager DEB Packages (x86_64) DEB
EULA PDF

What's New in DCGM 1.4.6

  • Added support for Tesla M10, and bandwidth test on Tesla P4
  • Integration with open-source tools such as Prometheus, collectd to report GPU metrics
  • Additional GPU metrics reported by DCGM (e.g. PCIe stats, Memory, Performance states, Video encoder/decoder clocks)
  • Supports the NVIDIA® Tesla® V100 (32GB) GPU accelerator
  • Many other improvements and bug fixes -- see release notes for details

Please download the DCGM 1.4.6 Package for your distribution below. Note that this version of DCGM requires at least an R384 Tesla driver that can be downloaded from NVIDIA Driver Downloads page

Downloads
User Guide & Install Instructions PDF
DCGM API Reference Guide PDF
NVIDIA Validation Suite User Guide PDF
DCGM Release Notes PDF
RPM Packages (x86_64) RPM
RPM Packages (Power8) RPM
DEB Packages (x86_64) DEB
DEB Packages (Power8) DEB
EULA PDF

What's New in DCGM 1.4.2

  • Integration with open-source tools such as Prometheus, collectd to report GPU metrics
  • Additional GPU metrics reported by DCGM (e.g. PCIe stats, Memory, Performance states, Video encoder/decoder clocks)
  • Supports the NVIDIA® Tesla® V100 (32GB) GPU accelerator
  • Many other improvements and bug fixes -- see release notes for details

DCGM 1.4.2 Downloads (May 2018)

Please download the DCGM 1.4.2 Package for your distribution below. Note that this version of DCGM requires at least an R384 Tesla driver that can be downloaded from NVIDIA Driver Downloads page

Downloads
User Guide & Install Instructions PDF
DCGM API Reference Guide PDF
NVIDIA Validation Suite User Guide PDF
DCGM Release Notes PDF
RPM Packages (x86_64) RPM
RPM Packages (Power8) RPM
DEB Packages (x86_64) DEB
DEB Packages (Power8) DEB
EULA PDF

What's New in DCGM 1.3.3

  • DCGM features are now available on non-Tesla GPUs
  • Includes additional GPU diagnostics to stress GPU hardware
  • All functionality of NVVS is now accessible via the DCGM command line interface
  • Supports the NVIDIA® Tesla® V100 Hyperscale PCIe GPU accelerator
  • Many other improvements and bug fixes -- see release notes for details

DCGM 1.3.3 Downloads

Please download the DCGM 1.3.3 Package for your distribution below. Note that this version of DCGM requires at least an R384 Tesla driver that can be downloaded from NVIDIA Driver Downloads page

Downloads
User Guide & Install Instructions PDF
DCGM API Reference Guide PDF
NVIDIA Validation Suite User Guide PDF
DCGM Release Notes PDF
RPM Packages (x86_64) RPM
RPM Packages (Power8) RPM
DEB Packages (x86_64) DEB
DEB Packages (Power8) DEB
EULA PDF

What's New in DCGM 1.2.3

  • Added support for the NVIDIA® Tesla® V100 GPU accelerator
  • Performance improvements - up to 40x speedups for reporting of GPU metrics
  • Added new policy triggers for XID events
  • Bug fixes

DCGM 1.2.3 Downloads

Please download the DCGM 1.2.3 Package for your distribution below. Note that this version of DCGM requires at least an R384 Tesla driver that can be downloaded from NVIDIA Driver Downloads page

Downloads
User Guide & Install Instructions PDF
DCGM API Reference Guide PDF
NVVS User Guide PDF
DCGM Release Notes PDF
RPM Packages (x86_64) RPM
RPM Packages (Power8) RPM
DEB Packages (x86_64) DEB
DEB Packages (Power8) DEB
EULA PDF