NDR 400G InfiniBand Architecture

Delivers Extreme Performance for the Exascale AI Era

Learn More

Powering Record-Breaking Performance in Network Communications

The seventh generation of the NVIDIA® Mellanox® InfiniBand architecture, featuring NDR 400 Gb/s InfiniBand, gives AI developers and scientific researchers the fastest networking performance available to take on the world’s most challenging problems. NVIDIA Mellanox InfiniBand® is paving the way with software-defined networking, In-Network Computing acceleration, remote direct-memory access (RDMA), and the fastest speeds and feeds—including impressive advancements over the previous HDR InfiniBand generation.

 

Introducing NVIDIA NDR 400 Gb/s InfiniBand (1:48)

Enhancing HPC and AI Supercomputers and Applications

Accelerated In-Network Computing

Today’s high-performance computing (HPC), AI, and hyperscale infrastructures require faster interconnects and more intelligent networks to analyze data and run complex simulations with greater speed and efficiency. NVIDIA Mellanox InfiniBand enhances and extends its In-Network Computing with preconfigured and programmable compute engines, such as the third generation of NVIDIA Mellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARPv3), MPI Tag Matching, MPI All-to-All, and programmable cores, delivering the best cost per node and ROI.

In-Network Computing accelerated by NVIDIA Mellanox InfiniBand architecture
Data center with NVIDIA Mellanox NDR InfiniBand architecture with SHARPv3 technology

Limitless Scalability

NDR InfiniBand with SHARPv3 technology creates virtually unlimited scalability for large data aggregation through the network, with support for up to 64 parallel flows—32X higher AI acceleration power compared to the previous HDR InfiniBand generation. MPI All-to-All and MPI Tag Matching hardware engines, along with other capabilities such as enhanced congestion control, adaptive routing, and self-healing networking, all provide critical enhancements to HPC and AI clusters, enabling them to achieve even higher performance.

Delivering Data at the Speed of Light

Host Channel Adapters

NDR InfiniBand host-channel adapters (HCAs) with PCIe Gen4 and Gen5 support various form factors, delivering single or dual network ports at 400 gigabits per second (Gb/s) for NDR, 200 Gb/s for NDR200, and lower speeds.

The NDR InfiniBand HCAs include advanced In-Network Computing capabilities with MPI All-to-All and MPI Tag Matching hardware engines and other fabric improvements for enhanced quality of service (QoS), including adaptive routing, congestion control, and more. The HCAs also include programmable compute cores, which enable offloading pre-processing data algorithms and application control paths to the network, providing higher performance, scalability, and enhance overlapping between compute and communication.

Edge Switches

The NDR InfiniBand family of edge switches comprises 64 NDR ports or 128 NDR200 ports on physical 32 Octal Small Form-factor (OSFP) connectors. The compact 1U edge switches include air-cooled and liquid-cooled versions that are either internally or externally managed.

The NDR InfiniBand family of edge switches delivers an aggregated 51.2 terabits per second (Tb/s) of bi-directional throughput, with a capacity of more than 66.5 billion packets per second. As an ideal rack-mounted InfiniBand solution, the NDR InfiniBand edge switch allows maximum flexibility for a variety of topologies, including Fat Tree, DragonFly+, multi-dimensional Torus, Hypercube, and more.

Modular Switches

The NDR InfiniBand family of modular switches provides these port configurations:

  • 2,048 ports of NDR (400 Gb/s) or 4,096 ports of NDR200 (200 Gb/s)
  • 1,024 ports of NDR (400 Gb/s) or 2,048 ports of NDR200 (200 Gb/s)

The largest modular switch carries a total bi-directional throughput of 1.64 petabits per second, 5X over the previous-generation HDR InfiniBand modular switch.

The mid-size modular switch delivers a total bi-directional throughput of 819Tb/s.

Transceivers and Cables

The NDR InfiniBand connectivity options include a variety of transceivers and multi-fiber push-on connectors (MPOs), active copper cables (ACCs), and direct attached cables (DACs) with one to two, and one to four splitter options, providing maximum flexibility to build any topology with the best performance.

World-Leading Networking Performance, Scalability, and Efficiency

Performance

  • 400 Gb/s bandwidth per port
  • 64 NDR 400 Gb/s ports or 128 NDR200 200 Gb/s ports in a single switch
  • 2,048 NDR ports or 4,096 NDR200 ports in a single modular switch
  • Over 66.5 billion packets per second (bi-directional) on a single NDR switch device

Breaking Our Own Records

  • 2X bandwidth per port versus HDR
  • 3X the switch radix versus HDR
  • 32X higher AI acceleration power per switch versus HDR
  • Over one million 400 Gb/s nodes in a four-switch tier (three hops) DragonFly+ network, 6.5X higher than HDR InfiniBand

Key Features

  • Full transport offload
  • RDMA, GPUDirect® RDMA, GPUDirect Storage
  • Programmable In-Network Computing engines
  • MPI All-to-All hardware acceleration
  • MPI Tag Matching hardware acceleration
  • NVIDIA Mellanox SHARPv3
  • Advanced adaptive routing, congestion control, and QoS
  • Self-healing networking

Read the full architecture brief to learn more about NDR.

Architecture Brief