# Behind NVIDIA Engineering’s Gigawatt‑Scale AI Factories: Architecture, Deployment, and Performance ## Abstract Key takeaways Dive into a comprehensive analysis of the core NVIDIA DSX components that drive ecosystem enablement at scale: design with DSX Sim, power management with DSX MaxLPS, and service and workload orchestration with DSX OS. Get a detailed look at NVIDIA's Vera Rubin reference architecture and its deployment at scale. Gain practical insights into performance characteristics of next-generation AI workloads and the software stack that drives them. ## AI Summary - Presented the importance of reference architectures for successful and fast AI factory deployments, emphasizing the need for a scalable and reliable design. - Highlighted the significance of fine-grained control and monitoring in data center integration, particularly for liquid cooling and power management, to ensure optimal performance and resilience. - Showed the flexibility of the reference design, which can be deployed in various configurations, including row-based and containerized setups, and discussed the use of tools like DSX Air for fabric design and management. ## Transcript **[00:00:05 – 00:00:06]** Good afternoon. **[00:00:06 – 00:00:08]** Welcome to GCC Berlin. **[00:00:08 – 00:00:11]** Thank you for being with us here today. **[00:00:11 – 00:00:14]** My name is Haidung. I'm in the DGX product team, and **[00:00:14 – 00:00:17]** I'm your host for this session. **[00:00:17 – 00:00:20]** Before we begin, I have two announcements. **[00:00:20 – 00:00:24]** First, please fill out the survey in a mobile app to give **[00:00:24 – 00:00:29]** us feedback on the content and the speakers so that we can continue **[00:00:29 – 00:00:31]** to give you valuable content. **[00:00:31 – 00:00:35]** And then second, very important, please visit the exhibit hall **[00:00:35 – 00:00:39]** to see all the amazing demos that Jensen talked about. **[00:00:39 – 00:00:42]** And don't miss the reception today from five to seven, **[00:00:42 – 00:00:47]** the only time we will have drinks and networking for you today. **[00:00:47 – 00:00:50]** And then it's my honor to introduce the speaker of this **[00:00:50 – 00:00:54]** session, Julie Bernauer. **[00:00:54 – 00:00:59]** She is Senior Director of Applied Systems Engineering at NVIDIA. **[00:00:59 – 00:01:03]** She has been at the company since DGX1, the very first **[00:01:03 – 00:01:07]** AI supercomputer that Jensen delivered to OpenAI. **[00:01:07 – 00:01:12]** Do you help you successfully deploy AI factories at scale? **[00:01:13 – 00:01:16]** Julie's team actually built the early versions of all **[00:01:16 – 00:01:21]** the AI factories at NVIDIA with Vera Rubin and beyond. **[00:01:21 – 00:01:24]** We spent billions of dollars on these clusters so that **[00:01:24 – 00:01:29]** her team can beat them up to test reliability and performance. **[00:01:29 – 00:01:33]** So please welcome Julie. **[00:01:36 – 00:01:39]** Thank you, Aydoun. Thank you guys for being here. **[00:01:39 – 00:01:41]** That was not very flattering, saying I've been here for **[00:01:41 – 00:01:43]** a long, long time. **[00:01:43 – 00:01:46]** But, yeah, there was a lot of ground covered since we shipped **[00:01:46 – 00:01:49]** that first server, and I'm going to try to show you a little bit of **[00:01:49 – 00:01:52]** that today and where we're going. **[00:01:52 – 00:01:54]** First, I'm going to start with the model growth. **[00:01:54 – 00:01:58]** So this slide is already, like, a little bit more than **[00:01:58 – 00:02:01]** a couple months old, and it feels like it changed already several **[00:02:01 – 00:02:06]** times, but I decided to keep it, because there's always the who. **[00:02:06 – 00:02:08]** Why do you build bigger? **[00:02:08 – 00:02:09]** Why do you need more? **[00:02:09 – 00:02:12]** And we're actually trying to show you here how we **[00:02:12 – 00:02:16]** get more efficient by having better way of architecture, **[00:02:16 – 00:02:18]** architecting, and having the reference for you to use. **[00:02:18 – 00:02:22]** So here, a mixture of experts, our models that are in this **[00:02:22 – 00:02:26]** case open source and that are helping a lot of the understanding, **[00:02:26 – 00:02:30]** but they have a tendency to need a lot of communications. **[00:02:30 – 00:02:33]** And in the new models we see, we see all those models talking **[00:02:33 – 00:02:37]** to each other very much and really benefiting from the new builds **[00:02:37 – 00:02:42]** that we're going to see, and in particular, the scale-out fabrics. **[00:02:42 – 00:02:47]** So we heard a lot in the previous talk, and you'll hear a lot **[00:02:47 – 00:02:52]** of this today, again, about the reference, right? **[00:02:52 – 00:02:55]** At the time we started, we wanted to make sure every single **[00:02:55 – 00:02:58]** one of you was successful, and that's still our objective today. **[00:02:58 – 00:03:01]** For that, we need to deploy, we need to deploy fast, and **[00:03:01 – 00:03:06]** it has to be architected so that it could scale to a gigawatt **[00:03:06 – 00:03:08]** scale without any problem. **[00:03:08 – 00:03:11]** And so at the core of the cluster that is built inside **[00:03:11 – 00:03:14]** an AI factory, you have an architecture that integrates **[00:03:14 – 00:03:19]** a cluster design that has a design for compute, networking, storage, **[00:03:19 – 00:03:24]** and all the facilities looked at each other end-to-end, together. **[00:03:24 – 00:03:26]** For that, you need the infrastructure to be able **[00:03:26 – 00:03:27]** to be fully automated. **[00:03:27 – 00:03:32]** We're going to look at state-laced provisioning and recovery. **[00:03:32 – 00:03:35]** You basically cannot have an operator or an agent for **[00:03:35 – 00:03:37]** everything you do. **[00:03:37 – 00:03:41]** Integration, of course, you need a software stack that **[00:03:41 – 00:03:44]** is also end-to-end, making sure your OS, your scheduling, all your **[00:03:44 – 00:03:48]** identity management is integrated with it and making sure that **[00:03:48 – 00:03:52]** all the components work together with the existing ecosystem. **[00:03:53 – 00:03:55]** Every single one of us have a special application in mind, **[00:03:55 – 00:04:01]** and we need to keep that in mind as we go and integrate those things. **[00:04:01 – 00:04:03]** Then you have to run it. **[00:04:03 – 00:04:06]** Someone said in a previous talk, like, when people want **[00:04:06 – 00:04:09]** it to happen, they assume it's going to happen. **[00:04:09 – 00:04:11]** There's a little bit of a couple of steps here that **[00:04:11 – 00:04:13]** you might be familiar with. **[00:04:13 – 00:04:16]** And we're going to talk about telemetry and diagnostics, **[00:04:16 – 00:04:19]** how you monitor how things are done, how you diagnose **[00:04:19 – 00:04:22]** and then you recover from issues, but also how you maintain **[00:04:22 – 00:04:25]** resilience in a system, right? **[00:04:25 – 00:04:27]** We design at that scale for failure. **[00:04:27 – 00:04:31]** At this scale, statistics will run against you, bye, **[00:04:31 – 00:04:35]** just by the sheer numbers and how you make sure you can fail **[00:04:35 – 00:04:39]** and recover very fast and using a good reference, then you can keep **[00:04:39 – 00:04:41]** on deploying and deploy faster. **[00:04:41 – 00:04:43]** So you've heard a lot about the reference architecture. **[00:04:43 – 00:04:46]** Sometimes people are like, oh, but why should I do it **[00:04:46 – 00:04:48]** the way you guys have said, right? **[00:04:48 – 00:04:50]** Why are you mandating all this? **[00:04:50 – 00:04:52]** We're not. We're just telling you that if you want to go **[00:04:52 – 00:04:56]** faster, if we have a cookie cutter kind of design, it's **[00:04:56 – 00:04:58]** going to get easier for everyone. **[00:04:58 – 00:05:00]** And we're just offering a version of the cookie cutter **[00:05:01 – 00:05:04]** for you to go and make cookies. **[00:05:05 – 00:05:07]** I mentioned reliability. **[00:05:07 – 00:05:11]** One of the good old metrics we haven't looked at in a **[00:05:11 – 00:05:15]** while, but we start looking back at now that systems are getting **[00:05:15 – 00:05:21]** bigger and bigger, is the MTBI, for Mean Time Between Interrupts. **[00:05:21 – 00:05:25]** It's very important here because it's a good representation **[00:05:25 – 00:05:30]** of what I, as a user, or as an agent, or a set of agents **[00:05:31 – 00:05:35]** running on an infrastructure, have a feeling about are good **[00:05:35 – 00:05:39]** the infrastructure is and can support what I want to do. **[00:05:39 – 00:05:42]** So, you see that little weird graph here? **[00:05:42 – 00:05:44]** This is a real graph. **[00:05:44 – 00:05:47]** I'm not going to show you the scale, I'm not going to show **[00:05:47 – 00:05:52]** you the details of it, but I do mention that we deploy at NVIDIA. **[00:05:52 – 00:05:57]** We deploy before every single one of you do, so that you **[00:05:57 – 00:06:00]** don't have to see the early version of that curve. **[00:06:01 – 00:06:04]** Here, what we want to have is a good MTBI. **[00:06:04 – 00:06:10]** That means we want everyone to be easily above 100k ours MTBI, **[00:06:10 – 00:06:14]** so that if you're a user, you can run at a minimum for that amount **[00:06:14 – 00:06:17]** of time without seeing a failure. **[00:06:17 – 00:06:21]** And with a lot of components that have a certain DPPI and **[00:06:21 – 00:06:25]** so on, making sure we maintain that resiliency is going to take **[00:06:25 – 00:06:28]** a little bit of thinking ahead. **[00:06:29 – 00:06:33]** Why do we care and why do we do that? **[00:06:33 – 00:06:39]** I often use an analogy of a Formula One car, right? **[00:06:39 – 00:06:42]** The first time you take it to the track, you get a good feeling **[00:06:42 – 00:06:46]** that it has a lot of potential, but getting it to running reliably **[00:06:46 – 00:06:48]** every Sunday is a different story. **[00:06:48 – 00:06:52]** And so here, if I look at agentic workload performance, **[00:06:52 – 00:06:55]** we're trying to make sure that with a lot of interactivity from **[00:06:55 – 00:06:59]** one generation to the next, with minimal updates to the environment, **[00:06:59 – 00:07:02]** ecosystem, and the design, we can actually leverage more **[00:07:02 – 00:07:05]** and more capability per megawatt. **[00:07:05 – 00:07:09]** It's not just do more and expect more, it's just tune **[00:07:09 – 00:07:13]** it so that you can have a better efficiency and use **[00:07:13 – 00:07:16]** your capacity as much as possible. **[00:07:16 – 00:07:20]** So on this, that's basically where we are with Vera **[00:07:20 – 00:07:22]** Rubin and VL72 today. **[00:07:22 – 00:07:25]** So it looks like a slide that is a little bit polished, **[00:07:25 – 00:07:29]** but this is actually measured data. **[00:07:29 – 00:07:35]** So overall, we do that for a lot of the agentic and performance **[00:07:35 – 00:07:37]** workloads we have for AI. **[00:07:37 – 00:07:42]** So on the right at the bottom, you can see an actual tuning **[00:07:42 – 00:07:45]** performance graph that was obtained for training. **[00:07:45 – 00:07:47]** And basically, we have that for end-to-end. **[00:07:47 – 00:07:50]** And the platform you're going to see today are the platform **[00:07:50 – 00:07:52]** where those results are obtained. **[00:07:52 – 00:07:55]** Those results are completely reproducible, so if you have **[00:07:55 – 00:07:58]** your own Infra and that's not what you see, I'm pretty sure you **[00:07:58 – 00:08:00]** can talk to someone to get there. **[00:08:00 – 00:08:06]** But the idea is we validated the system quality, reliability, **[00:08:06 – 00:08:09]** and resilience as early as we can, so that when you go and **[00:08:10 – 00:08:14]** deploy, you can get the software and the performance out of the box. **[00:08:14 – 00:08:17]** OK, so we've been doing that for a while. **[00:08:17 – 00:08:20]** This slide is a little bit, it's both a very happy slide **[00:08:20 – 00:08:23]** because it's an anniversary, but it's always a little bit **[00:08:23 – 00:08:25]** challenging to see that. **[00:08:25 – 00:08:29]** So this year marks the 10-year anniversary of DGX, which **[00:08:29 – 00:08:31]** was the first server we did. **[00:08:31 – 00:08:36]** The reason this slide is very, you know, close to our hearts **[00:08:36 – 00:08:40]** is because we went from one server to putting servers together in **[00:08:40 – 00:08:44]** a way that now it's all integrated and performance is core and part of **[00:08:44 – 00:08:47]** what we do at scale since day one. **[00:08:47 – 00:08:53]** And so here you see a bunch of the top supercomputers that we've built **[00:08:53 – 00:08:58]** along the way and the people we've enabled as we go, and we hope that **[00:08:58 – 00:09:01]** we're going to keep on having that. **[00:09:01 – 00:09:06]** Interestingly, you'll see also here the DGX Spark launch; **[00:09:06 – 00:09:08]** you've seen it this morning. **[00:09:08 – 00:09:12]** The code that runs on those machines is the actual same **[00:09:12 – 00:09:14]** that runs on our laptop, and that is still very, **[00:09:14 – 00:09:19]** very true, which makes the loop of deployment very, very fast. **[00:09:19 – 00:09:23]** And that's one of the perks of that ecosystem. **[00:09:24 – 00:09:27]** So, I'm going to start with the part that is often the **[00:09:27 – 00:09:33]** most difficult for us, which is data center integration. **[00:09:34 – 00:09:37]** This is the part of the integration, or the part **[00:09:37 – 00:09:40]** of the design, that is usually a lengthy one, the one that **[00:09:40 – 00:09:42]** we're trying to accelerate. **[00:09:42 – 00:09:44]** You've seen the DSX part this morning. **[00:09:44 – 00:09:46]** We're just trying to pull in to make sure it doesn't **[00:09:46 – 00:09:50]** take years to actually get a data center together. **[00:09:50 – 00:09:53]** So the idea is really to have something that is repeatable, **[00:09:53 – 00:09:57]** that you can use like a Lego block, right? **[00:09:57 – 00:10:02]** And so that as you go and commission that, the commissioning **[00:10:02 – 00:10:05]** time is very, very small, as soon as you can. **[00:10:05 – 00:10:08]** We had an intern who came to our team some time ago, **[00:10:08 – 00:10:12]** and she was like, so why are we trying to do that in six months **[00:10:12 – 00:10:14]** while people do it in three years? **[00:10:14 – 00:10:16]** And it's part of because we can and because that's what's **[00:10:16 – 00:10:21]** going to accelerate our research and our first-in-time token. **[00:10:21 – 00:10:25]** For that, we need to minimize the site adaptation. **[00:10:25 – 00:10:29]** It's actually very challenging to have to retrofit something. **[00:10:29 – 00:10:33]** And so if we can accommodate the data center constraint **[00:10:33 – 00:10:37]** in the layout in terms of power, cooling, and cabling, **[00:10:37 – 00:10:39]** that's actually easier. **[00:10:39 – 00:10:42]** And then make sure that for every single deployment we **[00:10:42 – 00:10:46]** have, the data that everyone has or that we showcase to you, **[00:10:46 – 00:10:53]** you can actually get on this site configuration that comes to you. **[00:10:53 – 00:10:57]** So for that, as a building block, today we're using **[00:10:57 – 00:10:59]** Vera Rubin and VL72. **[00:10:59 – 00:11:03]** So exactly like the previous generation, it's a rack-scale **[00:11:03 – 00:11:05]** infrastructure architecture. **[00:11:05 – 00:11:07]** So we have a rack. **[00:11:07 – 00:11:09]** You can see some on the demo floor. **[00:11:09 – 00:11:15]** It has like 18 compute trays made of 36 Vera CPUs. **[00:11:16 – 00:11:21]** 72 Rubin GPUs, they are connected to the east-west fabric using **[00:11:21 – 00:11:25]** ConnectX Compute Nix, and the whole rack has a multi-node **[00:11:25 – 00:11:27]** and the link connectivity between all the GPUs in the **[00:11:27 – 00:11:31]** rack, that I'm going to call the scale-out fabric for the following **[00:11:31 – 00:11:35]** of his talk, that is obtained through a copper back thing. **[00:11:35 – 00:11:38]** It has nine anvil-link switches, trays in the rack. **[00:11:38 – 00:11:41]** Those are the ones that you can see in the middle, right? **[00:11:41 – 00:11:46]** And this rack has dedicated power shelves and a rack busbar **[00:11:46 – 00:11:49]** for power distribution. **[00:11:49 – 00:11:52]** So how do we design to put it in there? **[00:11:52 – 00:11:55]** So that's the pod that is a real engineering talk. **[00:11:55 – 00:11:58]** Those are not renderings, right? **[00:11:58 – 00:12:02]** So you've seen this morning Jensen talking about how we integrate **[00:12:02 – 00:12:03]** in a data center and so on. **[00:12:03 – 00:12:06]** Those are straight as built and picture taken from Procore and the **[00:12:06 – 00:12:10]** data center definitions we have. **[00:12:10 – 00:12:14]** So we organize into pods, the same way we have the DSX pod. **[00:12:14 – 00:12:16]** This is the same reference architecture. **[00:12:16 – 00:12:19]** You're going to see that the number aligns, and we do it exactly the **[00:12:19 – 00:12:23]** same way, whether we're in a data center or in a container, right? **[00:12:23 – 00:12:27]** And we align the power, the liquid cooling, and the network **[00:12:27 – 00:12:28]** fabric in one layout. **[00:12:28 – 00:12:31]** So again, this idea of having Lego blocks. **[00:12:31 – 00:12:36]** And basically what we do is we plan the rack rows and the containment **[00:12:36 – 00:12:37]** around that rack design. **[00:12:37 – 00:12:40]** So that's also something you've seen on the floor with Vendor, **[00:12:40 – 00:12:44]** is basically those building blocks are then wrapped into **[00:12:44 – 00:12:47]** the design that is around it. **[00:12:47 – 00:12:51]** So this is the rack on the right, this is an example **[00:12:51 – 00:12:54]** of a pod on the left at the bottom. **[00:12:54 – 00:12:56]** We're going to see it a little bit later. **[00:12:56 – 00:13:01]** That's an NVL576 pod, that's an engineering experimental pod, **[00:13:01 – 00:13:07]** and in the middle it's a nice build with one of our internal designs. **[00:13:07 – 00:13:10]** So how do we do the data center integration? **[00:13:10 – 00:13:12]** So we have several ways. **[00:13:13 – 00:13:14]** Some people are like, I want to have rows. **[00:13:14 – 00:13:15]** I want to have a HAC. **[00:13:15 – 00:13:17]** I want to have a containment. **[00:13:17 – 00:13:20]** The reference architectures just provide guidelines on what the **[00:13:20 – 00:13:22]** building blocks should look like. **[00:13:22 – 00:13:26]** But there's a lot of options into how do you put it in there? **[00:13:26 – 00:13:31]** So on the top right here, I've shown a hothell containment. **[00:13:32 – 00:13:36]** But on the bottom left here, I'm showing you a row-based design. **[00:13:36 – 00:13:40]** The power distribution, the cooling loops, and the cable **[00:13:40 – 00:13:44]** ladders, they still follow the same coordinated repeatable layout **[00:13:44 – 00:13:46]** that scale to larger deployments. **[00:13:46 – 00:13:49]** And as some of you might have seen on the previous talk **[00:13:49 – 00:13:52]** where it was presented the same way, you can see at the **[00:13:52 – 00:13:57]** bottom right the cooling loops and the CDUs that are attached. **[00:13:57 – 00:14:06]** Okay, so I got my CDUs, I got my HAC or my rack containment units. **[00:14:06 – 00:14:10]** I still need to be able to understand what's going on, making **[00:14:10 – 00:14:12]** sure my flows are good and so on. **[00:14:12 – 00:14:17]** So in a data center today, everything is. **[00:14:19 – 00:14:23]** Some people in my team I think haven't even seen the data **[00:14:23 – 00:14:25]** center we work on. **[00:14:25 – 00:14:28]** It doesn't matter. We feel like it's talking to us like **[00:14:28 – 00:14:30]** every day, several times a day. **[00:14:30 – 00:14:34]** So for example, for liquid cooling, you can see it at **[00:14:34 – 00:14:38]** the bottom; we have electronic pressure independent valve **[00:14:38 – 00:14:41]** at each inlet from the secondary cooling loop that enables **[00:14:41 – 00:14:45]** full control of the liquid cooling to the rack, we monitor the **[00:14:45 – 00:14:49]** flow, we monitor everything that gets in there, the differential **[00:14:49 – 00:14:51]** pressure, the delta T, et cetera. **[00:14:51 – 00:14:54]** So we got the coolant temperature monitoring, and just in **[00:14:54 – 00:14:57]** case something happens, we discussed earlier failure **[00:14:57 – 00:15:01]** and resiliency, making sure we do have automatic shocked **[00:15:01 – 00:15:05]** off to prevent any issue in terms of blast radius or everything. **[00:15:05 – 00:15:08]** So in the past, we used to look at this like, oh, I'm going **[00:15:08 – 00:15:10]** to protect the room and so on. **[00:15:10 – 00:15:12]** Now what we're doing is we're bringing the control as close **[00:15:13 – 00:15:17]** as possible to every single unit to make sure we actually **[00:15:17 – 00:15:21]** can have a fine-grained control of all the units as they get deployed. **[00:15:22 – 00:15:26]** And we can have any kind of automated response to events. **[00:15:26 – 00:15:29]** It could be primary loop temperature. **[00:15:29 – 00:15:30]** I mean, I was here this summer. **[00:15:30 – 00:15:34]** That was a little bit challenging, right? **[00:15:34 – 00:15:36]** One can decide to slow down. **[00:15:36 – 00:15:40]** One can decide to change, update, control how we want **[00:15:40 – 00:15:43]** to manage what we're doing. **[00:15:43 – 00:15:46]** We don't want to talk about leaks, because that rarely happens, **[00:15:46 – 00:15:50]** but when it happens, we're super happy to make sure that we don't **[00:15:51 – 00:15:55]** have to have someone intervening at 3 a.m. because we got a **[00:15:55 – 00:15:56]** swimming pool in the data center. **[00:15:56 – 00:16:01]** So having that fine-grained control allows at that flow rate to **[00:16:01 – 00:16:07]** make sure that everything is always monitored and there's no incident. **[00:16:07 – 00:16:10]** So that's basically it. **[00:16:10 – 00:16:12]** Now, there's a bunch of, each time we deep dive, so those **[00:16:12 – 00:16:18]** are the high-level sensors for environmentals, but also if we deep **[00:16:18 – 00:16:22]** dive, like for example, I took the example here of a hot containment **[00:16:22 – 00:16:26]** unit, we might want to have even more fine-grained control and **[00:16:26 – 00:16:28]** understanding of what's going on. **[00:16:28 – 00:16:32]** In a hot health containment unit, you need to make sure you **[00:16:32 – 00:16:38]** understand your temperature, your humidity content, your noise level. **[00:16:38 – 00:16:40]** We still have operators, we want to make sure it's safe **[00:16:40 – 00:16:44]** for them to operate or to do whatever we need to do in there **[00:16:44 – 00:16:47]** when we have like a replacement of someone that needs to go **[00:16:47 – 00:16:49]** and Let's go and check something. **[00:16:49 – 00:16:52]** And basically making sure everything from temperature, **[00:16:52 – 00:16:57]** humidity, noise, airflow is also tracked under control **[00:16:57 – 00:16:58]** and always looked at. **[00:16:58 – 00:17:01]** The joke we have in engineering is that if you sneeze in a **[00:17:01 – 00:17:03]** data center, we should be able to detect it. **[00:17:03 – 00:17:06]** That's how we make sure everything is fine. **[00:17:06 – 00:17:09]** The telemetry is collected and visualized using standard tools. **[00:17:09 – 00:17:13]** So here I'm showing a very open source engineering example **[00:17:13 – 00:17:15]** with Prometheus and Grafana, but there's a lot of solutions **[00:17:15 – 00:17:17]** out there, right? **[00:17:17 – 00:17:21]** And the environment data that we've seen earlier is actually combined **[00:17:21 – 00:17:23]** with all the system metrics. **[00:17:23 – 00:17:27]** That means the CPUs, the GPUs, what's running on the machine, **[00:17:27 – 00:17:29]** to make sure we control power. **[00:17:29 – 00:17:32]** At the same time, we control the environmentals, the fabric **[00:17:32 – 00:17:39]** utilization, any kind of workload usage, understanding, and **[00:17:39 – 00:17:40]** CICD workload coming up. **[00:17:41 – 00:17:44]** So that gives us, and we're going to follow and see that **[00:17:44 – 00:17:47]** a little bit after, full control of what's going on. **[00:17:47 – 00:17:51]** We actually understand what's coming to work on a machine. **[00:17:52 – 00:17:55]** We know where we are in terms of consumption, environmental, **[00:17:55 – 00:18:01]** and we can actually decide to just slow down, accelerate, rise, **[00:18:01 – 00:18:05]** completely manage what we want to make the best use of the resources **[00:18:05 – 00:18:08]** we have at any point in time. **[00:18:08 – 00:18:13]** We had this example in California where sometimes in the afternoon **[00:18:13 – 00:18:18]** we're having power challenges and we have to slow down things. **[00:18:18 – 00:18:23]** And it's actually something we can do at the push of a button. **[00:18:23 – 00:18:26]** What happens at the gigawatt scale, like, you're like, **[00:18:26 – 00:18:29]** okay, she's very nice, but she's showing eight racks. **[00:18:29 – 00:18:31]** It's the same thing. **[00:18:31 – 00:18:32]** It's the Lego block. **[00:18:32 – 00:18:35]** If you can do a small set, if you've done that with your kids, **[00:18:35 – 00:18:40]** ideally a larger set is a little bit longer, but it's the same idea. **[00:18:40 – 00:18:42]** And that's exactly what's going on. **[00:18:42 – 00:18:46]** So same thing, same design, just duplicated, right? **[00:18:46 – 00:18:48]** The model extends to any size, and it loads for **[00:18:48 – 00:18:50]** incremental deployment. **[00:18:50 – 00:18:52]** So that's also something that is very convenient. **[00:18:52 – 00:18:55]** Sometimes we get power tranches like, oh, you're **[00:18:55 – 00:18:59]** gonna get 50 megawatt this year, 50 megawatt next **[00:19:00 – 00:19:02]** Quarter, et cetera, et cetera. **[00:19:02 – 00:19:06]** That allows us to ramp and to actually deploy as we want **[00:19:06 – 00:19:09]** and potentially in phases, just attaching and keeping **[00:19:09 – 00:19:11]** on attaching as we go. **[00:19:11 – 00:19:15]** The fabric topology and the deployment matters to ensure **[00:19:15 – 00:19:16]** we have best performance. **[00:19:16 – 00:19:19]** So it's always better to know how much you're going **[00:19:19 – 00:19:20]** to deploy by day one. **[00:19:21 – 00:19:25]** But by default, you can keep on extending once you know that. **[00:19:25 – 00:19:26]** Okay, so power. **[00:19:26 – 00:19:32]** At a gigawatt scale, if one is not using all the power in a rack, that **[00:19:32 – 00:19:35]** can start counting a lot, okay? **[00:19:35 – 00:19:43]** And so everything that is land power shell, at that scale, 0.01%, **[00:19:43 – 00:19:45]** to be simple, is a lot of money. **[00:19:46 – 00:19:48]** So making sure that the power is used in the best **[00:19:48 – 00:19:53]** managed way possible is actually very, very important. **[00:19:53 – 00:19:56]** And so that allows to put more racks in the data center **[00:19:56 – 00:19:59]** and to use the power we have in the most efficient manner. **[00:19:59 – 00:20:03]** So here, I just use an example that people can deploy today **[00:20:03 – 00:20:05]** that we have in a data center. **[00:20:05 – 00:20:08]** This is an example of how we do it if we don't manage **[00:20:08 – 00:20:11]** anything on the top right. **[00:20:11 – 00:20:14]** Basically, not every single one of your workload is gonna use the **[00:20:14 – 00:20:19]** 200 plus kilowatts that you will have deployed in every rack, right? **[00:20:19 – 00:20:23]** So you want to make sure that you can use that for **[00:20:23 – 00:20:25]** something else eventually. **[00:20:25 – 00:20:28]** So I've shown an example here with 130 kilowatt racks, which **[00:20:28 – 00:20:31]** is equivalent to some GB family. **[00:20:31 – 00:20:34]** And so basically, as you know all the jobs that come in **[00:20:34 – 00:20:37]** here, as you know what's coming, you can actually cap and make **[00:20:37 – 00:20:43]** sure that you schedule the jobs to actually do that as a dependency **[00:20:43 – 00:20:46]** of how much power you have in which zone in your data center. **[00:20:46 – 00:20:49]** So that's an example that people can download today and actually **[00:20:49 – 00:20:54]** use to manage the capability here. **[00:20:54 – 00:20:57]** And in practice, what does it look like with VR and VL72? **[00:20:57 – 00:21:02]** So again, those are not renderings. **[00:21:02 – 00:21:03]** They are actually real drawings. **[00:21:03 – 00:21:07]** Some of them even have the red markings of the page turn **[00:21:07 – 00:21:09]** for construction. **[00:21:09 – 00:21:14]** This is an example of a row-based design with rack containment units. **[00:21:14 – 00:21:17]** So that's one that is a little bit more, I'm going to call **[00:21:17 – 00:21:23]** that exotic or less known, So where each rack has its rack containment. **[00:21:23 – 00:21:27]** And that's also well-sitted to very large data centers. **[00:21:27 – 00:21:32]** Sometimes we have rooms that are very, very long, sometimes **[00:21:32 – 00:21:36]** industrial sites, and not very wide, and where the deployment **[00:21:36 – 00:21:38]** in HACS is not convenient. **[00:21:38 – 00:21:41]** So that's an example where we deployed using RCUs, so **[00:21:41 – 00:21:43]** rack containment units. **[00:21:43 – 00:21:46]** And the cooling is divided into loops that are four **[00:21:46 – 00:21:50]** to eight sections that are redundant the same way. **[00:21:50 – 00:21:53]** And we can just arrange that in a row with an optional **[00:21:53 – 00:21:55]** overhead walkway in this case. **[00:21:56 – 00:21:59]** And so the cable ladders and all the power comes from the top. **[00:21:59 – 00:22:03]** So you can see it here on the top right of the slide. **[00:22:03 – 00:22:05]** That's an example with the coolant units. **[00:22:05 – 00:22:09]** And then you can see how they go down to the RCUs and how it gets **[00:22:09 – 00:22:14]** deployed overall in a room that is very lengthy and not very wide. **[00:22:14 – 00:22:17]** But the same reference, and we're going to see number **[00:22:17 – 00:22:21]** of racks and so on, it's the same pattern all over again. **[00:22:21 – 00:22:23]** The next slide is a little bit more interesting. **[00:22:23 – 00:22:25]** That's a picture you might have seen everywhere. **[00:22:25 – 00:22:29]** So the same thing, that's not a rendering considering the cabling. **[00:22:29 – 00:22:34]** That's a prototype of a Vera Rubin Ultra announcement that **[00:22:34 – 00:22:40]** we had earlier this year, where the NVLink domain is 576 GPUs. **[00:22:40 – 00:22:44]** So the NVLink scale-out fabric between all those GPUs actually **[00:22:44 – 00:22:49]** works all together with 576 at the same time. **[00:22:50 – 00:22:54]** It's a large-scale scale-out domain that maps the same **[00:22:54 – 00:22:57]** thing, and it's a two-layer, all-to-all NVLink topology. **[00:22:57 – 00:23:00]** So this time it's not inside a cartridge, it's made through **[00:23:00 – 00:23:05]** optical cables, but it's the same way of doing it. **[00:23:05 – 00:23:10]** Right, and each pod has eight racks, right, with the racks **[00:23:10 – 00:23:15]** are the same 72 Vera Rubin racks that you've seen earlier. **[00:23:15 – 00:23:17]** So there's a block pass if you want to see how those larger **[00:23:17 – 00:23:21]** domains will get deployed, but we're going to keep the same ID. **[00:23:21 – 00:23:25]** The idea here is that if you prepare your data center and **[00:23:25 – 00:23:28]** so on for VR and, like, I don't know, you decide to change **[00:23:28 – 00:23:31]** the way you do it, you should be able to use the same pattern, **[00:23:31 – 00:23:35]** the same data center, and overall not have to redo it every time. **[00:23:35 – 00:23:39]** It should be cookie cutter again, the scalable unit and **[00:23:39 – 00:23:41]** the reference pod. **[00:23:41 – 00:23:45]** So in the previous talk, Charlie and Anish talked about the DSX pod. **[00:23:45 – 00:23:48]** And people were like, oh, that's something new. **[00:23:48 – 00:23:50]** It's going to be very different, and so on. **[00:23:50 – 00:23:53]** And it was mentioned how we're planning to use the DSX pod **[00:23:53 – 00:23:57]** to pull in on all the time it would take for people that don't have **[00:23:57 – 00:24:02]** that opportunity to do the tenant fit out for a data center or room, **[00:24:02 – 00:24:06]** and actually bring that back to manufacturing into the container. **[00:24:06 – 00:24:09]** And you're like, oh, it's going to be very different then. **[00:24:09 – 00:24:11]** Actually, it's not. **[00:24:11 – 00:24:13]** It's the same reference in the background. **[00:24:13 – 00:24:15]** It's the same number of racks. **[00:24:15 – 00:24:18]** It's the same LEGO brick, just happened that that LEGO **[00:24:18 – 00:24:20]** block is in a container this time. **[00:24:20 – 00:24:23]** And so the deployment that is here would be the same, **[00:24:23 – 00:24:25]** independently of how it's deployed. **[00:24:25 – 00:24:28]** In the back end, if you train your teams or if you're working **[00:24:28 – 00:24:34]** with your partner, it's the same ID and, again, the same operations. **[00:24:34 – 00:24:38]** Decentralized fabrics help accelerate all the cable management **[00:24:38 – 00:24:43]** and deployment, and we have the leaf spine in this case that **[00:24:43 – 00:24:47]** are located near the compute racks to limit the number of cables, and **[00:24:47 – 00:24:52]** that allows us to deploy in several containers or in several rooms. **[00:24:52 – 00:24:56]** Let's take a little bit more time to talk about the scale-up **[00:24:56 – 00:24:58]** and the scale-out fabric. **[00:24:59 – 00:25:02]** I'm going to start with the east-west fabric, which is **[00:25:02 – 00:25:05]** based here on Spectrum Ethernet. **[00:25:05 – 00:25:09]** It's the same idea we've deployed in the past. **[00:25:09 – 00:25:11]** So some of you might be familiar with InfiniBand. **[00:25:11 – 00:25:14]** The radixes are a little bit different, but at the end of **[00:25:14 – 00:25:16]** the day, it's the same idea again. **[00:25:16 – 00:25:19]** We take that block, and you'll notice the size of the blocks **[00:25:19 – 00:25:24]** happen to match the size of the networking that deploys, right? **[00:25:24 – 00:25:26]** So there's a very convenient aspects here. **[00:25:26 – 00:25:31]** The blob matches like two CDUs redundant, matches a leaf **[00:25:31 – 00:25:37]** spine block or a set of leaves, and matches also like two megawatt **[00:25:37 – 00:25:41]** for seven kind of deployment that can be duplicated all the way. **[00:25:41 – 00:25:45]** So it kinds of like align in a very convenient way. **[00:25:45 – 00:25:49]** So for Vera Rubin and the L72 scale deployment, we can **[00:25:49 – 00:25:55]** use the SN6600 or SN6810 with 128 ports and shuffle boxes, **[00:25:55 – 00:25:59]** or the SN6800 with 512 ports each. **[00:25:59 – 00:26:01]** It's just a different form factor. **[00:26:01 – 00:26:04]** The rack location of the switches is identical. **[00:26:04 – 00:26:07]** So you've seen in my previous slide, I put them in blue. **[00:26:07 – 00:26:09]** I got my little engineering color code, like compute is **[00:26:09 – 00:26:11]** green, networking is blue. **[00:26:11 – 00:26:15]** And it's identical, and it leverages the same topology **[00:26:15 – 00:26:17]** and the same fiber. **[00:26:17 – 00:26:22]** The same functionality is leveraged for scale, and we're going to see **[00:26:22 – 00:26:25]** how it's a multi-plane design, and the same operation and management **[00:26:25 – 00:26:29]** for software applies to those. **[00:26:29 – 00:26:32]** Jensen actually showed one this morning, the beautiful **[00:26:32 – 00:26:34]** CPU1, same ID, right? **[00:26:34 – 00:26:38]** It's the same block for multiple topology. **[00:26:38 – 00:26:43]** So that's an example of a SpectrumX and VL72 reference topology **[00:26:43 – 00:26:47]** using 128 port SN6600 switches. **[00:26:47 – 00:26:49]** So this is the engineering drawing. **[00:26:49 – 00:26:52]** I like to showcase those. **[00:26:52 – 00:26:55]** One thing that is interesting here is that it uses eight **[00:26:55 – 00:27:00]** disconnected planes for maximum fabric scalability. **[00:27:00 – 00:27:03]** So if you're coming from a world that is like InfiniBand, where **[00:27:03 – 00:27:07]** we deploy by rail for the first DAGX, that's an extension of it. **[00:27:07 – 00:27:11]** Well, now we deploy by plane, and every plane is independent, **[00:27:11 – 00:27:14]** allowing us to scale more. **[00:27:14 – 00:27:18]** The shuffle box, they are used to split each CX9 for the east-west **[00:27:18 – 00:27:24]** at the back of the machine into two 4200 gigs per second ports, right? **[00:27:24 – 00:27:28]** And here we see a 14 rack leaf group to maximize the **[00:27:28 – 00:27:31]** port utilization at the leaf switch level. **[00:27:31 – 00:27:36]** So you see again the same building block that we're duplicating. **[00:27:36 – 00:27:40]** In practice, that matches the physical row at the bottom **[00:27:40 – 00:27:45]** that I've shown you earlier, with 14 Vera Rubin racks, **[00:27:45 – 00:27:50]** corresponds to one of the leaf groups, right, and 32 leaf switches **[00:27:51 – 00:27:54]** per leaf group for Rails 8 plane. **[00:27:54 – 00:27:57]** So in addition to the airplane, now we have the four rails on top **[00:27:57 – 00:28:02]** of it, and each compute tray having is four GPUs, as it's eight CX9, **[00:28:03 – 00:28:08]** four SFP cages at 1.6 Tera each, and eight MPO12 at 800 gig each. **[00:28:10 – 00:28:13]** So at the end of the day, we have each GPUs that connects **[00:28:13 – 00:28:18]** to eight times 200 gig planes and four per core NX-69. **[00:28:18 – 00:28:20]** So usually that's where I lose everyone with my like **[00:28:20 – 00:28:25]** very complex like five-grade mask, but that's where all aligns to **[00:28:25 – 00:28:28]** the block that we're duplicating. **[00:28:28 – 00:28:29]** How do you design that? **[00:28:29 – 00:28:30]** Sounds like it's a headache. **[00:28:30 – 00:28:32]** I want to do it for 32K. **[00:28:32 – 00:28:34]** I want to do it for 64K. **[00:28:34 – 00:28:35]** We get help. **[00:28:35 – 00:28:40]** There is the tool called DSX Air that we use to design, deploy, and **[00:28:40 – 00:28:42]** operate those large-scale fabric. **[00:28:42 – 00:28:46]** So it's currently limited, I think, to 128K GPUs. **[00:28:46 – 00:28:47]** But if you have more than that, we're super happy **[00:28:47 – 00:28:49]** to help you there. **[00:28:49 – 00:28:51]** And it helps design a two-level, eight-plane, **[00:28:51 – 00:28:56]** east-west fabric for any kind of size you want until then. **[00:28:56 – 00:28:58]** And then you can simulate, you can view it. **[00:28:58 – 00:29:01]** We usually like to see how the links are done. **[00:29:01 – 00:29:04]** And it helps also accelerate the deployment, because usually **[00:29:04 – 00:29:07]** when you get that thing and everything got connected, **[00:29:07 – 00:29:11]** there's always one cable that is not fully in the thing, right? **[00:29:11 – 00:29:13]** And finding it can be very difficult, even if you've **[00:29:13 – 00:29:15]** got the bundle, etc. **[00:29:15 – 00:29:19]** So that's a very useful tool to do that. **[00:29:19 – 00:29:22]** I haven't talked about north-south and storage. **[00:29:22 – 00:29:24]** I'm going to try to do a little bit here. **[00:29:24 – 00:29:26]** So we got a north-south fabric. **[00:29:26 – 00:29:29]** In the deployment that I've shown you, that north-south fabric **[00:29:29 – 00:29:33]** is based on a previous generation of Spectrum that we use with 5610 **[00:29:33 – 00:29:35]** switches that you might have today. **[00:29:35 – 00:29:41]** And for all needs, we've been using a 2-1 blocking factor for storage **[00:29:41 – 00:29:45]** that leaves 12 GB per second bandwidth available for GPUs. **[00:29:46 – 00:29:50]** For our application, we figured it was good enough, but we're **[00:29:50 – 00:29:55]** always interested in knowing about a use case we haven't looked at. **[00:29:55 – 00:30:01]** But basically, we're using that for classical storage tiers and Usage. **[00:30:01 – 00:30:04]** The favorite one being the local scratch, which are the **[00:30:04 – 00:30:07]** fast temporary files that are clear after each job. **[00:30:07 – 00:30:12]** It could be the checkpoints, it could be the temporary **[00:30:12 – 00:30:15]** weights, this kind of thing. **[00:30:15 – 00:30:18]** And then we got local copies of the data set, Muggle, Chefs, Points **[00:30:18 – 00:30:20]** for fast performance training. **[00:30:20 – 00:30:23]** And we have a shared NFS file system. **[00:30:23 – 00:30:25]** Typically, where is my own dir? **[00:30:25 – 00:30:29]** Where can I keep my configuration files, my administrative file, **[00:30:29 – 00:30:34]** all the source code bits that I don't want to transfer every time? **[00:30:34 – 00:30:37]** You can have a high-peat stair storage, we have that too, **[00:30:37 – 00:30:41]** depending on the use case, and also have, optionally, **[00:30:41 – 00:30:43]** an object storage. **[00:30:43 – 00:30:46]** So, it's just to showcase that this north-south fabric can actually **[00:30:46 – 00:30:49]** have any kind of tier you want. **[00:30:49 – 00:30:52]** I don't want to say in any order you want, because you want to make **[00:30:52 – 00:30:54]** sure your local scratch and your fast tier are very close to your **[00:30:54 – 00:31:00]** compute resources, but that way the reference is very, very flexible. **[00:31:00 – 00:31:04]** Let's spend a little bit of time on software. **[00:31:04 – 00:31:08]** So those, most of you coming from enterprise, might be **[00:31:08 – 00:31:09]** very familiar with. **[00:31:09 – 00:31:12]** Those are the classical stacks that have been used on the **[00:31:12 – 00:31:15]** DGX product for a while. **[00:31:15 – 00:31:20]** AI workload management sits on top of a recovery engine. **[00:31:20 – 00:31:23]** We mentioned telemetry and observability, validation **[00:31:23 – 00:31:27]** and diagnostics, all the way down to the Cluster Management **[00:31:28 – 00:31:32]** itself, whether it's Firmware Automation, Inventory Management, **[00:31:32 – 00:31:35]** Provisioning, etc. **[00:31:35 – 00:31:37]** I'm not going to spend so much time on it, but it's **[00:31:37 – 00:31:41]** basically, there are a lot of options here, a lot of us **[00:31:41 – 00:31:45]** have our habits, we like to deploy with this tool, we like to make **[00:31:45 – 00:31:50]** sure our inventory is presented that way, they kind of interrupt, **[00:31:50 – 00:31:53]** fungible, like people like to say, you can interrupt most of it, **[00:31:54 – 00:31:57]** the building blocks are really made to be able to interact with what **[00:31:57 – 00:32:02]** you already have or what you might like for the ecosystem already. **[00:32:02 – 00:32:04]** Same thing in managing the control plane. **[00:32:04 – 00:32:06]** So a lot of people are like, how do you do the end? **[00:32:06 – 00:32:08]** And how many networks do you have? **[00:32:08 – 00:32:09]** That's a lot of different things. **[00:32:09 – 00:32:12]** That's a very complicated slide, but at the end of the **[00:32:12 – 00:32:15]** day, I'm trying to convince you that that is what we already **[00:32:15 – 00:32:19]** do today, having those different management blocks, right? **[00:32:19 – 00:32:23]** And it's like the same standard, the same scalable and secure **[00:32:23 – 00:32:26]** control plane, and that the management of it can **[00:32:26 – 00:32:30]** still be centralized. **[00:32:31 – 00:32:35]** This is also how do you bring your workload there. **[00:32:35 – 00:32:37]** Some people are in the room here from the gentlemen from **[00:32:37 – 00:32:40]** the CUDA X libraries. **[00:32:40 – 00:32:45]** Today, none of us wants to compile the latest framework **[00:32:45 – 00:32:46]** and make sure it gets to perf. **[00:32:46 – 00:32:48]** That gets very, very complex. **[00:32:49 – 00:32:53]** And so I mentioned earlier the idea of experimenting on my laptop, **[00:32:53 – 00:32:56]** which now is a GNOME laptop, having my previous datacenter that was **[00:32:56 – 00:33:00]** not a GNOME datacenter, and being able to bring that on the cluster. **[00:33:00 – 00:33:03]** I don't want to have to worry and recompile and make sure **[00:33:03 – 00:33:06]** my experiments are set for each. **[00:33:06 – 00:33:09]** And so basically I just download the container, it's tuned **[00:33:09 – 00:33:11]** for me, it's tuned for me every month, and I kind of know where I'm **[00:33:11 – 00:33:17]** at, and I can repro the green slide I've seen very efficiently, simply **[00:33:17 – 00:33:20]** not having to worry about it. **[00:33:21 – 00:33:25]** Also, at the end of the day, it supports the CUDA **[00:33:25 – 00:33:27]** base layer, right? **[00:33:27 – 00:33:32]** So there are some talks this week about CUDA X libraries and so on. **[00:33:32 – 00:33:36]** All of those also get optimised for both whatever we might **[00:33:36 – 00:33:42]** have as a previous infrastructure and what we are deploying now. **[00:33:42 – 00:33:46]** So, an example is the NICL, the communication library that is **[00:33:46 – 00:33:51]** used in most of frameworks, right, and it sits on top of NICL core **[00:33:51 – 00:33:53]** and it has a bunch of plug-ins. **[00:33:53 – 00:33:57]** I'm going to go a little fast here, but this is an example **[00:33:57 – 00:34:00]** of a feature that is leveraged on ng-link fabric that you've **[00:34:00 – 00:34:05]** seen earlier, right, to accelerate the performance by reducing **[00:34:05 – 00:34:08]** the synchronization overhead and get better bandwidth out **[00:34:08 – 00:34:13]** of those very big domains, and that helps all the the performance **[00:34:13 – 00:34:17]** of this go faster on that fabric. **[00:34:17 – 00:34:19]** An example also on how do I test? **[00:34:19 – 00:34:22]** I got my rack, how do I make sure that that you know new fabric **[00:34:22 – 00:34:25]** that I haven't worked with works? **[00:34:25 – 00:34:29]** There is a bunch that needs to be taken care of here, **[00:34:29 – 00:34:30]** instead of memory management. **[00:34:31 – 00:34:33]** The good thing is, the memory is the same. **[00:34:33 – 00:34:36]** Some of you might be remember, I mean, I was personally traumatized **[00:34:37 – 00:34:39]** the first time I did CUDA that I had to handle this **[00:34:39 – 00:34:42]** memory here, this memory here, this memory there. **[00:34:42 – 00:34:45]** What's super nice on the rack is that all memory is addressable **[00:34:45 – 00:34:48]** and I don't have to worry where that is and I don't have to **[00:34:48 – 00:34:50]** worry about the performance of it. **[00:34:50 – 00:34:53]** It's almost weird for teaching because then you explain that **[00:34:53 – 00:34:54]** you have to manage memory. **[00:34:54 – 00:34:56]** People are like, why should I care? **[00:34:56 – 00:34:59]** Luckily today we don't have to care about it anymore. **[00:34:59 – 00:35:02]** We just have that big chunk where we can load the model and **[00:35:02 – 00:35:03]** the model can live here forever. **[00:35:03 – 00:35:05]** That's very convenient. **[00:35:06 – 00:35:08]** I'm just going to go quick here. **[00:35:08 – 00:35:11]** Those are the operational tools to manage that fabric, typically **[00:35:11 – 00:35:14]** telemetry and Fabric Manager. **[00:35:14 – 00:35:16]** Again, the compute software here. **[00:35:16 – 00:35:19]** I'm being told that I need to move faster. **[00:35:20 – 00:35:23]** So this is how we test that NVLink fabric. **[00:35:23 – 00:35:26]** Some of you might be used to the bandwidth test or running **[00:35:26 – 00:35:27]** OSUs or all those tools. **[00:35:27 – 00:35:31]** This is the equivalent for an NVL fabric. **[00:35:31 – 00:35:36]** This is a real example on a... GB infrastructure, I **[00:35:36 – 00:35:41]** think it's a 576, so it's one of the, that's the graph that usually **[00:35:41 – 00:35:44]** in engineering may just cheer because that means the cluster **[00:35:44 – 00:35:49]** is up, and that's something that is available for anyone to use. **[00:35:49 – 00:35:53]** And two concepts that are important for NVLink. **[00:35:53 – 00:35:54]** What is a domain? **[00:35:54 – 00:35:58]** Here, the domain is a rack, but I've shown you a 576 domain 2. **[00:35:58 – 00:35:59]** And what is a partition? **[00:35:59 – 00:36:01]** A partition is a subset of a domain. **[00:36:01 – 00:36:04]** And if you want to try on hard-dressing all of those **[00:36:04 – 00:36:08]** and have, oh, that, my job doesn't need the whole rack, **[00:36:08 – 00:36:10]** that's not a problem as long as it gets aligned right. **[00:36:10 – 00:36:13]** And usually what we do is we provide the first domain **[00:36:13 – 00:36:17]** scheduling strategy that then the ecosystems adopt to make sure that **[00:36:17 – 00:36:21]** we address the racks correctly and we don't have scheduling problem or **[00:36:21 – 00:36:24]** we don't have to do that manually. **[00:36:24 – 00:36:26]** The job scheduling example here that I'm presenting is **[00:36:26 – 00:36:32]** done with Slurm through a Topology block plugin that has been in **[00:36:32 – 00:36:37]** the open-source stack since 2024. **[00:36:37 – 00:36:43]** In conclusion, those are new, but they are not different. **[00:36:43 – 00:36:46]** The idea is we're going to keep on simplifying through **[00:36:46 – 00:36:47]** practical guidance. **[00:36:47 – 00:36:50]** The reference is not a mandate, it's something to make it **[00:36:50 – 00:36:52]** simpler for everyone. **[00:36:52 – 00:36:57]** We do it so that we make sure it's doable, right? **[00:36:57 – 00:37:00]** Charlie said that we want to make sure we try it. **[00:37:00 – 00:37:04]** We spend a significant amount of time driving it ourselves **[00:37:04 – 00:37:07]** and making sure that we get that experience, experience **[00:37:07 – 00:37:11]** that we have through DGX1, we need to use a product that **[00:37:11 – 00:37:15]** we enjoy ourselves, and that's what we're trying to give to you. **[00:37:15 – 00:37:19]** We aim to still advance that in the community. **[00:37:19 – 00:37:22]** You'll see a bunch of examples that are open source examples that you **[00:37:22 – 00:37:25]** can download today, and we would like to work with you to understand **[00:37:25 – 00:37:29]** what's missing, what you might want to have, what's impractical, and **[00:37:29 – 00:37:31]** how we turn it into better results. **[00:37:31 – 00:37:35]** And so, you have a bunch of people here, if you have questions **[00:37:35 – 00:37:38]** about your next deployment, whether it's power cooling, **[00:37:38 – 00:37:42]** you know, or data center readiness, super happy to talk to you **[00:37:42 – 00:37:44]** about it, even here on the floor. **[00:37:44 – 00:37:47]** Thank you so much for your attention. **[00:37:52 – 00:37:57]** We can take two questions, so please raise your hands. **[00:38:05 – 00:38:06]** Thank you very much. **[00:38:06 – 00:38:09]** Great talk, lots of very useful information. **[00:38:09 – 00:38:12]** I have a question on the optical networking side. **[00:38:12 – 00:38:17]** So what's your opinion on the discussion about where **[00:38:17 – 00:38:19]** the laser sources are going to be? **[00:38:19 – 00:38:24]** So there's discussion on ELSFP with remote laser sources versus **[00:38:24 – 00:38:29]** integrating the laser in terms of reliability and power consumption. **[00:38:29 – 00:38:31]** I will give you a real engineering answer. **[00:38:31 – 00:38:32]** I am not sure. **[00:38:32 – 00:38:36]** So I might be the only one answering I don't **[00:38:36 – 00:38:38]** know to your question. **[00:38:38 – 00:38:40]** I don't know yet. **[00:38:40 – 00:38:43]** It's true that we needed more compact. **[00:38:43 – 00:38:47]** As you've seen, deploying so much fabric, so much **[00:38:47 – 00:38:49]** optical, is difficult. **[00:38:49 – 00:38:52]** We know we can't do it with copper anymore, right? **[00:38:52 – 00:38:56]** So some of the very compact models, I mean, there was a **[00:38:56 – 00:38:59]** rendering of a CPO switch in there. **[00:38:59 – 00:39:01]** That is very convenient. **[00:39:01 – 00:39:02]** That is reliable. **[00:39:02 – 00:39:06]** And in our hands, it's been working extremely well. **[00:39:06 – 00:39:10]** We don't have a lot of history about it, so that's where **[00:39:10 – 00:39:12]** I'm always, you know, a little bit cold feet. **[00:39:13 – 00:39:16]** But I do like that it's very convenient. **[00:39:16 – 00:39:20]** Like every engineer, when you see all those things combine into such **[00:39:20 – 00:39:23]** a small amount of space, you worry that one failure is going to have a **[00:39:23 – 00:39:28]** very large blast radius, and that's not at all what we're seeing. **[00:39:28 – 00:39:31]** So far so good, I'm going to say, I'm like, I understand **[00:39:31 – 00:39:35]** I barely answered the question, but we've been having super good **[00:39:35 – 00:39:39]** results with, like, for example, package optics and so on have been **[00:39:39 – 00:39:42]** working very, very well for us. **[00:39:44 – 00:39:46]** Yes, question, please, here. **[00:39:46 – 00:39:48]** Sorry. **[00:39:48 – 00:39:57]** Considering the fact that the VR NVL72 is 100% water-cooled, why do **[00:39:57 – 00:40:03]** you ask for respecting hot hails, hot corridor, and cold corridor? **[00:40:03 – 00:40:06]** Very, very, very good question. **[00:40:06 – 00:40:10]** The systems that I've shown you have 100 **[00:40:10 – 00:40:12]** liquid-cooled compute racks. **[00:40:12 – 00:40:18]** But we do use, next to it, a lot of non, or not yet, **[00:40:18 – 00:40:21]** fully liquid cooled racks. **[00:40:21 – 00:40:25]** And so, having a hot air containment with those racks **[00:40:25 – 00:40:30]** in the middle for convenience of deployment is actually very good **[00:40:30 – 00:40:32]** because it helps balance the rest. **[00:40:32 – 00:40:37]** So the air-cooled section that you have in those hacks **[00:40:37 – 00:40:40]** is not very much, but that's enough to balance it out. **[00:40:40 – 00:40:42]** We haven't discussed that, but there's another hidden **[00:40:42 – 00:40:44]** question behind the scenes. **[00:40:44 – 00:40:49]** I only mentioned the secondary, I haven't talked about the primary. **[00:40:49 – 00:40:53]** Sometimes, the secondary you use for air and liquid cool **[00:40:53 – 00:40:56]** has to be shared, for example, when you are in a previously **[00:40:56 – 00:40:59]** designed infrastructure, and sometimes you have the luxury to **[00:40:59 – 00:41:01]** do it from scratch and separate it. **[00:41:01 – 00:41:05]** If you share it, it's actually much easier to balance in a data **[00:41:05 – 00:41:09]** center that is already deployed, and it can be faster to deploy. **[00:41:09 – 00:41:12]** So, I'm going to say not every case is different. **[00:41:12 – 00:41:15]** But depending on what you take from the reference and what you need **[00:41:15 – 00:41:19]** to deploy for your application, you might want to use one or the other. **[00:41:19 – 00:41:26]** OK, so I will need an RDHX only for the non-DLC equipment. **[00:41:27 – 00:41:33]** But for the VR and VL72, no RDA checks. **[00:41:33 – 00:41:35]** At all. **[00:41:35 – 00:41:39]** For the VR and VR72, it's 100% liquid cool. **[00:41:39 – 00:41:44]** We do like to have a little flow in the room anyway, just to make sure **[00:41:45 – 00:41:46]** you don't have localized effect. **[00:41:47 – 00:41:50]** So you still need to have a fan wall that actually makes sure your **[00:41:50 – 00:41:53]** room is homogeneous to some extent. **[00:41:54 – 00:41:55]** Okay, thank you so much for being here.