# Redefining Scalable AI Performance: OCI Supercomputing in the Cloud (Presented by Oracle) ## Abstract Organizations building or scaling frontier AI models can’t be constrained by infrastructure. In this session, discover how Oracle Cloud Infrastructure (OCI) connects hundreds of thousands of NVIDIA GPUs. Oracle’s AI superclusters deliver up to 16 zettaFLOPS of peak performance, giving you the scale, low-latency GPU-to-GPU communication, and price-performance required for the largest AI training and inference workloads. In this session you will learn how OCI’s Acceleron ultra-low-latency RoCE networking and NVIDIA’s next-generation GPU instances dramatically increase LLM throughput, accelerate multimodal and scientific workloads, and improve cluster utilization, so you can train faster, deploy sooner, and operate more efficiently. Walk away with a clear understanding of how to move from experimentation to industrialized AI at global scale, with the performance, reliability, and sovereignty controls enterprise AI demands. ## AI Summary - The presentation highlighted the development of high-performance RDMA in OCI Cloud, driven by customer demand, particularly for HPC and ML workloads. - The team chose RDMA over Converged Ethernet (ROCE) as the primary mechanism for delivering RDMA, focusing on multi-tenancy, scalability, and cost efficiency. - OCI introduced Acceleron, an umbrella term for innovative network features, including RDMA on the front-end network, adaptive traffic load balancing, and link-aware optics. - The multi-planar fabric design was discussed, offering higher reliability, lower latency, and reduced power consumption compared to traditional three-tier CLO designs. - The presentation covered the importance of customized NIC functionality, such as NVIDIA's PRDMA SDK, and the use of packet spraying and out-of-order packet handling to enhance performance and reliability. - OCI also emphasized the need for ongoing innovation and interoperability between different RDMA NICs and switch vendors to further improve network performance and reliability. ## Transcript **[00:00:06 – 00:00:09]** And I'm going to let Taylor kick us off. **[00:00:09 – 00:00:12]** And I'll do a quick intro of Taylor while he comes up. **[00:00:13 – 00:00:19]** Taylor is the reason that we have high performance RDMA **[00:00:19 – 00:00:21]** in the OCI cloud. **[00:00:21 – 00:00:24]** I remember when I first joined OCI, David Becker, who's in **[00:00:24 – 00:00:29]** the audience here, he was the first network engineer to join OCI. **[00:00:29 – 00:00:33]** And David and I were, you know, working on our network architecture **[00:00:33 – 00:00:34]** with the rest of the team. **[00:00:34 – 00:00:37]** And Taylor walked over, he goes, hey, guys, have you thought **[00:00:37 – 00:00:38]** about doing RDMA in the cloud? **[00:00:38 – 00:00:41]** And we're like, yeah, you know, we'd like to, but I don't think we **[00:00:41 – 00:00:43]** can get the support to launch it. **[00:00:44 – 00:00:47]** And Taylor goes, I have just the answer for you. **[00:00:47 – 00:00:49]** I have a customer who needs it. **[00:00:49 – 00:00:52]** And it was a high-performance cloud, high-performance compute **[00:00:53 – 00:00:54]** set of customers, actually. **[00:00:54 – 00:00:56]** And so Taylor is the reason **[00:00:56 – 00:00:59]** That we got to do what we do, and eventually, that is **[00:00:59 – 00:01:05]** the network that now runs all the ML workloads at OCI. **[00:01:05 – 00:01:08]** So, Taylor deserves all the credit for it. **[00:01:08 – 00:01:10]** Go ahead, Taylor, please. **[00:01:10 – 00:01:11]** Ideas are cheap, for sure. **[00:01:11 – 00:01:15]** It's the ability to execute, and that's what JAG and David brought. **[00:01:15 – 00:01:22]** So, yes, we had an idea, but JAG is the reason that we're here today. **[00:01:23 – 00:01:24]** What's that? **[00:01:24 – 00:01:28]** Oh, I'll go over here. I'm happy to stand next to Jag. **[00:01:28 – 00:01:31]** So anyway, thank you for joining us today. **[00:01:31 – 00:01:32]** Fantastic introduction. **[00:01:32 – 00:01:34]** We had a couple of hiccups that we were able to **[00:01:34 – 00:01:36]** overcome pretty quickly. **[00:01:36 – 00:01:41]** So hopefully you listened to Pradeep Vincent, to his keynote **[00:01:41 – 00:01:42]** that just finished up a little bit. **[00:01:42 – 00:01:46]** This is really Jag's opportunity to double click on some of the **[00:01:46 – 00:01:50]** things that Pradeep talked about. **[00:01:50 – 00:01:54]** What I liked about how JAG started is so much of what **[00:01:54 – 00:02:01]** we do at Oracle, at Oracle Cloud, is driven by the customer demand. **[00:02:01 – 00:02:04]** We had a unique differentiator when we started. **[00:02:04 – 00:02:06]** We focused on performance from day one. **[00:02:06 – 00:02:12]** And it's because our customers in the HPC space, that's what **[00:02:12 – 00:02:14]** they wanted out of a cloud. **[00:02:14 – 00:02:16]** That's what they came to Oracle for, and that's what we had. **[00:02:16 – 00:02:19]** And so JAG made it possible. **[00:02:19 – 00:02:25]** For our customers to run HPC and to run RDMA across our **[00:02:25 – 00:02:29]** network and then watching what has come from that, this **[00:02:29 – 00:02:34]** concept of hundreds of thousands of GPUs communicating over **[00:02:35 – 00:02:40]** this single network is really what we're here to talk about today. **[00:02:40 – 00:02:44]** JAG, I'll turn the time over to you and let you drive the show. **[00:02:44 – 00:02:47]** I think they're advancing the slides manually so they **[00:02:47 – 00:02:48]** can see when you push the button. **[00:02:49 – 00:02:51]** So if you push the button, they'll move it for you. **[00:02:51 – 00:02:52]** So which button do I push? **[00:02:52 – 00:02:54]** The green. Okay. Thank you. **[00:02:54 – 00:02:58]** And thank you to the team at the back helping us with the slides. **[00:02:58 – 00:03:01]** For some reason, we couldn't get it working on my laptop. **[00:03:01 – 00:03:02]** So thank you, Taylor. **[00:03:02 – 00:03:03]** Thank you for attending. **[00:03:03 – 00:03:08]** So this talk, we kind of changed the title at the last minute **[00:03:08 – 00:03:10]** to make it more targeted. **[00:03:11 – 00:03:13]** So really, what we're here to talk about is **[00:03:13 – 00:03:20]** The network infrastructure that enables RDMA in OCI Cloud, **[00:03:20 – 00:03:23]** and that is the same network infrastructure that supports **[00:03:24 – 00:03:26]** ML, GPU-based ML workloads. **[00:03:26 – 00:03:29]** And here's a picture of, thank you, David. **[00:03:29 – 00:03:31]** David actually took this picture, I think. **[00:03:31 – 00:03:35]** This is a picture of one of our largest clusters. **[00:03:35 – 00:03:38]** Maybe I shouldn't name exactly where it is, but you can probably **[00:03:38 – 00:03:41]** put the two and two together. **[00:03:42 – 00:03:48]** So this is the Abilene location and maybe the picture doesn't **[00:03:48 – 00:03:51]** quite do justice, these are very large buildings, this is **[00:03:51 – 00:03:54]** a very large construction project. **[00:03:54 – 00:03:56]** So one of the things we won't get to talk about here today **[00:03:56 – 00:04:00]** is one of the complexities One of the advantages of building and **[00:04:00 – 00:04:03]** operating a cloud is construction. **[00:04:03 – 00:04:06]** So we are also a construction company, we build large data **[00:04:06 – 00:04:12]** centers that includes getting permits, site, running fiber, **[00:04:12 – 00:04:15]** physical infrastructure, building, building fast, building cleanly, **[00:04:15 – 00:04:18]** following the code, getting all the inspections done. **[00:04:18 – 00:04:24]** Power infrastructure, multiple power, backup systems, UPSs, **[00:04:24 – 00:04:28]** power generation facilities on-site for backup power, **[00:04:28 – 00:04:32]** power distribution systems, physical layer 1 infrastructure, **[00:04:32 – 00:04:34]** cable infrastructure, a lot of thought goes into it. **[00:04:34 – 00:04:39]** In fact, that's something that we all, including David in particular, **[00:04:39 – 00:04:42]** has been putting a lot of focus on because that turns out is very **[00:04:43 – 00:04:45]** important for ML infrastructure. **[00:04:45 – 00:04:48]** We'll briefly touch on it today. **[00:04:48 – 00:04:49]** So let me... **[00:04:49 – 00:04:52]** Move the slide over. A little bit on the background of OCI Cloud. **[00:04:52 – 00:04:57]** For those of you who may not be familiar, OCI is a tier one cloud. **[00:04:57 – 00:05:02]** We're in the Gartner magic quadrant of cloud providers. **[00:05:02 – 00:05:06]** It's fully featured with hundreds of services from object store **[00:05:06 – 00:05:11]** to block store, to monitoring services, to compute, different **[00:05:11 – 00:05:15]** compute SKUs, storage services, file system services. **[00:05:15 – 00:05:17]** A fully featured cloud that's global. **[00:05:17 – 00:05:19]** We have over 200. **[00:05:19 – 00:05:26]** Regions globally and all kinds of regions, as in sovereign **[00:05:26 – 00:05:30]** regions, government regions, commercial regions, private cloud **[00:05:30 – 00:05:33]** regions, dedicated for customers. **[00:05:33 – 00:05:36]** And we have our own backbone interconnecting all these regions. **[00:05:36 – 00:05:41]** We also support all these different types of customers, as in **[00:05:41 – 00:05:46]** commercials, sovereign government, and of late, multi-cloud, **[00:05:46 – 00:05:52]** where you can get Oracle database service in a competitor's **[00:05:52 – 00:05:53]** cloud, for example, in Azure. **[00:05:53 – 00:06:00]** So you can consume Oracle database inside of Azure using Azure's **[00:06:01 – 00:06:04]** All the services that you may have come to use at Azure, **[00:06:05 – 00:06:07]** but in the end, the services, the database is still running **[00:06:07 – 00:06:11]** inside of OCI, so that's a very unique service, and that's **[00:06:11 – 00:06:15]** the case with AWS and GCP as well. **[00:06:15 – 00:06:20]** So making a switch into going in to talk about the ML infrastructure, **[00:06:20 – 00:06:26]** so firstly, we, as you know, RDMA is the fundamental building block **[00:06:26 – 00:06:30]** for machine learning workloads and also for high-performance **[00:06:30 – 00:06:31]** compute workloads. **[00:06:31 – 00:06:34]** And there are two primary ways of doing RDMA. **[00:06:34 – 00:06:39]** The first, most familiar way of doing RDMA is using InfiniBand. **[00:06:39 – 00:06:42]** And we certainly also have InfiniBand deployments. **[00:06:42 – 00:06:44]** And for customers who **[00:06:44 – 00:06:48]** Specifically asked for InfiniBand, we have InfiniBand as well, **[00:06:48 – 00:06:53]** but our more common way of doing RDMA is using ROCKI, which stands **[00:06:53 – 00:06:57]** for RDMA on Converged Ethernet or RDMA over Converged Ethernet. **[00:06:57 – 00:07:01]** And we started the rocky journey, RDMA over Converged **[00:07:02 – 00:07:06]** Ethernet journey, back in 2017, 2018, so pretty early. **[00:07:06 – 00:07:11]** And as we just talked about it, Taylor had customers who **[00:07:11 – 00:07:13]** wanted to run high-performance compute workloads in OCI, **[00:07:13 – 00:07:17]** and HPC MPI workloads need RDMA, so we had a choice, **[00:07:17 – 00:07:18]** like how do we deliver that? **[00:07:19 – 00:07:22]** And we wanted a mechanism that we could deploy widely. **[00:07:22 – 00:07:26]** across all regions and you know we wanted to be able to have supply **[00:07:26 – 00:07:32]** chains where we can get components from different manufacturers and so **[00:07:32 – 00:07:37]** we went and we chose Rocky as our primary or our first mechanism of **[00:07:37 – 00:07:41]** delivering RDMA and as I said we've since also supported InfiniBand **[00:07:41 – 00:07:43]** and the reason we went down **[00:07:43 – 00:07:49]** The rocky road is we wanted multi-tenancy, and Rocky lends **[00:07:49 – 00:07:52]** to multi-tenancy well, and in fact we've done a lot of **[00:07:53 – 00:07:55]** innovative work on how to do **[00:07:55 – 00:08:00]** Multi-tenancy for RDMA without taking performance compromises. **[00:08:00 – 00:08:06]** And we have a number of video blogs there on the OCI website, publicly **[00:08:06 – 00:08:09]** available, where we describe what does that mean, what did it take **[00:08:09 – 00:08:14]** for us to do RDMA in a multi-tenant environment without compromising **[00:08:14 – 00:08:17]** security or performance. **[00:08:18 – 00:08:21]** And we also wanted multi-source, switches, NICs, and optics **[00:08:21 – 00:08:22]** from multiple suppliers. **[00:08:22 – 00:08:25]** So we have assurance around supply chain. **[00:08:25 – 00:08:28]** We wanted arbitrarily large scale. **[00:08:28 – 00:08:32]** We didn't want to have limits around addresses, number of **[00:08:32 – 00:08:33]** customers, number of tenants. **[00:08:33 – 00:08:35]** And we wanted it to be cost efficient. **[00:08:35 – 00:08:39]** So for those reasons, we went down the rocky road. **[00:08:39 – 00:08:42]** As I said, we also have InfiniBand now. **[00:08:42 – 00:08:46]** Another distinguisher of OCI versus our other competitors is we **[00:08:46 – 00:08:50]** started with a bare metal offering, which is a heavier lift, harder **[00:08:50 – 00:08:55]** thing to do, because you don't have a hypervisor on the host. **[00:08:55 – 00:08:58]** So it makes doing virtual networking, virtualization, hard. **[00:08:58 – 00:09:01]** So we chose to go that road first, because if you do a **[00:09:01 – 00:09:05]** bare metal well, doing virtual machines gets much easier. **[00:09:05 – 00:09:09]** So that's a key property that OCI Cloud has, that it's a **[00:09:09 – 00:09:11]** bare metal cloud. **[00:09:11 – 00:09:17]** Which meant we had to do a lot of early difficult work. **[00:09:17 – 00:09:20]** And that's the case with the GPU instances as well. **[00:09:20 – 00:09:22]** They're all bare metal. **[00:09:22 – 00:09:26]** And then, as I said, shortly after we launched OCI, we **[00:09:26 – 00:09:29]** also launched RDMA using Rocky. **[00:09:29 – 00:09:32]** Here is what the product presentation for that looks like. **[00:09:32 – 00:09:35]** So if you imagine you have a private **[00:09:35 – 00:09:38]** Cloud network, we call it virtual cloud network, which **[00:09:38 – 00:09:41]** is the network over here, you get your own virtual network, you have **[00:09:41 – 00:09:46]** a bunch of CPUs, you have a virtual NIC that connects your CPUs, your **[00:09:46 – 00:09:50]** bare metal machines, or VMs to your virtual cloud network, you connect **[00:09:51 – 00:09:53]** that to the public internet, or you connect it to your on-premise **[00:09:53 – 00:09:59]** network, and then, let's say you want RDMA, then what you do is you **[00:09:59 – 00:10:03]** connect that network to a dedicated backend RDMA network, and that, **[00:10:03 – 00:10:06]** as I said, we started with Rocky. **[00:10:06 – 00:10:08]** So we have a dedicated back-end network. **[00:10:08 – 00:10:10]** And the reason we did a dedicated back-end network is we didn't want **[00:10:10 – 00:10:13]** to take a performance compromise. **[00:10:13 – 00:10:16]** We tried not to nickel and dime our customers. **[00:10:16 – 00:10:18]** So we tried to offer the highest possible network **[00:10:18 – 00:10:20]** performance at the lowest possible cost to the customer. **[00:10:20 – 00:10:23]** We actually deliberately do try to do that. **[00:10:23 – 00:10:25]** So for that reason, because we didn't want performance **[00:10:25 – 00:10:28]** compromises, we went with a dedicated back-end network. **[00:10:28 – 00:10:31]** And that also is a unique thing about OCI, as opposed to **[00:10:31 – 00:10:34]** trying to club everything together on the same front-end network. **[00:10:34 – 00:10:37]** We actually built a dedicated back-end network from **[00:10:37 – 00:10:38]** literally day one. **[00:10:38 – 00:10:41]** So we launched that in 2018. **[00:10:42 – 00:10:47]** Now, over the last year or so, OCI has started branding **[00:10:47 – 00:10:52]** this term called Acceleron, and some customers and third parties **[00:10:52 – 00:10:55]** have questions about what is OCI, Oracle Acceleron, what is it? **[00:10:55 – 00:10:59]** Acceleron is an uber term we use to describe **[00:10:59 – 00:11:04]** All of the innovative work, unique work we're doing on behalf **[00:11:04 – 00:11:06]** of our customers in the network. **[00:11:06 – 00:11:08]** So we're doing a bunch of things. **[00:11:08 – 00:11:11]** If I were to start naming innovative things that OCI **[00:11:11 – 00:11:14]** has done, it's probably a couple of dozen completely innovative, unique **[00:11:14 – 00:11:17]** things we've done for performance, **[00:11:17 – 00:11:22]** For scalability, for reliability, and for isolation of customers, **[00:11:22 – 00:11:26]** performance isolation, security isolation, namespace isolation. **[00:11:26 – 00:11:29]** There's a lot of unique, innovative things we've been doing over **[00:11:29 – 00:11:32]** the last 10 years, and there's a whole host of them. **[00:11:32 – 00:11:35]** So we try to bring them all together under an umbrella, **[00:11:35 – 00:11:36]** which is Acceleron. **[00:11:36 – 00:11:39]** Acceleron is how we're going to innovate on behalf of **[00:11:39 – 00:11:41]** you, our customers. **[00:11:41 – 00:11:44]** And examples of that include **[00:11:45 – 00:11:46]** New host-facing functionality. **[00:11:46 – 00:11:49]** For example, we announced RDMA on our front-end network as well. **[00:11:49 – 00:11:52]** So the previous drawing we looked at, not only do we have RDMA on the **[00:11:52 – 00:11:56]** back-end, we now also have started offering RDMA on the front-end. **[00:11:56 – 00:11:58]** So that's part of Acceleron. **[00:11:58 – 00:12:04]** Similarly, new functionality in the physical network as **[00:12:04 – 00:12:07]** in new kinds of optics that are more power efficient, **[00:12:07 – 00:12:09]** and we'll talk about that shortly. **[00:12:09 – 00:12:11]** New virtual networking features. **[00:12:11 – 00:12:14]** All of that taken together is Acceleron, and that could **[00:12:14 – 00:12:18]** be for the front-end or that could be for the back-end. **[00:12:19 – 00:12:25]** Now, talking about ML workloads, AI workloads, they have some unique **[00:12:25 – 00:12:30]** requirements from the network, unlike traditional cloud workloads. **[00:12:30 – 00:12:33]** These workloads are very performance sensitive, so they see **[00:12:33 – 00:12:35]** the highest possible performance. **[00:12:35 – 00:12:38]** They seek the highest possible availability because these **[00:12:38 – 00:12:42]** workloads are very tightly time-synchronized and they **[00:12:42 – 00:12:43]** are many-to-many workloads. **[00:12:43 – 00:12:49]** So performance impact of one bad node or one bad link is **[00:12:49 – 00:12:52]** outsized because it slows down the entire workload. **[00:12:52 – 00:12:54]** These are such time-correlated and synchronized workloads **[00:12:54 – 00:12:57]** that small events have big effects. **[00:12:57 – 00:13:00]** So availability is really important, more so than the **[00:13:00 – 00:13:01]** front-end workloads. **[00:13:01 – 00:13:03]** Like on the front-end, let's say you have a TCP-based workload, **[00:13:03 – 00:13:06]** you have one slow node, the load balancer will take that **[00:13:06 – 00:13:09]** node out of rotation automatically. **[00:13:09 – 00:13:12]** That's not the case for the back-end workloads. **[00:13:12 – 00:13:13]** And then scale is really important. **[00:13:13 – 00:13:16]** Like front-end workloads may have hundreds, maybe thousands **[00:13:16 – 00:13:20]** of nodes, VMs often. **[00:13:20 – 00:13:22]** Back-end, these are bare-metal GPUs, and it could be hundreds **[00:13:22 – 00:13:25]** of thousands at the highest possible performance. **[00:13:25 – 00:13:27]** At the highest possible availability. **[00:13:27 – 00:13:31]** So the expectations are much higher. **[00:13:31 – 00:13:34]** Sorry, pressing the wrong button. **[00:13:34 – 00:13:37]** So let's look at all three of them, performance, **[00:13:37 – 00:13:39]** scalability, and reliability. **[00:13:39 – 00:13:41]** So on performance. **[00:13:42 – 00:13:44]** So how did we get the highest possible performance **[00:13:44 – 00:13:45]** for our customers? **[00:13:45 – 00:13:48]** So remember I talked about the fact that Acceleron is this **[00:13:48 – 00:13:50]** bunch of innovations we've done? **[00:13:50 – 00:13:51]** Here are some of them. **[00:13:51 – 00:13:57]** Firstly, we take traffic locality into account when we place **[00:13:57 – 00:13:57]** customers' workloads. **[00:13:57 – 00:14:00]** So we try to place your workload as close to each... **[00:14:00 – 00:14:03]** So, if you have a bare metal machine, you'll get network **[00:14:03 – 00:14:05]** locality information automatically, and you can use that to **[00:14:05 – 00:14:06]** place your workload. **[00:14:06 – 00:14:08]** You can use that to set up your nickel topology, for example, **[00:14:08 – 00:14:11]** to get the highest performance. **[00:14:22 – 00:14:25]** We publish, as I said, we publish number locality information. **[00:14:25 – 00:14:28]** We finally tune Rocky settings. **[00:14:28 – 00:14:34]** As you know, Rocky has dozens of features you can tune. **[00:14:34 – 00:14:36]** You can tune how aggressive your congestion control is. **[00:14:36 – 00:14:38]** You can tune the ECN thresholds. **[00:14:38 – 00:14:42]** You can tune how you respond to congestion notification packets. **[00:14:42 – 00:14:44]** So we do a lot of tuning. **[00:14:44 – 00:14:45]** And in fact, tuning is **[00:14:45 – 00:14:49]** We consider tuning to be one of our core competencies. **[00:14:49 – 00:14:51]** This is something we are continuously doing. **[00:14:51 – 00:14:53]** Every time we launch a new workload, every time we get a new **[00:14:53 – 00:14:56]** large customer, we have to retune. **[00:14:56 – 00:14:59]** And often these are settings on the NIC, but sometimes **[00:14:59 – 00:15:01]** in the network as well. **[00:15:01 – 00:15:03]** We have innovative traffic load balancing techniques. **[00:15:03 – 00:15:07]** I would put them in this category of adaptive load balancing. **[00:15:07 – 00:15:11]** In fact, I would say we have invented or influenced the **[00:15:11 – 00:15:13]** invention of at least two new kinds of traffic load **[00:15:14 – 00:15:18]** balancing techniques implemented by our switch vendors, and those were **[00:15:19 – 00:15:24]** targeted at addressing specific corner cases of performance **[00:15:24 – 00:15:27]** in a multi-tenant environment. **[00:15:27 – 00:15:29]** We've done a lot of work to mitigate link flaps, so we **[00:15:29 – 00:15:35]** won't have time to get into it, but the modern 400, 800 gig optics, **[00:15:35 – 00:15:40]** they use this technique called forward error correction, where you **[00:15:40 – 00:15:44]** expect lower signal-to-noise ratio, meaning you expect more bit errors, **[00:15:44 – 00:15:47]** but you encode extra information in the frame of the packet so **[00:15:47 – 00:15:49]** you can recover from those errors. **[00:15:49 – 00:15:51]** And it's a well-known technique, it gives you **[00:15:52 – 00:15:53]** Increase signal-to-noise ratio. **[00:15:53 – 00:15:56]** And the reason you needed that is because these links **[00:15:56 – 00:16:00]** use new modulation schemes like PAM-4, where the signal-to-noise **[00:16:00 – 00:16:01]** amplitude is lower. **[00:16:01 – 00:16:04]** So you're more sensitive to noise, so you expect more errors, **[00:16:04 – 00:16:07]** but you encode extra bits into the packet so you can recover from it. **[00:16:08 – 00:16:11]** Anyways, long ways to say that these links are different. **[00:16:11 – 00:16:14]** And what we're finding is that these links have ever **[00:16:14 – 00:16:17]** so slightly more bit errors. **[00:16:17 – 00:16:20]** Now, that might not matter much for TCP-based workloads, **[00:16:20 – 00:16:23]** because TCP expects packet loss and it retransmits. **[00:16:23 – 00:16:26]** Not the case with RDMA workloads, rocky workloads, where the **[00:16:26 – 00:16:30]** basic assumption was near lossless operation. **[00:16:30 – 00:16:33]** And these workloads, these transports, as in traditional rocky **[00:16:33 – 00:16:37]** transport, doesn't lend well to retransmission of dropped packets. **[00:16:37 – 00:16:39]** In fact, there's an outsized impact of dropped packets. **[00:16:39 – 00:16:43]** So you want to minimize the amount of packet drop. **[00:16:43 – 00:16:45]** And then what happens is these links, because they have more **[00:16:45 – 00:16:50]** bit errors, there's this other emergent phenomena, which, **[00:16:50 – 00:16:53]** you know, the industry didn't really think this through, **[00:16:53 – 00:16:57]** which is if you look at the optical specs like IEEE specs for **[00:16:57 – 00:17:01]** 400 gig, 800 gig, they have very tight expectations around how many **[00:17:01 – 00:17:03]** bit errors do you want to accept. **[00:17:03 – 00:17:06]** So normally, if you have three bit errors in three **[00:17:06 – 00:17:08]** consecutive symbols, **[00:17:08 – 00:17:11]** Three consecutive code blocks. **[00:17:11 – 00:17:13]** This is FEC terminology. **[00:17:13 – 00:17:16]** You're supposed to take the link down, which is the wrong **[00:17:16 – 00:17:17]** thing to do for ML workloads. **[00:17:17 – 00:17:20]** Because the moment you take the link down, you now have **[00:17:20 – 00:17:24]** lost capacity, you're dropping a lot more packets, and it takes much **[00:17:24 – 00:17:25]** longer for the link to recover. **[00:17:25 – 00:17:26]** And I have a slide coming up on that. **[00:17:26 – 00:17:28]** Sorry, I'm kind of stealing my own thunder. **[00:17:29 – 00:17:31]** Important problem for us to... **[00:17:31 – 00:17:35]** Solve, because that plays into reliability. **[00:17:35 – 00:17:38]** Similarly, there are new fabric topologies, topologies **[00:17:38 – 00:17:42]** that I believe we were the first to implement. **[00:17:42 – 00:17:44]** So you could say we invented, and we'll talk about that. **[00:17:45 – 00:17:49]** Novel NIC functionality, some of this is all credit to NVIDIA, **[00:17:49 – 00:17:50]** and we'll talk about that. **[00:17:50 – 00:17:54]** You might ask, well, with all that, how is your performance? **[00:17:54 – 00:17:55]** Do you have that question? **[00:17:55 – 00:17:59]** Well, I'm happy to say performance is very good. **[00:17:59 – 00:18:03]** This is a this is a slide from a paper published at **[00:18:03 – 00:18:05]** the SIG Metrics in 2022. **[00:18:05 – 00:18:10]** It's by Torsten et al team from ETH Switzerland. **[00:18:10 – 00:18:13]** And what they were looking for was they were trying to **[00:18:13 – 00:18:17]** To validate a hypothesis, and the hypothesis was that **[00:18:17 – 00:18:20]** you probably cannot run HPC workloads in the cloud, because **[00:18:20 – 00:18:23]** cloud will have high variability of performance, and they defined **[00:18:23 – 00:18:25]** variability of performance as what they call noise. **[00:18:26 – 00:18:30]** So their hypothesis was cloud has more noise than on-premise. **[00:18:30 – 00:18:33]** Well, what they found is that's not the case. **[00:18:33 – 00:18:34]** Noise exists everywhere. **[00:18:34 – 00:18:36]** Noise exists even in on-premise. **[00:18:36 – 00:18:38]** Cloud is not particularly more sensitive to noise. **[00:18:38 – 00:18:41]** But the side effect of that, which is the, I'm cherry picking **[00:18:41 – 00:18:42]** the evidence here. **[00:18:42 – 00:18:45]** The side effect of that is, we actually have very **[00:18:45 – 00:18:46]** good performance. **[00:18:46 – 00:18:49]** So this is where HP, CMPI workloads, they tested AWS, **[00:18:49 – 00:18:52]** Azure, GCP, Oracle, which is OCI, and they tested three **[00:18:53 – 00:18:56]** European supercomputers, Alps, Daint, and Deepest. **[00:18:56 – 00:18:59]** And these are different transports, different network topologies, **[00:18:59 – 00:19:02]** different layer two, layer three technologies. **[00:19:02 – 00:19:05]** And look who's at the top. **[00:19:05 – 00:19:08]** So the right curves are the throughput curves, **[00:19:08 – 00:19:10]** and red is Oracle. **[00:19:10 – 00:19:12]** And the top left curves are latency curves. **[00:19:13 – 00:19:17]** And you can't really tell, but we're almost at the bottom. **[00:19:17 – 00:19:20]** So latency is great. **[00:19:20 – 00:19:22]** Throughput is excellent. **[00:19:22 – 00:19:23]** So performance is great. **[00:19:23 – 00:19:27]** So I'm happy to say all this heavy lifting we're doing **[00:19:27 – 00:19:32]** on behalf of our customers is actually bearing fruit. **[00:19:32 – 00:19:33]** So that was on performance. **[00:19:33 – 00:19:34]** What about high availability? **[00:19:34 – 00:19:37]** You might ask, hey, what you guys are doing about high availability? **[00:19:37 – 00:19:39]** We do a lot of stuff. This is something we focus a lot **[00:19:39 – 00:19:40]** of our attention on. **[00:19:40 – 00:19:44]** In fact, we've done a lot of really creative, innovative work. **[00:19:44 – 00:19:47]** Not all of that is captured here, like for example, we've **[00:19:47 – 00:19:51]** done a lot of work on cleaning the air quality of the data center. **[00:19:51 – 00:19:55]** Turns out that has an impact on quality of the optics, **[00:19:55 – 00:19:59]** and David here is sitting grinning because he's very familiar with it. **[00:19:59 – 00:20:01]** We've done this thing called Optical Link Debalance, and **[00:20:01 – 00:20:03]** Andre, you guys helped with that. **[00:20:04 – 00:20:06]** This is a feature we asked for, and we'll cover that shortly. **[00:20:06 – 00:20:09]** There's new kinds of optics that we are pushing for called **[00:20:09 – 00:20:13]** Link-Aware Optics, and these are optics that give you information **[00:20:14 – 00:20:15]** about a missing leg in the link. **[00:20:15 – 00:20:19]** So if you imagine a bare-metal GPU and you imagine us to **[00:20:19 – 00:20:22]** be the cloud provider, we have information about the switches, **[00:20:22 – 00:20:25]** but we don't have information about the NIC and the GPU because that's **[00:20:25 – 00:20:27]** under the control of the customer. **[00:20:27 – 00:20:30]** So if there's a problem on the NIC or on the optic connecting to the **[00:20:30 – 00:20:33]** NIC, we are kind of blind to it. **[00:20:33 – 00:20:43]** And Link-Aware Optics gives you that extra information **[00:20:43 – 00:20:46]** about the NIC on the GPU or the transceiver on the NIC **[00:20:46 – 00:20:48]** of the GPU or of the host. **[00:20:48 – 00:20:52]** And we find that important for us because then we can distinguish **[00:20:52 – 00:20:56]** whether this is an event caused by the switch or the NIC, so **[00:20:56 – 00:20:59]** we can do the right repair action. **[00:20:59 – 00:21:02]** And then we have, we've worked on a new kind of network design **[00:21:02 – 00:21:04]** we'll look at shortly called Multi-Planar Fabric. **[00:21:04 – 00:21:09]** We believe that makes a big impact on availability, and kudos **[00:21:09 – 00:21:13]** to NVIDIA for working very closely with us to make that happen. **[00:21:13 – 00:21:16]** And we do a bunch of NIC and workload tuning. **[00:21:16 – 00:21:19]** Turns out tuning of the NICL parameters, tuning of the **[00:21:19 – 00:21:23]** congestion control and retry and timeouts on the NIC. **[00:21:23 – 00:21:25]** This has a big impact on workload availability and workload **[00:21:26 – 00:21:30]** reliability in case of small events like link flaps. **[00:21:30 – 00:21:32]** And then we run ISSU on the switches. **[00:21:32 – 00:21:35]** So on all our first-year switches, we run this thing called ISSU, **[00:21:35 – 00:21:38]** which is in-service software update, which is if we ever **[00:21:38 – 00:21:40]** want to upgrade the software on the switch, which generally **[00:21:40 – 00:21:43]** happens twice a year. **[00:21:43 – 00:21:45]** Maybe once or twice a year. **[00:21:45 – 00:21:48]** And we want to do that in a way where we don't impact the workload. **[00:21:48 – 00:21:51]** And this is a difficult thing for most NOS vendors to do, **[00:21:51 – 00:21:56]** because it requires a lot of complex machinery to pull off. **[00:21:58 – 00:22:00]** This is on optical link debounce. **[00:22:00 – 00:22:01]** I briefly touched on it. **[00:22:01 – 00:22:03]** What is optical link debounce? **[00:22:03 – 00:22:05]** So I will very quickly describe what the problem is, and then **[00:22:05 – 00:22:07]** we'll describe what debounce does. **[00:22:07 – 00:22:11]** So in modern optics, what happens is, let's say you had **[00:22:11 – 00:22:16]** those three consecutive bit errors and three consecutive code blocks. **[00:22:16 – 00:22:19]** IEEE says you must now take the link down, right? **[00:22:19 – 00:22:21]** So if you're following the IEEE spec, you're going to **[00:22:21 – 00:22:22]** take the link down. **[00:22:22 – 00:22:25]** Now, what might happen is you may just get Those are **[00:22:25 – 00:22:28]** those three bad code blocks. **[00:22:28 – 00:22:31]** But taking the link down has a very big impact on **[00:22:31 – 00:22:32]** the duration of the outage. **[00:22:32 – 00:22:34]** So what happens when you take the link down is there are all **[00:22:34 – 00:22:36]** these layers at the physical layer. **[00:22:36 – 00:22:39]** There's the PMD layer, media-dependent layer, there's PMA **[00:22:39 – 00:22:42]** layer, the forward error correction layer, the PCS layer, there's **[00:22:42 – 00:22:44]** the reconciliation sublayer. **[00:22:44 – 00:22:47]** All these layers, they have their independent what I call **[00:22:47 – 00:22:50]** link stabilization routines, which is there's this stuff **[00:22:50 – 00:22:52]** you've learned over the years about how long do you want **[00:22:52 – 00:22:56]** to wait for the layer to be. **[00:22:58 – 00:23:00]** And that all adds up, and then the same thing happens in **[00:23:00 – 00:23:04]** the Mac layer, same thing happens in the network layer, IP layer. **[00:23:04 – 00:23:09]** So if you add all that up, let's say the actual link was flapping **[00:23:09 – 00:23:12]** for, let's say, or link went down, let's say, for a microsecond. **[00:23:12 – 00:23:16]** This would become a 10 to 20 seconds worth of outage. **[00:23:17 – 00:23:21]** So it'll take 10 to 20 seconds for the link to go down and come back **[00:23:21 – 00:23:24]** up, and that's the wrong tradeoff. **[00:23:24 – 00:23:27]** And so what we did is we worked with our network vendors, switch **[00:23:27 – 00:23:31]** vendors, and NIC vendors, and asked them, hey, give us the ability **[00:23:31 – 00:23:37]** to configure how long a link down do we want to tolerate before. **[00:23:37 – 00:23:38]** You do all this. **[00:23:38 – 00:23:41]** So we're saying, hey, if the link is down for some reason, **[00:23:41 – 00:23:44]** link is down for 10 microseconds, just drop those packets. **[00:23:44 – 00:23:46]** Do not take the link down, because not only all of this **[00:23:46 – 00:23:49]** 10 to 20 seconds will have to happen, routing protocols **[00:23:49 – 00:23:50]** will have to reconverge, right? **[00:23:50 – 00:23:53]** BGP, OSPF will have to reconverge. **[00:23:53 – 00:23:56]** All the network will have to recompute twice, once when **[00:23:56 – 00:23:58]** the link goes down and the second when the link comes up. **[00:23:58 – 00:24:01]** And then there's all these micro loops that happen when **[00:24:01 – 00:24:02]** the network is reconverging. **[00:24:02 – 00:24:05]** It's completely unnecessary to do. **[00:24:05 – 00:24:07]** So we're saying just **[00:24:07 – 00:24:12]** Do not take the link down if there's a short enough outage. **[00:24:14 – 00:24:17]** Link-aware optics. So I talked a little bit about that earlier. **[00:24:17 – 00:24:20]** So if you have a lot of GPUs, which these large clusters **[00:24:20 – 00:24:25]** do, link-aware optics give you the awareness or visibility **[00:24:25 – 00:24:28]** into what's going on on the transceiver or the optic of **[00:24:28 – 00:24:31]** the NIC on the GPU, because that's under the control of the customer. **[00:24:31 – 00:24:34]** And that allows us to **[00:24:34 – 00:24:38]** You can figure out if this is a true outage or if it's **[00:24:38 – 00:24:41]** a repair activity or some activity on the host. **[00:24:41 – 00:24:46]** And LinkedWare Optics also allow you to sidestep **[00:24:46 – 00:24:49]** some of these issues. **[00:24:49 – 00:24:51]** Basically, the reason this is so important is if you, **[00:24:51 – 00:24:57]** this is from a presentation that Oracle and Credo did at OFC. **[00:24:57 – 00:25:01]** If you have, let's say, 200,000 GPUs, that's the **[00:25:01 – 00:25:02]** statistic at the bottom. **[00:25:02 – 00:25:05]** If you have 200,000 GPUs, then you expect the link flap **[00:25:05 – 00:25:07]** every 12 minutes. **[00:25:07 – 00:25:10]** So that's a lot of link flaps. **[00:25:10 – 00:25:12]** So this, I think, makes the point of the previous slide, **[00:25:12 – 00:25:16]** why link-to-bounce is so important and why having visibility **[00:25:16 – 00:25:19]** into whether it's a flap or it's a repair activity on **[00:25:19 – 00:25:20]** the GPU is so important. **[00:25:20 – 00:25:22]** And that's what link-aware optics gives you. **[00:25:22 – 00:25:24]** It gives you the visibility into whether this is a real **[00:25:25 – 00:25:30]** link flap or somebody did some activity on the GPU. **[00:25:30 – 00:25:33]** So that's on reliability, now let's look at scale. **[00:25:33 – 00:25:37]** So as I said, one of our goals is we wanna have unlimited scale. **[00:25:37 – 00:25:41]** So our goal is unlimited scale, arbitrarily large scale, top-end **[00:25:41 – 00:25:44]** performance, so we seek like 97% of line rate, which we **[00:25:44 – 00:25:45]** generally achieve. **[00:25:45 – 00:25:48]** So if customers do what we want them to do in terms of **[00:25:48 – 00:25:52]** tuning, customers get around 97% of line rate. **[00:25:52 – 00:25:54]** And the rest of the 3% is packet headers. **[00:25:54 – 00:25:56]** So you pretty much get all the line rate. **[00:25:56 – 00:25:59]** And the third is we want the world's highest reliability. **[00:25:59 – 00:26:01]** So that's our goal for ourselves. **[00:26:01 – 00:26:03]** That's our internal SLO. **[00:26:03 – 00:26:05]** That's my SLO. That's David's SLO. **[00:26:05 – 00:26:06]** So we have many scales of fabric. **[00:26:06 – 00:26:08]** This is a small scale. **[00:26:08 – 00:26:10]** Let's say you have a small data center in a geography **[00:26:10 – 00:26:12]** that doesn't need a lot of GPUs. **[00:26:12 – 00:26:13]** This is a network we deploy. **[00:26:13 – 00:26:18]** It's a two-tier single fabric. **[00:26:20 – 00:26:21]** This is our large scale. **[00:26:21 – 00:26:23]** This is a three-tier CLO. **[00:26:23 – 00:26:26]** And by the way, the basic math is two-tier CLO scales at the **[00:26:26 – 00:26:29]** square of the radix of the switch, three-tier CLO scales at the **[00:26:29 – 00:26:30]** cube of the radix of the switch. **[00:26:30 – 00:26:31]** So it's exponential. **[00:26:31 – 00:26:33]** You can also do four-tier. **[00:26:33 – 00:26:35]** We've not needed to do it, but there's no reason why you **[00:26:35 – 00:26:37]** couldn't, in theory, in principle. **[00:26:37 – 00:26:38]** This is a three-tier CLO. **[00:26:38 – 00:26:40]** These are all widely deployed throughout the world. **[00:26:40 – 00:26:44]** Every data center in the world has either one of them or **[00:26:44 – 00:26:46]** often multiple of them. **[00:26:47 – 00:26:49]** This is our newest design, and we worked closely with **[00:26:49 – 00:26:52]** NVIDIA on this, so a lot of credit goes to NVIDIA for helping **[00:26:52 – 00:26:54]** us validate and make this happen. **[00:26:54 – 00:26:58]** A lot of work went on on the NIC and on the GPU, so all **[00:26:58 – 00:27:00]** credit to them on it. **[00:27:01 – 00:27:03]** So this is what we call multi-planar fabric, and you might **[00:27:03 – 00:27:05]** ask, hey, why is this relevant? **[00:27:05 – 00:27:07]** Why is it that you didn't just do three-tiered flow? **[00:27:07 – 00:27:11]** And basically the way to think about that is this is sort **[00:27:11 – 00:27:12]** of the middle ground. **[00:27:12 – 00:27:16]** So T-tier closed scales are the square of the radix, 3-tier closed **[00:27:16 – 00:27:17]** scales are the cube of the radix. **[00:27:17 – 00:27:21]** This scales at more than the square of the radix, less **[00:27:21 – 00:27:25]** than the cube, and the reason it's relevant is it has a **[00:27:25 – 00:27:27]** lot of very interesting properties. **[00:27:27 – 00:27:29]** One other interesting property, and we won't have the time **[00:27:29 – 00:27:32]** to do justice to this, I'll quickly cover them. **[00:27:32 – 00:27:34]** The first most important property of this network **[00:27:34 – 00:27:37]** is that it gives you much higher reliability, which is **[00:27:37 – 00:27:39]** the number one reason to do this. **[00:27:39 – 00:27:42]** You notice here, each NIC, each GPU is connected to **[00:27:42 – 00:27:43]** multiple fabric planes. **[00:27:43 – 00:27:48]** So if there's a link flab or a link outage to one of the planes, **[00:27:48 – 00:27:50]** The workload will keep chugging along. **[00:27:50 – 00:27:52]** You lose capacity, you don't lose reachability. **[00:27:52 – 00:27:54]** So that's the most important thing. **[00:27:54 – 00:27:56]** This is why we really like this design. **[00:27:56 – 00:27:59]** And customers certainly have responded very well to it. **[00:27:59 – 00:28:02]** So high reliability is the number one. **[00:28:02 – 00:28:04]** The second one is it gives you lower latency compared **[00:28:04 – 00:28:06]** to the three-tier cloud, because you don't have the **[00:28:06 – 00:28:08]** third tier of the switches. **[00:28:08 – 00:28:10]** And, in fact, it gives you lower latency and you have **[00:28:10 – 00:28:11]** more consistent latency. **[00:28:12 – 00:28:15]** There's one less band of latency because you're not going through **[00:28:15 – 00:28:18]** the third hop ever. **[00:28:18 – 00:28:20]** And then the next is it saves power. **[00:28:20 – 00:28:24]** It turns out the really large customers really care about **[00:28:24 – 00:28:26]** how much power you're consuming in the network, as in on the **[00:28:26 – 00:28:28]** optics and the switches. **[00:28:28 – 00:28:29]** There's fewer optics here. **[00:28:29 – 00:28:31]** There's fewer switches here. That saves you a lot of power. **[00:28:31 – 00:28:34]** And lastly, it saves cost because you don't have another **[00:28:34 – 00:28:37]** tier of switches and optics. **[00:28:38 – 00:28:41]** So now the next one is innovation in the network. **[00:28:41 – 00:28:42]** So not only do we want to have the largest scale, **[00:28:42 – 00:28:45]** the highest reliability, the highest performance, **[00:28:45 – 00:28:47]** we want to continuously innovate on behalf of our **[00:28:47 – 00:28:50]** customers because we realize that's what our customers want from us. **[00:28:50 – 00:28:53]** They want us to innovate along all axis. **[00:28:53 – 00:28:57]** And we're going to list some example innovations here. **[00:28:57 – 00:28:59]** Firstly, **[00:29:00 – 00:29:04]** Customized NIC functionality, as an example, NVIDIA PRDMA, it's **[00:29:04 – 00:29:06]** an SDK that NVIDIA makes available. **[00:29:06 – 00:29:10]** We believe it's a very significant thing. **[00:29:10 – 00:29:12]** It gives you a lot of control. **[00:29:12 – 00:29:15]** You can do a lot of cool things on the NIC that you couldn't **[00:29:15 – 00:29:19]** do otherwise, because scope of functionality is much larger **[00:29:19 – 00:29:22]** with PRDMA, so it gives you a lot more functionality. **[00:29:22 – 00:29:25]** There's a bunch of new rocky features which are widely **[00:29:25 – 00:29:28]** deployed at this point, including selective acknowledgement. **[00:29:28 – 00:29:31]** So what that basically does is it reduces the impact of packet loss. **[00:29:31 – 00:29:34]** So as opposed to you losing one packet and having to retransmit **[00:29:34 – 00:29:37]** many more packets, what's sometimes called a go-back-end **[00:29:37 – 00:29:41]** problem, you retransmit only the packet that's lost. **[00:29:41 – 00:29:42]** And then delay-based congestion control. **[00:29:42 – 00:29:46]** NVIDIA has a new congestion control called RTTCC. **[00:29:46 – 00:29:48]** I think it's called ZTR RTTCC. **[00:29:48 – 00:29:51]** A lot of letters in that acronym. **[00:29:51 – 00:29:53]** It's a delay-based congestion control, and we found it actually **[00:29:53 – 00:29:55]** is superior to DCQCN. **[00:29:55 – 00:29:58]** So we're actually, in most of the new clusters, we **[00:29:58 – 00:30:00]** are deploying RTTCC. **[00:30:00 – 00:30:04]** Multiport NIC with, so kudos to NVIDIA for that. **[00:30:04 – 00:30:06]** Multiport NIC with single VF and IP. **[00:30:06 – 00:30:08]** This is basically the mechanism that you use **[00:30:08 – 00:30:10]** to connect to Multiplanar. **[00:30:10 – 00:30:11]** Packet spraying on the NIC. **[00:30:11 – 00:30:15]** This is along the lines of what UEC is also proposing, which is **[00:30:15 – 00:30:18]** you spray packets on all the planes and then you do auto replacement. **[00:30:18 – 00:30:22]** These are new NIC features and NVIDIA certainly supports them. **[00:30:22 – 00:30:25]** So thank you for that, NVIDIA. **[00:30:25 – 00:30:30]** Innovative NIC functionality, like connecting to multiple planes. **[00:30:30 – 00:30:32]** And what makes this possible is there's a little Ethernet **[00:30:32 – 00:30:38]** switch on the NIC that allows this connectivity to multiple planes. **[00:30:38 – 00:30:42]** I think I'm repeating myself at this point. **[00:30:44 – 00:30:48]** NVIDIA SpectrumX also is kind of innovative in a way that **[00:30:48 – 00:30:52]** it's able to do... I hope I'm not breaking any... No? **[00:30:52 – 00:30:55]** I'm not saying anything that you don't want me to say. **[00:30:55 – 00:30:57]** There's no secrets that I'm disclosing here, right? **[00:30:57 – 00:31:00]** So NVIDIA switches support the spraying of packets, and **[00:31:00 – 00:31:05]** it's a load-aware spraying, which has an impact on performance. **[00:31:05 – 00:31:07]** So we like that. **[00:31:07 – 00:31:09]** And it counts for availability and performance of the **[00:31:09 – 00:31:11]** plane in doing that. **[00:31:11 – 00:31:14]** Now, it can result in packets arriving out of order, which **[00:31:14 – 00:31:18]** has traditionally been a problem for RDMA, even for TCP. **[00:31:18 – 00:31:22]** So none of the applications handle out-of-order packets well. **[00:31:22 – 00:31:24]** And that's the change here, which is, by the way, UEC **[00:31:24 – 00:31:28]** also has that, which is the latest NVIDIA NICs are able to receive **[00:31:28 – 00:31:31]** packets out of order and place them in the host memory in order. **[00:31:31 – 00:31:35]** So applications don't know any different. **[00:31:35 – 00:31:36]** Which is great. Which is fantastic. **[00:31:36 – 00:31:37]** And we really like that. **[00:31:37 – 00:31:40]** We believe that is, I think, David, you call it the holy **[00:31:40 – 00:31:42]** grail of networking or something like that, holy grail of NICs, **[00:31:42 – 00:31:46]** which is to spray packets and receive them out of order and still **[00:31:46 – 00:31:50]** place them in order in host memory. **[00:31:52 – 00:31:54]** Novel switch functionality, so there are newer things **[00:31:54 – 00:31:59]** on the NIC, like for example, traditionally, Rocky has relied **[00:31:59 – 00:32:01]** on ECN as the signal of congestion. **[00:32:01 – 00:32:04]** So the way it works is, if the network encounters congestion, **[00:32:04 – 00:32:07]** it marks the packets, telling the receiving NIC, hey, there **[00:32:07 – 00:32:08]** was congestion along the way. **[00:32:08 – 00:32:10]** Receiving NIC sends a congestion notification packet back **[00:32:10 – 00:32:11]** to the sending NIC. **[00:32:11 – 00:32:13]** Sending NIC then does congestion control to slow down. **[00:32:13 – 00:32:15]** That's the traditional Rocky. **[00:32:15 – 00:32:17]** It's widely deployed. **[00:32:17 – 00:32:21]** The problem with that is, it takes a full round trip time. **[00:32:21 – 00:32:22]** for congestion control to take place. **[00:32:22 – 00:32:25]** Switches have to mark the package, receiving hosts have **[00:32:25 – 00:32:27]** to receive it, send it to the sender, sender then slows down. **[00:32:27 – 00:32:28]** There's a better way to do that. **[00:32:29 – 00:32:31]** And this also is coming out of UEC and NVIDIA switches **[00:32:31 – 00:32:35]** also have implemented, it's called packet trims, which **[00:32:35 – 00:32:38]** is, if the switch encounters congestion, what it can do is it **[00:32:38 – 00:32:42]** can take the header of the packet that's congested, send it to the **[00:32:42 – 00:32:46]** sending, receiving NIC or better yet to the sending NIC, telling **[00:32:46 – 00:32:49]** it, hey, this packet was congested. **[00:32:50 – 00:32:52]** You might want to act on this congestion. **[00:32:52 – 00:32:54]** Or, God forbid, you have to drop the packet. **[00:32:54 – 00:32:57]** Because in traditional Rocky, that would be a silent drop. **[00:32:57 – 00:33:00]** Here, you're sending the actual packet to the sender or the **[00:33:00 – 00:33:03]** receiving NIC, telling them exactly which packet encountered **[00:33:03 – 00:33:08]** congestion or was dropped, so the NICs can take the correct action. **[00:33:08 – 00:33:10]** And that avoids the go-back-end problem. **[00:33:10 – 00:33:14]** So in summary, we strive for arbitrary large scale. **[00:33:14 – 00:33:18]** Highest possible performance, and I'm glad to say we achieved that, **[00:33:18 – 00:33:22]** and industry-leading availability. **[00:33:22 – 00:33:24]** This implies relentless innovation. **[00:33:24 – 00:33:26]** Like, we are not finished. **[00:33:26 – 00:33:27]** There's a lot of things we're cooking. **[00:33:27 – 00:33:30]** In fact, if I could just drop a hint, there was something **[00:33:30 – 00:33:33]** we were working on we were going to launch, and we were going to try **[00:33:33 – 00:33:37]** and make open source, and NVIDIA launched something like that. **[00:33:37 – 00:33:40]** So some of our, **[00:33:40 – 00:33:43]** I'm a little disappointed at us. **[00:33:43 – 00:33:46]** So there's a new NICL feature NVIDIA launched, like port, **[00:33:47 – 00:33:51]** what is it called? **[00:33:51 – 00:33:54]** Port bonding? What is it called? **[00:33:54 – 00:33:58]** Port fusion? There's a new word for it, which is something we also **[00:33:58 – 00:34:00]** were working on internally, too. **[00:34:00 – 00:34:04]** To make available to the community, but no, I'm like, and there's **[00:34:04 – 00:34:07]** something I've been working on and in connection with our team. **[00:34:07 – 00:34:09]** So I kind of feel like we've been beating to the punch, but **[00:34:09 – 00:34:13]** yeah, there's a lot of things we're trying to do for our customers **[00:34:13 – 00:34:15]** and hopefully for the community as well, because our intent **[00:34:15 – 00:34:17]** there was to make it available. **[00:34:17 – 00:34:20]** For everyone using NICL. **[00:34:20 – 00:34:24]** In future, what we need is we need interoperability between **[00:34:24 – 00:34:25]** different RDMA NICs. **[00:34:25 – 00:34:28]** That's really important. Today, RDMA NICs are not interoperable. **[00:34:28 – 00:34:31]** So if I have an ask from NVIDIA, that would be my ask. **[00:34:31 – 00:34:34]** Please help us with interop. **[00:34:34 – 00:34:36]** And also interop between switch vendors, because **[00:34:36 – 00:34:39]** we're a cloud provider, we need to have multi-sources, **[00:34:39 – 00:34:43]** and for our customers' sake, we really want interop. **[00:34:43 – 00:34:45]** And we want innovations in the transport, kind of like **[00:34:45 – 00:34:46]** the work NVIDIA is doing. **[00:34:46 – 00:34:48]** We want more of that, and particular reliability is **[00:34:48 – 00:34:51]** really important, and I think NVIDIA knows this more than we **[00:34:51 – 00:34:53]** do, but we want more work on that. **[00:34:53 – 00:34:57]** And that is our talk, and I think we're on time, right? **[00:34:57 – 00:34:59]** We're happy to take questions. **[00:34:59 – 00:35:02]** We have five minutes. **[00:35:08 – 00:35:11]** Questions from the audience? **[00:35:14 – 00:35:16]** No questions. I guess you're super clear. **[00:35:16 – 00:35:18]** That's good. **[00:35:19 – 00:35:22]** Hey, Jag. What's our cluster sizing that can be offered **[00:35:22 – 00:35:23]** with the multi-scaler? **[00:35:23 – 00:35:25]** With the multi-planar? Yeah, multi-planar. **[00:35:25 – 00:35:31]** Yeah, so it depends on the radix of the switch. **[00:35:31 – 00:35:35]** So you can certainly do 64,000 in one fabric, but we have **[00:35:35 – 00:35:39]** a way of going beyond that. **[00:35:39 – 00:35:42]** You can probably put the two and two together, how we might do that. **[00:35:42 – 00:35:43]** We didn't have that in this deck. **[00:35:43 – 00:35:46]** I think we kind of went back and forth, should we or should we not? **[00:35:46 – 00:35:47]** We don't have it in the deck, but we have a way **[00:35:47 – 00:35:48]** of scaling beyond that. **[00:35:48 – 00:35:53]** So 64,000 for sure, but you can do a lot more than that using **[00:35:53 – 00:35:56]** well-known scaling techniques. **[00:35:56 – 00:35:57]** On the small scale, what's the? **[00:35:57 – 00:36:01]** Oh, on the small scale. I think small scale would be, **[00:36:02 – 00:36:08]** so our smallest, the one I showed earlier, is I wanna say 2,000 GPUs. **[00:36:09 – 00:36:12]** And we have a smaller than that as well, which **[00:36:12 – 00:36:15]** is artificially shrunk fabric. **[00:36:15 – 00:36:20]** And that, I think, is 256 GPUs, and that's for on-premise deployments. **[00:36:20 – 00:36:23]** Like, we have this product called DRCC, dedicated region **[00:36:23 – 00:36:24]** clouded customer. **[00:36:24 – 00:36:31]** And for DRCC, we have a SKU that, I think it's 256 GPUs **[00:36:31 – 00:36:37]** scale, because other fabrics are too big for that use case. **[00:36:38 – 00:36:39]** Question here. **[00:36:39 – 00:36:42]** So what kind of monitoring capabilities exist to help **[00:36:42 – 00:36:45]** customers understand the problem is at the networking stack **[00:36:46 – 00:36:48]** versus on the compute stack? **[00:36:48 – 00:36:50]** Very good question. It's a very hard problem to solve, **[00:36:50 – 00:36:53]** so just trying to understand what capabilities you expose **[00:36:53 – 00:36:56]** to end customers to find sort of needle in a haystack situation. **[00:36:56 – 00:36:59]** Very good question. Thank you. **[00:36:59 – 00:37:01]** So I'll give a couple of answers. **[00:37:01 – 00:37:03]** So it used to be that we would spend a lot of time hands-on **[00:37:03 – 00:37:05]** with the customer, and we still do. **[00:37:06 – 00:37:07]** But we've gotten better at it. **[00:37:07 – 00:37:11]** So now we have, keep me honest, David, have we launched it? **[00:37:11 – 00:37:16]** There's an active monitoring service we have. **[00:37:16 – 00:37:20]** Which customers can deploy, it's available for every customer, **[00:37:20 – 00:37:22]** they can deploy on their GPUs. **[00:37:22 – 00:37:25]** And what it does, it gives you both reachability and **[00:37:25 – 00:37:29]** latency information from any to any in your cluster. **[00:37:29 – 00:37:31]** So if there's a reachability problem, it'll immediately **[00:37:31 – 00:37:35]** tell you these two GPUs are not able to talk to each other. **[00:37:35 – 00:37:38]** I don't think it does throughput yet, but we're going to do **[00:37:38 – 00:37:39]** throughput as well. **[00:37:39 – 00:37:42]** That is not launched, but we're actually working on it, which is **[00:37:42 – 00:37:44]** on a per-QB basis, it'll tell you. **[00:37:44 – 00:37:47]** So our goal is on a per-QP basis, on a per-GPU basis, **[00:37:47 – 00:37:51]** you can tell what's your throughput and maybe we can grade it. **[00:37:51 – 00:37:54]** So that I don't think is launched, but latency and **[00:37:54 – 00:37:57]** reachability is there. **[00:37:58 – 00:38:03]** Hi, I just had a question on the Converge NIC right here. **[00:38:03 – 00:38:04]** Oh, yeah. **[00:38:05 – 00:38:09]** Is that a customized ConnectX NIC that you guys have created? **[00:38:09 – 00:38:13]** So we have multiple DPU Smart NIC vendors. **[00:38:13 – 00:38:15]** NVIDIA is one of them. **[00:38:15 – 00:38:19]** So I would say it's a Bluefield class of device. **[00:38:19 – 00:38:24]** If that makes any sense, yeah, so that it's a DPU device **[00:38:24 – 00:38:30]** and it's conversion because we have very strict requirements on **[00:38:30 – 00:38:33]** church and state separation as in **[00:38:33 – 00:38:35]** bare metal and our infrastructure. **[00:38:35 – 00:38:39]** So for that reason, it's very customized for the security **[00:38:39 – 00:38:40]** of the infrastructure. **[00:38:40 – 00:38:45]** So so our goal there is that conversion should be such that. **[00:38:45 – 00:38:47]** We cannot get into the customer's domain. **[00:38:47 – 00:38:50]** Customer can never get into the OCI domain. **[00:38:50 – 00:38:53]** So it's very customized for that reason. **[00:38:53 – 00:38:55]** And that's why we've taken this long to do. **[00:38:56 – 00:38:58]** So ConversionNIC is part of Acceleron. **[00:38:58 – 00:39:00]** That's why it's taken us this long. **[00:39:00 – 00:39:02]** This idea is not new to us. **[00:39:02 – 00:39:05]** I was trying to do ConversionNIC in 2018, I think. **[00:39:05 – 00:39:08]** We couldn't pull this off because that **[00:39:08 – 00:39:14]** We were not confident or satisfied with the security isolation story. **[00:39:14 – 00:39:19]** So there's been a lot of work that's been done in that space. **[00:39:20 – 00:39:21]** Yeah, thank you, Jag. **[00:39:21 – 00:39:23]** Great presentation. **[00:39:23 – 00:39:28]** You know, one of the things that I thought you didn't **[00:39:28 – 00:39:31]** touch upon is scale across that NVIDIA talks about and **[00:39:31 – 00:39:33]** multi-building architectures. **[00:39:33 – 00:39:36]** What is Oracle's sort of thought in that, and how are you guys **[00:39:36 – 00:39:38]** looking at larger scale than 64K? **[00:39:38 – 00:39:40]** JAGAD HALALEIRAJANI- Yeah, so we are doing that. **[00:39:40 – 00:39:45]** We didn't talk this, Taylor, David, and I, we didn't **[00:39:45 – 00:39:47]** quite discuss this. **[00:39:47 – 00:39:48]** Upfront, should we include it or not? **[00:39:48 – 00:39:50]** But yeah, we do that. **[00:39:50 – 00:39:52]** So there is, yeah, we do multi-building, yes.