# Enhance Data Center Efficiency and Throughput by Co-Scheduling on NVIDIA CPU+GPU Systems ## Abstract Coherent NVIDIA CPU-GPU systems provide a versatile platform to support a wide spectrum of workloads. However not all jobs require all capabilities at scale. Co-scheduling CPU-only, partially accelerated, and fully GPU-accelerated jobs on NVIDIA CPU-GPU nodes can boost throughput and energy efficiency, removing the need for multiple dedicated clusters. We discuss isolation mechanisms including CPU affinity via cgroups, cache and memory bandwidth partitioning through ARM’s MPAM extensions, and InfiniBand QoS for network traffic separation. Learn to identify co-schedule candidates and assess scheduling policy design options and incentives for user adoption. We'll present workload studies aligned with NERSC’s upcoming Doudna system, demonstrating projected improvements in utilization and efficiency while highlighting open challenges in enabling fine-grained resource sharing. ## AI Summary - The presentation explored the benefits and challenges of co-scheduling GPU and CPU jobs on HPC systems to better utilize resources and improve energy efficiency. - Experiments on a Grace Hopper testbed showed that co-scheduling could increase throughput by almost 50% and reduce energy consumption by 5-20%. - The use of InfiniBand Quality of Service (QoS) and Memory System Resource Partitioning and Monitoring (MPAM) helped mitigate performance impacts, bringing the impact on GPU jobs down to about 5%. - The Doudna system, based on the Vera Rubin architecture, was planned to incorporate co-scheduling to achieve a 10x performance improvement over the Perlmutter system. - The facility aimed to make co-scheduling transparent to users, with options to opt out for those requiring maximum performance. - Key issues to address included performance variability, storage system quality of service, and fault containment strategies for co-scheduled jobs. ## Transcript **[00:00:06 – 00:00:09]** Thank you. So, welcome everyone to the session. **[00:00:09 – 00:00:14]** I'm glad to see you all here, showing up. **[00:00:14 – 00:00:17]** We have, over the past years, seen a trend in accelerated **[00:00:17 – 00:00:20]** computing, that when you put more and more stuff to the **[00:00:20 – 00:00:25]** GPU, you might have systems with a lot of CPU resources, that **[00:00:25 – 00:00:29]** I tried to show in this cartoon here, with the gray boxes that **[00:00:29 – 00:00:33]** go unused because you're getting **[00:00:33 – 00:00:35]** CPUs with more and more cores. **[00:00:35 – 00:00:38]** It's an ever increasing trend in the core count in CPUs. **[00:00:38 – 00:00:42]** That's for one of the reasons is of course because just **[00:00:42 – 00:00:46]** scaling frequency and getting more performance by that stopped **[00:00:46 – 00:00:49]** working around 20 years ago. **[00:00:49 – 00:00:52]** And so since about for 20 years or so we're getting **[00:00:52 – 00:00:57]** more and more cores and CPUs now have on the order of 100 cores. **[00:01:01 – 00:01:05]** As you offload all your parallelism to the GPU, which is a massive **[00:01:05 – 00:01:09]** throughput processor, you don't necessarily need all **[00:01:09 – 00:01:11]** these cores on the CPU. **[00:01:11 – 00:01:14]** And again, on the other hand, HPC data centers need to support **[00:01:14 – 00:01:19]** a wide variety of workloads, from classical CPU-only codes, **[00:01:19 – 00:01:22]** some partially accelerated codes that have phases that run on the **[00:01:22 – 00:01:26]** CPU and the GPU, and of course you don't want the CPU parts to slow **[00:01:26 – 00:01:29]** your GPU parts down unnecessarily. **[00:01:29 – 00:01:31]** And then of course you have these very highly accelerated **[00:01:31 – 00:01:35]** codes, which basically just are launching kernels on the GPU and **[00:01:35 – 00:01:39]** don't do much else on the CPU side. **[00:01:39 – 00:01:42]** They also need to support AI applications more and more, **[00:01:42 – 00:01:46]** and for that reason they need a lot of flexibility in the system. **[00:01:46 – 00:01:49]** They need a high system memory bandwidth, and they **[00:01:49 – 00:01:54]** need enough CPU resources towards these AMDAR limiters. **[00:01:54 – 00:01:57]** And so, the idea is, can we make use of these resources **[00:01:57 – 00:02:00]** somehow that sometimes go unused? **[00:02:00 – 00:02:04]** One of the ideas here is, why don't we just put... **[00:02:04 – 00:02:08]** Different drops on the system in parallel and fill these resources. **[00:02:08 – 00:02:12]** So trying to sketch it here, you might have your GPU jobs **[00:02:12 – 00:02:14]** running, a GPU job here over four nodes, and then you put **[00:02:14 – 00:02:19]** different CPU jobs indicated by these different colors on them, and **[00:02:20 – 00:02:25]** then just make better use of your resources that you already have. **[00:02:25 – 00:02:29]** Sounds like a great idea at first, but are there any issues with it? **[00:02:29 – 00:02:33]** Well, of course, we're sharing the resources on the node **[00:02:33 – 00:02:34]** and on the system. **[00:02:34 – 00:02:36]** Some things are easy, like CPU costs. **[00:02:36 – 00:02:39]** We can just dedicate CPU costs to the GPU job, the **[00:02:39 – 00:02:41]** others to the CPU job. **[00:02:41 – 00:02:44]** We can also just divide the memory capacity, and then **[00:02:45 – 00:02:48]** we have tools like cgroups and these job schedulers support **[00:02:48 – 00:02:52]** that, that we can make sure that every job stays within its limits. **[00:02:52 – 00:02:56]** But then we also have shared resources like memory bandwidth, **[00:02:56 – 00:03:00]** the last level cache, the network, **[00:03:00 – 00:03:03]** I.O., Power, and we need to understand **[00:03:03 – 00:03:05]** the performance impact on that. **[00:03:05 – 00:03:08]** This is what I tried to sketch on this cartoon, where you **[00:03:08 – 00:03:11]** have these green resources that go to the CPU job. **[00:03:11 – 00:03:15]** And for some of the resources like the NIC or the access to the memory **[00:03:15 – 00:03:19]** with the system memory bandwidth, this is not a hard limit. **[00:03:19 – 00:03:22]** We are here now trying to look at the issues that come **[00:03:22 – 00:03:26]** with sharing the memory bandwidth, the L3 cache and the NIC. **[00:03:27 – 00:03:29]** I always skip that because it's typically already a shared **[00:03:29 – 00:03:33]** resource in your system and for power reasons this is **[00:03:33 – 00:03:36]** mostly only an issue if you're extremely power-constrained **[00:03:36 – 00:03:42]** and cannot run your node at the full power that the node can take. **[00:03:42 – 00:03:44]** So for that we decided we'd do some experiments. **[00:03:44 – 00:03:48]** We have a small cluster that we built out of four **[00:03:48 – 00:03:49]** Grace Hopper nodes. **[00:03:49 – 00:03:55]** Each has a Grace CPU with 72 cores, 480 gigabytes of LPDDR RAM. **[00:03:55 – 00:03:59]** On Hopper GPU, and we also have two network adapters **[00:03:59 – 00:04:03]** in the system, which allows us to do a bit more advanced experiments. **[00:04:03 – 00:04:06]** We then decided, okay, we will take two workloads on **[00:04:06 – 00:04:11]** the CPU, OpenFOAM, which mostly stresses the memory system. **[00:04:11 – 00:04:14]** On the CPU, and then we also use Toast, an application **[00:04:14 – 00:04:18]** that more stresses the network components of the system. **[00:04:18 – 00:04:21]** And we put different GPU workloads on there, so we have Chroma, which **[00:04:21 – 00:04:25]** is one of the fully accelerated apps, really just mostly using **[00:04:25 – 00:04:27]** one core to launch kernels. **[00:04:27 – 00:04:32]** GROMACS, Highly Accelerated Molecular Dynamics Code, which **[00:04:32 – 00:04:36]** has some parts on the CPU that can become performance critical. **[00:04:36 – 00:04:39]** And MILC as a more classical offload. **[00:04:39 – 00:04:42]** Most of the things are offloaded to the GPU, but there are **[00:04:42 – 00:04:45]** still things that run on the CPU and need the high memory **[00:04:45 – 00:04:47]** bandwidth on the CPU side. **[00:04:47 – 00:04:53]** We then dedicate 8 cores to the GPU jobs, 64 cores of the 72 that **[00:04:53 – 00:04:57]** we have with Grace to the CPU jobs. **[00:04:57 – 00:05:01]** And then we run just overlapping jobs using Slurm and just making **[00:05:01 – 00:05:05]** sure that they always overlap by looping the jobs for long enough. **[00:05:06 – 00:05:09]** And if we just try that, now let me explain that graph here. **[00:05:09 – 00:05:12]** So we run the jobs in isolation and look at **[00:05:12 – 00:05:14]** the performance that we get out. **[00:05:14 – 00:05:16]** And then we run these combinations. **[00:05:16 – 00:05:19]** So the left part is running with OpenFOAM on the CPU, **[00:05:19 – 00:05:21]** which stresses the memory system. **[00:05:21 – 00:05:25]** Right part is running with Toast on the CPU and stressing **[00:05:25 – 00:05:26]** more the network part. **[00:05:26 – 00:05:29]** But you see the impact on the GPU apps in this orange **[00:05:29 – 00:05:33]** bar is on the order of 20%. **[00:05:34 – 00:05:37]** A lot of them have a very small impact than MILC, a **[00:05:37 – 00:05:42]** bit higher with OpenFOAM, but in the order of 1 to 10%. **[00:05:42 – 00:05:46]** And then what can we do about that? **[00:05:46 – 00:05:48]** Is there something that we can do to improve these **[00:05:49 – 00:05:52]** performance impacts that we see by just co-scheduling **[00:05:52 – 00:05:59]** without any guardrails that we have for these jobs? **[00:05:59 – 00:06:02]** One thing that we can look at, we can use InfiniBand **[00:06:02 – 00:06:03]** quality of service features. **[00:06:04 – 00:06:07]** That means we can assign different service levels to these jobs, **[00:06:07 – 00:06:11]** a feature of the InfiniBand network, and then we can assign **[00:06:11 – 00:06:15]** weights to these lanes that these jobs then use. **[00:06:15 – 00:06:19]** It's basically like we're doing a sharing of the bandwidth **[00:06:19 – 00:06:22]** with a fixed ratio. **[00:06:22 – 00:06:25]** On this plot here, we're looking just at the Toast application on **[00:06:25 – 00:06:29]** the CPU and then using the GeoMean over the GPU apps, just because **[00:06:29 – 00:06:32]** otherwise we have a ton of bars on these and it makes it hard to see **[00:06:32 – 00:06:34]** what we're actually caring about. **[00:06:34 – 00:06:39]** On the left, we're using a shared NIC and do nothing, no QoS applied. **[00:06:39 – 00:06:43]** And to the very right, because we have two network adapters, **[00:06:43 – 00:06:46]** we give each application one network adapter. **[00:06:46 – 00:06:50]** And this is kind of just as a baseline, because in practice **[00:06:50 – 00:06:53]** you don't want to double your network for the full system. **[00:06:53 – 00:06:56]** This will become expensive quickly. **[00:06:56 – 00:06:59]** And then if we use QoS, what you see as soon as we turn **[00:06:59 – 00:07:03]** QoS and we're splitting the bandwidth 50-50, and then **[00:07:03 – 00:07:08]** we're giving 20-80, 10-90, just to play with it a bit, giving **[00:07:08 – 00:07:10]** more weight to the GPU application. **[00:07:10 – 00:07:13]** We can bring down the impact on the GPU application on **[00:07:13 – 00:07:18]** the order to 5%, same level that we see when we have a **[00:07:18 – 00:07:20]** dedicated NIC for the GPU job. **[00:07:20 – 00:07:24]** So this tells us with QoS on the InfiniBand level, we **[00:07:24 – 00:07:25]** can mitigate the impact. **[00:07:25 – 00:07:26]** Of course, there are some trade-offs. **[00:07:26 – 00:07:30]** The CPU job sees a bit higher impact here, but **[00:07:30 – 00:07:34]** overall, we're improving the sharing of the network. **[00:07:34 – 00:07:40]** And then for the memory, Grace, the CPU, and also **[00:07:40 – 00:07:45]** in the future, Vera, support ARM, AMPA, Memory System Resource **[00:07:45 – 00:07:46]** Partitioning and Monitoring. **[00:07:46 – 00:07:51]** That allows us to assign the memory traffic to partitions, **[00:07:51 – 00:07:53]** which are then monitored. **[00:07:53 – 00:07:57]** And by that, we can also control how much memory bandwidth **[00:07:57 – 00:08:00]** the job can consume of the system. **[00:08:00 – 00:08:02]** And we can also assign **[00:08:02 – 00:08:07]** percentage of the L3 capacity, the last level cache in the **[00:08:07 – 00:08:09]** system to these jobs. **[00:08:09 – 00:08:14]** What we're seeing on that plot here with left and right most **[00:08:14 – 00:08:18]** These are the same values, as you might have noticed already. **[00:08:18 – 00:08:21]** We don't do any mitigation for the memory system, we **[00:08:21 – 00:08:22]** just leave it as is. **[00:08:22 – 00:08:26]** I've run this experiment using the dedicated NIC so that **[00:08:26 – 00:08:30]** we are not impacted by the network as much as possible **[00:08:30 – 00:08:33]** and clearly can separate out the impact of the memory system. **[00:08:33 – 00:08:35]** And then from the left, we're adding **[00:08:35 – 00:08:38]** the partitioning of the L3 cache. **[00:08:38 – 00:08:40]** Which already improves quite a bit, and we're now looking **[00:08:40 – 00:08:43]** of course at OpenFOAM, because that's the workload that stresses **[00:08:43 – 00:08:46]** the memory system on the CPU. **[00:08:46 – 00:08:48]** And we see that we can reduce the impact of the **[00:08:49 – 00:08:53]** GPU drops when we kind of start partitioning the L3 cache. **[00:08:53 – 00:08:56]** And on the right you see when we're now also partitioning **[00:08:56 – 00:08:58]** the bandwidth, giving 70% of the bandwidth to the **[00:08:58 – 00:09:01]** CPU, 30% to the GPU. **[00:09:01 – 00:09:05]** We can reduce the impact on the GPU job quite significantly. **[00:09:05 – 00:09:09]** In the middle, both combinations are combined, and then the **[00:09:09 – 00:09:13]** GPU sees an impact of around 5%. **[00:09:13 – 00:09:15]** We see a bit higher impact on the CPU. **[00:09:15 – 00:09:18]** This is just because we need to limit the bandwidth that **[00:09:18 – 00:09:23]** the CPU job can take to make sure that the GPU gets the **[00:09:23 – 00:09:27]** system memory bandwidth it needs. **[00:09:28 – 00:09:33]** And, of course, finally, because if we want to deploy that **[00:09:33 – 00:09:38]** in practice on clusters, the goal is to have a good baseline **[00:09:38 – 00:09:39]** solution that we can apply. **[00:09:39 – 00:09:41]** We don't want to tune that for every application combination. **[00:09:42 – 00:09:45]** We just did these experiments to understand the mechanisms **[00:09:45 – 00:09:47]** step Step by step. **[00:09:47 – 00:09:50]** We will combine them and now I'm taking the GeoMean over everything **[00:09:50 – 00:09:53]** because in practice you will have multiple different applications **[00:09:53 – 00:09:58]** running in your HPC centers, not just one-on-one combinations **[00:09:58 – 00:10:00]** of your applications in most cases. **[00:10:00 – 00:10:06]** So from the similar plot as before, on the outside we're looking **[00:10:06 – 00:10:10]** at the unmitigated scenario, now again sharing the network **[00:10:10 – 00:10:13]** adapter, so competing for it. **[00:10:13 – 00:10:15]** On the left, we're adding the InfiniBand **[00:10:15 – 00:10:18]** QoS, reducing the impact. **[00:10:18 – 00:10:22]** From the right, we're adding the MPAM, both for the memory **[00:10:22 – 00:10:25]** bandwidth and the L3 capacity. **[00:10:25 – 00:10:28]** And then in the middle, we'll meet and get the impact again **[00:10:28 – 00:10:35]** down to roughly 5% on the GPU level, 14% on the CPU level. **[00:10:35 – 00:10:38]** So we can control with these methods the impact. **[00:10:38 – 00:10:41]** And of course, we have just shown you some **[00:10:42 – 00:10:45]** Values that we we played of course with these values a **[00:10:45 – 00:10:49]** bit and and shows a good starting point But of course in practice **[00:10:49 – 00:10:52]** you will be able to to tune that for whatever workload scenarios **[00:10:53 – 00:10:56]** you need to support These are just starting points to explain the **[00:10:56 – 00:11:01]** concept and nothing that is kind of fixed values that you need to use **[00:11:01 – 00:11:06]** It's just some good combinations that we identified here **[00:11:06 – 00:11:10]** You might also ask, but what happens if we don't just do **[00:11:10 – 00:11:14]** that nice one-on-one pairing and for that we've done some **[00:11:14 – 00:11:18]** experiment here just to explain you what you're looking at here. **[00:11:18 – 00:11:21]** You see on the bottom I've tried to sketch it a bit. **[00:11:21 – 00:11:26]** We're running GPU drops over four GPUs now, over four nodes. **[00:11:26 – 00:11:29]** Then on two of the nodes I put open foam to the left **[00:11:29 – 00:11:31]** node and you see the impact. **[00:11:31 – 00:11:35]** The paler color is the unmitigated scenario. **[00:11:35 – 00:11:39]** Doing nothing, just running it, shared, and the darker full **[00:11:39 – 00:11:46]** color is with the QoS and MPAM for L3 and Memory Bandwidth mitigation **[00:11:46 – 00:11:47]** strategies that we tried. **[00:11:47 – 00:11:52]** So for these, the left two is basically similar to what you saw **[00:11:52 – 00:11:54]** already, just that we're now using **[00:11:54 – 00:11:56]** That's split. **[00:11:56 – 00:12:02]** And to the right, for that plot, I decided, now we will put on two **[00:12:02 – 00:12:07]** of the CPU nodes, open form, and on those, stress the memory system. **[00:12:07 – 00:12:10]** On the other ones, we stress... **[00:12:10 – 00:12:14]** The network and what you see for the unmitigated case, **[00:12:14 – 00:12:20]** you see a quite large impact that is almost additive. **[00:12:20 – 00:12:24]** If you, for example, look at the milk bars 25 and 30 **[00:12:24 – 00:12:30]** and the combined impact without mitigations is over 50 percent. **[00:12:30 – 00:12:33]** But if we then turn on our mitigations methods again that **[00:12:33 – 00:12:39]** we've experimented with, you see that the impact in the worst case **[00:12:39 – 00:12:42]** And when you combine both, it's not much higher than **[00:12:43 – 00:12:47]** kind of, if you look, for example, OpenFOAM plus GROMAC, **[00:12:47 – 00:12:51]** the green bar, is around 5%. **[00:12:51 – 00:12:54]** Toast plus GROMAC is a little lower. **[00:12:54 – 00:12:59]** And the OpenFOAM and Toast and GROMAC combination is **[00:12:59 – 00:13:03]** just a little higher than the OpenFOAM and GPU combination. **[00:13:03 – 00:13:05]** And that also holds for the other bars, and you can look **[00:13:05 – 00:13:07]** at it, for example, milk. **[00:13:08 – 00:13:12]** Toast and GPU gets you maybe 9% and more like 11% in the case. **[00:13:12 – 00:13:17]** So the mitigation strategies also work in that case. **[00:13:17 – 00:13:21]** The final thing that I will discuss is energy savings. **[00:13:21 – 00:13:25]** So basically, you're already running your GPU node. **[00:13:25 – 00:13:27]** You have the CPU unused. **[00:13:27 – 00:13:32]** If you're now running something on a CPU, you get some throughput **[00:13:32 – 00:13:34]** on that CPU part. **[00:13:34 – 00:13:37]** You need to pay extra power for the CPU part because the CPU **[00:13:37 – 00:13:40]** will need more power when there's something more running on it. **[00:13:40 – 00:13:44]** But you don't need to pay all the power for the node because **[00:13:44 – 00:13:48]** that is already running or the data center that is already there. **[00:13:48 – 00:13:51]** What I've done here, I've looked at how much throughput **[00:13:51 – 00:13:55]** do I get for these combinations when I co-schedule them. **[00:13:55 – 00:13:59]** And then I work out how much energy would it just take me to run the... **[00:14:00 – 00:14:02]** GPU job in isolation, how much energy would it take **[00:14:02 – 00:14:07]** me to run the CPU job in isolation, and then just add them up. **[00:14:07 – 00:14:10]** Now looking here at the PSU level, this is specific measurements **[00:14:11 – 00:14:13]** for the node that we had used here. **[00:14:13 – 00:14:18]** So might be different, node designs will all behave somewhat **[00:14:18 – 00:14:22]** different, but with that, with co-scattering, we are making use of **[00:14:22 – 00:14:25]** the full power budget of the node. **[00:14:25 – 00:14:31]** And don't have to pay extra, basically, base power of **[00:14:31 – 00:14:32]** having more nodes. **[00:14:33 – 00:14:38]** And so we're saving at the PSU level between 5 and 20% of **[00:14:38 – 00:14:40]** energy when running co-scheduled. **[00:14:41 – 00:14:45]** With that, I will hand it over to my co-presenter, Brian, from NERSC. **[00:14:45 – 00:14:50]** We'll discuss how they are looking in using these techniques. **[00:14:50 – 00:14:52]** So today I'm speaking on behalf of NERSC. **[00:14:52 – 00:14:55]** For those of you who are not familiar with NERSC, it is **[00:14:55 – 00:14:59]** the Mission High Performance Computing Facility for the **[00:14:59 – 00:15:00]** US Department of Energy. **[00:15:00 – 00:15:05]** That means that we support a very diverse scientific workload **[00:15:05 – 00:15:13]** ranging from material science to particle physics to plasma physics. **[00:15:13 – 00:15:18]** We have about 11,000 users and we produce or support **[00:15:18 – 00:15:21]** the production of over 2,500 publications per year. **[00:15:23 – 00:15:28]** So we run large-scale HPC very frequently, and if you'll **[00:15:28 – 00:15:35]** move to the next slide, we do that by providing new advanced **[00:15:35 – 00:15:42]** computing systems on a cadence of roughly once every five years. **[00:15:46 – 00:15:50]** The Perlmutter system that you see on the lower right **[00:15:50 – 00:15:57]** was installed in 2019 that has close to 8,000 ampere GPUs **[00:15:57 – 00:16:04]** in it and the Doudna system that we are planning to deliver toward **[00:16:04 – 00:16:09]** the end of this year will be based on the Vera Rubin architecture. **[00:16:09 – 00:16:15]** So when we learned from NVIDIA about co-scheduling and the **[00:16:15 – 00:16:18]** QoS and MPAM technologies that went into supporting **[00:16:18 – 00:16:21]** it, we were very excited to evaluate it and find out whether it **[00:16:21 – 00:16:23]** would work well for our workload. **[00:16:23 – 00:16:27]** When we designed the Perlmutter system, it intentionally **[00:16:27 – 00:16:32]** included both a GPU accelerated partition and a CPU partition, **[00:16:32 – 00:16:34]** And that's because, um... **[00:16:34 – 00:16:38]** The variety of codes that we run at NERSC need both **[00:16:38 – 00:16:39]** types of processors. **[00:16:39 – 00:16:47]** Not all of the codes have been ported to GPUs yet. **[00:16:47 – 00:16:50]** Furthermore, both of these partitions are constantly **[00:16:50 – 00:16:54]** running at over 95% utilization. **[00:16:54 – 00:17:02]** So there's no way to get away from supporting our CPU-based codes. **[00:17:02 – 00:17:07]** And so the question now is whether co-scheduling is a **[00:17:07 – 00:17:13]** viable way for us to support both of those workloads. **[00:17:14 – 00:17:18]** And I'll investigate that on the subsequent slides. **[00:17:22 – 00:17:25]** So the first step is simply a back-of-the-envelope **[00:17:25 – 00:17:29]** calculation of how much more throughput can we get by **[00:17:29 – 00:17:31]** using co-scheduling than without. **[00:17:31 – 00:17:37]** This was a very simple analytical model where we basically assumed **[00:17:37 – 00:17:43]** that the performance of the GPU codes is unchanged by using **[00:17:43 – 00:17:48]** co-scheduling, and the performance of the CPU codes is proportional to **[00:17:48 – 00:17:51]** the number of cores that it used. **[00:17:51 – 00:17:54]** You can see just how simple it is by the resulting formula **[00:17:54 – 00:17:55]** on the last line. **[00:17:55 – 00:17:58]** I'm not going to walk you through it, but that you can see that there **[00:17:58 – 00:18:01]** are really only three factors. **[00:18:01 – 00:18:07]** One is the fraction of cores that you devote to your CPU jobs **[00:18:07 – 00:18:13]** on the co-scheduled CPU jobs, the number of cores on the different **[00:18:13 – 00:18:18]** architectures involved, and the prices of the different components. **[00:18:18 – 00:18:23]** For the configuration that we're anticipating at NERSC, **[00:18:23 – 00:18:28]** I'm confident that we'll be able to achieve about 50% **[00:18:28 – 00:18:34]** more throughput if this model pans out with actual measurements, **[00:18:34 – 00:18:37]** and that's what I'm going to get to in the next few slides. **[00:18:44 – 00:18:49]** So to begin with, we looked back at the existing Perlmutter **[00:18:49 – 00:18:56]** workload to evaluate whether there really are available host resources **[00:18:56 – 00:19:00]** for running CPU jobs on GPU nodes. **[00:19:00 – 00:19:03]** So these are... **[00:19:03 – 00:19:06]** We have ongoing telemetry running on the Perlmutter **[00:19:06 – 00:19:10]** system all the time that's sampling the available free memory and the **[00:19:10 – 00:19:19]** number of cores that are used at every 10 seconds across the system. **[00:19:22 – 00:19:27]** What I've shown here is a heat map of the number of node **[00:19:27 – 00:19:34]** hours that use a certain amount of memory or a given number of CPUs. **[00:19:34 – 00:19:38]** The point is that in the upper left, all of the utilization **[00:19:38 – 00:19:42]** is concentrated in this region where you're using fewer than **[00:19:42 – 00:19:46]** a quarter of the available cores on the node and less than **[00:19:46 – 00:19:49]** half of the available host memory. **[00:19:49 – 00:19:54]** That means that we have a lot of capability to run CPU **[00:19:54 – 00:19:58]** jobs on our GPU nodes. **[00:19:59 – 00:20:02]** So next slide, please. **[00:20:07 – 00:20:10]** So now we're getting into the actual performance measurements **[00:20:10 – 00:20:11]** related to co-scheduling. **[00:20:11 – 00:20:15]** These tests were not done on Perlmutter but on our two-node **[00:20:15 – 00:20:18]** Grace Hopper testbed. **[00:20:19 – 00:20:24]** There are two factors that contribute to the performance **[00:20:24 – 00:20:27]** of a co-scheduled job. **[00:20:28 – 00:20:31]** One is the reduction in performance that occurs just from the **[00:20:31 – 00:20:34]** fact that you're partitioning the resources on the node. **[00:20:34 – 00:20:38]** You have fewer cores available to you, that's going to decrease **[00:20:38 – 00:20:39]** your performance. **[00:20:39 – 00:20:43]** The other is that there's another job present that's **[00:20:43 – 00:20:46]** competing for your resources. **[00:20:47 – 00:20:50]** I'm attempting to separate those two factors here by just looking **[00:20:50 – 00:20:57]** at the difference in performance of a suite of jobs when it runs **[00:20:57 – 00:21:01]** on the node by itself using all of the available resources and when **[00:21:01 – 00:21:06]** it's running only on the resources that it will have to itself **[00:21:06 – 00:21:09]** when running in co-scheduled. **[00:21:09 – 00:21:13]** The table on the left shows how we have partitioned the **[00:21:13 – 00:21:16]** resources within this experiment. **[00:21:16 – 00:21:21]** The middle figure shows how that has affected five different **[00:21:21 – 00:21:26]** GPU applications from our benchmarking test suite. **[00:21:26 – 00:21:30]** You can see that there are some **[00:21:32 – 00:21:36]** Certainly measurable, but not tremendous decreases in **[00:21:36 – 00:21:42]** the performance of the GPU apps and that between the **[00:21:42 – 00:21:46]** CPU codes the performance has gone down by about 25%. **[00:21:46 – 00:21:49]** That's consistent with our earlier very simple model **[00:21:49 – 00:21:53]** where you've decreased the number of CPU cores available to the **[00:21:53 – 00:22:00]** job and so the performance would have gone down commensurately. **[00:22:01 – 00:22:10]** So on the next slide, I will describe to you the capabilities **[00:22:10 – 00:22:15]** of network QoS and **[00:22:16 – 00:22:21]** And MPAM to provide performance isolation between jobs running **[00:22:21 – 00:22:23]** on the same node. **[00:22:23 – 00:22:31]** So on the left cluster of columns, I show the performance of these **[00:22:31 – 00:22:35]** jobs when they're run in isolation on their dedicated partition. **[00:22:35 – 00:22:38]** That sets our expectation for what we should be able **[00:22:38 – 00:22:46]** to achieve when we start running with a co-scheduled job. **[00:22:46 – 00:22:50]** In the next slide, I'm sorry, not in the next slide, the next cluster **[00:22:50 – 00:22:57]** of columns, I have introduced a bandwidth virus that uses, if **[00:22:57 – 00:23:02]** run by itself, would use all of the available network bandwidth and all **[00:23:02 – 00:23:05]** of the available memory bandwidth. **[00:23:05 – 00:23:08]** So this is a worst case scenario for co-scheduling. **[00:23:08 – 00:23:10]** That's as bad as it can get. **[00:23:10 – 00:23:13]** And you can see that that really **[00:23:13 – 00:23:17]** Does diminish the performance if you don't protect it **[00:23:17 – 00:23:22]** with MPAM and IBQoS. **[00:23:22 – 00:23:29]** In clusters C, D, and E, I incrementally turn on the **[00:23:29 – 00:23:38]** isolation mitigations for memory bandwidth, CPU cache, and network **[00:23:38 – 00:23:44]** cache, and you incrementally see improved performance of **[00:23:44 – 00:23:47]** the co-scheduled GPU job. **[00:23:48 – 00:23:51]** In the last clusters, F and G, I have replaced our bandwidth **[00:23:51 – 00:23:59]** virus with two real CPU codes, QoS and HMMsearch. **[00:23:59 – 00:24:02]** And again, you can see the performance go up of the **[00:24:03 – 00:24:07]** co-scheduled GPU job go up again. **[00:24:08 – 00:24:12]** You might be very curious about why that performance can **[00:24:12 – 00:24:15]** exceed 100% for some of those jobs. **[00:24:15 – 00:24:19]** That's because in the reference column on the far left, we **[00:24:19 – 00:24:27]** were able to apply a strict maximum bandwidth on the network by using **[00:24:27 – 00:24:33]** A tool called ibportconfig, but in experiments fng, we're instead **[00:24:33 – 00:24:38]** using the ibqos technology, which does not set a maximum bandwidth, **[00:24:38 – 00:24:45]** but sets a promise, not quite a guarantee, that the co-scheduled **[00:24:45 – 00:24:50]** job will be able to use at least its assigned fraction of the **[00:24:50 – 00:24:52]** available Total network bandwidth. **[00:24:52 – 00:24:56]** So when you're observing more than 100% performance here, what you're **[00:24:56 – 00:25:00]** seeing is that those applications are able to take advantage **[00:25:00 – 00:25:06]** of the network bandwidth that was not being used by the CPU code. **[00:25:08 – 00:25:12]** On the next slide I do the same set of experiments for **[00:25:12 – 00:25:17]** our CPU benchmarks and it's very easy to see that these **[00:25:17 – 00:25:24]** are far less affected by running a co-scheduled job. **[00:25:28 – 00:25:33]** And next slide, please. **[00:25:35 – 00:25:39]** So combining all of the same **[00:25:39 – 00:25:42]** performance results from the last two slides, which showed **[00:25:42 – 00:25:49]** how performance decreased when co-scheduling, what they **[00:25:49 – 00:25:53]** didn't show was that the throughput by running jobs simultaneously **[00:25:53 – 00:25:55]** actually increased. **[00:25:55 – 00:25:58]** So our metric for that is something that we've developed **[00:25:59 – 00:26:01]** at NERSC over the last few years called Workflow SSI. **[00:26:01 – 00:26:05]** It's essentially an average over the different applications. **[00:26:05 – 00:26:10]** Weighted in a way that accounts for the existence **[00:26:10 – 00:26:12]** of different partitions. **[00:26:12 – 00:26:16]** And you can see that this goes up by almost 1.5x. **[00:26:18 – 00:26:22]** It's basically a coincidence that this 1.5x agrees with **[00:26:22 – 00:26:25]** the 50% that I estimated earlier. **[00:26:25 – 00:26:29]** The earlier estimate was **[00:26:29 – 00:26:33]** really based on changes to the architecture with **[00:26:33 – 00:26:35]** an idealized performance model. **[00:26:35 – 00:26:39]** Here we've made no changes to the architecture and all we've done **[00:26:39 – 00:26:44]** is explore a different operational mode for the same hardware. **[00:26:44 – 00:26:48]** Those two factors compound each other, so in practice you would be **[00:26:48 – 00:26:52]** able to see something that's higher than either of them, but it's **[00:26:52 – 00:26:58]** very difficult to estimate that without getting the real hardware. **[00:26:58 – 00:26:59]** This slide is measured data. **[00:26:59 – 00:27:07]** The earlier slide was back at the end of the model. **[00:27:08 – 00:27:09]** So, next slide. **[00:27:09 – 00:27:15]** So we now have a lot of confidence that co-scheduling works well **[00:27:15 – 00:27:21]** for both throughput and performance isolation using real data measured **[00:27:21 – 00:27:24]** on our Grace Hopper testbed. **[00:27:24 – 00:27:27]** Later this year, we're going to receive delivery of the **[00:27:27 – 00:27:31]** Doudna system, which I mentioned earlier is based on Vera Rubin. **[00:27:31 – 00:27:36]** Vera is going to have new enhancements to MPAM, **[00:27:36 – 00:27:39]** which should further improve performance isolation. **[00:27:39 – 00:27:43]** And this gives us a lot of confidence in relying **[00:27:43 – 00:27:48]** on co-scheduling to achieve the 10x performance commitment **[00:27:48 – 00:27:50]** relative to Perlmutter. **[00:27:50 – 00:27:55]** We've included that in our design of the system. **[00:27:56 – 00:28:00]** So, again, just to repeat that point, it's very important, **[00:28:00 – 00:28:05]** we believe in co-scheduling, we're planning to use it on **[00:28:05 – 00:28:08]** NERSC 10, also known as Doudna. **[00:28:10 – 00:28:13]** So the next slide, I will mention that until now I've **[00:28:14 – 00:28:17]** been talking only about the technical There are several **[00:28:17 – 00:28:25]** capabilities for co-scheduling and the performance aspects of it. **[00:28:25 – 00:28:27]** In order to deploy it, there are a lot of other things that we **[00:28:28 – 00:28:30]** need to consider and be aware of. **[00:28:31 – 00:28:34]** One of those is if we, as a... **[00:28:36 – 00:28:40]** As a facility, configure co-scheduling correctly. **[00:28:40 – 00:28:42]** We can make this almost entirely transparent to our users, **[00:28:42 – 00:28:45]** which is a great benefit. **[00:28:45 – 00:28:50]** But those users need to be aware of the performance changes so that **[00:28:50 – 00:28:54]** they aren't upset when they're not getting 100% of the CPU performance **[00:28:54 – 00:29:00]** that they may have expected by reading about what this is the **[00:29:00 – 00:29:03]** CPU hardware is. **[00:29:03 – 00:29:05]** We may need to brace them for differences **[00:29:05 – 00:29:08]** in performance variability. **[00:29:08 – 00:29:12]** If their experiment requires it, if they're really focused on hardware **[00:29:12 – 00:29:14]** performance, they can opt out. **[00:29:14 – 00:29:18]** That's something that we can arrange. **[00:29:19 – 00:29:24]** And another benefit of these technologies is they can be **[00:29:24 – 00:29:28]** tuned almost live to meet the goals of the center. **[00:29:28 – 00:29:35]** An example of that is that we can adjust the bandwidth available to **[00:29:35 – 00:29:42]** a job at start time, for example, if it is flagged as high priority. **[00:29:45 – 00:29:51]** There are more complications to quality of service when **[00:29:51 – 00:29:56]** co-scheduling is used with the storage system. **[00:29:56 – 00:29:59]** It needs to, there's a **[00:30:00 – 00:30:05]** That's something that we are navigating now, but it's a point of **[00:30:05 – 00:30:07]** interest that we need to resolve. **[00:30:07 – 00:30:10]** And lastly, if you have two different jobs running on **[00:30:10 – 00:30:14]** a given node and the node goes down for whatever reason, that's **[00:30:14 – 00:30:19]** two different potentially disjoint sets of nodes that are affected. **[00:30:19 – 00:30:23]** So we're aware of that. **[00:30:23 – 00:30:27]** We're developing strategies for fault containment. **[00:30:28 – 00:30:34]** So in my next and I believe final slides, I'm just going to **[00:30:34 – 00:30:36]** repeat the highlights of this work. **[00:30:37 – 00:30:40]** One, we've demonstrated that co-scheduling is an effective **[00:30:40 – 00:30:45]** way to increase the throughput of jobs on HPC systems and **[00:30:45 – 00:30:48]** to improve their energy efficiency. **[00:30:48 – 00:30:53]** We're planning to use co-scheduling on the Doudna system that's largely **[00:30:53 – 00:30:57]** based on early measurements that show a 50% increase on throughput **[00:30:57 – 00:31:00]** on a Grace-Hopper system. **[00:31:00 – 00:31:04]** We think that will improve on Vera-Rubin. **[00:31:05 – 00:31:09]** There are available resources within our workload to take **[00:31:09 – 00:31:11]** advantage of co-scheduling. **[00:31:11 – 00:31:15]** And we've identified certain points of interest that need **[00:31:15 – 00:31:21]** to be addressed as we deploy it. **[00:31:21 – 00:31:25]** With that, I thank you for your time and attention, and **[00:31:25 – 00:31:29]** we would welcome your questions. **[00:31:32 – 00:31:33]** Awesome. **[00:31:33 – 00:31:36]** Thank you so much. If you have questions, you can raise **[00:31:36 – 00:31:41]** your hand and I'll come walk around and I've got one here. **[00:31:48 – 00:31:53]** Hi, how was the co-scheduling overhead calculated? **[00:31:53 – 00:32:00]** So what we did, we ran the job in isolation, just on **[00:32:00 – 00:32:04]** the same hardware, running the same job, and just running **[00:32:04 – 00:32:09]** the GPU job, looking at the time it takes to complete that GPU job. **[00:32:10 – 00:32:13]** And then we did the same experiment, but co-scheduling. **[00:32:13 – 00:32:18]** So we ran the, if you look at the GPU job, We ran the GPU job. **[00:32:18 – 00:32:21]** Measure the time it took to execute and in the background **[00:32:21 – 00:32:24]** we made sure that the CPU job is constantly running while we **[00:32:24 – 00:32:28]** measure the time of the GPU job so that we have that constant overlap **[00:32:28 – 00:32:34]** of both jobs and then we just take the ratio of the run times. **[00:32:34 – 00:32:37]** Hi, here. **[00:32:37 – 00:32:41]** Nice presentation. This is Rajesh. **[00:32:41 – 00:32:44]** We have a data center and most of the people are trying to **[00:32:44 – 00:32:49]** execute their LLM models especially and high-end AI applications **[00:32:49 – 00:32:53]** in different domains of science like medical and atmospheric **[00:32:53 – 00:32:57]** sciences and biology, genomics. **[00:32:57 – 00:33:02]** So what kind of solutions you would like to give for efficiently **[00:33:02 – 00:33:06]** running these kind of applications especially in LLMs and AI and **[00:33:06 – 00:33:11]** HPC and CPU and GPU combinations? **[00:33:11 – 00:33:15]** So I think the general question is to understand basic things like **[00:33:15 – 00:33:20]** how many cores do your jobs require and how much are they impacted **[00:33:20 – 00:33:22]** by the cores, how much memory do they need, so that you can **[00:33:22 – 00:33:28]** make sure that the node has enough memory to hold both these jobs. **[00:33:29 – 00:33:33]** And then the tuning of these parameters is basically start **[00:33:33 – 00:33:35]** from something like the baseline that we've provided here. **[00:33:35 – 00:33:41]** And then it's quite easy to just run some experiments **[00:33:41 – 00:33:42]** and tune these parameters. **[00:33:42 – 00:33:45]** You might not get the absolute best performance with that **[00:33:45 – 00:33:49]** starting set of parameters, but you should get good performance. **[00:33:49 – 00:33:53]** There might be some jobs where you just need all of the node **[00:33:53 – 00:33:57]** resources and of course then you should just run them in isolation. **[00:33:57 – 00:34:00]** I don't know the details of your workloads well enough **[00:34:00 – 00:34:05]** to comment from here, but I'm happy to chat with you after the session. **[00:34:08 – 00:34:15]** So, in the combination of CPU and GPU, how MIG and MPS **[00:34:15 – 00:34:18]** can be collaborated together? **[00:34:19 – 00:34:26]** So you mean, you can still assign, so what we're doing is not on the **[00:34:26 – 00:34:31]** GPU level but on the CPU resources that you assign to the GPU job and **[00:34:32 – 00:34:36]** the network resources, so you can assign these jobs even if you have **[00:34:36 – 00:34:38]** multiple jobs running on the GPU. **[00:34:38 – 00:34:42]** You still assign them to CPU cores, and assign for these **[00:34:42 – 00:34:45]** CPU cores the MPAM, and you can also assign a service **[00:34:45 – 00:34:48]** level on the IBQOS for these drops. **[00:34:48 – 00:34:50]** This can be done. **[00:34:51 – 00:34:54]** Hi, thank you for a nice talk. **[00:34:54 – 00:34:58]** Is it correct that the CPUs and GPUs were running different **[00:34:58 – 00:35:01]** workloads and not parts of the same application? **[00:35:01 – 00:35:04]** In this scenario, we were looking at different workloads, yes. **[00:35:04 – 00:35:06]** Completely independent jobs. **[00:35:06 – 00:35:10]** So my real question is, crudely, workloads can be classified **[00:35:10 – 00:35:12]** as compute-bound and memory-bound. **[00:35:12 – 00:35:16]** Do we have any performance characterization for whether **[00:35:16 – 00:35:20]** the GPU side was memory versus compute and bound in combination **[00:35:20 – 00:35:22]** with the CPU, that job being compute and memory bound? **[00:35:22 – 00:35:26]** Did we see any variations in these combinations for **[00:35:26 – 00:35:29]** the metrics that we measured? **[00:35:30 – 00:35:33]** Yes, if you have something that you would characterize as on the **[00:35:33 – 00:35:38]** CPU side, because on the GPU side, there's only one thing running. **[00:35:38 – 00:35:43]** It has access to the full compute and full HBM memory resources. **[00:35:43 – 00:35:47]** On the CPU side, if you run something like OpenFOAM, which **[00:35:47 – 00:35:50]** is really memory bandwidth bound, **[00:35:51 – 00:35:55]** Not quite as pure as just running stream triad, but **[00:35:55 – 00:35:59]** very sensitive to the memory bandwidth that it has available. **[00:35:59 – 00:36:04]** You saw the stronger impact on OpenFOAM when we were having **[00:36:04 – 00:36:09]** other jobs like MILC running, which also uses the impact. **[00:36:09 – 00:36:12]** And so if you have something that just stresses the compute **[00:36:12 – 00:36:15]** resources, but uses very little of the memory bandwidth, of **[00:36:15 – 00:36:20]** course that memory bandwidth is available for the GPU job. **[00:36:20 – 00:36:25]** But then you should already see smaller interference between **[00:36:25 – 00:36:27]** these jobs because they are not fighting for the shared **[00:36:27 – 00:36:31]** resource of memory bandwidth. **[00:36:33 – 00:36:38]** Can you please use the microphone so I can... we get the questions? **[00:36:38 – 00:36:40]** Sorry, just the last part of this question. **[00:36:40 – 00:36:43]** Does the scheduling policy change depending on whether **[00:36:43 – 00:36:47]** the CPU and GPU jobs are compute-bound or memory-bound? **[00:36:48 – 00:36:49]** This is something that... **[00:36:49 – 00:36:53]** So, so far we didn't do a scheduling policy here, we **[00:36:53 – 00:36:56]** were just doing the experiments to understand the impact and **[00:36:56 – 00:36:58]** the mitigation strategies. **[00:36:58 – 00:37:01]** This is something that can be implemented in practice, yes. **[00:37:01 – 00:37:03]** So you could think of... **[00:37:04 – 00:37:06]** The configuration space that you can go in and how you **[00:37:06 – 00:37:13]** want to set up your queues and user incentives is something where we **[00:37:13 – 00:37:18]** suggested a baseline and then you could think of more solutions and **[00:37:18 – 00:37:23]** more fine-grained solutions, but we wanted to keep it simple for now. **[00:37:23 – 00:37:26]** Okay, I think that's all the time we have for questions today. **[00:37:26 – 00:37:29]** I want to encourage you to check out GDC Park and **[00:37:29 – 00:37:30]** the Exhibit Center. **[00:37:30 – 00:37:33]** If you want to speak with our guests, let them do mic, and **[00:37:33 – 00:37:34]** you can meet them over on the side.