# Achieve Truly Serverless GPUs With libfuse, CRIU, and CUDA-Checkpoint ## Abstract Unpredictable workloads make inference a natural fit for serverless computing (a.k.a. "functions-as-a-service"). But serverless execution only works if new function replicas spin up quickly. Learn the techniques we've developed at Modal to speed up the boots of GPU containers and achieve truly serverless execution for workloads running on thousands of GPUs. ## AI Summary - The presentation discussed the challenges of achieving high GPU allocation utilization in serverless environments, particularly for inference applications, which often have unpredictable and variable demand. - The speaker explained that Modal, a platform for managing GPU resources, used a buffer of ready instances to reduce the time required for spinning up new machines, significantly decreasing startup latency. - Modal implemented a custom file system in user space (FUSE) to optimize the loading of container images, using lazy loading and eager prefetching to improve performance and reduce the amount of data that needed to be transferred. - The platform utilized gVisor for checkpointing and restoring Linux processes, allowing applications to skip lengthy initialization steps and start handling requests more quickly. - A key focus was on accelerating the creation and restoration of CUDA contexts, which could take up to 200 seconds, by using CUDA Checkpoint to reduce this to around 30 seconds. - The speaker also highlighted the importance of efficient model weight loading and distribution, suggesting that RDMA could provide the maximum speed for this process, although it was challenging to implement in a dynamic worker pool. ## Transcript **[00:00:12 – 00:00:17]** Good morning, everyone, and yeah, I'm going to tell you a **[00:00:17 – 00:00:23]** bit about the technology stack that we put together at Modal to achieve **[00:00:23 – 00:00:27]** what we call truly serverless GPUs. **[00:00:27 – 00:00:31]** So first, I want to answer the question, why serverless GPUs? **[00:00:31 – 00:00:33]** If you're sitting here or you're watching this, you **[00:00:33 – 00:00:34]** maybe have your own answer. **[00:00:34 – 00:00:36]** I'll give you ours. **[00:00:36 – 00:00:40]** The goal that we have is to help people maximize their **[00:00:40 – 00:00:46]** GPU allocation utilization and sort of match the supply **[00:00:46 – 00:00:49]** of compute that they have, that they're paying for, that they **[00:00:49 – 00:00:55]** have allocated to the demand for compute that their application has. **[00:00:55 – 00:00:57]** So I think a lot of us are probably familiar with GPU **[00:00:57 – 00:00:59]** kernel utilization. **[00:00:59 – 00:01:02]** If you've ever run NVIDIA SMI, you get this little number **[00:01:02 – 00:01:03]** that says GPU utilization. **[00:01:03 – 00:01:06]** It's a number between zero and 100%. **[00:01:06 – 00:01:10]** GPU kernel utilization tells you, like, **[00:01:11 – 00:01:15]** At a given period of time, for how long was a kernel **[00:01:15 – 00:01:16]** running on the device? **[00:01:16 – 00:01:20]** Like, if you have 10 seconds, and during those 10 seconds **[00:01:20 – 00:01:23]** you were running a kernel every time, attention, Jim, rel **[00:01:23 – 00:01:27]** u, then you have 100% utilization. **[00:01:27 – 00:01:30]** If there are some gaps between your kernels, like looks here, **[00:01:30 – 00:01:33]** like about a third of the time a kernel is running, and most of **[00:01:33 – 00:01:38]** the time nothing is running, that's a lower GPU utilization, like 30%. **[00:01:38 – 00:01:40]** And if there's nothing running on the GPU at all, your kernel **[00:01:40 – 00:01:41]** utilization is 0%. **[00:01:41 – 00:01:45]** So this is a common metric, in part because it's really **[00:01:45 – 00:01:46]** easy to get a hold of. **[00:01:46 – 00:01:53]** And it helps you know, are you running code on the GPU? **[00:01:53 – 00:01:57]** So if you've used the PyTorch profiler and site systems, **[00:01:57 – 00:02:02]** they help you debug issues with this kernel utilization metric. **[00:02:03 – 00:02:06]** GPU allocation utilization is like a higher level version **[00:02:06 – 00:02:08]** of this same metric. **[00:02:08 – 00:02:14]** So at a given moment in time, maybe you might have code running on all **[00:02:15 – 00:02:16]** the GPUs that you're paying for. **[00:02:16 – 00:02:19]** That would be 100% GPU allocation utilization. **[00:02:19 – 00:02:22]** And if, for example, if this was a slice, a moment in time **[00:02:22 – 00:02:25]** in my, like, deployment and I was running, you know, this **[00:02:25 – 00:02:27]** was running on one GPU, this application was running on another **[00:02:27 – 00:02:31]** GPU, and then nothing was happening on a third GPU, that would **[00:02:31 – 00:02:37]** be this middle bar here, two-thirds GPU allocation utilization. **[00:02:37 – 00:02:44]** So what you're trying to capture with this is I have paid for this **[00:02:44 – 00:02:47]** hardware, whether you bought it and rack and stacked it, or whether **[00:02:47 – 00:02:49]** you are renting it in a cloud. **[00:02:50 – 00:02:54]** And the question is, are you using the thing that you paid for? **[00:02:54 – 00:02:56]** And, um... **[00:02:56 – 00:03:00]** I'd say, based off of both our conversations with customers **[00:03:00 – 00:03:03]** and on things like the State of AI Infrastructure Report, **[00:03:03 – 00:03:06]** I would say most people tend to operate in somewhere in **[00:03:06 – 00:03:11]** between this regime and this regime, where, like, at peak **[00:03:11 – 00:03:14]** often, actually, I think the median organization in the 2024 State of **[00:03:14 – 00:03:21]** AI Infrastructure Report said that they hit, at peak, 70% utilization **[00:03:21 – 00:03:25]** and, like, quite frequently ran a lot lower than this. **[00:03:26 – 00:03:28]** And one of the reasons for this, I think if you compare **[00:03:28 – 00:03:33]** this to what people frequently talk about, people frequently talk about **[00:03:33 – 00:03:35]** an even lower level utilization than kernel utilization. **[00:03:35 – 00:03:37]** They talk about model flops utilization. **[00:03:37 – 00:03:42]** And they hit numbers like 40% of the flops that they **[00:03:42 – 00:03:44]** are paying for, they're being used. **[00:03:44 – 00:03:46]** So it's maybe surprising to see like, wait, this is so **[00:03:46 – 00:03:46]** much higher level. **[00:03:46 – 00:03:49]** This is like, is the GPU even running code? **[00:03:49 – 00:03:50]** And people are reporting these lower numbers. **[00:03:50 – 00:03:52]** Like, what's going on there? **[00:03:52 – 00:03:54]** The basic answer is that inference is not training. **[00:03:54 – 00:03:57]** Inference is very different from training. **[00:03:57 – 00:04:01]** Inference applications have highly variable and unpredictable **[00:04:01 – 00:04:05]** demand because the demand on the system is driven by essentially **[00:04:05 – 00:04:08]** market forces, by whoever is consuming your inference, **[00:04:09 – 00:04:12]** whether that's what happens with internal teams, but it definitely **[00:04:12 – 00:04:15]** happens with consumer-facing inference applications. **[00:04:15 – 00:04:19]** So, this is a trace of requests per minute over a month from **[00:04:19 – 00:04:22]** one of our customers, and you can see that there's quite **[00:04:22 – 00:04:26]** a bit of variability in the number of requests per minute. **[00:04:26 – 00:04:29]** And some kinds of this variability are relatively predictable, **[00:04:29 – 00:04:34]** like you can probably detect these diurnal daily fluctuations, **[00:04:34 – 00:04:37]** if you squint you can maybe start to see some weekend versus **[00:04:37 – 00:04:39]** This is weekday patterns, but **[00:04:40 – 00:04:42]** A lot of, one of the interesting things about building an **[00:04:42 – 00:04:46]** application, especially in the era of social media, virality **[00:04:46 – 00:04:51]** phenomena, rapid hypergrowth, is that you actually have much, **[00:04:51 – 00:04:57]** even more extremely unpredictable demand over time, where **[00:04:57 – 00:05:00]** you might suddenly double your number of requests per minute, your **[00:05:01 – 00:05:02]** peak requests per minute goes up. **[00:05:02 – 00:05:06]** In this case, it goes up by a factor of two from one day **[00:05:06 – 00:05:09]** to another, and hits a new all-time I'm high. **[00:05:09 – 00:05:13]** And we see a lot of people encountering this problem. **[00:05:13 – 00:05:17]** And so the challenge as engineers of these systems is that we **[00:05:17 – 00:05:21]** need to find a way to efficiently service that demand within **[00:05:21 – 00:05:27]** our service level objectives on latency, for instance. **[00:05:27 – 00:05:33]** So, just to like poke at that a little bit further, fixed **[00:05:33 – 00:05:36]** over-provisioned GPU allocations are kind of the, like this **[00:05:36 – 00:05:38]** is the classic approach, this is kind of the only option **[00:05:38 – 00:05:44]** if you don't have like a cloud that you can run things on. **[00:05:44 – 00:05:47]** So the idea is you predict what your peak demand is gonna be, **[00:05:47 – 00:05:51]** you make a guess, then you go and you like get a hold of those GPUs **[00:05:51 – 00:05:54]** and you sign a long-term cloud. **[00:05:54 – 00:06:01]** You make a guess about capacity, and then most of the time, **[00:06:01 – 00:06:04]** because you have to be provisioned for that peak in order to **[00:06:04 – 00:06:08]** be able to hit your SLO during that key critical moment when **[00:06:08 – 00:06:13]** you're getting a bunch of attention for a big release or a viral **[00:06:13 – 00:06:18]** moment, you're actually sitting there with a bunch of idle GPUs. **[00:06:18 – 00:06:21]** And so there's this big gap between the demand from your **[00:06:21 – 00:06:23]** application and your supply. **[00:06:24 – 00:06:27]** And so you might think, okay, I'll do like automatic allocation. **[00:06:27 – 00:06:32]** I'll just like auto scale up and down and like especially if you're **[00:06:32 – 00:06:36]** building on top of the hyperscalers or neoclouds, this is like **[00:06:36 – 00:06:37]** the immediate thing to think of. **[00:06:37 – 00:06:40]** And it actually, the problem here is that it actually makes **[00:06:40 – 00:06:43]** things worse if it's slow. **[00:06:43 – 00:06:46]** So if you say, oh yeah, whenever request per minute goes up, **[00:06:46 – 00:06:49]** I will provision more GPUs, and then when it goes back **[00:06:49 – 00:06:52]** down, I'll spin them back down, what you end up with is this **[00:06:53 – 00:06:56]** kind of funky, like, convolution. **[00:06:57 – 00:07:02]** If you love linear time and variant systems, raise your hand. **[00:07:02 – 00:07:05]** So you end up with this thing where you actually, you **[00:07:05 – 00:07:09]** You are too late to handle an increase in demand, and **[00:07:09 – 00:07:12]** then you also hold on to the GPUs for too long. **[00:07:12 – 00:07:14]** You need to keep them around, maybe to handle the possibility **[00:07:14 – 00:07:16]** of another increase. **[00:07:16 – 00:07:21]** And you might completely miss some of these faster spikes. **[00:07:21 – 00:07:24]** So the goal that you want, if you want to be able to maximize **[00:07:24 – 00:07:27]** your GPU allocation utilization in the face of this like unpredictable **[00:07:27 – 00:07:32]** demand, is you need really, really fast automated sort of allocation **[00:07:32 – 00:07:35]** and deallocation of GPUs. **[00:07:35 – 00:07:37]** And that's what's depicted here. **[00:07:37 – 00:07:39]** And then depending on whether you're more like cost sensitive **[00:07:39 – 00:07:41]** or whether you're more SLO sensitive, you either sort **[00:07:41 – 00:07:44]** of like over, you know, kind of like aim to over provision a little **[00:07:44 – 00:07:51]** bit and absorb spikes or you under provision in order to save costs. **[00:07:55 – 00:08:00]** In order to make a system like this work, one of the things **[00:08:00 – 00:08:05]** that is a major benefit of getting a system like this working is you **[00:08:05 – 00:08:09]** can sort of move demand in between different tenants of that system. **[00:08:09 – 00:08:14]** And this actually improves the, like, if we improve our **[00:08:14 – 00:08:18]** GPU allocation utilization, like, As individual organizations, **[00:08:18 – 00:08:20]** or as like a, as. **[00:08:20 – 00:08:23]** Cloud providers, cloud platforms, like the one that I work on. **[00:08:23 – 00:08:27]** You actually increase the efficiency of the use of the **[00:08:27 – 00:08:29]** overall GPU supply. **[00:08:29 – 00:08:32]** So if you have a bunch of... So say a bunch of individual **[00:08:32 – 00:08:38]** organizations have to pre-allocate GPUs, and so they pre-allocate **[00:08:38 – 00:08:39]** for their peaks, right? **[00:08:39 – 00:08:41]** They all do it correctly, but they all need to pre-allocate **[00:08:42 – 00:08:44]** for their peaks, and their peaks probably happen at different times. **[00:08:44 – 00:08:45]** People go viral at different moments. **[00:08:45 – 00:08:48]** People have, like... **[00:08:48 – 00:08:53]** Audiences all around the world, peaks are relatively uncorrelated. **[00:08:53 – 00:08:57]** If you can service all of these out of a single GPU **[00:08:57 – 00:09:04]** allocation and you just stack all of these curves on top of **[00:09:04 – 00:09:09]** each other, you end up with a lower total number of GPUs consumed. **[00:09:09 – 00:09:12]** And so, in this current sort of like supply-constrained **[00:09:12 – 00:09:18]** environment where like, yeah, the GPUs can't be built fast enough, **[00:09:18 – 00:09:22]** no matter how hard people try, this sort of increased efficiency **[00:09:22 – 00:09:26]** gets, you know, more flops and more intelligence delivered, **[00:09:26 – 00:09:31]** you know, with the current supply. **[00:09:32 – 00:09:37]** So the platform that I work on that was built to solve **[00:09:37 – 00:09:39]** this problem is called Modal. **[00:09:39 – 00:09:43]** We're running on the order of 10,000, a couple tens of thousands **[00:09:43 – 00:09:46]** of GPUs at any given moment. **[00:09:46 – 00:09:48]** We have a couple of other things like compute **[00:09:48 – 00:09:50]** sandboxes for AI agents. **[00:09:50 – 00:09:54]** We aggregate this capacity off of a large number of clouds, **[00:09:54 – 00:10:03]** and we built this multi-tenant Rapid GPU allocation system. **[00:10:03 – 00:10:07]** And what we found is that in order to get that speed up, in order **[00:10:07 – 00:10:11]** to get that nice, clean, green and pink curves on top of each other, **[00:10:11 – 00:10:15]** we had to build a lot of this stuff ourselves and not use the kind **[00:10:15 – 00:10:19]** of standard tools that people have for managing these kinds of things. **[00:10:19 – 00:10:20]** Like the problems that I've, I just gave you a problem **[00:10:20 – 00:10:24]** that like, honestly, it's like the 1920s newsboy problem. **[00:10:24 – 00:10:27]** It's like, this is a problem that shows up all the time in sort of **[00:10:27 – 00:10:29]** like supply and demand in systems. **[00:10:29 – 00:10:33]** And so obviously it has shown up in computing systems. **[00:10:33 – 00:10:36]** You may not realize it if you spend a lot of time on **[00:10:36 – 00:10:38]** Twitter, but actually computers existed before artificial **[00:10:38 – 00:10:40]** intelligence, it's crazy. **[00:10:41 – 00:10:44]** And in that time, people built a lot of tools, whether **[00:10:44 – 00:10:48]** it's like Slurm for managing jobs on a cluster. **[00:10:49 – 00:10:51]** Docker for, like, packaging up applications and serving **[00:10:51 – 00:10:56]** them, Kubernetes for creating sort of, like, high variability **[00:10:56 – 00:11:00]** and high availability and resilient applications, but **[00:11:00 – 00:11:03]** we found that they, like, weren't built, architected in **[00:11:03 – 00:11:07]** the right way for these particular, like, performance considerations. **[00:11:07 – 00:11:10]** So I'm going to walk you through the other things **[00:11:10 – 00:11:13]** that we built on top of sort of lower level primitives, **[00:11:13 – 00:11:18]** the ones in the title of the talk that helped us deliver this. **[00:11:18 – 00:11:21]** So I'm going to talk about how we essentially optimize **[00:11:21 – 00:11:26]** the startup latency of applications running on GPUs. **[00:11:26 – 00:11:33]** And the basic principles that show up here are Hanson and **[00:11:33 – 00:11:37]** Crane's performance mantras from Brendan Gregg's book. **[00:11:37 – 00:11:41]** Don't do it, do it, but never do it again, and do it when **[00:11:41 – 00:11:42]** they're not looking. **[00:11:42 – 00:11:46]** So we'll see each of these come into play. **[00:11:46 – 00:11:51]** All right, so I wanted to walk folks through this diagram **[00:11:51 – 00:11:55]** that I'm going to return to over and over again. **[00:11:55 – 00:11:58]** So on the right, this is a picture of a computer. **[00:11:58 – 00:12:01]** So you have a CPU and its memory. **[00:12:01 – 00:12:04]** You have a GPU and its memory. **[00:12:04 – 00:12:08]** You have some sort of disk or durable storage. **[00:12:08 – 00:12:10]** And then you're communicating out. **[00:12:10 – 00:12:12]** In this case, we're assuming like cloud deployment. **[00:12:12 – 00:12:14]** That's what we. **[00:12:14 – 00:12:15]** That's how we do things. **[00:12:15 – 00:12:18]** So we're assuming some kind of like external storage that **[00:12:18 – 00:12:22]** is, you know, sort of far away but very capacious. **[00:12:22 – 00:12:23]** And that's this guy up here. **[00:12:23 – 00:12:24]** So we've got all these things together. **[00:12:24 – 00:12:28]** And I'm going to use these, like keep these colors consistent, **[00:12:28 – 00:12:32]** refer back to them so we can see as it shows up in all **[00:12:32 – 00:12:33]** the different things we talk about. **[00:12:33 – 00:12:35]** So that's this guy over here. **[00:12:35 – 00:12:38]** And then we are going to try and speed up. **[00:12:39 – 00:12:45]** This about two-kilosecond process of getting an AI application or **[00:12:45 – 00:12:51]** an inference application running in the cloud, so from zero to running. **[00:12:51 – 00:12:55]** And so we've observed that basically, yeah, it takes, **[00:12:55 – 00:12:59]** there's about 30 minutes of setup that's just sort of like raw. **[00:12:59 – 00:13:01]** Like, allocation of machine, booting of an operating **[00:13:01 – 00:13:04]** system, and then there's some application-specific **[00:13:04 – 00:13:07]** stuff that involves disk state, application-specific stuff **[00:13:07 – 00:13:11]** that involves CPU and memory state, and then application-specific **[00:13:11 – 00:13:13]** stuff that involves GPU state. **[00:13:13 – 00:13:15]** I'm going to talk through how we sped up each of **[00:13:15 – 00:13:19]** these to get this down to a twentieth of a kilosecond, **[00:13:19 – 00:13:22]** or a little under a minute. **[00:13:22 – 00:13:26]** And just to clarify, this is meant to be somewhat typical. **[00:13:26 – 00:13:29]** I'll give very specific examples and benchmarks **[00:13:29 – 00:13:30]** at the end of the talk. **[00:13:30 – 00:13:34]** But you can imagine this as spinning up a vLLM server or **[00:13:34 – 00:13:39]** an sglang server for a medium-sized language model or spinning **[00:13:39 – 00:13:48]** up a diffusion-generative model for audio or images or something. **[00:13:48 – 00:13:51]** That's kind of a typical application on our platform **[00:13:51 – 00:13:54]** And what I have in mind here. **[00:13:55 – 00:14:00]** All right, so what's going on, essentially, the four steps **[00:14:00 – 00:14:04]** that I have in that diagram are basically spin up a new instance, **[00:14:04 – 00:14:09]** eagerly load the file system into this state that it needs **[00:14:09 – 00:14:13]** to be for that application, start the application on the host side, **[00:14:13 – 00:14:17]** and then start the application on the device, like construct. **[00:14:17 – 00:14:19]** Device-side resources. **[00:14:19 – 00:14:24]** And so the TLDR of the way that we sped this up is to **[00:14:24 – 00:14:26]** switch from spinning up new instances to maintaining a **[00:14:26 – 00:14:31]** buffer of ready instances, lazy loading the file system and sort **[00:14:31 – 00:14:37]** of like a mix of eager prefetching and lazy loading, restoring Linux **[00:14:37 – 00:14:42]** processes instead of like starting applications, and then restoring **[00:14:42 – 00:14:46]** CUDA contexts instead of starting applications on the device. **[00:14:46 – 00:14:51]** So we'll go through all of these in detail through the talk. **[00:14:51 – 00:14:59]** So first, the biggest win here is adding a buffer to **[00:14:59 – 00:15:02]** handle instance spin-up. **[00:15:02 – 00:15:05]** So, yeah, we're going to go through and look at this little **[00:15:05 – 00:15:08]** block and ask repeatedly, like, what is going on in this block? **[00:15:08 – 00:15:09]** Why is it slow? **[00:15:09 – 00:15:11]** How can we make it faster? **[00:15:11 – 00:15:14]** So this big gray block here represents the period during **[00:15:14 – 00:15:18]** which a machine is allocated from the, like, cloud supply, **[00:15:18 – 00:15:22]** and an operating system is, like, booted and brought into the, **[00:15:22 – 00:15:26]** like, ready state on that machine. **[00:15:26 – 00:15:31]** So I'd say, why does this take such a long time? **[00:15:31 – 00:15:33]** There's a couple of pieces here. **[00:15:33 – 00:15:39]** One is that, especially if you're looking at a single cloud, **[00:15:39 – 00:15:42]** if you're looking at a single region or single availability **[00:15:42 – 00:15:46]** zone within a single cloud, and then you have a single specific **[00:15:46 – 00:15:50]** GPU type that you want to get a hold of, you end up with a lot of **[00:15:50 – 00:15:54]** tail latency on getting a hold of instances because there **[00:15:54 – 00:15:58]** might just not be any available if you're operating in like the spot **[00:15:58 – 00:16:01]** market or the on-demand market. **[00:16:01 – 00:16:03]** So there's like that tail latency and you might have **[00:16:03 – 00:16:06]** like a sort of like search process if you go multi- **[00:16:06 – 00:16:10]** AZ, multi-region, multi-cloud, now you have a search process **[00:16:10 – 00:16:11]** to find those things. **[00:16:11 – 00:16:14]** And if you do this at scale or for long enough, you'll **[00:16:14 – 00:16:19]** discover that the APIs are at best eventually consistent **[00:16:19 – 00:16:23]** and perhaps not consistent with the underlying actual supply. **[00:16:23 – 00:16:24]** And so there's quite a bit of failure. **[00:16:24 – 00:16:28]** So that adds a bunch of latency, operational complexity, and **[00:16:28 – 00:16:32]** then you have getting a proper machine image onto that machine, **[00:16:32 – 00:16:33]** booting the operating system. **[00:16:33 – 00:16:36]** So there's a bunch of stuff that shows up, slows you down. **[00:16:36 – 00:16:40]** Solution is to remove instance start from the hot path by running **[00:16:40 – 00:16:46]** a buffer where you've done all of that work to have a machine that **[00:16:46 – 00:16:52]** is ready to run an application, but it's not running anything yet. **[00:16:53 – 00:16:56]** So maybe you have eight GPUs of a particular type in a **[00:16:56 – 00:16:58]** particular region. **[00:16:58 – 00:17:00]** Five out of eight of them are running applications. **[00:17:00 – 00:17:03]** Three out of eight of them are not being used. **[00:17:03 – 00:17:06]** Key point, this does mean that you are not ever going **[00:17:06 – 00:17:11]** to hit 100% utilization if you operate with a buffer. **[00:17:11 – 00:17:14]** I think for the vast majority of applications, especially **[00:17:14 – 00:17:17]** inference applications, 100% utilization is a mirage. **[00:17:17 – 00:17:18]** You're never gonna hit it. **[00:17:18 – 00:17:22]** Buffers are required for any number of reasons, including **[00:17:22 – 00:17:24]** reliability and robustness. **[00:17:24 – 00:17:27]** And so there are a variety of operational **[00:17:27 – 00:17:31]** wins from having a buffer. **[00:17:31 – 00:17:38]** And then if you have this faster startup, then the application **[00:17:38 – 00:17:44]** demands mean that you can operate at the utilization of everything **[00:17:44 – 00:17:46]** except the buffer all the time. **[00:17:46 – 00:17:50]** And so that might get you up to 80%, 90% utilization. **[00:17:50 – 00:17:53]** At least in our experience, many of our users operate **[00:17:53 – 00:17:58]** at over 90% GPU allocation utilization thanks to this buffer. **[00:17:58 – 00:18:02]** So the idea is that if a request comes in and you want to run **[00:18:02 – 00:18:07]** it, you, represented by, oops, I changed the colors on this **[00:18:07 – 00:18:10]** one, so six out of eight of these GPUs are being used **[00:18:10 – 00:18:14]** because a user request come in and then we see our buffer's gone down **[00:18:14 – 00:18:17]** to two, we wanna maintain a buffer of like three or four in this **[00:18:17 – 00:18:22]** case, so we spin up some more GPUs. **[00:18:22 – 00:18:30]** In this case, I think the long-term play with this is you have **[00:18:30 – 00:18:35]** the smallest scale that you ever observe running in reservations **[00:18:35 – 00:18:40]** in the cloud providers or on-prem, and then you operate the buffer **[00:18:40 – 00:18:45]** out of spot if your application can handle it, or on-demand capacity. **[00:18:45 – 00:18:47]** And yeah. **[00:18:47 – 00:18:52]** Okay, so that process will take something that takes **[00:18:52 – 00:19:00]** about two kiloseconds, about 40 minutes, down to 0.3 kiloseconds **[00:19:00 – 00:19:00]** by pulling that out. **[00:19:00 – 00:19:04]** Basically, it now becomes a couple database lookups **[00:19:04 – 00:19:07]** or replacing the process of starting a machine, so it **[00:19:07 – 00:19:09]** shrinks down to nearly invisible. **[00:19:09 – 00:19:12]** So I'm gonna expand that out to 300 seconds because we're gonna **[00:19:12 – 00:19:17]** spend the rest of the time talking about how to speed this bit up. **[00:19:17 – 00:19:26]** OK, so now we've got that first high pole in the tent handled. **[00:19:26 – 00:19:27]** Oh yeah, sorry. **[00:19:27 – 00:19:31]** One of the key components to make this work really well **[00:19:31 – 00:19:33]** is actually health checking. **[00:19:33 – 00:19:39]** So GPUs operate generally at the limits of physics. **[00:19:39 – 00:19:43]** They're operating at their TDP, their **[00:19:43 – 00:19:48]** Their draw power, they are, they're also like, yeah, it's **[00:19:48 – 00:19:52]** actually, it's pretty hard to get like a petaflop of, **[00:19:52 – 00:19:56]** a petaflop per second running, and so these things like fail **[00:19:56 – 00:19:58]** at a pretty high rate, especially if you're used to operating **[00:19:58 – 00:20:02]** like a CPU or a disk system. **[00:20:02 – 00:20:06]** Or where those are the components whose reliability and performance **[00:20:06 – 00:20:07]** you're most concerned with. **[00:20:07 – 00:20:12]** And so we ended up having to build quite a bit to make **[00:20:12 – 00:20:16]** sure that the things in the buffer are ready to run applications, **[00:20:16 – 00:20:19]** to handle when one of the **[00:20:19 – 00:20:22]** GPUs that is running an application fails, and then you need to **[00:20:22 – 00:20:24]** start it back up again. **[00:20:24 – 00:20:28]** We ended up needing to build quite a bit of both active **[00:20:28 – 00:20:32]** pre-flight and then passive in-flight health checking. **[00:20:32 – 00:20:35]** And we talked a lot about how we did this in this blog **[00:20:35 – 00:20:39]** post and talked a little bit anonymized about our experiences **[00:20:39 – 00:20:43]** with different clouds and how well they do on this reliability stuff. **[00:20:43 – 00:20:47]** This is really important for making that sort of **[00:20:47 – 00:20:51]** A pool of instances like reliable and actually something you **[00:20:51 – 00:20:56]** can schedule work on to and plan work around. **[00:20:56 – 00:20:57]** Okay, so that's our big step. **[00:20:57 – 00:21:04]** Next step, we need to get the file system into the right state. **[00:21:05 – 00:21:11]** So during this next big chunk of application startup, a **[00:21:11 – 00:21:15]** container is created inside of the allocated machine. **[00:21:16 – 00:21:19]** So you spin up a container. **[00:21:19 – 00:21:23]** And a container image, in the end, so the thing that's **[00:21:23 – 00:21:27]** kind of your bottleneck here in a cloud deployment or in **[00:21:27 – 00:21:32]** a distributed deployment is actually the container image, the **[00:21:32 – 00:21:35]** file system state of the container. **[00:21:35 – 00:21:38]** So the container image is essentially like a root file **[00:21:38 – 00:21:41]** system that is designed to be sort of like packaged and run **[00:21:41 – 00:21:43]** inside of another root file system. **[00:21:43 – 00:21:45]** So if you were to look on like a host running some containers, **[00:21:45 – 00:21:47]** you would see something like, yeah, root file system up **[00:21:47 – 00:21:50]** here and then somewhere down there another sort of place where **[00:21:50 – 00:21:54]** there's another root file system that has like, you know, nothing **[00:21:54 – 00:21:57]** but the distro of the operating system and the application code. **[00:21:57 – 00:22:03]** Obviously, a full container runtime is more than that, but you need **[00:22:03 – 00:22:08]** permissions, resources, models, but the container image itself is what **[00:22:08 – 00:22:12]** slows you down, what takes a lot of time in getting an application **[00:22:12 – 00:22:15]** running on a cloud machine. **[00:22:15 – 00:22:19]** So that's what we're going to focus on for the performance questions. **[00:22:19 – 00:22:25]** So the key problem with traditional container runtimes, like Docker, **[00:22:25 – 00:22:29]** is that they load this file system eagerly, layer by layer. **[00:22:29 – 00:22:33]** So as you construct a Docker image with line by line in **[00:22:33 – 00:22:39]** a Docker file, you basically create this stack of TAR archives. **[00:22:39 – 00:22:43]** That then in order for the application to start, even **[00:22:43 – 00:22:47]** if it just wants to print like hello world, you need to load **[00:22:47 – 00:22:54]** every single block in that file system needs to be loaded in order **[00:22:54 – 00:22:57]** to create like this file system. **[00:22:57 – 00:23:01]** So in a cloud deployment, that's coming from your object **[00:23:01 – 00:23:05]** storage, and the speed at which you can do this is limited. **[00:23:05 – 00:23:10]** It also, if you zip it with gzip, you're limited to about 100 **[00:23:10 – 00:23:15]** megabytes a second of throughput by the single-threaded nature of gzip. **[00:23:15 – 00:23:17]** And the reason why this is a bad idea has been known for, **[00:23:17 – 00:23:19]** like, about a decade, actually. **[00:23:19 – 00:23:22]** There's this great Usenix paper, Slacker, that they **[00:23:23 – 00:23:27]** just measured, like, for a wide variety of repositories, **[00:23:27 – 00:23:30]** like, how big is the... **[00:23:31 – 00:23:34]** How big is this thing that we were reading? **[00:23:34 – 00:23:36]** And then compressed and uncompressed. **[00:23:36 – 00:23:40]** And then how much do we actually read when the application runs? **[00:23:40 – 00:23:46]** And it turns out you end up reading only tens of megabytes **[00:23:46 – 00:23:51]** out of a gigabyte-size file system in many cases. **[00:23:51 – 00:23:54]** For this HelloBench benchmark, we've also seen this for our **[00:23:54 – 00:23:57]** user code running in production. **[00:23:57 – 00:24:02]** So we wrote this custom file system in user space, or FUSE, **[00:24:02 – 00:24:07]** built with libfuse to load container images asynchronously. **[00:24:07 – 00:24:12]** So the basic idea is that when the container starts, you **[00:24:13 – 00:24:16]** When it's time to load the container image, you basically **[00:24:16 – 00:24:20]** mount a virtual file system saying, oh, yeah, all the files **[00:24:20 – 00:24:23]** are here, use the application, go ahead and get started. **[00:24:23 – 00:24:26]** And then in the background, you start fetching the files **[00:24:26 – 00:24:29]** that you think the application is going to need. **[00:24:29 – 00:24:30]** So some of this is heuristics. **[00:24:30 – 00:24:33]** It's like people are probably going to need libc. **[00:24:33 – 00:24:35]** Hard to write an application without libc, even Rust goes **[00:24:35 – 00:24:35]** through libc, right? **[00:24:36 – 00:24:39]** HelloBench, libc, gzip, containers. **[00:24:40 – 00:24:43]** We run a lot of Python, people are probably going to need **[00:24:43 – 00:24:44]** the Python that they installed. **[00:24:44 – 00:24:49]** People construct their Docker images on our platform, that **[00:24:49 – 00:24:50]** gives us some signal. **[00:24:50 – 00:24:55]** So that's why we try to load that first. **[00:24:55 – 00:24:58]** And then if there are requests that come in later that are things **[00:24:58 – 00:25:07]** that we haven't loaded, we'll load those when they are requested. **[00:25:08 – 00:25:13]** So, the key thing with FUSE, that's file system in user space, **[00:25:13 – 00:25:16]** the idea here is that you get to the actual kernel space operations **[00:25:16 – 00:25:20]** that make your file system run, are in this FUSE module, **[00:25:20 – 00:25:22]** and then you integrate with it. **[00:25:22 – 00:25:24]** So, the user, the like, **[00:25:24 – 00:25:27]** Application code just issues normal reads and writes. **[00:25:27 – 00:25:30]** They have no idea that there's anything else going on. **[00:25:30 – 00:25:33]** And the Fuse module sort of hands it over to your custom **[00:25:33 – 00:25:38]** file system that's running in user space in a different process. **[00:25:38 – 00:25:42]** And then you sort of issue writes against a specific file, and that **[00:25:42 – 00:25:47]** then goes back to the requester. **[00:25:47 – 00:25:51]** And so the big win here is like sort of like engineering **[00:25:51 – 00:25:54]** velocity and like, you know, you're integrate in in our **[00:25:54 – 00:25:57]** case We're like integrating with a wide variety of clouds that offer **[00:25:57 – 00:25:59]** a wide variety of storage systems We need to be able to extend **[00:25:59 – 00:26:04]** this file system really easily and then in general for the for a lot **[00:26:04 – 00:26:09]** of container images you are going to be issuing a Small number of **[00:26:09 – 00:26:14]** large reads and so the throughput of the file system matters most **[00:26:14 – 00:26:17]** We're able to achieve acceptable throughput with Fuse. **[00:26:17 – 00:26:20]** You do have this... People who've worked on file systems **[00:26:20 – 00:26:24]** are probably well aware, or their ears are twitching. **[00:26:24 – 00:26:28]** It's like you have double the context switches going into **[00:26:28 – 00:26:30]** kernel space, user space, back. **[00:26:30 – 00:26:33]** So that has a penalty on latency, so you wouldn't want to use **[00:26:33 – 00:26:38]** this in many small reads and writes type situations, like a **[00:26:38 – 00:26:41]** character device file or something. **[00:26:41 – 00:26:44]** But for container images, this is great. **[00:26:45 – 00:26:46]** So what's the benefit of doing this? **[00:26:46 – 00:26:51]** You can take this 300-second container start and shave **[00:26:51 – 00:26:54]** off, like, 100 seconds off of it. **[00:26:54 – 00:26:57]** And so that's a pretty big win. **[00:26:57 – 00:27:01]** The key performance component that makes this work, maybe we're, **[00:27:01 – 00:27:03]** like, looking really closely **[00:27:03 – 00:27:05]** In the previous diagram, you were like, wait a second, **[00:27:05 – 00:27:08]** now I have to wait to go to object storage every time I want a file? **[00:27:08 – 00:27:10]** Like, that's pretty bad. **[00:27:10 – 00:27:15]** So good news is that you don't, because not only are most **[00:27:15 – 00:27:19]** files in most container images not read, there's actually a tremendous **[00:27:19 – 00:27:20]** amount of sharing, right? **[00:27:20 – 00:27:24]** People build on top of many of the same open source. **[00:27:24 – 00:27:29]** Software tools, so they build on top of libc, they use Python **[00:27:29 – 00:27:31]** in a small number of versions. **[00:27:31 – 00:27:38]** So we maintain a cache on a variety of different layers in the **[00:27:38 – 00:27:42]** stack that is content addressed. **[00:27:42 – 00:27:45]** Talk-level content addressed. **[00:27:46 – 00:27:51]** And so the key idea here is that if you use a, people have tried to do **[00:27:51 – 00:27:56]** this with like a more Docker-style container serving, and the problem **[00:27:56 – 00:27:59]** is that the natural way to cache things in Docker is layer-wise. **[00:28:00 – 00:28:03]** And there are shared layers between **[00:28:05 – 00:28:13]** But two people using Python, one might go through the super common **[00:28:13 – 00:28:15]** images put out by the Python team. **[00:28:15 – 00:28:19]** Other people might, VLM or SGLang might decide to package **[00:28:19 – 00:28:21]** it differently, and now you've got completely different layers, **[00:28:21 – 00:28:25]** even though the contents of the files are identical. **[00:28:25 – 00:28:29]** And so this leads to a much, much higher cache hit rate. **[00:28:29 – 00:28:30]** And much, much less. **[00:28:31 – 00:28:35]** Which comes essentially from needing to maintain a much **[00:28:35 – 00:28:40]** smaller cache, because all you need is one copy of each of these files. **[00:28:40 – 00:28:45]** And then building that cache, we're deploying onto the cloud, **[00:28:45 – 00:28:50]** so it's sort of architected for the cloud, just like caches **[00:28:50 – 00:28:53]** that people operate for a database are architected for **[00:28:53 – 00:28:58]** the tiered CPU caches and memory. **[00:28:58 – 00:29:02]** So there's a couple of components that really matter, like for **[00:29:02 – 00:29:04]** ultra-low latency, so we want to make sure that people's, **[00:29:04 – 00:29:07]** you know, Python code starts running really quickly. **[00:29:07 – 00:29:09]** We want, yeah, all, like, libc and those standard libraries **[00:29:09 – 00:29:11]** available, we would try and make sure those are in the **[00:29:11 – 00:29:16]** page cache in memory, and they're generally for, like, you know, **[00:29:17 – 00:29:22]** However, we run for an extended period, so they show up in **[00:29:22 – 00:29:23]** that page cache pretty quickly. **[00:29:24 – 00:29:26]** And that gives you very, very short read latencies in the **[00:29:26 – 00:29:30]** nanosecond scale and gives you really high throughput, so you **[00:29:30 – 00:29:33]** can both handle super low latency things and super high throughput. **[00:29:33 – 00:29:36]** Of course, page cache is relatively small and it's relatively **[00:29:36 – 00:29:40]** expensive, so you have to lean on other things. **[00:29:40 – 00:29:44]** If you're used to working with hard drives or have a **[00:29:44 – 00:29:47]** model of disks that many people have that come from hard drives, it **[00:29:47 – 00:29:52]** may be surprising how low the read latency actually can be for SSDs. **[00:29:52 – 00:29:55]** And you can get pretty high capacity from most cloud **[00:29:55 – 00:29:57]** providers with pretty high throughput in the scale **[00:29:57 – 00:30:00]** of multiple gigabytes a second. **[00:30:00 – 00:30:03]** So we stuff those caches very, very full. **[00:30:04 – 00:30:08]** And then there's kind of like a tier of things outside of **[00:30:08 – 00:30:14]** that given by the cloud providers, caches within specific AZs that can **[00:30:14 – 00:30:18]** give you relative, like it's still an order of magnitude jump in your **[00:30:18 – 00:30:20]** read latency up to a millisecond, but you get actually really **[00:30:20 – 00:30:22]** incredible throughput from these. **[00:30:22 – 00:30:24]** So this is a great place to store some of the heftier **[00:30:25 – 00:30:28]** things that go in your container their image, like... **[00:30:29 – 00:30:33]** Maybe model weights or something like that. **[00:30:33 – 00:30:39]** But then for the actual capacity, the long tail of things, you **[00:30:39 – 00:30:42]** need to go out to things like blob storage or a regional CDN. **[00:30:42 – 00:30:44]** You want to make sure as much as possible that you beat **[00:30:44 – 00:30:49]** these out of the common paths so that they show up only in tail **[00:30:49 – 00:30:53]** latencies or for the large-scale work that the application **[00:30:53 – 00:30:55]** is doing, like loading up. **[00:30:55 – 00:31:00]** Loading a model or loading some other sort of like large artifact. **[00:31:01 – 00:31:06]** Okay, so the key performance detail is handling the like **[00:31:06 – 00:31:09]** read-ahead tuning in libfuse. **[00:31:09 – 00:31:12]** So we found the default was too small, the max was far **[00:31:12 – 00:31:14]** too big, and there was something sort of in the middle that **[00:31:14 – 00:31:16]** worked for most of our workloads. **[00:31:17 – 00:31:19]** I already talked a little bit about gzip, so I'm going **[00:31:19 – 00:31:22]** to skip ahead in the interest of time and start talking **[00:31:22 – 00:31:27]** about the sort of snapshot restore piece on hosts and on device. **[00:31:27 – 00:31:32]** So when we get to this stage, user code is creating processes **[00:31:32 – 00:31:35]** inside the container, and then once they're done setting **[00:31:35 – 00:31:39]** up those processes, they're ready to start handling requests. **[00:31:39 – 00:31:42]** In the end, a process is a data structure. **[00:31:42 – 00:31:44]** It's an instance of a program in flight, but it's also **[00:31:44 – 00:31:45]** a data structure. **[00:31:45 – 00:31:50]** There's stack, heap, registers, some file descriptors. **[00:31:50 – 00:31:54]** If you can get a hold of this state at the end of an application **[00:31:54 – 00:31:57]** starting, put it somewhere, and then restore it, you can **[00:31:57 – 00:31:58]** skip a bunch of work. **[00:31:58 – 00:32:03]** So toss that up in this storage system, and you can skip this **[00:32:03 – 00:32:07]** work and turn it into this tiny sliver of pulling something **[00:32:07 – 00:32:10]** from your storage. **[00:32:11 – 00:32:13]** And the key thing here, we're thinking about container startups, **[00:32:13 – 00:32:17]** so you only need this sort of like checkpoint restore at startup. **[00:32:17 – 00:32:21]** So in our system, we sort of define a container lifecycle and say, **[00:32:21 – 00:32:23]** oh, yeah, do this at the beginning. **[00:32:23 – 00:32:26]** And this is the stuff that happens before I'm ready to handle **[00:32:26 – 00:32:29]** requests that I can snapshot. **[00:32:29 – 00:32:30]** This is the stuff that happens before I'm ready to handle **[00:32:31 – 00:32:35]** requests that can't be snapshot, can't be stored at disk. **[00:32:35 – 00:32:38]** And then this is how I handle. **[00:32:38 – 00:32:41]** We don't have to go full transparent checkpoint restore, **[00:32:41 – 00:32:44]** but if you want to do migration and other ways to increase **[00:32:44 – 00:32:47]** your utilization of, say, **[00:32:47 – 00:32:51]** automatic migration to improve the efficiency of your buffered **[00:32:51 – 00:32:55]** allocation, that is also helpful. **[00:32:55 – 00:32:57]** We use gVisor for this. **[00:32:57 – 00:32:59]** It has a built-in checkpoint restore system. **[00:32:59 – 00:33:04]** You basically produce this big set of pages, like pages **[00:33:05 – 00:33:06]** from virtual memory. **[00:33:06 – 00:33:10]** You store that somewhere and then restore it. **[00:33:10 – 00:33:12]** For running on GPUs, an important piece here is this component **[00:33:12 – 00:33:17]** of gVisor called NVProxy It handles communication with **[00:33:17 – 00:33:19]** But otherwise, gVisor basically just offers **[00:33:19 – 00:33:23]** a narrower syscall interface to the user application, sort **[00:33:23 – 00:33:29]** of like a tiny, like a mini kernel sort of running in user space. **[00:33:29 – 00:33:32]** The benefit of using gVisor with checkpoint restore is **[00:33:32 – 00:33:36]** that because gVisor is sort of like modern thing written in **[00:33:36 – 00:33:40]** Go with a lot of like async await style concurrency, it's essentially **[00:33:40 – 00:33:42]** already a state machine, and state machines are really **[00:33:42 – 00:33:44]** easy to checkpoint and restore. **[00:33:44 – 00:33:46]** The thing that makes, you know, something like a Linux **[00:33:46 – 00:33:49]** process generally hard to check when to restore is because, **[00:33:49 – 00:33:51]** in principle, it's a state machine. **[00:33:51 – 00:33:55]** In practice, it's really complicated. **[00:33:55 – 00:34:00]** So the key performance target here is restoring this pages.image file. **[00:34:00 – 00:34:04]** The exact same system that we used for container images works here. **[00:34:04 – 00:34:07]** Like the performance characteristics are very similar. **[00:34:07 – 00:34:12]** So lazy loading and eager prefetching. **[00:34:12 – 00:34:15]** So that takes something that takes 200 seconds to something **[00:34:15 – 00:34:16]** that takes about 200 seconds. **[00:34:16 – 00:34:20]** This is not the high pole in the tent, it's just a necessary step. **[00:34:20 – 00:34:22]** You do speed up some micro benchmarks and some like very **[00:34:22 – 00:34:26]** short applications, useful for sort of cloud native development, **[00:34:26 – 00:34:30]** but it doesn't get you a big performance win on its own. **[00:34:30 – 00:34:33]** The key thing is that this unlocks the ability to do the same kind **[00:34:33 – 00:34:36]** of checkpointing and restoring with things that happen on the device. **[00:34:36 – 00:34:41]** So during this phase, user code is creating CUDA resources **[00:34:41 – 00:34:43]** inside a context on the device. **[00:34:43 – 00:34:45]** And this is now the part. **[00:34:45 – 00:34:46]** This is the high tent in the pole. **[00:34:46 – 00:34:47]** This is what we want to accelerate. **[00:34:47 – 00:34:48]** And the key idea is the same. **[00:34:49 – 00:34:51]** We want to store the data created inside of the CUDA **[00:34:51 – 00:34:57]** context and then restore it later so that we speed things up. **[00:34:57 – 00:35:01]** And there's a tool, the underlying tool driver functionality provided **[00:35:01 – 00:35:03]** for this is called CUDA Checkpoint. **[00:35:03 – 00:35:08]** There's both like a driver API and a command line tool. **[00:35:08 – 00:35:10]** And the key idea is that you run CUDA Checkpoint and that **[00:35:10 – 00:35:13]** takes stuff that's in a context on the device and puts it **[00:35:13 – 00:35:15]** on the host's memory. **[00:35:15 – 00:35:18]** And then you can run a C-checkpoint or CRIU, **[00:35:18 – 00:35:22]** whatever your checkpointing tool is, to put it on disk. **[00:35:22 – 00:35:25]** Then you can restore back to memory when it's time to use it. **[00:35:25 – 00:35:29]** And then CUDA checkpoint it, like in reverse, CUDA checkpoint **[00:35:29 – 00:35:33]** restore onto the device. **[00:35:33 – 00:35:36]** And that can take something that takes 200 seconds and **[00:35:36 – 00:35:39]** get it down to 30 seconds. **[00:35:39 – 00:35:44]** So, proof of principle, skipping a sleep, so let's say this is like **[00:35:44 – 00:35:46]** the ideal snapshottable program. **[00:35:46 – 00:35:50]** When it starts, it sleeps for some number of seconds **[00:35:50 – 00:35:51]** and doesn't do anything else. **[00:35:51 – 00:35:56]** So all that does is sort of like idle, and so you can **[00:35:56 – 00:35:59]** just sort of fast forward through this with a snapshot. **[00:35:59 – 00:36:02]** So that's depicted here. This is the non-snapshot program, takes **[00:36:02 – 00:36:06]** a little over 10 seconds, and then the snapshot program takes under **[00:36:06 – 00:36:10]** 10 seconds for something with a delay, in this case, of 10 seconds. **[00:36:10 – 00:36:12]** So you can just skip right by it. **[00:36:12 – 00:36:15]** So the speedup here is unbounded, which is very, very exciting. **[00:36:15 – 00:36:19]** You don't often find unbounded potential speedups. **[00:36:19 – 00:36:22]** So what is something that people do that looks like **[00:36:22 – 00:36:23]** a sleep but is actually useful? **[00:36:23 – 00:36:27]** So the Torch compiler takes a very long time. **[00:36:27 – 00:36:30]** It produces some state, both on, it requires access to **[00:36:30 – 00:36:33]** a device, produces state on the device, and produces a bunch of **[00:36:33 – 00:36:37]** difficult to serialize artifacts, and you can just skip right over **[00:36:37 – 00:36:39]** it with this checkpoint restore. **[00:36:39 – 00:36:43]** So you can skip through Torch compilation directly, so you **[00:36:43 – 00:36:47]** can see you speed up something that takes like over a minute to **[00:36:47 – 00:36:51]** compile even this relatively small like old-school DenseNet model. **[00:36:51 – 00:36:54]** You can skip that and start in like a few seconds. **[00:36:54 – 00:36:56]** And I'm showing you like this goes all the way out by the **[00:36:56 – 00:36:58]** way to like the fourth nine here. **[00:36:58 – 00:37:02]** So I'm showing you like this is end-to-end including like machine **[00:37:02 – 00:37:03]** allocation all these other things. **[00:37:03 – 00:37:06]** So that's why it extends all the way out this far. **[00:37:06 – 00:37:08]** Wanted to give you like the full sense of like if you're **[00:37:08 – 00:37:10]** running the system at scale and you need to care about that **[00:37:10 – 00:37:15]** third nine or fourth nine, what does the distribution look like? **[00:37:15 – 00:37:17]** So for a practical case, you have the vLLM and SGLing **[00:37:18 – 00:37:18]** inference servers. **[00:37:18 – 00:37:21]** This is starting a relatively small, like billion parameter **[00:37:21 – 00:37:25]** model that takes them a minute to do that with SGLing and vLLM. **[00:37:25 – 00:37:27]** There's CUDA graphs, there's Torch compilation, there's **[00:37:27 – 00:37:29]** all these things. **[00:37:29 – 00:37:32]** These take time and, but they don't produce that **[00:37:32 – 00:37:34]** much application state. **[00:37:34 – 00:37:36]** And so you can check that they're like an ideal target **[00:37:36 – 00:37:37]** for checkpointing and restoring. **[00:37:37 – 00:37:41]** Trick here is that you don't want to checkpoint or restore **[00:37:41 – 00:37:46]** a KV cache because that is really big and really easy to recreate. **[00:37:46 – 00:37:48]** So you want to take advantage of the sort of like sleep-wake **[00:37:48 – 00:37:49]** features that they have. **[00:37:49 – 00:37:51]** And if you're building your own inference engine, you're going to **[00:37:51 – 00:37:55]** want to put this in there to take advantage of snapshot and restore. **[00:37:55 – 00:37:58]** So, talked about those pieces. **[00:37:58 – 00:38:00]** What's left? Mostly model weight loading, so the next **[00:38:00 – 00:38:03]** step for us is to look at much faster distribution of model **[00:38:03 – 00:38:07]** weights, probably more caching of these things, RDMA is how **[00:38:07 – 00:38:10]** you get the absolute maximum speed. **[00:38:10 – 00:38:12]** These are hard to do in a global dynamic worker pool, **[00:38:12 – 00:38:14]** but we're experimenting with it. **[00:38:14 – 00:38:17]** So just to say a little bit about like how we've actually **[00:38:17 – 00:38:23]** deployed this for we've done about for a couple hundred **[00:38:23 – 00:38:30]** users a couple hundred thousand GPU hours across like tens of thousands **[00:38:30 – 00:38:37]** of applications we've operated this system and seen this like **[00:38:37 – 00:38:43]** a range of speedups between 2x and 10x just with the GPU snapshotting. **[00:38:43 – 00:38:48]** On top of the CPU memory snapshotting after about **[00:38:48 – 00:38:50]** six months of operating the system. **[00:38:50 – 00:38:53]** So I'll just say one use case to check out. **[00:38:53 – 00:38:58]** We wrote up our work with Reducto Enterprise Document Processing **[00:38:58 – 00:39:01]** talking about how they use GPU snapshots to handle extremely **[00:39:01 – 00:39:05]** bursty workloads when an enterprise customer drops their entire **[00:39:05 – 00:39:10]** knowledge base in them and asks for it to be indexed and processed. **[00:39:10 – 00:39:12]** So you can check that out on our blog. **[00:39:12 – 00:39:14]** So if you want to build this yourself, come talk to me. **[00:39:14 – 00:39:16]** I'll be out in the lobby after the talk. **[00:39:16 – 00:39:17]** We'll be at booth 4020. **[00:39:17 – 00:39:19]** If you're interested in building this kind of system, we love **[00:39:19 – 00:39:20]** to talk shop on it. **[00:39:20 – 00:39:23]** If you want to build this with us, check out modal.jobs. **[00:39:23 – 00:39:27]** We're hiring for all kinds of systems and platform engineering. **[00:39:27 – 00:39:31]** And if you want to try it, try the system and see how fast **[00:39:31 – 00:39:32]** it is, check it out on modal.com.