# Continuous Inference Benchmarking of the AI Frontier ## Abstract As large language model architectures evolve, traditional peak performance metrics are becoming less predictive of real-world efficiency. Effective inference benchmarking now requires a multidimensional approach that balances raw throughput against energy consumption, cost-per-token, and the critical "interactivity" thresholds required for user-facing applications. This session introduces InferenceX, SemiAnalysis’ open-source benchmark suite designed to navigate this complexity by sweeping the latest software optimizations across leading AI accelerators. It moves beyond static datasheets to examine nightly, reproducible results that reveal how frontier models perform under variable production workloads. Learn how the InferenceX benchmarks are designed, how to run them, and how to interpret the results to make informed decisions. ## AI Summary - InferenceX, an open-source benchmark, aimed to track the constantly evolving performance of machine learning models and hardware, showing the full Pareto frontier for a comprehensive view. - The project utilized a fleet of over a thousand GPUs donated by supporters such as OpenAI, Oracle, CoreWeave, and Crusoe, to run daily benchmarks and provide up-to-date performance data. - InferenceX compared different hardware and software configurations, highlighting the significant performance improvements from NVIDIA's TRT-LLM and the impact of software optimizations. - The benchmarking included various models and scenarios, with plans to expand to more complex workloads like multi-turn and agentic tasks, and to incorporate new models as they emerged. ## Transcript **[00:00:09 – 00:00:13]** Thank you everyone for attending the talk on InferenceX. **[00:00:13 – 00:00:16]** If you haven't heard of it, I'm surprised because you didn't watch **[00:00:16 – 00:00:17]** the keynote, that was pretty sick. **[00:00:17 – 00:00:21]** Jensen talked about InferenceX for like five minutes. **[00:00:21 – 00:00:22]** But what are we going to talk about today, right? **[00:00:22 – 00:00:26]** So we're going to talk about the motivations of InferenceX. **[00:00:26 – 00:00:27]** It's very important because it's open source, up-to-date, **[00:00:27 – 00:00:29]** and using frontier techniques. **[00:00:29 – 00:00:32]** We're going to explore the frontier, Pareto curve, **[00:00:32 – 00:00:35]** talk about our technical collaborations with NVIDIA. **[00:00:35 – 00:00:38]** And then talk about TRT-LLM from NVIDIA. **[00:00:38 – 00:00:42]** And then we're going to compare NVIDIA to the competing **[00:00:42 – 00:00:43]** hardware out there. **[00:00:43 – 00:00:44]** We're going to talk about software velocity and the **[00:00:45 – 00:00:46]** continuous improvement. **[00:00:46 – 00:00:48]** And then lastly, we're going to talk about how you can **[00:00:48 – 00:00:51]** use InferenceX data. **[00:00:51 – 00:00:53]** So first of all is the motivations. **[00:00:53 – 00:00:56]** The problem with all benchmarks out there in the industry, **[00:00:56 – 00:00:59]** at least before InferenceX, is that they were old and slow, right? **[00:00:59 – 00:01:03]** So you use some old model, you're running on old software, or **[00:01:04 – 00:01:07]** you're running on super hyper-tuned software that is specifically for **[00:01:07 – 00:01:11]** that old model, which is irrelevant because new models have come out. **[00:01:11 – 00:01:14]** Every week there's two VLLM updates, every week there's two **[00:01:15 – 00:01:17]** SGLang updates, every week there's probably a CUDA driver update, **[00:01:17 – 00:01:20]** every week there's a PyTorch update, and all these things. **[00:01:20 – 00:01:22]** Actually, there's a PyTorch nightly every night, right? **[00:01:22 – 00:01:24]** All of these things constantly change and **[00:01:24 – 00:01:25]** that changes the performance. **[00:01:25 – 00:01:29]** And so what we're aiming to do with InferenceX is track this **[00:01:30 – 00:01:33]** constantly evolving performance and show the entire curve. **[00:01:33 – 00:01:35]** Another thing is people love to pick out one point on the **[00:01:35 – 00:01:39]** curve, and that one point on the curve is cherry-picked, right? **[00:01:39 – 00:01:43]** You can easily, you know, people say things about statistics, **[00:01:43 – 00:01:46]** they're very useful for lying, and so the thing that we try **[00:01:46 – 00:01:48]** to do is show the entire curve. **[00:01:48 – 00:01:51]** And so, yes, people can then take individual points and lie with **[00:01:51 – 00:01:53]** them, but at the end of At the end of the day, you have access to the **[00:01:53 – 00:01:55]** full curve and it's open source. **[00:01:55 – 00:02:00]** The other aspect of InferenceX is that it is completely **[00:02:00 – 00:02:02]** open source, right? **[00:02:02 – 00:02:05]** Anyone can take what we've run and run it themselves **[00:02:05 – 00:02:09]** and reproduce the numbers, and you can see that for any day that **[00:02:09 – 00:02:11]** we've published the InferenceX. **[00:02:11 – 00:02:15]** So every day for the last handful of months, you can take the image, **[00:02:15 – 00:02:17]** run it, and get the performance. **[00:02:17 – 00:02:20]** And then lastly, you know, what's important is this entire **[00:02:21 – 00:02:24]** Pareto frontier and using it for competitive analysis. **[00:02:24 – 00:02:26]** It's one thing to just look at performance, but performance **[00:02:26 – 00:02:28]** in isolation ignores cost. **[00:02:28 – 00:02:31]** So you have to look at cost and you can't just look at cost alone **[00:02:31 – 00:02:32]** because power is also a metric. **[00:02:32 – 00:02:34]** So we look at all of these different metrics and you can **[00:02:34 – 00:02:36]** configure what you want to look at. **[00:02:36 – 00:02:39]** And then this is maintained by us, AMD and NVIDIA, our **[00:02:39 – 00:02:42]** huge contributors, and there's other folks there. **[00:02:42 – 00:02:46]** And the purpose of this is it's for the benefit of the ML community. **[00:02:46 – 00:02:49]** The other thing about InferenceX is that because it's a living, **[00:02:49 – 00:02:52]** breathing benchmark, there's no fixed release cadence, right? **[00:02:52 – 00:02:54]** It's literally every day we're running. **[00:02:54 – 00:02:56]** There's no unsubmitting results. **[00:02:56 – 00:02:57]** That's the thing about a lot of benchmarks. **[00:02:57 – 00:02:59]** People don't like what they're going to put out **[00:02:59 – 00:03:00]** there, they pull away. **[00:03:00 – 00:03:02]** And so if you look at certain benchmarks, NVIDIA's the only **[00:03:02 – 00:03:03]** one submitting to them. **[00:03:03 – 00:03:06]** That's kind of pointless, right? **[00:03:06 – 00:03:08]** And the other aspect of benchmarking is it's really, **[00:03:08 – 00:03:09]** really freaking expensive. **[00:03:09 – 00:03:13]** So we have over a thousand GPUs, over, you know, 40, **[00:03:13 – 00:03:16]** 30 million dollars of hardware that we're running every night **[00:03:16 – 00:03:20]** and that's donated to us graciously by all of our supporters, right? **[00:03:20 – 00:03:23]** So we have supporters like OpenAI, like Oracle, like **[00:03:23 – 00:03:25]** CoreWeave, like Crusoe. **[00:03:25 – 00:03:29]** NVIDIA and AMD, TensorFlow, and many others are supporting **[00:03:29 – 00:03:30]** us with hardware. **[00:03:30 – 00:03:34]** So this is galvanizing the ecosystem to give us the resources **[00:03:34 – 00:03:37]** to run these benchmarks, which are incredibly costly to run, **[00:03:37 – 00:03:41]** and the engineering team, which is incredibly costly to run. **[00:03:42 – 00:03:45]** So the whole point is, because you can see the day-to-day **[00:03:45 – 00:03:49]** performance chart here is, hey, how's performance evolved **[00:03:49 – 00:03:54]** over the last few months for any specific GPU, any specific model? **[00:03:54 – 00:03:57]** We support seven plus models now. **[00:03:57 – 00:04:00]** Anyways, the whole goal here is to push the ecosystem forward. **[00:04:00 – 00:04:02]** And so I'm going to stop yapping because I didn't **[00:04:02 – 00:04:03]** do any of this work. **[00:04:03 – 00:04:05]** This is the team that actually did the work. **[00:04:05 – 00:04:07]** Here you go, Cam. Yeah, hey guys. **[00:04:07 – 00:04:09]** I'm Cam Colucci. I'm one of the leads here at InferenceX. **[00:04:09 – 00:04:12]** I'm one of the main incurred costs for Dylan, apparently. **[00:04:12 – 00:04:13]** So, yeah. **[00:04:13 – 00:04:17]** So, just by a show of hands here, who, like, understands, like, **[00:04:17 – 00:04:20]** what the Pareto frontier is for inference, like, inferencing and... **[00:04:21 – 00:04:22]** Okay, so quite a few guys. **[00:04:22 – 00:04:25]** So, I think one of the main things that we typically like **[00:04:25 – 00:04:28]** to talk about is how the analogy of a bus versus a Ferrari. **[00:04:28 – 00:04:33]** So, a bus and a Ferrari might cost the same amount of money, **[00:04:33 – 00:04:37]** like, end-to-end, but the bus is amortized across multiple **[00:04:37 – 00:04:40]** users, whereas the Ferrari, you know, there's only one user and **[00:04:40 – 00:04:42]** he has to pay for the whole thing. **[00:04:43 – 00:04:45]** This is kind of like the idea of the Pareto Frontier inference, **[00:04:45 – 00:04:49]** you have high batch size, very high latency, but high **[00:04:49 – 00:04:50]** throughput over here. **[00:04:50 – 00:04:56]** And on this end you have a SQL user getting many tokens very fast, so **[00:04:56 – 00:04:57]** kind of the trend here would be... **[00:04:57 – 00:05:00]** You'd have high batch size at the top left with maybe a lower **[00:05:00 – 00:05:05]** degree of parallelism so you have more room for KV cache and whatnot. **[00:05:05 – 00:05:08]** And then down here as you go down, you have higher **[00:05:08 – 00:05:11]** parallelism degrees, so each user is effectively allocated more **[00:05:11 – 00:05:14]** compute, so that's a lower latency. **[00:05:14 – 00:05:17]** So, this is the way to think about inference. **[00:05:17 – 00:05:20]** If anyone ever just tries to show you a bar chart with **[00:05:20 – 00:05:24]** throughput here versus throughput here, you've got to look at **[00:05:24 – 00:05:26]** the whole curve because this is showing you the whole picture. **[00:05:26 – 00:05:29]** This is the point of InferenceX. **[00:05:29 – 00:05:30]** Okay, so moving on. **[00:05:30 – 00:05:35]** Oh, I guess I already talked about the throughput Pareto **[00:05:35 – 00:05:38]** optimal frontier, but really you can think about this in **[00:05:38 – 00:05:42]** terms of many different y-axis metrics, and the SEMI analysis **[00:05:42 – 00:05:47]** analyst team have many, many models where we calculate, **[00:05:47 – 00:05:50]** for instance, the total cost of ownership of all these chips. **[00:05:50 – 00:05:52]** And on the InferenceX website, **[00:05:52 – 00:05:55]** You can select from all of these different y-axis metrics, so we're **[00:05:55 – 00:05:59]** not only viewing the Pareto optimal frontier in terms of throughput, **[00:05:59 – 00:06:01]** but also in terms of things like cost per million total tokens. **[00:06:01 – 00:06:06]** So, whereas maybe an NVIDIA chip is overall higher throughput, **[00:06:06 – 00:06:12]** maybe an AMD chip might be cheaper, for example. **[00:06:12 – 00:06:15]** So yeah, just a bit on the setup, right? **[00:06:16 – 00:06:19]** Right now we are using random data with fixed input sequence **[00:06:19 – 00:06:20]** links and output sequence links. **[00:06:20 – 00:06:23]** So if you go back here, you can see this is on the top **[00:06:23 – 00:06:26]** left here, 8K, 1K on FP4. **[00:06:27 – 00:06:30]** Kind of, this is something we want to move away from in the future, **[00:06:30 – 00:06:35]** but for now, it's a really good baseline for inference and it's **[00:06:35 – 00:06:38]** representing three main scenarios, 1K, 1K, which is like chat **[00:06:38 – 00:06:44]** QA, 1K, 8K, long decode, and then 8K, 1K, which is like long context. **[00:06:44 – 00:06:46]** So, you know, this is an imperfect setup. **[00:06:46 – 00:06:47]** We're always working to make it better. **[00:06:47 – 00:06:51]** Yeah, right now we don't actually test prefix caching, but the **[00:06:51 – 00:06:53]** thing is we're iterating super, super fast on this. **[00:06:53 – 00:06:56]** It's very young, and in the next version, you're gonna **[00:06:56 – 00:06:59]** see all of these things, so. **[00:07:00 – 00:07:04]** Okay, now since we're at GTC, let's talk about InferenceX plus NVIDIA. **[00:07:04 – 00:07:09]** So I just want to take a second to thank the Dynamo and TRT **[00:07:09 – 00:07:12]** LLM teams because this would not be possible without them. **[00:07:12 – 00:07:17]** This is not like us three just making all the configs, right? **[00:07:17 – 00:07:20]** The NVIDIA team has been amazing in helping. **[00:07:20 – 00:07:22]** It would not be possible without them. **[00:07:22 – 00:07:25]** We're happy to say that we have a full Pareto optimal **[00:07:25 – 00:07:28]** frontier for all six NVIDIA Western GPU SKUs over the past four **[00:07:28 – 00:07:31]** years, which is pretty incredible. **[00:07:31 – 00:07:33]** No other inference benchmark does this. **[00:07:33 – 00:07:35]** We employ state-of-the-art inference techniques like **[00:07:35 – 00:07:38]** Disag Prefail and YEP. **[00:07:39 – 00:07:44]** And, yeah, this is something that just is new and novel and awesome, **[00:07:44 – 00:07:47]** and you guys should definitely check it out for yourself. **[00:07:47 – 00:07:50]** Thank you very much. **[00:07:50 – 00:07:52]** Quick way to put it in perspective, you know, over the last **[00:07:52 – 00:07:56]** six months, we've run over 115,000 sweeps, which equates **[00:07:56 – 00:07:59]** to around 2.65 GPU years. **[00:07:59 – 00:08:02]** And yeah, as Dylan said, we have graciously had Compute **[00:08:02 – 00:08:06]** donated to us, and we're managing a fleet of, I don't even know **[00:08:06 – 00:08:13]** how many, hundreds at least, of GPUs, and it's quite cool. **[00:08:14 – 00:08:17]** I'm going to pass it off to Kimbo now. **[00:08:18 – 00:08:20]** Thank you Cam. **[00:08:20 – 00:08:23]** So I'm going to talk about the advantages of rack scale **[00:08:23 – 00:08:28]** system like MVL72 on inference. **[00:08:31 – 00:08:33]** So what is a RackScale system? **[00:08:34 – 00:08:37]** In MVL72, a RackScale system consists of 72 GPUs. **[00:08:38 – 00:08:42]** They're linked through NVLink at 900 gigabytes per second **[00:08:43 – 00:08:46]** per GPU for Blackwell. **[00:08:46 – 00:08:52]** For a Blackwell rack and so the idea is that by having 72 GPUs **[00:08:52 – 00:08:58]** and within a scale of domain we can aggregate for 576 terabytes per **[00:08:58 – 00:09:06]** second of total bandwidth and this is significantly higher than other **[00:09:06 – 00:09:10]** previous generations and by doing so we could leverage parallelism **[00:09:10 – 00:09:14]** without leaving the NVLink domain. **[00:09:16 – 00:09:20]** So one technique we can use with this rack scale system **[00:09:20 – 00:09:25]** is that the wide EP technique, wide expert parallelism. **[00:09:25 – 00:09:29]** For example, we use, we benchmark DeepSeek R1 with wide EP. **[00:09:29 – 00:09:33]** By doing so, we don't, by applying this technique, we do not **[00:09:33 – 00:09:37]** need to leave the NVLink domain. **[00:09:37 – 00:09:41]** And as we see in the chart, we can see that Blackwell can achieve **[00:09:41 – 00:09:47]** 20x throughput at low interactivity regions compared to Hopper. **[00:09:48 – 00:09:51]** So if we translate this throughput improvement to **[00:09:51 – 00:09:56]** cost, as we see here, in different interactivity regions, **[00:09:56 – 00:10:04]** we could reduce the cost to 10%, and in some cases even to 1.5%. **[00:10:07 – 00:10:12]** So the chart previously was in log scale, so if we see **[00:10:12 – 00:10:16]** it in actual y-axis, a normal axis, we can see that the **[00:10:16 – 00:10:20]** curve is actually very steep. **[00:10:23 – 00:10:27]** So yeah, if we further apply more techniques such as reducing **[00:10:27 – 00:10:32]** the model precision, we can see even more gains in performance. **[00:10:32 – 00:10:39]** Concretely, if we use GP200, MV072, applying the FP4 precision, **[00:10:40 – 00:10:45]** comparing to FP8 with H100 with all the latest techniques, we **[00:10:45 – 00:10:52]** can still see from 20x all the way to 100x improvement in throughput. **[00:10:57 – 00:11:02]** So applying YDP, we can see that RAC scale really shines **[00:11:02 – 00:11:04]** in these kind of scenarios. **[00:11:04 – 00:11:10]** And as a general trend, we see that YDP really improves **[00:11:10 – 00:11:15]** the throughput at the middle of the The curve of the interactivities. **[00:11:16 – 00:11:18]** We believe this is for a combination of reasons, **[00:11:18 – 00:11:24]** including by having a rack scale system, we could have a higher **[00:11:24 – 00:11:31]** aggregate bandwidth and reducing the time for loading extra weights. **[00:11:33 – 00:11:38]** Okay, so in addition to YDP, we also applied disaggregate **[00:11:38 – 00:11:39]** serving techniques. **[00:11:39 – 00:11:44]** And by applying these two techniques, we see that across **[00:11:44 – 00:11:47]** different configurations, **[00:11:47 – 00:11:50]** this kind of rack system really improves, really beats a lot **[00:11:50 – 00:11:53]** of different comparing systems. **[00:11:53 – 00:11:59]** So for example here, by holding the framework constant, we **[00:11:59 – 00:12:04]** can see that GB200s on Dynamo and SGLang really outperforms **[00:12:04 – 00:12:11]** MI355 with their own MoRI framework on SGLang by a lot. **[00:12:12 – 00:12:15]** The second thing we want to talk about is the TRT-LLM. **[00:12:15 – 00:12:20]** So TRT-LLM is a framework that we also benchmarked that is **[00:12:20 – 00:12:22]** designed and developed by NVIDIA. **[00:12:23 – 00:12:27]** NVIDIA really leveraged software and hardware co-design to really **[00:12:27 – 00:12:32]** improve the performance of TRT-LLM. **[00:12:32 – 00:12:39]** So concretely we see that if we compare TRTLM with SGLang, at low **[00:12:39 – 00:12:48]** interactivity regions we can see peak 1.8x throughput improvement. **[00:12:50 – 00:12:57]** This slide is just talking about how TRTLM was one of the first **[00:12:57 – 00:13:01]** inference engines to really dive into this idea of hardware-software **[00:13:01 – 00:13:05]** co-design with the NVIDIA chips. **[00:13:05 – 00:13:09]** They've actually inspired, apparently, AMD with Rocknum to **[00:13:09 – 00:13:14]** create Atom, which is a specialized inference engine for AMD GPUs. **[00:13:14 – 00:13:17]** You can see here it's still a work in progress but we're tracking **[00:13:17 – 00:13:19]** the progress and it'll probably **[00:13:19 – 00:13:22]** Be better, hopefully, soon. **[00:13:23 – 00:13:25]** Okay. How do I use this? **[00:13:25 – 00:13:27]** Just this? Okay. **[00:13:27 – 00:13:30]** Finally learned how to use this. **[00:13:30 – 00:13:32]** Okay. **[00:13:33 – 00:13:36]** So if we compare different frameworks, here we also **[00:13:36 – 00:13:38]** benchmark VLM and SGLang. **[00:13:38 – 00:13:45]** On the VLM side, we also see that TRT-LLM outperforms VLM on B200s **[00:13:45 – 00:13:50]** across all configurations, lifting up the throughput interactivity **[00:13:50 – 00:13:54]** curve for every scenario. **[00:13:56 – 00:14:03]** Comparing even more other scenarios, we can see that **[00:14:03 – 00:14:08]** after applying disaggregated serving techniques, TRT-LLM **[00:14:08 – 00:14:09]** performs really well. **[00:14:09 – 00:14:15]** Concretely, if we compare SGLang with TRT-LLM, holding **[00:14:15 – 00:14:20]** the multi-node serving framework constant and **[00:14:20 – 00:14:26]** Applying MTP, we can still see 2.5x improvement across **[00:14:26 – 00:14:31]** different serving, across different interactivities. **[00:14:33 – 00:14:37]** That said, TRT-LLM doesn't always improve, it doesn't always **[00:14:37 – 00:14:41]** outperform for every scenario. **[00:14:41 – 00:14:48]** So here we see that on H200s, VLM actually outperforms TRT-LLM **[00:14:48 – 00:14:52]** on the lower interactivity regions, having higher throughput. **[00:14:52 – 00:14:56]** And we believe this is because TRT-LLM actually contributes **[00:14:56 – 00:15:00]** back to the open source community With different things, including **[00:15:00 – 00:15:02]** their GPU kernels. **[00:15:02 – 00:15:05]** And by doing so, we could, the open source community **[00:15:06 – 00:15:10]** could have better performance, and so this is definitely not, TRT-LLM **[00:15:10 – 00:15:16]** is definitely not only competing a sole competition with open source **[00:15:16 – 00:15:21]** frameworks such as VLM and SGLang. **[00:15:23 – 00:15:25]** Next, I'll talk about NVIDIA's competitors and **[00:15:25 – 00:15:27]** the competitive frontier. **[00:15:27 – 00:15:30]** So, NVIDIA's best chip, the NVL72, is RackScale, and there's **[00:15:30 – 00:15:33]** really no comparison chip that we have that's really **[00:15:33 – 00:15:35]** apples-to-apples in our benchmark. **[00:15:35 – 00:15:39]** And our nearest competitor chip is the MI355X, and we can't really **[00:15:39 – 00:15:43]** compare that to NVL72 because it's not really apples-to-apples, **[00:15:43 – 00:15:45]** and the closest competitor to the competitor is B200. **[00:15:45 – 00:15:49]** So looking at B200 against MI355X, **[00:15:51 – 00:15:58]** MI355 MoRI-SGLang outperforms B200-Dynamo-SGLang, but NVIDIA **[00:15:58 – 00:16:01]** takes the cake again with TRT, as mentioned by Kimbrough **[00:16:01 – 00:16:05]** earlier, due to the great TRT software-hardware co-design. **[00:16:06 – 00:16:11]** And MoRI is a new communication framework by AMD for KVH transfer **[00:16:11 – 00:16:15]** and disaggregated prefill, and it's built from the ground up, instead **[00:16:15 – 00:16:20]** of forking NCCL like RCCL did. **[00:16:21 – 00:16:28]** And yeah, once you can see B200 beats MI355 at this deep-seq FP8, **[00:16:28 – 00:16:32]** 8K input tokens, 1K output tokens. **[00:16:32 – 00:16:35]** But the story changes once we move over to FP4 with the **[00:16:35 – 00:16:39]** same model and 8K in, 1K out. **[00:16:39 – 00:16:43]** So we can see that AMD, in the bottom left, performs worse **[00:16:43 – 00:16:46]** now than B200, Dynamo, SGLang. **[00:16:47 – 00:16:50]** And this is a software problem with AMD, and definitely not **[00:16:50 – 00:16:51]** a hardware problem. **[00:16:51 – 00:16:56]** The flops and the memory bandwidth of AMD's MI355 is sufficient **[00:16:56 – 00:16:59]** and does not account for this difference in performance. **[00:16:59 – 00:17:02]** And this just shows the difference in software development that **[00:17:02 – 00:17:06]** AMD has with NVIDIA. **[00:17:06 – 00:17:09]** And I'm not trying to say that AMD isn't improving. **[00:17:09 – 00:17:14]** We have preliminary results of MI55 improving over its **[00:17:14 – 00:17:18]** previous and achieving up to five times improvement since **[00:17:18 – 00:17:20]** its introduction in the benchmark. **[00:17:20 – 00:17:22]** And this just shows how much AMD is trying to improve and **[00:17:22 – 00:17:26]** the head start that NVIDIA software actually already has. **[00:17:26 – 00:17:31]** And moving on to new models, this... **[00:17:31 – 00:17:32]** The gap is even more drastic. **[00:17:32 – 00:17:38]** In the recently released QN 3.5 397B, NVIDIA basically **[00:17:38 – 00:17:43]** surpassed AMD by almost three times at certain interactivity **[00:17:43 – 00:17:47]** levels, and this just largely shows the software gap that **[00:17:47 – 00:17:51]** NVIDIA has due to the rapid pace of software development, and it's **[00:17:51 – 00:17:53]** not an interesting hardware gap. **[00:17:54 – 00:17:58]** And in VROM, with the new MiniMax M2.5.23, **[00:18:00 – 00:18:04]** Out of 230 billion models, the same story is shown. **[00:18:04 – 00:18:09]** NVIDIA has great software and is able to beat AMD at **[00:18:09 – 00:18:11]** every interactivity level. **[00:18:11 – 00:18:18]** H200 even beats M355X at certain interactivities. **[00:18:18 – 00:18:21]** Moving on to talk more about the software velocity and the **[00:18:21 – 00:18:24]** continuous improvement of NVIDIA. **[00:18:24 – 00:18:33]** This is a graph of NVL72 with Dynamo TRT from December 19 **[00:18:33 – 00:18:36]** to January 31, about 40 days. **[00:18:36 – 00:18:40]** And within these 40 days, we see a two times performance increase. **[00:18:40 – 00:18:44]** So this is one crucial part of InferenceX, where we actually **[00:18:44 – 00:18:47]** are able to trace the history of these software improvements **[00:18:47 – 00:18:52]** and see how software improves over time for this hardware. **[00:18:52 – 00:18:56]** And likewise for SGLang, these are graphs of B200 SGLang, **[00:18:56 – 00:19:02]** the two on the bottom are FP8, showing improvement from **[00:19:02 – 00:19:03]** minor version 0.5.5 to 0.5.6. **[00:19:04 – 00:19:08]** And this is a minor version change, and it's already quite **[00:19:08 – 00:19:12]** a big improvement at certain areas, almost two times at **[00:19:12 – 00:19:14]** high throughput regions. **[00:19:14 – 00:19:17]** And likewise for FP4, there's also a big improvement from **[00:19:17 – 00:19:20]** a minor version change. **[00:19:20 – 00:19:24]** And likewise for VLLM, we see a similar increase as **[00:19:24 – 00:19:28]** SGLang from version 0.11.2 to 0.13. **[00:19:28 – 00:19:32]** And our benchmark also has the capability of showing the change **[00:19:32 – 00:19:34]** logs between such version changes. **[00:19:34 – 00:19:41]** In this case, we attribute the improvement to VLLM introducing **[00:19:41 – 00:19:45]** Marlin kernels to MXFP4 for H200. **[00:19:45 – 00:19:47]** And yeah, in a short span of three months, we see quite **[00:19:47 – 00:19:50]** a big improvement. **[00:19:51 – 00:19:58]** Yeah, so I'm gonna wrap it up here by just talking about **[00:19:58 – 00:20:01]** how people use InferenceX data, how you should, how you can, **[00:20:01 – 00:20:04]** and how we use InferenceX data. **[00:20:04 – 00:20:08]** So, yeah, the one example we'll use is the idea of tokenomics, **[00:20:08 – 00:20:11]** so basically analyzing the unit economics of inference **[00:20:11 – 00:20:14]** providers and, like, yeah, you can kind of gain insight **[00:20:14 – 00:20:18]** into, like, how pricing happens, although it's a bit mysterious, **[00:20:18 – 00:20:23]** but anyway, this is open rotor data from February 20th, and **[00:20:23 – 00:20:28]** if we just take the Crusoe serving DeepSeek R1 FP8 at an interactivity **[00:20:28 – 00:20:31]** of 35 tokens per second per user, so I think it's 36 up **[00:20:32 – 00:20:34]** there, but it's around 35, right? **[00:20:34 – 00:20:38]** Let's ignore prefix caching for simplicity, just analyzing **[00:20:38 – 00:20:40]** the baseline cost. **[00:20:40 – 00:20:43]** And now let's cross-reference this with the inference X **[00:20:43 – 00:20:45]** data that we collected. **[00:20:45 – 00:20:48]** So using the semi-analysis AI cloud total cost of ownership **[00:20:48 – 00:20:50]** model where we... **[00:20:50 – 00:20:54]** Where we get the pricing, or estimate the pricing for all **[00:20:54 – 00:20:59]** the SKUs, we can get on the left here, you're seeing the cost per **[00:20:59 – 00:21:03]** million input tokens at 35 tokens per second for DeepSeek R1 FP8, and **[00:21:03 – 00:21:08]** on the right, you see the cost per million output tokens for each SKU. **[00:21:08 – 00:21:11]** So the idea here is if you say, okay, well, let's assume **[00:21:11 – 00:21:15]** that Crusoe is using at least H200, which is reasonable **[00:21:15 – 00:21:19]** enough, and let's just assume that they're serving with **[00:21:19 – 00:21:25]** vanilla SGLang, no MTP, and no custom optimizations in there, **[00:21:25 – 00:21:27]** which obviously isn't true, but... **[00:21:27 – 00:21:31]** With this setup, they're serving at about $0.22 per million **[00:21:31 – 00:21:35]** input tokens and around $2.95 per million output tokens. **[00:21:35 – 00:21:38]** And cross-referencing the OpenRouter data that we just **[00:21:38 – 00:21:42]** showed, they're serving at around $1.35 per million input **[00:21:42 – 00:21:44]** and $5.40 per million output. **[00:21:44 – 00:21:46]** So just some simple math on the margins here. **[00:21:46 – 00:21:48]** So you can say that, at minimum, **[00:21:48 – 00:21:50]** At minimum, because remember, no optimizations, no prefix **[00:21:51 – 00:21:55]** caching, nothing, their margin would be at least 83% on input **[00:21:56 – 00:21:58]** and at least 45% on output, right? **[00:21:58 – 00:22:02]** So this is just like napkin math calculations, but it **[00:22:02 – 00:22:04]** kind of gives you an idea of like how you can use the **[00:22:04 – 00:22:10]** data to like have sort of a better holistic understanding of pricing. **[00:22:10 – 00:22:14]** And then finally, you know, Cloud Fast Mode came out, and I saw **[00:22:14 – 00:22:17]** a lot of stuff on Twitter that was like, oh, this must be a new chip, **[00:22:17 – 00:22:21]** or, you know, are they using LPX, like, what's obvious, I mean... The **[00:22:21 – 00:22:25]** point is, is like, for those of you who don't know, Cloud Fast Mode is **[00:22:25 – 00:22:31]** promising, you know, 6x the, 2.5x the speed for 6x the price, right? **[00:22:31 – 00:22:35]** And when you understand inference, you know, across the whole **[00:22:35 – 00:22:39]** frontier, you understand that, no, it's not a new chip, it's not, **[00:22:39 – 00:22:42]** you know, it doesn't have to be any crazy things, it's just another **[00:22:42 – 00:22:47]** point on the curve, right, because when you think about it, if... **[00:22:47 – 00:22:49]** I am receiving tokens faster myself. **[00:22:49 – 00:22:52]** I have to pay more because the throughput of the overall **[00:22:52 – 00:22:54]** system has to be less. **[00:22:54 – 00:22:55]** There has to be a trade-off. **[00:22:55 – 00:23:00]** Therefore, the cost of the system is amortized over less users. **[00:23:00 – 00:23:03]** So me, myself as a user, have to pay more. **[00:23:03 – 00:23:06]** And another thing on this chart, right, is you see the **[00:23:06 – 00:23:07]** lower curve, right? **[00:23:07 – 00:23:09]** This is cost per million tokens, right? **[00:23:09 – 00:23:12]** The lower chart is with MTP, right? **[00:23:12 – 00:23:16]** And then the more expensive frontier is without MTP. **[00:23:16 – 00:23:20]** So this just goes to show that even simple optimizations **[00:23:20 – 00:23:24]** like MTP and speculative decoding can drastically decrease the **[00:23:24 – 00:23:27]** cost of inference at a fixed interactivity level. **[00:23:27 – 00:23:32]** So yeah, with that, I think we're basically done, and I **[00:23:32 – 00:23:34]** want to thank everyone for coming. **[00:23:34 – 00:23:37]** And we're done quite early, so if anyone has any questions, **[00:23:37 – 00:23:41]** you're welcome to ask. **[00:23:45 – 00:23:48]** I do want to emphasize, since we froze the content of this **[00:23:48 – 00:23:51]** presentation, the team's already added numerous features, right? **[00:23:51 – 00:23:52]** They've added evals, right? **[00:23:52 – 00:23:56]** It turns out some GPUs and some providers of GPUs not **[00:23:56 – 00:24:00]** named NVIDIA, there's only one other one in the presentation, **[00:24:00 – 00:24:03]** they use certain kernel tricks and then that makes their evals **[00:24:03 – 00:24:05]** worse for the exact same benchmark. **[00:24:05 – 00:24:07]** So evals are very important to watch as well because there **[00:24:07 – 00:24:08]** are always tricks you can do. **[00:24:08 – 00:24:12]** And so that's something that Brian implemented over the last **[00:24:12 – 00:24:15]** few weeks since we've frozen the **[00:24:15 – 00:24:17]** Since we've frozen the content for this presentation, so **[00:24:17 – 00:24:22]** things are always being added, and you too can contribute. **[00:24:22 – 00:24:26]** It's completely open source, so we would love contributions. **[00:24:26 – 00:24:28]** Okay, thanks for the nice presentation. **[00:24:28 – 00:24:30]** I have a couple of questions. **[00:24:30 – 00:24:33]** First question is, how do you ensure that the numbers, **[00:24:33 – 00:24:36]** the performance you are getting is the best out of the hardware? **[00:24:36 – 00:24:37]** That is my first question. **[00:24:37 – 00:24:41]** And second one is, where does the benchmark run? **[00:24:41 – 00:24:45]** The inference runs in the NVIDIA data center or it runs in **[00:24:45 – 00:24:49]** the cluster managed by InferenceX? **[00:24:49 – 00:24:52]** Yeah, thanks. So, first question, how do we know it's the best? **[00:24:52 – 00:24:55]** I suppose, **[00:24:55 – 00:25:00]** The thing that's really fun is both NVIDIA's engineering **[00:25:00 – 00:25:04]** teams, and Kedar's in the crowd, we love him so much, and he's **[00:25:05 – 00:25:07]** leading the engineering teams at NVIDIA helping us here, or at least **[00:25:07 – 00:25:09]** coordinating, and same with AMD. **[00:25:09 – 00:25:13]** They're so competitive about this that... **[00:25:13 – 00:25:18]** They always are submitting updates to our repos, to their **[00:25:19 – 00:25:22]** drivers, to the VLM backend, to the HDLang backend, etc. **[00:25:22 – 00:25:26]** So to get the best possible performance, because people **[00:25:26 – 00:25:30]** are using this to drive purchasing decisions, which is awesome, right? **[00:25:30 – 00:25:33]** As shown by some of our supporters of the benchmark. **[00:25:33 – 00:25:35]** And so that's how we're sort of ensuring that performance **[00:25:35 – 00:25:36]** is best as possible. **[00:25:36 – 00:25:40]** We sweep a lot of things like parallelism configs, but also **[00:25:40 – 00:25:42]** AMD and NVIDIA are so incentivized to make sure they're putting **[00:25:42 – 00:25:46]** their best foot forward across the whole curve that it kind **[00:25:46 – 00:25:48]** of ensures that they are there. **[00:25:48 – 00:25:51]** Or they'd be leaving money on the table and they'd be looking bad in **[00:25:51 – 00:25:54]** comparisons, which people are using to drive purchasing decisions. **[00:25:54 – 00:25:58]** And to add to that, you know, like, you don't really have **[00:25:58 – 00:26:02]** to sweep every single configuration because you can kind of derive **[00:26:02 – 00:26:06]** the set of configurations that make sense from first principles. **[00:26:06 – 00:26:09]** For instance, you would never have a high throughput situation with, **[00:26:09 – 00:26:10]** like, extremely low batch size. **[00:26:10 – 00:26:12]** It just doesn't. **[00:26:12 – 00:26:17]** And to your second question, we have various clusters managed **[00:26:17 – 00:26:20]** by various providers, so yeah. **[00:26:20 – 00:26:27]** As mentioned, we use Crusoe, we use Nebius, we use NVIDIA and AMD **[00:26:27 – 00:26:28]** Direct, we use We use TensorWave. **[00:26:28 – 00:26:30]** Is there anyone else? **[00:26:30 – 00:26:32]** That's it. **[00:26:33 – 00:26:36]** Thank you. Thank you. **[00:26:37 – 00:26:38]** CoreWeave? Oh, CoreWeave, yes. **[00:26:38 – 00:26:40]** CoreWeave, yes. And OCI. **[00:26:40 – 00:26:41]** And Oracle, yes. There we go. **[00:26:41 – 00:26:44]** So we have seven providers, basically, of compute that **[00:26:44 – 00:26:48]** helps us make sure that it's not cherry picked. **[00:26:48 – 00:26:49]** Hi, my name is Gagandeep. **[00:26:49 – 00:26:51]** Thanks for the talk. I'm from Salesforce. **[00:26:51 – 00:26:56]** The question is regarding, apart from AMD and NVIDIA, let's **[00:26:56 – 00:27:00]** think about Chinese manufacturers like Alibaba or Huawei. **[00:27:00 – 00:27:03]** Do we have any data on them, or are you benchmarking them **[00:27:03 – 00:27:05]** as well, or how is it going? **[00:27:05 – 00:27:10]** So our roadmap of implementation is focused primarily on TPUs and **[00:27:10 – 00:27:15]** traniums, and so that's what we're focused on implementing first. **[00:27:15 – 00:27:19]** If someone can get us access to Huawei chips or Alibaba **[00:27:19 – 00:27:21]** chips or anyone else's chips for free, because that's the **[00:27:21 – 00:27:24]** thing, all the compute is donated, this benchmark is **[00:27:24 – 00:27:31]** completely free, we would love to implement it, but those providers, **[00:27:32 – 00:27:36]** we don't have direct contacts with Huawei, for obvious reasons. **[00:27:36 – 00:27:39]** So if anyone wants to provide Chinese chips, we would **[00:27:39 – 00:27:40]** love to try it out. **[00:27:40 – 00:27:44]** But right now we're focused on TPUs and Tranium. **[00:27:44 – 00:27:45]** Thanks for the talk. **[00:27:45 – 00:27:49]** The question is about the random data you're using, and whether **[00:27:49 – 00:27:54]** it creates a realistic distribution of export activations, and whether **[00:27:54 – 00:27:57]** that's a concern for you guys? **[00:27:57 – 00:28:01]** Yeah, I think Brian actually did a lot of research into **[00:28:01 – 00:28:04]** this, and the answer is yes. **[00:28:05 – 00:28:08]** You would think actually that the expert activation would **[00:28:08 – 00:28:11]** be kind of uniform with random data, but it turns out it's **[00:28:11 – 00:28:13]** actually not the case. **[00:28:14 – 00:28:17]** I don't actually know why, but someone in the crowd probably does. **[00:28:17 – 00:28:20]** But another thing is the acceptance rates are off, right? **[00:28:20 – 00:28:23]** But kind of like... **[00:28:23 – 00:28:25]** First of all, we're moving away from random data, right? **[00:28:25 – 00:28:27]** So that's going to be good for real data sets. **[00:28:27 – 00:28:31]** But kind of like when the playing field is level, it's **[00:28:31 – 00:28:35]** sort of, I feel like that the baseline benchmark results **[00:28:35 – 00:28:38]** are still like a great proxy of the actual throughput, right? **[00:28:38 – 00:28:41]** But that's a good point. **[00:28:42 – 00:28:43]** Hello. **[00:28:43 – 00:28:45]** First of all, thanks for your work you're doing. **[00:28:45 – 00:28:47]** Love your articles. **[00:28:47 – 00:28:48]** Keep it up. **[00:28:48 – 00:28:52]** I was in a previous talk, actually here, NVIDIA was **[00:28:52 – 00:28:58]** talking about Avira, and generally there's a strong signal around... **[00:28:58 – 00:29:01]** More computation across the inference stack, especially **[00:29:01 – 00:29:03]** with agentic workloads. **[00:29:03 – 00:29:06]** And that, I would assume, would add more dimensions **[00:29:06 – 00:29:07]** on the benchmarking. **[00:29:07 – 00:29:12]** So apart from clearly running the LLMs and those models, do you guys **[00:29:12 – 00:29:18]** think of a future roadmap where you include more peripherals and other **[00:29:18 – 00:29:21]** inputs generally on the stack? **[00:29:21 – 00:29:23]** And how would that work? Because that would be also **[00:29:23 – 00:29:26]** incredibly useful. **[00:29:28 – 00:29:31]** We have a roadmap of improvements and Cam's actually been **[00:29:31 – 00:29:33]** working on exactly this, a multi-turned benchmark. **[00:29:33 – 00:29:37]** Right, yeah, yeah, multi-turn and agentic, you know, extremely **[00:29:37 – 00:29:40]** long context, high prefix cache rates, etc. **[00:29:40 – 00:29:41]** And tool calling. **[00:29:41 – 00:29:44]** And tool calling, right, so you can expect that in the **[00:29:44 – 00:29:48]** next few months, hopefully. **[00:29:51 – 00:29:54]** First, I want to say thank you not only for today, but for the **[00:29:54 – 00:29:59]** consistent high-quality container you guys created for the community. **[00:29:59 – 00:30:00]** So, thank you. **[00:30:00 – 00:30:06]** And my question is that there is a key point at yesterday's **[00:30:06 – 00:30:11]** keynote that advanced hardware often delivers biggest advantage **[00:30:11 – 00:30:16]** over the old hardware when it runs large models. **[00:30:16 – 00:30:21]** And I just checked the website, and at the moment, we benchmarking **[00:30:21 – 00:30:22]** open source models. **[00:30:22 – 00:30:27]** But nowadays, closed source models are much bigger than **[00:30:27 – 00:30:28]** the open source models. **[00:30:28 – 00:30:33]** Do you guys think this is an issue or mismatch, or would it **[00:30:33 – 00:30:35]** be a matter now or in the future? **[00:30:35 – 00:30:37]** Thank you. **[00:30:37 – 00:30:41]** Yeah, it's a good point. And I think it's... **[00:30:41 – 00:30:44]** For us, it's mainly about the architecture, right? **[00:30:44 – 00:30:49]** So if we're seemingly benchmarking similar architectures to frontier **[00:30:49 – 00:30:50]** models, then that's good. **[00:30:50 – 00:30:53]** But also, like, we just added Kimi. **[00:30:53 – 00:30:55]** It's like one trillion parameters. **[00:30:55 – 00:30:59]** Yeah, we don't have anything that's like two trillion parameters, **[00:30:59 – 00:31:01]** but we also don't have access to... **[00:31:01 – 00:31:04]** You know, OpenAI's model weight, so it's kind of hard, but **[00:31:04 – 00:31:06]** that is a good point, but yeah. **[00:31:06 – 00:31:11]** The other thing that I would caution is there was a scaling **[00:31:11 – 00:31:13]** of model sizes rapidly. **[00:31:13 – 00:31:16]** OpenAI's GPT 4.5 was over 10 trillion parameters, but **[00:31:16 – 00:31:19]** it was terrible cost and it was way too slow to serve, **[00:31:19 – 00:31:21]** even with disaggregated serving. **[00:31:21 – 00:31:23]** And so then they stepped back to much smaller models. **[00:31:23 – 00:31:26]** So the current models that they serve are less than trillion **[00:31:26 – 00:31:28]** parameters currently. **[00:31:28 – 00:31:31]** That's not the case for Anthropic and Google DeepMind, but at **[00:31:31 – 00:31:34]** least GPT-5.4 is less than a trillion parameters. **[00:31:34 – 00:31:37]** We're always going to have a lag on what the open models have **[00:31:38 – 00:31:42]** versus what the frontier models have in terms of architecture, **[00:31:42 – 00:31:45]** quality, and maybe even size. **[00:31:45 – 00:31:47]** But, you know, NVIDIA recently announced the **[00:31:47 – 00:31:49]** Nematron Collective, which... **[00:31:49 – 00:31:52]** Hopefully, we'll catch the open source up closer to what **[00:31:52 – 00:31:54]** the closed model labs are doing. **[00:31:54 – 00:31:57]** And when those come out, we will implement them, right? **[00:31:57 – 00:32:01]** We implemented MiniMax on day one, and we intend to implement new **[00:32:01 – 00:32:05]** large models on day one as well. **[00:32:08 – 00:32:11]** So I think the pricing angle is quite interesting on InferenceX **[00:32:11 – 00:32:13]** for large kind of cloud use cases. **[00:32:13 – 00:32:16]** Do you see the same metrics and methodology being a framework to **[00:32:16 – 00:32:18]** help enterprises adopt self-hosted and kind of figure out what **[00:32:18 – 00:32:19]** the sweet spot is on break-even? **[00:32:19 – 00:32:24]** Have you thought about maybe going into that business and advisory? **[00:32:25 – 00:32:29]** So currently we offer, we divide the sort of throughput **[00:32:29 – 00:32:32]** by the total cost of ownership, right, so that would just **[00:32:32 – 00:32:34]** be, you know, if you purchase the hardware and you put it **[00:32:34 – 00:32:37]** in a data center you were running, you were renting, and you paid the **[00:32:37 – 00:32:40]** power, you paid the hands, you paid for the networking, etc., and then **[00:32:40 – 00:32:42]** you depreciate it over five years. **[00:32:42 – 00:32:44]** That's currently what's shown there. **[00:32:44 – 00:32:48]** Of course, when you are renting GPUs yourself, you have to **[00:32:48 – 00:32:50]** pay a margin to the provider. **[00:32:50 – 00:32:51]** And when you're an enterprise, you have to have people **[00:32:51 – 00:32:55]** who operate those GPUs and numerous other costs. **[00:32:55 – 00:33:00]** So that's certainly something that semi-analysis is... **[00:33:00 – 00:33:02]** Things like this are what semi-analysis would like to **[00:33:02 – 00:33:07]** do, right, or do already, you know, we don't pay for InferenceX **[00:33:07 – 00:33:11]** via, you know, anyone paying us for InferenceX, we pay for InferenceX **[00:33:11 – 00:33:14]** via all the consulting and advisory services that we offer, **[00:33:14 – 00:33:18]** so I don't mean to do an ad, but... **[00:33:18 – 00:33:21]** It is something that we're exploring, it's not something **[00:33:21 – 00:33:25]** that the engineering team or myself have worked on yet. **[00:33:26 – 00:33:30]** Hey guys, thanks for doing this and I appreciate all the good work. **[00:33:30 – 00:33:35]** Basically, my question is, recently we've seen GPU availability **[00:33:36 – 00:33:38]** basically fall to the floor. **[00:33:38 – 00:33:42]** And I think that's obviously in direct correlation to Anthropic's **[00:33:42 – 00:33:44]** insane revenue growth. **[00:33:44 – 00:33:46]** And I'm just wondering, in the conversations that you **[00:33:46 – 00:33:49]** guys are having with your hyperscaler clients, are they **[00:33:49 – 00:33:50]** buying this as a longer-term thing? **[00:33:51 – 00:33:54]** And is it having an impact on the way that they're thinking **[00:33:54 – 00:33:57]** about returning invested capital? **[00:33:57 – 00:34:03]** And basically, do they think that this GPU availability **[00:34:03 – 00:34:07]** freefall is going to persist? **[00:34:07 – 00:34:10]** And how is that having an impact on the planning and the conversations **[00:34:10 – 00:34:12]** that they're having with you? **[00:34:12 – 00:34:15]** Everyone is buying every GPU they can, and that's **[00:34:15 – 00:34:16]** why NVIDIA is sold out. **[00:34:16 – 00:34:18]** That's why AMD is selling most everything they make, **[00:34:18 – 00:34:22]** and that's why Google TPUs are sold out, etc., etc. **[00:34:23 – 00:34:25]** As far as InferenceX and what it showcases, I'm going to **[00:34:25 – 00:34:28]** pivot to that a little bit. **[00:34:28 – 00:34:32]** What's interesting is because performance has improved across **[00:34:32 – 00:34:35]** the Pareto frontier for models and initially the best model **[00:34:35 – 00:34:39]** we were benchmarking was DeepSeek, but now MiniMax and Kimi are **[00:34:39 – 00:34:43]** better than DeepSeek and MiniMax is cheaper than DeepSeek, right? **[00:34:43 – 00:34:49]** And so when you compare total throughput of tokens of MiniMax **[00:34:49 – 00:34:51]** on a number of GPUs versus DeepSeek, it's higher and **[00:34:51 – 00:34:55]** yet the cost is way lower and each of those tokens is more **[00:34:55 – 00:34:57]** valuable because they're smarter. **[00:34:58 – 00:35:01]** And so that is why GPUs are becoming less available. **[00:35:02 – 00:35:04]** And it's very difficult to rent GPUs at the prices you would get **[00:35:04 – 00:35:06]** them six months ago or a year ago. **[00:35:06 – 00:35:09]** It's because the intelligence per GPU generated is way better **[00:35:09 – 00:35:13]** because new models are coming out that are smarter and those models **[00:35:13 – 00:35:16]** are being optimized more and more and have a better Pareto frontier. **[00:35:16 – 00:35:19]** Even DeepSeek, the cost to serve a year ago versus **[00:35:19 – 00:35:21]** now is significantly lower because performance is **[00:35:21 – 00:35:23]** improved via software. **[00:35:23 – 00:35:25]** I love InferenceX. **[00:35:25 – 00:35:29]** One thing that was really cool in the keynote was Jensen **[00:35:29 – 00:35:32]** kind of formalizing this idea that higher TPS gives you more value, **[00:35:32 – 00:35:34]** and that it's not linear value. **[00:35:34 – 00:35:37]** It's not like, oh, 600 TPS is three times more valuable than 200, **[00:35:37 – 00:35:39]** it actually gets way more valuable. **[00:35:39 – 00:35:42]** And I feel like this creates a very complicated optimization **[00:35:42 – 00:35:46]** problem of, okay, well, if that's the case, where am I actually **[00:35:46 – 00:35:49]** going to be getting the most valuable inference performance? **[00:35:49 – 00:35:52]** I was just curious what you guys are thinking about after **[00:35:52 – 00:35:55]** the keynote in terms of that whole dimension of like revenue **[00:35:55 – 00:35:59]** and value and different value for different points. **[00:35:59 – 00:36:00]** Yeah, that's a good question. **[00:36:00 – 00:36:03]** I actually was talking to some of my colleagues about this **[00:36:03 – 00:36:06]** this morning and we were kind of debating like, okay, let's assume **[00:36:06 – 00:36:09]** we reach AGI, right, whatever that means, right, and tokens **[00:36:09 – 00:36:12]** are of basically infinite value. **[00:36:12 – 00:36:16]** Would it be more valuable to have extremely fast decode, **[00:36:16 – 00:36:20]** so extremely fast generation, or will AGI operate in a way **[00:36:20 – 00:36:23]** where everything is basically batched together and happens async? **[00:36:23 – 00:36:25]** And I guess it kind of depends on how you think of that, right? **[00:36:25 – 00:36:29]** So if everything will happen async, then, you know, general purpose **[00:36:29 – 00:36:33]** GPUs with super high throughput is gonna be, you know, important. **[00:36:33 – 00:36:37]** You want each person to get smarter tokens, i.e. **[00:36:37 – 00:36:40]** faster generation, and yeah, you're going to want extremely **[00:36:40 – 00:36:46]** fast decode, and each person is going to need to have that, but... **[00:36:46 – 00:36:49]** The big thing that Jensen sort of focused on here is, you **[00:36:49 – 00:36:51]** know, GPUs have this curve, right? **[00:36:51 – 00:36:55]** And you can clearly see the sweet spot for GPUs is somewhere **[00:36:55 – 00:36:57]** in this region, right? **[00:36:57 – 00:37:00]** Somewhere in the 40 to 60 tokens per second interactivity, **[00:37:00 – 00:37:05]** or below 60 tokens per second, at least for this model, DeepSeek. **[00:37:05 – 00:37:09]** His whole thing he was advocating is, oh, if we add LPUs, the **[00:37:09 – 00:37:11]** cost floor maybe stops here, right? **[00:37:11 – 00:37:12]** And then it goes across. **[00:37:12 – 00:37:16]** So the terrible, where GPUs really suck, right, really at the **[00:37:16 – 00:37:20]** far end of the curve, that there's no, they're not designed for that. **[00:37:20 – 00:37:22]** And that's why, that's what the whole LPU product that **[00:37:22 – 00:37:24]** he pitched is for is way out here. **[00:37:24 – 00:37:28]** Do you see a world where you guys build out like super **[00:37:28 – 00:37:31]** thorough pre-filled decode Disagg across all different combinations **[00:37:31 – 00:37:34]** of all different kinds of machines? **[00:37:35 – 00:37:37]** I will say yes, and the engineering team will say no. **[00:37:37 – 00:37:42]** Well, no, actually, there was just a few months ago, an **[00:37:42 – 00:37:45]** Amazon engineer opened up for the **[00:37:45 – 00:37:48]** Like for Nick, for the transfer engines, right? **[00:37:48 – 00:37:50]** Yeah, you can probably talk about that more, right? **[00:37:50 – 00:37:53]** Actually, I don't know what you're going to talk about. **[00:37:53 – 00:37:54]** Oh, okay. **[00:37:54 – 00:37:58]** I was going to say, yeah, you can do pre-fill on GPUs and decode **[00:37:58 – 00:37:59]** on training, but yeah, anyway. **[00:37:59 – 00:38:03]** To answer your question, so as we said, we're really thankful **[00:38:03 – 00:38:08]** of all the engineers that collaborated with us at NVIDIA, **[00:38:08 – 00:38:09]** AMD, and in the future. **[00:38:09 – 00:38:12]** Google and Amazon engineers. **[00:38:12 – 00:38:16]** I think they would definitely help us on sweeping all configurations **[00:38:16 – 00:38:20]** where they find useful and where they find would be the best. **[00:38:20 – 00:38:21]** And yeah. **[00:38:21 – 00:38:24]** The nice thing is because we run pre-fill and decode **[00:38:24 – 00:38:26]** separately, you can actually just **[00:38:26 – 00:38:28]** Make your own assumption on that, right? **[00:38:28 – 00:38:31]** You don't need to necessarily run them in combination. **[00:38:31 – 00:38:33]** You can do the pre-fill on Tranium, and you can compare that number to **[00:38:33 – 00:38:38]** the decode that we've run on GPU, and you can compare the pre-fill **[00:38:38 – 00:38:40]** across different chips separately. **[00:38:40 – 00:38:43]** So you kind of can do that already, because they're **[00:38:43 – 00:38:45]** completely disaggregated and it runs over ethernet. **[00:38:45 – 00:38:49]** There's no low-level tying together of these benchmarks.