# From Kernels to Clusters: How PyTorch Powers High-Performance AI (Presented by Meta)

## Abstract

Discover how PyTorch is redefining high-performance AI from the ground up. This session explores PyTorch’s latest advances—from custom kernel authoring and the Helion compiler stack to Monarch’s next-generation distributed runtime for reinforcement learning to Torchcomms’ vendor-neutral, scalable communication layer. We’ll highlight how PyTorch’s robust fault tolerance and elasticity enable reliable training at massive scale, and how these innovations together create a unified, production-ready stack for both research and deployment. Learn why PyTorch is the open foundation for AI innovation across clusters, clouds, and devices.

## AI Summary

- PyTorch, as a foundation for AI, aimed to remain at the center of the AI world by providing a common language for research and production, supporting various hardware and software ecosystems.
- Helion, a domain-specific language for writing kernels, improved developer efficiency through an auto-tuning system that optimized performance across different hardware and input scenarios.
- ExecuTorch, a specialized component of PyTorch, enabled the deployment of models on edge devices, from desktops to embedded systems, ensuring lightweight and efficient runtime execution.

## Transcript

**[00:00:04 – 00:00:09]** Hello everyone, I am Albin, I'm a research engineer at Meta.

**[00:00:09 – 00:00:12]** I've been there for almost seven years now, and I'm also

**[00:00:13 – 00:00:15]** a core maintainer of PyTorch.

**[00:00:15 – 00:00:18]** And so today, in this Meta talk, I'm going to mostly talk about

**[00:00:18 – 00:00:22]** PyTorch actually, and all the cool stuff we've been building there,

**[00:00:22 – 00:00:27]** and a bunch of the new technology that we have exciting for you.

**[00:00:28 – 00:00:30]** So, I wanted to start with something about, you know,

**[00:00:30 – 00:00:32]** what has been happening.

**[00:00:32 – 00:00:36]** For all the people here that have been here for a while, you've

**[00:00:36 – 00:00:39]** seen a lot of changes and you've seen a lot of things happening.

**[00:00:39 – 00:00:40]** And things have been accelerating.

**[00:00:40 – 00:00:45]** I think over the past year, things have gone very crazy

**[00:00:45 – 00:00:47]** for the people that have been here.

**[00:00:47 – 00:00:50]** I guess, almost as long as me, you know, you remember

**[00:00:50 – 00:00:54]** when MLP and convolution neural networks were a hot topic.

**[00:00:54 – 00:00:59]** And we moved to RNN, we moved to attention, and now we have

**[00:00:59 – 00:01:02]** a whole bunch of attention variants that are coming up.

**[00:01:02 – 00:01:04]** And we're getting into agentic AI.

**[00:01:05 – 00:01:08]** So what agentic AI means, I don't know.

**[00:01:08 – 00:01:12]** We're going to figure it out in the next six months, I guess.

**[00:01:12 – 00:01:14]** But, you know, it has created a lot of new things that people

**[00:01:14 – 00:01:17]** need to express when they do machine learning.

**[00:01:17 – 00:01:19]** And PyTorch is always there, you know, when you need to

**[00:01:19 – 00:01:21]** express something new.

**[00:01:21 – 00:01:23]** We need to allow you to do that.

**[00:01:23 – 00:01:26]** And along with all these models, people have completely changed

**[00:01:26 – 00:01:30]** the scale at which they work and how they train their model.

**[00:01:31 – 00:01:35]** We went from relatively basic, fully supervised learning where,

**[00:01:35 – 00:01:38]** you know, you had the data set, you had your samples, you trained

**[00:01:38 – 00:01:40]** for a while, and you were done.

**[00:01:40 – 00:01:42]** And now, well, that doesn't work anymore.

**[00:01:42 – 00:01:44]** Like, you need to actually adapt.

**[00:01:44 – 00:01:45]** You need to go beyond that.

**[00:01:45 – 00:01:47]** We do a lot of reinforcement learning.

**[00:01:47 – 00:01:48]** We do mixed things.

**[00:01:48 – 00:01:51]** We do a synchronous reinforcement learning, continuous learning

**[00:01:51 – 00:01:55]** if, you know, your production system needs to continue learning.

**[00:01:55 – 00:01:59]** And that puts a lot of strain, again, in this agentic world.

**[00:01:59 – 00:02:03]** The second big thing here is we also started to move

**[00:02:03 – 00:02:05]** to a different scale.

**[00:02:05 – 00:02:08]** So again, people that have been here for a while, we

**[00:02:08 – 00:02:10]** started with a bunch of CPUs.

**[00:02:10 – 00:02:13]** You started with a few GPUs.

**[00:02:13 – 00:02:16]** And we went to eight GPUs for the people that were very

**[00:02:16 – 00:02:19]** rich like five years ago.

**[00:02:19 – 00:02:23]** And a couple of years ago, NVL was 72, and now we see

**[00:02:23 – 00:02:28]** things growing even more, like cluster scale, 100,000 more GPUs.

**[00:02:28 – 00:02:31]** Obviously, you could think that this creates a lot of

**[00:02:31 – 00:02:35]** technical problems on how to manage these clusters.

**[00:02:35 – 00:02:39]** We need full tolerance, we need communication layers that are very

**[00:02:39 – 00:02:44]** reliable, we need to create a lot of new technology in that domain.

**[00:02:44 – 00:02:47]** And finally, hardware types.

**[00:02:47 – 00:02:53]** We've been going from CPU and then graphics cards that were not

**[00:02:53 – 00:02:56]** designed to do machine learning, but you know, people made it work.

**[00:02:56 – 00:02:59]** And now we have more and more accelerators that are

**[00:02:59 – 00:03:00]** specialized, right?

**[00:03:00 – 00:03:03]** We have less variability in the models that we run.

**[00:03:03 – 00:03:07]** We can get a lot more performance per watt.

**[00:03:07 – 00:03:11]** And so these accelerators are coming up a lot, and they

**[00:03:11 – 00:03:13]** have a lot of new paradigms, right?

**[00:03:13 – 00:03:16]** In the name of efficiency, people create new ideas, new

**[00:03:16 – 00:03:21]** things in the hardware, like Tensor Core, TMA, CUDA Graph are the

**[00:03:21 – 00:03:24]** kind of things you hear about here.

**[00:03:24 – 00:03:27]** And all of these things add a lot of complexity and you

**[00:03:27 – 00:03:31]** know in in the world of machine learning PyTorch often is here

**[00:03:31 – 00:03:34]** in the middle between all these hardware all these new innovation

**[00:03:34 – 00:03:37]** that come up and our users.

**[00:03:37 – 00:03:41]** And a big thing that the team we have do is like think hard whenever

**[00:03:41 – 00:03:43]** these new Two things happen.

**[00:03:43 – 00:03:47]** We need to make that available for our users, and what do

**[00:03:47 – 00:03:49]** they need to know about, and what should we hide from them?

**[00:03:49 – 00:03:53]** Because they don't need to know about all the complexity.

**[00:03:53 – 00:03:57]** We need to hide a bunch of things, but what should we hide from them?

**[00:03:58 – 00:04:01]** And that's where, funnily enough, we come back to PyTorch core

**[00:04:01 – 00:04:05]** principles from many years ago.

**[00:04:05 – 00:04:09]** With people that have might have seen this one before, but

**[00:04:09 – 00:04:13]** it's usability over performance.

**[00:04:13 – 00:04:15]** So we want to allow innovation.

**[00:04:15 – 00:04:18]** We want to allow people to do anything they need to do

**[00:04:18 – 00:04:20]** on top of PyTorch.

**[00:04:20 – 00:04:22]** And, you know, the progress in machine learning is

**[00:04:22 – 00:04:24]** not a linear thing.

**[00:04:24 – 00:04:29]** And so, you know, gaining 1% performance today is usually

**[00:04:29 – 00:04:33]** not worth it if it's going to make you miss the 10x improvement

**[00:04:33 – 00:04:35]** that's going to happen tomorrow.

**[00:04:35 – 00:04:38]** And so, you know, hunting down the small bits of performance is not

**[00:04:38 – 00:04:42]** always the right thing, You know, depending on your time horizon.

**[00:04:43 – 00:04:46]** Simple over easy is also a very important thing.

**[00:04:46 – 00:04:50]** People often underestimate how expensive it is to have

**[00:04:50 – 00:04:51]** a very complex system.

**[00:04:51 – 00:04:53]** You need a lot of people to maintain it.

**[00:04:53 – 00:04:57]** You need to do a lot of things whenever it breaks to fix it.

**[00:04:57 – 00:04:59]** It takes a lot of time.

**[00:04:59 – 00:05:02]** And for the people here that work in PyTorch, PyTorch is

**[00:05:02 – 00:05:05]** still a very complex system, even though we're striving

**[00:05:05 – 00:05:07]** to keep it as simple as possible.

**[00:05:07 – 00:05:11]** And going forward, this is true both for humans and LLMs.

**[00:05:11 – 00:05:13]** Both of them struggle with that.

**[00:05:13 – 00:05:17]** And actually, that is where the third principle is really

**[00:05:17 – 00:05:20]** good, is because everyone knows Python, you guys know

**[00:05:20 – 00:05:24]** Python, I know Python, Claude knows Python, and so we can actually

**[00:05:24 – 00:05:30]** leverage Python as an ecosystem to build everything we need to do.

**[00:05:30 – 00:05:34]** And people know what's happening there, they know what to expect.

**[00:05:34 – 00:05:37]** And they have, you know, imperative coding.

**[00:05:37 – 00:05:41]** So whenever you write enough code, it actually executes and

**[00:05:41 – 00:05:43]** you do what you expect it to see.

**[00:05:43 – 00:05:44]** It's debuggable, right?

**[00:05:45 – 00:05:48]** You, when something goes wrong, you can look at it and you

**[00:05:48 – 00:05:51]** know what went wrong and you can figure out how to fix that.

**[00:05:51 – 00:05:56]** And it's Pythonic, meaning there's a consistency across

**[00:05:56 – 00:05:59]** all the abstraction and the whole ecosystem you're using

**[00:05:59 – 00:06:01]** because you're not going to use only PyTorch, you're going

**[00:06:01 – 00:06:03]** to use a bunch of other libraries to manage your data, to manage

**[00:06:04 – 00:06:07]** your cluster, all of that, and you can have a lot of consistency

**[00:06:07 – 00:06:10]** in the principles being used there.

**[00:06:11 – 00:06:15]** And so this is basically where we want to go with PyTorch

**[00:06:15 – 00:06:18]** as a foundation, because PyTorch is a foundation now, as like

**[00:06:18 – 00:06:20]** an open language for AI.

**[00:06:20 – 00:06:25]** So to be able to work together and have AI move faster, it

**[00:06:25 – 00:06:29]** is important for all of us to speak the same language.

**[00:06:29 – 00:06:32]** And that is why we're focusing on, like, all the aspects

**[00:06:32 – 00:06:35]** of the AI world, and we're trying to make sure that PyTorch

**[00:06:35 – 00:06:39]** remains at the center and is the thing that everyone knows about

**[00:06:39 – 00:06:40]** and everyone is able to leverage.

**[00:06:40 – 00:06:44]** And when you talk with everyone else about AI, you have a

**[00:06:44 – 00:06:47]** common language where you can make good progress.

**[00:06:47 – 00:06:51]** That is true on the research side, first of all, first

**[00:06:51 – 00:06:55]** and foremost, but also across the ecosystem with hardware vendors,

**[00:06:55 – 00:07:00]** cloud vendors, higher-level libraries, fostering innovation

**[00:07:00 – 00:07:03]** through our work with The Linux Foundation, where we support and

**[00:07:03 – 00:07:07]** host a lot of open-source projects that are innovating around AI,

**[00:07:07 – 00:07:12]** and finally focusing on adoption all the way to AI frontier labs

**[00:07:12 – 00:07:16]** and these very important use cases.

**[00:07:16 – 00:07:19]** And so today, with the little time I have, I'm going to quickly

**[00:07:19 – 00:07:23]** dive into two of these projects that we've been building on.

**[00:07:23 – 00:07:26]** First one is Helion, which is a domain-specific language,

**[00:07:26 – 00:07:30]** which is a special language to write kernels.

**[00:07:30 – 00:07:36]** And the goal there has been to improve developer efficiency.

**[00:07:36 – 00:07:40]** So here on the image you saw, a lot of these languages have

**[00:07:40 – 00:07:43]** this trade-off where you're either very low level and

**[00:07:43 – 00:07:47]** very fast or you're very high level but you don't really

**[00:07:47 – 00:07:51]** know how to get the last drop of performance from the language.

**[00:07:51 – 00:07:55]** And so a lot of DSLs have been designed across the many

**[00:07:55 – 00:07:59]** years to basically strike different parts of that trade-off.

**[00:07:59 – 00:08:02]** Depending on what you want to do, how much performance

**[00:08:02 – 00:08:04]** you need from the kernel, all these kind of things,

**[00:08:04 – 00:08:08]** you're gonna use a different language basically in that stack.

**[00:08:09 – 00:08:12]** With PyTorch being at the top is very, very agnostic.

**[00:08:12 – 00:08:17]** And you know, PTX being the most specialized thing here for CUDA.

**[00:08:17 – 00:08:20]** I'm going to go quickly on this one, but I would say

**[00:08:20 – 00:08:25]** the most interesting part on Helion is really its auto-tuning system.

**[00:08:25 – 00:08:27]** It's the second point I have here.

**[00:08:27 – 00:08:33]** The way we actually try and change the game here is by saying we have

**[00:08:33 – 00:08:35]** all these lower-level language.

**[00:08:35 – 00:08:37]** They have all the knobs we need.

**[00:08:37 – 00:08:40]** The real question is, like, which knob should we use?

**[00:08:40 – 00:08:43]** And that's really the job of our auto-tuning system.

**[00:08:43 – 00:08:47]** That is, automatically find the best thing to use.

**[00:08:47 – 00:08:52]** And that gives us speed, because you're going to find the right

**[00:08:52 – 00:08:54]** pattern from the lower 11 language that you should be

**[00:08:54 – 00:08:57]** using for that particular hardware, for that particular kernel,

**[00:08:58 – 00:09:00]** with these particular inputs.

**[00:09:00 – 00:09:03]** It's going to give you hardware itinerary, because across

**[00:09:03 – 00:09:06]** different versions of the hardware, when the hardware

**[00:09:06 – 00:09:11]** evolves, the auto-tuner is going to be able to pick the right choice.

**[00:09:11 – 00:09:14]** And it improves your productivity, because you wrote one kernel,

**[00:09:14 – 00:09:16]** and that one kernel is fast across the board.

**[00:09:16 – 00:09:21]** You don't have to rewrite a kernel again and again.

**[00:09:21 – 00:09:23]** We have a booth and we are doing some demo if you want

**[00:09:23 – 00:09:27]** to look at the code, what it looks like, what is the result.

**[00:09:27 – 00:09:29]** Our team is at the booth and we'll be very happy

**[00:09:29 – 00:09:32]** to show you all of that.

**[00:09:32 – 00:09:35]** And on the complete other end of the spectrum, the other

**[00:09:35 – 00:09:38]** new project, big project we have, it's not really new,

**[00:09:38 – 00:09:42]** it's been in the works for a while, is on the edge side.

**[00:09:43 – 00:09:47]** So I don't know who here is trying to deploy models outside

**[00:09:47 – 00:09:51]** of servers, but you will know it's kind of a fun time.

**[00:09:51 – 00:09:55]** You start from desktop laptop, where it looks relatively

**[00:09:55 – 00:09:57]** similar to what you're used to.

**[00:09:57 – 00:09:59]** And then you move on to phones and glasses.

**[00:09:59 – 00:10:01]** You don't have as much memory.

**[00:10:01 – 00:10:02]** Compute goes down.

**[00:10:02 – 00:10:04]** Floating points go away.

**[00:10:05 – 00:10:07]** And then you get to wearable and embedded system.

**[00:10:07 – 00:10:10]** And basically, everything is an integer.

**[00:10:10 – 00:10:12]** You have no idea what to do about numbers.

**[00:10:12 – 00:10:15]** It's kind of crazy.

**[00:10:15 – 00:10:18]** And so ExecuTorch is the piece of PyTorch that is

**[00:10:18 – 00:10:22]** specialized to run things on all of these edge devices.

**[00:10:22 – 00:10:24]** So starting from the top, where you have your PyTorch model,

**[00:10:24 – 00:10:26]** the first step is always export.

**[00:10:26 – 00:10:30]** So going from your PyTorch model to something the rest of the system

**[00:10:30 – 00:10:32]** is going to be able to work on.

**[00:10:32 – 00:10:37]** And funnily enough, it is a very hard piece in PyTorch,

**[00:10:37 – 00:10:39]** because PyTorch is so flexible.

**[00:10:39 – 00:10:42]** We try to be very PyTonic, we try to allow you to do whatever,

**[00:10:42 – 00:10:44]** but that means that when you need to get in a more constrained

**[00:10:44 – 00:10:48]** environment, it is a big challenge.

**[00:10:48 – 00:10:52]** The middle box there is then how do we lower things and

**[00:10:52 – 00:10:53]** how do we optimize things.

**[00:10:53 – 00:10:57]** And that's where you specialize, you optimize, you quantize, you

**[00:10:57 – 00:11:00]** do all these things that are needed to have your model being small

**[00:11:01 – 00:11:05]** enough to run on the device, have the model being able to use the

**[00:11:05 – 00:11:09]** hardware that you're going to have available to its full potential.

**[00:11:09 – 00:11:12]** And the last piece is going to be the runtime where your runtime

**[00:11:12 – 00:11:17]** needs to be very lightweight and it needs to run anywhere across all of

**[00:11:17 – 00:11:20]** these things that we talked about.

**[00:11:20 – 00:11:24]** So today PyTorch we say is production ready, meaning

**[00:11:24 – 00:11:27]** like basically all of Meta wearables that you might have

**[00:11:27 – 00:11:28]** seen run on ExecuTorch.

**[00:11:28 – 00:11:31]** So trust me, it works pretty well.

**[00:11:31 – 00:11:34]** And so we're very happy to have more and more people

**[00:11:34 – 00:11:36]** using it in the industry as well.

**[00:11:37 – 00:11:42]** If you come to the booth, we are doing a super fun demo on the voice

**[00:11:42 – 00:11:47]** and the transcription with Parakeet and other models and we're super

**[00:11:47 – 00:11:51]** happy to be working with partners like LM Studio to do these kind

**[00:11:51 – 00:11:54]** of things to, you know, through PyTorch, make them available

**[00:11:54 – 00:11:58]** to hopefully more and more people.

**[00:11:58 – 00:12:02]** And there is a lot more that is happening in that stack.

**[00:12:02 – 00:12:05]** The way I think about all of these things is usually, you

**[00:12:05 – 00:12:10]** know, the core PyTorch box, that is the small box at the bottom,

**[00:12:10 – 00:12:14]** is kind of the slow moving piece.

**[00:12:14 – 00:12:16]** And you know, there, the name of the game is stability,

**[00:12:16 – 00:12:18]** correctness, you know.

**[00:12:18 – 00:12:20]** We want things to work and continue working, right?

**[00:12:20 – 00:12:21]** It's a shared language.

**[00:12:21 – 00:12:25]** We don't want you to have to redo the language every single day.

**[00:12:25 – 00:12:29]** And then around that, we have a very healthy ecosystem that

**[00:12:29 – 00:12:33]** we built through the foundation of ecosystem projects and

**[00:12:33 – 00:12:38]** platform projects, where we want to allow innovation to strive and have

**[00:12:38 – 00:12:40]** more and more things happening.

**[00:12:40 – 00:12:44]** In particular, we're looking for projects that are

**[00:12:44 – 00:12:46]** building more scale.

**[00:12:46 – 00:12:50]** So the big important ones here are Torchcomms that you

**[00:12:50 – 00:12:51]** might have heard about.

**[00:12:52 – 00:12:53]** That is very interesting.

**[00:12:53 – 00:12:57]** So it's like a way to organize communication layer and to

**[00:12:57 – 00:13:01]** abstract over a communication layer for distributed systems

**[00:13:02 – 00:13:06]** that go for much larger scale and heterogeneities.

**[00:13:06 – 00:13:12]** We have ethereogenous hardware and communications and we have Monarch

**[00:13:12 – 00:13:17]** which is an orchestrator which is basically you have a single

**[00:13:17 – 00:13:20]** process that is able to schedule a whole bunch of things happening

**[00:13:21 – 00:13:23]** in such a way that is going to be a lot more fault-tolerant,

**[00:13:23 – 00:13:28]** less problematic and you'll be able to scale to much bigger things.

**[00:13:29 – 00:13:33]** And at the top here, you can see TorchTitan, which is what

**[00:13:33 – 00:13:38]** I would call our showcase on how do you do these large-scale jobs in

**[00:13:38 – 00:13:40]** very much a PyTorch native level.

**[00:13:40 – 00:13:42]** So we do pre-training, mid-training,

**[00:13:42 – 00:13:44]** post-training, all of that.

**[00:13:44 – 00:13:48]** We do some very cool experiments at a relatively large scale.

**[00:13:48 – 00:13:52]** We leverage vLLM, which is a PyTorch Foundation project,

**[00:13:52 – 00:13:58]** and OpenEnv, with partnership with HuggingFace and many others there.

**[00:13:58 – 00:14:04]** And it allows us to show you how we see this world of like

**[00:14:04 – 00:14:08]** a PyTorch native training stack for you to fork and then build

**[00:14:08 – 00:14:13]** your own and, you know, really be able to build what you want

**[00:14:13 – 00:14:18]** to build on top of PyTorch using the same language we're all using.

**[00:14:18 – 00:14:24]** And the last one below is then hardware hardware ready

**[00:14:24 – 00:14:28]** through Helion, through Torchcomms, through ExecuTorch, through vLLM.

**[00:14:28 – 00:14:32]** You know, the hardware for machine learning are evolving fast.

**[00:14:32 – 00:14:37]** You know, a new version of Rubin has been announced here.

**[00:14:37 – 00:14:40]** It has completely new paradigms on how you're going to run compute.

**[00:14:40 – 00:14:42]** It's getting more and more specialized.

**[00:14:42 – 00:14:44]** It looks almost like an accelerator.

**[00:14:44 – 00:14:47]** So we need to have the flexibility to be able to target all this

**[00:14:47 – 00:14:51]** hardware and we cannot afford to rebuild the stack from

**[00:14:51 – 00:14:53]** scratch every time.

**[00:14:54 – 00:14:55]** So, yeah, I'm going to stop here.

**[00:14:56 – 00:14:58]** I could go for another hour.

**[00:14:58 – 00:15:04]** I would invite you to join us at our booth, which is right there.

**[00:15:04 – 00:15:07]** We actually have so many topics, we have a whole schedule.

**[00:15:07 – 00:15:10]** So if you go there, we have a schedule of all the things

**[00:15:10 – 00:15:14]** you can talk about with different times from open source, how to

**[00:15:14 – 00:15:17]** create projects, how to contribute.

**[00:15:17 – 00:15:22]** To Kernel Authoring, Distributed System, Compiler, Python Packaging,

**[00:15:22 – 00:15:25]** OpenAuth, like all of these kind of things we do.

**[00:15:25 – 00:15:28]** So come over, check the times, and meet the people.