# Building Open Models That Developers Can Inspect, Adapt, and Deploy

## Abstract

Key takeaways Understand how open models are built—from model architectures and training data to weights, post-training recipes, and evaluation. Learn how developers can inspect, adapt, and deploy open models for domain-specific applications across language, physical AI, biology, science, and robotics. Explore why open data, open weights, and repeatable recipes matter for transparency, sovereignty, regulatory readiness, and production deployment.

## AI Summary

- The presentation highlighted the evolution of AI models from simple token prediction to reasoning and tool use, emphasizing the importance of open models for developers.
- The speaker discussed the benefits of using open models, such as the ability to inspect, fine-tune, control deployments, and manage costs, especially relevant for companies in regions with strict data regulations.
- The presentation covered the availability of millions of pre-trained models and RL environments on the Hugging Face Hub, providing a strong starting point for companies to adapt models to their specific tasks.
- The speaker addressed the challenges of training open models, including the need to define verifiers and the importance of balancing model size, data quality, and test-time compute.
- The presentation concluded with a discussion on the potential risks of open models, which are similar to closed models, primarily centered around alignment issues.
- The speaker encouraged the audience to share their own models and environments on the Hugging Face Hub to contribute to the collective effort in advancing AI.

## Transcript

**[00:00:05 – 00:00:06]** Hello, everyone.

**[00:00:07 – 00:00:10]** Welcome to the final day of the first day of GTC.

**[00:00:10 – 00:00:12]** I hope you're all having a great time.

**[00:00:12 – 00:00:17]** My name's Eric Pounds, and I'm on the Nemotron team at NVIDIA.

**[00:00:17 – 00:00:20]** And this session, Building Open Models that Developers

**[00:00:20 – 00:00:24]** Can Expect, Adapt, and Deploy, is really exciting.

**[00:00:24 – 00:00:28]** We have a special guest, and we'll explore why sharing

**[00:00:28 – 00:00:31]** and training recipes matter and how developers can post-train

**[00:00:31 – 00:00:34]** open models for their own tasks.

**[00:00:34 – 00:00:37]** And I'm personally very excited for this session.

**[00:00:37 – 00:00:40]** Many of you that are developers in the room know that on Hugging

**[00:00:40 – 00:00:43]** Face, there's over 3 million models currently today and

**[00:00:43 – 00:00:45]** 1 million data sets.

**[00:00:45 – 00:00:48]** It is the place where you go to get open models and

**[00:00:48 – 00:00:49]** tools and work with them.

**[00:00:49 – 00:00:53]** And we're delighted to welcome Thomas Wolf, the

**[00:00:53 – 00:00:56]** co-founder and chief science officer at Hugging Face.

**[00:00:56 – 00:00:58]** And Thomas, thank you very much for being on our pregame

**[00:00:58 – 00:01:00]** today before the keynote.

**[00:01:00 – 00:01:04]** Thomas helped start Hugging Face's open source, open science,

**[00:01:04 – 00:01:05]** and robotics efforts.

**[00:01:05 – 00:01:10]** His work includes creating or advising Hugging Face transformers,

**[00:01:10 – 00:01:15]** data sets, small agents, and LeRobot, alongside the small

**[00:01:15 – 00:01:20]** LLM family, small LLM, small VLM, and small VLA, making

**[00:01:20 – 00:01:24]** research models and data sets more accessible to developers.

**[00:01:24 – 00:01:27]** And we'll reserve the final few minutes for Q&A, so please

**[00:01:27 – 00:01:32]** hold those, and we'd like to welcome to the stage Thomas Wolf.

**[00:01:34 – 00:01:37]** Thanks, Eric.

**[00:01:37 – 00:01:40]** Thanks everyone, thanks Eric,

**[00:01:40 – 00:01:45]** thanks for sticking out with me in this long day, the keynote

**[00:01:45 – 00:01:47]** yesterday, this morning, for those of you who were in the

**[00:01:47 – 00:01:50]** room and saw some microdeck.

**[00:01:50 – 00:01:52]** Today we are talking about something different,

**[00:01:52 – 00:01:54]** which is open models.

**[00:01:54 – 00:01:55]** Why should you use them?

**[00:01:55 – 00:01:59]** It's a good question.

**[00:02:02 – 00:02:06]** At least we hope we can talk about that.

**[00:02:11 – 00:02:16]** So the thing is lighting up but doesn't move the slides.

**[00:02:19 – 00:02:21]** Yes, OK, thanks.

**[00:02:22 – 00:02:24]** Let's get us started.

**[00:02:25 – 00:02:27]** Well, building a model has changed a lot in the past

**[00:02:28 – 00:02:33]** five years, for those of you who've been around since COVID, I guess.

**[00:02:33 – 00:02:36]** We started back in the day, we were just predicting next token.

**[00:02:36 – 00:02:37]** Everyone was amazed about that.

**[00:02:37 – 00:02:42]** And people were scared about GPT-2, writing news articles, and

**[00:02:42 – 00:02:44]** maybe we should not release this.

**[00:02:44 – 00:02:47]** Well, it turned out it was still fine.

**[00:02:47 – 00:02:50]** And the next challenge was to get it actually to do what you wanted,

**[00:02:50 – 00:02:52]** and not just to generate models.

**[00:02:52 – 00:02:56]** So yeah, so that was the challenge of following instruction.

**[00:02:56 – 00:02:59]** And for this, we still had pretty small context length.

**[00:02:59 – 00:03:01]** For those who remember, we had like 1,000 or 2,000 tokens

**[00:03:02 – 00:03:04]** in one context length and could basically answer a question.

**[00:03:04 – 00:03:07]** That was the first GPT-3.

**[00:03:07 – 00:03:10]** The breakthrough really happened when we started to have reasoning

**[00:03:10 – 00:03:14]** model 01, basically now one year, well, two years ago

**[00:03:14 – 00:03:18]** now, I guess, almost exactly, actually, when models were

**[00:03:18 – 00:03:22]** able to think before answering, and we saw this explosion of context

**[00:03:22 – 00:03:26]** length, which, unfortunately, because we pay by token, also

**[00:03:26 – 00:03:28]** meant an explosion of costs.

**[00:03:28 – 00:03:30]** But yeah, that's what comes with it.

**[00:03:30 – 00:03:33]** And the main challenge, I mean, the main next, the main

**[00:03:33 – 00:03:36]** last revolution happened this year with tool use, where the model

**[00:03:36 – 00:03:42]** were finally able to use tool for real, and we finally had agents,

**[00:03:42 – 00:03:45]** we've been promised agents for ages, but they really started to

**[00:03:45 – 00:03:48]** work in 2026, and now we actually need, because we have all these

**[00:03:49 – 00:03:53]** tools, we need this like million Context length in our models.

**[00:03:53 – 00:03:57]** Now, why should you care about an open model?

**[00:03:57 – 00:03:58]** Well, there's a couple of reasons.

**[00:03:59 – 00:04:04]** I highlight the four I really like, and I think they kind of

**[00:04:04 – 00:04:05]** summarize, basically, the reason.

**[00:04:05 – 00:04:08]** First one is sometimes you want to inspect the model.

**[00:04:08 – 00:04:10]** Like, it can be because, I don't know, your model is

**[00:04:10 – 00:04:11]** doing bad things.

**[00:04:11 – 00:04:13]** I don't know, you're open AI, you're attack hugging face.

**[00:04:13 – 00:04:16]** You may want to inspect what your model has been doing.

**[00:04:16 – 00:04:18]** So that's one thing you might want to do.

**[00:04:18 – 00:04:21]** And it's easy if you're the model provider.

**[00:04:21 – 00:04:24]** But if you're not, you may rather want to have an open model.

**[00:04:24 – 00:04:26]** So in the future where you might be responsible for what

**[00:04:26 – 00:04:29]** your model is doing, you actually want to know what it's doing.

**[00:04:29 – 00:04:31]** The second reason is you might want to fine-tune it.

**[00:04:31 – 00:04:33]** You might want to adapt it to your task.

**[00:04:33 – 00:04:37]** Maybe it's not a task that's yet covered by all these

**[00:04:37 – 00:04:40]** pre-training data sets that Anthropic and OpenAI has.

**[00:04:40 – 00:04:42]** Another one that I really like is maybe you want to

**[00:04:42 – 00:04:43]** control the deployments.

**[00:04:43 – 00:04:46]** Maybe you don't want to send all your company data to Anthropic

**[00:04:46 – 00:04:48]** or OpenAI or Google.

**[00:04:48 – 00:04:51]** Maybe you want to keep part of this private.

**[00:04:51 – 00:04:55]** You may also want to control costs and run it on your own hardware.

**[00:04:55 – 00:04:59]** And the last part, kind of relevant in Europe, is when

**[00:04:59 – 00:05:02]** Fable is suddenly not accessible to non-US citizens.

**[00:05:02 – 00:05:04]** Well, that's a big problem when you're in Germany or

**[00:05:04 – 00:05:06]** France, or in the UK, whatever.

**[00:05:06 – 00:05:10]** So, as long as you're not running your own model, someone

**[00:05:10 – 00:05:13]** has a key or switch on your tech, and this someone can

**[00:05:13 – 00:05:17]** be a government or a company.

**[00:05:17 – 00:05:19]** Now, what does open mean?

**[00:05:20 – 00:05:21]** It kind of means many different things, and

**[00:05:21 – 00:05:24]** people have been fighting against this for a long time.

**[00:05:24 – 00:05:27]** I don't think it matters that much what you call it, but it's

**[00:05:27 – 00:05:30]** good to know exactly what it means.

**[00:05:30 – 00:05:33]** And typically each range from, you know, just releasing the

**[00:05:33 – 00:05:35]** weights, but basically there is a non-commercial license.

**[00:05:35 – 00:05:38]** So if you're any commercial team, it's kind of hard to use that,

**[00:05:38 – 00:05:41]** but it's still very useful for researchers because they can, you

**[00:05:41 – 00:05:43]** know, work on top of these models.

**[00:05:43 – 00:05:45]** Sometimes you have a permissive license, so

**[00:05:45 – 00:05:46]** it's way more easy to use.

**[00:05:46 – 00:05:50]** Typically Apache, like DeepSeq is like Apache model, so anyone

**[00:05:50 – 00:05:52]** can use it and change it and do whatever they want.

**[00:05:53 – 00:05:56]** It can go then into the next step, and the next step I would say

**[00:05:56 – 00:05:58]** there's only a couple of companies in the world who really does that.

**[00:05:58 – 00:06:04]** I mean, Hugging Face is one, NVIDIA is another, is when also the team,

**[00:06:04 – 00:06:07]** you know, building models is also sharing the code and the recipe.

**[00:06:07 – 00:06:10]** And this is a very strong move because then everyone can retrain

**[00:06:10 – 00:06:12]** a model that's similar to that.

**[00:06:12 – 00:06:17]** And the full part, what we call fully open, I think there's

**[00:06:17 – 00:06:20]** only AI2, NVIDIA, and HuggingFace who have been doing that up

**[00:06:20 – 00:06:21]** to now, is you share everything.

**[00:06:22 – 00:06:25]** You share the code, you share the recipe, you share the data set, you

**[00:06:25 – 00:06:29]** share the checkpoints of training, the evaluation, et cetera.

**[00:06:29 – 00:06:32]** I think all of these have a use, a very important use,

**[00:06:32 – 00:06:36]** and already with open weights you can get a lot of the advantages I

**[00:06:36 – 00:06:39]** was listing in the previous slide.

**[00:06:39 – 00:06:42]** Now, are open models even competitive?

**[00:06:42 – 00:06:46]** Well, it turns out they are, if you're ready to wait a

**[00:06:46 – 00:06:47]** couple of months.

**[00:06:47 – 00:06:49]** So right now, the frontier, the open frontier, definitely

**[00:06:49 – 00:06:51]** is not years beyond.

**[00:06:51 – 00:06:54]** It's, I would say, a couple of months, so typically estimation

**[00:06:54 – 00:06:56]** around six to nine months.

**[00:06:56 – 00:07:00]** So the gap fluctuates with each model release.

**[00:07:00 – 00:07:02]** We just get a bunch of model releases that actually I don't

**[00:07:02 – 00:07:06]** have here, which are deep seek and mistral.

**[00:07:06 – 00:07:08]** It's going to come soon.

**[00:07:09 – 00:07:12]** But yeah, typically six to nine months is the good rule of thumb

**[00:07:12 – 00:07:14]** you can keep in mind, basically.

**[00:07:14 – 00:07:16]** So right now, we're in October, so we basically have open

**[00:07:16 – 00:07:22]** models to the power of the model of April, so GPT-5.4 Pro.

**[00:07:26 – 00:07:29]** Now, they're pretty close, that's what you can get.

**[00:07:29 – 00:07:32]** Now, what exactly gets shared with an open model?

**[00:07:32 – 00:07:35]** Just like we saw with the level you can open source, there

**[00:07:35 – 00:07:39]** is basically an associated stack that can be more or less open.

**[00:07:39 – 00:07:42]** And typically, the most common is inference.

**[00:07:42 – 00:07:43]** So we get the weights.

**[00:07:43 – 00:07:45]** You can do inference with it.

**[00:07:45 – 00:07:46]** We'll have some code base.

**[00:07:46 – 00:07:49]** If you're following yesterday, DeepSeq released their ultra

**[00:07:49 – 00:07:51]** fast inference code base.

**[00:07:51 – 00:07:53]** So these are examples of things that this team released,

**[00:07:54 – 00:07:57]** and they are very useful to basically use the model.

**[00:07:57 – 00:08:00]** Reproducibility is my personal favorite because

**[00:08:00 – 00:08:01]** I'm a former scientist.

**[00:08:01 – 00:08:04]** So I like being able to reproduce things, understand, you know,

**[00:08:04 – 00:08:05]** I'm like Feynman.

**[00:08:05 – 00:08:08]** If I cannot reproduce something, I don't understand it.

**[00:08:08 – 00:08:10]** So I like this. So it means I like when people share how

**[00:08:10 – 00:08:12]** they train it, basically.

**[00:08:12 – 00:08:15]** The data is the rarest thing to share because, of course,

**[00:08:15 – 00:08:19]** there's a lot of... No question there; it's usually the ultimate

**[00:08:19 – 00:08:21]** secret recipe that people keep.

**[00:08:21 – 00:08:23]** And of course we also have copyright laws that might

**[00:08:23 – 00:08:28]** be quite complex and are not settled at all right now.

**[00:08:29 – 00:08:33]** Now, why does this matter for a company?

**[00:08:33 – 00:08:36]** Well, I think when you're a company, you can differentiate.

**[00:08:36 – 00:08:39]** When you're, I would say, an AI-native company or a

**[00:08:39 – 00:08:42]** company that wants to embrace AI, that's one strong way

**[00:08:42 – 00:08:44]** you can differentiate yourself.

**[00:08:44 – 00:08:47]** I mean, an easy way, there may be a sketchy way to say

**[00:08:47 – 00:08:50]** that, is a lot of companies are going to be a wrapper of a big lab.

**[00:08:50 – 00:08:52]** And if you don't want to be a wrapper, you basically need

**[00:08:52 – 00:08:53]** to be an AI builder.

**[00:08:53 – 00:08:56]** So your company needs to understand how you can train.

**[00:08:56 – 00:09:00]** So how it works is basically...

**[00:09:00 – 00:09:03]** The nice thing is, because of places like Hugging Face,

**[00:09:03 – 00:09:06]** a lot of people are sharing pre-trained models.

**[00:09:06 – 00:09:08]** So you don't have to spend a billion dollars starting

**[00:09:08 – 00:09:11]** from scratch, which would be totally impossible for

**[00:09:11 – 00:09:12]** most companies, right?

**[00:09:12 – 00:09:15]** And I did, actually, this weekend, a back-up the envelope computation,

**[00:09:15 – 00:09:19]** and the pre-training costs that people have been sharing,

**[00:09:19 – 00:09:21]** if you sum all the pre-training costs of all the models in Hugging

**[00:09:21 – 00:09:25]** Face, you reach about a billion dollars of free compute that people

**[00:09:25 – 00:09:26]** are kind of giving in a way, right?

**[00:09:26 – 00:09:29]** They train this model and then they open source it.

**[00:09:29 – 00:09:32]** So you have this pre-trained model that the expensive part

**[00:09:32 – 00:09:34]** has been like pre-training them, and then the part that's rest

**[00:09:35 – 00:09:36]** for you is actually pretty cheap.

**[00:09:36 – 00:09:38]** And I think a lot of people underestimate how cheap it can be.

**[00:09:38 – 00:09:41]** It's kind of post-training this model on your domain.

**[00:09:41 – 00:09:43]** So you have your domain data, you have maybe your

**[00:09:43 – 00:09:44]** domain environments.

**[00:09:44 – 00:09:46]** I'm going to talk about that later.

**[00:09:46 – 00:09:49]** And you use this to actually tailor the model to your application.

**[00:09:49 – 00:09:52]** So you get a smaller model that's tailored for your domain,

**[00:09:52 – 00:09:57]** but that's also very, very efficient in a way.

**[00:09:57 – 00:10:00]** There is two companies who recently talked about it.

**[00:10:00 – 00:10:02]** I just select these two ones because I know them well,

**[00:10:02 – 00:10:04]** but there's actually a lot more.

**[00:10:04 – 00:10:08]** One is RAMP, which is the famous fintech in the US.

**[00:10:08 – 00:10:10]** It's been growing very, very fast.

**[00:10:10 – 00:10:14]** So in their case, they have a task that's pretty.

**[00:10:14 – 00:10:17]** Fundamentally in fintech, in this case, like receipt

**[00:10:17 – 00:10:20]** extraction, so they get the receipt and they need to extract, you

**[00:10:20 – 00:10:23]** know, what was paid to who, etc.

**[00:10:23 – 00:10:26]** And they were able to basically fine-tune it very cheap, one

**[00:10:26 – 00:10:29]** epoch of SFT, so SFT, we're going to see that later if you want, but

**[00:10:29 – 00:10:32]** basically 10,000 examples, and they get something that's way better

**[00:10:32 – 00:10:35]** than Claude Haiku or GPT-6 Luna.

**[00:10:35 – 00:10:36]** Anyway, for receipts extraction, you don't

**[00:10:36 – 00:10:38]** want to use like OptiX 5.5.

**[00:10:38 – 00:10:40]** That's way too overkill in any case.

**[00:10:40 – 00:10:43]** But even if you compare to the very cheap, the cheapest

**[00:10:43 – 00:10:45]** alternative of the big labs, you can match them, and then you

**[00:10:45 – 00:10:49]** have full control of wherever the costs are or where you deploy that.

**[00:10:49 – 00:10:51]** Another one is trajectory.

**[00:10:51 – 00:10:54]** So Trajectory is one of these companies helping your teams

**[00:10:54 – 00:10:56]** fine-tune their model if you want, and here they gave an

**[00:10:56 – 00:11:00]** example of what they did with Harvey, and in this case was

**[00:11:00 – 00:11:03]** one actual internal benchmark that Harvey has, well, actually, sorry,

**[00:11:03 – 00:11:06]** this I think has been also released externally, which is called

**[00:11:06 – 00:11:08]** the Harvey Legal Agent Bench.

**[00:11:08 – 00:11:11]** And basically, you see that you can reach the top performance

**[00:11:12 – 00:11:13]** for a fraction of the price.

**[00:11:13 – 00:11:17]** So here is like 12 times cheaper, and basically, you match your

**[00:11:17 – 00:11:20]** top closest performance on this legal agent bench, which

**[00:11:20 – 00:11:23]** is, if you take a look at it, it's very wide-ranging, actually.

**[00:11:23 – 00:11:25]** It's the opposite of receipt extraction.

**[00:11:25 – 00:11:28]** That's very narrow. Like, the legal bench is super broad.

**[00:11:28 – 00:11:34]** As I told you, the way to start this mission, if you

**[00:11:34 – 00:11:39]** want to do it, is to start from an open model and an open environment.

**[00:11:45 – 00:11:47]** Well, luckily, we have the Hugging Face Hub, so they're

**[00:11:47 – 00:11:49]** all available there.

**[00:11:49 – 00:11:52]** The models, I know that most of you, if you know Hugging Face,

**[00:11:52 – 00:11:54]** know that you can find models.

**[00:11:54 – 00:11:56]** I think something that less people know about is that you can also

**[00:11:57 – 00:11:59]** find RL environments on the Hub.

**[00:11:59 – 00:12:02]** There is a lot of them. We'll go over that later.

**[00:12:02 – 00:12:04]** There's actually several millions of them.

**[00:12:04 – 00:12:10]** And these are maybe now the shovels to do this new way

**[00:12:10 – 00:12:15]** of doing AI, which is training in RL environments.

**[00:12:17 – 00:12:19]** Now, how did open model get that good?

**[00:12:19 – 00:12:22]** So here I do a little bit of history.

**[00:12:22 – 00:12:24]** I've been in the field for 10 years, so I can't really

**[00:12:24 – 00:12:26]** avoid doing a little bit of history and talking about

**[00:12:26 – 00:12:28]** old things, I'm sorry.

**[00:12:28 – 00:12:31]** But how did we get there? Well, it started just after COVID when we

**[00:12:31 – 00:12:33]** got this scaling law graph, 2021.

**[00:12:33 – 00:12:38]** And basically what the scaling law was saying is that, you

**[00:12:38 – 00:12:41]** can predict how good your model is gonna be, which is

**[00:12:41 – 00:12:45]** measured typically during training by the loss, In function of compute

**[00:12:45 – 00:12:49]** and data, and it also told us that basically, you know, we trained a

**[00:12:49 – 00:12:52]** little bit, we did the beginning of the black curve here, but there's

**[00:12:52 – 00:12:56]** all, you know, sign to believe that this linear trend can go way,

**[00:12:56 – 00:12:59]** way lower, way lower to the bottom right, which means if you add

**[00:12:59 – 00:13:03]** compute, or if you add data, or if you increase the size of the model,

**[00:13:03 – 00:13:05]** you just get better performances.

**[00:13:05 – 00:13:08]** So the next steps was basically increasing data and increasing

**[00:13:08 – 00:13:11]** compute to get better performances.

**[00:13:11 – 00:13:14]** So we saw the pre-training data set grow a lot.

**[00:13:14 – 00:13:17]** So in the past, you know, like a billion tokens was

**[00:13:17 – 00:13:19]** something gigantic.

**[00:13:19 – 00:13:22]** For people who read the BERT paper, like Google back in the day, they

**[00:13:22 – 00:13:23]** saw we're pushing to the limits.

**[00:13:23 – 00:13:27]** Basically, we're going to train this massive, massive 300 million

**[00:13:27 – 00:13:31]** parameters model on this data set of a couple hundred million tokens.

**[00:13:31 – 00:13:33]** And now we passed, we're way past that.

**[00:13:33 – 00:13:36]** We're around, I mean, this is an estimation, but the estimation

**[00:13:36 – 00:13:40]** pre-training data set for closed model and also actually open

**[00:13:40 – 00:13:44]** models, they are not so far from that, is around 50 trillion tokens.

**[00:13:44 – 00:13:48]** Part of it is the internet, part of it is your user data,

**[00:13:48 – 00:13:53]** and part of it is also increasingly some form of synthetic data.

**[00:13:56 – 00:13:58]** Well, that's data.

**[00:13:58 – 00:14:01]** There is another way, which is, like scaling, to increase

**[00:14:01 – 00:14:01]** the size of the model.

**[00:14:01 – 00:14:03]** And we saw the size of the model also increasing

**[00:14:03 – 00:14:05]** strongly over the year.

**[00:14:05 – 00:14:06]** So I was telling about BERT.

**[00:14:06 – 00:14:10]** BERT is actually this very first point on the bottom left.

**[00:14:10 – 00:14:12]** It is 300 million parameters.

**[00:14:12 – 00:14:18]** And we've seen this increasing in size of the model,

**[00:14:18 – 00:14:19]** basically, over time.

**[00:14:19 – 00:14:22]** Right now, we estimate that.

**[00:14:22 – 00:14:25]** Oh, by the way, the vertical axis here is performance,

**[00:14:25 – 00:14:27]** and the horizontal is parameters.

**[00:14:27 – 00:14:30]** So that's not over time, but this one is actually how much better

**[00:14:30 – 00:14:33]** you get when your model is larger.

**[00:14:33 – 00:14:36]** Sorry for that, that's maybe confusing slightly.

**[00:14:36 – 00:14:38]** So basically the idea here is you want to go larger,

**[00:14:38 – 00:14:42]** you want to go to larger models, but you also don't have an

**[00:14:42 – 00:14:47]** automatic advantage to just simply make your model large.

**[00:14:47 – 00:14:48]** So you have...

**[00:14:48 – 00:14:50]** The model here on the bottom right is a very bad model

**[00:14:50 – 00:14:53]** because they are both extremely large and also extremely bad.

**[00:14:53 – 00:14:54]** So you don't want to be there.

**[00:14:54 – 00:14:57]** You want to stay on this Pareto frontier here.

**[00:14:57 – 00:14:59]** And you see like right now, I would say most of the Pareto

**[00:14:59 – 00:15:03]** frontier in terms of open model has been Chinese model.

**[00:15:03 – 00:15:06]** From DeepSeq is very famous and Quern are two very famous to

**[00:15:06 – 00:15:12]** be both, you know, extremely good and extremely efficient in a way.

**[00:15:13 – 00:15:17]** Well, there is another way also, which is training compute.

**[00:15:17 – 00:15:19]** So you can scale your compute.

**[00:15:19 – 00:15:23]** And here, that's pretty much almost linear.

**[00:15:23 – 00:15:24]** That's maybe not linear.

**[00:15:24 – 00:15:27]** But there is a very clear increase in training compute

**[00:15:27 – 00:15:32]** versus custom ECI is this way of comparing models over

**[00:15:32 – 00:15:37]** a differing benchmark from Epoch.

**[00:15:37 – 00:15:40]** The last way we saw this increase is through context window.

**[00:15:40 – 00:15:43]** I talked a little bit about that, and we've seen steadily increasing

**[00:15:43 – 00:15:46]** context window from about 1,000 token context window to now we

**[00:15:46 – 00:15:52]** have like 1 million token is like a standard, even in open models.

**[00:15:52 – 00:15:56]** And there is some attempt to go even wider, 10 million.

**[00:15:56 – 00:15:58]** And I heard the other day Jeff Dean was telling me that

**[00:15:58 – 00:16:01]** he would love to do a trillion token context window.

**[00:16:01 – 00:16:04]** I had no idea how he think he's going to do that, but let's see.

**[00:16:04 – 00:16:05]** That's his job to do.

**[00:16:06 – 00:16:08]** So basically the nice thing, of course, when you have 10

**[00:16:08 – 00:16:12]** or 100 million tokens is you can sidestep this main problem

**[00:16:12 – 00:16:14]** of continual learning, which is your model is always static,

**[00:16:14 – 00:16:17]** but basically if you put your whole code base in the model, the model

**[00:16:17 – 00:16:20]** will kind of learn your code base in a way and can generate this.

**[00:16:20 – 00:16:24]** So that's our current way, I would say, to basically deal with static

**[00:16:24 – 00:16:28]** models, is just put everything we can in a context window.

**[00:16:28 – 00:16:30]** It's very impressive. If you ever try to put your whole

**[00:16:30 – 00:16:32]** company in stack or something in a model, you'll see that

**[00:16:32 – 00:16:34]** you can fit a massive thing.

**[00:16:34 – 00:16:37]** Another angle we don't see here, but let's talk about

**[00:16:37 – 00:16:40]** it here, is that we now also want this model to reason a lot.

**[00:16:40 – 00:16:43]** So models spend a lot of time pondering how they should answer;

**[00:16:43 – 00:16:45]** they try something, they go back.

**[00:16:45 – 00:16:47]** And we see that basically this is called what we call

**[00:16:47 – 00:16:50]** test-time compute, which is if you let the model think

**[00:16:50 – 00:16:51]** and explore more things.

**[00:16:51 – 00:16:56]** In context, we know it actually ends up getting better conclusion

**[00:16:56 – 00:16:57]** and going back better.

**[00:16:57 – 00:17:00]** The main challenge here is at some point, you know, this can

**[00:17:00 – 00:17:02]** really explode your context window.

**[00:17:02 – 00:17:07]** So we saw the apparition about this year of a good way to compress.

**[00:17:07 – 00:17:09]** So now we're getting used to waiting for our models

**[00:17:09 – 00:17:12]** to compress their context window, which is very annoying.

**[00:17:12 – 00:17:15]** And I think we'll actually disappear very soon because people

**[00:17:15 – 00:17:17]** will do that, I think, very likely.

**[00:17:17 – 00:17:19]** But for now, we need to wait some time.

**[00:17:19 – 00:17:21]** I feel like, you know, the old days of the internet, you

**[00:17:21 – 00:17:24]** were waiting for your video to load the next thing and, you know, the

**[00:17:24 – 00:17:25]** streaming thing was still too slow.

**[00:17:25 – 00:17:27]** But that's where we are right now.

**[00:17:27 – 00:17:29]** So we let the model think for long and then we wait

**[00:17:29 – 00:17:32]** for them to compress their path thinking, a little bit like humans

**[00:17:32 – 00:17:36]** need to sleep maybe every night to compress their day of work.

**[00:17:36 – 00:17:41]** And hopefully you'll have enough to sleep on tonight from my talk.

**[00:17:41 – 00:17:44]** The new thing of this year I was talking about that is agent, that

**[00:17:44 – 00:17:45]** learned by acting in environments.

**[00:17:45 – 00:17:47]** So this used to be more like about robots.

**[00:17:47 – 00:17:49]** The first time we were talking about RL, we thought about

**[00:17:49 – 00:17:51]** robots trying things.

**[00:17:51 – 00:17:54]** What we see this year is that we have actually agent trying

**[00:17:54 – 00:17:57]** things in our environment, so enterprise environments.

**[00:17:57 – 00:17:59]** And there are a lot of them, and actually I want to talk

**[00:17:59 – 00:18:01]** a lot about that later, so I just keep this very simple

**[00:18:01 – 00:18:05]** view of what is an agent.

**[00:18:05 – 00:18:08]** Now, how can you do that yourself?

**[00:18:08 – 00:18:11]** How can you hill climb your own domain?

**[00:18:11 – 00:18:14]** Well, it was kind of dark magic for a long time, but

**[00:18:14 – 00:18:16]** now I think it's really opening up.

**[00:18:16 – 00:18:18]** And it's getting so easy that you have team offering that

**[00:18:18 – 00:18:21]** for quite cheap, but you all can also start doing that.

**[00:18:21 – 00:18:23]** And we see a lot of teams actually starting doing that

**[00:18:23 – 00:18:25]** themselves in their company.

**[00:18:25 – 00:18:27]** Well, the first thing I was telling you is you start

**[00:18:27 – 00:18:28]** from an open model.

**[00:18:28 – 00:18:32]** So you're lucky, HuggingFace has like 3 million open models,

**[00:18:32 – 00:18:35]** so the main problem is more like the problem of choice; you have too

**[00:18:35 – 00:18:38]** many choices, so how do you choose the one you want to start from?

**[00:18:38 – 00:18:40]** Well, there's a couple of obvious things.

**[00:18:40 – 00:18:43]** You want the, you want to see the most efficient model

**[00:18:43 – 00:18:44]** that fits your hardware.

**[00:18:44 – 00:18:46]** So let's say you're like RAMP, you're processing receipts.

**[00:18:46 – 00:18:49]** You don't need a 2 trillion parameters model for that.

**[00:18:49 – 00:18:50]** That's pretty simple.

**[00:18:51 – 00:18:53]** You can go for maybe the most downloaded.

**[00:18:53 – 00:18:55]** So for instance, the most downloaded and huggingface are

**[00:18:55 – 00:18:57]** usually the small, medium sizes.

**[00:18:57 – 00:18:59]** You can start from that. Another way is to see which hardware

**[00:18:59 – 00:19:03]** you have, or which hardware you can find, and to see which

**[00:19:03 – 00:19:06]** one fits basically there.

**[00:19:06 – 00:19:07]** You need to pick the right checkpoints.

**[00:19:07 – 00:19:10]** So here you need to start having a little bit of evaluation

**[00:19:10 – 00:19:12]** and being able to benchmark a couple of models.

**[00:19:12 – 00:19:14]** Some of them are optimized for code generation, some of them

**[00:19:14 – 00:19:17]** are multimodal, like if you want to process PDF images, you will

**[00:19:17 – 00:19:19]** want to start with a multimodal.

**[00:19:19 – 00:19:22]** So you need to pick the right type of model, that's called

**[00:19:22 – 00:19:24]** a checkpoint; it's not exactly training checkpoint here,

**[00:19:24 – 00:19:27]** it's more the type of model.

**[00:19:27 – 00:19:30]** And the last part, like I mentioned earlier in this talk,

**[00:19:30 – 00:19:34]** is that, unfortunately, not all models have a fully open license.

**[00:19:34 – 00:19:36]** So depending on what you're doing, if it's a research

**[00:19:36 – 00:19:38]** experiment, if it's more like something you want to deploy in

**[00:19:38 – 00:19:42]** production, you will need to check under which license your model is.

**[00:19:42 – 00:19:45]** In some cases, they might be permissive, but they might

**[00:19:45 – 00:19:50]** require you to basically contact the model builder to negotiate

**[00:19:50 – 00:19:51]** a license with them.

**[00:19:51 – 00:19:55]** That's actually getting quite common, but it's also pretty fine.

**[00:19:55 – 00:19:57]** And the thing to understand here is, the nice thing here

**[00:19:57 – 00:20:02]** is you're basically benefiting for this trillion token, a huge compute

**[00:20:02 – 00:20:05]** that someone has put in this model that they can share freely.

**[00:20:05 – 00:20:10]** So even if it's hard to choose, you should approach that with a happy

**[00:20:10 – 00:20:14]** mind of having found this treasure or gold of three things that's

**[00:20:14 – 00:20:18]** lying around for you to start from.

**[00:20:18 – 00:20:20]** Open ways are a strong starting point.

**[00:20:20 – 00:20:22]** I said they're about six to nine months behind the frontier.

**[00:20:22 – 00:20:24]** Another way to see that is if you plot them on,

**[00:20:24 – 00:20:27]** you know, like, for instance, the artificial analysis index.

**[00:20:27 – 00:20:31]** You see that, you know, even though the new very latest generation

**[00:20:31 – 00:20:34]** of frontier model has one step above, they're pretty good, right?

**[00:20:34 – 00:20:35]** They are the second step.

**[00:20:35 – 00:20:40]** So typically, like if you take one, I mean right now

**[00:20:40 – 00:20:43]** it's MiMo that just finished training, but if you take one that

**[00:20:43 – 00:20:46]** finished training a bit earlier or maybe have some users like

**[00:20:46 – 00:20:52]** GLM 5.3, that's definitely a very, very strong model to start from.

**[00:20:53 – 00:20:56]** And then the second question is, you know, on what are

**[00:20:56 – 00:20:58]** you going to train this model on?

**[00:20:58 – 00:21:01]** So sometimes you just need to train the model on a data set.

**[00:21:02 – 00:21:04]** Maybe you just have some receipts, you just fine-tune it on the

**[00:21:04 – 00:21:05]** receipts, and that's it.

**[00:21:05 – 00:21:06]** Sometimes it's more complex.

**[00:21:06 – 00:21:09]** You want the model, for instance, to, you know, follow a workflow

**[00:21:09 – 00:21:14]** in your company, internal tools, for instance.

**[00:21:15 – 00:21:17]** You want the model to take some action, to go there and

**[00:21:17 – 00:21:20]** to take some other action from this first action in other software.

**[00:21:20 – 00:21:23]** And so here you will need to explore open environments

**[00:21:23 – 00:21:28]** that are available, and that's what I wanted to show you here.

**[00:21:28 – 00:21:31]** Let's see how many environments are available on the Hub.

**[00:21:31 – 00:21:35]** Well, it turns out we have also 12 million environments, so now you'll

**[00:21:35 – 00:21:38]** again have a number of choices, too many choices, sorry for that.

**[00:21:38 – 00:21:41]** But the plus side of this is that, very likely, there

**[00:21:41 – 00:21:43]** is actually some environment that's quite close to what you

**[00:21:43 – 00:21:47]** want to train a model in, so you don't have to build them yourself.

**[00:21:47 – 00:21:50]** We're very lucky that, just like people have been sharing

**[00:21:50 – 00:21:54]** their model weights, which I think is amazing, great teams have also

**[00:21:54 – 00:21:55]** been sharing their environments.

**[00:21:55 – 00:21:58]** So teams like Prime Intellect, they're around here, I saw

**[00:21:58 – 00:21:59]** them the other day.

**[00:21:59 – 00:22:02]** Xiaomi is a Chinese team that actually trained the model

**[00:22:02 – 00:22:05]** I was showing you just before, the latest model.

**[00:22:05 – 00:22:07]** All of these companies have been sharing environments

**[00:22:07 – 00:22:12]** that basically can be more or less related to what you do.

**[00:22:14 – 00:22:17]** This is some example I took from the Hub.

**[00:22:17 – 00:22:21]** There is way more, right? I just try to give you some examples that

**[00:22:21 – 00:22:23]** are maybe more enterprise-focused.

**[00:22:23 – 00:22:25]** Some environments are very code-focused.

**[00:22:25 – 00:22:29]** This one, for instance, is really nice by ServiceNow.

**[00:22:29 – 00:22:32]** They call it the EnterpriseOpsGems.

**[00:22:32 – 00:22:33]** So they have like this.

**[00:22:33 – 00:22:34]** It's amazing, right?

**[00:22:34 – 00:22:38]** They have 1,150 tasks in eight domains.

**[00:22:38 – 00:22:41]** They check the database, and basically these are things

**[00:22:41 – 00:22:44]** that you're quite likely to see in an enterprise setting.

**[00:22:44 – 00:22:47]** So some of these tasks are around data creation.

**[00:22:47 – 00:22:50]** Some of the tasks are around user request and policy,

**[00:22:50 – 00:22:52]** how you handle that.

**[00:22:52 – 00:22:53]** Yeah, there's a lot there.

**[00:22:53 – 00:22:57]** And as you see on the right side, there is also other

**[00:22:57 – 00:22:58]** company like NVIDIA.

**[00:22:58 – 00:23:01]** We're lucky that they're organizing this.

**[00:23:01 – 00:23:06]** But they also open source great RL environments around email handling,

**[00:23:06 – 00:23:10]** calendar handling, databases, Mercor that you've much likely

**[00:23:10 – 00:23:14]** know as well, some banking if you're more interested in banking.

**[00:23:14 – 00:23:16]** So all of these are basically freely shared on the hub,

**[00:23:16 – 00:23:19]** and you can explore them and see if they actually the type

**[00:23:19 – 00:23:21]** of environments you're interested in, and maybe you want to

**[00:23:21 – 00:23:26]** find a model that works really well on this type of instruction.

**[00:23:26 – 00:23:27]** Now, how do you train?

**[00:23:27 – 00:23:30]** So this is maybe the slightly technical part, but we have

**[00:23:30 – 00:23:32]** a couple of the data set.

**[00:23:32 – 00:23:36]** I mean, HuggingFace is famous also for making libraries themselves.

**[00:23:36 – 00:23:38]** So you can do it yourself.

**[00:23:38 – 00:23:41]** Once you've picked an environment, you can pick

**[00:23:41 – 00:23:42]** our library called TRL.

**[00:23:42 – 00:23:44]** In the library, you have something called the GRPO

**[00:23:44 – 00:23:49]** trainer, which is kind of the working horse for RL training.

**[00:23:49 – 00:23:50]** It's very stable.

**[00:23:51 – 00:23:54]** Here, for instance, they start with Quent 3.

**[00:23:54 – 00:23:56]** You train it. There is one thing you will need to do

**[00:23:56 – 00:24:00]** here, which is specific to you, which is define what's the

**[00:24:00 – 00:24:04]** verifier, which is basically how do you, how your company or your team

**[00:24:04 – 00:24:07]** want to grade the final output.

**[00:24:07 – 00:24:09]** And this is where I would say there is still human choice

**[00:24:10 – 00:24:13]** needed, because you are the one, ultimately, that know which type

**[00:24:13 – 00:24:15]** of behavior you want in the model.

**[00:24:15 – 00:24:18]** So the verifier is the part you want to fill.

**[00:24:19 – 00:24:21]** And then, I mean, once you train your great model, I

**[00:24:21 – 00:24:24]** kind of encourage you to publish it again on the Hub if you can.

**[00:24:24 – 00:24:26]** So feel free to reshare them.

**[00:24:26 – 00:24:28]** Of course, if there is a lot of private data in that, maybe

**[00:24:28 – 00:24:31]** you don't want, but if you feel like you can share, please

**[00:24:31 – 00:24:35]** participate, you know, in this kind of collective endeavor.

**[00:24:35 – 00:24:38]** And this is some example of people, I mean teams, who

**[00:24:38 – 00:24:39]** have been training.

**[00:24:39 – 00:24:41]** This is all just from model cars and blogs.

**[00:24:41 – 00:24:42]** There's probably a lot better.

**[00:24:42 – 00:24:44]** Not everyone share, you know, the final results.

**[00:24:44 – 00:24:46]** Some of them just use it internally.

**[00:24:46 – 00:24:50]** So Liquid AI has been, you know, fine-tuning a coding agent,

**[00:24:50 – 00:24:53]** like a very pretty small one, 2.6.

**[00:24:53 – 00:24:57]** Liquid AI is known to make this very small and efficient model.

**[00:24:57 – 00:25:00]** Canada was doing an OCR, which is, you know, optical character

**[00:25:00 – 00:25:05]** recognition, spreadsheets, GeoGuessr is, you know, from

**[00:25:05 – 00:25:08]** image, can you find where it is, and NVIDIA as well

**[00:25:08 – 00:25:09]** has been, like, showing.

**[00:25:09 – 00:25:11]** The best way to show that your RL environments are great is

**[00:25:11 – 00:25:15]** usually just to train a great model on it, so that's what I did here.

**[00:25:15 – 00:25:20]** Yeah, this is a bunch of pre-training results that

**[00:25:20 – 00:25:23]** were really nice.

**[00:25:24 – 00:25:27]** Yeah, here is basically a few more recent things.

**[00:25:28 – 00:25:31]** So you saw MuseSpark was released.

**[00:25:31 – 00:25:33]** There is already post-training recipes.

**[00:25:33 – 00:25:37]** So if you feel the previous slide was slightly intimidating

**[00:25:37 – 00:25:41]** because you need to know TRL and how to handle it, well, also

**[00:25:41 – 00:25:45]** be assured that a lot of people are sharing recipes on how to do that.

**[00:25:45 – 00:25:48]** Meta first, with this recipe from Muse, but also Prime

**[00:25:48 – 00:25:51]** Intellect, Manchin, Bespoke Labs.

**[00:25:51 – 00:25:54]** All of these are very good sources of information on how to do that.

**[00:25:54 – 00:25:57]** And if you really feel like, okay, we don't have the time,

**[00:25:57 – 00:26:00]** we don't want to spend the time to do that, there is

**[00:26:00 – 00:26:04]** now a bunch of really nice teams who can help you actually do that.

**[00:26:04 – 00:26:08]** So you're not alone in this.

**[00:26:09 – 00:26:12]** Nemotron is typically what people use to train models.

**[00:26:12 – 00:26:14]** It's been developed over the years by NVIDIA.

**[00:26:14 – 00:26:17]** It's now the workhorse of training.

**[00:26:17 – 00:26:21]** It's typically where you will find the fastest, most efficient

**[00:26:21 – 00:26:23]** way to train models.

**[00:26:23 – 00:26:30]** Nemotron covers both, I would say, the training library and framework

**[00:26:30 – 00:26:32]** itself, and also everything that goes around the models and things.

**[00:26:32 – 00:26:36]** So basically, Nemotron is a combination of the ways that they

**[00:26:36 – 00:26:39]** share that are typically, right now, very good starting points.

**[00:26:39 – 00:26:42]** So Nemotron VL is probably the best vision language model

**[00:26:42 – 00:26:44]** you can start from today.

**[00:26:44 – 00:26:46]** Also the pre-training data, we worked a lot with them.

**[00:26:46 – 00:26:51]** At Helge Infels, we had FineWeb and now Nemotron CC is also really

**[00:26:51 – 00:26:53]** nice to pre-train your model.

**[00:26:53 – 00:26:56]** Most of the case, you won't want to pre-train on yourself, like you

**[00:26:56 – 00:26:58]** want to have another team do that.

**[00:26:58 – 00:27:00]** But it's still nice to know that you can inspect what

**[00:27:00 – 00:27:01]** was in your pre-training data.

**[00:27:01 – 00:27:04]** And there is, as I told you, there is a bunch of recipe, code,

**[00:27:04 – 00:27:05]** RL, environment documentation.

**[00:27:05 – 00:27:09]** So it's probably the best place you can start from nowadays.

**[00:27:09 – 00:27:15]** Well, to close the session, I would say it's pretty easy to train.

**[00:27:15 – 00:27:19]** You should try to do it, or you should at least get a team or a

**[00:27:19 – 00:27:21]** friendly team to help you do that.

**[00:27:21 – 00:27:25]** And if you do that, I think the best would be also that you share

**[00:27:25 – 00:27:29]** back again what you learned, your environment, if you improved the

**[00:27:29 – 00:27:33]** state, your model, and your eval.

**[00:27:33 – 00:27:36]** And with this being said, I think we are at the last slide.

**[00:27:36 – 00:27:38]** So we actually have some time for questions.

**[00:27:38 – 00:27:41]** It's quite late in the day, but if some of you are still

**[00:27:41 – 00:27:45]** awake enough to dare ask a question, I see one already there.

**[00:27:45 – 00:27:49]** We have microphones, so feel free to wait.

**[00:27:52 – 00:27:54]** Thank you for the great talk.

**[00:27:54 – 00:27:56]** My question is somewhat tangentially related to

**[00:27:56 – 00:28:00]** what we just talked about, and it was a great talk for a technical

**[00:28:00 – 00:28:05]** audience here, but I was wondering, given that at this point in time,

**[00:28:05 – 00:28:13]** given how AI has become a hot topic everywhere, well, we have access

**[00:28:13 – 00:28:18]** in, let's say, the domain that are also pushing a particular narrative

**[00:28:18 – 00:28:24]** that open models can be more dangerous for a number of reasons,

**[00:28:24 – 00:28:28]** and they use their platform to target, say, people in the policy

**[00:28:28 – 00:28:33]** space, so that they can influence legislature in one way or another.

**[00:28:33 – 00:28:36]** So my question to you, Thomas, is as someone with a substantial

**[00:28:36 – 00:28:41]** platform yourself, what could you, or by extension Hugging

**[00:28:41 – 00:28:48]** Face, do to better educate policy makers so they can put out better

**[00:28:48 – 00:28:55]** informed legislature that supports the proliferation of open models?

**[00:28:56 – 00:28:58]** What can we, ourselves, the engineering, let's say the

**[00:28:58 – 00:29:03]** technical community, do on the communication front to sort of do

**[00:29:03 – 00:29:04]** this in a more democratic process?

**[00:29:04 – 00:29:07]** Yeah, that's a very good question.

**[00:29:07 – 00:29:10]** I mean, this was the technical part of Egliface, but there

**[00:29:10 – 00:29:14]** is one part which is, and you rightly say so, which is education.

**[00:29:14 – 00:29:18]** We try to explain what it is, because there was this idea

**[00:29:18 – 00:29:21]** that open models are dangerous.

**[00:29:21 – 00:29:25]** My current view is that the open-close axis is rather

**[00:29:25 – 00:29:28]** orthogonal to the danger axis.

**[00:29:28 – 00:29:31]** Let's take this example of this summer.

**[00:29:31 – 00:29:33]** We were attacked by some closed source model.

**[00:29:33 – 00:29:36]** We're very happy to have open models to defend ourselves because

**[00:29:37 – 00:29:40]** the closed model we tried to use said, okay, we can't protect you.

**[00:29:40 – 00:29:44]** The logs you're trying to process are relating to a cyber attack.

**[00:29:45 – 00:29:46]** I'm not allowed to talk about cyber attacks.

**[00:29:46 – 00:29:49]** Basically, Opus and GPT and everything were saying no.

**[00:29:49 – 00:29:52]** There's no way we're going to help you do anything with that.

**[00:29:52 – 00:29:53]** We're like, OK, now we're under attack.

**[00:29:53 – 00:29:55]** We need to defend ourselves. So the only way we were able

**[00:29:55 – 00:29:58]** to do that was basically to spin out an open model.

**[00:29:58 – 00:30:00]** And for us, for me at least, there was really like a wake-up call

**[00:30:00 – 00:30:04]** to understand that if we are fully dependent on two providers, well,

**[00:30:04 – 00:30:07]** we can't have the mercy of what they decide we're allowed to do.

**[00:30:07 – 00:30:10]** And sometimes they just don't; they are not aware of the

**[00:30:10 – 00:30:12]** risk they can make us take, right?

**[00:30:12 – 00:30:15]** So I think the general idea that having a diversity of

**[00:30:15 – 00:30:17]** model, having some model that you can host yourself, you

**[00:30:18 – 00:30:22]** don't always have to send your data, you know, to actually hear

**[00:30:22 – 00:30:28]** like an external, like a foreign entity or foreign government,

**[00:30:28 – 00:30:30]** I think it's very, very important.

**[00:30:30 – 00:30:34]** There's another thing which is also the retiring all these models.

**[00:30:34 – 00:30:37]** So typically right now, you're in this weird state where

**[00:30:37 – 00:30:40]** you will maybe build a whole setup on top of, I don't know,

**[00:30:41 – 00:30:43]** let's say, Opus 5.

**[00:30:43 – 00:30:47]** And in six months, Anthropic say, oh, sorry, but we're now stopping

**[00:30:47 – 00:30:50]** Opus 5 because Opus 6 is here and we can't just maintain both.

**[00:30:50 – 00:30:52]** So you kind of force, you have this kind of obsolescence

**[00:30:52 – 00:30:55]** that you kind of force to update your thing all the time.

**[00:30:55 – 00:30:57]** And sometimes some setup work just really well.

**[00:30:57 – 00:30:59]** You don't want to change them every four months just because

**[00:30:59 – 00:31:02]** the AI race is accelerating, and they want a free GPU.

**[00:31:02 – 00:31:05]** So, open models have this thing that they are very resilient,

**[00:31:05 – 00:31:06]** they're very solid.

**[00:31:06 – 00:31:09]** Once you have them and they work, well, they're going to stay there.

**[00:31:09 – 00:31:11]** Nobody's going to stay. We're going to get you access.

**[00:31:11 – 00:31:14]** So, I think there is a very strong case for safety, which

**[00:31:14 – 00:31:17]** is around open models as kind of a safety thing.

**[00:31:17 – 00:31:21]** Even more in Europe, I think the fable event happening

**[00:31:21 – 00:31:23]** earlier this year saying, oh, maybe you won't get access,

**[00:31:23 – 00:31:28]** so maybe someone will decide for you the access, was a strong way.

**[00:31:28 – 00:31:31]** We don't, on our side, we don't really write policy or anything.

**[00:31:31 – 00:31:33]** What we try to do is educate.

**[00:31:33 – 00:31:36]** So when we see, like, policymakers in the US and the EU, we try

**[00:31:36 – 00:31:40]** to explain in simple terms what is open source, you know, what are the

**[00:31:40 – 00:31:41]** dangers, what are the interests.

**[00:31:41 – 00:31:44]** We try to help them balance the thing.

**[00:31:44 – 00:31:47]** And we also try to work a lot on our side, you know,

**[00:31:47 – 00:31:50]** on a lot more transparency.

**[00:31:50 – 00:31:54]** I think, ultimately, this is a whole question of transparency.

**[00:31:54 – 00:31:56]** And for me, open source is kind of synonymous with transparency.

**[00:31:56 – 00:31:58]** You can open the box.

**[00:31:58 – 00:32:01]** But yeah, there is a lot of work to do here to explain

**[00:32:01 – 00:32:02]** this thing in non-technical terms.

**[00:32:02 – 00:32:05]** I fully agree. I think the best way you can do it yourself

**[00:32:05 – 00:32:08]** is kind of to relay this thing that if you use open source model in

**[00:32:08 – 00:32:10]** your company, if you think they're useful for whatever reason, cost,

**[00:32:10 – 00:32:14]** privacy, you know, like, I think you should be vocal about that.

**[00:32:14 – 00:32:17]** I think this helps us a lot because we can say, it's not

**[00:32:17 – 00:32:20]** just Huggy Face, you know, there's also RAMP, Trajectory, ServiceNow,

**[00:32:20 – 00:32:22]** they're all using open models.

**[00:32:22 – 00:32:24]** So there is probably a business reason for that.

**[00:32:24 – 00:32:27]** These are not crazy companies.

**[00:32:31 – 00:32:36]** I can say your question again maybe, Claude.

**[00:32:38 – 00:32:41]** I'll try to be short on my answer because I'm pretty... Sure.

**[00:32:41 – 00:32:47]** Hey, Thomas. So, while we think of the context and a million tokens

**[00:32:47 – 00:32:51]** in the context seems awesome, but for big enterprises where the

**[00:32:51 – 00:32:54]** intelligence and knowledge sits in millions and millions of documents

**[00:32:54 – 00:33:00]** and different type of documents, engineering drawings and whatnot.

**[00:33:00 – 00:33:04]** What would be your suggestion to invest into open models

**[00:33:04 – 00:33:07]** and train them with the context or build the context engine

**[00:33:07 – 00:33:11]** and context layer for the enterprise and then openly use like

**[00:33:11 – 00:33:12]** and then that's a choice, right?

**[00:33:12 – 00:33:16]** Use the frontier model or open model, but invest into

**[00:33:16 – 00:33:18]** the context layer.

**[00:33:18 – 00:33:20]** Yeah, I would say it might depend in the future.

**[00:33:20 – 00:33:23]** I think right now it's safer to build kind of a context

**[00:33:23 – 00:33:24]** engine, like you were saying.

**[00:33:24 – 00:33:28]** So a very good way to navigate and do retrieval, so basically teach

**[00:33:28 – 00:33:31]** the model to search your database.

**[00:33:31 – 00:33:36]** And the first step is usually for your database, whatever

**[00:33:36 – 00:33:38]** they are, or your documents, or your internal knowledge,

**[00:33:38 – 00:33:40]** is to make it very accessible.

**[00:33:40 – 00:33:43]** So you typically want to build MCP API search.

**[00:33:43 – 00:33:45]** You want to have this very easy to search.

**[00:33:45 – 00:33:47]** As a first step, I think it's very safe, and then and then,

**[00:33:47 – 00:33:50]** like you say, you can choose between, and often you want

**[00:33:50 – 00:33:51]** the mix of closed and open models.

**[00:33:52 – 00:33:54]** Often you will want some closed model for a very complex task,

**[00:33:54 – 00:33:56]** you know, or maybe something you're just trying.

**[00:33:57 – 00:33:58]** If you're just trying something, you should start with the

**[00:33:58 – 00:34:00]** best way, you know.

**[00:34:00 – 00:34:03]** And then you want to move them to production, then cost is a

**[00:34:03 – 00:34:05]** reason, and then you're like, okay, maybe let's look at open model.

**[00:34:05 – 00:34:07]** We kind of now know what's the task.

**[00:34:07 – 00:34:10]** So you will constantly be moving between both in most

**[00:34:10 – 00:34:13]** cases, unless you're a very sovereign, different thing

**[00:34:13 – 00:34:15]** where you actually can't choose.

**[00:34:15 – 00:34:16]** So I think that's really the best way.

**[00:34:16 – 00:34:19]** It's still very fun when you find you the model on

**[00:34:19 – 00:34:20]** your own code base.

**[00:34:20 – 00:34:22]** I don't know if everyone tried it here, but I find your model

**[00:34:22 – 00:34:23]** in my own writing.

**[00:34:23 – 00:34:26]** It's like, there is like, the model deeply gets you, which is

**[00:34:26 – 00:34:30]** very interesting, but I think for a company, yeah, maybe in the future.

**[00:34:30 – 00:34:32]** Right now, the frontier is moving so fast that you will

**[00:34:32 – 00:34:35]** fine-tune it, and then in three weeks or one month again, you

**[00:34:35 – 00:34:37]** might want to fine-tune it again.

**[00:34:37 – 00:34:39]** So it seems like a little bit wasteful for now.

**[00:34:39 – 00:34:42]** Thank you.

**[00:34:42 – 00:34:43]** It's hard to choose. I think you were raising

**[00:34:44 – 00:34:47]** your hand for some time.

**[00:34:48 – 00:34:50]** Try to get on the L. There is someone very far.

**[00:34:50 – 00:34:51]** Okay, let's try also the very far people.

**[00:34:51 – 00:34:52]** Thank you, Thomas.

**[00:34:52 – 00:34:55]** I have a question about the physical AI.

**[00:34:56 – 00:34:59]** So what do you think, what is the real bottleneck for physical AI?

**[00:35:00 – 00:35:05]** Is it data or the simulation of the model itself?

**[00:35:05 – 00:35:07]** I think data, very clearly.

**[00:35:07 – 00:35:10]** Data, right now, we don't have enough.

**[00:35:11 – 00:35:13]** I mean physically it's very broad.

**[00:35:13 – 00:35:15]** Yesterday I was complaining about this term, so maybe I can complain

**[00:35:15 – 00:35:17]** again, but that means everything.

**[00:35:17 – 00:35:19]** It's like robotics. Some people, even some people rightfully

**[00:35:20 – 00:35:20]** say it's also biology.

**[00:35:20 – 00:35:24]** I mean biology, physical, everything is physical.

**[00:35:24 – 00:35:27]** For instance, bio is a data problem for sure.

**[00:35:27 – 00:35:30]** Robotics is also a data problem that is starting to get solved

**[00:35:30 – 00:35:32]** and seems to work really well.

**[00:35:32 – 00:35:35]** So there is some area where you can mitigate that with simulation.

**[00:35:35 – 00:35:36]** Simulation works really well, I think.

**[00:35:36 – 00:35:38]** I mean, thanks to NVIDIA, right?

**[00:35:38 – 00:35:43]** Like Isaac Lab, for instance, robotics is amazing.

**[00:35:43 – 00:35:46]** Duck, this morning, working on stage, it's all seemed

**[00:35:46 – 00:35:49]** to real direct transfer, nothing fine-tuned in real life.

**[00:35:49 – 00:35:52]** So, for some area, it works really well in simulation.

**[00:35:52 – 00:35:54]** Most of the area, you know, in particular, if you start

**[00:35:54 – 00:35:56]** to be like complex thing, complex texture, things that

**[00:35:56 – 00:36:01]** are more like squishy thing, all of that, simulation is really hard.

**[00:36:01 – 00:36:05]** And then data, data is a mode.

**[00:36:06 – 00:36:08]** Thank you, Thomas.

**[00:36:08 – 00:36:11]** I would like to know how much compute do you need

**[00:36:11 – 00:36:17]** to post-train, let's say, a 20 billion parameter model?

**[00:36:17 – 00:36:23]** Is it a lot of compute you need, or is it done with very

**[00:36:23 – 00:36:28]** little hardware, and how long would this take for training?

**[00:36:29 – 00:36:34]** And can you do this, like

**[00:36:34 – 00:36:41]** copying data from the internal wiki, and train the model on this?

**[00:36:41 – 00:36:46]** Or what would you suggest for a small model like 20

**[00:36:46 – 00:36:47]** billion parameter model?

**[00:36:47 – 00:36:51]** Say a hundred hundred dollar to a thousand dollar, I think.

**[00:36:52 – 00:36:54]** I think the easiest way, if it's a small pre-training,

**[00:36:54 – 00:36:57]** like we had the ramp example, so they find you on 10,000

**[00:36:57 – 00:37:00]** example, that's really small.

**[00:37:00 – 00:37:01]** Here you can use API.

**[00:37:01 – 00:37:04]** Usually, we want to go to the firework, you know, the

**[00:37:04 – 00:37:07]** bastion, the nebulous, the...

**[00:37:07 – 00:37:11]** The tinker, and you just pay for some costs.

**[00:37:11 – 00:37:13]** If you start to do massive pre-training, like in RL

**[00:37:13 – 00:37:17]** environments that are quite complex, and then you most

**[00:37:17 – 00:37:21]** likely need to budget a bit more, like in the 10,000 maybe.

**[00:37:21 – 00:37:24]** It's quite rare that you need to do several hundred; like

**[00:37:24 – 00:37:26]** if you start to be in the million rush, that's quite rare.

**[00:37:26 – 00:37:29]** I would say that's probably that you actually, yourself,

**[00:37:29 – 00:37:32]** you're almost a frontier company, or like you're trying to do

**[00:37:32 – 00:37:33]** something that are really complex.

**[00:37:34 – 00:37:36]** So, I mean, the takeaway is that often you should check because

**[00:37:36 – 00:37:41]** it's quite cheaper, I think, than what most people think, yeah.

**[00:37:41 – 00:37:44]** But yeah, everything depends on how many examples, you know,

**[00:37:44 – 00:37:48]** how much, how, even the size of the model, how much is it optimized?

**[00:37:48 – 00:37:51]** Some of them are actually very bad, and so, in terms

**[00:37:51 – 00:37:54]** of how architecturally shaped they are, and so they're

**[00:37:54 – 00:37:55]** actually very costly.

**[00:37:55 – 00:38:00]** Yeah, but there is now also some good ways to evaluate that.

**[00:38:00 – 00:38:04]** We have some budgeting, how you say, you know, kind of

**[00:38:04 – 00:38:07]** not leaderboard, but like formula on the hub spaces

**[00:38:07 – 00:38:10]** where you can evaluate that, or you should contact one

**[00:38:10 – 00:38:13]** of these fine tuning company, and they can quickly tell you, oh, this

**[00:38:13 – 00:38:16]** is going to be costly that much.

**[00:38:17 – 00:38:19]** Yeah, maybe there's someone there.

**[00:38:19 – 00:38:23]** Yeah, because we did all of this middle.

**[00:38:24 – 00:38:28]** We still have four minutes left, so it's fine.

**[00:38:28 – 00:38:31]** Thank you so much, Thomas, for the talk.

**[00:38:31 – 00:38:34]** My question is, do you think, to reach AGI, we just have

**[00:38:34 – 00:38:36]** to keep pushing, like, parameters?

**[00:38:36 – 00:38:38]** Like, for instance, Elon, he clearly said he wants to

**[00:38:38 – 00:38:41]** make a 10 trillion parameter model by end of next year.

**[00:38:41 – 00:38:46]** Then you have Sam, then you have Dario, further pushing this.

**[00:38:46 – 00:38:48]** Or do you think something else will come out of LLM

**[00:38:48 – 00:38:51]** because the natural language of compute isn't language,

**[00:38:51 – 00:38:54]** isn't our physical language?

**[00:38:54 – 00:38:55]** Yeah, that's a good question.

**[00:38:55 – 00:38:59]** I mean, the model size did scale quite quickly, but maybe

**[00:38:59 – 00:39:01]** less than we would have expected.

**[00:39:01 – 00:39:05]** There were some moments, like people were, if you follow

**[00:39:05 – 00:39:07]** a bit what the Big Labs has been doing, there was some challenge

**[00:39:07 – 00:39:10]** at some point, more fundamentally, how we train the model, like

**[00:39:10 – 00:39:12]** how the gradient behaves.

**[00:39:12 – 00:39:14]** But it seems like some of these labs have solved that now.

**[00:39:14 – 00:39:18]** And this is why we see suddenly this new generation of models,

**[00:39:18 – 00:39:21]** like Astra, I think; they are one step above in terms of sizing.

**[00:39:21 – 00:39:26]** So I would expect people to keep increasing the size of

**[00:39:26 – 00:39:27]** the model, but...

**[00:39:27 – 00:39:30]** Ultimately, it's like this three-axis thing.

**[00:39:30 – 00:39:34]** Like, you want to increase the size, but it's as much as the size,

**[00:39:34 – 00:39:38]** it's a data, it's a data game as well, and then test time computes.

**[00:39:38 – 00:39:39]** So you have these three axes.

**[00:39:39 – 00:39:44]** So yeah, it's never purely just scaling, and that's one error.

**[00:39:44 – 00:39:46]** For instance, when we trained the first model a very long

**[00:39:46 – 00:39:50]** time ago in 2020 called Bloom, the main error we did was

**[00:39:50 – 00:39:54]** to scale it way too big, and so you just have to keep this thing.

**[00:39:54 – 00:39:57]** So right now, data is also 50% of the thing.

**[00:39:57 – 00:40:00]** So the way we're reaching AGI is quite funny.

**[00:40:00 – 00:40:03]** We're just basically completing all the tasks that we see

**[00:40:03 – 00:40:04]** that are not yet solved.

**[00:40:04 – 00:40:07]** We just build new RL environments for this one, new RL environments

**[00:40:07 – 00:40:08]** for this one, new RL environments.

**[00:40:08 – 00:40:10]** So this is the new form of data.

**[00:40:10 – 00:40:13]** So both. I would expect the size to keep increasing, but

**[00:40:13 – 00:40:14]** the data needs as well.

**[00:40:14 – 00:40:18]** I think we only have one minute left, so tiny, tiny last question.

**[00:40:18 – 00:40:21]** Is it a short question?

**[00:40:24 – 00:40:29]** Hi, you mentioned challenges with open model and the benefits of it.

**[00:40:29 – 00:40:32]** What would you say some of the risks are?

**[00:40:32 – 00:40:34]** The risk for open models?

**[00:40:34 – 00:40:36]** I think it's mostly the same as closed model right now.

**[00:40:36 – 00:40:37]** It's alignment.

**[00:40:37 – 00:40:41]** So for a long time, we thought we had figured alignment,

**[00:40:41 – 00:40:44]** at least I thought, for when we are using what's called RLVR.

**[00:40:44 – 00:40:47]** So model were basically doing what you think they were doing.

**[00:40:47 – 00:40:49]** And now we start to have these cases where, for instance,

**[00:40:49 – 00:40:51]** one model will go hug in face.

**[00:40:51 – 00:40:52]** Well, it's not asked to do that.

**[00:40:52 – 00:40:54]** And so this is an open question.

**[00:40:54 – 00:40:58]** It's going to be the same for open models as closed model, I think.

**[00:40:58 – 00:41:00]** That's the main challenge.

**[00:41:00 – 00:41:01]** Yeah, to be honest.

**[00:41:01 – 00:41:03]** So it's the same. That's when I was saying, okay, close

**[00:41:03 – 00:41:09]** open is roughly orthogonal with, you know, like the acts of danger.

**[00:41:09 – 00:41:12]** I think the problem is roughly now the same.

**[00:41:12 – 00:41:14]** The reason we see it a bit less for open model is that

**[00:41:14 – 00:41:16]** they are six to nine months behind.

**[00:41:16 – 00:41:18]** So the main problem right now and the main reason I'm worried,

**[00:41:19 – 00:41:22]** for instance, for line months is more like the Opus 5.5 or like

**[00:41:22 – 00:41:24]** the frontier-frontier of mobile.

**[00:41:24 – 00:41:28]** Thank you. Which is a very serious thing, actually.

**[00:41:28 – 00:41:32]** Thanks, everyone. Thank you, Thomas, excellent session.

**[00:41:38 – 00:41:41]** Thank you. That brings us to the last session of today.

**[00:41:41 – 00:41:44]** I know there were a lot more questions, so perhaps Thomas

**[00:41:44 – 00:41:46]** is around to take more questions.

**[00:41:46 – 00:41:49]** I'd like to also invite all of you to come to our

**[00:41:49 – 00:41:50]** visit our show floor.

**[00:41:50 – 00:41:53]** It's open until 7 p.m. today.

**[00:41:53 – 00:42:00]** And today, and today alone, we have also our refreshments available

**[00:42:00 – 00:42:02]** on the show floor from 5 to 7 p.m.

**[00:42:02 – 00:42:05]** So please feel free to help yourselves.

**[00:42:05 – 00:42:09]** And see you back here in this room tomorrow for the continuation of

**[00:42:09 – 00:42:11]** the agentic AI and closed sessions.

**[00:42:11 – 00:42:12]** Thank you.

