# Scaling RL Runtime Infrastructure: The New Systems Bottleneck for Agentic AI

## Abstract

Key takeaways Reinforcement learning (RL) is becoming an end-to-end systems problem: As labs scale agentic RL, the bottlenecks are increasingly in rollouts, inference serving, tool execution, sandboxing, verification, scheduling, and memory movement—not only in model training throughput. The runtime stack for RL has different scaling challenges than pretraining or standard inference: These workloads are bursty, long-context, verifier-heavy, and often require thousands of parallel trajectories, dynamic batching, CPU/GPU coordination, KV cache management, and reliable sandbox lifecycle management. Hardware/software co-design will determine who can scale RL efficiently: The panel will surface what leading labs need from GPU systems, networking, serving runtimes, orchestration layers, NVIDIA NeMo™-RL/TRT-LLM/vLLM-style software, and observability/benchmarking in order to reduce cost per successful rollout and improve end-to-end throughput.

## AI Summary

_No AI summary available._

## Transcript

**[00:00:05 – 00:00:06]** Hi, everyone. Welcome.

**[00:00:06 – 00:00:09]** I hope you're having a great GTC.

**[00:00:09 – 00:00:10]** Welcome to our panel.

**[00:00:10 – 00:00:14]** It's called Scaling RL Runtime Infrastructure, the New Systems

**[00:00:14 – 00:00:18]** Bottleneck, and we have a whole bunch of really good

**[00:00:18 – 00:00:23]** panelists here, moderated by Vartika Singh from NVIDIA.

**[00:00:23 – 00:00:27]** Vartika Singh leads exploratory technical partnerships at

**[00:00:27 – 00:00:31]** NVIDIA across reinforcement learning, AI systems, and

**[00:00:31 – 00:00:33]** emerging compute workloads.

**[00:00:33 – 00:00:36]** And I'm going to leave it now with Vartika to introduce

**[00:00:36 – 00:00:37]** the rest of the panelists.

**[00:00:37 – 00:00:40]** Vatika Shivakumar.

**[00:00:56 – 00:00:59]** Both concepts evolve towards scaling learning, what

**[00:00:59 – 00:01:00]** scaling learning means.

**[00:01:00 – 00:01:04]** So we will be talking about RL in the context of scaling learning.

**[00:01:04 – 00:01:07]** So in the beginning, when we talk about RL, we talk

**[00:01:07 – 00:01:10]** about environments, we're talking about rewards and

**[00:01:10 – 00:01:13]** how they come together, and as we discuss more, we have been talking

**[00:01:13 – 00:01:17]** about experiences, interaction, learning, and how that changes

**[00:01:17 – 00:01:20]** the shape of the computation.

**[00:01:20 – 00:01:22]** Then, amongst all of you, we have also talked about

**[00:01:22 – 00:01:27]** different domains where mathematics and computational sciences have

**[00:01:27 – 00:01:31]** more easily and readily verifiable rewards, and then we have more

**[00:01:31 – 00:01:37]** ambiguous, delayed rewards in terms of scientific discovery, human

**[00:01:38 – 00:01:42]** interaction, so on so forth, and then do we keep constructing these

**[00:01:42 – 00:01:44]** environments like a long tail?

**[00:01:44 – 00:01:46]** Keep, do we address that?

**[00:01:46 – 00:01:52]** So we have five excellent researchers, practitioners.

**[00:01:52 – 00:01:54]** Some are affiliated with universities, some have

**[00:01:54 – 00:01:59]** started their own companies, so I will let them introduce

**[00:01:59 – 00:02:02]** themselves from expressing their perspective or what

**[00:02:03 – 00:02:05]** scaling learning means for them.

**[00:02:05 – 00:02:08]** So I would like to start with you, Zihang.

**[00:02:08 – 00:02:12]** You have a lot of experience working at tremendous scale.

**[00:02:12 – 00:02:14]** I will let you describe that.

**[00:02:15 – 00:02:20]** Sure, thanks. So I was in the founding team of XAI,

**[00:02:20 – 00:02:24]** where essentially my reinforcement learning experience is mostly

**[00:02:24 – 00:02:28]** related to scaling language models in the context of post-training,

**[00:02:28 – 00:02:33]** from earlier chart model to later agent models.

**[00:02:33 – 00:02:38]** So for me, a lot of time we're dealing with how to construct

**[00:02:38 – 00:02:44]** an environment, how to construct a reward, and also how to

**[00:02:44 – 00:02:46]** avoid reward hacking.

**[00:02:46 – 00:02:50]** And right now I'm working in a new company, which is called Epochal.

**[00:02:51 – 00:02:55]** So it's along the same line of how to create better training

**[00:02:55 – 00:02:57]** signal at the end of the day.

**[00:02:57 – 00:03:00]** There are a lot of long tails in terms of environment we

**[00:03:00 – 00:03:04]** need to cover, and at the end of the day, I believe

**[00:03:04 – 00:03:07]** that's the most important thing.

**[00:03:07 – 00:03:08]** Thank you, Sihang.

**[00:03:08 – 00:03:11]** So you're talking about scaling in the order of hundreds of

**[00:03:11 – 00:03:13]** thousands of GPUs.

**[00:03:13 – 00:03:18]** Yeah, so taking coding or coding agent as an example,

**[00:03:18 – 00:03:23]** if you can do one rollout for, on average, one hour, and you

**[00:03:23 – 00:03:27]** can do verification, you can run that in parallel, maybe millions

**[00:03:27 – 00:03:31]** of agents, which means every one hour you can get millions, even

**[00:03:31 – 00:03:37]** towards maybe 100 million bits of reward signal from the environment.

**[00:03:37 – 00:03:41]** So that's kind of the skill I've been working with, or

**[00:03:41 – 00:03:43]** I work with before.

**[00:03:43 – 00:03:48]** And that type of thing requires the system to be super robust,

**[00:03:48 – 00:03:50]** the environment to be super robust, because any reward

**[00:03:50 – 00:03:54]** hacking will ruin the entire run.

**[00:03:54 – 00:03:57]** That's basically the type of thing I'm thinking about.

**[00:03:57 – 00:03:59]** Yeah, interesting.

**[00:03:59 – 00:04:02]** And talking about the scale at which you worked at, and

**[00:04:02 – 00:04:06]** then transitioning to Academia, I would love for you, Laura,

**[00:04:06 – 00:04:11]** to talk more about, like, what the scale of computation,

**[00:04:11 – 00:04:15]** or lack thereof, how it defines your research in the academia,

**[00:04:15 – 00:04:20]** the more about open source, and the plurality of different

**[00:04:20 – 00:04:21]** components that come together.

**[00:04:21 – 00:04:24]** Yeah, thanks very much for the point and the question.

**[00:04:24 – 00:04:26]** Yeah, I will bring a quite different perspective, I think,

**[00:04:26 – 00:04:28]** today, that is more from academia.

**[00:04:28 – 00:04:36]** And I think the idea is that, for this scaling and paradigm

**[00:04:36 – 00:04:39]** AI, we really believe that the settings in which we are and

**[00:04:39 – 00:04:44]** we develop these advances clearly define the point of convergence

**[00:04:44 – 00:04:46]** of this AI ability, right?

**[00:04:46 – 00:04:51]** And this breakthrough today has been mostly built in companies

**[00:04:51 – 00:04:52]** and big BOLD, DRIVEOS, NVIDIA.

**[00:04:52 – 00:04:53]** Bold, DriveOS, NVIDIA.

**[00:04:53 – 00:04:59]** Let's say they favor this compute and maybe closed-source idea.

**[00:04:59 – 00:05:02]** So I come from, like, academia, so we bring,

**[00:05:02 – 00:05:04]** like, a completely different way.

**[00:05:04 – 00:05:08]** So we have funded now, thanks to a very big national UK

**[00:05:08 – 00:05:10]** funding, the BOLD lab.

**[00:05:10 – 00:05:14]** So it's British Open-ended Learning and Discovery, where we are

**[00:05:14 – 00:05:16]** basically taking two bets on that.

**[00:05:16 – 00:05:20]** So first is, what if we have converged to an I-paradigm

**[00:05:20 – 00:05:22]** that is not the only one?

**[00:05:22 – 00:05:26]** So what if we have, so, one island of a very large archipelago that

**[00:05:26 – 00:05:28]** we haven't explored yet, right?

**[00:05:28 – 00:05:31]** And we believe that in academia that's very good, to keep

**[00:05:31 – 00:05:37]** this curiosity-driven, exploratory approach to be able to go into

**[00:05:37 – 00:05:40]** the next scaling learning ability and open-endedness, but at the

**[00:05:40 – 00:05:46]** same time, we cannot keep working in this single PI CELOS lab.

**[00:05:46 – 00:05:48]** So we really want to build, that's the bat we are taking

**[00:05:48 – 00:05:53]** at BOLD, a fully open source, collaborative, probabilistic

**[00:05:53 – 00:05:57]** environment in which we really want to build open source,

**[00:05:57 – 00:06:02]** open models, open provenance, to really have everybody being

**[00:06:02 – 00:06:05]** able to come in together for this collective intelligence.

**[00:06:05 – 00:06:08]** And that's because, like, we really believe that we will

**[00:06:08 – 00:06:12]** come from a completely different compute and an environment

**[00:06:12 – 00:06:16]** and ecosystems, and maybe that will lead to different AI paradigms.

**[00:06:16 – 00:06:19]** So that's a little bit today the idea that we have beyond bold.

**[00:06:19 – 00:06:20]** So we merge our labs.

**[00:06:20 – 00:06:22]** We don't have PI labs anymore.

**[00:06:22 – 00:06:25]** We have one big team, collaborative, and we

**[00:06:25 – 00:06:29]** want to create this new ecosystem and blueprints that maybe I will

**[00:06:29 – 00:06:32]** also discuss more later, but that's a little bit the idea that we have.

**[00:06:32 – 00:06:35]** So one thing, as the panel moves forward, I would like

**[00:06:35 – 00:06:40]** to come back to you later on as you hear, is, do the

**[00:06:40 – 00:06:44]** research ideas that you initially pick up are also different because

**[00:06:44 – 00:06:48]** you're not in a closed, centrally

**[00:06:48 – 00:06:51]** staffed and resourced compute?

**[00:06:51 – 00:06:54]** So hold that thought, but we'd like to hear, like, do

**[00:06:54 – 00:06:57]** you take different directions completely compared to what?

**[00:06:57 – 00:07:02]** So we will go back, but quickly, I think yes and no, in the sense

**[00:07:02 – 00:07:05]** of we have different constraints, so we have different needs

**[00:07:05 – 00:07:09]** and different directions, aligned with still the open-endedness

**[00:07:09 – 00:07:12]** and scaling learning, but we also pitch a lot the greatness

**[00:07:12 – 00:07:14]** cannot be planned idea.

**[00:07:14 – 00:07:17]** I think because of this full exploratory, we actually start

**[00:07:17 – 00:07:20]** with one direction, but we don't know exactly where we end up.

**[00:07:20 – 00:07:23]** And I think that's very good in the academia, right?

**[00:07:23 – 00:07:25]** We are not bounded by.

**[00:07:25 – 00:07:29]** So I think maybe the initial point can be similar, but the ending

**[00:07:29 – 00:07:32]** point will be certainly different because of the constraints

**[00:07:32 – 00:07:34]** and needs and resources.

**[00:07:34 – 00:07:37]** So, like, stretching the same point to you, Khurram, right, you

**[00:07:37 – 00:07:39]** also come from Academia, but you've started a new company, if you would

**[00:07:40 – 00:07:43]** like to talk a little bit about it, and your vision of, like, million

**[00:07:44 – 00:07:47]** of independent agents, maybe not so much independent, but million

**[00:07:47 – 00:07:49]** of agents learning over time.

**[00:07:50 – 00:07:51]** So, my background is continual learning.

**[00:07:51 – 00:07:54]** I've been working in that area for seven, eight years,

**[00:07:54 – 00:07:56]** mostly in academia.

**[00:07:56 – 00:07:58]** Two years it was in industry.

**[00:07:58 – 00:08:03]** And recently, me and my co-founder, Rich Sutton, we made a new company.

**[00:08:03 – 00:08:07]** And the thesis we have is, if you look at current paradigm

**[00:08:07 – 00:08:12]** of training these agents, most of the learning is done by humans.

**[00:08:12 – 00:08:14]** And here's what I mean by that.

**[00:08:14 – 00:08:16]** Let's say you want to solve a task in robotics.

**[00:08:16 – 00:08:19]** This is an example. This is true in general.

**[00:08:19 – 00:08:22]** You would have humans, for example, go and learn how to operate that

**[00:08:22 – 00:08:24]** robot, so that's teleoperation.

**[00:08:24 – 00:08:27]** So human is doing the learning, you're collecting the data,

**[00:08:27 – 00:08:30]** and then once humans have learned this task and you've

**[00:08:30 – 00:08:33]** collected the data, you can distill that knowledge that human

**[00:08:33 – 00:08:35]** discovered into a neural network.

**[00:08:35 – 00:08:38]** So the learning happened outside the training paradigm.

**[00:08:38 – 00:08:40]** The learning was done by human.

**[00:08:40 – 00:08:42]** The second way people do it is they go make a simulation.

**[00:08:42 – 00:08:46]** But again, here, humans are the one figuring out what is

**[00:08:46 – 00:08:48]** the important aspect of a task that you need to put in the simulation.

**[00:08:49 – 00:08:51]** So again, humans are going outside, they're doing some learning

**[00:08:51 – 00:08:54]** about how to capture the task in an environment, and only once

**[00:08:54 – 00:08:58]** humans have figured it out, then we let our agents do some limited kind

**[00:08:58 – 00:09:02]** of learning inside that simulator.

**[00:09:02 – 00:09:05]** So most of the learning is right now done by humans.

**[00:09:05 – 00:09:08]** The bet that we are making is that we want to remove

**[00:09:08 – 00:09:09]** humans out of the loop completely.

**[00:09:09 – 00:09:13]** We would want our algorithms to do most of the learning,

**[00:09:13 – 00:09:15]** and so that is really what we mean by scaling learning.

**[00:09:16 – 00:09:19]** So we want all the compute to go towards learning, not

**[00:09:19 – 00:09:25]** towards distilling this human knowledge into neural networks.

**[00:09:25 – 00:09:29]** And if you can crack this problem, if you can have learning

**[00:09:29 – 00:09:33]** at scale, then it becomes possible to have highly specialized models.

**[00:09:33 – 00:09:36]** So we don't need to have, like, five or 10 models.

**[00:09:36 – 00:09:39]** We can have millions and millions of distinct models, each

**[00:09:39 – 00:09:41]** specialized to different use cases.

**[00:09:41 – 00:09:44]** Maybe the model I use learns about my preferences or things

**[00:09:44 – 00:09:48]** that I want, or about a niche use case, and yeah, that's

**[00:09:48 – 00:09:49]** how we're thinking about it.

**[00:09:49 – 00:09:50]** Thank you.

**[00:09:50 – 00:09:53]** And what are the problems that you're thinking of cracking when

**[00:09:53 – 00:09:55]** you say you cracked the problem?

**[00:09:55 – 00:09:58]** So the problem really is, how do we go from this paradigm

**[00:09:58 – 00:10:03]** where we have to outsource the learning to humans to

**[00:10:03 – 00:10:06]** the agents learning on their own from their own experience, right?

**[00:10:06 – 00:10:10]** So right now, if you remove humans, if everyone stopped working

**[00:10:11 – 00:10:14]** at big labs or robotics companies, robots would stop improving,

**[00:10:14 – 00:10:18]** and that's because learning is still being done by the humans, and

**[00:10:18 – 00:10:19]** then you distill that into robots.

**[00:10:19 – 00:10:22]** So the problem we want to crack is, you convert experience

**[00:10:22 – 00:10:24]** directly into learning.

**[00:10:24 – 00:10:26]** You don't have a human in the loop that is actually

**[00:10:26 – 00:10:29]** doing the learning and then passing it to a, I guess, agentic system.

**[00:10:29 – 00:10:31]** Okay.

**[00:10:31 – 00:10:35]** So, if I build on that, like, you're talking about various

**[00:10:35 – 00:10:39]** different experiences, different systems, different mechanisms,

**[00:10:39 – 00:10:42]** and, switching back to Nando, I mean, you can talk about

**[00:10:42 – 00:10:47]** something else, but my question to you, because you bring up your rich

**[00:10:47 – 00:10:51]** experience of data set curation, building verifiable rewards,

**[00:10:51 – 00:10:54]** environments, things like that.

**[00:10:54 – 00:10:59]** Yeah, so I guess I'm at the other end of what you were saying, which

**[00:10:59 – 00:11:02]** is, and this is just for practical reasons, if you want to teach

**[00:11:02 – 00:11:08]** agents to be good at math, at code and so on, then the challenge is

**[00:11:08 – 00:11:12]** really to find all the problems and find problems that are hard enough.

**[00:11:12 – 00:11:16]** So this is more like what you were saying, finding all

**[00:11:16 – 00:11:20]** these environments, building environments, ensuring they're

**[00:11:20 – 00:11:22]** hard enough, and of course, hard enough is with respect to

**[00:11:22 – 00:11:25]** the current policy that you have.

**[00:11:25 – 00:11:28]** And so, you know, you can't just go and buy environments and

**[00:11:28 – 00:11:30]** hope that those are the right ones.

**[00:11:30 – 00:11:33]** So you have to try them, and you have to have the right

**[00:11:33 – 00:11:35]** levels of difficulty.

**[00:11:35 – 00:11:39]** And of course, trying involves a lot of computation, so at

**[00:11:39 – 00:11:41]** the end of the day, scale is about being able to do

**[00:11:41 – 00:11:47]** this with, you know, 500 million

**[00:11:47 – 00:11:50]** question answers, for example.

**[00:11:50 – 00:11:52]** And then there is this huge long tail.

**[00:11:52 – 00:11:56]** If you want a model not to just be good at coding, but to be very good

**[00:11:56 – 00:12:04]** at coding reinforcement learning infrastructure, for example,

**[00:12:04 – 00:12:08]** then you need to train the models for that, so you need to bring and

**[00:12:08 – 00:12:12]** create the right tasks and so on.

**[00:12:12 – 00:12:14]** That's a huge.

**[00:12:15 – 00:12:19]** I believe that one could contemplate automating

**[00:12:19 – 00:12:22]** that, but the trade-off is a computational one.

**[00:12:22 – 00:12:25]** So the more you automate, the more you're allowing the models to do

**[00:12:25 – 00:12:29]** the computation, unless we figure out more efficient algorithms.

**[00:12:29 – 00:12:32]** And so that's one of the big challenges, is also how to

**[00:12:32 – 00:12:36]** speed up the inference, also because these reasoning traces

**[00:12:36 – 00:12:41]** are very long, and so it's a lot of tokens that you have to produce,

**[00:12:41 – 00:12:44]** and so that's very expensive.

**[00:12:45 – 00:12:46]** I'm part of, I have a new start-up.

**[00:12:46 – 00:12:51]** We focus on doing efficient parallel inference, so hopefully we

**[00:12:51 – 00:12:58]** will help a bit with this, and, you know, but these are other ideas.

**[00:12:58 – 00:13:01]** Some people decompose the task into subtasks, and then

**[00:13:01 – 00:13:04]** each agent takes one task, and that's basically the whole

**[00:13:04 – 00:13:05]** idea of the swarms.

**[00:13:05 – 00:13:09]** It's just basically parallelization.

**[00:13:11 – 00:13:13]** Yeah, those are all, like, formidable challenges that

**[00:13:13 – 00:13:18]** we face when we try to build models to do useful things.

**[00:13:18 – 00:13:21]** And of course, it depends on the application you have, too.

**[00:13:21 – 00:13:23]** In some applications like robotics, I think learning

**[00:13:23 – 00:13:27]** from experience makes a little more sense, or environments.

**[00:13:27 – 00:13:30]** And then there are these sort of more immediate applications

**[00:13:30 – 00:13:34]** that startups have, like build a good code model, where I think

**[00:13:34 – 00:13:36]** the approach is a bit different, so you need to take advantage.

**[00:13:36 – 00:13:38]** NVIDIA, Ineffable Intelligence.

**[00:13:39 – 00:13:42]** Thank you, and 40 minutes is not a good enough time to

**[00:13:42 – 00:13:47]** draw out all you guys' experience, but Lasse, putting it all

**[00:13:47 – 00:13:51]** on you, like combining experience, interaction, learning, like, how

**[00:13:51 – 00:13:55]** does it define what you are about, your company, and your perspective

**[00:13:55 – 00:13:57]** on learning and scaling?

**[00:13:57 – 00:14:00]** Right, so I'm from Ineffable Intelligence, and we are

**[00:14:00 – 00:14:04]** trying to make, like, a general learning algorithm.

**[00:14:05 – 00:14:08]** To a large extent, we are actually quite aligned with Oak Labs, but

**[00:14:08 – 00:14:10]** there are also a few differences.

**[00:14:10 – 00:14:15]** But some observations we have done is that one is

**[00:14:15 – 00:14:19]** that language models are bottlenecked by data, especially

**[00:14:19 – 00:14:22]** for pre-training, but also to some extent for the reinforcement

**[00:14:22 – 00:14:24]** learning post-training.

**[00:14:24 – 00:14:29]** The other part is, one of our co-founders, Dave Silver,

**[00:14:29 – 00:14:35]** he was part of AlphaGo, and the first AlphaGo algorithm was able to

**[00:14:35 – 00:14:42]** beat the best human at Go, but the next project, AlphaGo Zero, AlphaGo

**[00:14:42 – 00:14:48]** was based initially on trained on human data, or games, but

**[00:14:48 – 00:14:51]** AlphaGo Zero is actually trained with no human data, only with

**[00:14:51 – 00:14:54]** reinforcement learning, so there's no human biases in how, how you are

**[00:14:54 – 00:14:57]** going to do the best action and go.

**[00:14:58 – 00:15:00]** And the observation there was actually, like, this

**[00:15:00 – 00:15:03]** algorithm became very quickly much better than the one

**[00:15:03 – 00:15:06]** that was trained on human data.

**[00:15:06 – 00:15:08]** And at that point, you can just keep scaling, right?

**[00:15:08 – 00:15:11]** So there's basically no limit.

**[00:15:11 – 00:15:14]** So one way we are differentiating ourselves from, like, Oak

**[00:15:14 – 00:15:19]** Labs is that we are trying to do all this in simulation at first.

**[00:15:19 – 00:15:24]** So we are trying to have a small set of environments,

**[00:15:24 – 00:15:29]** some that are simple, but still has emergent, complex emergent

**[00:15:29 – 00:15:33]** behavior, and some that are more realistic or physics-based.

**[00:15:34 – 00:15:37]** And in these environments, the simpler ones we can use for

**[00:15:37 – 00:15:43]** really scaling, learning, really get some data out and some ideas on

**[00:15:44 – 00:15:46]** how we should shape this algorithm.

**[00:15:46 – 00:15:51]** And then each of these steps, we will eventually have a general

**[00:15:51 – 00:15:56]** algorithm that's very efficient at learning, even in the real world.

**[00:15:56 – 00:15:58]** So the end goal is kind of the same.

**[00:15:58 – 00:16:01]** We are targeting superintelligence with this algorithm, but the

**[00:16:02 – 00:16:04]** approach is very different.

**[00:16:04 – 00:16:05]** Thank you.

**[00:16:05 – 00:16:09]** So we are talking about scale, we are talking about learning,

**[00:16:09 – 00:16:12]** and all of you are approaching it from different dimensions,

**[00:16:12 – 00:16:13]** more or less same.

**[00:16:13 – 00:16:18]** So if we talk about the next theme that I want to talk about,

**[00:16:18 – 00:16:21]** is where do you get bottleneck?

**[00:16:21 – 00:16:22]** So starting with you, Nanda, right?

**[00:16:22 – 00:16:25]** So you're talking about verifiable rewards and environments.

**[00:16:25 – 00:16:32]** At what point compute stops being a bottleneck for learning the whole

**[00:16:32 – 00:16:34]** universe, the entire universe?

**[00:16:34 – 00:16:38]** At which point, the computer stops being a bottleneck?

**[00:16:38 – 00:16:44]** I find it very hard to envision that right now, because

**[00:16:44 – 00:16:49]** it's, .. compute has always been a bottleneck.

**[00:16:49 – 00:16:53]** I've been training neural networks now for longer than some

**[00:16:53 – 00:16:57]** of you have been alive, since 1994.

**[00:16:57 – 00:17:01]** I was doing back propagation on a device.

**[00:17:01 – 00:17:05]** And I can't think of a single year in my life when I felt

**[00:17:05 – 00:17:08]** that I had enough compute.

**[00:17:08 – 00:17:14]** OK, let's frame it. So I think with what's going to happen is

**[00:17:16 – 00:17:20]** And I'm actually going to take a serious answer on this,

**[00:17:20 – 00:17:26]** I think the way science and engineering advances is just

**[00:17:26 – 00:17:29]** when you think you're finally going to get enough compute,

**[00:17:29 – 00:17:33]** that means you now have the chance to solve bigger problems.

**[00:17:33 – 00:17:35]** You can automate further.

**[00:17:35 – 00:17:40]** You can now move to do what these guys want to do.

**[00:17:40 – 00:17:44]** And so we'll always want to do something more amazing.

**[00:17:44 – 00:17:50]** And I think that it's just this, solving problems just means

**[00:17:50 – 00:17:52]** that there's new problems to solve.

**[00:17:52 – 00:17:55]** And there are bigger ones. Let's bound the kind of problems.

**[00:17:55 – 00:17:58]** Yeah, just maybe very quickly as a follow up, I think I agree that,

**[00:17:58 – 00:18:02]** I mean, there will always be a bottleneck, but maybe a provocative

**[00:18:02 – 00:18:07]** question is, or reflection is also, maybe, you know, we

**[00:18:07 – 00:18:09]** have so much that we are building.

**[00:18:09 – 00:18:12]** So we were discussing about that also before, like, we

**[00:18:12 – 00:18:14]** are building models that we don't even have the time to

**[00:18:14 – 00:18:16]** verify understanding of thoughts.

**[00:18:16 – 00:18:17]** We cannot have that.

**[00:18:17 – 00:18:20]** We don't have the computer. We don't have the time to do that.

**[00:18:20 – 00:18:24]** So what if maybe at some point the problem will not be, like,

**[00:18:24 – 00:18:27]** how do we compute more and now we get better, larger problem,

**[00:18:27 – 00:18:32]** but maybe there is a question, how are we doing that from the right

**[00:18:32 – 00:18:33]** perspective, in the right way?

**[00:18:33 – 00:18:36]** Should we change way in which maybe we look at rewards, we look

**[00:18:37 – 00:18:41]** at verifiable reproducibility and everything in this paradigm change?

**[00:18:41 – 00:18:43]** So I don't know if your question was also related to this aspect

**[00:18:43 – 00:18:46]** of, yeah, it is the problem, but we keep on going, like building

**[00:18:46 – 00:18:50]** faster cars, but maybe we actually don't have routes and streets

**[00:18:50 – 00:18:52]** yet to, to actually play that.

**[00:18:52 – 00:18:55]** So it's not that extreme, but maybe at some point that

**[00:18:55 – 00:18:59]** could be, like, yeah, there is bottleneck in the compute, but also

**[00:18:59 – 00:19:02]** maybe in our understanding ability to, to leverage even more on that.

**[00:19:02 – 00:19:05]** Yeah, exactly.

**[00:19:06 – 00:19:09]** Yes, I just want to say, so again, our approach has been

**[00:19:09 – 00:19:12]** to also have some very simple environments where we can

**[00:19:12 – 00:19:15]** really utilize the compute well.

**[00:19:15 – 00:19:19]** But it is what you're saying, without human data, we have

**[00:19:19 – 00:19:21]** nothing to bootstrap on, right?

**[00:19:21 – 00:19:23]** So we need to do everything from scratch, and that

**[00:19:24 – 00:19:25]** takes extra compute.

**[00:19:25 – 00:19:29]** And once we get there, we will be able to just scale.

**[00:19:29 – 00:19:34]** But I think another dimension, when does it compute enough, is also

**[00:19:34 – 00:19:35]** a bit on how you use a computer.

**[00:19:35 – 00:19:39]** So doing the simple things, so that way you're getting a

**[00:19:39 – 00:19:44]** lot of data very quickly, a lot of data points, is one thing, but also

**[00:19:44 – 00:19:46]** what type of computer we're using.

**[00:19:46 – 00:19:49]** So if you look at the LLM companies now, they have to

**[00:19:49 – 00:19:53]** do what we did 10 years ago with reinforcement learning, where

**[00:19:53 – 00:19:59]** they have giant clusters of CPU machines that are just operating

**[00:19:59 – 00:20:01]** like machines or like sandboxes.

**[00:20:01 – 00:20:04]** So we try to not put ourselves in that position.

**[00:20:04 – 00:20:06]** We've been in it before.

**[00:20:06 – 00:20:07]** Horrible.

**[00:20:07 – 00:20:10]** So, you know, we try to do as much as we can on like

**[00:20:10 – 00:20:15]** the GPUs because obviously like the GPUs is where the flops are coming

**[00:20:15 – 00:20:19]** from, right, and increasingly so.

**[00:20:19 – 00:20:21]** Extending the same thing, right?

**[00:20:21 – 00:20:26]** So you're talking about compute, over provisioning the CPUs,

**[00:20:26 – 00:20:29]** and not talking about you.

**[00:20:29 – 00:20:33]** You're anticipating future experiences from human interaction.

**[00:20:33 – 00:20:39]** How much do you build into the learning system as you wait for it?

**[00:20:39 – 00:20:41]** How do you scale there?

**[00:20:42 – 00:20:43]** That's for you, Karim. That's for me?

**[00:20:43 – 00:20:47]** Yeah. So in some sense, the ideal computer that we would

**[00:20:47 – 00:20:49]** want doesn't exist yet.

**[00:20:49 – 00:20:52]** So we are anticipating that we are moving in that direction.

**[00:20:52 – 00:20:56]** And purely learning from experience and getting to very strong

**[00:20:56 – 00:21:00]** capabilities would require computers that might be possible

**[00:21:00 – 00:21:01]** in five, ten years.

**[00:21:01 – 00:21:03]** So as Nanda was saying, more computer unlock, the things

**[00:21:03 – 00:21:06]** that we want to do.

**[00:21:06 – 00:21:12]** But even now you can solve some of the algorithmic challenges,

**[00:21:12 – 00:21:16]** and those solutions also inform how to make the compute for the future.

**[00:21:16 – 00:21:18]** Because just more compute is not enough.

**[00:21:18 – 00:21:20]** You have to make the right kind of computer to take

**[00:21:20 – 00:21:23]** advantage of learning from experience in real time.

**[00:21:23 – 00:21:28]** And so we are right now, our hope is that we can just focus

**[00:21:28 – 00:21:33]** on the algorithms for now, and then in, I foresee at least,

**[00:21:33 – 00:21:37]** that maybe in three, five years, we might actually be at a point

**[00:21:37 – 00:21:40]** where we can take these algorithms, take the computers from five years

**[00:21:40 – 00:21:43]** ago, five years in the future, and build systems that can do useful

**[00:21:44 – 00:21:47]** things by learning from experience.

**[00:21:48 – 00:21:49]** Nando, any addition there?

**[00:21:49 – 00:21:54]** No, I agree with that point.

**[00:21:54 – 00:21:58]** So moving to what we were discussing last night from

**[00:21:58 – 00:22:04]** there, so maybe you can continue on the same theme, is, if

**[00:22:04 – 00:22:07]** you want to elaborate a little bit more on how it changes

**[00:22:07 – 00:22:09]** the shape of the computation.

**[00:22:09 – 00:22:12]** And I think this is where you and Lasse will have a

**[00:22:13 – 00:22:14]** positive spectrum.

**[00:22:14 – 00:22:19]** So we're... We think that the world is large, there's

**[00:22:19 – 00:22:23]** a long tail, so the best place to learn is in the real world on the

**[00:22:23 – 00:22:25]** actual distribution, not in some...

**[00:22:25 – 00:22:27]** I think you would agree with that too when it comes to

**[00:22:27 – 00:22:28]** making the useful systems.

**[00:22:28 – 00:22:31]** Okay, so whenever you're making algorithms that learn in the

**[00:22:31 – 00:22:34]** real world, that means we don't have to use computation

**[00:22:34 – 00:22:35]** to simulate, for example.

**[00:22:35 – 00:22:38]** So the shape of computation is quite different.

**[00:22:38 – 00:22:39]** It all goes towards the agent.

**[00:22:39 – 00:22:41]** There is no need to simulate the world.

**[00:22:41 – 00:22:45]** Now, inside the agent, we're also betting on a kind of

**[00:22:45 – 00:22:48]** algorithm that learn immediately from their data stream.

**[00:22:48 – 00:22:50]** So your sensory data is coming in, you process it, you learn

**[00:22:50 – 00:22:51]** it, you throw it away.

**[00:22:51 – 00:22:52]** You don't have to store it.

**[00:22:52 – 00:22:56]** And if you learn it this way, that opens new kind of algorithmic

**[00:22:56 – 00:22:58]** efficiencies that are not possible with the training

**[00:22:58 – 00:23:00]** paradigm of large batch learning.

**[00:23:00 – 00:23:02]** So what are these efficiencies?

**[00:23:02 – 00:23:05]** One is that if you're learning, if you have a large agent

**[00:23:05 – 00:23:08]** and then you're learning from sensory data, at every step,

**[00:23:08 – 00:23:10]** you're not going to have to touch all of your parameters.

**[00:23:10 – 00:23:12]** So the brain is also like that.

**[00:23:12 – 00:23:13]** You're going to touch a subset of them.

**[00:23:13 – 00:23:16]** So you can design your implementation so that

**[00:23:16 – 00:23:18]** they only touch the part of the network that is useful

**[00:23:18 – 00:23:20]** and not touch the rest of the part.

**[00:23:20 – 00:23:25]** And that changes your compute from accessing this static.

**[00:23:25 – 00:23:27]** It's a pattern of memory access, so you stream in the whole weights,

**[00:23:27 – 00:23:31]** that's what happens in current LLMs, to a much more dynamic

**[00:23:31 – 00:23:34]** set of computation, where you don't know what parameters you need

**[00:23:34 – 00:23:37]** to touch at a given point, but it's going to be a small subset, so if

**[00:23:37 – 00:23:41]** you only touch those dynamically, you can be much more efficient.

**[00:23:41 – 00:23:45]** And if I were to use an analogy, the computation looks more similar

**[00:23:45 – 00:23:47]** to the computation in video games.

**[00:23:47 – 00:23:49]** In video games, real-time rendering, you know, you're

**[00:23:49 – 00:23:51]** looking at one side, you're rendering those things, you're

**[00:23:51 – 00:23:54]** getting the objects, you get their vertices, you look somewhere

**[00:23:54 – 00:23:58]** else, suddenly the assets change completely, so it's very dynamic.

**[00:23:58 – 00:24:01]** And that is sort of what the computation looks like for

**[00:24:01 – 00:24:04]** our algorithms as well.

**[00:24:06 – 00:24:11]** Yeah, so I think from our perspective, I agree that

**[00:24:11 – 00:24:14]** if you could do everything in the real world and scale that

**[00:24:14 – 00:24:16]** up, then that's certainly better.

**[00:24:16 – 00:24:19]** Like where the disagreement comes in is that we do believe

**[00:24:19 – 00:24:23]** that you can create like a general learning algorithm first for

**[00:24:23 – 00:24:29]** like very simple environments and increase the complexity from there.

**[00:24:29 – 00:24:31]** And I don't even think there's a disagreement.

**[00:24:31 – 00:24:36]** I think that's a perfectly reasonable spread.

**[00:24:37 – 00:24:40]** Zihang, do you have anything?

**[00:24:40 – 00:24:44]** Yeah, so my take on this one is, for all the head

**[00:24:44 – 00:24:48]** problems, so actually the clusters are pretty useful.

**[00:24:48 – 00:24:53]** But if you really want to cover all the long tails, that becomes

**[00:24:53 – 00:24:55]** a little bit more challenging.

**[00:24:55 – 00:24:58]** When I say cover long tails, it's not just about training,

**[00:24:58 – 00:25:00]** also for evaluation.

**[00:25:00 – 00:25:04]** So I think, yeah, that's probably my feeling.

**[00:25:04 – 00:25:07]** If we only need to care about a few tasks, let's say we just

**[00:25:07 – 00:25:12]** want to make perfect coding agents, probably the current way works.

**[00:25:12 – 00:25:16]** If we really want to reach another level of intelligence

**[00:25:16 – 00:25:19]** that is able to solve different types of tasks on the fly

**[00:25:19 – 00:25:22]** and also learn on the fly

**[00:25:24 – 00:25:27]** I will challenge myself whether the current regime is possible or

**[00:25:27 – 00:25:32]** not, or at least it's not trivial to just extend what we have.

**[00:25:32 – 00:25:35]** Yeah, I like to visualize a graph where you have tasks

**[00:25:36 – 00:25:39]** on one axis and number of people who use that task.

**[00:25:39 – 00:25:42]** And if a number of people is large enough, then the current paradigm

**[00:25:42 – 00:25:43]** works, you know, like coding.

**[00:25:43 – 00:25:45]** There are many, many users.

**[00:25:45 – 00:25:48]** But as you go toward a long tail, maybe five people need

**[00:25:48 – 00:25:51]** a specific automation, then the current paradigm just

**[00:25:51 – 00:25:54]** breaks apart, and you really need to learn from experience.

**[00:25:54 – 00:25:57]** So like you're solving a specific task and you're

**[00:25:57 – 00:25:58]** solving the whole world.

**[00:25:58 – 00:26:00]** Those are two different things.

**[00:26:00 – 00:26:01]** And, like...

**[00:26:01 – 00:26:04]** I think kind of posing it to you differently, Nando,

**[00:26:04 – 00:26:08]** so you mentioned in the beginning that the sequential nature

**[00:26:08 – 00:26:12]** of the inference keeps things a little bit more limited

**[00:26:12 – 00:26:15]** as well as more expensive compared to where you're headed, like

**[00:26:15 – 00:26:18]** paralyzing the whole computation.

**[00:26:18 – 00:26:22]** Yeah, I mean, what we do is very specific.

**[00:26:22 – 00:26:28]** It's sort of aside from this discussion of whether you

**[00:26:28 – 00:26:31]** learn from scratch or try to learn from others.

**[00:26:31 – 00:26:35]** And it's more the question of how fast is that learning,

**[00:26:35 – 00:26:39]** so regardless of which approach you're using.

**[00:26:39 – 00:26:44]** So basically, I mean, transformers became popular because they

**[00:26:44 – 00:26:45]** parallelized training.

**[00:26:45 – 00:26:48]** So we moved from RNNs, which were sequential, to transformers

**[00:26:48 – 00:26:51]** because training got parallel.

**[00:26:51 – 00:26:54]** Inference was never parallelized.

**[00:26:54 – 00:26:58]** And the reason is because in inference, in game environments,

**[00:26:58 – 00:27:02]** robotics, you typically predict just a few things, just a

**[00:27:02 – 00:27:05]** few steps, just not something big.

**[00:27:05 – 00:27:09]** In language, chatbots only predict a few sentences.

**[00:27:09 – 00:27:12]** But that whole thing changed like this year, of course,

**[00:27:12 – 00:27:14]** you know, we're now predicting pages and pages of code and

**[00:27:14 – 00:27:18]** pages of reports and so on.

**[00:27:18 – 00:27:24]** And so I don't think it's no longer wise to put so much energy

**[00:27:24 – 00:27:26]** in doing sequential inference.

**[00:27:26 – 00:27:29]** We need parallel inference as well.

**[00:27:29 – 00:27:32]** And so that's the bit that we focus on.

**[00:27:32 – 00:27:36]** It's like trying to figure out how to design the algorithms.

**[00:27:37 – 00:27:40]** You know, there are these things that are orthogonal to

**[00:27:40 – 00:27:42]** which philosophy you adopt, right?

**[00:27:42 – 00:27:45]** And regardless of which philosophy you adopt, you still need

**[00:27:45 – 00:27:49]** to figure out what network you use, what layers, whether

**[00:27:49 – 00:27:55]** you put dropout there or not.

**[00:27:55 – 00:27:57]** And this too, RL can help meta-optimise.

**[00:27:57 – 00:28:01]** And even in fact, in AlphaGo, we were already meta-optimising

**[00:28:01 – 00:28:02]** using Bayesian optimisation.

**[00:28:02 – 00:28:06]** That was the thing that we deployed to Bitlyse at all.

**[00:28:07 – 00:28:11]** And so you can do that to some extent, but there's still a

**[00:28:11 – 00:28:16]** lot of progress we need to make in many directions to kind of enable

**[00:28:16 – 00:28:20]** research, whether it's specific to code or whether it's something

**[00:28:20 – 00:28:26]** that could help, say, LASA come up with this amazing algorithm

**[00:28:26 – 00:28:29]** from few, you know, if the tools,

**[00:28:30 – 00:28:34]** it's about creating tools that others can use more effectively.

**[00:28:35 – 00:28:39]** I'll come back to you again on your paralyzing the inference

**[00:28:39 – 00:28:41]** and hence the topic we were discussing in the morning.

**[00:28:42 – 00:28:46]** I just wanted to give you a chance on...

**[00:28:46 – 00:28:48]** Your academic research, like, does it really matter?

**[00:28:48 – 00:28:53]** Like, are you thinking about compute that much right now?

**[00:28:53 – 00:28:57]** Well, yes, in the sense of we still think about compute

**[00:28:57 – 00:28:59]** and we get a good support.

**[00:28:59 – 00:29:03]** But I think maybe the point to do here, to make here,

**[00:29:03 – 00:29:06]** is that sometimes having limited compute can also

**[00:29:06 – 00:29:08]** lead to different directions.

**[00:29:08 – 00:29:11]** So I think it's open-ended also, the shape of compute, and is

**[00:29:11 – 00:29:16]** highly linked to to take a system in which science is built, right?

**[00:29:17 – 00:29:21]** So in our case, for example, we had in BOLD, we had a very

**[00:29:21 – 00:29:24]** last year a very small moment in which we had access to

**[00:29:24 – 00:29:28]** a specific national compute in this trial phase, and that actually

**[00:29:28 – 00:29:31]** put us together, say, okay, let's have moonshot discussion, see

**[00:29:31 – 00:29:35]** how to exploit these resources that for us was very unique position.

**[00:29:35 – 00:29:37]** And that led us to think like, OK, maybe we do evolutionary

**[00:29:37 – 00:29:40]** strategies for optimizing neural network, but it's too costly.

**[00:29:40 – 00:29:43]** We don't have the time because we don't have these resources forever.

**[00:29:43 – 00:29:45]** We have it only for one month.

**[00:29:45 – 00:29:49]** So we didn't start like, OK, let's apply more low-rank

**[00:29:49 – 00:29:51]** matrix factorization.

**[00:29:51 – 00:29:54]** And that led to what we have now today at Agro.

**[00:29:54 – 00:29:58]** That is actually challenging the backpropagation in our

**[00:29:58 – 00:30:00]** cases in which you don't have differentiability.

**[00:30:00 – 00:30:04]** And now we are expanding that in distributed systems, so completely

**[00:30:04 – 00:30:07]** different compute style and shape.

**[00:30:07 – 00:30:12]** So we always need compute also on our side, but I think sometimes

**[00:30:12 – 00:30:18]** the situations lead to also then develop things differently.

**[00:30:19 – 00:30:27]** So coming back to your point, Nando, on you really have to think

**[00:30:27 – 00:30:31]** about different dimensions than when you're doing the research.

**[00:30:31 – 00:30:35]** So the point that you touched upon this morning was, on

**[00:30:35 – 00:30:38]** one hand, we're talking about continual learning.

**[00:30:38 – 00:30:41]** And coming back from the past, the scaling laws and

**[00:30:41 – 00:30:44]** understanding of a deployed model are very well understood.

**[00:30:44 – 00:30:46]** So then we are talking about reinforcement learning and

**[00:30:46 – 00:30:47]** continual learning.

**[00:30:47 – 00:30:50]** So you were mentioning, or you touched upon, the

**[00:30:50 – 00:30:54]** topic of outcome-based RL, verification of it.

**[00:30:54 – 00:30:59]** If you want to discuss a little bit more in detail?

**[00:30:59 – 00:31:04]** I mean, that is a problem that is still open, and I think

**[00:31:04 – 00:31:08]** that's what could be said there.

**[00:31:09 – 00:31:11]** I think there is some, fundamentally, some problems

**[00:31:11 – 00:31:13]** we never answered.

**[00:31:13 – 00:31:17]** So, we know how to teach agents through reinforcement

**[00:31:17 – 00:31:19]** learning in an environment to learn to communicate.

**[00:31:19 – 00:31:28]** Not teach, but as long as there's tasks, agents learn to communicate.

**[00:31:29 – 00:31:33]** No one has ever demonstrated how in that sort of setup

**[00:31:33 – 00:31:38]** you could learn, get agents that learn to learn language

**[00:31:40 – 00:31:43]** Craft the thing called language that we crafted.

**[00:31:43 – 00:31:46]** So this is still one of the big mysteries.

**[00:31:46 – 00:31:50]** So I'm really glad that there's folks looking at this because

**[00:31:50 – 00:31:54]** I think you need to answer this question to be able to

**[00:31:54 – 00:31:58]** make the sort of next big advances.

**[00:31:58 – 00:32:02]** And yeah, and language is just basically... A tool, and we learned

**[00:32:02 – 00:32:07]** to harness this tool somehow, and every time we have a new tool, the

**[00:32:07 – 00:32:11]** tool changes our mind, just like our phones now have changed the

**[00:32:11 – 00:32:14]** way we think and the way we behave.

**[00:32:14 – 00:32:17]** And language was this tool that changed this, and we

**[00:32:17 – 00:32:19]** became very good at it.

**[00:32:19 – 00:32:21]** And so it's not surprising that all the big advances

**[00:32:21 – 00:32:22]** are happening at the same time.

**[00:32:22 – 00:32:29]** Code, math, because this is all symbolic manipulation of this tool.

**[00:32:30 – 00:32:32]** But yeah, there's lots of questions we haven't figured

**[00:32:32 – 00:32:34]** out, so that would be interesting.

**[00:32:34 – 00:32:38]** And of course, that is today's reality as well, which is,

**[00:32:38 – 00:32:42]** how do you do this efficiently in a computational, reasonable way?

**[00:32:42 – 00:32:48]** And I think it requires... that we shouldn't just be thinking

**[00:32:48 – 00:32:52]** about compute, but how do you make, the real problem is energy, is how

**[00:32:52 – 00:32:57]** to make better use of the energy to do, to get the same amount

**[00:32:57 – 00:33:00]** of compute, so try to come up with, say, with parallel inference

**[00:33:00 – 00:33:05]** or the distributed algorithms, like basically the swarms, and

**[00:33:05 – 00:33:12]** then that opens up the whole game theoretic way of doing things, so

**[00:33:13 – 00:33:16]** It gets very interesting.

**[00:33:16 – 00:33:18]** AlphaGo is good because you learn to play against each

**[00:33:18 – 00:33:20]** other, self-play.

**[00:33:20 – 00:33:26]** So we get these Nash-type races, but then there's also cooperation,

**[00:33:26 – 00:33:28]** so that's all very exciting.

**[00:33:28 – 00:33:30]** Right.

**[00:33:31 – 00:33:33]** So you're talking about complexity as you try to handle more

**[00:33:33 – 00:33:36]** and more things, and then complexity of measurement.

**[00:33:36 – 00:33:43]** So let's say, if you would like to spend some time on, you're talking

**[00:33:43 – 00:33:46]** about, so we're talking about language, and LLMs, and text-based.

**[00:33:46 – 00:33:49]** And then you're talking about handling all kinds

**[00:33:49 – 00:33:50]** of experiences, right?

**[00:33:50 – 00:33:53]** How Silver says, era of experience.

**[00:33:53 – 00:33:57]** So distributed computation, how is everything coming together?

**[00:33:57 – 00:34:01]** Like, are you even worried about all the ways you will have

**[00:34:01 – 00:34:04]** to measure the system, whether it's actually performing well, whether

**[00:34:04 – 00:34:06]** it's behaving as it's intended to?

**[00:34:06 – 00:34:08]** How do you evaluate?

**[00:34:08 – 00:34:09]** What was the last part, sorry?

**[00:34:09 – 00:34:11]** So how are you measuring the behavior?

**[00:34:11 – 00:34:13]** How are you evaluating the behavior of the system?

**[00:34:13 – 00:34:16]** So as your experiences, interactions, and then the

**[00:34:16 – 00:34:23]** learners, they are learning, uh, that's like a critical part of...

**[00:34:24 – 00:34:27]** So we are trying to design our environments in ways where

**[00:34:27 – 00:34:32]** we are not shaving the walls to solve a task in a specific

**[00:34:33 – 00:34:36]** way, or to come up with language in a very specific way.

**[00:34:36 – 00:34:40]** We're trying to design the walls such that... like the

**[00:34:40 – 00:34:43]** algorithms will come up with something like language, right?

**[00:34:43 – 00:34:46]** And there's like a lot of like interesting papers

**[00:34:46 – 00:34:49]** on like how language or communication arose, right?

**[00:34:49 – 00:34:53]** Not just for humans, but also for bacteria and so on.

**[00:34:53 – 00:34:55]** So I think that's like the interesting part for us, right?

**[00:34:55 – 00:34:59]** Like, you know, we don't want to have like a reward for

**[00:34:59 – 00:35:00]** coming up with a computer.

**[00:35:00 – 00:35:05]** We want like a reward, where a computer may be a necessary

**[00:35:05 – 00:35:11]** part or a necessary technology to actually get to that reward.

**[00:35:12 – 00:35:16]** Oh, OK. So I have to move fast now.

**[00:35:16 – 00:35:19]** So leaving it up to you on evaluations, if you want to

**[00:35:20 – 00:35:22]** touch a point for a minute.

**[00:35:22 – 00:35:26]** I think for me, it's just like for anything, like I said,

**[00:35:26 – 00:35:30]** if we already know the environment, we already have the reward, we can

**[00:35:30 – 00:35:33]** design the environment really well.

**[00:35:33 – 00:35:36]** I think evaluation is easy, but for long tail, I just don't

**[00:35:36 – 00:35:40]** know how to evaluate that properly.

**[00:35:41 – 00:35:44]** And also another thing I think related to what Nando said,

**[00:35:44 – 00:35:48]** or also Lawson said, I think one advantage we're taking,

**[00:35:48 – 00:35:51]** basically we're taking advantage of pre-training in the sense

**[00:35:51 – 00:35:55]** the interface is language; everybody understands language.

**[00:35:55 – 00:35:58]** Language is just one way to tokenize this world, one way.

**[00:35:59 – 00:36:03]** But there are many ways, and they might be much more efficient.

**[00:36:03 – 00:36:06]** But even if we get there, how do we know?

**[00:36:06 – 00:36:09]** What language or what models are thinking or what models

**[00:36:09 – 00:36:13]** are doing, that would be another even more challenging problem.

**[00:36:13 – 00:36:17]** Even now, if you only look at the channel of thought

**[00:36:17 – 00:36:20]** from the model, I'm not 100% sure that's what the model

**[00:36:20 – 00:36:23]** is really thinking or speaking.

**[00:36:23 – 00:36:27]** So, yeah, for me, human understanding is one part,

**[00:36:28 – 00:36:30]** and long tail, how to actually know the reward is the other

**[00:36:30 – 00:36:34]** part, which I don't really know.

**[00:36:34 – 00:36:38]** Zihang, there's one question that we wanted to record for eternity.

**[00:36:38 – 00:36:42]** I'm sorry, I'm getting benchmarks and all.

**[00:36:42 – 00:36:46]** If we meet again three years from now, what would we be

**[00:36:46 – 00:36:50]** talking about in terms of learning?

**[00:36:51 – 00:36:57]** I would say whether humans will be able to understand

**[00:36:57 – 00:37:00]** what models are doing will be something we will be talking

**[00:37:00 – 00:37:07]** about in three years, especially when models are everywhere.

**[00:37:07 – 00:37:13]** And we kind of understand part of it, but we don't really understand.

**[00:37:13 – 00:37:18]** And then, no matter if it's for improvement or for evaluation,

**[00:37:18 – 00:37:20]** we have to spend more time there.

**[00:37:21 – 00:37:22]** Still evaluating.

**[00:37:22 – 00:37:25]** I think evaluation and reward, they are the same thing at

**[00:37:25 – 00:37:27]** the end of the day.

**[00:37:27 – 00:37:29]** What gets measured gets improved.

**[00:37:29 – 00:37:32]** Just how do you measure the thing?

**[00:37:32 – 00:37:35]** How do you know you are measuring the correct thing?

**[00:37:35 – 00:37:38]** Yeah, the correct way.

**[00:37:38 – 00:37:40]** Laura, you wanna?

**[00:37:40 – 00:37:42]** Yeah, I can jump on that.

**[00:37:42 – 00:37:45]** So the vision I have, and I also connect to what we've

**[00:37:45 – 00:37:51]** seen before very quickly, is that maybe we will have a different

**[00:37:51 – 00:37:54]** way of also looking at this aspect of Eval and benchmarking

**[00:37:54 – 00:37:57]** in the future that is linked to the evolution of AI toward

**[00:37:57 – 00:38:01]** open-endedness and interaction in the society with human,

**[00:38:01 – 00:38:02]** other agents, and maybe itself.

**[00:38:02 – 00:38:05]** So I think the hope is that beyond the evolution that

**[00:38:06 – 00:38:10]** we see in AI, we will also see a strong provenance and not only open

**[00:38:10 – 00:38:15]** source, but also open provenance...

**[00:38:15 – 00:38:19]** Understanding on how the models have been unfolded, right?

**[00:38:19 – 00:38:21]** Like beyond the chain of thoughts, like how we reach the point.

**[00:38:21 – 00:38:25]** So we will have more people asking the how and why.

**[00:38:25 – 00:38:27]** And there will be this.

**[00:38:27 – 00:38:31]** We would like to hope that every research that we do will have

**[00:38:31 – 00:38:34]** the entire process of how we led there and what was the reasoning

**[00:38:34 – 00:38:38]** and all the thinking of the human on top of the agent behind.

**[00:38:38 – 00:38:42]** And that will create a collective intelligence that will create

**[00:38:42 – 00:38:46]** ideally even more breakthrough, and even more possible also

**[00:38:46 – 00:38:49]** in academia, maybe.

**[00:38:49 – 00:38:53]** Pinnando, what will you be thinking about three years from now?

**[00:38:53 – 00:38:56]** I mean, I would love, I guess, to wake up in a time where

**[00:38:56 – 00:39:03]** we see these models creating tools, and not using tools,

**[00:39:03 – 00:39:07]** creating tools that we can't dream of today, because there was

**[00:39:07 – 00:39:12]** a time where people couldn't dream of beyond the wildest dreams about

**[00:39:12 – 00:39:22]** electron macroscopes or telescopes or computers or even mathematics

**[00:39:24 – 00:39:27]** But we created those tools,

**[00:39:27 – 00:39:30]** and it's, it would be amazing if we had models that could

**[00:39:30 – 00:39:34]** create some amazing tools, and, and hopefully that we could

**[00:39:34 – 00:39:38]** somehow understand something out of it, because it would be very sad

**[00:39:38 – 00:39:41]** if we don't get anything from it.

**[00:39:41 – 00:39:45]** It's a new tool.

**[00:39:45 – 00:39:46]** Let's see.

**[00:39:46 – 00:39:50]** Yeah, well, so part of our mission statement is creating

**[00:39:50 – 00:39:52]** new technology and tools, right?

**[00:39:52 – 00:39:56]** And being optimistic, I would want to have this super learner,

**[00:39:56 – 00:39:59]** super intelligent algorithm that we actually agree on

**[00:39:59 – 00:40:02]** is super intelligent, and actually see this emerging

**[00:40:02 – 00:40:09]** of new technology and deploy it in the real world as well.

**[00:40:10 – 00:40:12]** I think, wrap it up for us.

**[00:40:13 – 00:40:16]** I mean, I hope in three years, we are talking about

**[00:40:17 – 00:40:18]** our specific agents.

**[00:40:18 – 00:40:20]** Like, my agent can do this because I taught it this thing,

**[00:40:20 – 00:40:22]** and you will tell me what your agent can do.

**[00:40:22 – 00:40:25]** And I think that would be possible.

**[00:40:25 – 00:40:26]** I think that is where we are headed.

**[00:40:26 – 00:40:30]** If we can crack computationally efficient continual learning,

**[00:40:30 – 00:40:33]** we're headed towards a future where all of us would maybe

**[00:40:33 – 00:40:35]** start with the base agent, but then we will specialize

**[00:40:35 – 00:40:39]** them, and they will learn things that we want them to learn.

**[00:40:39 – 00:40:41]** So it's very much possible that we will all be replaced

**[00:40:41 – 00:40:43]** by our agents in this room.

**[00:40:43 – 00:40:46]** No, I don't think so. I think more likely everyone would

**[00:40:46 – 00:40:49]** have at least one, maybe multiple agents, and they would be proud

**[00:40:49 – 00:40:52]** of showing what the agents can do.

**[00:40:52 – 00:40:55]** So their agents would do really well on the things they care

**[00:40:55 – 00:40:57]** about, and they'll show it off to each other.

**[00:40:57 – 00:40:59]** Well, thank you so much.

**[00:40:59 – 00:41:02]** Thanks for your time, and it was a great panel.

**[00:41:02 – 00:41:03]** Thank you. Thanks, everyone.

