# Beyond Deployment: Engineering Agents That Learn on the Job

## Abstract

Key takeaways Enterprise AI agents are ready for production—This session covers the architecture patterns that make them reliable, observable, and designed to improve without human intervention. The latest NVIDIA Agent Toolkit innovations enable agents that learn while they operate, using traces, telemetry, and feedback loops to get better on their own, with a full audit trail of every decision. Intelligent model routing selects the right model for each task at runtime—balancing capability and efficiency across simple tool calls and deep reasoning steps within the same workflow.

## AI Summary

_No AI summary available._

## Transcript

**[00:00:05 – 00:00:06]** All righty.

**[00:00:07 – 00:00:09]** Good morning, everyone. Guten Morgen.

**[00:00:09 – 00:00:13]** Welcome to day two of the Agentic AI in Clause track

**[00:00:14 – 00:00:16]** here at GDC Berlin.

**[00:00:16 – 00:00:20]** If you joined us yesterday or you're here for the first time,

**[00:00:20 – 00:00:21]** we're pretty glad to have you.

**[00:00:21 – 00:00:23]** So thanks for coming out. I know it's early.

**[00:00:24 – 00:00:27]** My name is Chris, your emcee, and this room is home to

**[00:00:27 – 00:00:31]** Agentic AI in Clause sessions throughout the whole day.

**[00:00:31 – 00:00:33]** So that's the topic for the day.

**[00:00:34 – 00:00:36]** We explored yesterday what it takes to bring agents into

**[00:00:37 – 00:00:39]** production, and today we're going to dive a little bit

**[00:00:39 – 00:00:43]** deeper into agents and open models, and we're going to be hearing

**[00:00:43 – 00:00:48]** from experts and partners about how they're putting agents to work.

**[00:00:48 – 00:00:51]** We're going to explore how they can learn and improve,

**[00:00:51 – 00:00:54]** how to customize open models for specialized tasks, and

**[00:00:54 – 00:00:58]** how to build in governance and control so that you can deploy

**[00:00:58 – 00:01:01]** these things in the real world.

**[00:01:01 – 00:01:05]** You can expect insights that are practical, technical,

**[00:01:05 – 00:01:07]** and lessons from people who are actually building these

**[00:01:07 – 00:01:09]** systems in production.

**[00:01:09 – 00:01:13]** To get us started today, we're going to kick off with what happens

**[00:01:13 – 00:01:16]** after you deploy your agent.

**[00:01:16 – 00:01:20]** We're going to learn how we can keep learning from the experience

**[00:01:20 – 00:01:22]** it gets while it's doing work.

**[00:01:22 – 00:01:25]** Our first session is called Beyond Deployment, engineering

**[00:01:25 – 00:01:26]** agents that learn on the job.

**[00:01:26 – 00:01:30]** Presenting is going to be Adel El Hallak, NVIDIA's vice

**[00:01:30 – 00:01:34]** president of product management, and Matt Penn, a principal solution

**[00:01:34 – 00:01:37]** architect of agentic AI at NVIDIA.

**[00:01:37 – 00:01:40]** They're going to explore how to build reliable, observable

**[00:01:40 – 00:01:44]** agents that learn from traces and feedback, with auditable

**[00:01:44 – 00:01:48]** decisions and model routing that balances capability and efficiency.

**[00:01:49 – 00:01:51]** There is going to be some time for QA, so we're going

**[00:01:52 – 00:01:53]** to be taking some questions.

**[00:01:53 – 00:01:58]** Please join me in welcoming Adel and Matt.

**[00:02:02 – 00:02:05]** All right. Good morning, Berlin.

**[00:02:05 – 00:02:10]** Guys, okay, quick caveat.

**[00:02:10 – 00:02:11]** We haven't slept much.

**[00:02:11 – 00:02:14]** We're very jet-lagged.

**[00:02:14 – 00:02:18]** We have a whole bunch of demos that we're going to show, and Matt

**[00:02:18 – 00:02:22]** and I haven't gotten the chance to rehearse this presentation.

**[00:02:22 – 00:02:25]** So what could possibly go wrong, right?

**[00:02:25 – 00:02:27]** Look, we've got a lot to cover. We're going to kick it off.

**[00:02:27 – 00:02:29]** All right.

**[00:02:29 – 00:02:32]** The agenda is as follows.

**[00:02:32 – 00:02:36]** Hopefully, you're going to recognize adoption of agents

**[00:02:36 – 00:02:40]** is accelerating and accelerating in the enterprise.

**[00:02:40 – 00:02:42]** We're going to share with you the learnings that we

**[00:02:42 – 00:02:45]** have in deploying an Open Agent Stack internally at NVIDIA.

**[00:02:45 – 00:02:49]** And more importantly, we're going to talk about all the levers,

**[00:02:49 – 00:02:53]** all the traces, trajectories, all the techniques that you

**[00:02:53 – 00:02:58]** can leverage to optimize and improve your agent systems.

**[00:02:58 – 00:03:01]** We've got a lot to cover, so let's just kick it off.

**[00:03:01 – 00:03:04]** Okay, so Jensen broke down the anatomy of an agent

**[00:03:04 – 00:03:05]** yesterday, right?

**[00:03:05 – 00:03:07]** He talked about five things.

**[00:03:07 – 00:03:12]** It starts with the model or a set of models, the brain,

**[00:03:12 – 00:03:13]** if you will, right?

**[00:03:13 – 00:03:15]** The brain of the agents.

**[00:03:15 – 00:03:19]** He then talks about a harness, and what a harness is essentially is

**[00:03:19 – 00:03:21]** the exoskeleton of an agent, right?

**[00:03:21 – 00:03:26]** If the models are a brain, then the harness is what gives it its

**[00:03:26 – 00:03:31]** abilities to utilize tools, be able to plan, reason that orchestration

**[00:03:31 – 00:03:35]** loop, be able to store context, be able to manage memory.

**[00:03:36 – 00:03:38]** We then talk about a couple of things that you need to

**[00:03:38 – 00:03:40]** be able to connect to the agents.

**[00:03:40 – 00:03:43]** You wanna give it access to your corpus of data, your

**[00:03:43 – 00:03:47]** knowledge base, your enterprise systems, where the IP of every

**[00:03:47 – 00:03:48]** enterprise resides.

**[00:03:48 – 00:03:51]** We call that the knowledge, your knowledge foundations.

**[00:03:51 – 00:03:54]** And finally, for agents to be useful, they gotta

**[00:03:54 – 00:03:56]** have access to tools.

**[00:03:56 – 00:04:02]** And then the final thing is, and that's a given, is a runtime

**[00:04:02 – 00:04:05]** that we call Open Shell that ensures that every agent remains

**[00:04:06 – 00:04:10]** in a sandbox and doesn't exceed the policies or the permissions or the

**[00:04:10 – 00:04:12]** access that you give it to, right?

**[00:04:12 – 00:04:16]** So the NVIDIA Agent Toolkit is the set of libraries and the models

**[00:04:16 – 00:04:20]** and the reference implementations that we build, that we provide,

**[00:04:20 – 00:04:25]** work with the ecosystem so folks can build open agent stacks.

**[00:04:25 – 00:04:27]** Okay.

**[00:04:28 – 00:04:31]** This is an outdated slide.

**[00:04:32 – 00:04:33]** This goes all the way to August.

**[00:04:33 – 00:04:36]** If you saw yesterday, Jensen, I think, went one month beyond

**[00:04:36 – 00:04:37]** or a couple months beyond.

**[00:04:37 – 00:04:41]** The growth is at 34x.

**[00:04:41 – 00:04:45]** And really what this tells you is that, frankly, this

**[00:04:45 – 00:04:46]** isn't token maxing.

**[00:04:46 – 00:04:50]** This is because agents are useful now.

**[00:04:50 – 00:04:51]** Agents are useful now.

**[00:04:51 – 00:04:55]** And so I expect this acceleration to continue to grow.

**[00:04:55 – 00:04:58]** That's why we're super excited about all the energy and the

**[00:04:58 – 00:05:01]** ecosystem we have here in Berlin.

**[00:05:02 – 00:05:04]** So useful AI has arrived.

**[00:05:04 – 00:05:06]** I took a couple of examples.

**[00:05:06 – 00:05:10]** These aren't the fancy video demo kind of examples, but

**[00:05:10 – 00:05:14]** these are examples, hopefully they're going to play.

**[00:05:14 – 00:05:18]** These are examples of agents that we deploy internally at NVIDIA.

**[00:05:18 – 00:05:22]** The first one is a visual inspection agent.

**[00:05:22 – 00:05:25]** Basically, you're just asking your agent to build a pipeline,

**[00:05:25 – 00:05:29]** a video pipeline, that allows me to do vision, language,

**[00:05:29 – 00:05:31]** action, understanding, be able to archive it.

**[00:05:31 – 00:05:34]** And it's not enough to just deploy that pipeline.

**[00:05:34 – 00:05:38]** Go test the pipeline, validate it, do the QA against it before

**[00:05:38 – 00:05:39]** you hand it over to me.

**[00:05:39 – 00:05:41]** On the right-hand side, we have an SRE agent.

**[00:05:42 – 00:05:43]** Matt told me, like, there's nobody in the enterprise that

**[00:05:43 – 00:05:45]** doesn't want an SRE agent, right?

**[00:05:45 – 00:05:49]** If you look at the prompts that were provided here earlier, you

**[00:05:49 – 00:05:50]** asked, hey, why did my job fail?

**[00:05:50 – 00:05:53]** You forgot to attach the data set to the manifest.

**[00:05:53 – 00:05:56]** Okay, hey, I noticed one of the temperatures of the GPU

**[00:05:56 – 00:05:58]** is rising more than the others.

**[00:05:58 – 00:05:59]** Why is that?

**[00:05:59 – 00:06:03]** It breaks down analysis.

**[00:06:03 – 00:06:05]** You then ask it to create a dashboard

**[00:06:05 – 00:06:07]** so that you're able to monitor.

**[00:06:07 – 00:06:11]** So these are the types of examples that we utilize here at NVIDIA,

**[00:06:11 – 00:06:14]** and we've got plenty of those.

**[00:06:14 – 00:06:18]** Today we're going to talk about improving an agent system.

**[00:06:18 – 00:06:23]** But I think it is critical that we start off by just

**[00:06:23 – 00:06:26]** reminding everybody that it is super critical that every

**[00:06:26 – 00:06:32]** agent that we deploy Has the right control mechanisms, because

**[00:06:32 – 00:06:38]** advancing AI also means advancing safety and advancing security.

**[00:06:38 – 00:06:44]** Two-thirds of enterprises have said they've frankly had incidents.

**[00:06:44 – 00:06:48]** We see about half of them where the agents exceed the

**[00:06:48 – 00:06:51]** policies they've been provided.

**[00:06:51 – 00:06:55]** And only about one-fifth today, less than one-fifth today,

**[00:06:55 – 00:06:57]** have the controls in place to ensure they're able to.

**[00:06:57 – 00:07:03]** So we're not going to spend too much time talking about

**[00:07:03 – 00:07:08]** OpenShell, but it is the only non-negotiable thing at NVIDIA

**[00:07:08 – 00:07:09]** in terms of deploying agents.

**[00:07:09 – 00:07:13]** Every one of our agents runs in that secure runtime.

**[00:07:13 – 00:07:16]** Now this is NVIDIA's AI factory.

**[00:07:16 – 00:07:20]** This is how we generate our local intelligence, and this is

**[00:07:20 – 00:07:22]** how we run our agents internally.

**[00:07:23 – 00:07:25]** A couple of things to call out here.

**[00:07:25 – 00:07:27]** The Secure Agent Workspace is something we've published.

**[00:07:27 – 00:07:30]** It shows our know-how of how we give agents identity, all

**[00:07:30 – 00:07:35]** the core services that are required for your actual agents.

**[00:07:35 – 00:07:37]** The other piece to call out over here is we're

**[00:07:37 – 00:07:40]** running on Kubernetes, so we partnered with Red Hat.

**[00:07:40 – 00:07:42]** We're running on Kubernetes.

**[00:07:42 – 00:07:46]** And then, as you can see, we have access to our We have

**[00:07:46 – 00:07:48]** our MCP gateway, which is our tool gateway; we have access

**[00:07:48 – 00:07:54]** to our inference hub, which is our LLM gateway; and frankly, there's a

**[00:07:54 – 00:07:57]** lot more of the ecosystem partners that we leverage over here, whether

**[00:07:57 – 00:08:00]** for observability or beyond.

**[00:08:00 – 00:08:07]** And so, super exciting, but this is how we run our agents internally.

**[00:08:07 – 00:08:09]** I'm not Sean, right?

**[00:08:09 – 00:08:13]** I'm not Sean. Sean is a TME on our team, a colleague of

**[00:08:13 – 00:08:16]** ours, and I want to talk about, we're going to talk about this

**[00:08:16 – 00:08:19]** use case throughout the session.

**[00:08:20 – 00:08:24]** And anybody use Muse or OpenClaw over here?

**[00:08:24 – 00:08:25]** Show of hands.

**[00:08:25 – 00:08:29]** And so more often than not, I think a lot of folks are utilizing

**[00:08:29 – 00:08:31]** those on a personal basis.

**[00:08:31 – 00:08:34]** You're connecting it maybe to your Gmail once you trust it.

**[00:08:34 – 00:08:37]** You're giving it access to your file system maybe.

**[00:08:37 – 00:08:42]** So what we're doing over here is, we've set up the equivalent,

**[00:08:42 – 00:08:45]** if you will, an open agent stack that allows you to build

**[00:08:45 – 00:08:49]** a personal assistant or a chief of staff, right, that

**[00:08:50 – 00:08:53]** is connected to all your business systems, that is connected

**[00:08:54 – 00:08:55]** to all your business systems.

**[00:08:55 – 00:08:57]** And so what are we going to do over here?

**[00:08:57 – 00:09:00]** First thing we're going to do, as I said, every agent

**[00:09:00 – 00:09:01]** needs to be deployed in a sandbox.

**[00:09:01 – 00:09:04]** So first thing we do is we deploy an OpenShell.

**[00:09:04 – 00:09:06]** This is all real, this is operating at NVIDIA.

**[00:09:06 – 00:09:11]** Next question is, hey, now that I've stood you up, what can you do?

**[00:09:11 – 00:09:14]** And it's gonna surface all the tools that we talked about earlier

**[00:09:14 – 00:09:18]** that I showed you in that reference architecture in the MCP gateway.

**[00:09:18 – 00:09:22]** It says, hey, I'm going to tell you what you have.

**[00:09:22 – 00:09:24]** And what Sean basically says, OK, I can see you

**[00:09:24 – 00:09:26]** can connect to my Outlook.

**[00:09:26 – 00:09:32]** Go to my Outlook and create a task, give me my task list of the day.

**[00:09:32 – 00:09:33]** Go through my emails and tell me what I've got to

**[00:09:33 – 00:09:35]** focus on for today.

**[00:09:35 – 00:09:39]** And it will generate a pretty good list.

**[00:09:39 – 00:09:42]** But if you guys are like us, we use email.

**[00:09:42 – 00:09:46]** We also spend a lot of time on Slack.

**[00:09:46 – 00:09:50]** There's also Teams chats that happen, there's Google Docs,

**[00:09:50 – 00:09:53]** and so if you really want to know what you're responsible

**[00:09:53 – 00:09:57]** for the day, you need to be able to cross-integrate with all those

**[00:09:57 – 00:09:59]** sources and cross-reference them.

**[00:09:59 – 00:10:05]** Now, what you'll notice happens over here is Sean says, okay, in

**[00:10:05 – 00:10:09]** addition to getting the information that's required for my tasks from

**[00:10:09 – 00:10:12]** Outlook, do so from Slack as well.

**[00:10:12 – 00:10:16]** The next thing he does is he says, hey, I think I'm gonna be asking

**[00:10:16 – 00:10:18]** for this task list every day.

**[00:10:19 – 00:10:23]** And what he explicitly does over here is he asks it to

**[00:10:23 – 00:10:27]** encapsulate that as a skill, a repeatable skill.

**[00:10:27 – 00:10:29]** Now, some harnesses do this automatically.

**[00:10:29 – 00:10:31]** They sit there and observe you.

**[00:10:31 – 00:10:33]** Hermes became very popular for that, right?

**[00:10:33 – 00:10:35]** It sits there, like, observing.

**[00:10:35 – 00:10:38]** Once it recognizes a pattern, it encapsulates it as a skill.

**[00:10:38 – 00:10:42]** And a skill essentially becomes a set of instructions.

**[00:10:42 – 00:10:45]** For you to be able to, rather than it reasoning and thinking

**[00:10:45 – 00:10:47]** through it, it's a set of instructions, I don't say it's

**[00:10:47 – 00:10:51]** prescriptive, but it ensures with higher levels of efficiency that

**[00:10:52 – 00:10:54]** it executes that same repetitive task over and over again.

**[00:10:55 – 00:10:59]** Now, look at the metrics on the right-hand side.

**[00:10:59 – 00:11:03]** This is 20 billion tokens per day.

**[00:11:03 – 00:11:05]** That shows you the level of scale that we have those deployed

**[00:11:06 – 00:11:07]** internally at NVIDIA.

**[00:11:07 – 00:11:10]** And I think I see our VP of IT sitting in the back

**[00:11:10 – 00:11:11]** over there, Rama.

**[00:11:11 – 00:11:13]** So, later on, guys, hi, Rama.

**[00:11:13 – 00:11:16]** She's the one that enables all this for us at NVIDIA.

**[00:11:16 – 00:11:22]** The other thing is, .. we started off with frontier models.

**[00:11:22 – 00:11:23]** Everybody does.

**[00:11:23 – 00:11:25]** They're incredible.

**[00:11:25 – 00:11:28]** There's general purpose intelligence is unbelievable.

**[00:11:28 – 00:11:31]** But once we got our use case working, we started thinking

**[00:11:31 – 00:11:35]** about, like, hey, how can we further optimize with open models?

**[00:11:35 – 00:11:39]** There are times when we have to route or escalate to the

**[00:11:39 – 00:11:42]** frontier proprietary models, but a lot of times for these kind

**[00:11:42 – 00:11:44]** of tasks, we can use open models.

**[00:11:44 – 00:11:48]** So we've been able to reduce costs by 84%.

**[00:11:48 – 00:11:54]** And it wasn't just a cost function, and using open models

**[00:11:54 – 00:11:57]** and using open models that are actually hosted on our

**[00:11:57 – 00:12:02]** AI factory, we've also been able to reduce latency in the process.

**[00:12:02 – 00:12:05]** So we're going to be going through this use case.

**[00:12:05 – 00:12:06]** I'm going to hand it over to Matt.

**[00:12:06 – 00:12:10]** He's going to talk about how else we can optimize this

**[00:12:10 – 00:12:11]** personal assistant.

**[00:12:12 – 00:12:16]** Matthew McDonald Jr.

**[00:12:16 – 00:12:19]** So we just walked through some pretty cool numbers.

**[00:12:19 – 00:12:21]** And there was a lot of work that was done in order to

**[00:12:21 – 00:12:22]** obtain those numbers.

**[00:12:23 – 00:12:24]** And for the rest of this talk, we're going to talk about

**[00:12:24 – 00:12:27]** some of the levers that we turned in order to obtain

**[00:12:27 – 00:12:28]** those optimization results.

**[00:12:28 – 00:12:32]** We're going to talk about the importance of observability.

**[00:12:32 – 00:12:34]** You cannot improve what you cannot measure.

**[00:12:34 – 00:12:38]** And then we're going to talk about two parallel ways that you might

**[00:12:38 – 00:12:39]** optimize your open agent stack.

**[00:12:39 – 00:12:41]** We're going to talk about the harness and some of the

**[00:12:41 – 00:12:43]** tooling that you can use in order to optimize that harness.

**[00:12:43 – 00:12:46]** Then we're going to talk about the model.

**[00:12:46 – 00:12:53]** How can you perhaps use a group of models in order to optimize

**[00:12:53 – 00:12:57]** your cost and other dimensions?

**[00:12:57 – 00:13:01]** So starting with observability.

**[00:13:01 – 00:13:06]** So observability, you wouldn't build a production

**[00:13:06 – 00:13:09]** software application without standard logging.

**[00:13:09 – 00:13:12]** Agents are the next generation of software applications.

**[00:13:12 – 00:13:15]** We cannot put away our standard software development principles

**[00:13:15 – 00:13:16]** here just because we're building with agents.

**[00:13:16 – 00:13:21]** And the thing about agents is they're extremely probabilistic.

**[00:13:21 – 00:13:24]** Very hard to determine what goes wrong when every time

**[00:13:24 – 00:13:26]** you run it, you get a slightly different answer.

**[00:13:26 – 00:13:28]** Even if the answer is still meaningfully accurate,

**[00:13:28 – 00:13:31]** the tokens are probably a little bit different.

**[00:13:31 – 00:13:35]** It's also really difficult to tell how it arrived at that answer.

**[00:13:35 – 00:13:36]** How it arrived at that answer?

**[00:13:36 – 00:13:39]** Did it violate any corporate policies on the way to

**[00:13:39 – 00:13:40]** getting that answer?

**[00:13:40 – 00:13:42]** Just because the answer was right didn't mean I didn't

**[00:13:42 – 00:13:45]** accidentally commit something to another database, and it's

**[00:13:45 – 00:13:46]** going to ruin somebody else's day.

**[00:13:46 – 00:13:50]** And then the last piece is if you add a new tool, you add

**[00:13:50 – 00:13:55]** a new skill, you add a new model that improved one task outcome,

**[00:13:55 – 00:13:59]** did I get regressions across other tasks that I'm responsible for?

**[00:13:59 – 00:14:01]** So doing this is pretty difficult.

**[00:14:01 – 00:14:05]** It's very difficult if you cannot see what is going on.

**[00:14:05 – 00:14:07]** So the first piece of technology we're going to talk about

**[00:14:07 – 00:14:11]** here today is a library that's part of the NVIDIA Agent Toolkit

**[00:14:11 – 00:14:13]** called NeMo Relay.

**[00:14:13 – 00:14:17]** And NeMo Relay is designed to work across all of the

**[00:14:17 – 00:14:19]** open agent harnesses, some of the frontier harnesses

**[00:14:19 – 00:14:24]** as well, in order to give you kind of a standard observability

**[00:14:24 – 00:14:26]** data format that comes out.

**[00:14:26 – 00:14:31]** And you can take that data, and you can use it for dashboarding;

**[00:14:31 – 00:14:34]** you can use it to sync to the popular observability

**[00:14:34 – 00:14:37]** platforms, your Langsmiths of the world, your Langfuses,

**[00:14:37 – 00:14:40]** your Datadogs, your ArisePhoenixes, so on and so forth, or some of

**[00:14:40 – 00:14:42]** your agent optimization platforms.

**[00:14:42 – 00:14:46]** You see a few here, Weights and Biases, and a number of others.

**[00:14:46 – 00:14:49]** Having a standard way that you're instrumenting the data across all

**[00:14:49 – 00:14:55]** of the agents that are deploying your enterprise, it's a given that

**[00:14:55 – 00:14:56]** you can really take advantage of.

**[00:14:56 – 00:14:58]** You can take advantage of it for debugging, you can take

**[00:14:59 – 00:15:02]** advantage of it for post-training, you can take advantage of

**[00:15:02 – 00:15:07]** it for identifying hotspots in your agent performance.

**[00:15:07 – 00:15:09]** So let's go ahead and take a look at what a notional

**[00:15:09 – 00:15:11]** agent workflow might look like.

**[00:15:11 – 00:15:13]** Really simple agent across the top asking a simple math

**[00:15:13 – 00:15:17]** problem, and we can see the steps that the agent would have taken.

**[00:15:17 – 00:15:20]** If I just gave you the answer and the answer was correct,

**[00:15:20 – 00:15:23]** you would have had no idea if it followed the actual

**[00:15:23 – 00:15:26]** math principles along the way.

**[00:15:26 – 00:15:28]** So one of the formats that we provide is something called

**[00:15:28 – 00:15:31]** ATOF, something that we came up with, but it's open, and

**[00:15:31 – 00:15:39]** it stands for the Agent Trajectory Observability Format.

**[00:15:39 – 00:15:41]** And it's a lossless format, so it's really, really good

**[00:15:41 – 00:15:45]** if you want to see everything that happened, the starts, the ends,

**[00:15:45 – 00:15:50]** the timings, everything in there, so that you can dive really deep.

**[00:15:50 – 00:15:54]** But you may also be familiar with something called Harbor.

**[00:15:54 – 00:15:58]** One of the things Harbor is known for is evaluations, and the format

**[00:15:58 – 00:16:02]** is called ATOF, which is the Agent Trajectory Interchange Format.

**[00:16:02 – 00:16:05]** This is much more human readable.

**[00:16:05 – 00:16:08]** It's a little lossless, but it's great for performing evaluations.

**[00:16:08 – 00:16:11]** And then the last piece is a lot of these observability

**[00:16:11 – 00:16:14]** platforms accept open telemetry.

**[00:16:14 – 00:16:18]** All right, so if you have an open hotel collector, chances are you

**[00:16:18 – 00:16:21]** can take the traces here and ingest them in any of the platforms.

**[00:16:21 – 00:16:24]** So the idea here is provide a standard set of tooling

**[00:16:24 – 00:16:27]** that makes it very easy for everybody to say, yes, I'm

**[00:16:27 – 00:16:32]** going to collect observability from my agent so that we can provide

**[00:16:32 – 00:16:34]** enterprise-grade observability.

**[00:16:34 – 00:16:36]** So what can you do with this?

**[00:16:36 – 00:16:40]** So I saw a few hands go up when we talked about the users of Hermes

**[00:16:40 – 00:16:42]** Agent or News Research's Agent.

**[00:16:42 – 00:16:45]** And they actually like to relay so much that they put

**[00:16:45 – 00:16:47]** it directly into their library.

**[00:16:47 – 00:16:49]** So now every one of you who raised your hand can now take

**[00:16:49 – 00:16:54]** advantage of NeMo Relay and the unified observability collection.

**[00:16:54 – 00:16:56]** They used it in order to optimize their harness.

**[00:16:56 – 00:17:01]** They saw benefits across efficient models, benefits across the

**[00:17:01 – 00:17:02]** large frontier models.

**[00:17:02 – 00:17:04]** And the mileage varied, obviously the different smarts

**[00:17:04 – 00:17:06]** and different types of models.

**[00:17:06 – 00:17:10]** But the things that they were able to identify was basically

**[00:17:10 – 00:17:14]** passing things like failure hints back to the model.

**[00:17:14 – 00:17:16]** Some of the more efficient models struggled to deal with

**[00:17:16 – 00:17:20]** loading large files from disk that would effectively truncate early,

**[00:17:20 – 00:17:23]** and therefore, if the answer was at the end of the file, it would

**[00:17:23 – 00:17:28]** never make it in the context, and then your answer would be wrong.

**[00:17:28 – 00:17:31]** Dealing with white space has always been a problem for us

**[00:17:31 – 00:17:36]** developers, and then deduplication of redundant skills and tools

**[00:17:36 – 00:17:41]** and stuff, being better stewards of our precious context window.

**[00:17:41 – 00:17:47]** So they use Relay, they have a rich set of evaluations,

**[00:17:47 – 00:17:50]** they used their agent systems in order to take this data and

**[00:17:50 – 00:17:54]** look for opportunities to improve their library, they merged a bunch

**[00:17:54 – 00:17:58]** of pull requests, and now Hermes is even better than it was before.

**[00:17:58 – 00:18:01]** The challenge is their strategy isn't as accessible to the rest

**[00:18:01 – 00:18:06]** of us, so we're going to now focus on Harness, see how we can all

**[00:18:06 – 00:18:08]** do the same kinds of things that they're doing, what tooling that

**[00:18:08 – 00:18:11]** you might use in order to do that.

**[00:18:12 – 00:18:16]** So at this point, we're going to be very focused on tuning the

**[00:18:16 – 00:18:20]** harness, the exoskeleton that Adil talked about earlier in this talk.

**[00:18:21 – 00:18:23]** And there are two libraries that we're going to talk about here.

**[00:18:23 – 00:18:26]** Number one, Nemo Compass.

**[00:18:26 – 00:18:31]** So we talked about the ATOF format that traces the evidence of

**[00:18:31 – 00:18:33]** what your agent did during runtime.

**[00:18:33 – 00:18:35]** And we would like to be able to use that evidence in order

**[00:18:35 – 00:18:39]** to identify when things are going well, and more importantly,

**[00:18:39 – 00:18:41]** when things are not going well.

**[00:18:41 – 00:18:44]** And we're probably also interested in running that through evaluation,

**[00:18:44 – 00:18:49]** having a score of, like, did this, actually telling us, did the

**[00:18:49 – 00:18:51]** answer turn out correctly or not?

**[00:18:51 – 00:18:53]** So what we do with Compass is we have a set of algorithms that run.

**[00:18:53 – 00:18:56]** They perform things like anomaly detection and clustering to

**[00:18:56 – 00:19:00]** bin related failures together, detect where the user might

**[00:19:00 – 00:19:03]** be saying, hey, this, you know, how many of us have screamed at our

**[00:19:03 – 00:19:04]** agent and we're getting frustrated?

**[00:19:04 – 00:19:06]** Like, just give me the right answer, right?

**[00:19:06 – 00:19:10]** Detect that and identify those as failure modes.

**[00:19:10 – 00:19:13]** There's something called an ethos, which is similar to

**[00:19:13 – 00:19:17]** something like a system prompt, but instead of telling the

**[00:19:17 – 00:19:20]** agent what to do, it describes the agent's ideal behavior.

**[00:19:20 – 00:19:22]** We can use this to look at the traces and see where the

**[00:19:22 – 00:19:24]** agent's actually diverging from what it's supposed to do.

**[00:19:24 – 00:19:27]** And we can use that to identify different failures and look at

**[00:19:27 – 00:19:29]** tool issues and so on and so forth.

**[00:19:29 – 00:19:32]** So what Nemo Compass is going to let you do, run all of

**[00:19:32 – 00:19:36]** these things against your traces and give you a synthesized

**[00:19:36 – 00:19:39]** list in a nice, clean format that says, hey, here are the areas

**[00:19:39 – 00:19:40]** that you might want to focus on.

**[00:19:40 – 00:19:43]** Now, the second half of this is eval.

**[00:19:43 – 00:19:46]** Typically, folks will start with a baseline eval.

**[00:19:46 – 00:19:52]** But I know for years, building software before agents, getting

**[00:19:52 – 00:19:55]** code coverage is a bit of a pain in the butt.

**[00:19:55 – 00:19:57]** If you've got a lot of it, you're very happy.

**[00:19:57 – 00:19:59]** But if you're not, you're always chasing.

**[00:19:59 – 00:20:01]** And the same is true for agents.

**[00:20:01 – 00:20:05]** You want your unit tests and your integration tests to

**[00:20:05 – 00:20:08]** be able to have as much coverage of the agent's behavior that

**[00:20:08 – 00:20:11]** represents what you're going to see in the real world.

**[00:20:11 – 00:20:12]** The chances are you don't have great coverage.

**[00:20:12 – 00:20:16]** So what NeMo eval author is going to help us do is it's

**[00:20:16 – 00:20:19]** going to look at our traces.

**[00:20:19 – 00:20:24]** It's going to look at our existing eval and recommend

**[00:20:24 – 00:20:27]** and build us additional evals that are going to give us

**[00:20:27 – 00:20:30]** better coverage for what the agent is going to be doing in the wild.

**[00:20:31 – 00:20:32]** It's going to have access to the code base.

**[00:20:32 – 00:20:34]** It's going to have access to the ethos.

**[00:20:34 – 00:20:36]** And what's going to come out is you're going to have better

**[00:20:36 – 00:20:41]** coverage for your agent and identify knobs to turn to

**[00:20:41 – 00:20:42]** make your agent even better.

**[00:20:42 – 00:20:46]** So the takeaway here is take a look at NeMo Compass and eval author.

**[00:20:46 – 00:20:48]** We're going to show you a few ways that you can get

**[00:20:48 – 00:20:50]** activated on it here pretty soon.

**[00:20:50 – 00:20:52]** But first, we're talking about a really cool use case.

**[00:20:53 – 00:20:55]** So for those that tuned into the keynote yesterday, you

**[00:20:55 – 00:20:57]** probably heard Jensen mention Dassault Systems.

**[00:20:58 – 00:21:00]** They're an excellent industrial software vendor.

**[00:21:00 – 00:21:02]** They specialize in CAD.

**[00:21:02 – 00:21:08]** And I'm gonna focus on one specific workflow here today, designing

**[00:21:08 – 00:21:12]** a door that's gonna be part of a vehicle that's being engineered.

**[00:21:12 – 00:21:14]** So at some point, somebody has to design the door.

**[00:21:14 – 00:21:18]** So the first part in this process is they have these virtual

**[00:21:18 – 00:21:22]** twin physics, kind of surrogate models that estimate physics,

**[00:21:22 – 00:21:27]** and they look at previous cars and their doors, and they kind of

**[00:21:27 – 00:21:31]** recommend a couple of door designs without going through the full

**[00:21:31 – 00:21:34]** finite element analysis process.

**[00:21:34 – 00:21:35]** And then the engineer gets a set of doors.

**[00:21:35 – 00:21:38]** They say, OK, I think I'm going to go with this door,

**[00:21:38 – 00:21:42]** and then we're going to send it to the really, really heavyweight

**[00:21:42 – 00:21:45]** Katya-based design, where it's going to have this iterative

**[00:21:45 – 00:21:50]** process where it's going to design the door, the materials,

**[00:21:50 – 00:21:52]** and then it's going to run that through a high-fidelity simulation.

**[00:21:52 – 00:21:55]** It's going to run it through crash simulations and load tests.

**[00:21:55 – 00:21:58]** It's going to say, hey, this door, this door is good, or

**[00:21:58 – 00:22:01]** this door is going to be lacking in this area under these specific

**[00:22:01 – 00:22:04]** conditions, and it goes back and forth until it gets the job right.

**[00:22:05 – 00:22:08]** So those last two pieces are orchestrated by this

**[00:22:08 – 00:22:11]** Leo agent, this Leo companion that Dassault has.

**[00:22:11 – 00:22:14]** But the challenge is that this step three is very difficult.

**[00:22:14 – 00:22:15]** It's very difficult to get right.

**[00:22:15 – 00:22:18]** So this is a great opportunity for Dassault Systems, an early

**[00:22:18 – 00:22:22]** adopter of the NeMo Compass tooling, in order to see if

**[00:22:22 – 00:22:24]** they can boost the performance of their critical agents.

**[00:22:24 – 00:22:28]** They can put out these doors much more quickly, much more accurately.

**[00:22:28 – 00:22:30]** So they followed a very similar process, right?

**[00:22:30 – 00:22:34]** They collected golden trajectories that represented what a good

**[00:22:34 – 00:22:36]** engineer would do.

**[00:22:36 – 00:22:41]** Then they ran their agent and they collected the traces,

**[00:22:41 – 00:22:44]** and they compared these things against the golden data sets,

**[00:22:44 – 00:22:47]** and they said, hey, things are looking good or things

**[00:22:47 – 00:22:49]** are not looking good, and they're not looking good for these reasons.

**[00:22:49 – 00:22:52]** Let's make some tweaks to the harness, and let's

**[00:22:52 – 00:22:53]** run this loop again.

**[00:22:53 – 00:22:56]** They ran this overnight, I think it ran 12 hours or so, and they

**[00:22:56 – 00:23:00]** came out in the morning and they saw some pretty impressive results.

**[00:23:00 – 00:23:02]** So let's go ahead and take a look at this workflow here.

**[00:23:03 – 00:23:06]** So we're going to see, this is Leo agent in action designing a door.

**[00:23:06 – 00:23:09]** So the first thing that this engineer does is he

**[00:23:09 – 00:23:13]** picks design five.

**[00:23:13 – 00:23:17]** You can see it starts to build out the high fidelity door here.

**[00:23:17 – 00:23:19]** It's picking materials.

**[00:23:19 – 00:23:21]** It's layering on the details.

**[00:23:21 – 00:23:24]** There's an applied bill of materials involved here.

**[00:23:24 – 00:23:30]** And at some point, we're going to run a pretty high-fidelity

**[00:23:30 – 00:23:34]** simulation as to how this automatically-designed door

**[00:23:34 – 00:23:37]** is going to perform under load.

**[00:23:38 – 00:23:39]** Should be any second now, so here we go.

**[00:23:39 – 00:23:45]** All right, so here we're doing some static load testing on the door.

**[00:23:46 – 00:23:49]** We're then going to simulate A crash.

**[00:23:49 – 00:23:51]** So really, many of us have been in a car, you'd like

**[00:23:52 – 00:23:53]** the door to work pretty well.

**[00:23:53 – 00:23:55]** I've got two young kids, I'd like the doors to be really

**[00:23:55 – 00:23:57]** good in the backseat.

**[00:23:57 – 00:24:00]** So as it turns out, once these simulations are done, it ends

**[00:24:01 – 00:24:01]** up finding an issue.

**[00:24:01 – 00:24:04]** And I believe the issue is, it's kind of small for me

**[00:24:05 – 00:24:07]** to read, but I think it was with some of the vibration testing.

**[00:24:07 – 00:24:10]** And so we need to go back to the Katia-based agent and

**[00:24:10 – 00:24:15]** say, we need to do a better job along this dimension.

**[00:24:15 – 00:24:18]** So it goes ahead and goes through the process, improves

**[00:24:18 – 00:24:21]** the door according to the failure mode, and then it's

**[00:24:21 – 00:24:25]** going to come out eventually the back side and then have the door

**[00:24:25 – 00:24:29]** that we're going to leave and get the approvals and ship the door.

**[00:24:29 – 00:24:32]** So by following the harness engineering process, they saw

**[00:24:32 – 00:24:37]** a 28% improvement in door design.

**[00:24:37 – 00:24:40]** But what I think is really great is they saw a 25% improvement

**[00:24:41 – 00:24:44]** in other kind of design scenarios following the same process.

**[00:24:44 – 00:24:47]** So it wasn't over-fit to this one specific design process,

**[00:24:47 – 00:24:51]** they followed the optimization strategy, and they saw lift

**[00:24:51 – 00:24:52]** across the other use cases.

**[00:24:52 – 00:24:55]** They also saw 35% fewer tool calls.

**[00:24:55 – 00:24:58]** So, better agent, more efficient.

**[00:24:58 – 00:25:02]** And the tooling that they used was Numotron Super 3.5

**[00:25:02 – 00:25:05]** VL, which is great.

**[00:25:06 – 00:25:08]** And I just think it's a really cool use case.

**[00:25:08 – 00:25:11]** So if we zoom out here for a second, we've talked about

**[00:25:11 – 00:25:16]** news research, we've talked about our internal personal assistant,

**[00:25:16 – 00:25:19]** but we also spent a lot of time with the other Harness providers.

**[00:25:19 – 00:25:21]** So LangChain is one of them as well.

**[00:25:21 – 00:25:23]** They have a pretty robust eval set as well.

**[00:25:23 – 00:25:26]** And they go through the sweep all the time in order to lift

**[00:25:27 – 00:25:31]** the performance of their deep agents kind of Harness.

**[00:25:31 – 00:25:32]** And we just saw this as well.

**[00:25:32 – 00:25:35]** So there's some common levers that people tend to pull.

**[00:25:35 – 00:25:37]** No matter what your evals say, you kind of start to

**[00:25:37 – 00:25:39]** see some commonalities.

**[00:25:39 – 00:25:42]** You start to see opportunities to improve your prompts, your

**[00:25:42 – 00:25:44]** skills, your soul.

**[00:25:44 – 00:25:46]** You see tool optimizations.

**[00:25:46 – 00:25:50]** Sometimes exposing a smaller set of tools to your agent gives

**[00:25:50 – 00:25:52]** it better performance, especially when you're using some of these

**[00:25:52 – 00:25:54]** smaller, more efficient models.

**[00:25:54 – 00:25:57]** Sometimes your arguments need to be a little bit better,

**[00:25:57 – 00:26:00]** you need to fix your schemas, or sometimes you need to pass hints

**[00:26:00 – 00:26:03]** back so that the agent can recover.

**[00:26:03 – 00:26:06]** Sometimes you get really, really large dumps back from

**[00:26:06 – 00:26:09]** some of your tools, and sometimes it's a bespoke format that's

**[00:26:09 – 00:26:10]** required to be parsed.

**[00:26:10 – 00:26:14]** I know internally one of our MCP tools returned this big

**[00:26:14 – 00:26:17]** pile of JSON back that we were struggling to parse agent-side.

**[00:26:17 – 00:26:19]** So we just, you know, one of the tweaks we did with

**[00:26:20 – 00:26:23]** Hermes was we used the exposed plugin and lifecycle hooks in order

**[00:26:23 – 00:26:27]** to just tweak the parsing a little bit, and then it worked wonders.

**[00:26:27 – 00:26:29]** And then sometimes it makes sense to handle

**[00:26:29 – 00:26:31]** the control flow directly.

**[00:26:31 – 00:26:36]** So every time I get a request, maybe I'm going to do a quick,

**[00:26:36 – 00:26:39]** rag-like search across all of my tools in order to load the small

**[00:26:39 – 00:26:42]** subset that's most appropriate to the current active context.

**[00:26:42 – 00:26:43]** These are the types of things that you can do and get a

**[00:26:43 – 00:26:49]** lot of mileage out without doing one single post-training rep.

**[00:26:51 – 00:26:54]** So, Adil mentioned blueprints.

**[00:26:54 – 00:26:58]** So, one of the blueprints that we've put out recently is to

**[00:26:58 – 00:27:00]** show you how to do this yourself.

**[00:27:00 – 00:27:02]** What it does is it uses a common name.

**[00:27:02 – 00:27:04]** The one common thing you'll see across all of our blueprints

**[00:27:04 – 00:27:07]** is OpenShell, but we have an agent running an OpenShell with

**[00:27:07 – 00:27:10]** Hermes that simulates our internal personal assistant use case.

**[00:27:10 – 00:27:13]** We try to take out some of the learnings and put it on GitHub

**[00:27:13 – 00:27:14]** for others to take advantage of.

**[00:27:14 – 00:27:17]** So, it's a use case where somebody says, hey, set up

**[00:27:17 – 00:27:21]** this email for me that I'm going to send to this person.

**[00:27:21 – 00:27:23]** But the agent sends the email.

**[00:27:23 – 00:27:26]** We want to fix this agent so that it is actually

**[00:27:26 – 00:27:27]** listening to the request.

**[00:27:27 – 00:27:31]** So it shows you how to, it comes with a bunch of evals,

**[00:27:31 – 00:27:34]** it comes with, it comes with some mock MCP servers with data,

**[00:27:34 – 00:27:37]** and you can kind of run through the full process, build out your eval,

**[00:27:37 – 00:27:40]** look for the opportunity to improve the harness, and then test the

**[00:27:40 – 00:27:44]** improved harness and see it pass all of those integration tests.

**[00:27:44 – 00:27:46]** And again, it's on GitHub, and it kind of shows you how

**[00:27:46 – 00:27:48]** to piece all these together.

**[00:27:48 – 00:27:52]** So just as a reminder, it's Compass, eval author, Hermes,

**[00:27:52 – 00:27:55]** running an OpenShell that emulates our internal deployment,

**[00:27:55 – 00:27:58]** so you can kind of get hands-on.

**[00:28:00 – 00:28:02]** So the other half of the agent, we talked about Harness for

**[00:28:02 – 00:28:05]** a while, but there's this big thing called a model, right?

**[00:28:05 – 00:28:09]** And we're going to talk a little bit about how you can

**[00:28:09 – 00:28:13]** pick the right model at the right time during runtime, okay?

**[00:28:13 – 00:28:15]** You might hear the term router quite a bit.

**[00:28:15 – 00:28:20]** A good place for a router to exist is in your LLM gateway.

**[00:28:20 – 00:28:25]** So like a light LLM or a ramp or a Kong, or maybe you've

**[00:28:25 – 00:28:26]** built your own gateway.

**[00:28:26 – 00:28:28]** It's a way, kind of, to standardize your inference

**[00:28:29 – 00:28:32]** endpoint, but you can put some smarts behind that router.

**[00:28:32 – 00:28:35]** We have a library called NeMo Switchyard that our contribution

**[00:28:35 – 00:28:37]** to the routing ecosystem is a set of algorithms.

**[00:28:37 – 00:28:40]** And these algorithms would be picked up typically at your LLM

**[00:28:40 – 00:28:43]** gateway to allow you to get more efficiency out of the distribution

**[00:28:43 – 00:28:46]** of models while maintaining the accuracy or whatever dimension

**[00:28:46 – 00:28:47]** that you're trying to optimize for.

**[00:28:47 – 00:28:50]** And by a show of hands, anybody utilizing routers

**[00:28:50 – 00:28:52]** today for their systems?

**[00:28:53 – 00:28:54]** Okay, awesome.

**[00:28:54 – 00:28:59]** That's great. Anybody using Switchyard, just out of curiosity?

**[00:28:59 – 00:29:01]** They will now. They'll try it now.

**[00:29:01 – 00:29:03]** So a couple of the big algorithms.

**[00:29:03 – 00:29:07]** So we have an LLM classification routing algorithm.

**[00:29:07 – 00:29:11]** And you can almost think about this as a version of intent routing.

**[00:29:11 – 00:29:14]** Basically what we do, actually we'll get into the algorithm

**[00:29:14 – 00:29:15]** on the next couple of slides.

**[00:29:15 – 00:29:17]** We've got a stage-based route, which actually is the most popular.

**[00:29:17 – 00:29:21]** A lot of the LLM gateways are picking this one up first.

**[00:29:21 – 00:29:23]** We've got a demo on this in a slide.

**[00:29:23 – 00:29:27]** And then the Escalation Router, which kind of lets you gradually

**[00:29:27 – 00:29:30]** go up to a bigger model once the task difficulty is seen.

**[00:29:30 – 00:29:32]** But let's go ahead and take a look at a quick demo.

**[00:29:32 – 00:29:35]** So Ramp's done some work to adopt Switchyard.

**[00:29:35 – 00:29:38]** Here they're going to be using two different Frontier models

**[00:29:38 – 00:29:47]** and going to be comparing it to using GPT-55 Sol versus using Sol

**[00:29:47 – 00:29:52]** and Luna in combination to drive costs down while driving speed up.

**[00:29:52 – 00:29:54]** Right, so you can kind of see here in this plot, well,

**[00:29:54 – 00:29:56]** it kind of went a little bit quick, but what was on the

**[00:29:56 – 00:30:01]** plot was if you used Luna alone, you're, sure, you're cheaper, but

**[00:30:01 – 00:30:03]** you're taking a hit on accuracy.

**[00:30:03 – 00:30:07]** But if we can find, at the right time, to combine the

**[00:30:07 – 00:30:10]** two models, we can have the same accuracy, but we're going to reduce

**[00:30:10 – 00:30:15]** cost significantly, about 60% per task, and have lower runtime.

**[00:30:15 – 00:30:18]** And the way they've done it, and it's part of their, part of

**[00:30:18 – 00:30:21]** their secret sauce, is maintaining high cash hit rates, which is

**[00:30:21 – 00:30:25]** going to be great for cost and it's going to be great for performance.

**[00:30:25 – 00:30:28]** So now let's take a quick look at some of these algorithms

**[00:30:28 – 00:30:30]** in a little bit more depth.

**[00:30:30 – 00:30:33]** So the first one is that classification algorithm.

**[00:30:33 – 00:30:36]** And what we do here is we

**[00:30:36 – 00:30:41]** basically give each model a model capability card that

**[00:30:41 – 00:30:43]** kind of lives in prompt land.

**[00:30:43 – 00:30:48]** And we do that, and we take a LLM as a judge approach.

**[00:30:48 – 00:30:51]** The judge says, hey, does, you know, given this problem

**[00:30:51 – 00:30:54]** that we're facing, does this model have the capability to do this?

**[00:30:54 – 00:30:58]** And then there's some thresholding, and then the model is selected

**[00:30:58 – 00:31:00]** and used for that particular turn.

**[00:31:00 – 00:31:03]** I would use this router for chat type applications.

**[00:31:03 – 00:31:07]** I probably wouldn't use it alone anyway for something

**[00:31:07 – 00:31:12]** like a coding agent, but where it can be used, it's quite good.

**[00:31:12 – 00:31:16]** The next one was the one that we demoed in the ramp example.

**[00:31:16 – 00:31:17]** This is the stage router.

**[00:31:17 – 00:31:20]** And where the previous one with limitation was you can't

**[00:31:20 – 00:31:24]** really get us, you don't know how difficult the problem is

**[00:31:24 – 00:31:25]** when you submit your prompt, right?

**[00:31:25 – 00:31:28]** You can kind of get a sense of the problem difficulty

**[00:31:28 – 00:31:30]** as you start to make tool calls.

**[00:31:30 – 00:31:33]** As the tool cars start to fail, you can kind of get a sense of,

**[00:31:33 – 00:31:36]** you know, there's some heuristics here that if I was observing

**[00:31:36 – 00:31:39]** one of my employees, I can kind of tell when somebody needs a hand.

**[00:31:39 – 00:31:43]** So what we do with the Stage Router is we observe the runtime,

**[00:31:43 – 00:31:47]** we collect these metrics, and we use these heuristics to say, hey,

**[00:31:47 – 00:31:51]** I think now is the time that we need to escalate to a higher model.

**[00:31:51 – 00:31:53]** But, you know, depending on your configuration, you have

**[00:31:53 – 00:31:57]** the ability to kind of escalate back down once we've observed

**[00:31:57 – 00:31:59]** the complexity of the problem has started to stabilize.

**[00:31:59 – 00:32:02]** So that's Stage Routing Algorithm.

**[00:32:02 – 00:32:04]** And the last one we'll get into is Escalation.

**[00:32:04 – 00:32:07]** It's kind of similar.

**[00:32:07 – 00:32:10]** The idea here is you start with your default model.

**[00:32:10 – 00:32:16]** We maintain a window of messages that have been submitted.

**[00:32:16 – 00:32:21]** We track effectively contiguous errors of the same type with

**[00:32:21 – 00:32:24]** a counter, and when some things get above a threshold, we

**[00:32:24 – 00:32:30]** say, hey, it's time to, actually, at every turn, we run a LLM

**[00:32:30 – 00:32:32]** classifier that says, hey, is it time to escalate or

**[00:32:32 – 00:32:35]** not, using that data as evidence?

**[00:32:35 – 00:32:38]** And if it says yes, then we can pin it to the big model, and

**[00:32:38 – 00:32:40]** depending on your configuration, you can recover, but that's

**[00:32:40 – 00:32:41]** the idea, escalation router.

**[00:32:41 – 00:32:43]** The main, one of the main differences is we make a judge

**[00:32:43 – 00:32:47]** call every time here, and on the escalation side we

**[00:32:47 – 00:32:51]** leverage heuristics and thresholds in order to make the decision.

**[00:32:52 – 00:32:53]** So I think we're getting pretty close to time here, and I think

**[00:32:53 – 00:32:55]** we'll have time for questions.

**[00:32:55 – 00:32:58]** I just wanted to kind of summarize here just really quickly.

**[00:32:58 – 00:33:02]** We talked about how we're deploying agents internally,

**[00:33:02 – 00:33:05]** the stack that we're using, OpenShell is core to that.

**[00:33:05 – 00:33:08]** We talked about how we made our agents more efficient.

**[00:33:08 – 00:33:10]** We talked about observability, the importance of it.

**[00:33:10 – 00:33:13]** The tooling we're using for observability being Nemo Relay.

**[00:33:13 – 00:33:15]** We talked about the different levers that you can pull

**[00:33:15 – 00:33:16]** for optimization.

**[00:33:16 – 00:33:18]** We talked about the harness side of things.

**[00:33:18 – 00:33:21]** We talked about the tool you can use to optimize the harness side

**[00:33:21 – 00:33:24]** of things, Compass, EvalAuthor.

**[00:33:24 – 00:33:26]** And then we talked about model routing as another dimension in

**[00:33:26 – 00:33:30]** order to bring the right model at the right time into your process.

**[00:33:30 – 00:33:32]** Now you're typically not done.

**[00:33:32 – 00:33:36]** If you deploy an agent, you tend to find more people that

**[00:33:36 – 00:33:37]** want to use that.

**[00:33:37 – 00:33:40]** Maybe more use cases become onboarded, and you kind of

**[00:33:40 – 00:33:41]** go through the process again.

**[00:33:41 – 00:33:44]** You want to make sure you have your observability set up; you want to

**[00:33:44 – 00:33:46]** make sure you have coverage for the new use cases that you're adopting,

**[00:33:46 – 00:33:47]** kind of go through this process.

**[00:33:47 – 00:33:49]** But once you build this muscle, there's a lot of things that

**[00:33:49 – 00:33:52]** you can do moving forward.

**[00:33:52 – 00:33:54]** So the last thing, I think our last major slide here,

**[00:33:54 – 00:33:59]** before we'll go into questions, is NemoClaw Blueprints.

**[00:33:59 – 00:34:01]** There'll be a number of these that'll go out there.

**[00:34:01 – 00:34:04]** Some of them will be full-blown applications, some of them

**[00:34:04 – 00:34:07]** will be tutorials like the ones that we just saw.

**[00:34:07 – 00:34:09]** But the core that you'll see in every single one of these is the

**[00:34:09 – 00:34:13]** Secure Runtime and different ways to put these pieces together from

**[00:34:13 – 00:34:20]** harnesses, models, sandbox, tools, and your knowledge foundation.

**[00:34:20 – 00:34:22]** So I think that's the last slide here.

**[00:34:22 – 00:34:28]** I think we're ready to probably open it up for questions.

**[00:34:41 – 00:34:44]** Yes, hello, thank you for the lovely talk.

**[00:34:44 – 00:34:47]** My question is regarding the model routing.

**[00:34:47 – 00:34:51]** Have you looked into what has recently become popular,

**[00:34:51 – 00:34:53]** this decision type of models?

**[00:34:53 – 00:34:57]** Because I think those, you could put a lot of information about what

**[00:34:57 – 00:35:02]** you want to route in the context and then have a fast evaluation

**[00:35:02 – 00:35:06]** compared to an LLM evaluation.

**[00:35:06 – 00:35:08]** I think, I think you've been paying attention to the lesson.

**[00:35:08 – 00:35:11]** I think that's great. I don't know if we've released anything

**[00:35:11 – 00:35:13]** on Switchyard yet, but I know our team is actively looking

**[00:35:13 – 00:35:16]** on building those things in for routing and then lots

**[00:35:16 – 00:35:17]** of other use cases as well.

**[00:35:17 – 00:35:20]** Super excited about the direction things are going.

**[00:35:20 – 00:35:21]** I'm super excited about that in particular.

**[00:35:21 – 00:35:25]** And at build time, using decision models to figure out what tools to

**[00:35:25 – 00:35:26]** load for your particular use case.

**[00:35:27 – 00:35:31]** I mean, decision models are unlocking a lot.

**[00:35:31 – 00:35:32]** We have another GTC coming up in a month.

**[00:35:32 – 00:35:34]** You should watch the sessions.

**[00:35:34 – 00:35:36]** It's in Washington.

**[00:35:36 – 00:35:38]** We'll be talking a lot more about some of those things.

**[00:35:38 – 00:35:41]** But yeah, decision models are fantastic.

**[00:35:41 – 00:35:42]** Great question.

**[00:35:42 – 00:35:43]** Thanks for the talk.

**[00:35:43 – 00:35:45]** It was really interesting. Quick question.

**[00:35:45 – 00:35:49]** What are your thoughts on deploying agents on your local

**[00:35:49 – 00:35:51]** machine versus the cloud?

**[00:35:51 – 00:35:56]** Any advantages or disadvantages with two approaches?

**[00:35:56 – 00:35:58]** I can take a quick shot and I don't know if you want to...

**[00:35:59 – 00:36:03]** yeah, so I think when you look at the anatomy of an agent, what we're

**[00:36:03 – 00:36:08]** doing internally, we are deploying the harness, the exoskeleton,

**[00:36:08 – 00:36:11]** on a Vera-based system.

**[00:36:11 – 00:36:15]** Internally on our AI factory, we got, I think, Grace Blackwell

**[00:36:15 – 00:36:19]** and other systems, so we're running some of our models there.

**[00:36:19 – 00:36:21]** Local AI is a thing, right?

**[00:36:21 – 00:36:24]** So some folks have Spark, some folks have stations

**[00:36:24 – 00:36:26]** underneath their desks.

**[00:36:26 – 00:36:28]** They use a, like, you know, when you're routing, you want

**[00:36:28 – 00:36:32]** to default to local when you can, but if you actually require

**[00:36:32 – 00:36:38]** frontier intelligence, you can escalate, you can escalate there.

**[00:36:38 – 00:36:43]** The times when we are explicitly restricted from going to cloud

**[00:36:43 – 00:36:47]** for data providence reasons, that's when we have to deploy

**[00:36:47 – 00:36:48]** fully internally, right?

**[00:36:49 – 00:36:54]** And, you know, that's a data privacy or a compliance reason

**[00:36:54 – 00:36:57]** as opposed to anything else.

**[00:36:57 – 00:36:58]** Great question.

**[00:36:58 – 00:37:02]** The only thing I would add is if you are, whether it's your personal

**[00:37:02 – 00:37:04]** computer or your work computer, if it's your work computer, you

**[00:37:04 – 00:37:07]** probably want to have more control over what's installed on there.

**[00:37:07 – 00:37:10]** Open shell can be installed, so you can have the same level

**[00:37:10 – 00:37:14]** of privacy or protection over what your agent does on your

**[00:37:14 – 00:37:16]** computers if you would just install open shell there as well

**[00:37:17 – 00:37:19]** as to add one layer of security.

**[00:37:19 – 00:37:22]** I actually ran that entire blueprint that Matt took you

**[00:37:22 – 00:37:27]** through with the Nemotron 3.5 VL on a DGX Spark, fully.

**[00:37:27 – 00:37:30]** Now, when I'm doing the authoring of the evals, I think I used

**[00:37:30 – 00:37:33]** Codex to generate some of those, and Compass to generate some

**[00:37:33 – 00:37:36]** of those evals and improvements, but the system as a whole

**[00:37:36 – 00:37:39]** was running directly on a Spark.

**[00:37:41 – 00:37:47]** Yeah, so one of the questions I have is back to the

**[00:37:47 – 00:37:49]** decision models, right.

**[00:37:49 – 00:37:52]** What we see most of the time now in open weight decision

**[00:37:52 – 00:37:54]** models is that you have a standard model, right, and

**[00:37:54 – 00:38:00]** you have one pass and then you have an attention head on it.

**[00:38:00 – 00:38:06]** So I was wondering, is NVIDIA also working in hosting the default

**[00:38:06 – 00:38:10]** model, and then only having the additional heads hosted separately,

**[00:38:11 – 00:38:15]** so you have two interfaces only hosting it one time in memory?

**[00:38:15 – 00:38:18]** Kind of like that.

**[00:38:19 – 00:38:24]** It's a great idea. I don't know if we've done it internally yet.

**[00:38:29 – 00:38:32]** We should talk more about that afterwards.

**[00:38:32 – 00:38:34]** We have one question. So I have a question related to

**[00:38:34 – 00:38:35]** a couple of things.

**[00:38:35 – 00:38:38]** One is, when your context is really in the agent, the

**[00:38:38 – 00:38:42]** context really builds up a lot, and it starts kind of hallucinating.

**[00:38:42 – 00:38:47]** So is there some tooling to assist there from NVIDIA?

**[00:38:48 – 00:38:53]** And then is there some tooling around, okay, I want to have

**[00:38:53 – 00:38:59]** ethos where no agent runs more than two minutes, thank you.

**[00:39:02 – 00:39:05]** I don't know that we have a specific dedicated approach

**[00:39:05 – 00:39:07]** to context compaction.

**[00:39:07 – 00:39:13]** We put out a NVIDIA object-oriented agents kind of experimental

**[00:39:13 – 00:39:16]** harness implementation that plays some some clever tricks

**[00:39:16 – 00:39:17]** for context management.

**[00:39:17 – 00:39:21]** One of the most interesting things was pass-by reference

**[00:39:21 – 00:39:25]** as opposed to throwing the whole thing into context.

**[00:39:25 – 00:39:28]** But a lot of times, frankly, we lean on the harnesses and

**[00:39:28 – 00:39:33]** their compaction strategies and the extension interfaces that they have

**[00:39:33 – 00:39:35]** if we want to add anything custom.

**[00:39:35 – 00:39:39]** But from a compaction perspective, I think that's... or sometimes

**[00:39:39 – 00:39:42]** people just build their own harness, and they're like, I just

**[00:39:42 – 00:39:47]** have this really good math idea, and I'm just going to do it myself.

**[00:39:47 – 00:39:50]** And I'm forgetting what the second question was that you had.

**[00:39:50 – 00:39:53]** I apologize.

**[00:39:53 – 00:39:56]** Kill switch after kill switch.

**[00:39:57 – 00:40:01]** Well, I think that the total solution that Jensen put on stage,

**[00:40:01 – 00:40:06]** where we had your agent running on Vera in your secure runtime,

**[00:40:06 – 00:40:08]** and then you're observing what's happening out of band on your

**[00:40:09 – 00:40:12]** BlueField with Sentry, and then the ecosystem tooling that we'll

**[00:40:12 – 00:40:15]** see, you know, come over the years will provide the actual answer,

**[00:40:15 – 00:40:17]** I think, to the kill switch.

**[00:40:17 – 00:40:20]** I think today, how I would probably do it is I would

**[00:40:20 – 00:40:23]** use the open shell kind of supervisor interface in order to,

**[00:40:23 – 00:40:26]** you know, give the supervisor extra tooling in order to kill things

**[00:40:26 – 00:40:29]** that are happening in the sandbox.

**[00:40:31 – 00:40:33]** Are we done? One more question.

**[00:40:33 – 00:40:37]** In the back over there. Yeah, I got a question here.

**[00:40:37 – 00:40:40]** The light is so bright, I can't see you guys.

**[00:40:40 – 00:40:41]** I'm tanning one and sweating one.

**[00:40:41 – 00:40:45]** Thanks Matt and Anna.

**[00:40:45 – 00:40:47]** Very great talk.

**[00:40:47 – 00:40:51]** I think my question is around harness and it's a bit of

**[00:40:51 – 00:40:56]** a thinking around, instead of saying like, say we observe

**[00:40:56 – 00:41:00]** and monitor and then we fix, is it more like a proactive way of doing

**[00:41:00 – 00:41:05]** that, saying for example for every agent step and one for example

**[00:41:05 – 00:41:10]** especially making decisions, we kind of give a confidence score to

**[00:41:10 – 00:41:14]** each of that, and say we set up a threshold to this confidence score

**[00:41:14 – 00:41:20]** and saying lower those scores, threshold, we wouldn't take the

**[00:41:20 – 00:41:22]** decision, we would rerun, retry.

**[00:41:22 – 00:41:26]** So what do you think about along this line and any thoughts

**[00:41:26 – 00:41:29]** you can share or any developments already along here?

**[00:41:29 – 00:41:33]** I mean, the classification-based routers effectively do

**[00:41:33 – 00:41:35]** a version of that.

**[00:41:35 – 00:41:40]** I could imagine a new either routing level implementation

**[00:41:40 – 00:41:42]** that does that, or I could imagine hooking directly into

**[00:41:42 – 00:41:45]** the harness at the extension points and writing a quick detector

**[00:41:45 – 00:41:49]** that's gonna give me a probability, and you can, depending on what

**[00:41:49 – 00:41:51]** you do with that number, I think, will be up to your application.

**[00:41:51 – 00:41:56]** I don't think I've seen somebody within my kind of cone of

**[00:41:56 – 00:41:58]** influence to that, not to mean it hasn't been done,

**[00:41:58 – 00:41:59]** but I think it's a clever idea.

**[00:41:59 – 00:42:02]** Okay, thanks.

**[00:42:02 – 00:42:04]** I think we're over time, you guys.

**[00:42:04 – 00:42:06]** You would do that in classical ML, right?

**[00:42:06 – 00:42:07]** If you have a lot of uncertainty, right?

**[00:42:07 – 00:42:09]** All right.

**[00:42:09 – 00:42:09]** Thanks a lot.

