# Use Reinforcement Learning to Turn AI Agents Into Continuous Learners

## Abstract

Static prompt engineering breaks under real-world complexity. This session shows how Reinforcement Learning (RL) enables agents to make better decisions, use tools reliably, and adapt in production. This talk distills lessons from Atlassian on how it built Rovo, its enterprise AI solution for search, AI chat, and agents. The team explains how it moved from static prompt engineering to policy training via RL thereby improving intent interpretation, retrieval quality and how they use NVIDIA NeMo RL to evolve query rewriting from brittle rules to adaptive learning. The session then expands to platform RL, where Google Cloud Vertex Training Cluster showcases how its integration of NVIDIA NeMo libraries has helped customers build best-in-class models. The team will deep dive on Vertex’s managed GPU infrastructure and how NeMo RL and other techniques can be used to build high performance agents."

## AI Summary

- The presentation discussed how Atlassian built AI workflows using Rovo, an AI teammate that enhances search, chat, and agent platforms across products like Jira and Confluence.
- Rovo utilized reinforcement learning to optimize query rewriting in the search and retrieval process, improving recall and other search relevance metrics.
- The Jira Issue Solver, a long-horizon agent, was designed to resolve complex Jira issues end-to-end without human intervention, using a suite of tools and maintaining safety controls.
- The team used supervised fine-tuning and reinforcement learning with the GRPO algorithm to achieve a 94% success rate in in-domain tasks and a 12% improvement in out-of-domain tasks.
- For the agent customization pipeline, the data science team evaluated and selected a large 350 billion parameter teacher model to distill knowledge into smaller student models.
- The Vertex AI Training Cluster provided a turnkey solution for training, handling orchestration, hardware resiliency, and pre-configured data science recipes, significantly improving model performance and throughput.

## Transcript

**[00:00:05 – 00:00:12]** to introduce how we are building Agenda AI workflows at Lazien.

**[00:00:21 – 00:00:23]** For those who are not familiar with Atlassian's product,

**[00:00:23 – 00:00:28]** and especially Rovo, our AI product suite, Rovo is an

**[00:00:28 – 00:00:32]** AI teammate that powers our intelligent search, chat, and

**[00:00:32 – 00:00:36]** agent platform across Atlassian's product family, such as Jira,

**[00:00:36 – 00:00:41]** Confluence, JSM, and Bitbucket.

**[00:00:41 – 00:00:45]** And it operates on top of what we call the Teamwork

**[00:00:45 – 00:00:50]** Graph, which is a foundational data layer that connects

**[00:00:50 – 00:00:55]** Many different kinds of data and activity traces people and teams

**[00:00:55 – 00:00:59]** across different organizations.

**[00:00:59 – 00:01:08]** Rovo, our goal is to turn everyday work for every team

**[00:01:08 – 00:01:15]** into Agent-intelligent workflows without human supervision.

**[00:01:16 – 00:01:20]** So today, we are going to focus on two specific reinforcement

**[00:01:20 – 00:01:22]** learned workloads.

**[00:01:22 – 00:01:27]** One is a simple application for optimizing query rewriting

**[00:01:28 – 00:01:32]** in Rovo search and retrieval augmented generation.

**[00:01:32 – 00:01:36]** The second one is an agenda planner for long horizon

**[00:01:36 – 00:01:39]** Jira-centric workloads.

**[00:01:41 – 00:01:45]** So why do we need reinforcement learning in enterprises?

**[00:01:45 – 00:01:50]** Simply put, we found that static rules and LLM prompts

**[00:01:50 – 00:01:55]** don't generalize well across different use cases, different

**[00:01:55 – 00:01:59]** customers, and different domains.

**[00:02:00 – 00:02:05]** It's also extremely hard for humans to manually encode

**[00:02:05 – 00:02:09]** trade-offs between different competing goals such as precision,

**[00:02:09 – 00:02:13]** recall, latency, and effort.

**[00:02:13 – 00:02:17]** We need policies that can learn from itself, from

**[00:02:17 – 00:02:20]** enterprise-scale data.

**[00:02:21 – 00:02:25]** For example, we want to learn how to better interpret and

**[00:02:25 – 00:02:31]** rewrite users queries that they enter into the system and

**[00:02:31 – 00:02:38]** we want the system to learn how to sequence tools over long horizons

**[00:02:38 – 00:02:41]** to complete tasks and reach goals.

**[00:02:41 – 00:02:45]** So, NVIDIA's reinforcement learning stack, which includes

**[00:02:45 – 00:02:52]** NeMo RL and NeMo JEM, gives us much faster training speed

**[00:02:52 – 00:02:54]** compared to competing products.

**[00:02:54 – 00:02:58]** And it also offers us a standard unified way to create and

**[00:02:59 – 00:03:02]** manage RL environments.

**[00:03:02 – 00:03:10]** So our very first use case that we use NeMo RL for is for optimizing

**[00:03:10 – 00:03:13]** query writing for RoboSearch.

**[00:03:13 – 00:03:17]** So the goal is to rewrite user queries to maximize downstream

**[00:03:17 – 00:03:21]** search relevance and task success.

**[00:03:21 – 00:03:26]** So the action it takes is to propose structured

**[00:03:26 – 00:03:30]** rewrite proposals, which may include spell correction,

**[00:03:30 – 00:03:34]** normalization, expansion, and design migration that's

**[00:03:34 – 00:03:37]** commonly seen in search engines.

**[00:03:37 – 00:03:43]** So the state includes user queries and product, tenant,

**[00:03:43 – 00:03:45]** and user context.

**[00:03:46 – 00:03:49]** So the reward function would be based on the recall of

**[00:03:49 – 00:03:55]** relevant docs and it could also include other type of

**[00:03:55 – 00:04:00]** search relevance proxy metrics such as mean reciprocal rank

**[00:04:00 – 00:04:06]** or normalized discounted cumulative gain and other type of downstream

**[00:04:06 – 00:04:08]** task success metrics.

**[00:04:08 – 00:04:13]** We may also include penalties such as output format correctness,

**[00:04:13 – 00:04:17]** length of the query, and safety.

**[00:04:18 – 00:04:23]** So let's take a look at how the actual RL training workflow

**[00:04:23 – 00:04:27]** works and integrate it with the RoboSearch environment.

**[00:04:27 – 00:04:32]** So we start with an input dataset, which includes queries

**[00:04:32 – 00:04:35]** and target documents.

**[00:04:36 – 00:04:40]** So these are sampled and batched and used as a

**[00:04:40 – 00:04:43]** prompt for our policy.

**[00:04:43 – 00:04:45]** So the active policy is a query rewrite model, which

**[00:04:46 – 00:04:52]** is based on Google's Gemma 3 4 billion-base model.

**[00:04:52 – 00:04:57]** It takes the prompt and generates a query-write proposal.

**[00:04:57 – 00:05:00]** And the rewritten query is then sent to the search environment,

**[00:05:00 – 00:05:07]** which goes through retrieval using BM25 and vector search or

**[00:05:07 – 00:05:11]** hybrid search, and then re-ranking.

**[00:05:11 – 00:05:16]** And next, the re-work function evaluates the outcome.

**[00:05:16 – 00:05:22]** It calculates success signals such as recall, nDCG, MMR

**[00:05:22 – 00:05:27]** that I just mentioned, and it also applies necessary penalties

**[00:05:27 – 00:05:30]** such as format, length, and safety.

**[00:05:30 – 00:05:34]** So this is where Reinforcement Learning kicks in.

**[00:05:34 – 00:05:41]** With NeMo RL, we leverage GRPO, Group Relative Policy Optimization.

**[00:05:42 – 00:05:48]** It samples a group of different rewrites for the same prompt.

**[00:05:49 – 00:05:52]** And it computes our advantage by comparing the different

**[00:05:52 – 00:05:53]** rewrites against each other.

**[00:05:53 – 00:05:57]** And then it incorporates the environment rewards

**[00:05:57 – 00:06:02]** and the log probabilities to compute the advantage.

**[00:06:02 – 00:06:10]** And then, because we use our own policy optimization, we strictly

**[00:06:10 – 00:06:18]** leverage the action that the model just took to optimize itself.

**[00:06:19 – 00:06:25]** So it executes this GRPO update where the gradients flows

**[00:06:25 – 00:06:29]** back to the policy to update the policy itself.

**[00:06:29 – 00:06:34]** And at the same time, the outputs get routed to logging

**[00:06:34 – 00:06:40]** and checkpointing and evaluation and eventually deployment.

**[00:06:40 – 00:06:44]** So this is basically how the workflow works.

**[00:06:44 – 00:06:48]** So the training infrastructure that we build is on a

**[00:06:48 – 00:06:50]** Kubernetes cluster.

**[00:06:50 – 00:06:57]** So the user just need to write up this YAML config, and they

**[00:06:57 – 00:07:01]** submit this config together with their custom code through

**[00:07:01 – 00:07:07]** this job submission microservice into the training cluster.

**[00:07:07 – 00:07:10]** The training cluster,

**[00:07:10 – 00:07:16]** NVIDIA leverages a pool of GPU instances that uses NVIDIA

**[00:07:16 – 00:07:19]** GPUs such as H200s.

**[00:07:20 – 00:07:25]** And everything executes on the Docker image that leverages

**[00:07:25 – 00:07:29]** the NeMo RL-based image.

**[00:07:30 – 00:07:38]** So in evaluation, we found that our reinforcement-learned model beats

**[00:07:38 – 00:07:40]** the base model across the board.

**[00:07:40 – 00:07:46]** So in the standard data set, our GRPO model improved

**[00:07:46 – 00:07:49]** recall at 10 by 6%.

**[00:07:49 – 00:07:53]** And we observed similar gains on the hard queries data set, where it

**[00:07:53 – 00:07:57]** increased the recall at about 5%.

**[00:07:58 – 00:08:02]** The main takeaway here is that the reinforcement learning actually

**[00:08:02 – 00:08:08]** works by leveraging NeMo RL to optimize the downstream search

**[00:08:09 – 00:08:11]** relevance objective directly.

**[00:08:11 – 00:08:16]** We can achieve better document recall than we ever could

**[00:08:16 – 00:08:23]** with regular prompt tuning or instruction tuning alone.

**[00:08:23 – 00:08:29]** So our next use case is for optimizing one-hour

**[00:08:29 – 00:08:34]** long-horizon agents that we call the Jira Issue Solver.

**[00:08:35 – 00:08:42]** So the goal of this Jira Issue Solver is to resolve complex

**[00:08:42 – 00:08:47]** Jira issues end-to-end without human intervention.

**[00:08:47 – 00:08:52]** So, as the creator of Jira, we think this is a very important

**[00:08:52 – 00:08:59]** problem because if we are building AI that will eventually

**[00:08:59 – 00:09:02]** replace our jobs.

**[00:09:03 – 00:09:08]** We feel it's very important that this AI will handle and

**[00:09:08 – 00:09:12]** clear your Jira backlog first.

**[00:09:13 – 00:09:20]** So to achieve this, we need to tackle the long-horizon problem,

**[00:09:20 – 00:09:26]** which involves more than 10 coherent steps, and tool switching,

**[00:09:26 – 00:09:29]** and maintaining long-term memory.

**[00:09:33 – 00:09:36]** So the agent will need to go through the entire lifecycle

**[00:09:36 – 00:09:42]** of resolving ticket, including triage, investigation, mitigation,

**[00:09:43 – 00:09:46]** verify, and close.

**[00:09:47 – 00:09:55]** We give a suite of tools to the agent at its disposal, including

**[00:09:55 – 00:10:02]** tools across Jira, Confluence, and regular development tools

**[00:10:02 – 00:10:10]** such as feature flags, CICD, and metrics, and so on and so forth.

**[00:10:10 – 00:10:17]** So most importantly we also build in the permission profiles

**[00:10:17 – 00:10:20]** and safety controls into the system because we need to

**[00:10:20 – 00:10:24]** ensure there's no successful unauthorized high-risk actions

**[00:10:24 – 00:10:27]** taken during testing.

**[00:10:27 – 00:10:31]** So that's very important for enterprises.

**[00:10:31 – 00:10:36]** So the bottleneck, one of the bottlenecks we aim to address

**[00:10:36 – 00:10:43]** with this system is hallucination and state amnesia that's very

**[00:10:43 – 00:10:47]** commonly seen in single-term LLMs.

**[00:10:50 – 00:10:55]** So to tackle this problem, we use what we call the agent

**[00:10:55 – 00:10:58]** flow style optimization loop.

**[00:10:58 – 00:11:01]** This is originally a framework that's proposed by researchers

**[00:11:01 – 00:11:04]** at Stanford University.

**[00:11:04 – 00:11:09]** It can optimize agent policy inside a multi-step,

**[00:11:09 – 00:11:11]** two-using environment.

**[00:11:12 – 00:11:19]** Instead of using one giant LLM that needs to consider reasoning

**[00:11:19 – 00:11:25]** and true execution simultaneously, we break down the work into

**[00:11:25 – 00:11:29]** several specialized modules.

**[00:11:29 – 00:11:34]** As you can see in this slide, this system includes a team of

**[00:11:34 – 00:11:39]** coordinated modules, including the Planner, which is a reinforcement

**[00:11:39 – 00:11:44]** learning geometry model that outputs strict JSON actions,

**[00:11:44 – 00:11:50]** which includes thoughts, action types, and arguments.

**[00:11:50 – 00:11:54]** And there is Executor, which is a deterministic

**[00:11:54 – 00:11:59]** Python layer that enforces permissions and runs tools.

**[00:11:59 – 00:12:04]** And there's a verifier that's another deterministic module

**[00:12:04 – 00:12:10]** that checks metrics, flags, Jira status and provides a

**[00:12:10 – 00:12:17]** feedback signal that indicates whether the issue was properly

**[00:12:17 – 00:12:21]** resolved before accepting success.

**[00:12:22 – 00:12:27]** So the agent has access to a suite of different tools,

**[00:12:27 – 00:12:31]** like, as I mentioned, that's across Jira, Confluence, Ops,

**[00:12:31 – 00:12:33]** Coms, and so on and so forth.

**[00:12:33 – 00:12:37]** So as a very simple example of how the agent utilizes

**[00:12:37 – 00:12:42]** tools, suppose you got a ticket that reports search results

**[00:12:42 – 00:12:44]** are intermittently empty.

**[00:12:45 – 00:12:48]** And this agent will kick in and go to investigate the problem.

**[00:12:48 – 00:12:55]** And the planner, what the planner executes is a series of two calls.

**[00:12:55 – 00:12:59]** For example, it will first get the issue, analyze the

**[00:12:59 – 00:13:01]** issue, and then it will search Jira, call the search Jira

**[00:13:01 – 00:13:03]** tool, to find similar issues.

**[00:13:03 – 00:13:06]** In this case, it found, oh, there's another issue that's

**[00:13:06 – 00:13:10]** potentially a duplicate of this current ticket at hand.

**[00:13:10 – 00:13:15]** And then it leaves a comment on that new ticket that says,

**[00:13:15 – 00:13:20]** oh, likely we found a duplicate, and then we will post additional

**[00:13:20 – 00:13:23]** update in this original ticket.

**[00:13:23 – 00:13:26]** And then it adds a label to the new ticket that says duplicate,

**[00:13:26 – 00:13:32]** and then close it to done and duplicate status.

**[00:13:33 – 00:13:37]** So as I mentioned, the objective of this agent is to resolve

**[00:13:37 – 00:13:43]** Jira issues end-to-end using tools, ensuring no safety violations.

**[00:13:43 – 00:13:49]** To achieve this, we use SFT, supervised fine tuning, to

**[00:13:49 – 00:13:54]** learn tool selection and closer sequencing using a series

**[00:13:54 – 00:13:57]** of oracle traces.

**[00:13:57 – 00:14:01]** And on top of that, we apply Reinforcement Learning with

**[00:14:01 – 00:14:07]** this flow GRPO algorithm for on-policy optimization.

**[00:14:07 – 00:14:12]** So the key of this Reinforcement Learning algorithm is the

**[00:14:12 – 00:14:14]** reward that we design.

**[00:14:14 – 00:14:21]** So the reward function considers three key behaviors.

**[00:14:21 – 00:14:24]** Number one, Success.

**[00:14:24 – 00:14:29]** Which is to ensure that we could reach the final goal

**[00:14:29 – 00:14:32]** of resolving tickets.

**[00:14:32 – 00:14:34]** And second, Recovery.

**[00:14:34 – 00:14:40]** This is to teach the model how to recover from errors

**[00:14:40 – 00:14:43]** and tool failures.

**[00:14:43 – 00:14:46]** And lastly,

**[00:14:46 – 00:14:50]** There's, we incorporate anti-regression anchors.

**[00:14:50 – 00:14:56]** This is to ensure, to prevent the model from regressing or

**[00:14:56 – 00:15:01]** unlearning the fundamental skills that it has already acquired.

**[00:15:01 – 00:15:05]** So all three elements are super important.

**[00:15:06 – 00:15:09]** So the key engineering lessons that we learned through building

**[00:15:09 – 00:15:11]** this agent is that

**[00:15:11 – 00:15:17]** First, structured outputs and the schema constraints

**[00:15:17 – 00:15:22]** of the outputs are prerequisites to building such systems.

**[00:15:22 – 00:15:28]** And safety must be considered at the system level and enforced

**[00:15:29 – 00:15:33]** at the system level rather than afterthought.

**[00:15:33 – 00:15:38]** And long-horizon agents need explicit progress and

**[00:15:38 – 00:15:43]** closure signals to ensure it doesn't enter loops.

**[00:15:43 – 00:15:51]** So we evaluated this system across 64 and synthetic worlds.

**[00:15:53 – 00:15:56]** Each with the different workflow types and run the

**[00:15:56 – 00:16:01]** tool types and metrics, logs, and permission profiles.

**[00:16:02 – 00:16:06]** This includes both in-domain and out-of-domain tasks.

**[00:16:07 – 00:16:16]** So for in-domain evaluation, we use 360 in-domain tasks

**[00:16:16 – 00:16:17]** to evaluate the system.

**[00:16:17 – 00:16:23]** It turned out that the combining supervised fine-tuning and

**[00:16:23 – 00:16:27]** reinforcement learning achieved overall success rate of 94%.

**[00:16:27 – 00:16:31]** This is a 16% of an increase over the base model.

**[00:16:36 – 00:16:44]** The SFT plus RL model also achieved a much lower number

**[00:16:44 – 00:16:48]** of steps, average steps to achieve, to resolve tickets

**[00:16:48 – 00:16:49]** compared to the base model.

**[00:16:49 – 00:16:56]** So it decreases 21% from an average of 4.33 steps to an average

**[00:16:56 – 00:17:03]** of 3.4 steps for resolving tickets in the in-domain task data set.

**[00:17:03 – 00:17:09]** But the real test is on out-of-domain tasks.

**[00:17:09 – 00:17:14]** Here, we actually found that the supervised fine-tuning

**[00:17:14 – 00:17:15]** failed miserably.

**[00:17:15 – 00:17:22]** So it actually degraded the model performance by 21% compared

**[00:17:22 – 00:17:24]** to the base model.

**[00:17:24 – 00:17:31]** However, combining SFT and RL brought the success rate up 12%,

**[00:17:31 – 00:17:34]** Again, compared to the base model.

**[00:17:34 – 00:17:37]** I think that these results are very intuitive to understand

**[00:17:37 – 00:17:43]** because the supervised fine-tuning fundamentally is teaching

**[00:17:43 – 00:17:50]** the model how to achieve success through a happy path.

**[00:17:50 – 00:17:56]** It learns basic tool selection and sequencing directly from

**[00:17:56 – 00:17:58]** perfect oracle traces.

**[00:17:58 – 00:18:03]** It knows exactly what success looks like when everything

**[00:18:03 – 00:18:11]** goes right, but the reality of the enterprise workflows is messy.

**[00:18:11 – 00:18:18]** So it has two failures, it has API timeouts, it has all

**[00:18:18 – 00:18:19]** kinds of unexpected things.

**[00:18:19 – 00:18:25]** But SFT fundamentally cannot handle or anticipate all kinds

**[00:18:25 – 00:18:28]** of these unexpected failures.

**[00:18:29 – 00:18:34]** That's exactly where reinforcement learning is important.

**[00:18:34 – 00:18:39]** Reinforcement learning adds necessary robustness for these

**[00:18:39 – 00:18:41]** unexpected scenarios.

**[00:18:41 – 00:18:45]** It also teaches the model how to recover from errors and failures.

**[00:18:45 – 00:18:49]** That's what we call this recovery learning.

**[00:18:49 – 00:18:56]** That's extremely important for real-life enterprise scenarios.

**[00:18:56 – 00:19:02]** So to summarize the key learnings from this journey, first of

**[00:19:02 – 00:19:07]** all, we found that long-horizon agents fails first on the

**[00:19:07 – 00:19:11]** tool interface, not on reasoning.

**[00:19:12 – 00:19:16]** So this indicates a strict schemas and constraints outputs

**[00:19:17 – 00:19:21]** and safety gates are very important from the get-go.

**[00:19:21 – 00:19:26]** So in fact, they are mandatory, we think.

**[00:19:26 – 00:19:32]** And our biggest unlock was the implementation of closure controls

**[00:19:32 – 00:19:38]** such as completion keys and loop breakers to ensure the system will

**[00:19:38 – 00:19:44]** stop or keep progressing and stop investigation and finish tickets

**[00:19:44 – 00:19:48]** within a finite number of steps.

**[00:19:50 – 00:19:53]** Supervised fine-tuning helps in-domain but overfits header

**[00:19:53 – 00:19:56]** policy on out-of-domain tasks.

**[00:19:56 – 00:20:00]** Reinforcement Learning adds robustness but needs a recovery

**[00:20:00 – 00:20:05]** curriculum paired with strong anchors to ensure the model

**[00:20:05 – 00:20:11]** knows how to get back on track when it faces two failures.

**[00:20:11 – 00:20:14]** And the final suggestion is that when you build this kind of

**[00:20:14 – 00:20:19]** model, this kind of systems, always judge your progress and quality on

**[00:20:19 – 00:20:26]** other domain tasks and shift your learning to a choice-only policy

**[00:20:26 – 00:20:32]** such as tool choice and ranking to ensure stable generalization.

**[00:20:32 – 00:20:37]** For future use cases, we are super happy to continue to

**[00:20:37 – 00:20:43]** work with NVIDIA to further optimize search and better balance

**[00:20:43 – 00:20:46]** latency and quality trade-offs by building this intelligent

**[00:20:46 – 00:20:52]** reasoning router and further increase the quality of the GRP

**[00:20:52 – 00:20:56]** algorithm for multi-turn rollouts.

**[00:20:56 – 00:21:01]** Ultimately, by building with NVIDIA and building on NVIDIA's

**[00:21:01 – 00:21:04]** AI software stack, these steps will bring us closer to our

**[00:21:04 – 00:21:09]** core goal of turning everyday work into truly autonomous,

**[00:21:09 – 00:21:11]** intelligent workflows for everyone.

**[00:21:11 – 00:21:13]** Thank you. That concludes my talk.

**[00:21:13 – 00:21:17]** If you're interested in Rovo, feel free to scan this QR

**[00:21:17 – 00:21:20]** code or go to this website.

**[00:21:20 – 00:21:23]** And with that, I'm handing over to Rob to introduce how

**[00:21:23 – 00:21:29]** to build custom agents in Vertex AI Training Clusters.

**[00:21:29 – 00:21:30]** Welcome, Rob. Thank you, Kang.

**[00:21:30 – 00:21:32]** Appreciate it.

**[00:21:32 – 00:21:34]** Hello, everyone.

**[00:21:34 – 00:21:35]** Are we moving through slides?

**[00:21:35 – 00:21:37]** Yes, we are. Hey, folks.

**[00:21:37 – 00:21:38]** My name is Rob Van Dusen.

**[00:21:38 – 00:21:40]** I'm a product manager with Google.

**[00:21:40 – 00:21:43]** I help support our Vertex AI team.

**[00:21:43 – 00:21:46]** For those of you who don't know, Vertex AI is Google

**[00:21:46 – 00:21:51]** Cloud's place to build, train, evaluate, and deploy machine

**[00:21:51 – 00:21:53]** learning and AI models.

**[00:21:53 – 00:21:56]** And today, I'm going to talk about a new feature that recently

**[00:21:56 – 00:21:57]** launched inside Vertex.

**[00:21:57 – 00:22:00]** This is a new service called the Vertex AI Training Cluster,

**[00:22:00 – 00:22:05]** and I'm going to focus today on how our data science team

**[00:22:05 – 00:22:09]** used that platform, along with NVIDIA, NeMo RL, and

**[00:22:09 – 00:22:16]** distillation techniques in order to customize and train an agent.

**[00:22:16 – 00:22:18]** Alright, so briefly before I get started, just a little

**[00:22:18 – 00:22:19]** bit of definitional work.

**[00:22:19 – 00:22:22]** What is an agent?

**[00:22:22 – 00:22:26]** An agent is a piece of software that's going to own the execution

**[00:22:26 – 00:22:31]** of a task on behalf of a user.

**[00:22:31 – 00:22:33]** And the key here is that it's going to behave in an intelligent

**[00:22:33 – 00:22:35]** and an autonomous way.

**[00:22:35 – 00:22:39]** This is a departure from past software paradigms where software

**[00:22:39 – 00:22:42]** is a little bit more deterministic and is focused on helping

**[00:22:42 – 00:22:43]** a person do a task.

**[00:22:43 – 00:22:47]** With an agent, it's owning the task.

**[00:22:47 – 00:22:50]** So zooming in a little bit, this is a high level sort

**[00:22:50 – 00:22:55]** of architecture of what an AI agent is, components here.

**[00:22:55 – 00:22:57]** In green, we have the model.

**[00:22:57 – 00:23:00]** Usually this is a generative AI model.

**[00:23:00 – 00:23:04]** This is sort of the reasoning engine of the agent.

**[00:23:04 – 00:23:07]** And then in yellow we have an orchestration component,

**[00:23:07 – 00:23:11]** this is a piece of the agent that's going to help manage state, memory,

**[00:23:12 – 00:23:14]** and help with some coordination tasks inside the agent.

**[00:23:15 – 00:23:17]** And then finally in blue we have the tools.

**[00:23:17 – 00:23:21]** These are the tools that the agent has access to and can call.

**[00:23:21 – 00:23:27]** These are APIs, functions, databases, or even other agents.

**[00:23:27 – 00:23:30]** Now, as you can imagine, the model is a very important

**[00:23:30 – 00:23:31]** part of this agent.

**[00:23:31 – 00:23:34]** It's the center where all the reasoning and logic is

**[00:23:34 – 00:23:35]** going to be happening.

**[00:23:35 – 00:23:39]** And there are a lot of quality off-the-shelf models that

**[00:23:39 – 00:23:42]** you could use for agentic purposes.

**[00:23:42 – 00:23:46]** I have to pitch Gemini, of course, because I am a product

**[00:23:46 – 00:23:49]** manager at Google, but there's also some excellent open-source

**[00:23:49 – 00:23:52]** models that you could use as well.

**[00:23:52 – 00:23:55]** Now, however, you as customers, you could imagine scenarios

**[00:23:55 – 00:24:00]** where those open-source or off-the-shelf models are not going

**[00:24:00 – 00:24:01]** to quite cut up for your use case.

**[00:24:01 – 00:24:06]** You might have some specific data that's proprietary that you might

**[00:24:06 – 00:24:08]** want to infuse into the model.

**[00:24:08 – 00:24:10]** You might have some specific or unique tools that only

**[00:24:10 – 00:24:13]** exist inside your environment that an open source model

**[00:24:13 – 00:24:14]** hasn't been exposed to.

**[00:24:14 – 00:24:19]** So you might want to customize your agent to operate better

**[00:24:19 – 00:24:21]** in that environment, right?

**[00:24:21 – 00:24:25]** Now, the challenge is that that's pretty hard to do.

**[00:24:25 – 00:24:29]** It can be a real heavy lift to customize a model for

**[00:24:29 – 00:24:31]** agentic use cases.

**[00:24:31 – 00:24:33]** And that's true for several reasons.

**[00:24:33 – 00:24:36]** You as a machine learning engineer, if you want to customize one

**[00:24:36 – 00:24:40]** of these models, you have to deal with some very complex techniques.

**[00:24:40 – 00:24:44]** You have to choose the right pre- or post-training technique to

**[00:24:44 – 00:24:45]** use to get good Good performance.

**[00:24:46 – 00:24:47]** You might have to deal with different types of

**[00:24:47 – 00:24:50]** model parallelism and optimizing framework code

**[00:24:50 – 00:24:51]** for good performance.

**[00:24:51 – 00:24:54]** If you're going to be customizing this agent on a continuous

**[00:24:54 – 00:24:58]** basis, it's very important that you're achieving good throughput

**[00:24:58 – 00:25:00]** in your training process.

**[00:25:00 – 00:25:02]** Otherwise, you might be wasting compute time, you

**[00:25:02 – 00:25:03]** might be wasting money.

**[00:25:03 – 00:25:07]** And then you also have to contend with sort of the evolving

**[00:25:07 – 00:25:08]** state of the art.

**[00:25:08 – 00:25:09]** It seems like every couple of months there's a new

**[00:25:09 – 00:25:13]** technique, there's a new way to do this process.

**[00:25:13 – 00:25:15]** And so as a machine learning engineer, you have to stay

**[00:25:15 – 00:25:17]** very close to the research space in order to be implementing

**[00:25:17 – 00:25:19]** those new techniques.

**[00:25:20 – 00:25:23]** And then underneath all of that in the framework level, you

**[00:25:23 – 00:25:27]** have to deal with infrastructure management and orchestration.

**[00:25:27 – 00:25:30]** As a data scientist, worrying about these things is the worst.

**[00:25:30 – 00:25:34]** You want to focus on science, you want to focus on differentiation

**[00:25:34 – 00:25:37]** and building a product that's going to perform better than

**[00:25:37 – 00:25:38]** your competitors.

**[00:25:38 – 00:25:41]** You don't want to be spending your time on these undifferentiated

**[00:25:41 – 00:25:44]** things like making sure my hardware is up and available,

**[00:25:44 – 00:25:49]** making sure that my orchestrator is well configured.

**[00:25:49 – 00:25:51]** So what I'm going to talk about today is how the Vertex

**[00:25:51 – 00:25:56]** AI Training Cluster helps simplify this workflow, and

**[00:25:56 – 00:25:59]** in particular, how you can use recipes built on top of NeMo

**[00:25:59 – 00:26:07]** RL to do this model customization process with NVIDIA GPUs.

**[00:26:09 – 00:26:11]** Today, I'm going to talk about a pipeline that our

**[00:26:11 – 00:26:13]** data science team built.

**[00:26:13 – 00:26:16]** This is an agent customization pipeline.

**[00:26:16 – 00:26:19]** It's built on top of a Vertex AI Training Cluster infrastructure,

**[00:26:19 – 00:26:22]** and it leverages NeMo RL recipes.

**[00:26:22 – 00:26:25]** There are three components to this pipeline.

**[00:26:25 – 00:26:29]** There's a data prep step, there's a distillation step,

**[00:26:29 – 00:26:33]** distillation being the main technique that we used for

**[00:26:33 – 00:26:36]** this agent customization pipeline.

**[00:26:36 – 00:26:40]** And then there's an evaluation and a results component here as well.

**[00:26:40 – 00:26:44]** Now for this particular example, we were focused on training

**[00:26:44 – 00:26:49]** a model to better use tools, so agentic tool use, and accordingly

**[00:26:49 – 00:26:53]** we selected the TAU2 bench, which is an evaluation set

**[00:26:53 – 00:26:55]** focused on tool use.

**[00:26:55 – 00:26:57]** But this pipeline here is completely customizable.

**[00:26:57 – 00:27:01]** You could set it for your own use case, and in fact,

**[00:27:01 – 00:27:04]** all of the related data and code

**[00:27:04 – 00:27:08]** Underlies this presentation is available to our customers

**[00:27:08 – 00:27:11]** and we also include an explanatory blog if you want to go a little bit

**[00:27:11 – 00:27:14]** deeper on some of the details here.

**[00:27:17 – 00:27:20]** All right, so before I dive in, maybe just a quick bit

**[00:27:20 – 00:27:22]** of definitional work on the technique that we used.

**[00:27:22 – 00:27:24]** So we focused on distillation.

**[00:27:24 – 00:27:27]** We'll call it with distillation.

**[00:27:27 – 00:27:30]** This is a technique where we start with a more capable

**[00:27:30 – 00:27:33]** teacher model, and we try and transfer a subset of its

**[00:27:34 – 00:27:39]** capabilities or knowledge down to a student model, usually a smaller,

**[00:27:39 – 00:27:41]** more efficient student model.

**[00:27:42 – 00:27:45]** Two types of distillation that are often used, we have on-policy

**[00:27:45 – 00:27:47]** and off-policy distillation.

**[00:27:47 – 00:27:51]** With on-policy distillation, we have both the student and

**[00:27:51 – 00:27:55]** the teacher substantiated or instantiated in memory,

**[00:27:55 – 00:28:01]** and we pass tasks or prompts to the student model.

**[00:28:01 – 00:28:04]** Student model starts to generate responses, and the teacher

**[00:28:04 – 00:28:06]** model in a real-time way is Everybody's going to evaluate

**[00:28:06 – 00:28:08]** the quality of those responses.

**[00:28:08 – 00:28:11]** And then based on that feedback, we would update the weights

**[00:28:11 – 00:28:13]** of the student model.

**[00:28:13 – 00:28:15]** On-policy distillation can be fairly complex,

**[00:28:15 – 00:28:18]** both from a framework and from an infrastructure

**[00:28:18 – 00:28:21]** orchestration standpoint.

**[00:28:21 – 00:28:23]** You have to have both models instantiated, you have to make sure

**[00:28:23 – 00:28:25]** that they're communicating well.

**[00:28:26 – 00:28:29]** It can be difficult to pull off well.

**[00:28:29 – 00:28:33]** This stands in contrast to off-policy distillation, which

**[00:28:33 – 00:28:36]** is a little bit more of a batch-oriented framework.

**[00:28:36 – 00:28:40]** With off-policy distillation, you take all of your tasks

**[00:28:40 – 00:28:45]** or prompts, you pass them into that larger, more capable teacher model.

**[00:28:45 – 00:28:48]** You get some responses, and then you take those responses,

**[00:28:48 – 00:28:50]** you pair them with the tasks, and that's your training data

**[00:28:50 – 00:28:53]** set that you will use to improve the student model.

**[00:28:53 – 00:28:56]** So, off-policy distillation, since it's a bit more of a

**[00:28:56 – 00:28:59]** batch-oriented process, it's a bit more straightforward.

**[00:28:59 – 00:29:02]** And so, actually, this pipeline relies more on the off-policy

**[00:29:02 – 00:29:03]** distillation standpoint.

**[00:29:03 – 00:29:05]** So, that's what we're going to talk about today.

**[00:29:05 – 00:29:07]** But I will note that if you look at the literature, you'll

**[00:29:07 – 00:29:09]** see a lot of examples where customers are doing both of these

**[00:29:09 – 00:29:12]** things, or researchers, rather, are doing both of these things,

**[00:29:12 – 00:29:15]** sometimes starting with off-policy distillation and then moving

**[00:29:15 – 00:29:19]** on to on-policy distillation.

**[00:29:19 – 00:29:21]** All right, so one of the first steps that you have

**[00:29:21 – 00:29:26]** in the data prep standpoint is choosing a teacher model.

**[00:29:26 – 00:29:29]** Here on this slide, you can see we evaluated a series

**[00:29:29 – 00:29:30]** of different teacher models.

**[00:29:30 – 00:29:34]** These are represented across the y-axis here, ranging from a small,

**[00:29:34 – 00:29:39]** I think about 2 billion parameter post-train QEN model, up to a much

**[00:29:39 – 00:29:41]** larger 350 billion parameter model.

**[00:29:42 – 00:29:45]** And then across the y-axis, we have our student models,

**[00:29:45 – 00:29:49]** also ranging from about 2 billion parameters in size up

**[00:29:49 – 00:29:51]** to 14 billion parameters in size.

**[00:29:51 – 00:29:54]** And the numbers inside the grid here represent the performance

**[00:29:54 – 00:29:56]** on the TAU2 bench.

**[00:29:56 – 00:30:00]** Again, the TAU2 bench, this is an evaluation data set

**[00:30:00 – 00:30:03]** focused on agentic tool use.

**[00:30:03 – 00:30:06]** And you can see from the results here, it kind of

**[00:30:06 – 00:30:08]** conforms to intuition.

**[00:30:08 – 00:30:13]** Our larger teacher model yielded better accuracy with our student

**[00:30:13 – 00:30:16]** models across the board.

**[00:30:16 – 00:30:18]** For the purposes of this pipeline, we moved forward with using

**[00:30:18 – 00:30:22]** that larger 350 billion parameter model because of these results.

**[00:30:22 – 00:30:25]** But it's important to call out, this is not always the case.

**[00:30:25 – 00:30:27]** We experimented with some other use cases, some other

**[00:30:27 – 00:30:31]** evaluation datasets, where actually smaller teacher models

**[00:30:31 – 00:30:34]** actually yielded better results.

**[00:30:34 – 00:30:37]** So, really, the takeaway here is that you, as a data scientist,

**[00:30:37 – 00:30:41]** as you are going through each step of this pipeline, it's important

**[00:30:41 – 00:30:44]** that you run these empirical studies to determine what is going

**[00:30:44 – 00:30:47]** to yield the best results for me.

**[00:30:47 – 00:30:49]** All right, so we've selected our teacher model.

**[00:30:49 – 00:30:52]** Now, the next step is to synthesize some data that we're gonna

**[00:30:52 – 00:30:54]** use for training, right?

**[00:30:54 – 00:30:57]** First step here is task generation.

**[00:30:57 – 00:31:02]** And so what we did, we generated a series of tasks and prompts

**[00:31:02 – 00:31:05]** using an LLM that were based off the TAU2 bench.

**[00:31:05 – 00:31:08]** And you can see an example of the task that we've generated over here

**[00:31:08 – 00:31:10]** on the right side of the screen.

**[00:31:10 – 00:31:16]** Each task includes an environment or a scenario, a database state.

**[00:31:17 – 00:31:19]** Some tools that the agent has access to and can call.

**[00:31:19 – 00:31:23]** And then, critically, success criteria, whether or not the

**[00:31:23 – 00:31:25]** response is correct.

**[00:31:25 – 00:31:28]** In this example on the right,

**[00:31:28 – 00:31:31]** our agent here is a telecom technical support agent, and

**[00:31:31 – 00:31:35]** they're trying to help a customer with an issue where they can't

**[00:31:35 – 00:31:36]** get any service on their phone.

**[00:31:36 – 00:31:41]** Maybe they're calling in from a landline, I'm not sure.

**[00:31:41 – 00:31:44]** We have a list here of the available tools that the agent

**[00:31:44 – 00:31:47]** has access to, and then the success criteria.

**[00:31:47 – 00:31:51]** Here, with this task, it's whether or not the agent successfully

**[00:31:52 – 00:31:57]** recommends that the client use these tools to improve

**[00:31:57 – 00:31:59]** the reception that they have.

**[00:32:00 – 00:32:02]** Alright, so we've created all those tasks, then the

**[00:32:02 – 00:32:04]** next step is to generate responses.

**[00:32:04 – 00:32:08]** We take those tasks, we pass them into that 350 billion parameter

**[00:32:08 – 00:32:11]** model, and outcome our responses.

**[00:32:11 – 00:32:14]** Those responses are paired with the tasks, and that becomes

**[00:32:14 – 00:32:18]** our training dataset.

**[00:32:18 – 00:32:21]** Now, a couple more quick notes on Dataprep here.

**[00:32:21 – 00:32:23]** First one's on rollouts.

**[00:32:23 – 00:32:27]** So, a rollout is where we generate a set of responses to

**[00:32:27 – 00:32:29]** those tasks that we just generated.

**[00:32:29 – 00:32:32]** But we found that,

**[00:32:33 – 00:32:36]** Increasing the number of rollouts that you do, that is, producing

**[00:32:36 – 00:32:42]** more responses per task, can actually improve the performance or

**[00:32:42 – 00:32:44]** the accuracy of the student model.

**[00:32:44 – 00:32:47]** So you can see with this chart here, as we increase the number

**[00:32:47 – 00:32:51]** of rollouts going up from 1 to 16, the accuracy of our

**[00:32:51 – 00:32:55]** student model improves.

**[00:32:55 – 00:32:57]** You can see here we stopped at 16.

**[00:32:57 – 00:32:59]** There's really nothing magical about the number 16.

**[00:32:59 – 00:33:03]** It's just that my data science team was maybe just a little

**[00:33:03 – 00:33:07]** bit GPU-poor at the time that we did this work.

**[00:33:07 – 00:33:10]** So, you as customers, I encourage you to push the limits here,

**[00:33:10 – 00:33:11]** go a little further.

**[00:33:11 – 00:33:14]** It's really just a trade-off between experiment time

**[00:33:14 – 00:33:16]** and compute time.

**[00:33:16 – 00:33:20]** But again, indicating that as a data scientist, do these

**[00:33:20 – 00:33:23]** empirical studies throughout your pipeline to make sure that you're

**[00:33:23 – 00:33:27]** generating or you're achieving the best accuracy that you can.

**[00:33:27 – 00:33:30]** All right, so last slide here on data prep.

**[00:33:30 – 00:33:32]** This relates to rejection sampling.

**[00:33:32 – 00:33:38]** Rejection sampling is a process by which we eliminate responses

**[00:33:38 – 00:33:42]** from the teacher that are incorrect, that fail that success

**[00:33:42 – 00:33:44]** criteria that we mentioned before.

**[00:33:44 – 00:33:46]** And the intuition here, I think, is pretty straightforward.

**[00:33:46 – 00:33:51]** It's, hey, if we make sure that the student is only exposed

**[00:33:51 – 00:33:53]** to the responses from the teacher that are correct,

**[00:33:54 – 00:33:55]** our student will be better off.

**[00:33:56 – 00:34:00]** But, this is actually an example where rejection

**[00:34:00 – 00:34:03]** sampling didn't actually help.

**[00:34:03 – 00:34:05]** You can see here the blue bars are where we kept all responses.

**[00:34:05 – 00:34:10]** We see high accuracy, and the green bars are where we

**[00:34:10 – 00:34:11]** employed rejection sampling.

**[00:34:11 – 00:34:14]** We cut out the incorrect responses.

**[00:34:14 – 00:34:15]** We actually get lower accuracy.

**[00:34:15 – 00:34:17]** So we think what's happening here is that there may be

**[00:34:17 – 00:34:20]** more useful information, even in the reasoning traces of

**[00:34:20 – 00:34:24]** responses from our teacher that were incorrect responses.

**[00:34:24 – 00:34:27]** So again, just further emphasizing that as a data scientist,

**[00:34:27 – 00:34:30]** do these empirical studies.

**[00:34:30 – 00:34:32]** The techniques that you use won't necessarily always

**[00:34:32 – 00:34:34]** conform to intuition.

**[00:34:34 – 00:34:36]** All right, so we've talked about data prep.

**[00:34:36 – 00:34:39]** The next step is to actually do the training, right?

**[00:34:39 – 00:34:41]** So we generated tasks.

**[00:34:41 – 00:34:42]** We passed them to our teacher.

**[00:34:42 – 00:34:44]** We generated responses.

**[00:34:44 – 00:34:47]** That comprises our training data set.

**[00:34:48 – 00:34:52]** For the actual training work, our team turned to NeMo RL

**[00:34:52 – 00:34:55]** and the NeMo framework.

**[00:34:55 – 00:34:59]** We found that it's a really powerful framework that

**[00:34:59 – 00:35:00]** enables easy scaling.

**[00:35:00 – 00:35:02]** It's also highly flexible.

**[00:35:02 – 00:35:04]** Our team is very grateful to our collaboration that

**[00:35:04 – 00:35:07]** we had with NVIDIA in working on things like this.

**[00:35:07 – 00:35:13]** They helped us implement the TAO2 benchmark inside NeMo Gym.

**[00:35:13 – 00:35:17]** And in fact, all of the recipes that are available inside

**[00:35:17 – 00:35:20]** Vertex Training Cluster are actually based on NeMo RL

**[00:35:21 – 00:35:24]** because of those strengths.

**[00:35:25 – 00:35:28]** Underneath the framework layer where we used NeMo RL to train our

**[00:35:28 – 00:35:31]** model, we have the infrastructure.

**[00:35:31 – 00:35:35]** To run all of these experiments, our data science team used between

**[00:35:35 – 00:35:38]** 16 and 64 H100s, H200s, and B200s.

**[00:35:40 – 00:35:48]** And we've put on top of this cluster a Vertex Training Cluster.

**[00:35:48 – 00:35:51]** And the reason we did this is because our data scientists

**[00:35:51 – 00:35:54]** really didn't want to think about orchestration management.

**[00:35:54 – 00:35:56]** They really wanted to focus on data science.

**[00:35:56 – 00:36:00]** And what Vertex Training Cluster offers is a turnkey

**[00:36:00 – 00:36:02]** cluster solution.

**[00:36:02 – 00:36:05]** There's a handful of features that help customers very

**[00:36:05 – 00:36:07]** quickly get started with their training workflows.

**[00:36:07 – 00:36:11]** In particular, we have automated cluster setup and creation.

**[00:36:11 – 00:36:16]** You as a customer can go from GPU reservation to

**[00:36:16 – 00:36:18]** a production training environment in about 20 minutes.

**[00:36:18 – 00:36:21]** We handle all the compute, networking, and storage

**[00:36:21 – 00:36:24]** configuration on your behalf.

**[00:36:24 – 00:36:26]** We also, on the infrastructure layer, we also help with

**[00:36:27 – 00:36:28]** hardware resiliency.

**[00:36:28 – 00:36:32]** So Vertex Training Cluster has a resiliency system that is

**[00:36:32 – 00:36:36]** monitoring your cluster for errors, and it's going to automatically

**[00:36:36 – 00:36:38]** remediate any interruptions to the training that might occur.

**[00:36:38 – 00:36:43]** So you hit a GPU failure or a GPU error, our system will detect it,

**[00:36:43 – 00:36:47]** remediate it, improve the uptime and good put of your training work.

**[00:36:47 – 00:36:50]** And then finally, another value prop of Vertex Training

**[00:36:50 – 00:36:54]** Cluster are pre-configured data science recipes and workflows.

**[00:36:54 – 00:36:58]** I'm talking about one of them today, this agentic

**[00:36:58 – 00:37:00]** customization pipeline.

**[00:37:00 – 00:37:03]** But we have a number of other pre- and post-training recipes available

**[00:37:03 – 00:37:08]** that you as a VTC customer can pick up and achieve state-of-the-art

**[00:37:08 – 00:37:13]** performance on GCP infrastructure just right out of the box.

**[00:37:13 – 00:37:15]** Alright, so wrapping up and looking at results with the

**[00:37:15 – 00:37:18]** last minute I have here.

**[00:37:18 – 00:37:21]** Distillation yielded very significant improvement with

**[00:37:21 – 00:37:24]** our model on this TAU-2 benchmark.

**[00:37:24 – 00:37:27]** So, here we're doing a comparison with the green bars.

**[00:37:27 – 00:37:33]** This is a post-train QEN model that is of equal size to our

**[00:37:33 – 00:37:37]** blue bars, which are the same models that went through this

**[00:37:37 – 00:37:39]** agentic customization pipeline.

**[00:37:39 – 00:37:41]** And you can see when it comes to tool usage, we improved

**[00:37:41 – 00:37:48]** performance by almost 30 points, so a very powerful technique here.

**[00:37:48 – 00:37:50]** And I want to give a quick shout out to some of the

**[00:37:50 – 00:37:54]** customers who are using some of our recipes today.

**[00:37:54 – 00:37:58]** Customer AI Singapore used Vertex Training Cluster infrastructure

**[00:37:58 – 00:38:05]** in some of our recipes to train the recent CLion foundational model.

**[00:38:05 – 00:38:08]** They were able to improve their throughput by 30% by

**[00:38:08 – 00:38:11]** using some of these NeMo recipes.

**[00:38:11 – 00:38:14]** We also have enterprise customers such as CrowdStrike who rely on

**[00:38:14 – 00:38:20]** VTC for their infrastructure needs and their training infrastructure.

**[00:38:20 – 00:38:24]** And then finally, just some quick takeaways from this pipeline

**[00:38:24 – 00:38:26]** that our data science team found.

**[00:38:26 – 00:38:28]** Distillation is a very powerful technique.

**[00:38:28 – 00:38:30]** It works well.

**[00:38:30 – 00:38:33]** We found in the data prep stage that for complex tasks,

**[00:38:33 – 00:38:38]** it looks like a large teacher model is also your best choice.

**[00:38:38 – 00:38:40]** We found that rollouts improved model accuracy pretty

**[00:38:40 – 00:38:44]** significantly, but in our use case, rejection sampling did not.

**[00:38:44 – 00:38:47]** For those three points, again, it's important that as a data

**[00:38:47 – 00:38:50]** scientist, do these empirical studies to make sure that

**[00:38:50 – 00:38:53]** the decisions you're making are going to yield the best accuracy.

**[00:38:53 – 00:38:56]** And then finally, just emphasizing the importance

**[00:38:56 – 00:38:58]** of reliable infrastructure.

**[00:38:58 – 00:39:02]** Help your data scientists just focus on differentiated science and

**[00:39:02 – 00:39:05]** not on infrastructure management.

**[00:39:06 – 00:39:09]** Quick shout out to the NeMo team, our collaboration was very helpful.

**[00:39:09 – 00:39:11]** Thank you, folks.

**[00:39:11 – 00:39:14]** And if you'd like to learn more, a couple of QR codes here.

**[00:39:14 – 00:39:17]** First one relates to Vertex AI Training Cluster, some

**[00:39:17 – 00:39:19]** of our documentation if you want to try this out.

**[00:39:19 – 00:39:22]** And then the second one relates to a blog post, which provides a

**[00:39:22 – 00:39:26]** lot more detailed information about this presentation that I just gave.

**[00:39:26 – 00:39:28]** And with that, that's all I have.

**[00:39:28 – 00:39:30]** Thank you, folks. Really appreciate your time.