# Crack the Code: Enable AI Assistants for Massive C++ Codebases

## Abstract

Most AI code assistants struggle with large, complex codebases because they lack reliable project-wide context, leading to shallow or incorrect suggestions. This talk presents a technical approach to improving AI code generation by combining NVIDIA NeMo Retriever NIM with the Model Context Protocol (MCP). We show how long-context embedding models from NeMo Retriever, paired with AST-based chunking that preserves logical code boundaries, enable higher-quality retrieval across massive C++ projects. By exposing this retriever as an MCP server, AI assistants can dynamically pull in relevant project-specific context, such as engine APIs, architectural patterns, and recent changes, at generation time. The result is more accurate, context-aware AI code generation that aligns with a project’s structure and conventions.

## AI Summary

- The presentation discussed the challenges of using off-the-shelf AI coding assistants for developing AAA games, which often involve large, complex C++ codebases with millions of lines of code.
- The team evaluated three options: doing nothing, providing better context to existing models, and fine-tuning a custom model. They chose to provide better context due to its feasibility and cost-effectiveness.
- The solution involved abstract syntax tree (AST) parsing to create context-aware chunks of code, which were then embedded using NeMo Retriever and stored in a Milvus vector database for hybrid search.
- The implementation included a Model Context Protocol (MCP) server for integrating with various coding agents and enforcing access controls, ensuring that developers could use the solution securely.
- The pilot project with a game development team showed significant improvements in developer productivity and sentiment, even with some delays in data freshness.
- The team planned to open-source core parts of the solution to help other organizations adopt similar context-aware AI coding assistance.

## Transcript

**[00:00:04 – 00:00:06]** Thanks, Paul.

**[00:00:09 – 00:00:11]** So after that announcement about the reception, I don't

**[00:00:11 – 00:00:16]** know why you guys are all still here, but let's begin.

**[00:00:16 – 00:00:24]** Actually, before we do that, just a quick, hold on a second, there's

**[00:00:24 – 00:00:28]** no talking notes in here, but okay.

**[00:00:28 – 00:00:31]** Can I see a show of hands how many people here work in the game

**[00:00:32 – 00:00:36]** industry currently or previously worked in the game industry?

**[00:00:36 – 00:00:39]** Couple of you, a few of you there, okay.

**[00:00:39 – 00:00:41]** Well, so...

**[00:00:43 – 00:00:46]** So making games is actually really hard for those who

**[00:00:46 – 00:00:49]** know, but for the others, it's actually really, really hard.

**[00:00:49 – 00:00:51]** It's very difficult.

**[00:00:51 – 00:00:52]** It takes a long time, right?

**[00:00:52 – 00:00:58]** It takes three, five years, sometimes seven, ten, twelve.

**[00:00:58 – 00:01:00]** It's hundreds and thousands of developers, right,

**[00:01:01 – 00:01:03]** working on this game.

**[00:01:03 – 00:01:07]** When the world has changed with all the streaming services,

**[00:01:07 – 00:01:12]** with social media, with all these mobile games, with ads

**[00:01:12 – 00:01:16]** that little kids seem to gravitate towards, there's so many different

**[00:01:16 – 00:01:18]** things that people could spend their time on, right?

**[00:01:18 – 00:01:23]** So making AAA games, it's so expensive and it's so hard

**[00:01:23 – 00:01:25]** that it really makes...

**[00:01:26 – 00:01:28]** We're not just competing with other games, right?

**[00:01:28 – 00:01:32]** We're actually competing with all forms of entertainment

**[00:01:32 – 00:01:34]** that's easily accessible today.

**[00:01:34 – 00:01:39]** So, GenAI offers a way for us to build better games faster,

**[00:01:39 – 00:01:47]** and which, you know, it really helps with lowering our risk.

**[00:01:47 – 00:01:53]** All right, well, now you all can see my talking notes as well.

**[00:01:53 – 00:01:55]** So what do I do?

**[00:02:00 – 00:02:02]** So here's our agenda today, right?

**[00:02:02 – 00:02:04]** So we'll first talk about the problems that we're

**[00:02:04 – 00:02:06]** trying to solve for.

**[00:02:06 – 00:02:10]** We will talk about the different options that we evaluated

**[00:02:10 – 00:02:14]** and why we ultimately decided on what we decided on.

**[00:02:14 – 00:02:16]** And then James will come in and talk about the actual

**[00:02:16 – 00:02:18]** technical solution.

**[00:02:18 – 00:02:21]** And then we'll talk about the results and we'll have

**[00:02:21 – 00:02:23]** some time for Q&A.

**[00:02:24 – 00:02:30]** So last summer, we had an internal study

**[00:02:30 – 00:02:34]** And what we found is that with AI coding assistance, it worked really

**[00:02:34 – 00:02:37]** well for game-adjacent use cases.

**[00:02:37 – 00:02:40]** And what I mean by game-adjacent use cases, we're talking about

**[00:02:40 – 00:02:44]** like all the tooling that helps support creating games, right?

**[00:02:44 – 00:02:51]** But to actually help with C++ core game code bases, it doesn't work.

**[00:02:51 – 00:02:55]** Like our study shows that like maybe at best a 2%

**[00:02:55 – 00:02:57]** productivity uplift, right?

**[00:02:57 – 00:03:00]** Which is not great.

**[00:03:01 – 00:03:05]** So why kind of the off-the-shelf AI co-assistance doesn't work

**[00:03:05 – 00:03:07]** for game development?

**[00:03:07 – 00:03:11]** First of all, off-the-shelf coding assistants, they work really

**[00:03:11 – 00:03:12]** well for small projects, right?

**[00:03:12 – 00:03:15]** Projects are maybe thousands, a couple thousand lines of

**[00:03:15 – 00:03:17]** codes, things like that.

**[00:03:17 – 00:03:20]** It works really well on common languages like us.

**[00:03:20 – 00:03:23]** Python, JavaScript, things like that.

**[00:03:24 – 00:03:25]** And it's really good for prototyping.

**[00:03:25 – 00:03:28]** So anytime you need to write new codes, it works really

**[00:03:28 – 00:03:32]** well, like it doesn't need to know what you're trying to build.

**[00:03:32 – 00:03:36]** It doesn't need kind of legacy context.

**[00:03:36 – 00:03:39]** But it really struggles with the core game code bases that

**[00:03:39 – 00:03:40]** we have at EA, right?

**[00:03:40 – 00:03:45]** So many of our games, they've been around for a very long time.

**[00:03:45 – 00:03:49]** You know, they're C++ code bases.

**[00:03:49 – 00:03:53]** We're talking about double-digit million lines of code versus

**[00:03:53 – 00:03:55]** thousands lines of code.

**[00:03:55 – 00:03:58]** We have, we use a proprietary game engine.

**[00:03:58 – 00:04:01]** And then the size, you know, like I said, double-digit

**[00:04:01 – 00:04:03]** million lines of code, right?

**[00:04:03 – 00:04:06]** And these games were built

**[00:04:06 – 00:04:09]** Upon decades of knowledge, right, so because building

**[00:04:10 – 00:04:11]** a game is hard, right?

**[00:04:11 – 00:04:15]** It takes 3, 5, 7, 10 years of time to build these games.

**[00:04:15 – 00:04:18]** So some of these codes been around for a very long time.

**[00:04:18 – 00:04:22]** And then with game development, what's different from kind of

**[00:04:22 – 00:04:26]** your general software development is that game development requires

**[00:04:26 – 00:04:30]** nuance optimization, memory optimization, right?

**[00:04:30 – 00:04:33]** So like in order for the game to run very smoothly, like

**[00:04:33 – 00:04:35]** we need to do a lot of extra work.

**[00:04:35 – 00:04:38]** Right, unlike, you know, just clicking a couple of dropdowns and

**[00:04:38 – 00:04:41]** clicking a button here and there.

**[00:04:42 – 00:04:48]** So we had three options, first one being doing nothing.

**[00:04:48 – 00:04:51]** Maybe at some point in the future, the public models

**[00:04:51 – 00:04:53]** are going to be so much better.

**[00:04:53 – 00:04:55]** We actually don't really need to do anything.

**[00:04:55 – 00:04:58]** We just focus on all the game-adjacent use cases,

**[00:04:58 – 00:05:01]** and they'll be fine.

**[00:05:01 – 00:05:05]** And then the second option is, wow, now they're all

**[00:05:05 – 00:05:07]** showing up as 1, 1, 1.

**[00:05:08 – 00:05:10]** But the second option is, you know, we could provide the

**[00:05:10 – 00:05:12]** model with better context, right?

**[00:05:12 – 00:05:16]** So like, you know, if we actually tell the model with the right

**[00:05:16 – 00:05:21]** context, maybe, just maybe the model itself, it's good enough

**[00:05:21 – 00:05:23]** to be able to figure things out.

**[00:05:23 – 00:05:26]** And then lastly, we could fine tune our own model.

**[00:05:26 – 00:05:29]** But that's super expensive.

**[00:05:29 – 00:05:33]** We ultimately landed on...

**[00:05:33 – 00:05:37]** Option number two, which is the second one on the last slide.

**[00:05:37 – 00:05:44]** And the reason is because what we ultimately decided

**[00:05:44 – 00:05:48]** on is that, well, with this option, with just providing the

**[00:05:48 – 00:05:53]** model with better context, we can easily build a proof of concept.

**[00:05:53 – 00:06:00]** So in our project together, it only took a few weeks to actually

**[00:06:00 – 00:06:03]** get it running and functional.

**[00:06:03 – 00:06:06]** And it's relatively inexpensive to run.

**[00:06:06 – 00:06:12]** So it's basically just creating an index, which James is

**[00:06:12 – 00:06:14]** going to talk about.

**[00:06:14 – 00:06:17]** And once we have the index, then it's just basically just

**[00:06:17 – 00:06:18]** a service, right?

**[00:06:18 – 00:06:24]** So you will have developers asking the index, retrieve

**[00:06:24 – 00:06:28]** some information, and be able to use that and feed it into,

**[00:06:28 – 00:06:31]** you know, your LLM.

**[00:06:33 – 00:06:36]** But wait, why are we building all these things?

**[00:06:37 – 00:06:40]** Like, shouldn't there be something out there, right?

**[00:06:40 – 00:06:43]** And it's true. So the off-the-shelf coding assistants, they typically

**[00:06:43 – 00:06:45]** already have a local index.

**[00:06:46 – 00:06:47]** They have a remote index.

**[00:06:47 – 00:06:51]** But if you actually look at the numbers on the slide, what they

**[00:06:51 – 00:06:55]** offer is significantly smaller than the typical use case, the typical

**[00:06:55 – 00:06:59]** game code basis that we have.

**[00:07:00 – 00:07:03]** And for that reason, we decided to build.

**[00:07:03 – 00:07:07]** And I will pass this to James, who will talk about the

**[00:07:07 – 00:07:10]** Actual nuts and bolts of the solution.

**[00:07:10 – 00:07:12]** Thanks, Xander.

**[00:07:12 – 00:07:15]** All right. Before I advance the slides, it's helpful to

**[00:07:15 – 00:07:19]** zoom out and talk a little bit about what an AI coding agent

**[00:07:19 – 00:07:24]** is going to do on a massive code base without what we built, which

**[00:07:24 – 00:07:27]** is if you ask it to do something, it's going to use command line

**[00:07:27 – 00:07:29]** tools to do exact string matches.

**[00:07:29 – 00:07:31]** It'll call the find function.

**[00:07:31 – 00:07:35]** It'll call cat and grep, and it'll try to find a foothold

**[00:07:35 – 00:07:38]** somewhere in the code base of relevant code, and then

**[00:07:38 – 00:07:39]** it'll begin navigating from there.

**[00:07:39 – 00:07:43]** So it'll look at files that that file imports.

**[00:07:43 – 00:07:47]** And you can imagine with an LLM with a reasonably high

**[00:07:47 – 00:07:51]** temperature, it's a very stochastic process as it tries to navigate

**[00:07:51 – 00:07:56]** the file tree and figure out what all it needs to modify to create.

**[00:07:56 – 00:07:58]** Whatever you're asking for.

**[00:07:58 – 00:08:03]** So it's hard for it to find all the context it needs.

**[00:08:04 – 00:08:06]** Our solution, which will be a

**[00:08:06 – 00:08:08]** Illuminated a little bit further on the next slide

**[00:08:08 – 00:08:14]** is to basically do abstract syntax tree parsing of all the code.

**[00:08:14 – 00:08:17]** This is built with TreeSitter, so it supports a huge range

**[00:08:17 – 00:08:19]** of coding languages.

**[00:08:19 – 00:08:22]** And then chunk, basically create a RAG engine from there.

**[00:08:22 – 00:08:26]** So we're using abstract syntax tree parsing so we make sure

**[00:08:26 – 00:08:32]** not to split functions or classes where they should not be split.

**[00:08:32 – 00:08:36]** We create our initial chunks, and then we're using a NeMo

**[00:08:36 – 00:08:41]** Retriever NIM dense embedding model to create these embeddings.

**[00:08:41 – 00:08:42]** That's step two there.

**[00:08:42 – 00:08:45]** We're actually creating sparse embeddings and dense embeddings.

**[00:08:45 – 00:08:47]** For the dense embeddings, we're using that model that's

**[00:08:47 – 00:08:49]** listed at the very bottom there.

**[00:08:49 – 00:08:55]** The Llama 3.2 Envy Embed QA 1 billion version 2 model.

**[00:08:55 – 00:08:59]** And for the sparse vectors, we've explored the sparse

**[00:08:59 – 00:09:01]** embedding neural networks, TF-IDF.

**[00:09:01 – 00:09:06]** We've tested the built-in BM25 implementation in Milvus.

**[00:09:06 – 00:09:08]** There's lots of ways to do this.

**[00:09:08 – 00:09:11]** The key for step two is that we're doing hybrid search.

**[00:09:11 – 00:09:15]** So we're doing lexical and semantic search side by side.

**[00:09:15 – 00:09:19]** Ultimately, all these vectors get put into a Milvus Vector Database,

**[00:09:19 – 00:09:23]** where we create an index, and then we can ask questions in real-time.

**[00:09:23 – 00:09:27]** Rather than having to navigate the file tree and search for what's

**[00:09:27 – 00:09:31]** relevant, we just ask a question and get back several thousand

**[00:09:31 – 00:09:35]** lines of code in real-time.

**[00:09:36 – 00:09:40]** All right, so the last slide was the full embedding workflow.

**[00:09:40 – 00:09:43]** I forgot to mention on the last slide that we added

**[00:09:43 – 00:09:44]** incremental indexing.

**[00:09:44 – 00:09:47]** So after you initially do a full load of your large

**[00:09:47 – 00:09:50]** code base, you can do incremental indexes based on changes.

**[00:09:50 – 00:09:53]** So as you make changes and basically commit your changes,

**[00:09:53 – 00:09:56]** we can re-index and update the vector database using

**[00:09:56 – 00:09:59]** just those files that were changed.

**[00:09:59 – 00:10:03]** This is the full Retriever workflow, so I'll talk through

**[00:10:03 – 00:10:07]** the blue line first until we hit the Vector Database up there in

**[00:10:07 – 00:10:11]** the top right, and then I'll talk back to the response that the agent

**[00:10:11 – 00:10:15]** receives following that green line.

**[00:10:15 – 00:10:19]** So we can ask, in this case the agent can ask, a question about

**[00:10:19 – 00:10:21]** the code base in plain language.

**[00:10:21 – 00:10:24]** We don't have to use the find function on the command line

**[00:10:24 – 00:10:27]** and search for an exact string match for something.

**[00:10:27 – 00:10:31]** So we can ask, where's the code for the kickoff meter?

**[00:10:31 – 00:10:33]** That code goes through an MCP server.

**[00:10:33 – 00:10:37]** We use MCP as a way to expose this functionality to any agent.

**[00:10:37 – 00:10:39]** So we have GitHub Copilot up here on the slide, but

**[00:10:39 – 00:10:40]** this would work with Cursor.

**[00:10:41 – 00:10:43]** This would work with Cloud Code.

**[00:10:43 – 00:10:47]** This would work with a custom coding agent built in Langraph.

**[00:10:48 – 00:10:50]** So we ask an initial question, where's the

**[00:10:50 – 00:10:51]** code for the kickoff meter?

**[00:10:51 – 00:10:58]** That passes through the API using Azure Entra ID, single sign-on

**[00:10:58 – 00:11:02]** to control our rule-based access control to make sure that that

**[00:11:02 – 00:11:07]** person actually has access to ask questions about that code base and

**[00:11:07 – 00:11:12]** filter out parts of the code base that they may not have access to.

**[00:11:13 – 00:11:17]** Assuming they do have access, we take that request and create

**[00:11:17 – 00:11:20]** the sparse and dense vectors.

**[00:11:20 – 00:11:24]** So we call our embedding model, our NIMO Retriever NIM.

**[00:11:24 – 00:11:29]** And we also use our TF-IDF encoder to create the sparse

**[00:11:29 – 00:11:30]** and dense vectors.

**[00:11:30 – 00:11:34]** And those ultimately are sent to the Milvus database to

**[00:11:34 – 00:11:36]** do our hybrid search.

**[00:11:36 – 00:11:40]** From there, we're gonna get two sets of results back, our lexical

**[00:11:40 – 00:11:43]** matches and our semantic matches, and those are at the bottom right.

**[00:11:43 – 00:11:46]** So we'll have these two chunks of basically big segments

**[00:11:46 – 00:11:50]** of code that correspond to the question that was initially asked.

**[00:11:50 – 00:11:52]** Those will go through a re-ranking.

**[00:11:52 – 00:11:54]** This could be a dense re-ranking model.

**[00:11:55 – 00:11:58]** We also have NIM or Retriever NIMs for dense re-ranking.

**[00:11:58 – 00:12:00]** It's also gonna be like reciprocal rank fusion that's

**[00:12:01 – 00:12:02]** built into Milvus.

**[00:12:02 – 00:12:04]** But some re-ranking step, and then we'll ultimately

**[00:12:04 – 00:12:07]** take the top K relevant results.

**[00:12:07 – 00:12:10]** The agent actually can control this top K parameter.

**[00:12:10 – 00:12:16]** So it can ask for five or 10 or 15 code chunks back from the database.

**[00:12:16 – 00:12:20]** All right, this is the high level architecture and we'll zoom in

**[00:12:20 – 00:12:23]** in the following slides on little kind of sub components of this and

**[00:12:24 – 00:12:27]** why we made the decisions we did.

**[00:12:29 – 00:12:35]** All right, first off, MCP, or Model Context Protocol.

**[00:12:35 – 00:12:39]** So we selected MCP as a way to centralize and make this

**[00:12:40 – 00:12:43]** code search functionality portable across agents, and

**[00:12:43 – 00:12:48]** also power things like single sign-on authentication, other

**[00:12:48 – 00:12:52]** tools that we needed to be able to enforce as part of

**[00:12:52 – 00:12:56]** rolling this out to a larger team.

**[00:12:57 – 00:12:59]** This gives us like rapid extensibility.

**[00:12:59 – 00:13:03]** We have a single place where we can modify and redeploy and

**[00:13:03 – 00:13:05]** improve this capability over time.

**[00:13:05 – 00:13:08]** And people can connect to it using whatever agent they

**[00:13:08 – 00:13:11]** want or build their own agents.

**[00:13:17 – 00:13:21]** Next up is that Abstract Syntax Tree Aware Parsing.

**[00:13:21 – 00:13:25]** So we have an example in the diagram over there on the right

**[00:13:25 – 00:13:29]** that we can't do what we would do with just plain text normally

**[00:13:29 – 00:13:33]** with RAG, where we set like a...

**[00:13:33 – 00:13:37]** A chunk size and a stride and we just arbitrarily slice through

**[00:13:37 – 00:13:39]** the text doesn't work with code.

**[00:13:39 – 00:13:44]** You end up cutting functions in half and you really need

**[00:13:44 – 00:13:47]** the full function, the full class intact to be able to

**[00:13:47 – 00:13:52]** see input and outputs, especially with data types and all the

**[00:13:52 – 00:13:56]** other complexity that C++ can add.

**[00:14:00 – 00:14:04]** And the other thing that using the NeMo Retriever NIM helped enable is

**[00:14:04 – 00:14:07]** very long context length, or very long embedding length, I'm sorry.

**[00:14:08 – 00:14:10]** And this gives us a richer representation, but it can also

**[00:14:10 – 00:14:14]** add some other challenges, which we'll talk about in the next slide.

**[00:14:14 – 00:14:19]** So the embedding NIM that we're using, that LAMA embed

**[00:14:19 – 00:14:22]** QA, or LAMA 3.2 embed QA model,

**[00:14:22 – 00:14:26]** It has very long context length, so it can accept inputs as long

**[00:14:26 – 00:14:32]** as 8,192 tokens worth of code, and a long embedding length, so we get

**[00:14:32 – 00:14:37]** back 2,048 as our embedding length.

**[00:14:37 – 00:14:40]** And then again, using that two-stage retrieval, so we

**[00:14:40 – 00:14:43]** do the hybrid search and then our re-ranking, but that two-stage

**[00:14:43 – 00:14:48]** retrieval helps really balance cost and quality, so we can

**[00:14:48 – 00:14:53]** get very good results in a scalable, fast way.

**[00:14:56 – 00:15:00]** All right, one of the challenges potentially of using an embedding

**[00:15:00 – 00:15:05]** model that provides a much longer context length is it

**[00:15:05 – 00:15:07]** can slow down vector search.

**[00:15:07 – 00:15:11]** It can slow down index building for your ANN algorithms,

**[00:15:11 – 00:15:14]** your approximate nearest neighbor algorithms.

**[00:15:14 – 00:15:18]** And it also increased your size on disk quite a bit.

**[00:15:18 – 00:15:22]** That's not something we're afraid of because NVIDIA has a

**[00:15:22 – 00:15:25]** GPU accelerated vector search solution called QVS.

**[00:15:25 – 00:15:29]** It also supports things like quantization, so if you want to

**[00:15:29 – 00:15:33]** change from float values to integer values on your long embeddings,

**[00:15:33 – 00:15:36]** all that stuff is built into QVS.

**[00:15:36 – 00:15:40]** But we can also build those indexes and actually do vector search

**[00:15:40 – 00:15:46]** at very, very high throughputs with long embedding lengths all on GPU.

**[00:15:54 – 00:15:58]** So before we talk about the results, we should talk about

**[00:15:58 – 00:16:02]** how we're actually measuring.

**[00:16:02 – 00:16:08]** So our testing strategy, first, we have to test this solution

**[00:16:08 – 00:16:10]** with our game developers, right?

**[00:16:10 – 00:16:15]** So we selected a game team as our Lighthouse partner.

**[00:16:15 – 00:16:18]** And for phase one, what they did was they identified about

**[00:16:18 – 00:16:20]** 30 use cases, right?

**[00:16:20 – 00:16:25]** And each one of those use cases, they were basically just comparing

**[00:16:25 – 00:16:29]** with and without our solution, let's call it context foundation.

**[00:16:29 – 00:16:34]** And based on the results of these testing.

**[00:16:34 – 00:16:37]** We then fine-tune our indexes and our instructions, right?

**[00:16:37 – 00:16:41]** So we're actually excluding certain type of files, we're adding

**[00:16:41 – 00:16:45]** different instructions, so that we can fine-tune it in a way that

**[00:16:45 – 00:16:49]** it optimizes the retrieval results.

**[00:16:49 – 00:16:53]** And then for phase two, what we did was we actually did A-B testing.

**[00:16:53 – 00:16:57]** So we had this pilot group within our game team.

**[00:16:57 – 00:17:02]** They spent the first sprints, which is three weeks, without

**[00:17:02 – 00:17:05]** any coding assistant, without our context foundation, our

**[00:17:05 – 00:17:07]** custom rack solution.

**[00:17:07 – 00:17:12]** And then the second week, they had a coding assistant along with

**[00:17:12 – 00:17:17]** the context foundation, and then we measured them against the velocity,

**[00:17:17 – 00:17:20]** how many hours they planned, and then how many hours did

**[00:17:20 – 00:17:26]** they actually complete the work, and the changes in line of code

**[00:17:26 – 00:17:29]** and all sorts of different things.

**[00:17:30 – 00:17:33]** And the result is that, first, what's interesting

**[00:17:33 – 00:17:35]** is a sentiment shift.

**[00:17:35 – 00:17:39]** So before we started the first...

**[00:17:39 – 00:17:43]** Control sprints, where there were no code assists, there

**[00:17:43 – 00:17:45]** were no context foundation.

**[00:17:45 – 00:17:47]** People were skeptical, like they're thinking, I don't

**[00:17:47 – 00:17:51]** have my talking notes here, so I'm just pulling this from my memory.

**[00:17:51 – 00:17:55]** I think 40% of the users, or of the participant, they

**[00:17:55 – 00:17:59]** said they don't believe there will be any efficiency gain, right?

**[00:17:59 – 00:18:01]** And then,

**[00:18:01 – 00:18:05]** But what happened was after the second sprint, we sent

**[00:18:05 – 00:18:08]** out a survey, they came back and they're like, oh, this is

**[00:18:08 – 00:18:10]** great, don't take this away, right?

**[00:18:10 – 00:18:14]** And then some other people who are moving on to a different project,

**[00:18:14 – 00:18:17]** they contact us and say, hey, make sure you give MySpot to somebody

**[00:18:18 – 00:18:19]** else who can use it, right?

**[00:18:19 – 00:18:23]** So like they really like that shift was dramatic.

**[00:18:23 – 00:18:28]** The second thing is our solution, it's really accurate

**[00:18:28 – 00:18:30]** and it's very fast.

**[00:18:30 – 00:18:37]** So we had an example where we asked, by the way, our

**[00:18:37 – 00:18:41]** game team pilot Lighthouse partner is American football, so it's our

**[00:18:41 – 00:18:44]** Madden and college football team.

**[00:18:44 – 00:18:47]** And one of the example that they share with us is that

**[00:18:47 – 00:18:51]** they asked the coding assistant,

**[00:18:51 – 00:18:56]** Weather affects the accuracy and the power of kicking

**[00:18:56 – 00:18:58]** a football, right?

**[00:18:58 – 00:19:03]** And then so the first time without the solution, it took

**[00:19:03 – 00:19:06]** 20 minutes and then it timed out.

**[00:19:06 – 00:19:07]** It wasn't able to answer anything.

**[00:19:07 – 00:19:10]** It didn't even try to, like, hallucinate and

**[00:19:10 – 00:19:11]** create something, right?

**[00:19:11 – 00:19:13]** It just timed out.

**[00:19:13 – 00:19:17]** With our solution, within a few seconds, it returned,

**[00:19:17 – 00:19:20]** I think, about 10 files.

**[00:19:20 – 00:19:23]** And it says, these are all the relevant files.

**[00:19:23 – 00:19:27]** And we had a human verify that eight of the 10 files

**[00:19:27 – 00:19:31]** were actually correct, two were irrelevant, but it doesn't matter.

**[00:19:31 – 00:19:36]** The LLM is actually so powerful and so good that with the eight

**[00:19:36 – 00:19:40]** correct files, we're still able to actually look up and find the

**[00:19:40 – 00:19:46]** right information, and therefore able to complete the task.

**[00:19:46 – 00:19:51]** And then during the evaluation period, that's phase two,

**[00:19:51 – 00:19:55]** actually phase one as well, the team that were using the tool, they

**[00:19:55 – 00:19:59]** kept thinking about, oh, I can use it for this, I can use it for that.

**[00:19:59 – 00:20:02]** A couple of examples are

**[00:20:02 – 00:20:06]** And this one's interesting, because we hear so much about

**[00:20:06 – 00:20:09]** people saying that AI is going to be replacing people's job

**[00:20:09 – 00:20:11]** and this and that.

**[00:20:11 – 00:20:15]** But what we really found is that because with a custom

**[00:20:15 – 00:20:20]** rack, with a context foundation, it's able to retrieve and actually

**[00:20:20 – 00:20:23]** find information effectively.

**[00:20:23 – 00:20:26]** And so what ended up happening is it allowed us to onboard

**[00:20:26 – 00:20:29]** junior engineers and people who are new to the project

**[00:20:29 – 00:20:33]** a lot quicker because they now have something that they can ask 24-7.

**[00:20:34 – 00:20:37]** They don't have to rely on their subject matter experts

**[00:20:37 – 00:20:40]** to be available after they're done with their meetings and they

**[00:20:40 – 00:20:41]** can finally ask those questions.

**[00:20:41 – 00:20:43]** That's no longer the case, right?

**[00:20:43 – 00:20:45]** They can use this for that.

**[00:20:46 – 00:20:50]** We've also tried to use it for some kind of high-level refactoring.

**[00:20:50 – 00:20:53]** It does really well. There are a couple other use cases

**[00:20:53 – 00:20:55]** that I can't remember off the top of my head now that I don't have

**[00:20:55 – 00:20:58]** my talking notes, so bear with me.

**[00:20:58 – 00:21:02]** But basically, we keep hearing people saying that, like,

**[00:21:02 – 00:21:04]** hey, we should try it with this.

**[00:21:04 – 00:21:05]** We should try it with that.

**[00:21:06 – 00:21:10]** Another one was creating design documents or system documents based

**[00:21:10 – 00:21:13]** on design and vice versa, right?

**[00:21:13 – 00:21:15]** And lastly, we're seeing

**[00:21:15 – 00:21:22]** Real productivity uplift, which as of last week, I saw

**[00:21:22 – 00:21:25]** some of those data that I can't share with you, but they're

**[00:21:25 – 00:21:28]** actually meaningful productivity uplifts that we're seeing

**[00:21:28 – 00:21:33]** by using this custom rack solution.

**[00:21:35 – 00:21:38]** All right, well, looks like we're going a lot faster,

**[00:21:38 – 00:21:42]** but here's the key takeaways.

**[00:21:43 – 00:21:46]** Today's foundational models are very, very good.

**[00:21:46 – 00:21:48]** But context matters.

**[00:21:48 – 00:21:53]** So without context, it can only get you to a certain point, especially

**[00:21:53 – 00:21:58]** when we're building games, doing very difficult things, and having

**[00:21:58 – 00:22:01]** many, many million lines of code.

**[00:22:01 – 00:22:04]** Having the right context really matters.

**[00:22:04 – 00:22:09]** And then experimentation is also very important, Because

**[00:22:09 – 00:22:15]** The space is just evolving so fast, and if we don't experiment,

**[00:22:15 – 00:22:18]** we don't know what works and what doesn't, right?

**[00:22:18 – 00:22:22]** And then finally, know what you're measuring, know how

**[00:22:22 – 00:22:26]** you're going to be measuring, because with every experimentation,

**[00:22:26 – 00:22:29]** You want to be able to fail fast, or you want to be able

**[00:22:29 – 00:22:31]** to double down on your investment.

**[00:22:32 – 00:22:35]** And then lastly, NVIDIA, they've been great.

**[00:22:35 – 00:22:38]** They are such a great strategic partner for

**[00:22:38 – 00:22:40]** us in this journey here.

**[00:22:40 – 00:22:44]** Just wanted to thank James and Wendy, Rick, and all the

**[00:22:45 – 00:22:47]** others for having me here.

**[00:22:47 – 00:22:50]** And that's it.

**[00:23:06 – 00:23:07]** Great talk, thank you.

**[00:23:07 – 00:23:10]** I am really interested to know if you could share some

**[00:23:10 – 00:23:14]** of the metrics that you gathered, like percentage increase in

**[00:23:14 – 00:23:18]** productivity or velocity or whatever you were measuring.

**[00:23:19 – 00:23:23]** That's not something that I can share because I will get in

**[00:23:23 – 00:23:26]** trouble for sharing actual numbers.

**[00:23:26 – 00:23:30]** But I can tell you just based on the two things, right,

**[00:23:30 – 00:23:34]** like we're capturing, we're looking at velocity, we're looking at

**[00:23:34 – 00:23:40]** all those metrics, right, and we're seeing real impact and, you know.

**[00:23:41 – 00:23:43]** And I encourage you or any other teams out there to do

**[00:23:43 – 00:23:45]** the same thing, right?

**[00:23:45 – 00:23:47]** Because you can always look at line of code, you can look

**[00:23:47 – 00:23:52]** at other passive metrics that comes out from your telemetry, right?

**[00:23:52 – 00:23:56]** But that's not a good indication of what it actually translates

**[00:23:56 – 00:24:02]** to people actually doing more work, saving more time.

**[00:24:03 – 00:24:06]** You know, just writing more codes doesn't translate to

**[00:24:06 – 00:24:08]** better productivity, right?

**[00:24:08 – 00:24:09]** It means maintaining more code.

**[00:24:10 – 00:24:14]** It means that there's more chances of something going wrong.

**[00:24:14 – 00:24:18]** So I think the best way is to look at kind of how do you

**[00:24:18 – 00:24:20]** measure productivity before, right?

**[00:24:20 – 00:24:21]** That's your benchmark.

**[00:24:21 – 00:24:25]** And then you layer in all these changes and look at that.

**[00:24:26 – 00:24:29]** And then I think the other part of it that I did share

**[00:24:29 – 00:24:33]** is about kind of the sentiment Don't shift, right?

**[00:24:33 – 00:24:38]** We can force people to use code assist, but if they don't see the

**[00:24:38 – 00:24:40]** value, they're not going to use it.

**[00:24:40 – 00:24:46]** And we see a huge swing in people's opinion on why they really

**[00:24:46 – 00:24:48]** want to use it every day now.

**[00:24:49 – 00:24:52]** That's all I can share for now.

**[00:24:53 – 00:24:56]** I was super curious on, like, how you did reading and that

**[00:24:56 – 00:24:59]** sort of, like, aspect of it.

**[00:25:00 – 00:25:03]** How do you tell, like, create those things in that system?

**[00:25:03 – 00:25:07]** How do you do stuff like barfletries?

**[00:25:08 – 00:25:11]** That's something we're exploring in the future.

**[00:25:11 – 00:25:14]** Right now we kind of operate like on the branch level.

**[00:25:14 – 00:25:20]** So once something is indexed, we keep track of, I'll use

**[00:25:20 – 00:25:23]** Git terminology, because I'm more familiar with Git than perforce,

**[00:25:23 – 00:25:26]** but we use basically a Git commit and we can calculate a change log

**[00:25:26 – 00:25:29]** from like a certain commit to the next and create basically a Python

**[00:25:30 – 00:25:32]** list of every file that's changed.

**[00:25:32 – 00:25:34]** And then we just.

**[00:25:34 – 00:25:37]** Obviously, if it's a deletion, we can delete those vectors

**[00:25:37 – 00:25:37]** from the database.

**[00:25:37 – 00:25:42]** If it's an update or an addition, we'll go through and re-chunk.

**[00:25:42 – 00:25:46]** And to keep things simple, we'll just re-embed the full

**[00:25:46 – 00:25:49]** document rather than trying to isolate a specific chunk

**[00:25:49 – 00:25:52]** that changed within that document.

**[00:25:53 – 00:25:58]** If 10 people are working on the same code based on one

**[00:25:58 – 00:26:00]** change, it's like 50 files.

**[00:26:00 – 00:26:05]** How fast other 49 developers will see this change?

**[00:26:06 – 00:26:08]** It takes milliseconds.

**[00:26:08 – 00:26:11]** So you could do that on the individual branch level, like

**[00:26:11 – 00:26:14]** as part of a commit hook.

**[00:26:14 – 00:26:19]** But you get control over that.

**[00:26:19 – 00:26:22]** You can do it nightly if you want, or you can make it part

**[00:26:22 – 00:26:25]** of just a live CI CDE.

**[00:26:26 – 00:26:31]** I'll also add that from my conversation with the developers

**[00:26:31 – 00:26:34]** using the tool, they're saying that even when... Because

**[00:26:34 – 00:26:39]** in the beginning of our pilot, we're not refreshing the data

**[00:26:39 – 00:26:41]** as quickly, right?

**[00:26:41 – 00:26:46]** But even then, they're saying that even with some stale data within

**[00:26:46 – 00:26:50]** the context, within the index, it's still good enough, because

**[00:26:50 – 00:26:54]** as soon as it's able to pinpoint roughly, like, hey, these are

**[00:26:54 – 00:26:58]** the files, and then it just lets the LLM figure it out, like it will

**[00:26:58 – 00:27:03]** go crawl the right files and then find the information that you need.

**[00:27:03 – 00:27:07]** So even if there is a bit of a delay in data freshness,

**[00:27:07 – 00:27:09]** it wasn't a problem.

**[00:27:09 – 00:27:13]** You see interesting behavior with a real agent where it'll

**[00:27:13 – 00:27:17]** call the MCP search tool first and use that as a starting

**[00:27:17 – 00:27:20]** point, and then it'll fall back and still kind of hunt on its own.

**[00:27:20 – 00:27:23]** So if there's been actual files on disk that have been

**[00:27:23 – 00:27:26]** updated, it typically can see that.

**[00:27:26 – 00:27:28]** It'll find it.

**[00:27:29 – 00:27:31]** Thank you for your time.

**[00:27:32 – 00:27:33]** Sorry about that. Hi.

**[00:27:33 – 00:27:36]** First, thank you. This is really insightful.

**[00:27:36 – 00:27:40]** So can you share more insight on how do you implement the

**[00:27:41 – 00:27:47]** chunking and how you made the language aware tokenization and

**[00:27:47 – 00:27:51]** stuff, because that's one of the stuff that we more struggle with.

**[00:27:51 – 00:27:54]** Like, you chunk the stuff with the code that spills

**[00:27:54 – 00:27:59]** out the model is nonsense or it starts hallucinating.

**[00:27:59 – 00:28:02]** So how do you approach that?

**[00:28:02 – 00:28:05]** The research paper that we sort of base everything on is called

**[00:28:05 – 00:28:09]** CAST, so lowercase C, capital AST.

**[00:28:10 – 00:28:13]** And then there's a library called ASTChunk that is built on

**[00:28:13 – 00:28:16]** TreeSitter that does a lot of this.

**[00:28:16 – 00:28:21]** That's the library we use pretty heavily.

**[00:28:29 – 00:28:31]** Awesome, but also at the same time we have kind of like

**[00:28:31 – 00:28:34]** a scene called Greybeard Developers that are really interested

**[00:28:34 – 00:28:40]** in trying to apply coding agents to the codebase, but also there's some

**[00:28:40 – 00:28:41]** safety and security issues, right?

**[00:28:41 – 00:28:44]** So I'm really fascinated by the discussion and we have

**[00:28:44 – 00:28:48]** some resources to be able to get started in this kind

**[00:28:48 – 00:28:51]** of like context enrichment and this kind of more secure coding

**[00:28:51 – 00:28:55]** agent for a very specific 20, 30-year-old codebase to try to...

**[00:28:56 – 00:28:58]** Modernize and put a lot of work into.

**[00:28:58 – 00:29:02]** So I was wondering if you could kind of size the effort,

**[00:29:02 – 00:29:06]** just ballpark size the effort in a big picture way of how to get

**[00:29:06 – 00:29:10]** started, how many people it takes, what kind of skills and expertise

**[00:29:10 – 00:29:15]** to develop a team to help kickstart an effort like this, to help

**[00:29:15 – 00:29:21]** bring some of the more contextually aware capability into a company.

**[00:29:23 – 00:29:26]** A bit of good news, we are planning to open source core

**[00:29:26 – 00:29:34]** parts of this, we're starting those discussions, so look for that.

**[00:29:34 – 00:29:37]** I would say we used a lot of off-the-shelf tools.

**[00:29:37 – 00:29:41]** We used the AST chunk library that handles a

**[00:29:41 – 00:29:42]** lot of the chunking for us.

**[00:29:43 – 00:29:48]** Fast MCP is phenomenal for creating an MCP server

**[00:29:48 – 00:29:52]** pretty quickly to test.

**[00:29:52 – 00:29:55]** There's a whole bunch of really nice security authentication

**[00:29:55 – 00:29:59]** features in their latest release.

**[00:29:59 – 00:30:03]** I think you don't have to build everything from scratch.

**[00:30:03 – 00:30:05]** That's the nice thing about using the NeMo, Retriever,

**[00:30:05 – 00:30:09]** NIM is it's a Docker-run command and then you have an OpenAI

**[00:30:09 – 00:30:12]** API-compatible endpoint.

**[00:30:19 – 00:30:22]** I think for proof of concept and to show that there's value

**[00:30:22 – 00:30:25]** in it, absolutely, yeah.

**[00:30:26 – 00:30:31]** Hello, how did you keep your proprietary code private?

**[00:30:31 – 00:30:35]** And the second question, if you used any kind of on-premises

**[00:30:35 – 00:30:37]** hardware and what were the specs?

**[00:30:37 – 00:30:38]** Thank you.

**[00:30:39 – 00:30:47]** Yeah, so the indexes, they're all hosted within our EA-managed

**[00:30:47 – 00:30:50]** cloud instances through Azure.

**[00:30:50 – 00:30:51]** So they're all there.

**[00:30:51 – 00:30:57]** So it's all contained within our ecosystem.

**[00:30:57 – 00:31:00]** Sorry, I forgot, was there a second part to that question as well?

**[00:31:00 – 00:31:03]** On-prem hardware.

**[00:31:03 – 00:31:08]** So the MCP is all remote, so there's no on-prem

**[00:31:08 – 00:31:09]** hardware required.

**[00:31:09 – 00:31:11]** Is that right?

**[00:31:11 – 00:31:12]** Yeah, everything is all remote.

**[00:31:12 – 00:31:15]** I guess an important call-out is...

**[00:31:15 – 00:31:18]** As NVIDIANS, we never had any access to their code.

**[00:31:18 – 00:31:20]** I can't access their MCP server.

**[00:31:21 – 00:31:25]** We shared examples, we shared code snippets, but they built

**[00:31:25 – 00:31:27]** this all internally.

**[00:31:27 – 00:31:31]** So unless you're part of their organization, there's no way

**[00:31:31 – 00:31:34]** to access their system.

**[00:31:38 – 00:31:42]** Use things like design documents or other kind of information,

**[00:31:42 – 00:31:45]** not to only look or search for code, but also just try

**[00:31:45 – 00:31:50]** to understand it a little bit so that the developer can get kind

**[00:31:50 – 00:31:54]** of all the kind of answers to the question, like how something works.

**[00:31:54 – 00:31:57]** Yeah, that's the nice part about using something that's

**[00:31:57 – 00:32:00]** based on that TreeSitter library is it can chunk.

**[00:32:00 – 00:32:04]** We would do the AST parsing for Markdown, HTML, any sort

**[00:32:04 – 00:32:08]** of documentation file, RST files.

**[00:32:08 – 00:32:11]** It all makes it into the vector database.

**[00:32:12 – 00:32:14]** Hey, my name is Rasheek.

**[00:32:14 – 00:32:16]** I'm a grad student from NC State University.

**[00:32:16 – 00:32:17]** Thank you all for the talk.

**[00:32:17 – 00:32:20]** I just wanted to ask you all, I found the AST parsing that

**[00:32:20 – 00:32:22]** you did very interesting when you were doing chunking.

**[00:32:22 – 00:32:25]** Have you all considered applying that to maybe code that's

**[00:32:25 – 00:32:28]** been generated by a coding assistant and then verifying

**[00:32:28 – 00:32:32]** that it does a certain task or making sure it does it a certain

**[00:32:32 – 00:32:36]** way by doing similar chunking and passing it through some tests?

**[00:32:36 – 00:32:41]** Just something interesting from the AST parsing that you mentioned.

**[00:32:42 – 00:32:46]** We haven't looked into that yet.

**[00:32:46 – 00:32:49]** Catch me outside after I talk about that a little more.

**[00:32:49 – 00:32:51]** It's interesting.

**[00:32:53 – 00:32:55]** For EA's use case, they have so many phenomenal senior

**[00:32:55 – 00:32:59]** engineers that they can do their own code review and stuff.

**[00:32:59 – 00:33:06]** But that is interesting, thinking about building a AST parsing

**[00:33:06 – 00:33:09]** kind of code review agent.

**[00:33:13 – 00:33:16]** I mean, you didn't talk about in your talk, so you don't

**[00:33:16 – 00:33:18]** have to say, but the other end of the process, once you

**[00:33:18 – 00:33:22]** have all the right context, is there any learnings that

**[00:33:22 – 00:33:26]** you have about the build and test cycle that's involved

**[00:33:26 – 00:33:29]** in agentic coding that's to do with a very large code base that takes

**[00:33:29 – 00:33:31]** a long time to build and test?

**[00:33:31 – 00:33:37]** Not right now, so we actually just wrapped up our A-B testing

**[00:33:37 – 00:33:42]** a week ago, so that's why I think some of the productivity

**[00:33:42 – 00:33:45]** data, I've only seen a glimpse of the preliminary data, the

**[00:33:45 – 00:33:49]** results, so no, we haven't.

**[00:33:49 – 00:33:51]** I don't think that far ahead yet.

**[00:33:51 – 00:33:53]** I know, as I mentioned, people are thinking about the different

**[00:33:53 – 00:33:54]** use cases, right?

**[00:33:54 – 00:33:59]** Like, so like, can we use this context foundation as

**[00:33:59 – 00:34:00]** part of an agent, right?

**[00:34:00 – 00:34:07]** Like for example, one use case, I was talking to our QV

**[00:34:07 – 00:34:11]** quality verification group, right?

**[00:34:11 – 00:34:14]** And they're talking about like how we have testers,

**[00:34:14 – 00:34:16]** game testers reporting.

**[00:34:16 – 00:34:19]** Bugs, right, and say, hey, this doesn't look right, right?

**[00:34:19 – 00:34:24]** Can we use that, have an agent, and actually look at, based

**[00:34:24 – 00:34:30]** on the description of the defects, find the relevant files within the

**[00:34:30 – 00:34:35]** game and actually confirm, like, is this the correct behavior or not?

**[00:34:35 – 00:34:38]** And if it's not, then what are the possible fixes?

**[00:34:38 – 00:34:39]** So that's kind of one way.

**[00:34:39 – 00:34:42]** So there's so many different type of use cases that people

**[00:34:42 – 00:34:46]** are starting to think about now because we're finally

**[00:34:46 – 00:34:50]** able to have the right context and be able to leverage it

**[00:34:50 – 00:34:53]** in ways that we couldn't before.

**[00:34:55 – 00:35:00]** When you switch models, are you seeing any drastic difference

**[00:35:00 – 00:35:05]** in the accuracies, for instance, between Opus or any other

**[00:35:05 – 00:35:08]** open source models?

**[00:35:10 – 00:35:13]** Sorry, I didn't quite catch that.

**[00:35:13 – 00:35:17]** While you're, you know, retrying the code and using that to

**[00:35:17 – 00:35:22]** build any code, if you switch the models, like from Opus

**[00:35:22 – 00:35:26]** to GPT-4 or any models, is there any drastic difference

**[00:35:26 – 00:35:30]** in the accuracy in the code?

**[00:35:30 – 00:35:34]** Yeah, yeah, I think the, I don't have a lot of information

**[00:35:34 – 00:35:41]** to answer that question, but from maybe Bill, do you know, have you,

**[00:35:41 – 00:35:44]** you've tried the solution, right?

**[00:35:47 – 00:35:52]** Uh, yeah, we're the Lighthouse partner, um, we just, one model

**[00:35:52 – 00:35:55]** is really what we, um, stuck with.

**[00:35:55 – 00:35:58]** But, uh, because we're on limited time, um, you know, the dev team,

**[00:35:58 – 00:36:02]** this is our live dev team, and, um, American football is, like,

**[00:36:02 – 00:36:06]** in alpha right now, so we had to be very careful with the dev team's

**[00:36:06 – 00:36:10]** time, and our time, even as the senior engineers kind of helping,

**[00:36:10 – 00:36:13]** um, oversee a lot of this, so, um.

**[00:36:13 – 00:36:14]** We're just very happy.

**[00:36:14 – 00:36:17]** I can't say it enough, the results were significant.

**[00:36:17 – 00:36:20]** So it was really cool.

**[00:36:26 – 00:36:27]** Sorry, one more question.

**[00:36:27 – 00:36:30]** I was actually wondering, could this just be a generic solution

**[00:36:30 – 00:36:34]** that NVIDIA does for any type of code base, as long as the AST

**[00:36:34 – 00:36:37]** is able to recognize the code base, like the language in question?

**[00:36:38 – 00:36:41]** Why wouldn't it work for a generic, huge code base?

**[00:36:41 – 00:36:46]** Because you're chunking based on what the AST returns as,

**[00:36:46 – 00:36:49]** these are functions, these are class declarations or whatnot.

**[00:36:49 – 00:36:53]** But that could apply to any code base, not necessarily like, so

**[00:36:53 – 00:36:56]** that could be a generic solution.

**[00:36:57 – 00:37:00]** I think that's our goal, is to provide a template for

**[00:37:00 – 00:37:04]** generic solution, but as Sandak can elaborate on, there's

**[00:37:04 – 00:37:07]** a lot that goes on on the back end, like maintaining this for

**[00:37:07 – 00:37:10]** large software development teams.

**[00:37:10 – 00:37:14]** There's a lot of infrastructure that you've got to spin up.

**[00:37:14 – 00:37:19]** Yeah, I think one of the main learning that we have is that

**[00:37:20 – 00:37:24]** we knew it was, well, we've read paper that says that

**[00:37:24 – 00:37:25]** this might work, right?

**[00:37:25 – 00:37:28]** But we haven't tried it, so here's kind of an example

**[00:37:28 – 00:37:31]** of like we've actually tried it and it actually works.

**[00:37:31 – 00:37:35]** So yeah, because we went in as a hypothesis that like this

**[00:37:35 – 00:37:37]** might work, but we're not sure.

**[00:37:37 – 00:37:39]** So, but now we know.

**[00:37:39 – 00:37:45]** Yeah, I think we have time for two more questions, so.

**[00:37:45 – 00:37:48]** Thank you. Thank you for the insightful thing.

**[00:37:48 – 00:37:49]** Just my question is simple.

**[00:37:49 – 00:37:53]** So a lot of the code bases, the old code bases will have

**[00:37:53 – 00:37:54]** a lot of dead code, right?

**[00:37:54 – 00:37:57]** Like they'll be never be retrieved by the application, but if

**[00:37:57 – 00:38:00]** you chunk the entire code base, how do you make sure that

**[00:38:00 – 00:38:04]** whatever that's written is actually being used by the application or

**[00:38:04 – 00:38:06]** you know it's actually, you know...

**[00:38:06 – 00:38:08]** You've been used at the runtime of the application.

**[00:38:08 – 00:38:12]** How do you make sure that whatever that's written is

**[00:38:13 – 00:38:13]** actually good code?

**[00:38:13 – 00:38:18]** SHYAM GOLLAKOTA- Right.

**[00:38:20 – 00:38:24]** Yeah, I guess we don't because, I mean, there's a lot of

**[00:38:24 – 00:38:26]** code in there, right?

**[00:38:26 – 00:38:28]** Some of them are good, some of them are bad.

**[00:38:28 – 00:38:34]** But I think with this, there's been discussions around like using this

**[00:38:34 – 00:38:39]** to help with a lot of refactoring, right, a lot of code understanding,

**[00:38:39 – 00:38:42]** right, so that we can actually try to make sense of them all,

**[00:38:42 – 00:38:48]** maybe start to write better codes or replace some of the older ones.

**[00:38:48 – 00:38:50]** That wasn't really possible before, right?

**[00:38:50 – 00:38:55]** Like, yeah, because the person who originally wrote those code,

**[00:38:55 – 00:38:58]** they might no longer be with the company, makes it really difficult.

**[00:38:58 – 00:39:02]** So yeah, so like this by itself doesn't solve that problem,

**[00:39:02 – 00:39:06]** but it allow us to get a step closer to fixing it.

**[00:39:06 – 00:39:11]** Back to a point I tried to make earlier, this isn't just like LLM.

**[00:39:12 – 00:39:14]** Sending RAG to it, saying generate an answer using this

**[00:39:14 – 00:39:16]** context, it's an agent.

**[00:39:16 – 00:39:18]** So you get this interesting hybrid where you're like,

**[00:39:19 – 00:39:22]** hey, use this as a starting point and it'll go explore and it can

**[00:39:22 – 00:39:24]** figure out if code's dead or not.

**[00:39:24 – 00:39:28]** It'll make its own tool calls and continue to explore the code base.

**[00:39:28 – 00:39:29]** Awesome, thank you so much.

**[00:39:29 – 00:39:32]** Can we get a big round of applause for Sanda and James?