# GPU Access to Unbounded Dataset Sizes Breaks Open Document Ingestion and Search ## Abstract Vector databases with 1 trillion vectors cannot fit in memory, and VecDB ingestion and search requires the massive concurrency, memory bandwidth, and latency tolerance of GPUs. Come see new programming models, new storage server reference designs, new SSD offerings, and new customer-oriented scaled results. ## AI Summary - The presentation focused on the challenges and advancements in vector search, particularly for handling unbounded and large datasets that do not fit in memory. - The team introduced CooVS, a library for vector search and unstructured data processing, which aimed to optimize the performance of vector search algorithms across different hardware architectures. - The presentation covered the Kegra and Vamana algorithms, highlighting the stages of graph creation, reordering, pruning, and merging, which are crucial for improving the scalability and efficiency of vector search. - A new memory tier, Storage Next, was discussed, aiming to optimize for IOPS per watt and IOPS per dollar to handle large-scale vector search workloads more effectively. - The team demonstrated the performance improvements of SCADA (Scaled Accelerated Data Access) in vector search, showing significant speedups and the ability to handle datasets up to 100 billion vectors. - The presentation concluded with a discussion on the need for standardized benchmarking tools and synthetic data generation to evaluate vector search algorithms at scale. ## Transcript **[00:00:10 – 00:00:14]** We're delighted to have you here and welcome you to another session. **[00:00:14 – 00:00:18]** We are today really going to focus on being able to **[00:00:18 – 00:00:23]** break out of memory to be able to handle unbounded data **[00:00:23 – 00:00:27]** sets, especially for ingestion and search and dealing with documents. **[00:00:27 – 00:00:30]** I'm delighted to be here with Vikram and Corey and we'll **[00:00:30 – 00:00:31]** kind of trade things off. **[00:00:32 – 00:00:35]** I'd like to start first talking about the kind of legacy solutions **[00:00:35 – 00:00:40]** here where the computing agent talks to all of it finds all **[00:00:40 – 00:00:43]** of its data in memory where you're bringing chunks into **[00:00:43 – 00:00:47]** memory and working on them and doing all your accesses **[00:00:47 – 00:00:52]** inside memory And so the focus that you have on storage is really for **[00:00:52 – 00:00:56]** cheap capacity terabytes per dollar and that works great for images **[00:00:56 – 00:00:58]** KV context and streaming inputs **[00:00:59 – 00:01:02]** The vision of kind of where we wanna go with this is towards **[00:01:03 – 00:01:05]** data that absolutely just won't fit in memory. **[00:01:05 – 00:01:07]** If you look at all the trends on data, and we'll talk a **[00:01:07 – 00:01:11]** little bit about that, it simply won't fit in memory. **[00:01:11 – 00:01:14]** And so instead of bringing in big chunks, **[00:01:14 – 00:01:17]** And then throwing away most of it because you needed some **[00:01:17 – 00:01:20]** little tiny piece, you want to reach directly into storage **[00:01:20 – 00:01:22]** and to be able to access that. **[00:01:22 – 00:01:25]** And that's important for a different set of applications like **[00:01:25 – 00:01:28]** vector database, vector database indexes, and graph neural networks. **[00:01:28 – 00:01:31]** And we'll focus on those today. **[00:01:31 – 00:01:35]** So as we get ready here, we're going to kind of focus first **[00:01:35 – 00:01:38]** on deep diving into what's really going on inside Vector **[00:01:38 – 00:01:44]** Search and how you create the indexes for those, and we'll **[00:01:44 – 00:01:46]** walk through the index build steps. **[00:01:46 – 00:01:50]** We'll also talk about some technology that we have of SCADA **[00:01:50 – 00:01:53]** for Scaled Accelerated Data Access that many of you are collaborating **[00:01:53 – 00:01:56]** on, look at some performance and some technology and tools, and **[00:01:56 – 00:01:58]** then do some assessment of that. **[00:01:58 – 00:01:59]** Over to Vikram. **[00:01:59 – 00:02:03]** Thank you, CJ. So, vector search is a very important problem **[00:02:03 – 00:02:04]** in today's machine learning world. **[00:02:04 – 00:02:08]** The simplest way to understand what vector search is, it's **[00:02:08 – 00:02:11]** trying to find similar set of items comparing dense vectors, **[00:02:11 – 00:02:15]** and trying to understand what is the similarity between the set of **[00:02:15 – 00:02:18]** documents or set of corpus of the material that you're working with. **[00:02:18 – 00:02:21]** Vector search is used in many of the use cases that you're **[00:02:21 – 00:02:23]** already familiar with. **[00:02:23 – 00:02:27]** Retro-logmented generation, semantic search via Google search, **[00:02:27 – 00:02:29]** and also in recommendation systems. **[00:02:29 – 00:02:31]** So it's a very popular topic, but not very well understood, **[00:02:31 – 00:02:34]** and the challenges are not very clear either. **[00:02:34 – 00:02:36]** Vector search is not a new problem. **[00:02:36 – 00:02:39]** It has been there for several decades, but if you just look **[00:02:39 – 00:02:43]** into the last few set of years, how the vector size data sets **[00:02:43 – 00:02:46]** have been growing, it's going almost like an exponential growth. **[00:02:46 – 00:02:51]** Main reason is the scale and the velocity in which the people are **[00:02:51 – 00:02:54]** starting to use and deploy them. **[00:02:54 – 00:02:56]** And the other factor that is also happening is, thanks **[00:02:56 – 00:02:59]** to transformer models, the embeddings are no longer like a bag **[00:02:59 – 00:03:02]** of word, kind of like embeddings. **[00:03:02 – 00:03:05]** Now what we have is embeddings that are getting retrieved, **[00:03:05 – 00:03:08]** coming from transformer layers itself, and that increases **[00:03:08 – 00:03:10]** the complexity even more further. **[00:03:11 – 00:03:14]** To accelerate vector search, we have been building inside NVIDIA **[00:03:14 – 00:03:19]** something called Cuvier's, and let me welcome Corey to speak about **[00:03:19 – 00:03:22]** Cuvier's and vector search in GPU. **[00:03:22 – 00:03:24]** Awesome, thank you Vikram. **[00:03:25 – 00:03:28]** So first, a note on where GPU makes sense for vector search, because **[00:03:28 – 00:03:30]** it doesn't always make sense. **[00:03:30 – 00:03:32]** We tend to find that we have to build indexes, as I'll get into **[00:03:32 – 00:03:34]** for a moment, for vector search. **[00:03:34 – 00:03:37]** They are more like machine learning models than regular **[00:03:37 – 00:03:39]** traditional database indexes. **[00:03:39 – 00:03:42]** So as a result, we end up with some trade-offs. **[00:03:42 – 00:03:44]** Building the indexes themselves so that we can have efficient **[00:03:44 – 00:03:47]** search can be a heavy, high... **[00:03:47 – 00:03:50]** There's a throughput operation that is often better to offload to GPUs. **[00:03:50 – 00:03:53]** We can do it significantly faster than on the CPU. **[00:03:53 – 00:03:57]** Many preprocessing steps, like, for instance, dimensionality reduction, **[00:03:57 – 00:04:00]** compression, quantization, clustering that's often involved **[00:04:00 – 00:04:03]** and kind of ordering the vectors in different ways that they partition **[00:04:03 – 00:04:05]** well can be done well on the GPU. **[00:04:05 – 00:04:09]** But also online and streaming processes, for instance, **[00:04:09 – 00:04:13]** recommender systems that require extremely low latency. **[00:04:13 – 00:04:15]** GPU can do a lot more in parallel in terms of the **[00:04:15 – 00:04:18]** distance computations, being able to saturate the GPU **[00:04:18 – 00:04:20]** with many concurrent clients. **[00:04:20 – 00:04:23]** And, you know, being able to do really, really high **[00:04:23 – 00:04:24]** throughput search, right? **[00:04:24 – 00:04:26]** I've got millions of vectors I need to calculate the nearest **[00:04:26 – 00:04:28]** neighbors for very, very quickly. **[00:04:28 – 00:04:31]** This is not always an offline thing. **[00:04:31 – 00:04:36]** It can often be, but GPUs really accelerate that. **[00:04:36 – 00:04:38]** And because of these performance benefits and the fact that **[00:04:38 – 00:04:42]** we can do this on a whole range of GPUs, they can also **[00:04:42 – 00:04:46]** have a cost story there as well. **[00:04:46 – 00:04:49]** So, as you might have noticed in the keynote on Monday, we have **[00:04:49 – 00:04:52]** a library for vector search and unstructured data processing called **[00:04:52 – 00:04:54]** CooVS, it is our premier library. **[00:04:54 – 00:04:57]** The goal of CooVS really is to shoulder the burden of keeping **[00:04:57 – 00:05:00]** these algorithms up to date, right? **[00:05:00 – 00:05:02]** Everybody could implement these algorithms themselves, but **[00:05:02 – 00:05:05]** as we release new architectures, right, you would have to **[00:05:05 – 00:05:07]** take on the burden then of updating your algorithms. **[00:05:07 – 00:05:09]** We make sure that you're always going to get the best performance **[00:05:09 – 00:05:12]** out of these algorithms, no matter what architecture you're using. **[00:05:12 – 00:05:15]** So we have many different APIs that we expose. **[00:05:15 – 00:05:19]** We build the core in C++, expose it through C, support **[00:05:19 – 00:05:21]** Python, Go, Rust. **[00:05:21 – 00:05:23]** We even have support for Java. **[00:05:23 – 00:05:27]** We integrate into other libraries, such as the popular phase library **[00:05:27 – 00:05:31]** for vector search out of Meta, that provide interoperability so **[00:05:31 – 00:05:34]** that you can, for instance, build indexes on the GPU and then serve **[00:05:34 – 00:05:37]** them on the CPU if you'd like. **[00:05:38 – 00:05:41]** Vikram, do you want to do you want to cover this slide? **[00:05:45 – 00:05:46]** Sorry. **[00:05:46 – 00:05:48]** Sorry. **[00:05:48 – 00:05:50]** If you think about the vector search problem, **[00:05:50 – 00:05:52]** there are two particular set of stages of operation. **[00:05:52 – 00:05:56]** One is the index build stage, and another is the search stage. **[00:05:56 – 00:05:59]** The index build stage, essentially you bring in a chunk of data, **[00:05:59 – 00:06:02]** you ingest them, you generate embeddings, and after you **[00:06:02 – 00:06:05]** generate the embeddings, you start generating the indexes **[00:06:05 – 00:06:07]** for your set of embeddings. **[00:06:07 – 00:06:11]** The choice of the index that you pick is like IVF or HNSW or graph, **[00:06:11 – 00:06:15]** they have its own trade-off, which Corey is gonna speak about later. **[00:06:15 – 00:06:18]** The search phase is more about once you have an index, you **[00:06:18 – 00:06:20]** have to load it in the memory and you have to determine where the set **[00:06:20 – 00:06:22]** of requests is going to be landing. **[00:06:22 – 00:06:26]** This is your query versus your set of index that comes into picture. **[00:06:26 – 00:06:29]** The objective of IndexBuild and search are completely different. **[00:06:29 – 00:06:33]** So for instance, the objective for IndexBuild is to prepare your **[00:06:33 – 00:06:36]** data to do fast lookup, whereas the search is to similar, retrieve **[00:06:36 – 00:06:38]** a similar set of vectors that are really close Thanks for watching. **[00:06:38 – 00:06:40]** I'll see you next time. **[00:06:40 – 00:06:44]** The frequency is index built are usually infrequent compared **[00:06:44 – 00:06:47]** to the search operation because search is essentially a throughput **[00:06:47 – 00:06:51]** problem and operations both sides you will observe a lot **[00:06:51 – 00:06:55]** of batching and a real-time need. **[00:06:55 – 00:06:57]** There are other characteristics of this we can get into details **[00:06:57 – 00:07:00]** later but this gives you an idea that they are two different **[00:07:00 – 00:07:04]** types of workloads but are very closely related. **[00:07:06 – 00:07:08]** And so getting back into the approximate nature of these **[00:07:08 – 00:07:11]** algorithms, and I mentioned these trade-offs earlier, **[00:07:11 – 00:07:13]** you're always going to have this trade-off whenever you **[00:07:13 – 00:07:16]** have an approximation, these ML models, between your search **[00:07:16 – 00:07:19]** speed, the quality of your model, and the complexity of the model. **[00:07:19 – 00:07:23]** The complexity of the model tends to relate directly to the amount **[00:07:23 – 00:07:26]** of time that you spend building. **[00:07:26 – 00:07:30]** Thank you. And so, you know, at the most naive, we tend to **[00:07:30 – 00:07:35]** have, you know, exhaustive search, which can guarantee, it's not **[00:07:35 – 00:07:39]** actually always exact, that depends on a few factors, but we call this **[00:07:39 – 00:07:43]** a flat index, where we perform what we call a brute force, we basically **[00:07:43 – 00:07:46]** take a set of query vectors and we compare it against... **[00:07:47 – 00:07:48]** Every single one of the vectors in the index. **[00:07:48 – 00:07:50]** And you can imagine that that doesn't scale particularly easily. **[00:07:50 – 00:07:54]** To make them scale better, we build these machine learning models. **[00:07:54 – 00:07:57]** And there's two basic architectures to the machine learning models **[00:07:57 – 00:07:58]** that are widely used today. **[00:07:58 – 00:08:00]** They are not mutually exclusive. **[00:08:00 – 00:08:04]** Tree-based improves scale immensely by clustering the **[00:08:04 – 00:08:06]** vectors using some simple algorithm like k-means so **[00:08:06 – 00:08:08]** that we can group vectors together. **[00:08:08 – 00:08:12]** Oftentimes, we might build an index on each of the clusters separately. **[00:08:12 – 00:08:15]** Graph-based indexes really enable really low latency search, so you **[00:08:15 – 00:08:19]** see a lot of the popular databases today, they will implement, **[00:08:19 – 00:08:23]** integrate graph-based algorithms. **[00:08:23 – 00:08:27]** And so, you know, I think rather than trying to, you know, list **[00:08:27 – 00:08:31]** out every single algorithm that is available today for indexing and **[00:08:31 – 00:08:35]** search, a lot of these algorithms were constructed kind of in concert **[00:08:35 – 00:08:38]** with each other over time, and there's certain features that have **[00:08:38 – 00:08:40]** become Some common to all of them. **[00:08:40 – 00:08:43]** And I think it's a lot easier for me to present this in terms **[00:08:43 – 00:08:46]** of a framework, kind of talking about the primitives and how those **[00:08:46 – 00:08:47]** break down into building blocks. **[00:08:47 – 00:08:51]** So, first we start with a simple vector search index. **[00:08:51 – 00:08:53]** I have an API for the vector search index that allows me **[00:08:53 – 00:08:56]** to take a series of vectors, train the index, extend the **[00:08:57 – 00:08:59]** index, means that it's dynamic, I can add vectors later to the **[00:08:59 – 00:09:01]** index, and I can search the index. **[00:09:01 – 00:09:04]** Let's just say that this is like a class, right? **[00:09:04 – 00:09:06]** Like a plain old object-oriented class **[00:09:07 – 00:09:09]** for representing a vector index. **[00:09:09 – 00:09:13]** And inside of the payload, I can have some general index algorithm. **[00:09:13 – 00:09:15]** That could be a graph, a tree, it could be a flat index. **[00:09:15 – 00:09:18]** Can you go forward? **[00:09:19 – 00:09:24]** The most trivial and naive way to scale this would be to **[00:09:24 – 00:09:26]** create many smaller indexes, right? **[00:09:26 – 00:09:27]** I've got a large data set. **[00:09:27 – 00:09:29]** I've got billions of vectors. **[00:09:29 – 00:09:32]** It would take a long time to train one index or maybe I don't **[00:09:32 – 00:09:34]** even have memory on a single node to be able to train that index. **[00:09:35 – 00:09:38]** So instead, I break the data down into smaller bits and I train a **[00:09:38 – 00:09:40]** separate index on each of the bits. **[00:09:40 – 00:09:43]** This is a common scaling technique. **[00:09:43 – 00:09:45]** A lot of well-established databases, rather than **[00:09:45 – 00:09:49]** re-architecting for vector types, have integrated **[00:09:49 – 00:09:50]** vector search in this way. **[00:09:50 – 00:09:53]** The challenge here is that because you've uniformly distributed your **[00:09:53 – 00:09:57]** data across each of these indexes, you now have to search every single **[00:09:57 – 00:09:59]** one, every time a query comes in. **[00:09:59 – 00:10:01]** You can imagine that that impacts scale, right? **[00:10:01 – 00:10:04]** Eventually you've got so many of these little shards that **[00:10:04 – 00:10:08]** you're spending too much time, it's not worthwhile to search. **[00:10:09 – 00:10:12]** And so improving the scale just slightly, if we take **[00:10:12 – 00:10:16]** a clustering algorithm, structured indexes tend to scale very **[00:10:16 – 00:10:17]** well when we sort the data. **[00:10:18 – 00:10:19]** How do you sort vectors? **[00:10:19 – 00:10:22]** Well, it's a distance computation across different dimensions. **[00:10:22 – 00:10:24]** Putting the vectors in lexicographic order isn't going **[00:10:24 – 00:10:26]** to do anything to improve scale. **[00:10:27 – 00:10:30]** So kind of the analog to that would be clustering, grouping **[00:10:30 – 00:10:32]** the vectors that are similar and closer together. **[00:10:32 – 00:10:36]** And so when we do this, we create what's called an inverted **[00:10:36 – 00:10:39]** file index or a global index. **[00:10:39 – 00:10:43]** Each of the partitions could store any type of simple index, **[00:10:44 – 00:10:45]** as we're showing in this diagram. **[00:10:45 – 00:10:47]** You could put a tree in there. **[00:10:47 – 00:10:50]** You could put another partition-based algorithm in there. **[00:10:50 – 00:10:52]** You could have a flat index in there. **[00:10:52 – 00:10:54]** You can compress the vectors. **[00:10:54 – 00:10:58]** You can also assign vectors to not just its closest partition, but **[00:10:58 – 00:11:00]** you can spill across partitions. **[00:11:00 – 00:11:03]** We find that that improves quality. **[00:11:03 – 00:11:06]** There are algorithms like, for instance, Google Scan, **[00:11:06 – 00:11:10]** Meta's IVF-PQ, there is Microsoft's DiskANN. **[00:11:10 – 00:11:13]** They pretty much all borrow from this framework. **[00:11:13 – 00:11:15]** Obviously, the framework wasn't created first, but this framework **[00:11:16 – 00:11:18]** kind of summarizes and can be used to compose every single **[00:11:19 – 00:11:20]** one of those algorithms. **[00:11:20 – 00:11:21]** Go forward. **[00:11:21 – 00:11:24]** And so you can see easily that we could use multi-levels **[00:11:24 – 00:11:25]** of clustering, right? **[00:11:25 – 00:11:27]** And now we can dispatch like vectors to certain **[00:11:27 – 00:11:28]** nodes and collect them. **[00:11:28 – 00:11:31]** So we can keep going up and building a tree. **[00:11:31 – 00:11:33]** And one of the other benefits to building a tree here, and **[00:11:34 – 00:11:38]** some of the some of the grape that **[00:11:38 – 00:11:42]** You know, newcomers to the field feel about IVF is that as soon as **[00:11:42 – 00:11:46]** those clusters start to drift over time, you've got trending topics. **[00:11:46 – 00:11:48]** Everybody today is talking about Taylor Swift, you know, **[00:11:48 – 00:11:51]** everybody tomorrow is talking about, you know, insert whatever **[00:11:51 – 00:11:53]** happened in the news lately. **[00:11:53 – 00:11:56]** You know, eventually those clusters might get imbalanced. **[00:11:56 – 00:11:58]** You don't have to rebalance, sorry, you don't have to retrain **[00:11:58 – 00:12:00]** the clusters from scratch if you have a tree, right? **[00:12:00 – 00:12:02]** You can split nodes of a tree, you can join them together **[00:12:02 – 00:12:04]** for smaller clusters. **[00:12:04 – 00:12:07]** Go ahead, forward. **[00:12:07 – 00:12:09]** And so improving upon this framework, as I mentioned, **[00:12:10 – 00:12:12]** you know, there are quantized variants of a lot of the popular **[00:12:12 – 00:12:15]** indexing algorithms today, but those quantized variants **[00:12:15 – 00:12:16]** are just a mere detail. **[00:12:16 – 00:12:17]** It's just another primitive. **[00:12:17 – 00:12:20]** You can quantize the vectors and build a graph. **[00:12:20 – 00:12:23]** You can do an IVF and then you can quantize or compress. **[00:12:23 – 00:12:25]** Each of the different partitions, right, and then build a **[00:12:25 – 00:12:28]** graph on that or use any other type of index. **[00:12:28 – 00:12:31]** And so that's another thing, pre-processing, right, as **[00:12:31 – 00:12:34]** Jensen mentioned, unstructured data processing and quantization **[00:12:34 – 00:12:36]** and dimensionality reduction fall into that category as well. **[00:12:36 – 00:12:38]** So we achieve really great speedups. **[00:12:38 – 00:12:40]** You can compress them quite small and still be able to **[00:12:40 – 00:12:43]** maintain a lot of quality. **[00:12:44 – 00:12:47]** And so, graph-based methods are, by far, as I mentioned, some **[00:12:47 – 00:12:49]** of the most popular methods today. **[00:12:49 – 00:12:52]** Kind of improving upon this framework even further, you **[00:12:52 – 00:12:58]** know, there are common improvements upon pretty much all of the **[00:12:58 – 00:12:58]** graph-based algorithms. **[00:12:58 – 00:13:01]** You know, the graphs might be constructed in different **[00:13:01 – 00:13:03]** ways, they might have slightly different properties in terms **[00:13:03 – 00:13:06]** of how the neighborhoods look when the graph is built. **[00:13:06 – 00:13:09]** But in general, they aim to reduce redundant paths so **[00:13:09 – 00:13:11]** that with as few hops as possible through the graph, You can find **[00:13:11 – 00:13:15]** the nearest neighbors that you're looking for for some value of K. **[00:13:15 – 00:13:17]** Oftentimes, the graph-based algorithms do need to keep **[00:13:17 – 00:13:19]** the vectors around for traversal, but they don't always have **[00:13:19 – 00:13:20]** to be kept in memory. **[00:13:20 – 00:13:22]** You can compress, right? **[00:13:22 – 00:13:26]** And then you can have the actual full precision vectors on disk. **[00:13:26 – 00:13:29]** You can have them in, you know, yeah, if you're doing this on GPU, **[00:13:29 – 00:13:31]** you can have them in host memory. **[00:13:31 – 00:13:34]** That is an unfortunate detail about the graph-based algorithms that **[00:13:34 – 00:13:36]** a lot of people don't consider. **[00:13:36 – 00:13:39]** If you're willing to suffer some quality, you can get rid of the **[00:13:39 – 00:13:43]** original vectors and you can just keep the compressed vectors around. **[00:13:43 – 00:13:45]** Another common optimization is reordering the vectors **[00:13:45 – 00:13:48]** in the graph so that you can put **[00:13:48 – 00:13:51]** Similar vectors together when they're stored in memory or **[00:13:51 – 00:13:55]** on disk so that you can improve **[00:13:55 – 00:13:56]** the pageability of the vectors. **[00:13:56 – 00:13:59]** So instead of having a complete random access as you're going **[00:13:59 – 00:14:01]** through the graph, you can pull a chunk and know that its neighbors **[00:14:01 – 00:14:02]** are going to be within that chunk. **[00:14:02 – 00:14:04]** These are just all common methods. **[00:14:04 – 00:14:08]** Two general approaches to building the graph. **[00:14:08 – 00:14:10]** One is called this Navigating Spreading Out Graph. **[00:14:10 – 00:14:12]** We use this in an algorithm that we've built, a novel **[00:14:13 – 00:14:17]** algorithm in Kagra, sorry, in QVS called Kagra, which is very **[00:14:17 – 00:14:20]** amenable to massive parallelism, and that's one of the reasons **[00:14:20 – 00:14:22]** we've adopted this approach. **[00:14:22 – 00:14:25]** We didn't create the approach, but we improved upon it. **[00:14:25 – 00:14:27]** We start with a pre-built, we basically do that high **[00:14:27 – 00:14:31]** throughput index building task that I mentioned earlier, **[00:14:31 – 00:14:33]** where you do a search, I wanna find the nearest neighbors **[00:14:33 – 00:14:36]** for 100 million vectors, because that's the size of my index. **[00:14:36 – 00:14:39]** Other methods that are widely adopted today, mostly on the **[00:14:39 – 00:14:43]** CPU, this HNSW algorithm, that I think most folks these **[00:14:43 – 00:14:45]** days have at least heard. **[00:14:45 – 00:14:47]** It's a hierarchical algorithm, it's greedy, so it can be **[00:14:47 – 00:14:51]** constructed one vector at a time. **[00:14:52 – 00:14:54]** Thanks Vikram. I'll leave it to Vikram now to talk about **[00:14:54 – 00:14:55]** Kagra in more detail. **[00:14:55 – 00:14:58]** Yeah, thank you so much, Corey. **[00:15:01 – 00:15:05]** So, I don't want to go into the details of the Kegra algorithm, but **[00:15:05 – 00:15:07]** I want to give you a kind of like a mental picture because it's very **[00:15:07 – 00:15:11]** critical to understand for the next set of evaluations and others that **[00:15:11 – 00:15:12]** we're going to be speaking about. **[00:15:12 – 00:15:15]** So, the Kegra algorithm intuition is that it's split **[00:15:15 – 00:15:17]** into primary four stages. **[00:15:17 – 00:15:20]** The first phase is creation of the initial graph using **[00:15:20 – 00:15:22]** the vectors that you have received. **[00:15:22 – 00:15:24]** The next phase is reordering and pruning the set of graphs. **[00:15:24 – 00:15:27]** The next is to create reversing of the graph and then creating **[00:15:27 – 00:15:29]** a merging of the graph. **[00:15:29 – 00:15:32]** It's important to construct a mental model on why these stages **[00:15:32 – 00:15:35]** are needed, and that would give us a better idea of what exactly is **[00:15:35 – 00:15:37]** happening in the k-graph algorithm. **[00:15:37 – 00:15:40]** The idea of creating an initial graph construction is to create **[00:15:40 – 00:15:44]** a directed graph where each node connects to its nearest **[00:15:44 – 00:15:46]** neighbors by distance. **[00:15:46 – 00:15:49]** The idea here is that, okay, I want to identify who are **[00:15:49 – 00:15:52]** my most similar points next to me. **[00:15:52 – 00:15:56]** With this, you actually produce a dense locality similar graph **[00:15:56 – 00:16:00]** where every node points to many nearby nodes and this **[00:16:00 – 00:16:04]** graph, we call it locally correct but globally weak because we have **[00:16:04 – 00:16:08]** many connections that are redundant and can lead into the same set of **[00:16:08 – 00:16:10]** regions that you work with and some part of the graph may not be easily **[00:16:11 – 00:16:13]** reachable whereas the others can. **[00:16:13 – 00:16:17]** To address that, we do something called reordering and pruning, **[00:16:17 – 00:16:19]** and the idea here is that because you have many redundant **[00:16:19 – 00:16:22]** operations, and you can raise them concurrently at many different **[00:16:22 – 00:16:27]** places, you want to reduce the set of graphs only to export new areas **[00:16:27 – 00:16:29]** and in the most fastest fashion. **[00:16:29 – 00:16:32]** And this is done by creating something called detour edges, **[00:16:32 – 00:16:35]** and through the detour edges you actually move to the set of **[00:16:35 – 00:16:36]** graphs that you are interested in. **[00:16:37 – 00:16:39]** Edges that do not add new reachability are basically **[00:16:39 – 00:16:43]** pushed down in the importance, and while the edges that open access **[00:16:43 – 00:16:45]** to new regions are promoted up. **[00:16:45 – 00:16:47]** This reordering helps you to generate a much more **[00:16:47 – 00:16:51]** complex graph, and allows you to navigate more friendly. **[00:16:51 – 00:16:53]** And also enables you to go to different directions of **[00:16:53 – 00:16:55]** the dataset very easily. **[00:16:55 – 00:16:57]** After you have done that, you actually do something called **[00:16:57 – 00:17:01]** a spruning, where connections are still directional, but **[00:17:01 – 00:17:04]** you only keep those connections that provide the maximum value, **[00:17:04 – 00:17:08]** and you also want to enable a path towards going back. **[00:17:08 – 00:17:11]** So the idea about creating a reverse graph approach here **[00:17:11 – 00:17:14]** is that there are some nodes that are easily reachable, **[00:17:14 – 00:17:15]** but hard to leave. **[00:17:15 – 00:17:17]** So you want to fix that by creating a reversing operation, and **[00:17:17 – 00:17:21]** that is the reverse graph stage. **[00:17:21 – 00:17:25]** If a node A considered is important with respect to **[00:17:25 – 00:17:29]** node B, then B is also allowed to connect back to node A. **[00:17:29 – 00:17:32]** This step improves our overall connectivity so that the searches **[00:17:32 – 00:17:35]** that are happening from a random location can be easily reached **[00:17:35 – 00:17:36]** across a different data set. **[00:17:36 – 00:17:39]** That's the core principle for reversing the graph. **[00:17:39 – 00:17:42]** Once you have both the pruned graph and also the reverse graph, **[00:17:42 – 00:17:44]** you have to merge the two graphs. **[00:17:44 – 00:17:49]** And this merging is basically is the last stage of the operation. **[00:17:49 – 00:17:52]** But if you look into how the graph index build operation happens, **[00:17:52 – 00:17:55]** you can mathematically formulate what is the amount of input **[00:17:55 – 00:17:56]** data that you have to work with. **[00:17:57 – 00:17:59]** You can also formulate what is the amount of memory footprint that **[00:17:59 – 00:18:02]** you need while doing the execution. **[00:18:02 – 00:18:03]** And you can also formulate how much is the amount of data **[00:18:04 – 00:18:06]** that you will be generating after this operation has been completed. **[00:18:06 – 00:18:10]** They're all kind of like very nicely can be fit into **[00:18:10 – 00:18:11]** a bunch of equations. **[00:18:11 – 00:18:14]** In this case, I'm showing you what is the active memory **[00:18:14 – 00:18:17]** consumption of the Kegra index build algorithm, considering **[00:18:17 – 00:18:21]** a few sets of different sizing and number of vectors in this space. **[00:18:21 – 00:18:25]** 1 billion is kind of like a very small data set today for vector **[00:18:25 – 00:18:28]** index build and search operations. **[00:18:29 – 00:18:31]** People are already, as you saw in the initial graph, **[00:18:31 – 00:18:34]** people are already at tens of billion scale and going **[00:18:34 – 00:18:35]** towards 100 billion scale. **[00:18:36 – 00:18:38]** And if you notice here is that 1 billion already cannot fit in the **[00:18:38 – 00:18:45]** GPU memory capacity and it barely fits on the larger GPUs, but not **[00:18:45 – 00:18:47]** on the regular set of GPUs at all. **[00:18:47 – 00:18:51]** Whereas 100 billion cannot and there are way more lower **[00:18:51 – 00:18:55]** bounds that it's impossible to even to create over here. **[00:18:55 – 00:18:57]** So the idea here is that the x-axis basically shows you **[00:18:57 – 00:19:01]** the amount of the number of vectors in your dataset. **[00:19:01 – 00:19:03]** Y-axis tells you how much is the memory footprint that **[00:19:03 – 00:19:07]** you need in a terabyte, and how does it actually scale. **[00:19:07 – 00:19:11]** And y-axis is in log scale, so it grows extremely high. **[00:19:12 – 00:19:15]** That's the idea. The other factor is, I'm trying to **[00:19:15 – 00:19:17]** create here what's the read versus write ratios. **[00:19:17 – 00:19:21]** The initial graph build is heavy, read heavy. **[00:19:21 – 00:19:23]** Reordering pruning is also read heavy because you have to re-read **[00:19:23 – 00:19:26]** many, many times to identify what sort of graph points really matter **[00:19:26 – 00:19:28]** versus what graph do not matter. **[00:19:28 – 00:19:31]** The reverse graph is 50-50 because you are reading a **[00:19:31 – 00:19:33]** ton of data and also writing the same amount of data. **[00:19:33 – 00:19:35]** And the last one is merging. **[00:19:35 – 00:19:37]** It's slightly read-heavy, because you're reading more **[00:19:37 – 00:19:38]** amount of data and then merging. **[00:19:38 – 00:19:40]** And when you do the merge, you have to move the data **[00:19:40 – 00:19:45]** in and out, so that causes you a bit more read-heavy work. **[00:19:45 – 00:19:48]** The same thing can also be done for Vamana Build Algorithm. **[00:19:48 – 00:19:51]** So, in the case of Vamana Build Algorithm, what happens **[00:19:51 – 00:19:54]** is that you start with an empty graph with just with one **[00:19:54 – 00:19:58]** starting vector, which we call the starting vector as Mediord Vector. **[00:19:58 – 00:20:00]** We iteratively insert batches of new vectors. **[00:20:00 – 00:20:03]** We perform beam search when you insert the new set of **[00:20:03 – 00:20:05]** the vectors into the current graph. **[00:20:05 – 00:20:06]** We perform robust prune-on-top. **[00:20:06 – 00:20:09]** We do a setup of it on the all-inserted graph, we create **[00:20:09 – 00:20:12]** reverse edges, and once we exit the max-degree, we perform **[00:20:12 – 00:20:14]** another set of pruning. **[00:20:14 – 00:20:16]** And we logarithmically increase the number of batch sizes **[00:20:16 – 00:20:18]** when you keep on doing the insertion, and you try to **[00:20:18 – 00:20:24]** get to a maximum fraction of the n, which is a happy parameter in the **[00:20:24 – 00:20:27]** system, trying to see what happens. **[00:20:27 – 00:20:28]** The idea here that I want to tell you is that both the **[00:20:29 – 00:20:32]** graph algorithms are identical to the Kegra algorithm and Vamana. **[00:20:32 – 00:20:34]** They are similar in the method. **[00:20:34 – 00:20:37]** So you can also compute how much is the input and how **[00:20:37 – 00:20:40]** much is the output and how much is the temporary memory that you need. **[00:20:40 – 00:20:44]** And the read-write ratio is very similar to the Kegra approach. **[00:20:44 – 00:20:47]** Now, clearly we are seeing a memory being one of the **[00:20:47 – 00:20:49]** biggest limiters in this space. **[00:20:49 – 00:20:51]** But if you look at the current data hierarchy that we have, **[00:20:51 – 00:20:53]** the traditional memory hierarchy is kind of broken for the **[00:20:53 – 00:20:55]** set of workloads that we're looking into, especially in **[00:20:55 – 00:21:00]** the case of the VectorDB workload. **[00:21:00 – 00:21:02]** The way in which we have been thinking about this is that **[00:21:02 – 00:21:05]** we want to increase the capacity per watt and also capacity **[00:21:05 – 00:21:09]** per dollar, but this is not a great idea for the kinds of **[00:21:09 – 00:21:11]** workloads that we are looking into. **[00:21:11 – 00:21:14]** What we really need is a new memory tier, and the new memory **[00:21:14 – 00:21:17]** tier that CJ, me, and a bunch of people have been working towards **[00:21:17 – 00:21:21]** is called Storage Next, where we are trying to optimize for IOPS per **[00:21:21 – 00:21:23]** watt and IOPS per dollar metric. **[00:21:23 – 00:21:26]** The idea here is that this is kind of like you want to **[00:21:26 – 00:21:28]** kind of scale up your every node and increase your memory **[00:21:28 – 00:21:32]** capacity so that you can build much more larger graph and expand way **[00:21:32 – 00:21:35]** more if you want to go to like a hundred million scale or even more. **[00:21:35 – 00:21:38]** There are a few characteristics of this new memory tier that **[00:21:38 – 00:21:41]** we are working across a bunch of storage-next-par collaborators. **[00:21:41 – 00:21:44]** We want to increase the IO's per watt and dollar with respect **[00:21:44 – 00:21:48]** to the smaller grain access granularity like 512 byte to **[00:21:48 – 00:21:52]** 1 kilobyte in kind of like ranges. **[00:21:52 – 00:21:55]** Now, the reasoning here is, if you look into how GPUs **[00:21:55 – 00:21:58]** have been evolved over a period of time, you have been thinking **[00:21:58 – 00:22:01]** that the GPUs have been used as extremely limited, like **[00:22:01 – 00:22:03]** an offload-based solution. **[00:22:03 – 00:22:06]** That is a CPU who orchestrates the GPU movement and the data **[00:22:06 – 00:22:10]** movement on how and when the data needs to go to the GPU. **[00:22:10 – 00:22:12]** And the CPU is entirely in the control path of the operation, **[00:22:12 – 00:22:14]** and GPU is used as an assistant. **[00:22:15 – 00:22:17]** But if you look into the work that is being happening in **[00:22:17 – 00:22:20]** the end-to-end execution, majority of the work happens on the GPU. **[00:22:20 – 00:22:23]** And you're moving petabyte of data to the GPU, and you're **[00:22:23 – 00:22:25]** trying to execute to that, and CPU is doing all the orchestration. **[00:22:25 – 00:22:29]** It makes no sense, because the CPU is doing the control path, **[00:22:29 – 00:22:33]** trying to handle a very skinny straw on how the GPU has to work. **[00:22:33 – 00:22:34]** Makes no sense to us. **[00:22:34 – 00:22:37]** So we are like, okay, what does it mean for us if you want to... **[00:22:37 – 00:22:40]** Extend that into GPU native workload. **[00:22:40 – 00:22:44]** And the idea here is that we want to enable vector search **[00:22:44 – 00:22:45]** with what it really needs. **[00:22:45 – 00:22:47]** And if you look into what vector search algorithms seek, it's some **[00:22:48 – 00:22:50]** sort of like a view-based memory in the notion of like an infinite **[00:22:50 – 00:22:54]** memory pool for a very large memory capacity that you're working with. **[00:22:54 – 00:22:58]** You need the ability to fetch on demand on what data that you want **[00:22:58 – 00:23:00]** during the index build operation. **[00:23:00 – 00:23:01]** And you want to be able to scale both the compute and **[00:23:01 – 00:23:05]** data pipelines as you want, as independently as possible. **[00:23:05 – 00:23:09]** To do this, we created SCADA, and we call SCADA Scaled **[00:23:09 – 00:23:10]** Accelerated Data Access. **[00:23:10 – 00:23:14]** It comprises of two components, client and the server. **[00:23:14 – 00:23:17]** The client is a user-level accelerated, untrusted operation, **[00:23:17 – 00:23:21]** where GPU threats can do GPU-initiated IO. **[00:23:21 – 00:23:23]** It's not just the GPU can do it, it's both GPU and **[00:23:24 – 00:23:26]** CPU can do initiation here, it does not matter. **[00:23:26 – 00:23:29]** But if you do it in the GPU, you get the maximum performance, **[00:23:29 – 00:23:34]** and it has a caching library In the GPU, that allows you to **[00:23:34 – 00:23:37]** cast the most frequently used data. **[00:23:37 – 00:23:41]** The server side is also GPU-accelerated, and you **[00:23:41 – 00:23:44]** have a data service that is running on the GPU over there, trying **[00:23:44 – 00:23:47]** to access the data in the storage. **[00:23:47 – 00:23:49]** We also have a user-space NVMe driver that is running **[00:23:49 – 00:23:51]** on the GPU, trying to access the NVMe devices and mapping **[00:23:51 – 00:23:53]** and managing them really well. **[00:23:53 – 00:23:55]** We also added two additional components, which is a control **[00:23:55 – 00:23:56]** path and also a data path. **[00:23:56 – 00:23:59]** The data path is fully RDMA-enabled, whereas the control **[00:24:00 – 00:24:04]** path is... We built a new algorithm protocol called autobatch buffer, a **[00:24:04 – 00:24:08]** buffering protocol here that allows the creation of batches in dynamic **[00:24:08 – 00:24:11]** fashion from the GPU and then submitting them to the server side **[00:24:11 – 00:24:13]** and processing them over there. **[00:24:13 – 00:24:14]** There are more details in the slide, happy to share **[00:24:15 – 00:24:18]** later, which explains more about the details of different components **[00:24:18 – 00:24:21]** over here, but that's the idea. **[00:24:21 – 00:24:25]** We then tested this system through initial micro-benchmarks **[00:24:25 – 00:24:26]** with the H100 system. **[00:24:26 – 00:24:31]** We created the H100 system, trying to see what are the limits and **[00:24:31 – 00:24:34]** scales of it, and we wanted to see, let's compare with the CPU on how **[00:24:34 – 00:24:35]** much of the throughput we can get. **[00:24:35 – 00:24:39]** So the x-axis right side shows you the comparison between **[00:24:39 – 00:24:41]** two same generation of GPU and CPU. **[00:24:42 – 00:24:46]** The CPU is Gen5, and the Gen5 can only do with a highly **[00:24:46 – 00:24:50]** performant full stack with FIO, with IOU ring, with all **[00:24:50 – 00:24:53]** the software overhead removed, no file system, nothing, right? **[00:24:54 – 00:24:55]** Everything remote, clean, clean sweep. **[00:24:55 – 00:24:59]** So it's just bare metal, brute force implementation. **[00:24:59 – 00:25:01]** It can barely hit like 43 million IOPS at 512 byte **[00:25:01 – 00:25:02]** access granularity. **[00:25:02 – 00:25:09]** The same thing if I want to do it on the H100 GPU, with less than **[00:25:09 – 00:25:12]** 20 percent, that was like three or four months ago when we spoke **[00:25:12 – 00:25:15]** about, now it is like less than 10 percent of the GPU utilization, **[00:25:15 – 00:25:16]** we can hit 98 million IOPS. **[00:25:16 – 00:25:19]** We can't go more because we are limited by the PCI bandwidth **[00:25:19 – 00:25:20]** and nothing else. **[00:25:20 – 00:25:24]** So we can hit the full peak of the system right now. **[00:25:25 – 00:25:26]** We then, okay, it was great. **[00:25:26 – 00:25:28]** So this is the first time we are showing this. **[00:25:28 – 00:25:31]** So we integrated QVS with SCADA. **[00:25:31 – 00:25:33]** We have been working on it more like a proof of concept. **[00:25:33 – 00:25:36]** This is an unoptimized research, and this is a research work. **[00:25:36 – 00:25:38]** And we are trying to push towards, okay, what can we **[00:25:38 – 00:25:39]** do with respect to it? **[00:25:39 – 00:25:41]** So there are two primary charts here. **[00:25:41 – 00:25:44]** The biggest, I think, is the lower part is the 10 million **[00:25:44 – 00:25:49]** dataset, whereas the 100 million is a much more larger dataset. **[00:25:49 – 00:25:52]** There are three primary axes, one is the Cuvius all-in HBM **[00:25:52 – 00:25:54]** memory, another is Cuvius in the host memory, and the **[00:25:54 – 00:25:57]** third one is with SCADA, and we are not using the CPU memory in this **[00:25:57 – 00:25:59]** case, and we are using eight SSDs. **[00:25:59 – 00:26:02]** At smaller data sets, you'll notice that the SCADA solution **[00:26:02 – 00:26:07]** is somewhere around 2.8x slower compared to all data **[00:26:07 – 00:26:09]** in the host memory. **[00:26:09 – 00:26:12]** Repeat the same experiment with 10x more larger data set. **[00:26:12 – 00:26:14]** What you'll notice here is that the SCADA solution is **[00:26:14 – 00:26:20]** roughly, I mean, it scales slightly better and it's only 2x slower **[00:26:20 – 00:26:23]** compared to the baseline system. **[00:26:23 – 00:26:25]** And you can't run it on the HBM. **[00:26:25 – 00:26:30]** The idea here is that you are able to scale better and we **[00:26:30 – 00:26:32]** have not done any optimization yet. **[00:26:32 – 00:26:35]** So we are hoping to narrow this gap even further in the coming **[00:26:35 – 00:26:39]** days, and we are also hoping to grow to much more larger indexes. **[00:26:39 – 00:26:42]** Now if you look into the breakdown of where this time is spent, **[00:26:42 – 00:26:45]** this experiment is done with the big GNN 100 million dataset. **[00:26:45 – 00:26:48]** There are four primary studies I've said, and their memory **[00:26:48 – 00:26:51]** access patterns are very well aligned with SCADA's access **[00:26:51 – 00:26:52]** patterns that we really like. **[00:26:52 – 00:26:55]** And the time currently is being spent in the K-means **[00:26:55 – 00:26:58]** algorithm, that is the first stage in the pipeline, and the majority **[00:26:58 – 00:27:01]** of the time here spent is because we have not optimized our SCADA **[00:27:02 – 00:27:05]** stack yet for this kernel code. **[00:27:05 – 00:27:09]** We have been speaking about GNN for a while, so we are **[00:27:09 – 00:27:13]** not stopping GNN, we are continuing the work on the GNN, and end-to-end **[00:27:13 – 00:27:16]** results on the SCADA GNN is already showing very promising results. **[00:27:16 – 00:27:19]** We have only about 20% lower performance, even with the **[00:27:19 – 00:27:22]** unlimited size, compared to the CPU memory. **[00:27:22 – 00:27:23]** We will continue to invest more. **[00:27:23 – 00:27:26]** Next, I will hand over to CJ to speak more about where **[00:27:26 – 00:27:28]** this is headed towards. **[00:27:28 – 00:27:29]** Thanks Vikram. **[00:27:29 – 00:27:32]** So if you just kind of step back and look at the big picture **[00:27:32 – 00:27:36]** of this, we're seeing that if you look at what is a GPU **[00:27:36 – 00:27:41]** good for, it's essentially as a thousand times the resources that **[00:27:41 – 00:27:42]** you need for being able to do this. **[00:27:42 – 00:27:47]** If you look at the threads, it has more than 10x the bandwidth. **[00:27:47 – 00:27:50]** And those same threads are used for latency tolerance, and if **[00:27:50 – 00:27:54]** you look at doing something in the disk versus in the memory, you're **[00:27:54 – 00:27:57]** looking at one to two orders of magnitude cheaper, and you're still **[00:27:57 – 00:28:00]** reasonable in terms of the power. **[00:28:00 – 00:28:04]** The implications here burst out of these memory constraints. **[00:28:04 – 00:28:05]** Which we're just starting to do, right? **[00:28:05 – 00:28:08]** We're just in a billion scale, 10 billion scale. **[00:28:08 – 00:28:11]** We're just outside of what you could do in memory. **[00:28:11 – 00:28:14]** In the next couple months, we're looking to burst upwards **[00:28:15 – 00:28:19]** of petabyte scale, which you just can't possibly do in memory. **[00:28:21 – 00:28:25]** We want then to be able to reach directly and access the **[00:28:25 – 00:28:27]** data at a very fine granularity. **[00:28:28 – 00:28:32]** If you were to bring everything in at, say, the size of a 64K page, **[00:28:32 – 00:28:36]** if you did things like that, if you had just a couple 5TB references, **[00:28:36 – 00:28:37]** then your RAF would be like 128. **[00:28:37 – 00:28:41]** You'd be moving 128 times as much data as you need. **[00:28:41 – 00:28:44]** And if that's the bottleneck, then that's a concern. **[00:28:44 – 00:28:48]** And we're seeing, as Vikram just showed, that we can saturate **[00:28:48 – 00:28:54]** Gen6 bandwidth, which is at 512 bytes, as 200 million IOPS. **[00:28:54 – 00:28:57]** And that drives us into wanting to have drives that **[00:28:57 – 00:28:58]** go faster for that. **[00:28:58 – 00:29:01]** I'm going to talk a little bit about some of the related **[00:29:01 – 00:29:03]** technology, it's a little bit exciting here. **[00:29:03 – 00:29:05]** We've been working with a wide variety of different **[00:29:05 – 00:29:11]** partners as part of this ecosystem as we move forward and evaluate and **[00:29:11 – 00:29:14]** have others replicate the results and are working on what it is **[00:29:14 – 00:29:18]** that we need to do for the broader system, not just in the drives and **[00:29:18 – 00:29:22]** the storage controllers, but also for the other interactions with **[00:29:22 – 00:29:24]** the storage providers and so on. **[00:29:24 – 00:29:27]** So a couple of the aspirations that we have for the front **[00:29:27 – 00:29:29]** end are obviously the high IOPs. **[00:29:29 – 00:29:34]** In Gen7, we're looking for a total of 400 million IOPs per GPU. **[00:29:34 – 00:29:39]** You may need some number of GPUs, so that'll determine **[00:29:39 – 00:29:42]** sort of ultimately what that... **[00:29:42 – 00:29:46]** Capacity is for each of those drives as well. **[00:29:46 – 00:29:50]** You want to be more efficient and not throwing away power **[00:29:50 – 00:29:52]** for the bits that you move in. **[00:29:52 – 00:29:58]** And we're looking at something called PMR for the persistent **[00:29:58 – 00:30:02]** memory regions for addressability, rather than only using **[00:30:02 – 00:30:04]** NVMe of having some of that on the front end. **[00:30:05 – 00:30:07]** Being able to have an address translation cache so that **[00:30:07 – 00:30:11]** you don't have to, if the IOMMU is on and enforcing things, **[00:30:11 – 00:30:13]** then you can force it that way. **[00:30:13 – 00:30:16]** And protection over the LBA ranges so that if you have **[00:30:16 – 00:30:20]** multiple agents that are trying to access the same device, that it's **[00:30:20 – 00:30:23]** clear who owns it and who needs to keep their hands off of it. **[00:30:23 – 00:30:26]** I'd like to briefly talk about something that we've been **[00:30:26 – 00:30:30]** working on with our friends at Phison, kind of put out a **[00:30:30 – 00:30:34]** challenge to them, and that really rose to the occasion with us. **[00:30:34 – 00:30:38]** Essentially, if you have an IO buffer that you're trying **[00:30:38 – 00:30:43]** to move data out of, normally you DMA out of the drive into **[00:30:43 – 00:30:44]** an IO buffer, and then **[00:30:44 – 00:30:47]** Nick can pick that up for RDMA to send it over the network, **[00:30:47 – 00:30:50]** if you have a desegregated system. **[00:30:50 – 00:30:53]** Is it possible to actually eliminate that IO buffer? **[00:30:53 – 00:30:57]** Can you actually directly DMA into the device? **[00:30:57 – 00:30:59]** And the answer is, looks like yes. **[00:30:59 – 00:31:05]** So see, basically, you could do that and basically directly map the **[00:31:05 – 00:31:11]** LBAs into the address space through the bar and be able to get accesses **[00:31:11 – 00:31:16]** that go outside of the MDME spec even down as far as four bytes **[00:31:16 – 00:31:18]** for being able to access things. **[00:31:18 – 00:31:21]** And there's a prototype for this process, so we're looking **[00:31:21 – 00:31:22]** to see where that is. **[00:31:22 – 00:31:24]** We may not worry so much, may take the P out of that **[00:31:25 – 00:31:27]** and not worry about the persistent. **[00:31:27 – 00:31:30]** We'll talk at a future time about some of those usage models. **[00:31:30 – 00:31:34]** Our folks at Kioxia have done some really influential work. **[00:31:34 – 00:31:35]** We really appreciate this. **[00:31:35 – 00:31:39]** They created an emulator that's based on generating load in **[00:31:39 – 00:31:44]** the hopper and then emulating a drive on the Grace. **[00:31:44 – 00:31:50]** And so we've been able to do that with SCADA and essentially have **[00:31:50 – 00:31:52]** some different emulation targets. **[00:31:52 – 00:31:55]** And so these are some initial results. **[00:31:55 – 00:31:56]** This is really fun because I was... **[00:31:56 – 00:31:59]** Imagining what this graph shape would look like, it **[00:31:59 – 00:32:01]** came out exactly that way, so it was really cool. **[00:32:01 – 00:32:07]** So the principle is, if there's some cap of the amount of IOPS that **[00:32:07 – 00:32:12]** you can get, so that's the vertical scale, the question is, how quickly **[00:32:12 – 00:32:14]** can you get there with concurrency? **[00:32:14 – 00:32:16]** You're going to start at zero, and then you're going to climb, **[00:32:16 – 00:32:19]** give it more and more concurrency, to use Little's Law to cover **[00:32:19 – 00:32:23]** the latency, and if you have enough concurrency, you do fine. **[00:32:23 – 00:32:26]** If you have a longer latency, though, it's going to take **[00:32:26 – 00:32:29]** more concurrency to be able to cover that latency. **[00:32:29 – 00:32:33]** And we've also done some studies of it looking at the effect **[00:32:33 – 00:32:35]** of tail latency and looking at those distributions. **[00:32:35 – 00:32:37]** And we can clearly look at this. **[00:32:37 – 00:32:41]** And this ends up being a motivator for using SLC versus TLC. **[00:32:41 – 00:32:44]** So hopefully we have lots of concurrency all the time, **[00:32:44 – 00:32:47]** but sometimes we know the reality is that there's going **[00:32:47 – 00:32:47]** to be a variance of that. **[00:32:48 – 00:32:51]** And so having shorter latency helps it be To be more robust. **[00:32:51 – 00:32:55]** In NSYS, we have a lot of tools at NVIDIA for being able to **[00:32:55 – 00:32:58]** look at the entirety of the system and everything that's going on. **[00:32:58 – 00:33:00]** And if you want to find out, hey, where's my IO fit into **[00:33:00 – 00:33:04]** that, then we have some systems that can be able to do that. **[00:33:04 – 00:33:08]** I'd like to give some credits to maybe the gentleman in **[00:33:08 – 00:33:11]** Israel and Spain that have been working on this. **[00:33:11 – 00:33:16]** And so we now have SCADA support so that you can look at what **[00:33:16 – 00:33:20]** happens over time and you can start to analyze the latency and **[00:33:21 – 00:33:24]** we've had some great interactions with the H3P team for being **[00:33:24 – 00:33:27]** able to do this as well. **[00:33:27 – 00:33:31]** We also have, and I'll mention that too, that we have some **[00:33:31 – 00:33:36]** integrations with plug-ins for some of the Broadcom switches **[00:33:36 – 00:33:39]** and we're adding that for others to be able to look at stats of what's **[00:33:39 – 00:33:41]** going on inside the switches. **[00:33:41 – 00:33:44]** We also have some new and forthcoming tech here being **[00:33:44 – 00:33:50]** able to look inside, as we create batches of requests from the SCADA **[00:33:50 – 00:33:54]** client to the server, being able to analyze those and collect a bunch **[00:33:54 – 00:33:57]** of different stats for those, be able to see the effect of locality **[00:33:57 – 00:34:01]** as you have different batches, that you make fewer requests **[00:34:01 – 00:34:04]** and it's less bright yellow there. **[00:34:04 – 00:34:07]** Also, being able to look at **[00:34:07 – 00:34:11]** online actively, we took a tiny data set and said, hey, **[00:34:11 – 00:34:16]** if you were to overfetch, if the IO size were to be, say, 4 megabytes **[00:34:16 – 00:34:20]** instead of 512 bytes, how much RAF would you get and how much spatial **[00:34:20 – 00:34:23]** locality is there throughout the lifetime of the application? **[00:34:23 – 00:34:25]** And so we can start to look at those things. **[00:34:25 – 00:34:29]** And again, great contributions from Phil Carns and Javier **[00:34:29 – 00:34:31]** and Alex for that. **[00:34:31 – 00:34:34]** So, I'll turn this back over to Corey as we start to look **[00:34:35 – 00:34:38]** at the success criteria for being able to do a vector database build. **[00:34:38 – 00:34:39]** Awesome. Thank you, CJ. **[00:34:39 – 00:34:44]** So, yeah, I guess in order for us to... **[00:34:45 – 00:34:46]** To be able to evaluate this at scale, we need **[00:34:47 – 00:34:48]** some success criteria. **[00:34:48 – 00:34:49]** We need to know what we're evaluating. **[00:34:49 – 00:34:52]** And so, just doing a quick run-through of the criteria **[00:34:52 – 00:34:54]** that we would tend to use. **[00:34:55 – 00:34:56]** Well, first of all, we're talking about, you know, **[00:34:56 – 00:34:58]** tomorrow's scales, right? **[00:34:58 – 00:35:01]** We know that 90% of all new data is unstructured. **[00:35:01 – 00:35:06]** Most organizations can't tap into but, like, 10% or less of that. **[00:35:06 – 00:35:08]** We are seeing the writing on the wall. **[00:35:08 – 00:35:10]** 100 billion vectors is being discussed on the weekly now. **[00:35:10 – 00:35:13]** Three years ago it was a billion vectors, we're up to 100 billion. **[00:35:13 – 00:35:17]** I'm hearing trillion now discussed at least a few times a month now. **[00:35:17 – 00:35:20]** A lot of the biggest organizations have that scale problem. **[00:35:20 – 00:35:23]** We measure quality using recall, which is essentially if I were **[00:35:23 – 00:35:27]** to have an exact nearest neighbors, what does the approximate nearest **[00:35:27 – 00:35:28]** neighbors look like when compared? **[00:35:28 – 00:35:29]** How many neighbors did I find? **[00:35:29 – 00:35:35]** Right, it's oftentimes not, order is not taking an account to that. **[00:35:35 – 00:35:37]** The robustness, right? So measuring a cherry-picked **[00:35:37 – 00:35:40]** set of parameters and saying I've got great performance **[00:35:40 – 00:35:41]** is not extremely convincing, right? **[00:35:41 – 00:35:43]** These are machine learning models. **[00:35:43 – 00:35:45]** I can tell you that I trained a linear regression model **[00:35:45 – 00:35:48]** in zero time, but if I don't tell you that it predicted **[00:35:48 – 00:35:51]** the wrong thing 100% of the time, that is completely useless, right? **[00:35:51 – 00:35:54]** I need to vary the parameters, and I need to go through what **[00:35:54 – 00:35:57]** we call a Pareto, a grid search of parameters in order to really **[00:35:58 – 00:36:02]** show you, okay, across all of these quality levels and for the best **[00:36:02 – 00:36:05]** search latency, I see that this is consistently the best solution. **[00:36:05 – 00:36:09]** Obviously, being able to meet those performance SLA **[00:36:09 – 00:36:12]** So it's great to have a single cherry-picked example, but **[00:36:12 – 00:36:13]** we really need to be robust here. **[00:36:13 – 00:36:17]** We need to do our diligence and make sure that we are **[00:36:17 – 00:36:19]** being fair, especially to the CPU. **[00:36:19 – 00:36:23]** So, cost-effectiveness, I mentioned earlier, TCO is a big deal. **[00:36:23 – 00:36:25]** I already mentioned this one, so in the interest of time, I'm **[00:36:25 – 00:36:27]** just going to really brush over it. **[00:36:27 – 00:36:28]** I do a brute force. **[00:36:28 – 00:36:31]** I get the nearest neighbors. I do my approximate algorithm. **[00:36:31 – 00:36:32]** I get the nearest neighbors. I compare them. **[00:36:32 – 00:36:35]** How many from the approximate did I get from the brute force? **[00:36:35 – 00:36:37]** That's recall. Simple division. **[00:36:37 – 00:36:39]** This is the Pareto front that I referred to. **[00:36:39 – 00:36:42]** So, this is oftentimes, and needs to be really, it's not **[00:36:42 – 00:36:42]** something that you do every day. **[00:36:43 – 00:36:45]** That's something that I can do a hyperparameter optimization on. **[00:36:45 – 00:36:47]** A hyperparameter optimization is something a customer would **[00:36:47 – 00:36:50]** take to production with because they know they have an SLA and they **[00:36:50 – 00:36:51]** know they need to meet that SLA. **[00:36:51 – 00:36:53]** When I'm comparing something side by side, it needs to **[00:36:53 – 00:36:55]** be apples to apples, which means I need to do a range **[00:36:56 – 00:36:58]** of different parameter settings. **[00:36:58 – 00:37:00]** These benchmarks do take a while. **[00:37:00 – 00:37:03]** I take basically the optimum tradeoff between quality and **[00:37:03 – 00:37:07]** search performance, and that's how I construct this Pareto front. **[00:37:08 – 00:37:11]** And you can see the expectation here is that, as I mentioned **[00:37:11 – 00:37:15]** earlier, the trade-offs, as I get higher quality, my search **[00:37:15 – 00:37:18]** latency and my throughput are going to drop. **[00:37:18 – 00:37:21]** My latency is going to increase. **[00:37:21 – 00:37:23]** So, how do we measure index build then, right? **[00:37:23 – 00:37:27]** Which one out of all of these points on this Pareto front **[00:37:27 – 00:37:28]** is fair to report? **[00:37:28 – 00:37:31]** We tend to look at recall in terms of windows, right? **[00:37:31 – 00:37:34]** I might care about 95% to 99% recall. **[00:37:34 – 00:37:37]** I might care about 85% to 90% recall. **[00:37:37 – 00:37:39]** Most people are above 90%. **[00:37:39 – 00:37:41]** Some people can get away with 70%. **[00:37:41 – 00:37:42]** It depends on the use case. **[00:37:42 – 00:37:44]** There's some genomics use cases where they can get really, **[00:37:44 – 00:37:49]** really horrible recall, but still find its success acceptable. **[00:37:49 – 00:37:52]** What we do is we average the build times in each of these **[00:37:52 – 00:37:56]** windows and we try to get kind of a summary of what you can generally **[00:37:56 – 00:38:01]** expect a build time to take in each of these recall windows. **[00:38:01 – 00:38:03]** This is kind of part of a methodology that we had **[00:38:03 – 00:38:05]** to build ourselves. **[00:38:05 – 00:38:07]** If you look at all the standardized benchmarking tools, there **[00:38:07 – 00:38:09]** really is no standard, but if you look at all the attempts **[00:38:09 – 00:38:13]** to standardize a benchmarking tool for vector search, you **[00:38:13 – 00:38:15]** notice none of them really put anything into build time, **[00:38:15 – 00:38:17]** but at these scales, you need to. **[00:38:17 – 00:38:18]** Ingest is a big deal. **[00:38:18 – 00:38:20]** If I can't get those vectors in there in a reasonable amount **[00:38:20 – 00:38:24]** of time, that is completely useless to my use case. **[00:38:29 – 00:38:34]** Alright, so that being said, there are a lot of 1 million **[00:38:34 – 00:38:38]** scale data sets that you can download from the internet. **[00:38:38 – 00:38:40]** There's a lot of 10 million, there's a lot of 100 million, **[00:38:40 – 00:38:42]** there's even a couple billion. **[00:38:42 – 00:38:45]** I don't see 100 billion scale data sets just sitting around. **[00:38:45 – 00:38:49]** And even if they were, they would take a long time to download, **[00:38:49 – 00:38:51]** they would take a long time to transfer to different hardware, **[00:38:51 – 00:38:53]** they take a lot of storage. **[00:38:53 – 00:38:56]** And so we are putting a lot of effort into generating synthetic **[00:38:56 – 00:39:00]** data that can actually mock the distributions of real-world data. **[00:39:00 – 00:39:03]** And so what you're seeing here is we took 10 million samples **[00:39:03 – 00:39:07]** from a common data set, we modeled that so that we could mimic it, **[00:39:07 – 00:39:10]** and then we scaled the data set up. **[00:39:10 – 00:39:12]** You know, by generating the synthetic data. **[00:39:12 – 00:39:14]** And then we compared the Pareto across a wide range **[00:39:14 – 00:39:19]** of parameters, and we find that those Pareto's almost overlap. **[00:39:19 – 00:39:22]** This is actually, this is a huge improvement over what **[00:39:22 – 00:39:25]** happened when we just tried generating naive data. **[00:39:25 – 00:39:26]** This is looking really promising. **[00:39:26 – 00:39:27]** We're seeing this as a really good approach. **[00:39:27 – 00:39:29]** We don't actually have to materialize the data to be **[00:39:29 – 00:39:31]** able to do this at any point. **[00:39:31 – 00:39:35]** I'm going to skip ahead. So we did, you know, a billion by 1024 here. **[00:39:35 – 00:39:38]** We find that a lot of the time is spent doing things **[00:39:38 – 00:39:42]** that aren't actually generating the data, which is nice. **[00:39:42 – 00:39:44]** I'm willing to talk to anybody about this after the talk. **[00:39:44 – 00:39:46]** I know we're running short on time. **[00:39:46 – 00:39:50]** I do want to show the ground truth generation. **[00:39:50 – 00:39:53]** We are able to, in a scalable way, compute the ground truth **[00:39:53 – 00:39:57]** on 10 billion, 100 billion vectors extremely fast. **[00:39:57 – 00:39:59]** We do an IVF-like approach to do this. **[00:39:59 – 00:40:02]** But what's more is that it's hard to be able to materialize **[00:40:02 – 00:40:05]** the 100 billion dataset before you do the ground truth generation. **[00:40:05 – 00:40:07]** And so we can actually do that ground truth generation **[00:40:07 – 00:40:08]** without materializing the dataset. **[00:40:08 – 00:40:10]** We can do it in place. It's pretty neat. **[00:40:10 – 00:40:11]** I'm happy to talk to anybody about this. **[00:40:11 – 00:40:14]** But we want to try to standardize a benchmark, and we want to **[00:40:14 – 00:40:18]** start working with industry and academia to do this. **[00:40:18 – 00:40:21]** And with that, I will leave it to- Thank you for joining this session. **[00:40:21 – 00:40:25]** If you're staying in the room for the next session, please remain **[00:40:25 – 00:40:29]** in your seat and have your badge ready to be scanned by our team. **[00:40:29 – 00:40:32]** Break out of the bounds that we've had to be able to work on these **[00:40:32 – 00:40:33]** new and emerging applications. **[00:40:33 – 00:40:36]** So thank you very much. We'll be around for questions. **[00:40:37 – 00:40:37]** Thank you everybody for coming.