# From Analytics to AI in Highly Regulated Industries: Accelerating Financial Services

## Abstract

Key takeaways Explore how leading financial institutions are using GPU acceleration to process terabytes of data—from sub-second query execution to training transaction foundation models at scale. Understand why identifying which workloads benefit most from GPU acceleration is as important as the acceleration itself, and how financial institutions approach this in practice. Learn what it takes to deploy GPU-accelerated workloads in highly regulated financial services environments, from data governance to production infrastructure.

## AI Summary

_No AI summary available._

## Transcript

**[00:00:05 – 00:00:10]** And the moderator for the session, who I will introduce here,

**[00:00:10 – 00:00:13]** is my coworker, Yuliana Zamora.

**[00:00:13 – 00:00:17]** She is a senior developer relations manager for capital

**[00:00:17 – 00:00:21]** markets here at NVIDIA, so she will be moderating this session.

**[00:00:21 – 00:00:25]** And please welcome her, and she will introduce the panel.

**[00:00:32 – 00:00:33]** Thank you, everyone.

**[00:00:33 – 00:00:36]** Welcome to GTC Berlin, and welcome to the session.

**[00:00:36 – 00:00:38]** Thank you so much for being here.

**[00:00:38 – 00:00:42]** And so, in financial services and regulated sectors, moving

**[00:00:42 – 00:00:47]** from traditional analytics to AI in production is more

**[00:00:47 – 00:00:49]** than just a technical update.

**[00:00:49 – 00:00:52]** It's an entire architectural shift.

**[00:00:52 – 00:00:57]** We are having to deal with massive amounts of transactional

**[00:00:57 – 00:01:02]** data, strict compliance, data sovereignty, and the

**[00:01:02 – 00:01:05]** auditability requirements.

**[00:01:05 – 00:01:10]** Given that, we have to change the entire pipeline, and we

**[00:01:10 – 00:01:15]** have to understand where all the technical updates are needed.

**[00:01:15 – 00:01:21]** Today, we are welcoming three AI and engineering leads from...

**[00:01:21 – 00:01:35]** First, I'm going to introduce you to Temi Afeye, a data scientist

**[00:01:35 – 00:01:36]** from Lloyds Banking Group.

**[00:01:36 – 00:01:40]** Please, everyone, welcome Temi.

**[00:01:46 – 00:01:47]** My name is Temi.

**[00:01:47 – 00:01:51]** I work at Temi Lloyds Banking Group.

**[00:01:51 – 00:01:55]** So today I'm just going to talk a bit about some work

**[00:01:55 – 00:01:57]** we've been doing.

**[00:01:57 – 00:02:01]** So really, we want to talk about data processing at scale, and

**[00:02:01 – 00:02:04]** I just thought it was nice to talk about it from the perspective of a

**[00:02:04 – 00:02:07]** use case we're actually working on.

**[00:02:07 – 00:02:11]** So my team is here, Vivek, Usman, and Siu.

**[00:02:11 – 00:02:15]** We've been working on building a transaction foundation model, and

**[00:02:15 – 00:02:19]** the reason we've been doing that is because, I mean, first of all, it's

**[00:02:19 – 00:02:22]** becoming a thing in the industry.

**[00:02:22 – 00:02:25]** Revolut's done it, Neobank's done it, and we're beginning

**[00:02:25 – 00:02:29]** to see, like, really good usage and coverage from it.

**[00:02:29 – 00:02:32]** So, essentially, what I was thinking is, can we

**[00:02:32 – 00:02:37]** take behavioural transaction data, can we learn from

**[00:02:37 – 00:02:41]** customers, can we look at random transactions and order

**[00:02:41 – 00:02:45]** transactions like income, shopping?

**[00:02:46 – 00:02:50]** Can we begin to learn those kind of representations and then turn them

**[00:02:50 – 00:02:53]** into simple customer embeddings?

**[00:02:53 – 00:02:57]** And then also, can we apply this down to downstream tasks

**[00:02:57 – 00:03:02]** like borrowing propensity, and retention, and also life events?

**[00:03:02 – 00:03:06]** So we've built what is our group's first foundation model

**[00:03:06 – 00:03:07]** that will be named ATUM.

**[00:03:07 – 00:03:11]** But I'm not here to talk about ATUM, which is obviously really

**[00:03:11 – 00:03:14]** great, as we're seeing; I'm here to talk about scaling

**[00:03:14 – 00:03:18]** the transactions and the amount of data that we use to train it.

**[00:03:18 – 00:03:22]** So we've trained ATUM on about 18 billion transactions, covering

**[00:03:22 – 00:03:25]** about 28 million customers.

**[00:03:25 – 00:03:28]** That's about 216 billion fuel tokens.

**[00:03:28 – 00:03:32]** So to do this, first questions we have to ask ourselves is,

**[00:03:32 – 00:03:36]** how do you go from raw transactions to an aligned training window?

**[00:03:36 – 00:03:40]** How do we preserve customer order and tokenize our rules while we

**[00:03:40 – 00:03:42]** process these records in parallel?

**[00:03:42 – 00:03:47]** Again, that's a huge challenge because you have to process

**[00:03:47 – 00:03:51]** customer sequences, and you have to keep those ordered,

**[00:03:51 – 00:03:54]** and you have to train on that.

**[00:03:54 – 00:03:57]** I want to take us a bit into our simple workflow and how

**[00:03:57 – 00:04:00]** we've done that, and I'm sure the other speakers will talk

**[00:04:00 – 00:04:02]** a lot more in a larger picture.

**[00:04:02 – 00:04:07]** So currently, our stock is on Google Cloud.

**[00:04:07 – 00:04:10]** In choosing the tools that we use, we have to think about speed,

**[00:04:10 – 00:04:14]** we have to think about scalability, we have to think about simplicity,

**[00:04:14 – 00:04:18]** and we have to use tools that are familiar to us as developers.

**[00:04:18 – 00:04:23]** So we generally start from our BigQuery table with

**[00:04:23 – 00:04:25]** our BigQuery API.

**[00:04:25 – 00:04:28]** We write a lot of customer transaction files into GCS

**[00:04:28 – 00:04:32]** packets, and then we keep those in buckets together.

**[00:04:32 – 00:04:35]** Now, the big task of what happens inside one GPU is,

**[00:04:35 – 00:04:40]** how do we actually take one person, a customer's history, and put those

**[00:04:40 – 00:04:43]** into simple and ordered windows?

**[00:04:43 – 00:04:49]** So generally, we use RAPIDS library and open source

**[00:04:49 – 00:04:50]** libraries like CuPy.

**[00:04:50 – 00:04:54]** So in here, in the simple GPU worker here, we have to do a

**[00:04:54 – 00:04:56]** whole lot of cleaning and ordering.

**[00:04:56 – 00:04:59]** You could do that with, like, you know, pandas, and then

**[00:04:59 – 00:05:04]** obviously polars, but in order to actually scale this and accelerate

**[00:05:04 – 00:05:07]** our scale, we do that with QDF.

**[00:05:07 – 00:05:11]** So essentially, using the same APIs that we are familiar

**[00:05:11 – 00:05:13]** with, but parallelizing it.

**[00:05:13 – 00:05:18]** Then we have to encode and then tokenize the transactions,

**[00:05:18 – 00:05:22]** and that actually also has a whole lot of calculative operations with

**[00:05:22 – 00:05:26]** it, so, for instance, we have to take different amounts, and we have

**[00:05:26 – 00:05:28]** to put those into different bins.

**[00:05:28 – 00:05:32]** We have to hash the IDs so that we can always have

**[00:05:32 – 00:05:35]** consistent transactions assigned to the same customer.

**[00:05:35 – 00:05:38]** Then we have to also create time and gap features that help

**[00:05:38 – 00:05:42]** the model learn a lot about the transactions that we're building.

**[00:05:43 – 00:05:46]** And so, again, typically, if you wanted to do this with

**[00:05:46 – 00:05:52]** a CPU backend, you use the familiar non-Py that we're familiar with.

**[00:05:52 – 00:05:57]** But then, we also tried to do this with CuPy, a whole

**[00:05:57 – 00:06:00]** lot of mathematical operations also parallelized so we can

**[00:06:00 – 00:06:03]** do so many transactions at once.

**[00:06:04 – 00:06:07]** And then, at the same time, we use CuPy to actually assemble

**[00:06:07 – 00:06:11]** the windows, and then in one GPU, we have one customer window.

**[00:06:11 – 00:06:14]** And then second thing for here is, for us, is simplicity

**[00:06:14 – 00:06:16]** of the tools that we use.

**[00:06:16 – 00:06:20]** We're very familiar with, like, the Pandas API, so it makes it easier

**[00:06:20 – 00:06:27]** to actually use our QDF, the same APIs but exposed at the backend,

**[00:06:27 – 00:06:29]** and simple code here and there.

**[00:06:29 – 00:06:33]** Also with, like, QPI, same backend, same processes, same

**[00:06:33 – 00:06:39]** calculations, but parallelized at a very good speed.

**[00:06:39 – 00:06:42]** And, for instance, an assumed example, you can compute from,

**[00:06:42 – 00:06:44]** like, 40 seconds to 10 seconds.

**[00:06:44 – 00:06:49]** That's a 40 second, four times increased speed, and actually

**[00:06:49 – 00:06:54]** getting a job that would take a long time in doing this.

**[00:06:54 – 00:06:57]** Then the next challenge for us is, how do you scale this?

**[00:06:57 – 00:07:01]** So now we've seen how it works in one GPU.

**[00:07:01 – 00:07:04]** So we have to do this for about 18 billion transactions,

**[00:07:04 – 00:07:06]** and we have to do this on scale.

**[00:07:06 – 00:07:10]** So currently, internally, how we run this is, so, how

**[00:07:10 – 00:07:15]** do we get a pipeline to allocate and And orchestrate the job?

**[00:07:15 – 00:07:21]** So we use Vertex AI custom training job defined by Kubeflow pipelines.

**[00:07:21 – 00:07:25]** Then we use RAID to orchestrate the jobs.

**[00:07:25 – 00:07:30]** So what RAID does is, it gives each GPU worker the simple buckets.

**[00:07:30 – 00:07:34]** And then, so, we have a whole stretch of GPU workers working

**[00:07:34 – 00:07:38]** in tandem to actually parallelize this and then use this, and

**[00:07:38 – 00:07:41]** use this across multiple workers.

**[00:07:41 – 00:07:44]** So at the end of the day, we have, we have written about

**[00:07:44 – 00:07:48]** 18 billion transactions, converting them to about 215 billion tokens,

**[00:07:48 – 00:07:52]** and then written them back into our storage buckets with a simple

**[00:07:52 – 00:07:54]** tokenizer state and manifest.

**[00:07:54 – 00:07:57]** So in doing that, we have, we have approached this the

**[00:07:57 – 00:07:59]** same way we've been trying to approach all the problems

**[00:07:59 – 00:08:03]** we'll be solving, first with speed, second with scalability,

**[00:08:03 – 00:08:05]** taught with simple tools.

**[00:08:05 – 00:08:08]** Simplicity is so important to us, and the NVIDIA RAPID

**[00:08:08 – 00:08:10]** library gives us access to that.

**[00:08:10 – 00:08:15]** We can use the same tools that we are used to, but then we can apply

**[00:08:15 – 00:08:18]** them on simple or multiple threads.

**[00:08:18 – 00:08:22]** And really, I've told, like, a huge story in less than five

**[00:08:22 – 00:08:30]** to six minutes, but I've just had a bit more links here just to say

**[00:08:30 – 00:08:35]** about Atom, and then I mentioned on the NVIDIA's ecosystem, and also

**[00:08:35 – 00:08:41]** just some links around background and transfer and buffer sharing.

**[00:08:41 – 00:08:43]** Thanks. That's really it.

**[00:08:51 – 00:08:55]** Thank you so much, Sami; that was a great presentation.

**[00:08:55 – 00:08:57]** Our next presenter is Valeria Verzi.

**[00:08:57 – 00:08:59]** She comes from Klarna. She's the head of Next

**[00:09:00 – 00:09:01]** Gen Modeling Techniques.

**[00:09:01 – 00:09:05]** Everyone, please welcome Valeria.

**[00:09:08 – 00:09:11]** Hello, everyone. Thanks for having me.

**[00:09:11 – 00:09:13]** Thanks, Timmy, for introducing some of the tools I'm

**[00:09:13 – 00:09:15]** going to talk about.

**[00:09:15 – 00:09:20]** So, before starting, I want to talk just a little bit about Klarna.

**[00:09:20 – 00:09:25]** Many of you probably know Klarna, but I want to give you the scale

**[00:09:25 – 00:09:26]** of what I'm talking about today.

**[00:09:26 – 00:09:32]** So, Klarna is an AI-powered payment network and shopping ecosystem.

**[00:09:32 – 00:09:36]** We were recognized with banking licenses in 2017, so we're

**[00:09:36 – 00:09:37]** actually a bank now.

**[00:09:37 – 00:09:41]** We have more than 120 million consumers and

**[00:09:41 – 00:09:44]** 1.2 million retail partners.

**[00:09:45 – 00:09:49]** We handle more than 3.8 million transactions per day.

**[00:09:49 – 00:09:52]** Remember this number because we're going to get back to this.

**[00:09:52 – 00:09:56]** Each one of these transactions carries a lot of data, so

**[00:09:56 – 00:09:58]** leaves a lot of data behind.

**[00:09:58 – 00:10:02]** And when you have to make the best out of this amount

**[00:10:02 – 00:10:04]** of data, you need to have the...

**[00:10:04 – 00:10:07]** They are the correct computational tools,

**[00:10:07 – 00:10:10]** and that's why I'm here today.

**[00:10:10 – 00:10:13]** I want to tell you a brief story about how we manage

**[00:10:13 – 00:10:17]** this computational demand of this amount of data, how

**[00:10:17 – 00:10:22]** we are using GPUs, and how NVIDIA helped us make this possible.

**[00:10:22 – 00:10:23]** So here we are.

**[00:10:23 – 00:10:27]** So again, 3.8 million transactions per day.

**[00:10:27 – 00:10:32]** If we do a bit of math, that's 44 transactions per second.

**[00:10:32 – 00:10:34]** That's a huge number.

**[00:10:35 – 00:10:38]** Obviously, when we train models, we don't use all

**[00:10:38 – 00:10:39]** of these transactions.

**[00:10:39 – 00:10:41]** It wouldn't make sense.

**[00:10:41 – 00:10:43]** But we train model in production.

**[00:10:43 – 00:10:48]** What does it mean? It means that the models need to see

**[00:10:48 – 00:10:49]** a lot of this data.

**[00:10:49 – 00:10:52]** We want the model to be sharp, to be reliable.

**[00:10:52 – 00:10:58]** We want them to be compliant, and all this puts constraints

**[00:10:58 – 00:10:59]** into the process.

**[00:11:00 – 00:11:03]** And to meet these constraints, we need the right infrastructure.

**[00:11:03 – 00:11:08]** If you don't have the right infrastructure, then either

**[00:11:08 – 00:11:13]** you have to heavily downsample your data, obviously losing in richness

**[00:11:13 – 00:11:19]** of your model and in fairness of your model, or you have to give

**[00:11:19 – 00:11:23]** up on the number of experiments and iteration that you can do on the

**[00:11:23 – 00:11:27]** model, most likely a bit of both.

**[00:11:27 – 00:11:31]** So this is the reason why we started looking into GPUs

**[00:11:31 – 00:11:36]** and into scaling our code for model training.

**[00:11:36 – 00:11:40]** And I want to give you the first lesson that we've learned

**[00:11:40 – 00:11:45]** in this process, that is, when we work on acceleration,

**[00:11:45 – 00:11:49]** the first thing you need to look at is what you want to accelerate.

**[00:11:49 – 00:11:52]** Not everything is meant to be on a GPU.

**[00:11:53 – 00:11:57]** And so we profile our code, and obviously the first answer

**[00:11:57 – 00:11:59]** was very straightforward.

**[00:11:59 – 00:12:02]** The first thing that you want to optimize when you work

**[00:12:02 – 00:12:05]** on a machine learning, AI model, is the training code.

**[00:12:06 – 00:12:09]** The training part is the part that is computationally more

**[00:12:09 – 00:12:14]** expensive, where the workload is heavier, of course.

**[00:12:14 – 00:12:18]** And for this reason, by design, our training code is on GPU.

**[00:12:18 – 00:12:22]** What it wasn't, it was not parallelized on GPU.

**[00:12:22 – 00:12:25]** So we did quite some work.

**[00:12:25 – 00:12:32]** We implemented DDP on our training code and cleaned it up, and we were

**[00:12:32 – 00:12:37]** able to achieve our training cycle on four or eight eGPU instances

**[00:12:38 – 00:12:44]** with 95%, 96% of utilization on GPUs, so super optimized.

**[00:12:44 – 00:12:51]** Next up, we moved from NVIDIA A100 GPUs to the newest, well,

**[00:12:51 – 00:12:54]** not newest anymore from the speech yesterday from Johnson,

**[00:12:54 – 00:12:57]** but what we thought it was the latest generation back, well,

**[00:12:57 – 00:13:04]** NVIDIA GPUs, and we measured again.

**[00:13:05 – 00:13:07]** Uplift, 1.03.

**[00:13:10 – 00:13:13]** Because our model we tested, it was the small version.

**[00:13:13 – 00:13:16]** It was too small to actually scale on GPU.

**[00:13:16 – 00:13:20]** But our target was to have an infrastructure that was

**[00:13:20 – 00:13:22]** ready for scaling.

**[00:13:22 – 00:13:26]** So as soon as we started scaling 40x more data, 14x larger

**[00:13:26 – 00:13:32]** model, we saw nearly 2% faster model training on the back-well

**[00:13:32 – 00:13:36]** GPUs, which is a huge achievement.

**[00:13:36 – 00:13:38]** Why was this possible?

**[00:13:38 – 00:13:42]** Because the training, we said, it was designed to run on GPUs, which

**[00:13:42 – 00:13:47]** means that we had a heavy data processing step that was preparing

**[00:13:47 – 00:13:52]** the data to be dense and ready for GPU mathematical computation.

**[00:13:52 – 00:13:57]** And so we profiled again, and obviously, the next

**[00:13:57 – 00:14:03]** bottleneck was optimizing the data preprocessing step.

**[00:14:03 – 00:14:09]** That step, it was built on different pillars.

**[00:14:09 – 00:14:13]** First pillar is, take your data, crunch your data, try

**[00:14:13 – 00:14:15]** to make them dense enough.

**[00:14:15 – 00:14:17]** That's not only on CPU.

**[00:14:17 – 00:14:21]** There's nothing that a GPU can do on that, on reading,

**[00:14:21 – 00:14:23]** writing, flattening your data.

**[00:14:23 – 00:14:27]** GPUs are very good in parallelizing and math operation.

**[00:14:27 – 00:14:30]** They're not built for reading and writing or flattening.

**[00:14:30 – 00:14:35]** The second step, though, it was cleaning your data,

**[00:14:35 – 00:14:39]** preparing, creating features, tokenizing the data.

**[00:14:39 – 00:14:42]** That's absolutely a GPU work, and that's where we

**[00:14:42 – 00:14:48]** implemented the NVIDIA QDF, and we saw a 2.4 improvement

**[00:14:48 – 00:14:57]** in performance of time of our preprocessing step, and so on.

**[00:14:57 – 00:15:03]** So every time we were implementing optimization, we kept on

**[00:15:03 – 00:15:09]** measuring, profiling, and improving again and again.

**[00:15:09 – 00:15:13]** All this in a regulated environment, which means

**[00:15:13 – 00:15:17]** a lot of things, but two things

**[00:15:17 – 00:15:20]** that impacted on our project.

**[00:15:20 – 00:15:26]** One is that we want to be aware of how data is flowing

**[00:15:26 – 00:15:27]** through our model.

**[00:15:27 – 00:15:32]** And when you move from CPU to GPUs, the math behind remains the same,

**[00:15:32 – 00:15:36]** but the way math is calculated, it's completely different.

**[00:15:36 – 00:15:39]** CPUs are sequential; GPUs are not.

**[00:15:39 – 00:15:43]** And that's why they're so powerful, but that's also why the by-to-by

**[00:15:43 – 00:15:45]** comparison with CPU doesn't hold.

**[00:15:45 – 00:15:50]** So every time we move from CPU to GPU, or from one type of GPU

**[00:15:50 – 00:15:56]** to another type, we made sure that the outputs were coherent, that

**[00:15:56 – 00:16:00]** we knew what was going on in our data and how the data was flying.

**[00:16:00 – 00:16:02]** Second impact of regulation.

**[00:16:02 – 00:16:08]** Our models are compliant by design, which means, among all,

**[00:16:08 – 00:16:12]** that they are fully reproducible, because otherwise you can't

**[00:16:12 – 00:16:13]** put them in production.

**[00:16:13 – 00:16:14]** And that's it.

**[00:16:14 – 00:16:20]** So we developed this deterministic approach, which basically switched

**[00:16:20 – 00:16:22]** off mathematical approximation.

**[00:16:22 – 00:16:24]** Mathematical approximation is something very powerful

**[00:16:25 – 00:16:28]** in GPUs because it allows you to iterate very, very fast.

**[00:16:28 – 00:16:31]** For experimentation, great, that was working.

**[00:16:31 – 00:16:34]** But then, when we wanted to compare the last models and

**[00:16:34 – 00:16:37]** put them into production, we switched on that determinism.

**[00:16:37 – 00:16:40]** Mathematical approximation is off; we take a bit more time,

**[00:16:40 – 00:16:43]** but we have a production-ready code at the end that we can put

**[00:16:43 – 00:16:46]** into production with NVIDIA Triton.

**[00:16:46 – 00:16:51]** And for closing, I want to leave you just with one final

**[00:16:51 – 00:16:55]** number that, for me, is the number of this project.

**[00:16:55 – 00:17:02]** We started this taking more than 20 hours to do one experiment.

**[00:17:02 – 00:17:06]** It was crazy. So one experiment for us, it was five iterations because

**[00:17:06 – 00:17:10]** we want to be compliant, but more than 20 hours for a team, it means,

**[00:17:10 – 00:17:14]** for my team, meant we have to launch the job, wait for two days,

**[00:17:14 – 00:17:19]** wait for the job to finish, hoping nothing breaks in the middle, and

**[00:17:19 – 00:17:23]** then coming back and see results probably on the Monday morning.

**[00:17:23 – 00:17:25]** Now, it takes less than 10 hours.

**[00:17:25 – 00:17:30]** It means that we end our day, we launch our job, we wake

**[00:17:30 – 00:17:34]** up in the morning, and we review all together the results.

**[00:17:34 – 00:17:40]** So, thanks, NVIDIA, for making this possible, and thanks for listening.

**[00:17:45 – 00:17:46]** Thank you so much, Valeria.

**[00:17:46 – 00:17:49]** Great talk.

**[00:17:51 – 00:17:54]** Our next speaker is Reynolds Crumins, Principal

**[00:17:54 – 00:17:56]** Engineer from HSBC.

**[00:17:56 – 00:18:00]** Please, everyone, welcome Reynolds.

**[00:18:05 – 00:18:06]** Hi, everyone.

**[00:18:07 – 00:18:08]** My name is Reynolds.

**[00:18:08 – 00:18:14]** I work at HSBC within our CTO data technology.

**[00:18:14 – 00:18:19]** Today, I'll be talking about how we can actually get more value

**[00:18:19 – 00:18:24]** of the GPUs when it comes to our data processing and the pipelines.

**[00:18:25 – 00:18:29]** One of the things that we observed, so, GPUs have a

**[00:18:29 – 00:18:34]** cycle of the usage within the kind of AI workloads, but

**[00:18:34 – 00:18:37]** one of the most expensive GPUs is the one that's being underutilized.

**[00:18:37 – 00:18:41]** So this is where we started to look into opportunities

**[00:18:41 – 00:18:46]** on how we can actually use them during the off-peak hours.

**[00:18:46 – 00:18:50]** So this is where we started to look into some of our data products and

**[00:18:50 – 00:18:54]** the data pipelines that get run.

**[00:18:54 – 00:18:57]** So initially, we did a classic journey of looking at the industry

**[00:18:57 – 00:19:02]** benchmarks around some of the existing POCs, like TPC-H standard

**[00:19:02 – 00:19:08]** datasets, and we essentially reproduce all of these internally.

**[00:19:08 – 00:19:11]** So NVIDIA was kind enough to help us to get some of

**[00:19:11 – 00:19:13]** the GPUs allocated.

**[00:19:13 – 00:19:15]** As you all know, they're in quite high demand.

**[00:19:15 – 00:19:22]** And we run all of our experiments on our managed GKE within HSBC.

**[00:19:22 – 00:19:27]** So we are GCP driven for this specific use case.

**[00:19:27 – 00:19:31]** And what we afterwards did, we actually went to find an

**[00:19:31 – 00:19:35]** actual use case within a bank where this can be applied because, as

**[00:19:35 – 00:19:40]** the previous speakers pointed out, it is the right tool for the job.

**[00:19:40 – 00:19:43]** So not all of your queries can be kind of lifted and

**[00:19:43 – 00:19:47]** shifted from the CPU to GPU.

**[00:19:47 – 00:19:50]** And this is where I want to kind of go on to the third point,

**[00:19:50 – 00:19:55]** one step further, where what you really want to do is to actually

**[00:19:55 – 00:19:58]** focus on the metadata, because that's what matters for what

**[00:19:58 – 00:20:02]** can be accelerated and what can't.

**[00:20:02 – 00:20:03]** And here are the results.

**[00:20:03 – 00:20:09]** So what you see, the main figure there, is the 5x improvement.

**[00:20:09 – 00:20:15]** So this is what we actually got around our payments data use case.

**[00:20:15 – 00:20:19]** So we had about a couple of hundred million transactions

**[00:20:19 – 00:20:24]** that we essentially compared on the GPU versus on the GPU.

**[00:20:24 – 00:20:27]** And in some instances, we actually were able to see

**[00:20:27 – 00:20:31]** 10x to 12x enhancement of this.

**[00:20:31 – 00:20:37]** And our baseline figure that we looked into is that 5.7x

**[00:20:37 – 00:20:38]** figure that you can see.

**[00:20:38 – 00:20:40]** And in terms of our testing benchmark, it's essentially

**[00:20:41 – 00:20:46]** the same infrastructure with a GPU acceleration and without.

**[00:20:46 – 00:20:51]** So again, the advantage starts with the workload selection.

**[00:20:51 – 00:20:54]** So first of all, what you want to really look into are

**[00:20:54 – 00:20:59]** your queries, how well written they are, what they use.

**[00:20:59 – 00:21:01]** Are they properly partitioned?

**[00:21:01 – 00:21:02]** Can they be paralysed?

**[00:21:02 – 00:21:04]** And etc.

**[00:21:04 – 00:21:07]** And also, the data types, formats, and all the

**[00:21:07 – 00:21:09]** nitty-gritty bits do matter.

**[00:21:09 – 00:21:12]** So, in our instance, we actually converted a number of our datasets

**[00:21:12 – 00:21:16]** into Apache Iceberg format, which is what we used internally

**[00:21:16 – 00:21:19]** to run all of these benchmarks.

**[00:21:19 – 00:21:23]** And, well, last but not least is the memory fit.

**[00:21:23 – 00:21:30]** So we initially started our experimentation with the L4s,

**[00:21:30 – 00:21:34]** which have about 24 gigs of VRAM, if I recall correctly.

**[00:21:34 – 00:21:37]** And then we sort of noticed some of the queries didn't

**[00:21:37 – 00:21:39]** work as well as we expected.

**[00:21:39 – 00:21:41]** So we did a vertical scaling.

**[00:21:41 – 00:21:45]** So essentially, we switched over to RTS6000s.

**[00:21:45 – 00:21:50]** And all of our experimentation was done in a single query execution.

**[00:21:50 – 00:21:52]** So what does that mean for us in the future?

**[00:21:52 – 00:21:55]** So one of the things that we are looking into is, can

**[00:21:55 – 00:21:58]** we horizontally scale it reliably?

**[00:21:58 – 00:22:03]** And what are the actual gains when we start going to multi-node

**[00:22:03 – 00:22:06]** clusters and et cetera?

**[00:22:07 – 00:22:09]** So this is all great from the engineering point of view,

**[00:22:09 – 00:22:12]** but what does this actually mean for the people who use

**[00:22:12 – 00:22:14]** our data assets and data products?

**[00:22:14 – 00:22:18]** So first of all, there is a user experience opportunity.

**[00:22:18 – 00:22:22]** So when we can accelerate all of these queries, we actually spend

**[00:22:22 – 00:22:27]** less time waiting for the analytics teams to actually see the results.

**[00:22:27 – 00:22:30]** When we have certain data sets which are customer facing or

**[00:22:30 – 00:22:34]** our frontline customers, they can actually get the insights quicker,

**[00:22:34 – 00:22:35]** which is one of the advantages.

**[00:22:35 – 00:22:38]** But there's also the other advantage, which is the kind

**[00:22:38 – 00:22:42]** of FinOps side of things, where we can actually, by

**[00:22:42 – 00:22:47]** paying, let's say, 30% more for the compute, we can actually get about

**[00:22:48 – 00:22:51]** 5x enhancements in certain cases.

**[00:22:51 – 00:22:54]** So again, a lot of this kind of performance comes down

**[00:22:54 – 00:22:58]** to your metadata and the right tool for the job.

**[00:22:58 – 00:23:01]** But the query speedup is just the starting point, right?

**[00:23:01 – 00:23:04]** So we are still quite early on in this journey, but we

**[00:23:04 – 00:23:08]** did run this on a production dataset with the actual data

**[00:23:08 – 00:23:11]** and one of our payments use case.

**[00:23:11 – 00:23:15]** So, given that we are in a regulated industry,

**[00:23:15 – 00:23:17]** correctness matters, right?

**[00:23:17 – 00:23:21]** If we don't get it right, there are certain implications.

**[00:23:21 – 00:23:25]** So, this is where we've spent a lot of time of verifying the

**[00:23:25 – 00:23:28]** results against the CPU reference.

**[00:23:28 – 00:23:32]** And what we also need to do is to check if we go over

**[00:23:32 – 00:23:38]** to the GPU, does that mean our users will have failures,

**[00:23:38 – 00:23:45]** how quickly we can recover, and do we introduce any of the operational

**[00:23:45 – 00:23:48]** issues if we switch over?

**[00:23:48 – 00:23:51]** And, obviously, there is the whole security aspect of this,

**[00:23:51 – 00:23:54]** encryption and et cetera.

**[00:23:54 – 00:23:57]** But last but not least is the economics, right?

**[00:23:57 – 00:24:01]** It has to make financially sense for us to be able to

**[00:24:01 – 00:24:06]** maximize the value of the kind of computes that we apply to this.

**[00:24:06 – 00:24:10]** But I think the key takeaway from my kind of talk is focus

**[00:24:10 – 00:24:13]** on accelerating the right workload, prove the value

**[00:24:13 – 00:24:17]** iteratively, and at the same time apply all the controls.

**[00:24:17 – 00:24:22]** And yeah, because that's what matters in a regulated industry.

**[00:24:29 – 00:24:31]** Thank you so much, Reynolds, let's see.

**[00:24:31 – 00:24:34]** Perfect, I think we have time for one question, two questions?

**[00:24:34 – 00:24:35]** Yeah, yeah, we have time.

**[00:24:35 – 00:24:36]** Okay, perfect.

**[00:24:36 – 00:24:38]** Any questions for any of the speakers?

**[00:24:38 – 00:24:41]** The speakers can come up.

**[00:24:44 – 00:24:47]** Okay, perfect.

**[00:24:47 – 00:24:50]** Don't be shy, don't bite.

**[00:24:54 – 00:24:56]** Hey, thanks, everyone. It was a great talk.

**[00:24:57 – 00:24:59]** So, for HSBC, what was the system you were using under

**[00:24:59 – 00:25:03]** the hood in terms of the actual GPU query acceleration?

**[00:25:03 – 00:25:06]** It's the CUDA essentially.

**[00:25:06 – 00:25:12]** So, it's all based on the NVIDIA stack for us.

**[00:25:12 – 00:25:15]** Can you specifically mention which engine that you guys were using?

**[00:25:15 – 00:25:19]** So we use specifically Starburst, which is one of our partners

**[00:25:20 – 00:25:23]** for the Trino and we basically partnered with them quite

**[00:25:23 – 00:25:26]** significantly on this.

**[00:25:29 – 00:25:32]** Jake, oh, it's over here.

**[00:25:37 – 00:25:39]** Thank you, really great insights.

**[00:25:39 – 00:25:41]** So this is actually a question for Timmy.

**[00:25:41 – 00:25:45]** There was a really interesting lab yesterday regarding foundational

**[00:25:45 – 00:25:47]** models in finance.

**[00:25:47 – 00:25:52]** And so it's interesting, it becomes an umbrella where you can split

**[00:25:52 – 00:25:55]** it off to different use cases.

**[00:25:55 – 00:25:58]** So it can be for credit risk and things like that.

**[00:25:58 – 00:26:02]** Have you considered Lloyds and cybersecurity applications?

**[00:26:02 – 00:26:06]** Because the banking industry is going a bit.

**[00:26:06 – 00:26:08]** So, no, no.

**[00:26:08 – 00:26:10]** Short answer, no.

**[00:26:10 – 00:26:14]** The side of the bank that we work in is in retail.

**[00:26:14 – 00:26:19]** So, most of the applications that we get to work with,

**[00:26:19 – 00:26:22]** I mean, the downstream use cases are really around

**[00:26:22 – 00:26:28]** things like churn, propensity, borrowing, and all of that.

**[00:26:28 – 00:26:31]** Are there applications of this for cybersecurity?

**[00:26:31 – 00:26:32]** I believe so.

**[00:26:32 – 00:26:35]** I mean, but not really.

**[00:26:35 – 00:26:38]** I mean, for instance, just think about like say fraud or

**[00:26:38 – 00:26:40]** sporting anomalies and all of that.

**[00:26:40 – 00:26:45]** Yes, but our scope is still not expanded towards that.

**[00:26:45 – 00:26:48]** So we've been working on use case really around like retail

**[00:26:48 – 00:26:54]** and a bit about around commercials, but mostly retail.

**[00:27:08 – 00:27:10]** Thank you all, really interesting talks.

**[00:27:10 – 00:27:15]** I have a question for Temi as well, about these downstream

**[00:27:15 – 00:27:17]** use cases for your embeddings.

**[00:27:17 – 00:27:21]** How are you actually leveraging the embeddings in the use cases?

**[00:27:21 – 00:27:23]** Are these more traditional ML models?

**[00:27:23 – 00:27:27]** You mentioned churn, or are there other types of use cases

**[00:27:27 – 00:27:28]** that you're looking at?

**[00:27:28 – 00:27:32]** TEMI SHOKUNBI- So this is trained on structured data.

**[00:27:32 – 00:27:37]** So essentially, we want to have a challenger against our classical

**[00:27:37 – 00:27:38]** machine learning use cases.

**[00:27:38 – 00:27:42]** So what we do is we've trained this whole big backbone.

**[00:27:42 – 00:27:46]** And we have simple classical machine learning

**[00:27:46 – 00:27:49]** eggs, like your logistic regression and all of that.

**[00:27:49 – 00:27:53]** Then we benchmark them against existing models and production.

**[00:27:53 – 00:27:56]** So the thing is we want to see, do the embeddings

**[00:27:56 – 00:27:58]** provide better signal?

**[00:27:58 – 00:28:00]** Or do they perform at best at par?

**[00:28:00 – 00:28:03]** And then also, the very good part of it is that we also

**[00:28:03 – 00:28:06]** want to encourage reproducibility and reusability amongst our

**[00:28:06 – 00:28:07]** data science teams.

**[00:28:07 – 00:28:11]** So we can have one backbone where people can actually

**[00:28:11 – 00:28:14]** build on, and that would actually improve or increase all the

**[00:28:15 – 00:28:16]** time to production.

**[00:28:16 – 00:28:19]** So you wouldn't have like weeks, you know, and engineering

**[00:28:19 – 00:28:20]** features and all of that.

**[00:28:20 – 00:28:23]** So we've been running a series of like tests and

**[00:28:24 – 00:28:27]** experiments and all of that.

**[00:28:27 – 00:28:29]** Yeah, we're getting really some good results, some we're

**[00:28:29 – 00:28:34]** still learning which use cases the TFM is good for, which

**[00:28:34 – 00:28:37]** we have to like maybe modify the architecture to be better.

**[00:28:37 – 00:28:38]** Thank you so much