# From Idle to Optimized: Managing GPUs, Kubernetes, and Automated Compute at Scale (Presented by HPE) 

## Abstract

As AI workloads grow in size and complexity, infrastructure teams are under pressure to deliver high GPU utilization, predictable performance, and strong operational controls across a mix of Kubernetes and bare-metal environments. Limited visibility into GPU behavior can quickly lead to inefficiency, cost overruns, and governance gaps. This session examines practical approaches to observing, provisioning, and optimizing GPU-accelerated infrastructure for AI workloads. We’ll discuss strategies for end-to-end GPU monitoring, automation patterns for Kubernetes and bare-metal environments, and techniques for aligning performance optimization with policy and governance requirements. The focus is on repeatable, scalable practices that infrastructure and platform teams can apply in real-world AI environments.

## AI Summary

- The presentation discussed how AI is transforming decision-making in enterprises, moving the conversation beyond model training to running AI systems at scale.
- It highlighted the challenges of integrating AI into existing enterprise workflows, such as data pipeline bottlenecks, infrastructure management, and security concerns.
- The speaker introduced HPE's Private Cloud AI solution, which aims to provide a turnkey, integrated platform to help enterprises operationalize AI more effectively.
- The solution includes a curated catalog of models, automated workflows, and a unified control plane to manage AI systems, ensuring they remain busy and productive.
- The presentation covered the importance of day two operations, including version compatibility, security updates, and lifecycle management, which HPE handles through the GreenLake platform.
- Different configurations of the Private Cloud AI solution were discussed, ranging from small developer systems to large-scale deployments, with options for on-prem and air-gapped environments.

## Transcript

**[00:00:10 – 00:00:14]** Every technology wave changes how businesses compete.

**[00:00:15 – 00:00:20]** Internet changed connectivity, electricity changed manufacturing,

**[00:00:20 – 00:00:25]** mobile changed how businesses interact with customer.

**[00:00:25 – 00:00:28]** AI, artificial intelligence, is changing something

**[00:00:28 – 00:00:31]** more fundamental.

**[00:00:31 – 00:00:37]** It's transforming how decisions are made inside enterprises.

**[00:00:37 – 00:00:41]** Today, AI is reading document.

**[00:00:41 – 00:00:43]** It's writing code.

**[00:00:43 – 00:00:46]** It's helping in fraud detection.

**[00:00:46 – 00:00:50]** It's helping doctors diagnose disease.

**[00:00:50 – 00:00:53]** It's optimizing supply chain.

**[00:00:53 – 00:00:56]** It's automating customer support.

**[00:00:56 – 00:01:00]** And something else has changed over the last two years.

**[00:01:00 – 00:01:04]** The conversation we used to have in this space has moved

**[00:01:04 – 00:01:07]** beyond model training.

**[00:01:07 – 00:01:14]** Today, the challenge is about how do we run AI systems at scale.

**[00:01:15 – 00:01:17]** How do we serve inference?

**[00:01:17 – 00:01:20]** How do we serve millions of requests?

**[00:01:20 – 00:01:23]** How do we connect enterprise knowledge, enterprise

**[00:01:23 – 00:01:25]** data with models?

**[00:01:25 – 00:01:31]** How do we manage cost, governance, security?

**[00:01:31 – 00:01:36]** And if you think about all this together, in other

**[00:01:36 – 00:01:42]** words, organization in the AI era that will win.

**[00:01:42 – 00:01:47]** Would not be simply training models or have access to the

**[00:01:47 – 00:01:49]** biggest trained model.

**[00:01:49 – 00:01:55]** It would be about building a platform that can help run

**[00:01:55 – 00:02:01]** AI systems at scale inside their enterprise workflow.

**[00:02:01 – 00:02:09]** Let me share something that I hear from customers all the time.

**[00:02:09 – 00:02:16]** A company invests in powerful GPU cluster, state of art

**[00:02:16 – 00:02:21]** technology, powerful hardware.

**[00:02:21 – 00:02:24]** Data science team is excited.

**[00:02:24 – 00:02:26]** Infrastructure team is excited.

**[00:02:26 – 00:02:32]** And for a few weeks, everything is running at peak performance.

**[00:02:32 – 00:02:38]** After some time, the GPU starts to idle.

**[00:02:38 – 00:02:44]** You know why, not because organization ran out of idea.

**[00:02:44 – 00:02:49]** It's because everything around that environment became bottleneck.

**[00:02:50 – 00:02:52]** Team is waiting for data pipeline.

**[00:02:52 – 00:02:55]** Someone is waiting for data access.

**[00:02:55 – 00:03:00]** Infrastructure team is busy doing drivers, firmware update.

**[00:03:00 – 00:03:04]** Security team is reviewing policies.

**[00:03:04 – 00:03:08]** And all of a sudden, you see this state of art technology,

**[00:03:08 – 00:03:14]** powerful GPU that we have ever built so far, is sitting idle.

**[00:03:14 – 00:03:16]** Let me ask you a question.

**[00:03:16 – 00:03:22]** How many of you have seen something like this inside your own setup?

**[00:03:24 – 00:03:29]** Exactly, and this is one of the paradoxes of AI

**[00:03:29 – 00:03:32]** system, enterprise AI.

**[00:03:32 – 00:03:37]** We have the most powerful compute power ever built with

**[00:03:37 – 00:03:43]** us sitting idle, and the challenge is how do we convert that compute

**[00:03:43 – 00:03:47]** power into enterprise intelligence?

**[00:03:47 – 00:03:50]** And that's what this session will cover today.

**[00:03:50 – 00:03:52]** By the way of introduction,

**[00:03:52 – 00:03:56]** I'm Abhishek, my focus is on helping customers

**[00:03:56 – 00:03:58]** operationalize AI.

**[00:03:59 – 00:04:02]** As we think, the innovation is moving rapidly, and at

**[00:04:02 – 00:04:07]** the same time, integrating that into enterprise AI is

**[00:04:07 – 00:04:11]** becoming incredibly challenging.

**[00:04:14 – 00:04:20]** Across the industry, we see a consistent pattern.

**[00:04:20 – 00:04:27]** Enterprises are able to experiment with AI successfully.

**[00:04:27 – 00:04:32]** But when it comes to moving those experiments into production,

**[00:04:32 – 00:04:35]** that's where the friction begins.

**[00:04:35 – 00:04:41]** Only a fraction of AI initiatives convert into

**[00:04:41 – 00:04:43]** the real production workflow.

**[00:04:44 – 00:04:46]** The reasons are familiar to all of us.

**[00:04:46 – 00:04:49]** Infrastructure challenges, security and data privacy

**[00:04:49 – 00:04:55]** concern, lack of skill, and all of this while innovation is

**[00:04:55 – 00:05:00]** moving faster than we can adopt it.

**[00:05:00 – 00:05:06]** The real challenge here is, how do we operationalize AI?

**[00:05:06 – 00:05:09]** How do we serve business with applications such as co-pilot,

**[00:05:09 – 00:05:11]** chatbot, and agents?

**[00:05:11 – 00:05:15]** And if you think about operationalizing AI,

**[00:05:16 – 00:05:22]** What needs to happen is we need to coordinate stack across

**[00:05:22 – 00:05:24]** different layers.

**[00:05:24 – 00:05:28]** The first is how do we think about the compute power?

**[00:05:28 – 00:05:30]** How do we think about GPUs?

**[00:05:30 – 00:05:32]** How do we think about networking?

**[00:05:32 – 00:05:34]** How do we think about storage?

**[00:05:34 – 00:05:38]** How do we think about having the data that is

**[00:05:38 – 00:05:40]** designed for all of this?

**[00:05:40 – 00:05:41]** And if you think

**[00:05:42 – 00:05:44]** We need to think about data pipeline.

**[00:05:44 – 00:05:46]** We need to think about safety and governance.

**[00:05:46 – 00:05:49]** We need to think about how different system would

**[00:05:49 – 00:05:50]** interact with each other.

**[00:05:50 – 00:05:57]** What would be the strategy behind model and data mesh?

**[00:05:57 – 00:06:01]** And there are different providers, vendors, frameworks

**[00:06:01 – 00:06:06]** that are available for you to choose, make this infrastructure

**[00:06:06 – 00:06:09]** stack all by yourself.

**[00:06:09 – 00:06:14]** One thing, if we take a step back and think about this

**[00:06:14 – 00:06:20]** AI infrastructure, one thing becomes more and more clear

**[00:06:21 – 00:06:26]** that this is not the same as application infrastructure.

**[00:06:26 – 00:06:27]** It is different.

**[00:06:28 – 00:06:34]** It continuously consumes data, it continuously produces prediction,

**[00:06:34 – 00:06:39]** and it continuously evolves over a period of time.

**[00:06:40 – 00:06:43]** If I ask you another question, and if we think from an

**[00:06:43 – 00:06:47]** outcome-based approach, what's the outcome if your

**[00:06:47 – 00:06:53]** organization or enterprise invest in application infrastructure?

**[00:06:53 – 00:06:56]** The outcome is software.

**[00:06:56 – 00:07:00]** What's the outcome if your enterprise, they invest in

**[00:07:00 – 00:07:02]** AI infrastructure?

**[00:07:02 – 00:07:07]** The outcome is intelligence, more decisions, more insights,

**[00:07:07 – 00:07:09]** more predictions.

**[00:07:09 – 00:07:14]** And that's what AI infrastructure is all about.

**[00:07:17 – 00:07:20]** Companies trying to build AI infrastructure, broadly

**[00:07:20 – 00:07:24]** they have two choices.

**[00:07:24 – 00:07:30]** The one is they heavily develop their infrastructure on cloud.

**[00:07:30 – 00:07:36]** Second is they build that infrastructure all by themselves.

**[00:07:36 – 00:07:39]** Each options have trade-off.

**[00:07:39 – 00:07:42]** If they go completely on cloud,

**[00:07:42 – 00:07:45]** They have the risk of cost overrun.

**[00:07:45 – 00:07:47]** They have data sovereignty challenge.

**[00:07:47 – 00:07:54]** If they build everything on-prem, they need to have deep expertise

**[00:07:54 – 00:07:59]** and continuous coordination for various integration.

**[00:07:59 – 00:08:06]** What they really need is, they need cloud-like experience at the same

**[00:08:06 – 00:08:11]** time running this infrastructure where their data lives.

**[00:08:11 – 00:08:17]** And that's what this strategy of

**[00:08:18 – 00:08:20]** Private cloud AI is.

**[00:08:20 – 00:08:26]** When we were designing private cloud AI, we thought about

**[00:08:26 – 00:08:29]** key barriers that enterprises face.

**[00:08:29 – 00:08:33]** The first one, build tax.

**[00:08:33 – 00:08:39]** Most enterprises, they spend months bringing together multiple

**[00:08:39 – 00:08:46]** systems before actually running their single AI workload.

**[00:08:46 – 00:08:52]** The second challenge we thought about was day two operation.

**[00:08:52 – 00:08:57]** Once the stack is built, it requires a complete or continuous

**[00:08:57 – 00:09:01]** investment to make sure there is lifecycle management around it.

**[00:09:02 – 00:09:05]** And because of so many moving parts, things become too

**[00:09:05 – 00:09:08]** complex too soon.

**[00:09:08 – 00:09:12]** And the third is, we were very much grounded in our thought that the

**[00:09:12 – 00:09:20]** infrastructure, AI infrastructure, needs to run where data lives.

**[00:09:20 – 00:09:24]** And that's why we co-engineered this product with NVIDIA and

**[00:09:24 – 00:09:31]** came up with Turnkey AI solution, Private Cloud AI.

**[00:09:32 – 00:09:37]** We understand that different organization enterprises, even

**[00:09:37 – 00:09:44]** teams inside different enterprises, they start at different places.

**[00:09:44 – 00:09:48]** Someone starts with sandbox environment, where they want

**[00:09:48 – 00:09:51]** to experiment and do pilot quickly.

**[00:09:51 – 00:09:55]** Someone is already at a place where they are deploying

**[00:09:55 – 00:09:57]** inference at scale.

**[00:09:58 – 00:10:01]** Some organizations want to connect that inference with

**[00:10:01 – 00:10:04]** their enterprise knowledge.

**[00:10:04 – 00:10:07]** Some organizations, they want to fine-tune the model or

**[00:10:07 – 00:10:11]** train the model for their domain-specific knowledge.

**[00:10:11 – 00:10:16]** Private Cloud AI comes in different flavors, different sizes,

**[00:10:16 – 00:10:22]** so that enterprises can start with the validated stack where they are.

**[00:10:22 – 00:10:28]** It also provides a timeless expansion option.

**[00:10:28 – 00:10:33]** Where enterprises can start where they are, but with the

**[00:10:33 – 00:10:38]** growth of their workload, they can scale GPUs.

**[00:10:38 – 00:10:43]** The architecture shown here allows enterprise to scale the compute

**[00:10:43 – 00:10:47]** power as their AI workload grows.

**[00:10:47 – 00:10:52]** The key part in this slide is operational simplicity.

**[00:10:52 – 00:10:55]** They don't need to worry about

**[00:10:55 – 00:10:59]** Thinking about another control plane when their workload grows.

**[00:10:59 – 00:11:01]** The data architecture remains the same.

**[00:11:01 – 00:11:05]** Everything get managed from the same singular control plane

**[00:11:05 – 00:11:09]** that they are already used to.

**[00:11:14 – 00:11:22]** Some customers in finance sector and in healthcare, government,

**[00:11:22 – 00:11:27]** in critical infrastructure, they want to run everything in

**[00:11:27 – 00:11:29]** a completely isolated environment.

**[00:11:29 – 00:11:35]** And with Private Cloud AI, we also provide a flavor of

**[00:11:35 – 00:11:37]** air gap deployment.

**[00:11:37 – 00:11:42]** Where the entire AI unit can be run in an isolated environment

**[00:11:42 – 00:11:44]** where enterprise data lives.

**[00:11:44 – 00:11:48]** Think about running one unit in submarine without having

**[00:11:48 – 00:11:50]** any internet connectivity.

**[00:11:50 – 00:11:55]** So the AI, it's a solution that brings AI infrastructure

**[00:11:55 – 00:11:59]** to the place where organization most need without worrying about

**[00:11:59 – 00:12:02]** security and compliance challenges.

**[00:12:03 – 00:12:05]** See you.

**[00:12:07 – 00:12:10]** This is the part about software stack.

**[00:12:10 – 00:12:14]** When I think about Tanki AI Solution or Tanki AI

**[00:12:14 – 00:12:19]** Factory, it's not just hardware and hardware stack.

**[00:12:19 – 00:12:24]** It's about what it means to deliver a solution that data

**[00:12:24 – 00:12:29]** scientists can start using it from day one without worrying

**[00:12:29 – 00:12:34]** about requiring various tools, doing various configuration.

**[00:12:34 – 00:12:38]** With Private Cloud AI, the software stack converts this

**[00:12:38 – 00:12:41]** AI infrastructure into AI platform.

**[00:12:41 – 00:12:43]** It has three main parts.

**[00:12:43 – 00:12:47]** The first part focuses on models and blueprints.

**[00:12:47 – 00:12:51]** Private Cloud AI comes with prepackaged, optimized NVIDIA

**[00:12:51 – 00:12:54]** model, and at the same time, customers have option to pull

**[00:12:54 – 00:12:57]** the models from NVIDIA's registry.

**[00:12:57 – 00:13:01]** They can also connect with open source model providers.

**[00:13:01 – 00:13:03]** They can also bring their own model.

**[00:13:03 – 00:13:07]** At the same time, there are set of validated blueprint that are

**[00:13:07 – 00:13:10]** easy to deploy on private cloud AI.

**[00:13:10 – 00:13:14]** The second layer focuses on workflow and automation.

**[00:13:14 – 00:13:18]** So when you think about inference, inference does not run in vacuum.

**[00:13:18 – 00:13:23]** It requires some pre-processing, post-processing data pipeline.

**[00:13:23 – 00:13:26]** Private Cloud AI provides the capability to create

**[00:13:26 – 00:13:31]** automated workflow for evaluation, deployment, monitoring,

**[00:13:31 – 00:13:34]** and the bottom layer provides the capability for having

**[00:13:34 – 00:13:39]** the observability for the entire stack from a single pan of glass.

**[00:13:40 – 00:13:43]** Getting into a layer deeper.

**[00:13:44 – 00:13:48]** If we get into what these three layers are, firstly, I

**[00:13:48 – 00:13:52]** already talked about access to the models, be it the large language

**[00:13:52 – 00:13:57]** model, embedding model, optimized model for a particular type of GPU.

**[00:13:57 – 00:14:00]** It also provides capability to connect with Hugging Face,

**[00:14:00 – 00:14:02]** bring your own model.

**[00:14:02 – 00:14:05]** There are some pre-validated blueprint.

**[00:14:05 – 00:14:08]** If you focus on the second layer, which is the software and the

**[00:14:08 – 00:14:13]** data foundation, and if I walk you through a journey of an enterprise,

**[00:14:13 – 00:14:18]** In enterprise, there are multiple teams working on different

**[00:14:18 – 00:14:21]** parts of the project.

**[00:14:21 – 00:14:24]** It all starts with connecting with data.

**[00:14:24 – 00:14:29]** And in enterprise, data is very distributed.

**[00:14:29 – 00:14:32]** You have some data sitting on cloud, some data sitting

**[00:14:32 – 00:14:36]** on-prem, some data in SQL, some in file system.

**[00:14:36 – 00:14:41]** So Private Cloud AI provides an interface to connect with

**[00:14:41 – 00:14:46]** structured data, connect with unstructured data, and also

**[00:14:46 – 00:14:48]** connect with file system.

**[00:14:48 – 00:14:52]** That's the first part about providing model connection.

**[00:14:52 – 00:14:56]** Once the enterprise or the team sets up connectivity to

**[00:14:56 – 00:15:01]** the data, the second part is about how do they process that data.

**[00:15:01 – 00:15:05]** Sometimes data processing may mean cleaning the data so

**[00:15:05 – 00:15:06]** that the data is of high quality.

**[00:15:06 – 00:15:11]** Sometimes it may mean running an automated pipeline so that

**[00:15:11 – 00:15:15]** once the data is transformed it can be readily used for inference.

**[00:15:15 – 00:15:18]** And sometimes it may mean creating a visualization of

**[00:15:18 – 00:15:21]** that data so that it can be continuously monitored.

**[00:15:21 – 00:15:26]** Private Cloud AI provides the capability to have the data

**[00:15:26 – 00:15:32]** pipeline, data processing, using frameworks such as Spark, Airflow,

**[00:15:32 – 00:15:36]** Superset and at the same time provides the capability to create

**[00:15:36 – 00:15:41]** model evaluation, model deployment pipeline through Kubeflow.

**[00:15:41 – 00:15:45]** And all this inference runs behind KSERV, which is running

**[00:15:45 – 00:15:47]** on the Kubernetes runtime layer.

**[00:15:47 – 00:15:50]** This all comes prepackaged, validated,

**[00:15:50 – 00:15:52]** lifecycle managed by HPE.

**[00:15:52 – 00:15:56]** At the same time, there are evolving capabilities in terms

**[00:15:56 – 00:16:00]** of enterprises can convert their connected data to a

**[00:16:00 – 00:16:04]** knowledge base and use that knowledge base to ground the

**[00:16:04 – 00:16:09]** inference from the model that they have deployed on the platform.

**[00:16:09 – 00:16:12]** It also comes with the playground so that the data scientists,

**[00:16:12 – 00:16:16]** machine learning engineers, they can see the response

**[00:16:16 – 00:16:20]** of their deployed system before actually sharing the endpoint

**[00:16:20 – 00:16:22]** with the application team.

**[00:16:23 – 00:16:27]** And if you think about all this layer, this all layer

**[00:16:27 – 00:16:33]** comes on the validated hardware stack with the observability layer

**[00:16:33 – 00:16:36]** that is connected with OpsRamp.

**[00:16:39 – 00:16:41]** The unified control plane, these are some of the screenshots

**[00:16:41 – 00:16:42]** from the product.

**[00:16:42 – 00:16:48]** The unified control plane eliminates the build or stitching

**[00:16:48 – 00:16:51]** tags that I shared before.

**[00:16:51 – 00:16:56]** The curated catalog helps you to deploy framework, deploy

**[00:16:56 – 00:17:01]** models without actually worrying about various workflows.

**[00:17:01 – 00:17:05]** And the security is built in security at various layers so

**[00:17:05 – 00:17:10]** that you don't need to worry about security and compliance challenges.

**[00:17:16 – 00:17:23]** When we think about AI or AI infrastructure in general,

**[00:17:23 – 00:17:26]** things can't run in vacuum.

**[00:17:26 – 00:17:30]** What AI infrastructure really requires is there are three

**[00:17:30 – 00:17:33]** parts that needs to come together.

**[00:17:33 – 00:17:35]** The one is infrastructure.

**[00:17:35 – 00:17:42]** Second is software that converts that infrastructure into platform.

**[00:17:42 – 00:17:45]** And third part is ecosystem.

**[00:17:45 – 00:17:49]** Different organizations, they have different use cases,

**[00:17:49 – 00:17:52]** and they have different needs.

**[00:17:52 – 00:17:56]** That requires vertically integrated solution, which

**[00:17:57 – 00:17:59]** are proven in some stack.

**[00:17:59 – 00:18:04]** With Private Cloud AI, we have over 40 partners across

**[00:18:05 – 00:18:12]** Different layers that helps customers in deploying a selected

**[00:18:12 – 00:18:17]** solution which are pre-validated on private cloud AI so that

**[00:18:17 – 00:18:23]** they don't need to worry about finding an expertise very specific

**[00:18:23 – 00:18:27]** for their particular use cases.

**[00:18:27 – 00:18:35]** Industry is also recognizing the value of integrated solution, so

**[00:18:35 – 00:18:43]** that enterprises don't worry about bill tax, don't worry about day two

**[00:18:43 – 00:18:48]** operation, don't worry about data sovereignty, security challenges.

**[00:18:48 – 00:18:53]** HPE was ranked number one in IDC market scape for

**[00:18:53 – 00:18:56]** private cloud AI system.

**[00:18:56 – 00:19:01]** Where we saw customers who wanted to have a tightly integrated

**[00:19:01 – 00:19:07]** solution that includes storage and all the management integration,

**[00:19:07 – 00:19:12]** so that the complexity that is related to ensuring data pipelines

**[00:19:12 – 00:19:17]** and AI workloads, it's eliminated.

**[00:19:24 – 00:19:33]** We believe that the future of enterprise AI is AI factory, where

**[00:19:33 – 00:19:39]** enterprises will have different need for different types of

**[00:19:39 – 00:19:43]** use cases and deployment scenario.

**[00:19:43 – 00:19:49]** To get started, enterprises would need an integrated solution, which

**[00:19:49 – 00:19:54]** we offer through Private Cloud AI, which It is a Tanki AI factory.

**[00:19:54 – 00:19:59]** Then they would want to scale to much larger scale that

**[00:19:59 – 00:20:04]** we support through AI Factory, and all this solution needs to

**[00:20:04 – 00:20:10]** be available in a sovereign setup where customers don't really worry

**[00:20:10 – 00:20:14]** about any security challenges.

**[00:20:14 – 00:20:20]** And that's where HP's AI Factory story fits in.

**[00:20:20 – 00:20:24]** If I really think about, right, And

**[00:20:24 – 00:20:30]** And think broadly, when we think about any change,

**[00:20:30 – 00:20:35]** every change brings a new type of infrastructure.

**[00:20:35 – 00:20:39]** Going back to our example, electricity grid, it powered

**[00:20:39 – 00:20:43]** manufacturing, internet connected the world.

**[00:20:43 – 00:20:48]** AI shift will be powered by AI factory.

**[00:20:48 – 00:20:53]** And it's not about organizations with having access to

**[00:20:53 – 00:20:59]** Most GPUs that will win in this environment.

**[00:20:59 – 00:21:06]** It will be companies who can keep those GPUs busy.

**[00:21:06 – 00:21:10]** They will most probably advance ahead.

**[00:21:10 – 00:21:15]** And that's what I would say this story comes together about

**[00:21:15 – 00:21:20]** how to convert idle to optimized.

**[00:21:20 – 00:21:24]** GPU, how to convert idle to optimized AI system.

**[00:21:24 – 00:21:27]** Thank you very much.

**[00:21:34 – 00:21:38]** Thank you for that great talk, Abhishek.

**[00:21:38 – 00:21:41]** Now we'll open up the stage for Q&A.

**[00:21:41 – 00:21:45]** Does anybody have any questions to begin with?

**[00:21:46 – 00:21:50]** We have one question. Perfect.

**[00:21:50 – 00:21:53]** Can we have the mic over there?

**[00:21:54 – 00:21:58]** No, I thought there were more mics.

**[00:22:01 – 00:22:05]** Hi, so the private cloud offering by HPE sounds really intriguing,

**[00:22:05 – 00:22:08]** but, you know, you're not always going to have just

**[00:22:08 – 00:22:09]** complete greenfield deployments.

**[00:22:09 – 00:22:12]** How does private cloud AI integrate with your existing

**[00:22:12 – 00:22:14]** infrastructure around GPUs?

**[00:22:14 – 00:22:16]** Does it integrate at all?

**[00:22:17 – 00:22:20]** Yeah, your question is about, sorry?

**[00:22:20 – 00:22:23]** If you've already got, let's say you've already got infrastructure

**[00:22:23 – 00:22:26]** deployed, GPU infrastructure deployed, and you wanted to

**[00:22:26 – 00:22:30]** integrate HPE private cloud, is that possible?

**[00:22:30 – 00:22:34]** The HPE Private Cloud AI today, it's a turnkey solution.

**[00:22:34 – 00:22:38]** So it's about how can you bring that turnkey solution

**[00:22:38 – 00:22:40]** inside enterprise.

**[00:22:40 – 00:22:43]** There can be various flavors when you think about integration.

**[00:22:43 – 00:22:46]** One integration I talked about was about data integration, in which

**[00:22:46 – 00:22:49]** you have data strategy already figured out inside enterprise.

**[00:22:49 – 00:22:52]** So you use Private Cloud AI to connect with data.

**[00:22:52 – 00:22:55]** If you think from the other spectrum, you have GPUs

**[00:22:55 – 00:22:58]** and GPUs cluster, which are already deployed by,

**[00:22:58 – 00:23:03]** In enterprise, the private cloud AI would help you orchestrate

**[00:23:03 – 00:23:06]** or deploy some of the models that would be required for

**[00:23:06 – 00:23:09]** inference, but your level of integration would be limited.

**[00:23:09 – 00:23:12]** Because private cloud AI, the way it exists today, it's more about

**[00:23:13 – 00:23:16]** a vertically integrated solution so that you have everything

**[00:23:16 – 00:23:21]** running on this platform and you can expand as your workload grows.

**[00:23:22 – 00:23:24]** Thanks for that question. Thank you.

**[00:23:24 – 00:23:26]** Any other questions?

**[00:23:27 – 00:23:33]** Does the AI solution have a Kubernetes layer, and which

**[00:23:33 – 00:23:35]** flavor does it have?

**[00:23:35 – 00:23:36]** Sorry, what is the question?

**[00:23:36 – 00:23:45]** Does the solution have a Kubernetes solution in built-in in the stack?

**[00:23:46 – 00:23:48]** I think he's asking, do you have a Kubernetes solution built

**[00:23:48 – 00:23:51]** into the stack, and if so, what flavor of Kubernetes are you using?

**[00:23:51 – 00:23:55]** So in this private cloud AI solution, it's a completely

**[00:23:55 – 00:23:56]** end-to-end built solution.

**[00:23:56 – 00:23:59]** So as part of the runtime, it's Kubernetes.

**[00:23:59 – 00:24:04]** HPE has a product, Morpheus Kubernetes solution.

**[00:24:04 – 00:24:06]** They're calling it HPE Kubernetes solution under

**[00:24:06 – 00:24:08]** the Morpheus umbrella.

**[00:24:08 – 00:24:10]** So in private cloud AI,

**[00:24:10 – 00:24:13]** The Kubernetes is vended by Morpheus.

**[00:24:13 – 00:24:18]** Then it integrates with other layer of the software and hardware stack.

**[00:24:18 – 00:24:22]** And that Morpheus Kubernetes is a flavor of what you get

**[00:24:22 – 00:24:29]** from open source Kubernetes, but HP flavor with more security past.

**[00:24:30 – 00:24:33]** Perfect.

**[00:24:33 – 00:24:36]** When you ask the question, can you also introduce yourself,

**[00:24:36 – 00:24:38]** announce your name, and then ask the question?

**[00:24:38 – 00:24:40]** Yeah, thank you.

**[00:24:40 – 00:24:42]** My name is Abhishek. I'm from Google.

**[00:24:42 – 00:24:43]** I'm a product manager.

**[00:24:44 – 00:24:46]** So GPU reliability has been very bad.

**[00:24:46 – 00:24:48]** GPUs break back and forth.

**[00:24:48 – 00:24:52]** So how do you prevent disruptions to training and other things?

**[00:24:52 – 00:24:55]** What kind of tools that you use to detect these failures?

**[00:24:55 – 00:24:57]** And how do you? I mean, you talked a little about that,

**[00:24:57 – 00:25:00]** but just want to know a little bit more details if you could share.

**[00:25:00 – 00:25:02]** Yeah, I think that's a good question.

**[00:25:02 – 00:25:05]** So there are a few layers. When we think about reliability and

**[00:25:05 – 00:25:07]** fill over, there are a few layers.

**[00:25:07 – 00:25:12]** So first is starting from infrastructure.

**[00:25:12 – 00:25:14]** Small, medium, large, it comes in a flavor where

**[00:25:14 – 00:25:17]** it's a multiple node cluster.

**[00:25:17 – 00:25:19]** So one is at the node layer.

**[00:25:19 – 00:25:23]** If one node fails, the second node can do the failover.

**[00:25:23 – 00:25:26]** The second layer is more on the scheduling layer.

**[00:25:26 – 00:25:30]** So we use NVIDIA Sky as the scheduling engine for

**[00:25:30 – 00:25:32]** running GPU workload.

**[00:25:32 – 00:25:37]** And what that means is the scheduler is topology aware,

**[00:25:37 – 00:25:42]** and it takes care of bin packing, or if one node fails, how

**[00:25:42 – 00:25:44]** to do the transition.

**[00:25:44 – 00:25:47]** And all this thing is running at a layer of Kubernetes.

**[00:25:47 – 00:25:51]** So Kubernetes provide that additional layer of pod restart

**[00:25:51 – 00:25:53]** and pod high availability.

**[00:25:53 – 00:25:58]** And on top of it, this is, we are mostly focusing on inference,

**[00:25:58 – 00:26:00]** fine-tuning workflow for now.

**[00:26:00 – 00:26:04]** There are some training operator that is still work in progress

**[00:26:04 – 00:26:06]** that we would introduce in Private Cloud AI that would

**[00:26:06 – 00:26:13]** help catering to the training workload more about deep learning.

**[00:26:16 – 00:26:22]** Two questions here, so the first one is why in the large

**[00:26:22 – 00:26:28]** expansion the solution will need to stay configured over the same rack.

**[00:26:28 – 00:26:34]** So one rack for 42 units of rack for two servers or two units

**[00:26:34 – 00:26:39]** of rack each one is too expensive, I don't understand why it happened

**[00:26:39 – 00:26:40]** this, if you want explain me.

**[00:26:41 – 00:26:45]** So another question is, is the largest expansion

**[00:26:45 – 00:26:50]** is available worldwide and especially in Latin America?

**[00:26:50 – 00:26:54]** Yeah, so all this configuration are available worldwide, mostly

**[00:26:54 – 00:26:57]** where the HP solution is available.

**[00:26:57 – 00:27:00]** So it would be available in Latin America, that's second.

**[00:27:00 – 00:27:03]** The first part is when we think about designing this

**[00:27:03 – 00:27:08]** stack, and if you think about some of the expense and option.

**[00:27:08 – 00:27:11]** You think about the number of GPUs, such as 16 GPUs,

**[00:27:11 – 00:27:16]** 32 GPUs, 64 GPUs, you'd want that to extend to 128 GPUs.

**[00:27:16 – 00:27:19]** There are needs for networking requirements, so that when

**[00:27:19 – 00:27:25]** you really run the GPU workload at scale, not just the north-south

**[00:27:25 – 00:27:28]** traffic, but the east-west traffic, and when you think

**[00:27:28 – 00:27:33]** about distribution of any model across multiple GPUs, network

**[00:27:33 – 00:27:35]** does not become the bottleneck.

**[00:27:35 – 00:27:38]** And that's where you'd see rack with higher configuration.

**[00:27:39 – 00:27:42]** And there's also some amount of duplication, so that in

**[00:27:42 – 00:27:46]** scenarios when there is a node failover, the other node

**[00:27:46 – 00:27:52]** can take from the failed node, and there is a level of reliability.

**[00:27:52 – 00:27:56]** All those goes into design, and that's how you'd see this rack.

**[00:27:56 – 00:27:59]** If you really think about different price point, if you're

**[00:27:59 – 00:28:03]** interested, I can connect offline, and we can follow up with you.

**[00:28:03 – 00:28:05]** We have done some internal cost comparison, there's also

**[00:28:05 – 00:28:08]** some external validation on the cost comparison.

**[00:28:08 – 00:28:12]** We strongly believe the integrated solution in stack provides

**[00:28:12 – 00:28:18]** you much better TCO compared with other solution and even some

**[00:28:18 – 00:28:20]** of the hyperscaleless public cloud.

**[00:28:20 – 00:28:24]** So, like I, myself, talking to customers, I haven't come

**[00:28:24 – 00:28:27]** across customers thinking about investing in large, extra

**[00:28:28 – 00:28:32]** large, and if they make a price or cost comparison, cost becomes

**[00:28:32 – 00:28:33]** one of their top three challenges.

**[00:28:33 – 00:28:37]** Thank you.

**[00:28:38 – 00:28:41]** We'll take the next question there, here in Canada.

**[00:28:41 – 00:28:45]** Can you share a little more details about day two operations?

**[00:28:45 – 00:28:49]** So when you deliver the turnkey solution to the customer,

**[00:28:49 – 00:28:55]** how do you handle the reliability operational issues?

**[00:28:55 – 00:28:58]** Few things, when you think about day two operation in

**[00:28:59 – 00:29:04]** any AI infrastructure, right, and if you go from top to down, in our

**[00:29:04 – 00:29:09]** software layer, we start with the models, which are optimized models.

**[00:29:09 – 00:29:15]** Then it comes down to a layer that helps you deploy those models.

**[00:29:15 – 00:29:18]** Then it comes down to a layer where we provide access control

**[00:29:18 – 00:29:21]** around those deployed model.

**[00:29:21 – 00:29:24]** Then it comes down to a layer where there's a runtime that

**[00:29:25 – 00:29:27]** orchestrates this entire system.

**[00:29:27 – 00:29:30]** Then it comes down to a layer about control plane and worker

**[00:29:30 – 00:29:34]** plane where the GPU workload is running on bare metal but some

**[00:29:34 – 00:29:38]** of the things in the control plane are virtualized running on VM.

**[00:29:38 – 00:29:42]** And then it comes down to different operators.

**[00:29:42 – 00:29:45]** Think about GPU operator, network operator, that makes this entire

**[00:29:45 – 00:29:50]** workload topology aware and makes your things are rightly distributed

**[00:29:50 – 00:29:52]** for optimum performance.

**[00:29:52 – 00:29:56]** And then it comes down to firmware, which are required

**[00:29:57 – 00:29:59]** to support this entire stack.

**[00:29:59 – 00:30:03]** And then layered down, it comes down to ILO of server rack.

**[00:30:03 – 00:30:07]** And all this layer that I mentioned, right, the software

**[00:30:07 – 00:30:11]** for all these layers comes with particular version.

**[00:30:11 – 00:30:16]** And with passage of time, there's a need to fix security

**[00:30:16 – 00:30:17]** vulnerabilities, CVs.

**[00:30:17 – 00:30:21]** There's a need to provide additional features and capability.

**[00:30:21 – 00:30:26]** And all that means there's a high chance that all these versions

**[00:30:26 – 00:30:28]** will not work with each other.

**[00:30:28 – 00:30:33]** What HP does is HP takes care of the version compatibility.

**[00:30:33 – 00:30:38]** Make sure does the validation of all different layers and

**[00:30:38 – 00:30:43]** pushes the software update as one big bundle software update.

**[00:30:43 – 00:30:48]** Customers can manage this lifecycle through GreenLake platform.

**[00:30:48 – 00:30:51]** For the connected, so they see that a new version of

**[00:30:51 – 00:30:55]** the software is available, and then they can apply those.

**[00:30:55 – 00:30:58]** When they apply those, depending on the portion of the layers,

**[00:30:58 – 00:31:03]** those new softwares are downloaded and applied in a rollover basis.

**[00:31:03 – 00:31:06]** For the air gap disconnected scenario, the software updates

**[00:31:06 – 00:31:10]** are available offline for customers to get sanitized

**[00:31:10 – 00:31:12]** and then apply locally.

**[00:31:12 – 00:31:18]** HP takes care of all this lifecycle management and provides updates to

**[00:31:18 – 00:31:21]** customers for the deployed cluster.

**[00:31:21 – 00:31:23]** Hope that answers your question.

**[00:31:23 – 00:31:24]** Thank you so much, Abhishek.

**[00:31:24 – 00:31:26]** We have just a few minutes left, so we'll just have the

**[00:31:26 – 00:31:28]** last two questions over here.

**[00:31:28 – 00:31:29]** Yeah, just a quick question.

**[00:31:29 – 00:31:31]** I'm David from Leighton.

**[00:31:32 – 00:31:36]** I would like to know, can you give us some insight about

**[00:31:36 – 00:31:40]** the different configuration, like small, large, and extra

**[00:31:40 – 00:31:47]** large pricing configurations and also maybe hardware configuration?

**[00:31:47 – 00:31:51]** Yeah, I will not talk about pricing, we can connect offline

**[00:31:51 – 00:31:53]** to talk about pricing.

**[00:31:53 – 00:31:57]** But when you think about different configuration, it starts with

**[00:31:57 – 00:32:03]** a small developer system, which is two GPU single node,

**[00:32:03 – 00:32:10]** comes with RTX 6K, and some around 30 terabytes of storage data.

**[00:32:10 – 00:32:15]** When it comes to small, that's where the full stack builds in.

**[00:32:15 – 00:32:19]** We have the storage, which

**[00:32:19 – 00:32:23]** is currently powered by GreenLake for file system.

**[00:32:23 – 00:32:26]** And there are different configuration options.

**[00:32:26 – 00:32:29]** For example, it starts with four GPUs.

**[00:32:29 – 00:32:30]** We have H200.

**[00:32:31 – 00:32:34]** We also have RTX 6K for you to choose from.

**[00:32:34 – 00:32:40]** You can expand small by adding four more GPUs as one additional

**[00:32:40 – 00:32:41]** node to make it eight node.

**[00:32:41 – 00:32:45]** You can also start with medium, where it starts from eight node.

**[00:32:45 – 00:32:49]** Again, you have the option of using H200 or RTX 6K.

**[00:32:49 – 00:32:52]** You can expand the medium to add eight additional node,

**[00:32:52 – 00:32:55]** so to make it total 16.

**[00:32:55 – 00:33:00]** GPUs and for large you can start with 16 GPU and then you can

**[00:33:00 – 00:33:06]** expand to 32 and you we also have enabled this timeless expansion

**[00:33:06 – 00:33:11]** unit where you can start with one large rack and add additional

**[00:33:11 – 00:33:15]** three racks to go up the GPU count.

**[00:33:16 – 00:33:19]** The last question is over there.

**[00:33:23 – 00:33:26]** Hi, my name is Avinash. I'm going to be interning at HPE

**[00:33:26 – 00:33:28]** and this is a bit of a basic question but I wanted to know more

**[00:33:28 – 00:33:33]** about like I'm gonna be working on the GreenLake over the summer

**[00:33:33 – 00:33:35]** and I wanted to know most basically like where I can learn more

**[00:33:35 – 00:33:38]** about like how to like optimize like these AI agents because I

**[00:33:38 – 00:33:41]** have access to like all these tools as a student but I don't really

**[00:33:41 – 00:33:45]** know like how to optimize it so I want to know like how where I can

**[00:33:45 – 00:33:48]** learn and like how I can learn more about like this GreenLake solution.

**[00:33:48 – 00:33:51]** Yeah, that's a good question. So, the first thing is, when we think

**[00:33:51 – 00:33:53]** about optimizing anything, right, we need to be very clear about

**[00:33:54 – 00:33:57]** what are we optimizing and why we are optimizing, what's the outcome.

**[00:33:57 – 00:33:59]** So, when you think about optimizing AI agents, are you optimizing

**[00:33:59 – 00:34:02]** for performance, are you optimizing for speed, are you optimizing

**[00:34:02 – 00:34:07]** for cost, or are you optimizing for different metrics altogether?

**[00:34:07 – 00:34:08]** That's one part.

**[00:34:08 – 00:34:11]** Second part is the way it happens is the private cloud

**[00:34:11 – 00:34:14]** AI through AI essential provides you a platform or an integrated

**[00:34:14 – 00:34:18]** development environment to deploy the agents and once you deploy the

**[00:34:18 – 00:34:23]** agents you can evaluate the outcome of those agents and you can observe

**[00:34:23 – 00:34:27]** What are the different metrics for those agents?

**[00:34:27 – 00:34:31]** And then think about changing the parameter by tracking it.

**[00:34:31 – 00:34:34]** There are tools such as MLflow, which comes as part of Private

**[00:34:34 – 00:34:37]** Cloud AI that enables you to do that experimentation,

**[00:34:37 – 00:34:41]** track all those experimentation, and then decide what part

**[00:34:41 – 00:34:44]** of the agent or what agent is working well, while at the same

**[00:34:44 – 00:34:48]** time you can monitor and observe everything together on OpsRamp.

**[00:34:48 – 00:34:50]** This all platform

**[00:34:50 – 00:34:54]** Enables the configuration that is required for you to deploy

**[00:34:54 – 00:34:56]** the workload by first building it.