Building AI Factories: How Red Hat and NVIDIA Turn Enterprise Data Into Intelligence - Ep. 293

0:10Justin Boitano:Welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. My guests today are Red Hat's Chris Wright and NVIDIA's Justin Boitano, and we're talking AI factories. Why should enterprises build AI factories, and how can they do so with confidence in building AI factories that they can trust? By way of introductions, and I'll keep it brief because both of these guys' work speaks for itself, really. Chris Wright is Chief Technology Officer and Senior Vice President of Global Engineering at Red Hat. And Justin Boitano is Vice President and General Manager of Enterprise Computing at NVIDIA. Gentlemen, welcome to the NVIDIA AI Podcast.

0:46Justin Boitano:Thank you so much for taking the time to join us. Thanks for having us.

0:50Chris Wright:Thanks for having me, Noah.

0:50Justin Boitano:Let's get right into it. And Justin, I'll start with you, but always both of you guys feel free to jump in. you know, as the spirit moves you, so to speak, as we go. But Justin, why don't we start with you? Can you talk a little bit about, well, maybe first give kind of a working definition of what we mean, what you mean when we talk about an AI factory, and then get into kind of at a high level, why would an enterprise be interested? Why are enterprises building AI factories? And what are some of the tangible benefits that an enterprise can expect to see from an AI factory?

1:22Chris Wright:Sure, Noah. Yeah, you know, and I think it's important to understand kind of the context of where we are as an industry. And, you know, building digital intelligence to power the productivity of organizations is going to be as critical in this decade as, you know, energy in running our companies. This is the next industrial revolution and companies are always asking us, you know, how do we build these factories that basically take data in and then produce the intelligence that, you know, helps them run their businesses more efficiently. And so as we talk about, like, what is an AI factory? We think of them as really kind of five layers of technology that need to come together.

2:02Chris Wright:At the base layer, you've got to make sure that you have got the data centers with power to bring into these factories. You've got to have chips is the easy way to talk about it. But we're at this point of building rack-scale infrastructure that's six chips with extreme co-design to build the best token efficiency from the power available to you. The next layer, You typically want to have the software infrastructure to orchestrate everything. And then you want to have models that run that intelligence. And then ultimately, the apps and the agents on top. And so what every business needs to do, though, is take this intelligence and build use case specific business outcomes that help them drive innovation, build products faster, and ultimately grow revenue top line through deploying this intelligence at scale.

2:54Justin Boitano:Right. Right. And so these five layers you're referring to, this is the cake, right? The five-layer cake? That's right. This is a five-layer cake. Excellent. Chris, the world is, I feel like we can say this so often, but things are changing so quickly. Right now, as we record this, there's a lot of talk about OpenClaw and Autonomous Agents and kind of long-running agents. Can you speak kind of a little bit sort of to that and how Nvidia and Red Hat are working together to help enterprise and enterprise IT departments kind of step into this new world? Yeah, actually, OpenClaw is a great example because there's so much enthusiasm about, I guess, what's possible, what you could do.

3:34Justin Boitano:It's captured the kind of the builder's imagination, but also built quite quickly, certainly leveraging AI to help produce code quickly, but not with the enterprise in mind. So when we think about what Justin was describing, that kind of data in to a factory context that produces business value as an output, we're talking about enterprises. That's their data. Those business outcomes are really either driving net new growth or focused on the productivity and efficiency. All of that needs to be done responsibly, safely, respecting access controls, delivering audit trails, things that are maybe not as fun in the builder world, but fundamental to the enterprise world.

4:22Justin Boitano:And so a lot of what we're doing is taking these building blocks, the layers of that five-layer cake, and making them accessible to the enterprise together. So obviously, NVIDIA's got world-class hardware. We're bringing a software layer that enables the higher levels of that cake. And then we're building the right guardrails and security considerations into this combined solution so that our customers can then feel confident about bringing this into their enterprise as they're all trying to figure out how to do AI transformation, go from a traditional company to really an AI native company. and in that context, not introduce undue risk or, you know, essentially undermine the core of their business.

5:11Justin Boitano:Right. So there's research that shows that only 1 % of organizations right now have reached the stage of an optimized, AI-fueled, AI-native, as you were talking about, Chris, enterprise, while over half of organizations still remain in the early stages of transformation. But at the same time, projections have global AI investment exceeding a trillion dollars total by 2029, just a few years out. And of that trillion dollars, these projections are saying agentic systems are going to account for roughly half of that spending. That's a big shift from, you know, a year ago, two years ago. You guys know the timeframe better than I would.

5:50Justin Boitano:But, you know, when agents were kind of this buzzword that, you know, nobody necessarily knew, there are all these different definitions, et cetera, et cetera. And now we're talking about all of this, you know, resource and spending going in specifically to agentic systems. Chris, what can we glean from this? And I know you spoke to it a little bit just now, but what are the kinds of things that the AI factory can do for an enterprise, infrastructure-wise, but confidence-wise, as you were talking about, when it comes specifically to figuring out how to deploy and integrate these agentic systems?

6:24Justin Boitano:Well, if you think about that notion of transforming the enterprise and leveraging internal data and focusing on your core business, how do you improve it or grow it? There's a whole set of things that are underneath that. Obviously, the data piece that we talked about, but also it is the existing tools that operate your business that are not going to just go away. They're fundamental. They're the baseline, the business as usual components, pretty critical and fundamental. So part of this is how do you carry that forward and really modernize your entire infrastructure to bring these two worlds together, this highly modern AI native world and the traditional set of applications that literally run the business?

7:08Justin Boitano:Because you need to bring AI capabilities, not just in the net new, but also in the existing content that runs all the enterprise. And to me, that's exactly what the AI factory does. it helps bridge these two worlds. I mean, in the end, we've got models, but we also have, as Justin described at the beginning, agentic content or AI-enabled applications, and then also the traditional applications. So bringing all of that together and then doing it in a context with consistency across the enterprise so that you're not asking every team to go figure out their own, choose your own adventure path forward.

7:44Justin Boitano:And that consistency, you build best practices across your organization. And then you're ultimately improving your chances for success and reducing the failure rates. There's so many studies that suggest a lot of AI projects can fail. There's a number of reasons for that. One of those is having the right tools and having the best practices, access to the data, and essentially combining forces as a company to produce an output rather than devolving into sort of the next generation of shadow IT and everybody building their own thing and creating this highly fragmented internal environment, which is then kind of difficult to get your arms around if not just produces very little success.

8:32Justin Boitano:Yeah. Justin, are you seeing similar things?

8:35Chris Wright:Well, I got to say, what's interesting is in the last three months, it feels like the market has really started moving even faster. I'll just say this. You look at coding companies.

8:46Justin Boitano:Every one of you guests who comes on this podcast says that same thing, moving faster, moving faster.

8:50Chris Wright:Well, but you can actually really feel it now. And I say that because the first area of agents, like product market fit, really was in software development. And we see it really as a software company ourselves. We can feel these agents doing so much more work for our developers and running longer, more complex software tasks. So you give them design goals and they can work towards those goals. And at the same time, like you said, this moment of Claws came out. And Claws basically take it to a new frontier of full autonomy. And so we're getting to this point where agents are going to have a lot more agency within our enterprise.

9:38Chris Wright:A lot of those studies that you mentioned where people were having a hard time getting AI to work, I think was at a previous era of the world where people were trying to do chatbots, just very basic chatbots. And that was before reasoning, and it was before this level of autonomy that I'm talking about. And so I feel like a lot of what enterprises might have been experimenting with might be a couple generations behind where state of the art is right now. And so as we deploy agents internally now that can use a very, I'll call it, deep agent-like reasoning framework. They can plan and reason and act across many different business systems to do deep research, as an example, to understand the intent of what a user might be asking and help them get to the information across the enterprise in a way that's faster and more efficient than ever previously thought imaginable.

10:31Chris Wright:And the nice thing about running this on a factory, an AI factory within the context of an enterprise, as Chris mentioned, delivers data privacy and security by running that all across open models in this on-prem world. And then you can do things where you still potentially use the frontier models, but you can use the frontier models in a way where you might only use it for the planning stage of the agent, and all the search and summarization is using open models. And so that drives a lot of cost efficiency. In some of our newer blueprints, we see a 30x cost reduction, by doing a hybrid model architecture across your private unstructured information.

11:09Chris Wright:And so that is a use case. The enterprise search, I think, is a broadly generalized use case that gets us from, say, these early adopters that were seeing the benefits of agents for coding into really how knowledge workers are going to start to use agents to help them do their jobs in a much more productive and efficient way.

11:30Justin Boitano:As somebody who sits more on the knowledge worker than a software developer side of the fence myself, getting me more towards that and away from Vibe coding is probably a good idea, but that's just my own sort of personal use case there. But that does make me want to double click a little bit on, you know, on security and governance and things like this, which, you know, I think, Chris, you mentioned at the top with the advent of, I mean, joking aside, with the advent of, you know, coding tools, Vibe coding tools and these more advanced Agenta coding tools in the hands of anybody, including folks like me, it's easy to spin something up.

12:08Justin Boitano:I don't know that it has, you know, a hole in it waiting for a prompt injection attack or whatever the case may be, right? And get into that shadow IT world, Chris, you were talking about. So I want to ask you both, and Justin, I'll start with you because you were talking about it a little bit just now. When you talk about planning and building an AI factory, what are the non-negotiable capabilities that have to be built in, that the enterprise must have to move from, you know, kind of first experiments and prototypes with AI to production, getting into industrial-scale production AI use cases with confidence.

12:42Justin Boitano:And, you know, Justin, you mentioned some of these, but there's security, there's governance, reliability, obviously moving to scale. Can you talk a little bit about some of these factors?

12:52Chris Wright:Yeah, and I think I'll say in the software development world, we're really good at separating the notion of development versus production. And I think that's obviously the best practice as enterprises get going is to separate the two. And on the one hand, you want to help your internal, I'll call it AI development teams, do discovery in a development environment, but separate that access control from production data until you've basically proven the verification or done the functional verification of the outcome that you're trying to get to. you've QA'd it, you've pen-tested it. It's got things like role-based access control so that if a user is using that agent, it inherits their permissions to access business systems.

13:34Chris Wright:And you're going to promote the agent from this development environment into a prod in that way. And so I think the worst thing that enterprises can do is overanalyze this, though, and try and get to, like, how do I prove the TCO up front before I start to make the investment? You've got to believe that AI is this new frontier and the companies that are able to harness it and put it to work for them are going to have a massive competitive advantage. And so the sooner you get going, the better. And you can start in this dev environment with, I'll call it, narrow use cases that are aligned to your core business goals and then scale as you start to see success.

14:16Chris Wright:But to your point, you want to make sure these agents have a clear set of governance. There's clear ability to trace the data systems that they access and that you can continuously evaluate them against known business outcomes that you're trying to achieve. And then that accuracy against certain use cases is what allows you to promote it then into production.

14:42Justin Boitano:Chris, can I ask you how things like, well, inference, obviously we did an episode recently about, it was energy focused, but talking about the coming wave of inference and the shift of the load moving to some extent from training to inference, maybe in this calendar year or whatever kind of the next wave is. But talking about things like high-performance inference and also hybrid cloud agility, how does the AI factory sort of figure in and support these two things in particular? Simply put, inference is your production environment. So training, whether it's pre-training or post-training, those are things that happen pre-production.

15:18Justin Boitano:and inference is where you're bringing this intelligence to life. So scale, efficiency, security, robustness, reliability, compliance with policy, compliance with SLAs or SLOs, these are like the table stakes. And an AI factory is a significant investment for an enterprise. The expectations are it produces significant business outcomes. And so we're focused on optimizing that production of outcomes, which you could back up and say those are business intelligence, or you could back up a little bit more and say it's simply tokens. Optimize that throughput of tokens in the context of cost, in the context of power consumption, because we're also power constrained.

16:06Justin Boitano:And so how do we do that? That's through this scaled out inferencing, which is part of the AI factory. It's really the core underlying platform that you're running all of your models and above that, the agents and AI applications on top of. So to me, it's the critical substrate and the agility that comes with flexibility and choice of where and how you deploy your models or your workloads, that notion of pre-production environments and production environments and where production data versus non-production data is used, you get some choice in where you deploy. And that, to me, is really the hybrid cloud.

16:49Justin Boitano:You've got optionality. There's cloud environments. There's enterprise environments. There's even edge environments where you may want to deploy your workloads. And taking advantage of all of that with a consistent footprint, like we're building with this AI factory, it gives you the best of all of your alternatives. And so, you know, I think we're bringing the efficiency, we're bringing the flexibility, we're ensuring that we have those confinements, whether it's confidential computing or guardrails or any kind of sandbox technology that I think becomes really critical as we're building and delivering these new capabilities.

17:33Justin Boitano:And if you go back in time before the focus on AI, we developed through decades of experience what Justin highlighted, that pre-production, dev, test, prod kind of best practices. There's a whole set of learnings and rigor and discipline that we've built in building and delivering applications into production that we're bringing as part of an AI factory for building and delivering AI applications into production. I'm speaking with Chris Wright of Red Hat and NVIDIA's Justin Boitano, and we're talking about the AI factory and how enterprises can build AI factories that they can go to production with with confidence and can scale up to the future and really help transform companies into AI natives, as we've been talking about.

18:23Justin Boitano:I want to get into a little bit about specifics, infrastructure and software and platform components. And Justin, I'll start with you. For customers who are thinking about an initial AI factory footprint and might want to start small but have that ability to scale as they scale, how should those customers think about sizing and selecting NVIDIA infrastructure and software?

18:46Chris Wright:Yeah, I think, you know, as the customer starts to try and build the AI factory, they got to think through the five-layer cake that I mentioned previously. So it's where do I have data center power? What is the power density of the data center? Do I want to run air cooling or liquid cooling? That seems to be a decision point right now. A lot of enterprises still run air-cooled data centers. And so, you know, platforms like our RTX 6000s give you, you know, very good price performance. that's kind of a general purpose GPU to do experimentation with. So if you don't know where to start, that kind of gives you a great platform for many different use cases.

19:21Chris Wright:And then from there, you start to ask yourself, well, what's the orchestration management platform that I want to run my business on? And that's why we worked very closely with the Red Hat team. Red Hat's AI factory takes care of really the next few layers of the technology stack from software orchestration and management, model delivery, all the, I'll say, commercial security patching, lifecycle management of all of that open source software so that you can run it with confidence and kind of get the factory up and running. And then you get up into the application layers. And the application layers, the way we try and make it easy for customers to start is we provide reference blueprints, which are examples of proven use cases that even we run on our AI factories at NVIDIA for things like enterprise search that make it easy to then connect into your enterprise documents and do document ingestion and then start to provide benefits to your users.

20:16Chris Wright:And then from there, you can start to expand into, you know, your own developed use cases and such. But that thinking through that full stack is really the easiest way to get going. And then I think taking some of these proven examples is like kind of the quick way to get an early win with kind of your executive leadership team. Yeah, yeah. With them the benefits. And then from there, usually you pivot into, you know, what's the most important business outcome for the company to be competitive? You got to ask yourself for NVIDIA, we're a chip company, we're a software company, and we're a supply chain company when you really boil it down.

20:49Chris Wright:And so we then go super deep into those use cases to make sure that we're enabling, you know, tens of thousands of chip designers, software engineers, or all the people dealing with all the components that allow us to have supply and availability of building this rack scale infrastructure. and if we're world-class at those, then we can be world-class in market. And I think that's generally how companies should think about it.

21:10Justin Boitano:Chris, on the Red Hat side, what are the key platform components that you see as foundational for this first AI factory deployment? Thinking about things like OpenShift, Red Hat AI Enterprise, AI factory with NVIDIA. You know, when thinking about this first enterprise AI deployment, what are the key platform elements to start with? And also, how should customers think about sequencing them? Yeah, I think for us, the stack starts with hardware, hardware enablement, and then the distributed nature of Rackscale architecture. How do you get access to that whole distributed system? And then going up from there, we start getting into specifics of models and agentic applications and AI-enabled applications.

21:52Justin Boitano:So the bottom of the stack, very clearly, that's the world of Linux, right? Hardware enablement, device drivers, low-level system software, and near and dear to our hearts, we spend a lot of time in that space and making sure that we work closely together with NVIDIA to do that first phase, right against the metal enablement. The next layer above that is the distributed layer. Bringing that rack-scale architecture to life includes a distributed system like Kubernetes. Kubernetes is tried and true in the application space, and it's supporting well delivery of agents or models or other content as containers on this distributed system with access to all the accelerators down at the bottom of the stack.

22:36Justin Boitano:And then that Red Hat AI Enterprise layers on top. And this is where we start to integrate directly with some of the key capabilities that NVIDIA brings, like optimized models like NemoTron or some of the NIMMs. And that's where we bring that distributed inferencing stack that is the foundation for intelligence for the business. So we sometimes call this the metal-to-agent stack and starting with that layer right above the hardware, building up through inferencing and then supporting the models is what we're building together to enable those key reference architectures that Justin highlighted or the validated blueprints or the reproducible plays that you want to bring into the enterprise.