953: Beyond “Agent Washing”: AI Systems That Actually Deliver ROI, with Dell’s Global CTO John Roese

22:12Jon Krohn:It sounds like that internal governance aligns exactly with the AI slop conversation we were just having. A thousand percent. Yep. You want to avoid AI slop, you better have a governance structure to pick the things that are not sloppy. And by the way, even then, some of the slop, the implication of slop isn't just that it's potentially messy, it's potentially dangerous. If the slop touches something that's kind of regulatory third rail, and you don't realize that, that is an enormous problem. There were a few articles about some consulting companies working with governments that inadvertently, some of their team members use AI tools to generate content.

22:48They got sued. That's slop. And it's not slop because it was in a vacuum. There was a governance breakdown there. That should never have happened. And so governance is a counteraction to slop, and it definitely keeps you at least more deliberate about what you're doing. No one will choose to be sloppy. They will choose to do the right thing. But if they don't even make a choice because there's no governance, then you get where you're

23:13Jon Krohn:Yeah, yeah, yeah. Very well said again there. So I should have mentioned probably before I talked about your first prediction for 2026 is that you have five for 2026 plus a bonus. And I think that the bonus is something we talked about your preceding episode in your preceding episode of Fairbit as well. But just so that people have some sense of kind of where we are. So first one was governance, your governance, governance soapbox point. Number two is data management, which you call the true backbone of AI innovation. Fill us in on that, John. Yeah, yeah. So we've been obsessed about large language models and AI compute.

23:51Those are great, but we're doing pretty well in those spaces. And we have a lot of business around building out the compute infrastructure of the world. But as you put these into production, what you realize is that the compute can move around. You can do compute on a device. You can do it on edge. You can do it in a cloud. You can do it in a public service. You can do it in a data center. You have lots of choices to compute. That's actually in pretty good shape. But what's hard, especially as you move to things like agentic, is you have to create entirely new data structures to make these things work.

24:17Like an agent by itself does not actually work without the addition of things like knowledge graphs and additional information. And those are not created randomly. Those are created with a new data layer. Sometimes we call that layer the knowledge layer. You have your primary data, your systems of record, and you have your compute layer, which is the other end of the system. But right in the middle of it, there's a new knowledge layer forming. And that's where things like graph databases, vector databases, knowledge graphs, all of the necessary data, sometimes mathematical representation of data, sometimes graphical representation.

24:49But the bottom line is that new data layer is really the thing that allows systems to have context. You know, if you just take an LLM and try to run your business with it, it's not going to know anything about your business. You have to attach a vector database via RAG to make it intelligent. If you're an agent, it's different. You might use a RAG, but you might create knowledge graphs that are very specific to particular roles and also instantiate things like memory, a long and short-term memory, the ability to accumulate information and skills. All of that layer is, you know, in 2025 wasn't really talked about.

25:24But as you go into 2026, it's insufficient to just have your primary data and a bunch of AI compute. You've got to build out an effective data management layer in the middle called a knowledge layer. One of the interesting subtleties about that knowledge layer is usually in the past when we reflect or represent primary data to other systems, we do it close to the primary data, meaning you got your data in a database, then you put a front end on it to express it a different way. The knowledge layer is different. It's not an API. It's actually persisting data. It's a layer that lives. It's always warm or hot.

25:57And what we're seeing is you almost have to make a choice. Do you want the knowledge layer close to the static primary data, or do you want it close to the dynamic compute environment, the edge, the device? And it turns out it makes more sense to put the knowledge layer close to where the action is because that's where all the transactions are happening. The feeding of the knowledge layer from a database is like a one-time event. It's non-real-time. The consuming of a knowledge layer by an AI system is hot and real-time and perpetual. And so all of those dimensions of what does the data architecture look like for a robust AI system, pre-agentic or agentic, and it's different than anything we've ever built.

26:39It's an entirely new layer, and it has different characteristics. And we shouldn't walk in there with an assumption that just because it's called a vector database, it belongs where all the other databases are. It might actually belong close to where the compute is happening, and that's definitely going to be true for agents. So, you know, if you thought data was interesting before, now we have a whole bunch of new data to deploy, but we have to do it in an intelligent, architecturally consistent way. And we have to think about it from an AI first perspective, not a legacy data perspective. And that's going to be interesting this year.

27:10But I think it's going to occupy a lot of oxygen in the dialogue about how to do AI properly. Not just compute, not just data, but what is that knowledge layer in the middle? How do you build it? How do you make it optimized? How do you make it work best? Where should it be? All of those questions have to be answered for you to be successful, especially with Logitech.

27:26Jon Krohn:I love this idea of terming this the knowledge layer. I've never come across it terms so nicely, because I think about vector databases, reg databases, a knowledge graph. I've always had those as kind of different types of knowledge that are being derived from raw data and that are going to be needed downstream by my generative or my agentic application. but by terming those all as the same kind of thing, as a knowledge layer that sits between your raw data, your system of record, and your compute, that's a really nice mental framework for thinking about these kinds of things. And in particular, the point that you made there around having that knowledge layer be close, be on the edge, be on, you know, Dell makes a lot of AI PCs.

28:12Jon Krohn:That's been, it's been a big thing that you guys have launched in 2025 is new AI PC devices that allow you to have quite a bit of compute on the edge and quite a bit of data storage as well. So more plenty of data storage to be having your knowledge layer maybe updated on, I'm guessing here, you might even have some real world use cases, John, but in my mind, it would kind of make sense to me to have a 24 hour process. Like every 24 hours, a cron job goes off that goes over the systems of records and updates the knowledge layer on some edge device at midnight or something. And then that updated knowledge layer information is available for the generative system or the agentic system to be able to use it regularly with low latency.

28:59It gets even more dynamic when we start talking about fully autonomous agents because the agent uses that knowledge layer, specifically the things like knowledge graphs, as a place to keep track of what it's learned. Like we built autonomous agents where we've given an ensemble a task to develop a skill, to run a CNC machine, to do a research report, to whatever it is. And it turns out that it starts with that batch representation of knowledge. And then as it starts to work through the problem, it actually creates pathways through the knowledge graph that look like vectors that essentially allow it to understand how it got from point A to point B, how it turned on the CNC machine, how it found an answer.

29:38And all of that is dynamically being added to the knowledge graph as the agent is actually working. And so that's why I keep saying it's not a system that ever goes cold. It's hot or warm at the worst. And that's just a different way to think about it. I think you talk a lot about data in this environment, this dialogue. And the takeaway is there's a new layer forming. And you shouldn't treat these as just another database and use the same approach you do for any type of database on this AI stuff because it's not doing the same thing. It doesn't exist in the same place. It doesn't have the same behaviors.

30:11And so that's going to be fun. There hasn't been a lot of new data stuff for a while. This is all new data stuff, and it's actually pretty exciting to figure out, okay, what does it do? Where do I apply it? Then how do I optimize it? How do I make it work well? And that's where things like deploying knowledge graphs close to the point of presence and making them incredibly dynamic becomes incredibly powerful.

30:30Jon Krohn:Yeah, and I can't believe I was thinking of it so simple-mindedly to think of data only flowing from system of record, data layer, to knowledge layer. But, of course, in this agentic AI world that we're in now, the agent's going to be updating that knowledge layer as well the other way. You bet. Bidirectional. Yeah. And then I guess then you might have a cron job taking things out of the knowledge layer and putting that into your long-term system of record. Exactly. Remotely. Cool. All right. On to number three. This one is about agentic AI. And it looks like there's some topics here that we've already kind of brought up and maybe you want to dig into a bit more.

31:04Jon Krohn:You mentioned knowledge graphs, LLMs, MCP, and A2A, which you touched on briefly earlier in this episode, but we didn't go into too much detail. Yeah, you know, this is one that, you know, I've had a lot of agentic discussions over the last year and a half. And, you know, you can define it at a high level, but we thought we'd go a little deeper. And it's important to recognize that when you start thinking about an agent, and here's why it's in the predictions. There are two different philosophical approaches to what an agent is. there are a cast of characters who think that an agent is a feature of an LLL.

31:41The central piece is the large language model, and the large language model has agentic behavior. That might be true. I don't think that's very useful, to be perfectly honest. I think that isn't really fully exploiting the capability. We, on the other hand, and we are not alone, there are a lot of people who absolutely believe exactly what I'm about to say, but not everybody, that an agent is actually a software system that has multiple dimensions to it, of which an LLM is one of them. And in both cases, both groups are saying this should be an autonomous system, it should be able to do work independently.

32:15But the difference is, is it just a feature of an LLM or is it a software system that uses LLMs and other stuff to accomplish that task? We're in the second camp. And so we drew a little picture that I think has worked really well for people to say, okay, at a technical level, an agent that is that software system actually has four components. One of the components is in fact a large language model, and it needs that because that's where it gets basic communication skills, world knowledge, and in many cases, its reasoning capability is a function of the LLM. There are reasoning models that work really well.

32:48So you can't really build agents without LLMs. They might be large language or small language models, but you do need them, and they are one of the four fundamental components of an agent. The second part of an agent is what we just talked about in the previous prediction, which is that in addition to world knowledge, agents benefit by having specialized knowledge. And that specialized knowledge is expressed in some cases as a gentic rag, but in many cases as things like knowledge graphs, where you populate it with very specific information, but you don't just give it an information set. You give it a graph that it can actually manipulate, and it can use not just to see specialized information, but to actually develop skills around that specialized information, pathways through the knowledge graph that help it repeat tasks.

33:30And sometimes we call that short and long-term memory, that it can actually keep state over a very long period of time by instantiating what it's learned in the knowledge graph, not in the LLM. The third component, though, is that, and we talked about this earlier also, is that agents, this kind of system, need to perceive the world around them, meaning they have to access external data, and they need to be able to do things, which means they have to be able to use tools. MCP becomes the protocol of choice to do that. So in addition to having large language models and knowledge graphs, MCP, or a like interface that allows them to access external data sources and interact with tools is a critical component of this system.

34:09And then the fourth component, which is probably the most important in my view, is agents can talk to each other. Unlike traditional AI tools that operate in isolation, agents actually do better when you put them into an ensemble and you actually allow them to work together. They bring different skills, they collaborate. And the reason you do that isn't just because you want to distribute the work. It's because it allows you to start to introduce behaviors like consensus into their decision making. And we actually did some studies where we've seen that when you take a collection of agents, an ensemble of agents, and they share a base truth with a common knowledge graph, the knowledge graph is truth.

34:45The LLM is opinion. And the agents all share that base truth. If you give them a governmental system, because ensembles, you tell them, is this a democracy, an autocracy? How do you want these systems to behave? If you tell them that consensus matters, if one of the agents and their LLM start to hallucinate and do crazy things that are inconsistent with ground truth in the knowledge graph, the other agents ignore it. They don't let it impact the outcome. It's like, imagine you take five people in a team and one of them is absolutely nuts and starts saying crazy things. You have countermeasures.

35:19You agree collectively, that's a dumb idea. We're going to do the right thing. And so we've seen that behavior in agent ensembles. Well, that brings us to that fourth component, which is they have interworking protocols. And that's where agent to agent or A to A primarily lives. And it's an ability for you to actually express communication between agents so that they can work as teams, they can delegate tasks. And that's incredibly powerful to go after these bigger sets of activities that we want to pursue beyond just single process, single team. So that picture, and I'm sure we can throw it up on the screen or something, and you can see it, has become incredibly helpful to help people understand these are not just an LLF.

35:56These are these four, when these four capabilities, general world knowledge and communication skills, specialized knowledge and state memory, the ability to perceive and act in the real world, and the ability to communicate together in ensembles. When you take those four capabilities, put them together, you have an autonomous agent. And I think that's going to be kind of a fairly enduring definition that most of the world agrees with. And just caution, there's a few people who think all that stuff can just be done as magic inside of the black box of magic called an LL. We disagree.

36:25Jon Krohn:Yeah, we do. We do have a sizable chunk of listeners who are listening to the show in an audio only format. So they won't be able to benefit from this diagram. But we will get the diagram up on YouTube for those of you watching it on YouTube. and it just to kind of summarize it basically it has there's four components to this agentic software system that john went through so the llm is a component of the software system as opposed to the center of the system it has a knowledge graph or some other kind of graph for being able to store information and refer back to that mcp for accessing databases and tools and then a2a for allowing agents to talk to each other and a really cool thing that i like about the collaboration point that you made there where, you know, if one of the agents takes magic mushrooms and is completely, it just starts hallucinating all the time.

37:12Jon Krohn:The other ones start to ignore it. And that reminds me how, when I was talking about that MTur graph of how you have tasks now, you know, kind of on the two to four hour range, human tasks being handled at a 50 % accuracy rate. If you're willing to spend a bit more on inference costs and have multiple agents working together on the problem, you can very quickly get much, much, much higher than 50 % accuracy rate. You can probably get 99 % or more if you're willing to spend on the inference and have agents working together on the problem. Yeah. We don't know if this solves hallucination, but it sure mitigates a lot of it.

37:49And we're not inventing a different way to do it. In humanity, how do you mitigate insane behavior and ridiculous statements? You collectively look at it and you you dismiss it because it's obviously not consistent with ground truth. So the same principles apply to agents if you build them the way we just described it. So we think it's promising. Nothing is going to be 100%. Nothing is going to be absolutely accurate. That's just not possible in the world in a probabilistic system. But boy, having a few agents work together gives you a completely different outcome than hoping a single LLM in a vacuum is perfect.

38:27That's not going to happen.

38:29Jon Krohn:Nice. All right. And so I'm conscious that we might only have a few minutes left of your time for this recording. And it seems to me like number four is a critical one to be talking about, because this is one that you and I have been talking about all year round. You and I recorded an ad for television about the Dell AI factory with NVIDIA. And your point number four is that AI factories will redefine resilience and disaster recovery. Tell us about AI factories. Yeah, this one's simple. We have about 3 ,000 customers building out AI factories today with Dell in different states, but we haven't figured out how to make them resilient yet.

39:03And the reality is we could just back the whole thing up, but it turns out that AI factories are different than traditional environment, kind of going back to that data discussion. If the primary data sources are not actually used in most of the AI interaction, let's take an example. if I take all my service information, I vectorize it, and I create an agent that can use all that information to solve service problems. At no point does that agent go back to the primary data while it's doing its job. And so it opens up some really interesting thoughts about where do we provide resiliency? Which things need to be resilient?

39:36If I lose access to my primary data, but I'm not even using it to do the AI stuff, then maybe I don't need seven nines reliability for the primary data anymore. I need it for the AI side. In addition, where that backup occurs, what does a backup to an AI factory look like? You know, it needs to have access to a lot of GPUs, but you don't want to have them sit in their idle. So there's interesting opportunities about using kind of elastic infrastructure in some of the CSPs and other environments. But this discussion of now that we built our AI factories, we need to make it resilient and we shouldn't go into that discussion thinking an AI factory is a traditional IT infrastructure.

40:13So we have to kind of look at different models, find different efficiencies, use some of the new infrastructure that's being built where GPUs exist at scale to maybe create different kinds of resiliency models. In addition to that, the whole cyber resiliency world of ransomware protection and data vaults and cyber vaults all has to be applied to protect knowledge graphs, vector databases, agents. Agents are a software system. You need to store them somewhere. And so we think this is going to be a big topic. We think that the guidance is not to go into it blindly thinking we will bring cyber resiliency to our AI factories exactly how we did it 20 years ago.

40:48That is not going to be the case. And there's a huge opportunity to be highly disruptive and innovative and to really build resiliency for AI factories in a modern way, which we're pretty excited about.

40:58Jon Krohn:So you mentioned that 3 ,000 different Dell clients are using AI factories. It sounds to me like an AI factory is kind of, it's bringing together a lot of the services that Dell and then NVIDIA GPUs as well can offer. So it's data, it's services, it's open ecosystem infrastructure, all combined together to allow whoever's taking advantage of an AI factory to get an AI ROI flywheel going on their own. Yeah, and that's where NVIDIA and Dell got together, well, no, going almost two years ago with this AI factory concept to say, hey, we had to absorb the complexity to us. Customers shouldn't have to figure this stuff out.

41:36They should be able to take reference architectures and templates and make these things fast moving. No one should have to design their own server and figure out their own software stack. These things should be as consumable as possible so you can spend your time on the use cases and the data and the application. So NVIDIA and Dell have a long history here. And now our next challenge is, okay, now they exist. Let's make them super resilient and survivable in a modern way, which should be a very exciting journey.

42:00Jon Krohn:Cool. Yeah. So number four about AI factories redefining resilience. Awesome. And five, your fifth of five predictions for 2026. It's around sovereign AI. Tell us about that. Yeah. Last year, we talked about sovereign a lot, saying there's different ways to do it. You could build government for government, where you build data centers for your government. You could build government for industry where you build data centers to enable your indigenous industries to be able to move fast on infrastructure they couldn't afford to build themselves. Then there was government with industry where the government plays a convening role.

42:33All good. Been happening all year long. What we're predicting is clearly there is a dramatic acceleration. There will be no country in the world that does not have a sovereign AI strategy as we go into 2020, 26 and into 2027. It's mainstream. What's exciting about it though, is that's all maybe predictable and understandable. What's exciting is now that you build out these sovereign infrastructures, what do you do with them? And it turns out you do a lot more than just government stuff. We're looking at in the future, what do we need them for? I don't know, resiliency. If I want to basically create a backup for a critical infrastructure AI capability, maybe doing it in a sovereign infrastructure in the country that the critical infrastructure is in would be a good idea.

43:18That seems like a good use case. If I want to create agents that live somewhere long-term, but those agents are in some way tied to a government, they are authorized or certified by that government. They're a Swiss lawyer or they're given validation by the fact that they meet a expectation and a certification a government provides. Maybe their knowledge graphs and all of their backend systems and data should be certified and live physically in an environment that has some connection to that sovereignty and on and on. And so what we're excited about is now that we've built a few of these things, sure, it's making government better, it's accelerating training infrastructure, but there's a lot more we can do with them.

43:55And that's pretty exciting when we think about the fact that government's role may in fact increase as we bring more and more AI systems online, especially in the enterprise, and especially where they intersect with things that have a national interest, critical infrastructure, defense, intelligence, healthcare, these kinds of services, it seems like it's almost impossible that we're going to be able to fully separate government from private sector in AI. And that's probably a good thing because collectively we have a lot more resources to apply to it.

44:23Jon Krohn:Nice. Thank you for that perspective. I don't spend enough time thinking about sovereign AI and we don't talk about it that much on the show. So it's nice to have your input there as your fifth and final prediction for 2026. Now I do see that you also had a bonus one on keeping an eye on quantum, but that is actually something we talked about a fair bit at the end of your previous episode on the show, 887. And so it looks like, I think you might have a hard stop coming up in terms of recording. So we might jump right to your book recommendation for us. Okay. So my, by the way, quantum, it's coming.

44:55It's just accelerating. Don't take your eye off of it. But for a book recommendation, this is a book that I got recently and it's called The Book, The Ultimate Guide to Rebuilding Civilization. And what's inside of it is if the world ends and civilization collapses, it has the knowledge of mills, drills, and lathes, spices, how fungus works, different medical systems, how to build an x-ray machine. And so I found that amusing because it's kind of like plan B. If the AI thing fails, maybe civilization collapse. And if it's successful, maybe civilization will be so different that we have to rebuild it.

45:34But having a book to help me do that seemed like a good idea to have my coffee table.

45:38Jon Krohn:I like it a lot. It's it seems like the kind of knowledge that I would love to have for myself. And I don't have kids yet, but, you know, someday would like to. And it seems like a kind of a fun thing to go over and help them understand how the world works. For people who didn't see it in the video, it is gigantic. It is maybe the biggest book I've ever seen. fantastic okay the book the ultimate guide to rebuilding civilization we'll have a link to that in the show notes as well as anything else we talked about in this episode uh and just the final thing from you john is obviously it is crystal clear to anybody who's been listening to this episode that you are a fountain of crisp knowledge on ai where else should they be following you after this episode to get more of your thoughts well you know we obviously anything Dell related, but we do a YouTube AI insights with John Rose.

46:26It's really just a narrative. We've been running it for over a year and it's kind of the, you know, weekly updates of what we're learning as we're going and, you know, very specific stuff. But, you know, one of the things we decided to do is to share what we were doing because it seems like, you know, we're figuring out some things that might be helpful to people. So AI insights with John Rose, probably the best way to see what I'm up to and, you know, and podcasts like this.

46:48Jon Krohn:Nice. I love that. Thanks, John, thank you so much for your time again today. It's an honor to get some time of yours and your no doubt extremely busy schedule. And yeah, hopefully we'll be able to check in with you again soon. Thanks, John.

47:04Jon Krohn:What a brilliant mind. What a brilliant leader. In today's episode, Dell's global CTO and chief AI officer, John Rose, detailed how we should be wary of agent washing where companies rebrand existing tools as agentic, despite a lack of autonomy. How Dell achieved something unprecedented in their 41-year history, revenue grew by$10 billion last year, while absolute costs declined by 4%, thanks to disciplined AI deployment with 10 to 1 and even 30 to 1 ROI ratios on some projects. John talked about a new knowledge layer and how it's forming between primary data and AI compute, including things like vector databases and knowledge graphs, and how this knowledge layer should live close to where the AI action happens.

47:47Jon Krohn:He also talked about how true autonomous agents have four components, an LLM for reasoning and communication, knowledge graphs for specialized knowledge and memory, MCP for perceiving and acting in the world, and A2A protocols for agent-to-agent communication. I hope you enjoyed all of John's predictions for 2026. As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for John Rose's social media profiles, as well as my own at superdatascience.com slash 953. Thanks to everyone on the Super Data Science podcast team, podcast manager Sonja Breivich, media editor Mario Pombo, partnerships manager Natalie Zajski, researcher Serge Massis, writer Dr.

48:27Jon Krohn:Zar Karche, and our founder Kirill Aromenko. Thanks to all of them for producing another super episode today, for enabling that super team to create this free Super Data Science podcast for you. We are deeply grateful to our sponsors. You can support the show by checking out our sponsors' links in the show notes. And if you ever want to sponsor the show, yourself. You can see how to do that at johnkrone.com slash podcast. Otherwise, help us out by sharing this episode with someone who would enjoy listening to it, review it on your favorite podcasting app or on YouTube. Subscribe, obviously, if you're not a subscriber, but most importantly, just keep on listening.

49:01Jon Krohn:I'm so grateful to have you listening. And I hope I can continue to make episodes you love for years and years to come till next time. Keep on rocking it out there. And I'm looking forward to enjoying another round of the super data science podcast with you very soon. Thank you.