Taming the 'Agent Zoo': How to Scale AI Without Losing Control

0:00I'm not a traditional data analytics person, but I've learned through FIRE just how important it is to the point where I'm picking up ontology books that I need to read. Because I do think next year the word of 27 is going to be ontology. Like it is so important to understand meaning and how things relate. I've learned real fast that if you don't have that context figured out, automating with AI is very, very difficult. Businesses are gaining access to AI faster than they're organizing the knowledge and systems needed to put it to work. A capable model is a starting point, but trusting it with a business process means understanding what it needs to know, how you'll check its work, and what it will cost when that process runs thousands of times.

0:38On today's show, Jason Campagna joins us with lessons from helping enterprises work through those decisions and testing them inside WWT and in his own experiments. We're lucky to have Jason because he's been spending the last six to eight weeks traveling the world helping enterprise organizations think through workflow design, business context, and the architecture needed to come together to make AI useful, reliable, and economical at scale. From Worldwide Technology, this is the AI Proving Ground Podcast. Let's jump in.

1:14Jason, welcome back to the AI Proving Ground Podcast. How are you today? Great. Good to be here. A little weary from all the travel, but doing well. Yeah, you've been all over the place, all over the world with AI days, with client technical advisory boards, client advisory boards. You are a jet setter these days. It's been a busy year on that front. Yeah. Putting the global in the global solutions and architectural teams, it feels like these days. Absolutely. And funny enough, started when worldwide wasn't really worldwide. So it's really actually quite awesome from a global perspective to be so involved.

1:45Excellent. Excellent. Well, I do know, you know, one of the presentations that you've provided during our AI days is what signaled us to ask you to appear on the show, specifically talking about how to scale AI in today's very complex landscape. At the base of your argument, of your presentation, you've said scaling AI depends on the workload, which feels like a little bit of a, well, no duh type of statement, but at the same time, incredibly complex and maybe not something that everybody's caught on to quite yet, fair or not fair. Yeah, very fair. And I think, you know, you look at like archetypes of use cases or patterns of use cases become that idea of a workload, you know, kind of like a grouping of them that have a certain architectural pattern.

2:28The best example of this is like traditional generative AI versus agentic AI versus maybe even physical AI. And, you know, agentic agents, they use a lot more tokens and they can do so on our behalf while we're sleeping. And they also create a lot more CPU demand, not only GPU demand. And so you have a different mix. That architecture potentially evolves a little bit. And then, of course, there's a heavy impact depending upon the workload on things like data and how you're managing the agents or managing the AI models and all that good stuff that intelligence represents. Yeah. Do we have a circle yet around kind of those types, those archetypes of workloads?

3:03You mentioned agentic. You mentioned physical. You mentioned generative. Are there kind of a set of others there or is it just kind of an unlimited bound? Yeah, I wouldn't – I would be careful not to put a boundary on it at the moment. If you like – you ask NVIDIA this question, it quickly becomes a verticalized conversation. It also becomes a not just an LLM conversation like CNN, GNNs, world models. There are other models out there we don't really talk about a lot because we're all embedded in CHPT or CLOT or what have you. But those, you know, if you're a large pharmaceutical organization, the kinds of models you might use are not necessarily only LLMs.

3:41And so those workload patterns span model types. And, you know, and of course there becomes a factory architecture that goes along for the ride of, well, how do I actually support that? And when we say factory, we are referring to, you know, the portfolio approach to a factory. So on-prem versions of a factory, co-location, you know, the idea of a neocloud or an NCP, and of course a public cloud. I mean, the large pharmas of the world, certainly on the cutting edge, are you thinking that some of those other types of models, the CNNs, the world models, things like that, those are going to be eventually used in all enterprises across the board?

4:15Yeah, I think it's a matter of time. Right now you're starting to see more like SLM, TLM, small language model, tiny language models. And we quickly probably have to unpack the idea of workforce AI. Like we all use our tool of choice, platform of choice. It's more of an assistant bridging into an agent. But it's arguably, you know, it's funny, everyone seems to use 25 % as the savings. I see it with customer slides, with our slides, our metrics that's kind of like, okay, we're going to save some time. You don't really start to get into agentic and some of these other kinds of use cases until you unpack the workflow and really dissect the workflow to understand, well, where can I use a model?

4:54Where does a human need to be on point? Where is a human in the loop approving a model? Where is a human on the loop, to use the Singapore phrasing of it? The idea of a human is watching it happen, but not necessarily needing to approve each action. But you start to unpack a workflow, like an insurance claims workflow, for example, that we recently worked with a customer on. There's 78 agents in that workflow, not one, not two. Unpacking that is actually how you extract a low cost per run and higher accuracy. Those are actually concurrent, interestingly enough. And so that idea of like an SLM or a TLM or a different model, depending upon that workflow, it may be that there is a different style model in the future, depending upon what a customer is trying to get done.

5:40You mentioned that 75 or 78 agents from within at least one customer. I mean, whether it's agents that we're building internally or those that we're kind of buying from SaaS products. I mean, agents are coming from all over the place. You've referred to this as the agent zoo, which I think is a funny term there. Is Agent Zoo, I mean, is there a point of diminishing returns of too many agents or is this a good thing? Yeah, it's a sprawl problem. Every customer, including us, we all have. Every SaaS product we've ever heard of is coming at us. As we were talking ahead of time, like my Oura ring on sleep from international travel.

6:14There's an embedded AI in the Oura app that is a perfect example of, okay, well, do I go there or do I want to pull my data into Claude or how does that look? and all these different ways of doing things that are maybe right or wrong. You know, basically every product we use has some version of that. And then, of course, well, we get to build our own custom agents, which, of course, we're doing internally as well with our Atom approach generally. You know, that is the problem set that customers are dealing with. And most importantly, dealing with identity management and control systems across those platforms.

6:49How do I deal with the ones that are over in Azure AI Foundry and the ones I have over here in Bedrock, and then I have a couple over here that are in, you know, pick your platform. That is a key challenge, and there really isn't a good answer right now. So it's a multiple choice answer, whether we like it or not. Yeah. I want to stick with the agent zoo type of analogy here. Think about a zoo that you might visit in any city across the world. They're segmented, right? There's the reptile house, there's elephants on one side, and then, you know, there's certainly ways to navigate and get around.

7:21What can we learn from kind of that physical zoo on how we should manage agents? It's funny how you phrase that. In our AI days, I've joked about this now going on a year plus. Almost kind of got like a little shade in the early days. It's like, wait a minute, that's too messy. But it is the reality our customers are dealing with. And just like you said in the zoo, the decision intelligence platforms is kind of a pattern of animals in the zoo. And another one might be more the traditional automation platforms. And yet another one might be more, you know, the machine learning, data and analytics kind of platforms.

7:54Like there are like subspaces to that agentic zoo. And there's a crazy ton of overlap. One of the most common conversations we've been having lately is harnesses and loops. What exactly is a harness? And unfortunately for all of us, harnesses use it several layers. And that idea of like a core, like multiple agents in one runtime environment and the harness associated with that is how I personally have kept it sane. But you come up a layer, you know, a verification loop or an overall control plane. Gartner calls this the most valuable real estate in AI. That idea of a control tower or a control agentic control platform of some kind.

8:31And why is that? Why is that the most kind of important spot? Everyone's trying to make sure that the agents are doing what they're supposed to be doing, track token consumption, understand how to improve the system, all the things associated with that, like more orchestration, agentic platform layer, if you will. Yeah, yeah. I mean, so you have the agent zoo here, you have different workloads that are demanding different requirements of the infrastructure. Let's talk a little bit more about what all this does demand from an architecture perspective. Do we treat different workloads with different architectural decisions?

9:06Are we just looking for efficiency or cost optimization or what goes into all this decision making? I think there's like a couple of levers we're talking about here. Economics is definitely one of them. Especially in an agentic use case, as the run count increases of that workflow. You know, let's say it's$2 per run with, you know, frontier models or foundational, you know, like an Opus or Fable or similar. You know, let's say it's$2 per run. And then the same thing dissected with small language models in an on-prem instance might be$0.02 per run versus$2 per run. But, you know, so if you think of like, how do we, how are we unpacking that?

9:49What's it actually look like to apply that to the problems that, you know, becomes a key part of our conversation? I don't know architecturally, it's like a direct, okay, this workload, and then there's a whole different architecture. Right. It's not like, it's not quite pattern like that. Yeah. But economics is obviously a key lever. So anyway, sovereign data is another very significant level. Where is my data? I believe data has gravity. And a lot of times you're going to need to build inferencing right where the data is. It'll simply be more cost effective. And SLMs come into that pretty quickly.

10:21Latency requirements. You know, there are certain things that just, okay, cloud's going to be a problem because X. And it's not that cloud's not secure. It's that it has other demands among other aspects. Yeah. Well, I mean, certainly, once an organization or once we've figured out where we want AI to perform or where we want it to run, it still has to understand what and how and why it's doing what it is we've asked it to do or what an organization does. This is where we get into context a lot, and I feel like I've seen the word context really start to show up in a lot of different venues as not just a challenge but as a key unlock, giving it more context.

11:02But this is all stuff that's typically in our heads. Maybe it's somebody that – Different teams. Different teams. Somebody that maybe retired a while ago or went on to a different company. This seems like a near impossible task to try to curate all the context that would be needed. How do we start to build foundations or get started to get all that context on paper, so to speak, so that you can enable NKI? It's probably the hardest challenge of our time. Data is not context. Context is the right information at the right time to feed a model. And less data is actually more in many cases. Not only from a cost perspective, interestingly enough, there's a side effect of that, but the quality of output goes way up.

11:46I know we've experimented internally with the transcripts of this podcast, among other things, as data sources to inform our approach. Putting metadata on that and linking themes across it, linking partnerships across it, all of that, you get way better results. It may be useful to kind of just briefly step back, like overall architectural framing, you know, to your point about, if you look at AI native enterprise architecture, what do we mean by that these days? I've long been a student of complicated architectures early in my career with Citrix, which is about as bad as I get from a complicated architectural perspective, VDI, virtualization, of course, cloud, cloud-native development, all that kind of fun stuff.

12:26I do think that enterprise architecture is being upset in some really unique ways right now. The traditional swim lanes of applications, data, infrastructure, and security, those four, Security is sometimes in, sometimes not. These days, it's very much in. But those four have been around for a long time. We're structured that way. Our customers are structured that way. And then suddenly intelligence has come along. And intelligence, we're using as an umbrella term, meaning the harnesses, agentic frameworks, platforms, models, model gardens, AI gateways. The idea of building a computational graph, to throw a kind of an interesting word, or a cognitive architecture, which I know those like.

13:05It's like AI 101 or building an agentic architecture 101. You have to actually build how the things connect, what information they need when, how they feed to each other, what the ins and outs are for each thing that all of us humans do natively. But to build that into an actual thing is a big effort and controlling its ins and outs. That's all the intelligence layer from our perspective. Yeah. And we've been doing this internally for the last several months, if not years. We have an AI initiative, Strata, that we use. It does a lot of the stuff that you're talking about here. Maybe just brief us on what Strata is so that we can tee up a segment where we offer up some unique insights that we have so that others might be able to replicate.

13:47Yeah. So if you think of like intelligence data infrastructure as those three layers supporting the build of an experience from an application perspective, all secure, right? You know, data is feeding the models, arguably, and feeding the right context. And so, you know, the traditional, and there's been a long joke of like data lakes, data mesh that, I don't know, was it 20 years now? We've been talking about that. Now you actually really have to do that. And so I kind of view it as like, there's two parts. Part A is the standardization, semantic alignment, you know, cleaning and making data AI ready in a broad manner.

14:23And then there's like, well, how do I traverse and deal with that sea of all this information? And I don't, I'm not a data and analytics person, but I know like Rachel and others are, they're tuned into that. Like Mike Smith and others, like they're tuned into all these data sources. I view Strata as that advisor for all of us. It's an agent that sits on top of our data infrastructure that can provide insights into what you might need for whatever you're trying to get accomplished. And you don't have to go decode all the stuff that sits in the background. I think we're going to see an evolution of that to be also, well, not the data source do I need, but what context do I need?

15:01And while we're not like full-fledged knowledge graphs or we were talking about the ontology pipeline here, You know, the idea of an ontology is where we're all headed. And it's hard. It's not an easy thing to actually link in, like, map entities, map the semantics across all the stuff. And then, of course, an AI can traverse that graph and then go downstream into specific data sources when it needs more. It's what everyone's chasing. And if you go outside of like Strata and Worldwide's approach, like Databricks or like Snowflake, they have their version of a data advisor that sits on top of their platforms.

15:37So to your earlier point on the agentic zoo, there are elements of the whole architecture that show up in literally every platform. And then the question is at a customer level, well, which one do I use? And the honest answer is probably a couple different ones depending upon where your data lives and how you want to evolve it. But feeding the model, that idea of the context window, like what do I put in the context window? And we don't often dissect that in great detail. Some of it's like the soul of your AI that you're working with, like your Claude MD file that you can kind of tweak, the prompt itself, what tools you have available, not that are loaded, but that are available, right?

16:13That's like constitution and identity. Then there's working memory that's really more like your notepad that you have there. Like what are you working with right now that you have in your mental frame? And then, of course, in the technology world, the idea of a second brain, I can extend that greatly into a broader memory system. So like those 78 agents for that insurance use case, there's probably three different memory systems that are associated with chunks of that. But it's not like every single one of them has to go figure out all the data. There's more of a contextualized memory source for that specific workload.

16:45And then last is the idea of canonical data sources that arguably something like Stratus sits on top of that pulls all that together. I was going to ask you where model selection comes in, but I'm kind of gathering that there is not going to be a singular or even a handful of models that you're selecting from. You're going to have a vast array of models and tools and things that you can call from. And is the idea that strata or that context and that intelligence layer is going to help route to where you need to go best? Yeah. So think of strata as more of the agent at the data level to inform what data or what context you need.

17:21it's a that's a it's a little different that question it's really more in the intelligence stack so to speak ai gateways are all the rage right now how do you see the flow of tokens and then i do think that of and and how do i know who's using those tokens so think like identity management security mashed up with raw token flow and sitting in the middle of the api call and so i can see that the api call went out to claude for an opus model or sonnet i can see that The API call went out to this other thing. And so that gateway sits in the middle of all that. Yeah. And that gateway is often part of the agentic platforms that sit in that intelligence space.

17:59Where that's headed, though, is why can't I redirect that simple call to an on-prem or perhaps not saying on-prem, let's just say an open weight model that I can either control the weights of, I can fine-tune for my organization, and it can funnel and probably cheaper as well. lower latency, arguably closer to data, and I redirect that API call away from what is viewed as an expensive model to a cheaper model or something similar. I think we're headed for a lot of decision being made at the token flow level. It's out of ban from the workflow. It'll have its own problems, but it's definitely headed that way.

18:39Yeah. Let's go back to the Agent Zoo example here. What does that intelligence layer, What does the Strata type of mindset, what does it do to help make that zoo operate, function, be more secure, so on and so forth? In the case of Strata, I view it as any AI tool you're using, in theory, could have an MCP call to our Strata advisor and then dig up the right things to bring that information forward. So for me, there's a little bit of a segregation there between that data and context layer and multiple AI platforms that we're using just like our customers are using. And then, of course, there's fun access controls that goes along for the ride that have to be very top of mind.

19:22They make it very security first in approach. But those are different conversations. But that agentic zoo simplifying the data layer, which I think is what Strata represents, because it's really hard. I mean, you've tried to divert. We've talked about this out of band, like traversing and finding the right information you want is really difficult. For me, the dream of Stratus greatly simplifies that so that we can build more and consume that information with the right intelligence layer, models, harnesses, however you want to look at it, so to speak. Well, if you had an organization or somebody that was just asking you, well, how can I replicate this within my environment to get closer to that dream of having the right context, having the right information kind of at your fingertips?

20:05What are some of the hard lessons that we learned? You know, don't have to go too deep, but just what are some of the tenets that you would say, you know, here's kind of what you need to do to get yourself on the right path? Yeah, I think there's like two – for me, there's two layers of it. One is for the broader workforce and enablement of meat and potatoes data. Like for me, Salesforce is a great example of that for us internally or some similar. It's like, okay, get an MCP connector online to connect to whatever that thing is. If your healthcare might be an epic, you know, like that kind of a dialogue for me is like a workforce conversation that is accessibility of canonical data sources that exist.

20:41Then there's another version of the world that is, well, what context do I need for this workflow? So Nate Mackey talks quite a bit in the AI Day keynotes about, you know, define a function and software development principles of like test driven development. But one of the things that he has in there is like, well, for our supply chain side, there is a very specific function where you could take a picture and then using the computer vision, different model, by the way, you can look at that picture and decide whether the work was done or not. And it was a 92 % success rate. It's very, very targeted.

21:12And that's a scenario where, well, what do I need to be able to make this decision? What data do I need? What model do I need? And what verification loop do I need to make sure that's happened? By the way, verification loop ideally is deterministic or maybe needs to escalate to a human. But either way, those primitives in the workflow conversation mean I might need to go collect the data. I need someone to go take the picture. And so you start to look at like individual use cases, individual workload patterns, individual workflows, and you start to go, well, what data do I actually need to get?

21:47In some cases, I got to generate it. In other cases, I've got to go consolidate it from another source or I need to clean it up or whatever it is that will inform that broader approach of how data is evolving. And this is really hard, by the way, and well above my pay grade from a data analytics person. But I personally have dug into this because of need. I want to be able to fire my AI platform at will. On the personal front, I ditched Claude and went to GrokBot and went into Cursor. And it was about an hour migration, and all my agents came back up as if I had never left because all my contacts, my memory systems were stored out of Claude.

22:24Yeah. Yeah, you've mentioned and I've talked to you about how you are fairly advanced from a personal side. I mean, certainly more than capable and fantastic talking to our customers about how to advance their own AI strategies. But you've been doing a lot on your own personally. What have you learned there? Just curious that would resonate or would be relevant to an enterprise audience. So what's interesting is even at the personal level, like the architectural primitives are the same. And so the idea of, well, what harnesses am I using? How am I having a verification loop? What context am I feeding?

22:57What data? What infrastructure am I on, right? And including open weight models on like a Halo or a DGX Spark. I have some local inferencing in my own home lab that has, you know, nothing compared to certain others around here that have like crazy labs. Like Papillo comes to mind. But, you know, testing a bit between the different models. But I learned really fast that I was kind of sick and tired of going from Gemini to ChatsyBT to Claude and having different understanding of me and my world and my family and what I like and areas of effort and everything else. And it pushed me down this data journey.

23:32You know, I'm not a traditional data analytics person, but I've learned through FIRE just how important it is to the point where I'm picking up ontology books that I need to read. Because I do think next year the word of 27 is going to be ontology. Like, it is so important to understand meaning and how things relate, systems of action versus – that entire thing is fascinating to me, but also extremely valuable in actually getting outputs. And being able to automate with AI, I've learned real fast that if you don't have that context figured out, automate with AI is very, very difficult. Yeah. The idea of true agentic autonomy.

24:07Well, we'll write down the ontology prediction for 2027. Yeah. We usually don't ask people until December about predictions, but we'll make an exception. It's going to show up in priorities for next year. We're working on that report. All right. Well, we look forward to reading that. I wanted to touch on, you mentioned how you've got private inference going on for your own personal AI stuff. And I've heard you cite that as examples for savings. How much opportunity is available in savings and getting more out of the architecture and the infrastructure that we already have or that we're going to be building?

24:36I mean, so as you get tighter on context, you're feeding the model with less information. And in theory, inferencing costs go down. So they're real. And also accuracy goes up. I think I probably rattled that off before. I do think that that's kind of like the main event for as you unpack a workflow. And getting to smaller models is a theme we do continue to see. and you know you get into this crazy rabbit hole matrix style of the mixture of expert model approach that is really showing you don't even see it but it's in the background of fable it's in the background of kimi k3 it's in the background of these larger foundational frontier models where in reality when you ask a question it like turns on different parts that mixture of experts and it may do a lot of things we don't see in a broader identric architecture you may you may unpack that yourself deliberately with a bunch of little SLMs because you can tune that one to understand that picture that was taken.

25:30Is it, was worked on or not? Yes. Okay. Well, next step of the workflow. And each of those can be tuned and they're smaller and now suddenly have a conversation of, well, where am I running those? Should I put them next to the data that I have? Should I put them next in the manufacturing facility? Should I, where, where do I want to host those? And then there is simply a raw economics conversation between owned infrastructure, co-located or on-prem. And then of course, NeoClouds, it's kind of in the middle and then on up from a public cloud or foundational model API perspective. This is the same conversation we've been having.

25:59I'm an old cloud guy, right? We've been having this conversation for years, private cloud, public cloud, hybrid cloud. For AI, it is a portfolio approach. I don't see many organizations that are not going to have multiple layers of that. The thing that no one's really talking about is we're all consuming lots of tokens right now. Or a few seem to be talking about it. We're consuming lots of tokens. There's lots of workforce AI in there. There's some workflow reinvent, but that agentic cycle, if you will, is like, we're, we're like the first pitch of the first inning kind of a thing. Yeah. As more and more agents are consumed that we trust them because we can feed them context and all of that.

26:35I think there's going to be a light shined on, well, how many times am I doing that? Like cost per run is a thing that I know Tim Brooks and others talk about at the AI days. That cost per run is a critical aspect. Certain kinds of specific workflows probably deserve more attention on the economics front, latency front, accuracy front, because they're very impactful to the mission of that business and how we apply AI for that specific experience. Yeah. Interesting. Well, Jason, I know we're running up on time here. We've borrowed enough of your time and you've got a lot of reading to do with that ontology pipeline that looks like some dense type of stuff.

27:08But we always appreciate you stopping by here in the studio. I'm sure we'll have you on again soon. Of course. Looking forward to it. Awesome. Thank you. Okay. Thanks to Jason for joining. the takeaway scaling AI starts with understanding the work to find the task, give AI the context it needs and verify the result. Those decisions shape both reliability and cost. This episode of the AI Proving Ground podcast was co-produced by Naz Baker and Kara Kuhn. Our audio video engineer is John Knobloch. My name is Brian Felt. Thanks for listening. We'll see you next time.

27:46Thank you.