From AGENTS.md to Enterprise Deployment

0:01Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, on to the show.

0:41Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a principal AI and autonomy research engineer. How are you doing, Chris? Hey, doing great today. How's it going? It's going great. And I know this is going to be a great conversation for a few different reasons today. One of those being our guest is one of the speakers at the upcoming Midwest AI Summit, which is going to be amazing. There's going to be a bunch of amazing speakers there. I'm going to do a bit of emceeing, see if I don't mess that up.

1:19Would encourage our listeners to check that out October 15th in Indianapolis. You can get 20 % off with Practical AI 20. So check that out. So amazing speaker, going to be at the Midwest AI Summit. Also a fellow podcaster, which is great. One of the hosts of Cloud Foundry Weekly. and a tech marketing whiz at Broadcom Tansu VMware. So welcome, Nick. Great to have you. Thanks for having me. Great to be on the show. Yeah. Also excited to see you all in person in Indianapolis. Yeah, it's going to be fun. We're representing the good representation of the Silicon Prairie here. I like it. So, yeah, Nick, I know you're going to be talking.

2:10One of the things I thought was cool about your talk at the Midwest AI Summit is you bring in this concept of taking agents.md and shipping that to a production environment where you run an actual agent in an enterprise environment, which seems to be really fascinating. We recently had on someone from the Agentic AI Foundation who is kind of stewarding the model context protocol and agents.md and those sorts of things. So very relevant kind of carry on from that conversation. But I'm wondering as we set that up, what exactly does it mean to be operating in an enterprise environment? So the customers that you're working with, your kind of day-to-day, what makes an enterprise environment an enterprise environment?

3:06What are some of the concerns or characteristics of that type of environment? And I know you're always experimenting. You do a lot of home lab stuff as well. So always have a whole range of experience in working in different sorts of environments. So yeah, if you could just help us understand from a general perspective, what does that mean when you say kind of enterprise? Right. So yeah, if we think of the enterprise, we think about, you know, a few things. One, let's just assume you're going to have limited or no internet access. And, you know, I've been at VMortensia for about five years. And before that, I was at one of these large enterprises for about 14 years.

3:51So when we had vendors come in, it would always be like, they would always assume that we could just go to the internet. And it was always like, well, try again, because that's not going to work here. So that's kind of always like the biggest hurdle where you're going to be regulated and or, you know, you're not going to, what you can do at home is not going to be what you can do at work, so to speak. So you're going to be highly regulated. You're going to have to deal with potential PCI or SOX compliance or HIPAA or FIPS. And I could probably go on and on about all the different compliance tiers that you'd have to deal with.

4:25So it's just a whole different ballgame where you could potentially do something that has dramatic consequences. And you have a highly controlled, highly regulated environment where even some of the customers I work with, they're like, you know, they actually have the real air gap where we're carrying in things, you know, physically into the data center because there is no internet access type of thing. So it's a wild world, but it frustrates a lot of, I think, new folks to the enterprise base. But if you've been in a while, you know how to deal with it and how to handle it type of thing. And I guess, you know, people have been dealing with these types of environments for for quite a while, right?

5:07Because there've been enterprises for quite a while. And maybe just give us a sense of like leading up to AI agents, and we'll talk about agents here in a second, but leading up to AI agents, like what were some of the ways that you could, I guess, ease, you know, put some ointment on those pains of working in that sort of environment? How would you get an application into that sort of environment or or have it be enterprise ready quote unquote and then we can shift and talk about you know what changes with with agents yeah so i mean i think so you know i currently work for vmortanzu and we run a you know a platform or a platform as a service uh you know geared for private cloud whether that be on your like on your own bare metal in your data center or you could deploy it on a private cloud in your own like VPC on a hyperscaler, that type of thing.

6:03But we think about that, you know, we want to, in the scope of an enterprise, right, you have all these regulations, controls, et cetera, you want to make that path to production kind of the least resistant and easiest path. And if you make that path to production, like all the checkboxes and everything good to go, that's going to be a highly adopted path. So the whole concept of, you know, a Tinsu platform, and it's actually based on an open source project called Cloud Foundry, which is, I think, I mean, it's, you know, predates Kubernetes and Docker. So, you know, 2011-ish is when it kind of started originally out of VMware and spun out and then became part of an open source project.

6:38So it's been around and battle tested and seen a lot of things where you have that concept of like, I just want to take my code and send it to the platform. And the platform is going to know like the best practices for, it's going to build a container for the developer. So the developer doesn't have to worry about like, you know, securing the container or anything like that, the platform would take care of that. It will handle, you know, ingress, like certs, certificates, all the, you know, load balancers, all those things just handle health monitoring, potentially even kind of like sandboxing the apps from each other.

7:09Something that, you know, maybe some of these labs may learn from, but so that the apps can't escape type of thing. And, you know, if the app needs services like a database or messaging, middleware, any type of thing, even maybe a large language model, it can, what we call bind or kind of connect to a service on the fly. And that's all kind of handled underneath the cover. So it's just a few commands like you push, bind, scale your app with just a few simple constructs. And that's how you go to production. And what we see with enterprises, once that path, like once they take this, like this is a Tansy platform, it's been certified through all these different standards, it's just kind of like an unlock for these enterprises because it's so easy and so simple to use and they don't have to, you know, they don't have to recertify everything.

7:59It's just like this framework that they can use to get apps into production. And then the biggest thing is that you want these, the enterprises or the businesses, they don't want their developers, you know, hand crafting a new platform or a new way to deploy apps for every sub team and have like 100 different snowflakes, right? They want the same repeatable pattern across the board so that they can audit it and secure it and have their developers actually spend time writing the business logic versus like fiddling with infrastructure. So that's kind of like the whole premise behind, you know, the whole platform as a surface construct and, you know, Tainsit platform.

8:32And as you can imagine, go ahead. Yeah. No, no, go ahead. Finish up and then I'll follow up. All good. Okay. As you could imagine, those principles when we're talking about apps could probably play nicely into this agent world that we're living in. So it sounds like you're drawing a lot of commonality between kind of traditional app development and all of the structure that we're all used to, that we've been doing forever. And kind of now we're into the syngentic world, and maybe people have been thinking about that in a different way. But it sounds like you've really kind of said, and correct me if I'm wrong, I'm going to throw something out there.

9:11like it's kind of the same thing in the sense of there are differences but agents and apps should sort of be treated the same way is that am I getting that correctly in terms of of how you're structuring that and that maybe people need to look at what they've been doing for years and make that work for agents in the same way is that is that fair yeah I think it's a pretty fair assessment and we're starting to see that more and more just within our customer base and you know even like if you think about like apps back in the day, like someone would write some app on their laptop and like run it on their laptop, but it's like, we can't just run it on, you know, like you got to get it off your laptop somewhere.

9:50And the same thing with agents. You're like, I got this, I've got all these agents who are on my laptop. If I shut my lid, everything stops. Like I don't really want that to happen, right? I want them to run 24 seven, you know, 365 and do all the work I give them. So yeah, I think that's kind of precisely what we're going with. So I guess then it begs the question, why are people kind of throwing out what maybe their intuition uh like in certain ways people are saying oh we have agents now we need to do something totally different which is maybe driven by some things that make an agent not an app not just a sort of simple application but also it doesn't necessarily mean you know obviously we have to throw out everything we've learned about platform engineering and and how to deploy things.

10:39So from your mindset, where does the kind of agent equals just another app, where might that fall apart or what are the kind of unique characteristics that you're dealing with on the agentic side on the deployment pathway that you might not have had to consider before or at least are considering in a new way? Well, I mean, I think some of the biggest differences, right, where like you talk about Cloud Foundry and TNC platform, it was kind of based on this premise of a 12-factor application where storage and state are a little decoupled cleanly from the actual apps. I can scale up to a thousand instances and be fine and you handle session state and things.

11:20The traditional harnesses or agents were kind of built originally just like, I've got file system access and I can just write a bunch of MD files and that's like my memory. Everything's great. Well, you start to get into a cloud world that things spin up and down and are ephemeral. you're going to want to save those MD files somewhere they're going to go off into the ether. So that's probably I think the biggest challenge when we start talking about how do we take the app deployment methodology and then push that into an agent running on a cloud platform per se. It's probably the biggest challenge.

11:54And then just kind of getting the fact that there's some valid use cases for your laptop or whatever but then also as we're starting to see more and more enterprises want agents to be on demand and usable say in a CICD pipeline or within part of their e-commerce suite or just normal applications and you want those to be kind of really close to the apps because if you have them way off even just from a pure networking perspective if you have the agent or the LLM or whatever that's 30 hops away from the microservices trying to call it, there's physics that are going to add on to all of that latency and potentially at scale even be problematic.

12:33So they kind of want to collate all the, co-locate the intelligence with the app and the data as close as possible. That's what we've seen too. That's something that I didn't expect really because some of these responses are kind of long, but it's like what we've seen with our customers at scale. Like we want to get the LLMs and the agents closer to us and our actual apps using them and the people using them as well. And not to mention, if you're using some of these SaaS providers, they're maybe not the most reliable in terms of what traditional enterprises expect and demand. So you can't just have them going offline for hours because even at my past job, the very first thing I did there was work on a warehouse inventory system.

13:11And let's just say it was for a large retailer that was a grocery retailer in the US. And I was an intern out of college and the app was older than me at the time, still older than me. And it was a bunch of C and shell scripts on Unix systems. And if that thing went down, like within five minutes, like the warehouse was backing up trucks like on the interstate. So it's like, you have to like, the scale of enterprise versus just, you know, like consumer is a completely different scale. So that's why some of all the things we're trying to make them a little bit more enterprise grade and more used to what these enterprises expect from software.

13:54Some mornings as I'm drinking my morning coffee. I listened to live news updates to figure out what's going on in the world, and it seems like recently everyone is talking about AI kill switches. But the reality is that there is no one AI kill switch. We've moved from models to agents. Those agents are connected to multiple models. They have an agent harness. They're connected to MCP servers and tools, and those agents are acting within a fleet, delegating to one another and interacting with one another. In that environment, what you don't have is a single kill switch that solves all of your problems.

14:33But that doesn't mean that you can't exert your control. You just have to exert that control across various layers of that stack through, of course, component input output safeguards, but also controls on agents tied to their agent identity, things that track tool misuse or memory poisoning or goal drift, etc. And what we're providing at PredictionGuard, the company that I lead personally, is an AI control plane that you self-host in your own infrastructure that gives you that control plane or control layer for the relevant stack that agents operate on top of. I'd encourage you to check out what we're doing at predictionguard.com slash practical AI.

15:23Go ahead and book a call with myself and the team to learn more about how you can gain control of the agents operating in your environment and exert zero trust principles in the way that you operate those agents. So check us out at predictionguard.com slash practical AI. That's predictionguard.com slash practical AI. So Nick, I would love to dive back. I want to kind of build on what you've been sharing so far. And part of that is that we're all diving into this multi-agentic world where, you know, some, you know, late last year, you know, you'd have an agent. and then for most people I think getting into multi-agents it was probably developers in the environments that the AI providers were making available and you had multiple agents working on a project but now we're really seeing with these work products and other things that have come out we're seeing agents really multiplying across many different contexts and with a lot of different utility and it's moved out of being just frontier services and we're now doing that with our hosted agents as well that we're running.

16:44And so like this multi-agentic world has just taken off in a very short amount of time. And as you're, I guess I'm asking if you can kind of paint a picture as you start scaling this out in the enterprise and what that looks like, Like, could you describe, you know, and we have analogies, as you've pointed out, with kind of the traditional software development world and the containers and all the, you know, and how you're structuring deployment and stuff. Could you talk a little bit about what that looks like, kind of paint a picture so that somebody can really get that in their head? Me, first of all, about what that multi-agent world looks like when you were doing all these right things that you need to do in the enterprise.

17:33Yeah, so I think that the first thing that we want to focus on is that the enterprise needs to get whatever technology approved throughout their system. And then we are providing a harness called the Tenzu Agent Build Pack, and that helps through that process. So our existing customers already have our customer, we're already approved so that we can help provide them those tools. And we just make that process just another app deployment on the platform. So I probably haven't covered, they said, I'll explain what a build pack is, right? So when you, when you. Yeah, I was about to, I was about to ask you that.

18:08Right. If you could kind of dive into what the specifics of each of those things does. Sure. So when I would do demos before AI and before, you know, a year or two ago or whatever, we would always take like a Java app and it was like a simple kind of like, you know, music album, we called it Spring Music or whatever. But we take that Java code to be like a jar file, like compiled Java code. And we're like, all right, now we're going to take this. We have a little simple manifest file that tells us what the name of the app is, how much resources to give it. And we just run a command called CFPush.

18:35And that would then just take that code, the jar file, and upload it to the platform. And the platform would be like, oh, it's a Java app. So I'm going to use the Java Build Pack. And the Build Pack is basically kind of a command set to make a best practice container based off that use case. So the Java Build Pack knows to do all the JVM memory calculators, do the JDK and certificates. and everything that's best practiced to run Java in a container on a cloud platform. And that would be great. And then we would attach a database and all these things and show how that scales. Now, we've taken that same concept and then said, hey, we're going to have this agent.

19:11We're going to just CF push an agent. And instead of machine-readable code, we have a human-readable language, the agents MD. So basically, you tell what the agent it's going to do, and then you just push it to the platform, and it starts up within a minute type of thing. and you can scale that up, push as many agents as you want. And I think the coolest thing that I've seen is that once we're on the platform, then we can run models on the platform. So obviously a lot of our customers are air-gapped and some of them were smart enough to buy GPUs a few years ago. So some of them have lots of GPUs that can run all these local models and have a great experience.

19:46Some are wishing that they bought GPUs and hardware as the pricing of all this hardware goes through the roof. But then they can run local models and then tie the agent to that Or if they don't have, they might have a contract with one of their cloud providers or one of their approved LLMs of choice to register that. Then that kind of gave us a simple, here's your out-of-box agent with chat experience. And then along the way, we've done a lot of things with MCP, the model context protocol. Probably you guys covered that on the show before, but it's basically a way for an app to talk to an LLM.

20:21Very high level, right? So that's become useful as well as we then give the agent tools via MCP servers. And even at Broadcom, the only way approved way to use for our developers is to deploy MCP servers on Tansy platform. So that we have a whole set of MCP servers that are approved and hosted and they're kind of secured and maintained because they're very similar to applications as well. And then we have a concept of like the MCP gateway service that kind of registers that and secures all those servers. So then we can then extend the gateway to that agent. So then the gateway, this agent now has like secure access to a runtime, you know, battle test runtime.

20:58It can talk to an LLM that's approved and get tools via gateway. And then we're starting to add in like the memory service as well to have a kind of a shared memory service. So we're kind of expanding all that out where we're seeing all those best practice app lifecycle. Go to, you know, see a push, go, here's an agent, here's all the things it wants to do. And then, you know, it can be a security review agent. And like one of the things I showed, I was at a, I think a week or two ago, I can't quite remember. I was in Las Vegas for a conference, VMware Explorer. And I had like a five-phase demo where I walked through this.

21:33Maybe I'll show a little bit of this at the Midwest Summit here. but push the agent, bind all these services but the end goal was like the boss level was like all right, now we have these agents sitting in there and I can hook them up to a repo with an event listener or a webhook and say, all right, now do issue five on my repo, do a security review on this repository and then it sends it off to the agent via webhook and like, oh, the Tenzu agent fires up, gets it, fires up from the platform it gets a request and it just immediately pulls down the repo and starts doing a security review and then publishes or pushes up to the issue that I created on GitHub and says, this is security review.

22:13So you can see how that flow would be really useful in enterprise where like maybe on every git commit, you just have all these agents spinning up on the fly that could be a security review agent, potentially like a style or code quality review agent or potentially even like a, I don't know, like maybe a design review agent, all firing up on the fly as things like call. So it's kind of like, and then they all have their, they're all in their own little sandbox and everything's proper, right? Because part of, you know, the platform for a while, it's been like this very secured container runtime where like we have, you know, they can't talk to each other unless you enable them.

22:48So they can't go to the internet. Unless that's enabled, they can't, you know, escalate. Otherwise, they only get access to what you give them. So it's kind of that benefit as well. And then obviously they're containerized. They don't have access to the, underlying host system as well. So as we've seen some of these things of uncontrolled agent swarms, I think we have some remedies here that some of these AI labs could maybe take into account and implement some best practices that's learned throughout the past two decades. The unplanned swarms, the swarms that you don't necessarily want. I actually have a two-second follow-up and that is for my benefit.

23:25And we've talked a lot on the show over various episodes about agents using MCP to get access to different apps and services and tools and stuff like that out there. We haven't really talked about what an MCP gateway is, and I was wondering if you could share that for a moment just as a detail, because that's kind of the one little new piece of architecture that we've never touched on on the show. Yeah, so if you think about an MCP gateway as a way to easily control and scale access to MCP servers, especially within, you're in a cloud, I guess, platform. So we may have like 30 or 40 MCP servers that we run, or I don't even know how many it is, right?

24:06And then those get bound, those get registered and bound to certain specific MCP gateways. So maybe there's a, you know, and the MCP gateway is something we can spit up on the fly on the platform. There are many different MCP gateways out there. There's many different AI gateways out there. Some of them are like the same, one and the same. But you basically register MCP servers to this gateway and the gateway kind of controls access, so to speak, and regulates that. So it may say you can bind to this gateway and you only get these MCP servers you only get these MCP tools within it, that type of thing.

24:40So it's a way to easily abstract all the different MCP servers that may be in an organization and kind of direct them into either a certain agent or we even have a flow where we'll take those MCP gateways that we run that front all of our MCP servers. And then you can say, hey, go give me a way to connect this to my cursor or my cloud code or my whatever. And then it just injects it in there. And then your local agent has all the tools that are approved as well for your organization. And then it's just like a simple click and then it just registers. And then depending on the MCP server, it helps the handle the auth as well.

25:14Because some of them are like, well, if I'm using the GitHub MCP server, I want to pass my end user credentials all the way through to that MCP server because I don't want it just to be like, well, here's our generic service account for the entire organization and every GitHub action is going to be seen by that one account. And you have to have the individual identity pass through. So you have the concept of SSO and sign-in as well and identity for agents or humans running their agents on their desktop. So that's kind of a quick level of view. And then a great way about that too is because it's all flowing through that gateway, we get metrics out of it.

25:49so we can see all the tool calls and all the events happening coming from the agents to the MCP servers and back and forth. So you can kind of see what's happening, especially if maybe if there's something unexpected, like, well, this one agent made 200 ,000 tool calls to delete repo. Like, oh, maybe we should alert on that. That's probably not good. So another visibility or metric to see what agents are doing and what tool calls are doing as well. So I guess hopefully that made sense because that's kind of how I understand it. Yeah, that was great. And there were a couple of things that you mentioned in there, I think are additionally worth digging into.

26:27And you kind of have this, if I'm understanding right, in part of the build pack that you're working on, part of that centers around kind of pushing an agents.md file. You mentioned like people have these MD files hanging around. You mentioned memory, you mentioned identity. I'm wondering if you could kind of help us understand, like, if I'm working on my laptop, right, and I'm using a certain agent harness, I may have an MD file that is kind of generic. And then I have files that are created by the agent that it uses that are only corresponding to my single agent. and then I have memory and then calls into other things.