996: TrueFoundry’s Nikunj Bajaj on How to Get $100M Returns on AI Agent Deployments

0:00Jon Krohn:Imagine being able to deploy an AI agent and getting a return of over$100 million from that single deployment. My guest today has done that multiple times. Welcome to another episode of the Super Data Science Podcast. I'm your host, Jon Krohn. My guest today is Nikunj Bajaj, CEO and co-founder of True Foundry, a Bay Area-based startup that has raised over$20 million to solve the thorniest problems that enterprises that organizations in general face when deploying agents. And he's done it with great success, as I said at the outset of this episode. He's done this for some of the most demanding organizations in the world, including NVIDIA and Siemens.

0:43Jon Krohn:What an incredible guest. What a topic. I'm sure you're going to enjoy it.

0:51Jon Krohn:Nikunj, welcome to the Super Data Science Podcast. How are you doing today? Very good. Thank you so much for having me here, John. Yeah, of course. We're in person in San Francisco in a beautiful, sunny week in San Francisco. And here we are indoors. How does life get better than that? Nice. It's so great to have you on the show. Congratulations on all your recent success. It was really cool doing some research for your episode. Tell us about what True Foundry does and how you've had the success recently, all the capital you've been raising, all the great clients you've been landing. Absolutely.

1:21I think we owe all our success to agents. So over the last few years, most of our enterprise customers have gone from building agents that are like, you know, quick prototypes to now things are actually running in production. And as these agents start running in production, enterprises realized that they need a control plane to manage not only these agents, but all the underlying components of these agents. And what we have been building at True Foundry is this control plane layer that sits between the application layer and the infrastructure and model layer so that we have one place where you can connect, observe, and govern.

2:01I like to call it, we are a cog in a wheel, but in a very unique way.

2:06Jon Krohn:Tell us a bit about how it actually works. because I understand that you have a trillion tokens a day or something like that being processed, which is insane for your clients. Big number. It's hard for me to kind of wrap my head around that. It's a big number. Yeah. I don't know how many tokens I hit when I run out of, when my Claude code is like, you're out of credits, but I know it's a lot less than a trillion tokens. So yeah, so trillion tokens, tons of usage, obviously big clients. I think some of them I can name. I know there are some big ones I can't name, but I know NVIDIA is one that I can say.

2:37Jon Krohn:For sure. Siemens is one I can say. And secretly, there's a bunch of other logos that you would definitely recognize, listeners, that we can't say on air. And I would love to understand, you kind of described the functionality of what True Foundry does, but what is it like as a user to use it? If our listener here is, okay, great, I want AI orchestration or security or governance, what do they do? Is it open source or do they go to your website? What's the flow like? Very simple. You literally go to our website, truefoundry.com, hit sign up, connect your models, start using it. Right. So let me actually explain how this works from a, like, where do we fit in into a user's journey as well?

3:20Exactly. Imagine a user that started to build out an agent. To build that agent, they need three most important things. Number one, they need access to models. And now instead of directly hitting the models, they're going to hit this through a model gateway layer. they're going to interact with the environment, right? And this environment could be tools, functionalities, skills, actions you may want to take, data you may want to tap into. We call this goes through an environment gateway layer. And in the industry, this is more popular with the term MCP gateway layer. Okay. But we think that environment interaction is a little bit more generalized than just accessing MCP servers, basically.

4:03Jon Krohn:Right. So the MCP servers is for tools alone. But with this environment gateway, it's kind of tools and governance and all the other belts and muscles you offer. Tools, skills, any kind of data that you may want to tap into, all of those things. And we think that these are modus operandi today. They could also change in the future. So that's why we think of this systems-wise as an environment interaction, basically. And then the third thing, an agent will most likely need to connect with other agents or sub-agents. And this happens through an agent gateviller. So as a user is building out an agent, then interacting with these components, you go through a gateway layer.

4:39And then once all of your traffic starts flowing through a gateway layer, you start also recognizing the fact that now everything is observed. Your developer does not have to instrument their code at all. Everything is governed. It's like entering an office building, right? You're entering through a gate. You know who is entering. When do they enter? If you want someone to be blocked from entering, you could do all that, right? That's what we're doing for AI.

5:02Jon Krohn:I see. So it's, yeah, if you kind of anthropomorphize it, what you're doing with True Foundry is creating a system where you can tell what agent is kind of clocking in for their workday, when they clock in, when they clock out, what tools they have access to, who else they can speak to in the office space. I like that. I understand, yeah, what data they can access and, you know, some kind of directive for what they're supposed to be doing. Exactly. That's cool. And it sounds actually, the way you're describing it, it kind of sounds like it's click and point, even though it's intended for engineers, developers to be using, right?

5:36Yeah. So yes, we have like two sets of users who are directly using it, and one major set of persona that's actually managing it. So for sure, we have engineers who are building out agents. We call it pro-code agents when engineers are building it. They can write their code, but within their code, the API layer, they can invoke the models, MCPs, and other agents through our gateway. Right. But we also have a UI layer, a playground layer where, let's say, a person, a user who is not coding on a day to day basis, like a builder.

6:06Jon Krohn:Yeah. Or an IT manager. Yeah. They can actually simply prompt it. And it's still everything goes through the gateway. They can build an agent through texting and that agent gets published back to our agent registry. So the organization can keep scaling it. And then we have one more persona. We call it the platform engineering team. They are the ones who are in charge of managing the governance and the observability layer within the enterprise. and they are the ones who are going to buy our platform, set up our platform for internal usage. Nice. I like that. I'm starting to get a really good picture of how all this works.

6:36Jon Krohn:In our research, we came across that there are three specific components to this. So you've talked about the AI gateway, but it sounds like that can be broken down into these three components, model gateway, MCP gateway, and agent gateway. Do you want to tell us about that? Yes, for sure. Yes, that's exactly right. So first of all, just a quick industry picture. Like, you know, sometimes in the industry, people refer to AI gateway as a very specific term, which is basically just a routing layer across different LLMs. Like Open Router. Like Open Router, right? So to me, that's an LLM proxy layer, a very crucial point in the overall stack of AI, but still like one slice of the solution, right?

7:19We generalize the term AI gateway to truly include AI and all the underlying components of AI. which means the models, the MCPs, the agents. Those are the three major components that we talk about. And we did mention how as an enterprise or as a user, you're never sharing then the direct model API keys to a developer, everything. You only give them access to the gateway. And the gateway knows which user has access to which models. Same thing, gateway knows which users have access to which tools, which servers, and which agents have access to which other agents, right? So those three things are going through the gateway layer.

7:53And then across all of these three, you get end-to-end observability. So imagine this, right? Your agent ended up invoking a model that invoked an MCP server, implemented a certain guardrail that, do not send out my PII data to this particular model. Or if there was a prompt injection attack in the prompt, then just block that request altogether. So you can set up all those guardrails. And then you have a complete trace of what happened. And then as an enterprise, if you cared about putting certain kind of guardrails, So there's a kind of cost limits, FinOps layer, rate limiting, all of those things you're able to do through the gateway layer, basically.

8:29So that makes it a very powerful interface to build, govern your agents, basically.

8:34Jon Krohn:You guys have been working on this for years, and it sounds like there's a lot of functionality in there. It is interesting how this, it seems like you, I guess, were a visionary around building a platform that all of a sudden then about a year ago with where Agenta capabilities landed, all of a sudden, all this functionality, things like this gateway you're describing, all of a sudden became essential. Well, everybody likes to think that they are the visionaries. The reality is that we have been working in the space closely with our customers who have been building these agentic AI applications.

9:12And when you care about working with your customer and you're hearing their problems straight, and then you're ready to act on it fast, that's when you start looking like a visionary, but truly like they were the ones who were doing everything. So the story is that we were working, helping our customers, like, you know, build, deploy their agents on our platform from 2023, 2024 timeframe. And we noticed the fact that in the beginning, all of our customers, enterprise customers, actually thought of this AI gateway layer as indeed a thin proxy layer. And they thought they're going to build it in-house and everybody built it.

9:50Okay. As they started putting their agents to production, they realized, oops, it's more than a thin proxy layer. I see. Because now I need to build out this observability layer, this governance layer, this policy engine layer. And now it's not just one team within the enterprise that's working with these models and agents. So many different teams are doing it. All of them have their own cost centers. And they could also see that the environment or the ecosystem outside the enterprise was also exploding. Like MCPs are coming out, A2A protocol is coming out, new agent building tools like Claude is coming out.

10:26All of these tools have access to all of your internal tools and stuff. So they realized that the complexity grew very, very fast. And as we are seeing that our customers are struggling to solve this problem, we continue to expand the capability of our platform in conjunction with them. And here you go, like works well for us.

10:46Jon Krohn:As the person developing these specific solutions at Truth Foundry, you end up being in a position where you're seeing pain points across multiple different clients in different sectors. And so it gives you an advantage over them. When they see something, they're like, oh yeah, this is something quick I can build. But they're only experiencing their particular, the pain points that they've had so far. Whereas you're seeing this much broader range of problems. And so it allows you to make more informed, opinionated decisions about how the product should be designed and how it should work, which gives you the edge.

11:21Absolutely. And you get the best in class from everywhere, right? This one customer who may have perfect idea of how to solve a certain problem, but they are taking potentially some suboptimal routes to solve a certain other problem. But because we have seen a similar problem with another customer or we have experienced it ourselves, we are able to bring in the best in class everywhere. And actually, this is where one of the things that I like to believe is startups at a certain stage have the biggest advantage of this phenomena of getting feedback loop and closing it fast. If you're too small, you actually do not have enough customers to get meaningful data points out of.

11:59If you're too big, then people who are making the product decisions are not the ones who are closely interacting with all these customers on a day in day out basis, basically. The moment you merge the two, that's when I think magic unleashes.

12:11Jon Krohn:Nice. Well, congrats. You're seemingly nailing it right on the head because people are loving your product and flocking in. All right. So we were talking about those three kinds of gateways and then I think it took you off on a tangent. We had gotten through the model gateway aspect of it. Tell me about the MCP gateway. So I should also give a little bit of context. Probably most of our listeners know what MCP is, model context protocol, but just a quick primer in case you don't. It's a very popular protocol, similar to the way that HTTP is a protocol for the web. MCP is a protocol for how agents can access tools.

12:45Jon Krohn:And so there are thousands, conservatively, of what are called MCP servers out there. And each MCP server has a number, has one or more, sometimes hundreds or thousands of tools in a given MCP server. So if you think about a Google Maps MCP server, it allows you to put in a location and you can get the latitude and longitude out. That's one tool. You could find directions between two points. So all the kinds of things that you can do in the Google Maps app, a agent can do using the Google Maps MCP server. So that MCP server has access to all these different tools. And what that means is that thousands and thousands of these MCP servers exist.

13:28Jon Krohn:And across those thousands of MCP servers, there are millions, conservatively, of tools that agents could access. So anyway, a bit of context on MCP. So what does the MCP gateway at True Foundry allow you to do? Right. Well, first of all, thank you so much for that beautiful explanation of the MCP servers. I might steal that for some of our customers who are starting to get accustomed with these terminologies. The concept of MCP gateway fundamentally is very similar to the model gateway. Let me actually just explain a little bit more about the model gateway, the problems that it is trying to solve, and then overlay that concept on the MCP server bit as well.

14:05Right. So first of all, when you're working with a model gateway layer, you stop circulating the API keys of the different models to the end users, right? Which means now you have a single API key with which you're able to access multiple different models. As an organization, you can put some rate limits on it that you can only invoke this model 100 times in an hour, right? You can put budget limits on it that in a day, I do not want any dev environment to spend more than$2 ,000, right? You can start doing all that control layers and you get observability on the model side for free. You could also do interesting things around model fallbacks, for example.

14:43What if an OpenAI model in the US West region was down or slow? You can start routing your traffic to other regions, basically, right? So you can do all of these controls once your traffic flows through the gateway. Something similar happens in the MCP layer, except the authentication in the case of a model is just an API key. But in the case of an MCP, think about it, right? Like, you know, you're dealing with a lot of other private data. Like if it's a company Slack account, Gmail account, whatever it is, it has a lot of sensitive data.

15:15Jon Krohn:Yeah, and not only just reading it, but writing to it as well. Yes, right. So now you really need to, and each of these different tools may implement its own authentication authorization mechanism as well. So the problem becomes more complex in the world of MCP. You still want to figure out the authentication problem that who is the user or the agent or any virtual entity trying to access certain MCPs. But you also have to solve for the fact that what is the type of authentication mechanism that MCP server is implementing. And then you have to create a mapping between the two, right? But to this end, I think the point that you mentioned is not just reading, but also writing.

15:51and it's the agents autonomously writing code and like invoking these actions basically. So then you also want to have a sense of control that maybe there's an agent which could potentially be controlled by a consumer outside of my organization. There could be prompt injection attacks in there. So by definition, I actually only want to give it read-only permissions to like seven different MCP servers. Okay. So then you start building out these user interface layers that I'm going to put together these seven MCP servers, but only whitelist the read-only tools from these MCP server, create some virtual entity and expose to this agent.

16:30So now this agent has read-only version of a bunch of my MCP servers. And then of course, you want to build out tracking, you want to build out similar kind of resilience, tool selection, all that complexity or harness, as people are starting to call it now, starts to get baked into the MCP layer of it as well.

16:48Jon Krohn:On this podcast, I'm always going on about how Claude Code is mind-blowing, but now Claude Co-Work is making my jaw drop as well. For example, I recently wanted to quantify how healthy my sales pipeline is for my AI consulting business. I simply asked Claude to estimate my sales for the coming quarter, and it brought info from relevant Google Sheets and my Gmail to create a professional spreadsheet of clients with estimated revenue for each one. Whoa, this might have taken me a day. Instead, it was done flawlessly with Claude Co-Work in minutes. Claude is the AI for minds that don't stop at good enough.

17:19Jon Krohn:It's the collaborator that actually understands your entire workflow and thinks with you. Whether you're debugging code at midnight or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. Ah, and you'll appreciate that I can ask Cowork to show me data such as my sales spreadsheet, and it provides an interactive chart right in the conversation. For problems worth solving, get started with Claude at Claude.ai slash superdata. That's Claude.ai slash superdata. And check out Claude Pro, which includes access to all of the features mentioned in today's episode, claw.ai slash superdata.

17:53Jon Krohn:Really cool. Thanks for that, Tour. I suppose that this tool gateway would also probably help with tool sprawl, which I think is an issue that a lot of organizations face. You can end up in a situation where, because I mentioned it's so easy to hook up via MCP servers so many different tools, you can end up with, you know, too many tools being provided to given agents. And that can lead to problems. It can dilute the specificity of what an agent is supposed to be doing. And it can also potentially be providing read-write access in situations where we might not want it. That's right. Yes. So one of the things that happens with the tool sprawl is, first of all, fundamentally, tool sprawl doesn't break anything, right?

18:40You could have technically a million tools out there and you are individually managing them.

18:45Jon Krohn:You could have one agent and you're like, here's a million tools. Figure it out. So there's nothing theoretical about it. It's about the user experience, the developer experience that you provide of how easy it is to manage these agents, who has access to what, and when something goes wrong, how easy it is to debug it basically. That's where I think the productization of this MCP servers and controlling this tool sprawl comes in handy. One of the examples that I gave you about this virtual read-only MCP server entity that I talked about earlier, right? That's an example of, you know, that this agent has to operate in a certain scope.

19:23In that scope, they may need access to these 10 tools and you define that to the agent, right? Now, there are also other ways where you could, you do not know ahead of time, what are the tools that I want to give access, right? That's where you start creating certain kind of logical entities to manage these tools and you basically create what we call as a description that if the agent knows that this is the segment of tools that I really care about, they can read through that description and then in runtime, select these tools. So automated tool selection is the other problem that we solve for where the complexity goes on now from end users to the models themselves, basically.

20:02Jon Krohn:Nice, great explanation. So I think we've now covered two of the three kind of component gateways. We've covered the model gateway, we've covered the MCP gateway, that leaves us with the agent gateway. How is that different? That's the hard part. Okay, okay. The agent gateway is the hard part. So, well, first of all, remember when I started saying that an agent needs to talk to models, the environment, and other agents to get things done, right? Now, what's happening right now in the industry is that these agents are taking actions. And people are building out agents because they make them look more productive.

20:42and they're doing things that we sometimes do not want to solve for, right? And we keep giving access to these agents to a bunch of tools, a bunch of databases, a bunch of actions that they're able to take. What if something goes wrong with the agent? Most of the industry today does not have a kill switch, right? They may be, in the best case scenario, they may be able to observe what my agent is doing, and that too in the best case scenario, but they do not have a way of stopping it. So to solve for that problem, you want to put your agents behind a gateway. But if something goes wrong, you can manually or automatically kill the agent.