Baseten’s CEO on why every company will want to own its AI

0:00Alex Heath:Tuhin, for people who don't know what AI inference is beyond it's how you serve the models, I'm really curious to hear you explain it because it feels like the most exciting, frenetic part of the AI stack right now. And you're thick in the middle of it with Base 10. And I'd love to know what it means for people who know the term, but they don't know much else. Yeah, absolutely. So what is inference? So inference is the way you serve models. Quite often, customers will come to us. They'll say, hey, we have this model. We want to use this open weights model. We have post-trained this model. We need to figure out how to actually use it in our applications.

0:42And so when you work with a closed frontier model provider, you kind of get this for free. You know, they have the model, but they vertically own everything from the model all the way to the chips it runs on all the way up to the APIs that you are using to call that model. When it comes to post-trained models or your own models or open-weight models, you don't get that for free. And so you'll come to an inference provider and you'll say, hey, I need to run this in production. Here is some scale, approximate scale I need to run this at. And then you'll come to company base 10. now there is so much that goes into actually running a model um and you know for us like that the infrastructure level problems so that is like hey where is the actual compute going to come from to run this model um so we take care of that um there is the orchestration piece which is hey like how do i um get this user request to the right chip with the right model on it we solved that problem and then there's like the chip level details where it's like how can i make this model run very, very fast.

1:47And so really, when it comes to inference, you're really like, you're not just looking for an API. You're not just looking for GPUs. You're looking for like a performance, reliable system that puts it all together and puts it behind an API. And Baystone kind of takes care of all that. All right, that was kind of marketing. So let's double click on that. Let's actually double click on what does that actually mean. So Baystone's like three different layers of problems that we're solving for our customers. So the first one is this idea of infrastructure and compute. So you need chips to run these models on.

2:19So for our customers, if you're a customer running this model at decent scale, you'll probably need thousands of chips to run this. If you go to the market right now and say, I need 1 ,000 B200s or GP300s, good luck. It's not happening. As they're gone, someone's already taken them. And so you'll come to like, what we do is that we kind of take care of this compute procurement from the standpoint of what you need or that's necessary to run inference on for our customers. We do this using like a pretty complex architecture where we're able to pull together compute from all sorts of different regions of cloud.

2:57So we sit on top of 20 different clouds in 90 different regions, which is completely abstracted away from our customers. So that's the bottom heterogeneous compute layer. I didn't even know there were 20 different clouds. Yeah, exactly. Well, you call them clouds, neoclouds, data centers, hyperscalers. You call them to us. They are sources of compute to some extent. Yeah. Okay, let's go up one layer. That becomes the core inference layer, and that's kind of what I described, which is like, hey, I care about running these models really, really fast and really, really reliably. And so that's kind of like what Base 10 has done for a long time now.

3:30That is what we are known for. And our customers come to us not only saying, hey, we want to compute, but hey, we need the whole software stack on top of that, on top of this, so we can run this so it doesn't go down. We can run this so it's very fast. We can run this with the right numerics and so on and so forth. That's the second layer. We can talk as much as you want about that. The third layer is around all the primitives that come on top of that. And that's kind of like, these are inference adjacent primitives a lot of the time. What I mean by that, these are things that either make your inference more powerful, They give you different paths and different types of inference.

4:04They make your models more powerful, but they're all related to inference in some way. So that might be something like post-training models and RL on top of those models, which not only helps you train more models, but has a bunch of inference baked into it. It might be something like sandboxes, which is how do you actually execute code. We're going to get into that.

4:22Alex Heath:Yeah. How do you actually execute code generated by these models? It might be stuff like evals and routing. when people talk about inference, they're kind of talking about all these problems together, all these problems together. But there are lots of different parts to it. It's everything except training, right? It's serving AI in production. It's when you get a chat GPT response back or you use Muse or GrokBot or any of these things, they're all inference. And inference is growing like crazy, it seems like as a market. but it also feels like it's hard to make sense of it. There's a lot of players.

4:58Alex Heath:They're all approaching it in slightly different ways. And I'd love to better understand how Base 10 differs in its approach from not only your direct competitors and inference, but the traditional clouds. Because I think someone listening or watching this who is maybe not as in the weeds as we are would go, well, why would someone not just use AWS or Azure or Google Cloud to run this stuff because they have AI? Why do we need these AI clouds doing inference? It's a good question. Look, I think the most important thing with all this is that the market is just ginormous. And this is important for a reason.

5:37But if you kind of project forward four or five years and there's some silly number associated with how much AI spend is. Let's call it$5 trillion is what they say in five years from now. It doesn't matter what that number is. It's somewhere between$2 and$10 trillion. Let's say AI runs about like a 50 % gross margin. So that means there's about, you know, $1 to$5 trillion of inference spent in the market. And, you know, if you think about how that's going to be divvied up between hyperscalers, between closed frontier models, between inference providers like Base 10, So any which way you cut it, that's somewhere between$10 to$50 trillion of market cap that's going to be divided between the people who are capturing inference.

6:26If you then start to break down how different folks are approaching inference, everyone has a slightly different take, I'd say. So there's the hyperscalers who have historically been really good at infrastructure-like things, but they've also been really good partners. So I think the hyperscalers will have inference solutions and they do today, but they're also really good partners to the inference companies themselves. We have really strong relationships with all the folks you mentioned there. The reason why you'd use us over hyperscale, really a lot of it just comes down to building specialized software.

7:02And it's like, hey, we do one thing, they do many things. And so when you're in a market that is moving as fast as this market, where you need to run stuff in production and when your inference provider is down, your product is down, you would generally tend to who will move as fast as you in the most customer aligned way and give you the most attention. And that is base 10 today just because we do fewer things and allows us to be best in class at those few focused things that we really do. Yeah. And I think it goes beyond that because I think there's just so many layers of this problem that I described.

7:39And we're kind of solving it layer by layer there. Whereas I think a lot of other folks and like, you know, whether it's hyperscalers, whoever else are like, kind of like, hey, I've got some basic inference solution that works off the shelf, but isn't really dialed in to serving users in production at scale.

8:08Alex Heath:to Granola, the AI notepad for people in back-to-back meetings. It works everywhere you do and lets you focus on what matters. Try it at granola.ai slash sources and use the code sources for three months off. This episode is also brought to you by Jira Byadlassian, where teams and agents get the context, coordination, and control to move work forward. Try it free at jira.com. That's J-I-R-A.com. The criticism maybe of the market you're in is that these inference clouds, they're just reselling GPU capacity at a very thin margin, and they really have to prove the software layer still. And you just said the software is what matters, and you have an acquisition that you just announced that I want to talk about today on that.

8:51Alex Heath:But can you explain that a little more, why the software matters so much? Because as you said, the chips are hard to get. And if you've got chips, you see the base 10 ads in San Francisco and the billboards, and it's very clear. It's like you're saying, I've got compute. Then that seems valuable. But at the same time, you're saying, well, the software is actually what matters. Yeah, look, compute's necessary, but the software is what differentiates, right? So it's like, I don't want to downplay the role of compute, or compute's obviously the market constraining thing right now. At the same time, there is so much value to be driven, to be built on top of that.

9:24The way we think about this is if you wind back a few months from now, there's been this really big push towards owned intelligence. And what is the idea of owned intelligence? The idea of owned intelligence is like, hey, because of cost control, because of data sovereignty, because of, you know, credible permanence, which is the idea that you don't want models just to disappear in one day, is very, very important for every company in the world, to some extent, to own their own intelligence. Cool. What does that mean? Well, what that means is that they need to have the capability to not only run open source models, but an open way models, um, in addition to close frontier models, but to figure out like when to use, which, how to use their data and the, and the user feedback, um, from their customers to improve these models.

10:12What customers have realized is like, like inference is one part of that. And I would argue it's probably like the backbone of that to some extent, but there's all these things that you need to do to unlock that continual learning loop that makes customers kind of internalize their intelligence. So that might be inference. That might be how do you eval each of these models as they come out. It might be how do you post-train models with RO environments to make these models better. It might be like, hey, the execution environments for when these models are producing code and doing their own things for them to do this work.

10:48And then eventually it's like, how do you get a better model out of it all that you run it again and kick off this whole cycle again. That whole thing I just described to you is what we call a continual learning loop to some extent. And that is what the software layer that we are... That's the software stack. The software stack we are building is all those things I just described to you and all of them in the entirety, which in a lot of ways become their own hyperscaler or new cloud when we build it. And compute is necessary underneath that until you have that thing, until you have that loop working, the compute's kind of useless.

11:23Or it is exactly what you said. It is reselling of, you know, taking access to it and putting it somewhere else. But the minute that we build the software layer for that continual learning loop, what we are doing is like providing every enterprise, every customer in the world, some credible advantage they are getting from running their intelligence. And what that does for them is it protects their margins. And as a result of them protecting their margins, they will interpaste a lot of that margin to us because we are kind of making them independent again to some extent.

11:56Alex Heath:Yeah, that's interesting. I haven't heard continual learning framed in that way. I mean, when you spend time in the labs, you know, the frontier labs, they're talking about continual learning in the sense of the models themselves learning and getting better constantly and the research loop closing and being aided by AI. But you're talking about it in like a sovereign company AI sense of controlling your own destiny. which I hadn't heard before. Yeah, well, I mean, it's exactly that, right? This idea of owned intelligence, like Satya has talked a lot about this, Jensen's talked a lot about this, which is like, hey, your user data, your signal from these models, the improvements to these models themselves will become the core IP of every company in the world.

12:40Every enterprise, the extent to which you thrive in an AI first world is the extent to which you own all those things. in order to own those things, you need to own that loop that I just described to you. And that is what we are building for. That is like, you don't want all that data to be the input to someone else's continual line loop, which is the loop you described about how the model provider companies get better at training their models. And that to me is the software stock. That is the differentiated value that you're providing. That is when this gets a lot more interesting than just bare metal compute.

13:19Alex Heath:Does it feel to you like it does to me where in the last like eight to 10 months, it just really feels like AI has started becoming useful in a way that it was not before? I mean, I think that, you know, post-ChatGPT, obviously like that product took off and kind of kickstarted this whole boomerang, but it was this kind of like better chat bot, Google search thing for a while. And I mean, even in my own work, both, you know, as a podcaster, but investor now too, its utility is incredible. And I'm curious if you've been feeling that as well because you started Base 10 in what, 2019, which was well ahead of all of this.

13:58Alex Heath:The models were not capable at all. No company would be running LLMs in production even in like 2021, 2022. So yeah, I wanna go to the founding story and how you saw this when you did, but also do you feel that what I'm describing about the shift in AI right now? I think there was something in December that happened where these models just kind of had a capability shift. I think it's happening again right now, like in the last month where there's like another capability shift. But the last like six, seven months have been insane, right? Because so much has happened. Like obviously it felt like coding models finally got really good.

14:29The compute shortage happened. Open weight models started getting very, very good. And like, you know, it's like release after release after release. It's just like, oh, there's another one. There's another one. There's another one. And I think what has happened as a result of this is that one, people are paying attention to intelligence. So intelligence, people using intelligence and enterprising using intelligence has skyrocketed. As a result, earnings calls have become about using AI. Copies are being judged by how much they're spending on AI. That in turn is like making them rethink, hey, what is our long-term cost strategy and how do we, one, get AI in more places and two, how do we do it in some way that doesn't blow up our companies?

15:14And I think that is just taking over the entire narrative, but also just pushed adoption even further.

15:19Alex Heath:It's wild. It's wild. And I can't imagine being inside it the way you are. I mean, before we started this, you were saying, I may need to step away because I'll get paged because one of our customers needs help because everyone's growing like crazy. Before we get into more stuff, go back with me to 2019. You're starting this company with your co-founders. You were in banking before, is that right? How did you see this? because this market didn't exist then. Yeah, my background is that I grew up in Australia. I moved here for uni. Graduated uni in 2009. I did what everyone does in 2009. It's like, oh, you go work in banking.

15:58That's right. So I moved to New York. I worked in banking. I worked in infrastructure finance. Basically doing project financing for infrastructure projects like toll roads and bridges and airports, which is interesting. They're all data center people now, funnily enough. After that, I started working in technology. I ended up working at a lab in Boston, working on using machine learning to predict and diagnose neuromuscular disease. That was in 2012. From 2012 to 2019, machine learning was very, very early. It's kind of more classical models than large language models. If you go think about what machine learning was being used for 15 years ago, it was stuff that like paypal was doing and stuff that facebook was doing and a lot of it was like fraud classification it was recommendations and like we were kind of working in that paradigm for a long time i started a bunch of companies didn't really go anywhere 2019 um started another company and we were really thinking at the time like two things one is like you know my two co-founders of phil and amir um and then soon after that punkage um but phil and amir you know they're my best mates and so really the idea was like how do i how do i start a company with my friends that's number one number two number number two which was just as important was this idea that in 2019 we kind of knew that machine learning was going to be a big deal like opening ad had been around for a few years at that point um they were doing stuff like you know smart people were going to work there um but it was still very early like there's no there's no commercial proof what to happen but we're like look machine learning looks like it's gonna be pretty pretty big we don't know what that's going to look like.

17:37We had no idea. We're just like, oh. That's why I'm so in awe of all of our customers because I think application layer work is so challenging. Actually understanding that end user problem in a domain requires so much product expertise and domain expertise that frankly, we just didn't have in that way. But what we knew was a mission. We knew a bunch of our machine learning. We thought it was going to be a big deal. We're like, oh, well, if we believe in this and we know a bit, let's go to build a picks and shovels business alongside to support machine learning. And, you know, as long as the market is big enough, we'll have a shot of building something big.

18:12I think that what happened in 2022 was the market just accelerated in a way that none of us expected. And then like quarter after quarter, it just feels like, you know, we're like, surely this is it, right? And then, you know, and then it takes another turn. I was chatting with some folks that, you know, you work with, Guy and Effie and Jill about this Monday. And basically this idea that even in 2022, when they started thinking about machine learning and AI, it felt really early. It didn't feel like it was obvious then either. And it just feels like quarter after quarter, it's like it just come online in a way that it's kind of gone beyond all our wildest dreams.

18:51And so in 2019, when we started the company, we were definitely building an infrastructure company. I think that infrastructure company had many different facets. One facet of it, which seemed to us at the time was the least important facet, was being able to serve models. And then that just became the entire company.

19:11Alex Heath:And you raised at$5 billion valuation in January of 2026 and then$13 billion valuation in June. So in six months, you almost tripled your valuation. What changed in those months? Is it what we've been talking about? Yeah, it's all those things, right? Which is like, I think in the last like six months, what has changed is that we've gone from thinking our open weight model is going to be a thing is like to, oh, wow, this might be a majority of token volume. Do you think open will be that? I think open and custom and owned intelligence will become, I think the future is mixed intelligence where you're using close frontier, using open-witch models, using post-trained models.

19:56But I think custom frontier models will probably be some version, some percentage of the models that are uniquely served to needing the most powerful models in the world at all times. And you think about right now, think about what AI could solve three months ago versus what AI can solve now. And AI could solve a lot three months ago or two months ago even. If you can run that two-month-old intelligence for 90 % of your tasks at a fraction of the cost and own all that data sovereignty and ownership narrative that I talked about earlier, like that's a no-brainer and it's very rational.

20:31Alex Heath:How are you feeling about the state of open source though? Because the US is so behind, China is still dominating. I think a lot of people, a lot of tech leaders are worried about what's happening. And a bunch of CEOs were recently advocating for the government here in the US to protect open and open weights and open source. But yeah, curious what you make of what's going on right now. Yeah, look, I think it's both a very exciting time and an interesting and challenging situation, right? So look, we want more intelligence everywhere. And I think, you know, open weight models are fundamentally a very important part of the ecosystem.

21:09Like, you know, in absence of open weight models, you know, all that ownership narrative that I just told you about is gone. Like you need it. They're a necessary precondition. So we understand their importance to the independence and sovereignty of American enterprise. So that's why I'll say one. Two, so I feel really good that there are options now. So that's amazing. On where they come from, this seems temporary to me. I think there will be American open-way models. I hope from what we can tell, there'll be really good American open-way models in weeks, not even months or quarters. Soon enough, there'll be really good American open-way models.

21:44And I think, you know, once that flywheel kicks off of like, hey, we have stuff that is competitive with Chinese open-weight models, I think a lot of the narrative that you're seeing right now, which I think is a little bit more fear-driven and like, are we falling behind -driven than there's capability-driven, if that makes sense, I think that will kind of wash away as well. I'll give you a good example of this where all the open-wind stuff really kicked off 18 months ago with DeepSeek, when DeepSeek V3 came out, which was a Phanteks model came out over Christmas of 24. The first, you know, we were obviously serving that to a bunch of customers, but the first thing I did was I was like, hey, okay, we've met so many enterprises over the last six or seven years.

22:26I was like, let me go and talk to 10 of them right now and see how they feel about them. And the reality was that I'd say 90 % of them, if not more, were like, look we would never use this as a production this is you know like like we don't know where these are comfortable we don't really understand how this works like you know what are they being trained on um yada yada yada and and i'd say they were like deep tv 3 was interesting because i think it was like the first time you had something which approached the frontier but was still behind you know it's still like three six behind and like it wasn't clear how you'd run them and there's all this like uncertainty around uh was most of like and even though it's behind it was such a big

23:01Alex Heath:moment everyone was freaking out the air race has been reset yada yada yeah 100 and it was like it was like the paper that they trained it on like a fraction of the cost and blah blah blah fast forward a year now you have all sorts of models you know you have like you you even have some really good american models like inkling but then you have like um you have glm you have kimmy you have deep seek and you know hopefully a nematron the nematron family's getting really good from nvidia as well um all that narrative from 18 months ago is completely evaporated every enterprise we talk to now is very excited to run open source, open way models.

23:34And I think even, even like debating where they're coming from isn't really relevant. I think they're just like, this is just the new world. And turned out that like fear and certainty and doubt with more just capability driven as opposed to actual, what does this mean driven? So the minute they got good enough, everyone's like, can we use them? And, and like, you know, the infrastructure needs to exist. Open way models need to exist to, to run them. But like, we're in a really good spot right now in terms of owning that entire stack that's necessary to be able to run these models of production.

24:04I think as that stack matures, and obviously we are to some extent building that stack, we are building that stack until that stack matures. And as that stack matures, I think the barrier to adoption of open-way models is going to go down as well.

24:19Alex Heath:Does open have to keep growing for base 10 to keep growing? Are you really kind of tied at the hip with this? I think that's a good question. Like the intellectually honest answer is yes. You know, there's probably people who don't want me to say that, but the intellectually honest answer is yes, which is like we, you know, we believe in the open-weight ecosystem and we want to give back to it and push it as well. Like, you know, we launched something called Base Labs a few weeks ago, which is our research lab, which is like, hey, how can we make it? You know, we acquired a research team at the end of last year.