1. Podcasts
  2. The Generalist
  3. Why inference is harder than you think
The Generalist

17 Nov 2026 · 44m · 18 chapters

Episode artwork

Why inference is harder than you think

In short

The episode argues that “inference” (running AI models for user queries) is the hardest, most misunderstood bottleneck in the AI stack, and may become the dominant market as AI spreads. It explains why hyperscalers haven’t solved it (new workload, fast-changing models vs long reservation cycles, and missing cross-service primitives), and why specialized infrastructure is needed to serve models quickly, reliably, and at scale under compute shortages.

Guest

Tuhin Suravastava, co-founder and CEO of Base10 (BaseTan/Base10 mentioned). Background: previously worked in finance; did ML research in Boston tied to neuromuscular disease; co-founded earlier startups; later invested/learned infrastructure/project-finance concepts (Macquarie mentioned). Base10 serves training/inference for companies including Abridge, Cursor, and OpenEvidence across 20 clouds; raised a $1.5B round valuing it around $13B.

Key claims/examples

Inference is “critical path” and “final market” (tens of trillions). GPUs/chips fail; capacity is scarce (Base10 targets mid-90s utilization). Customers start with closed models, then route to open-weight models (e.g., DeepSeek V3; “GLM 5.3” cited as frontier-capable), then add dedicated deployments and post-training via “Loops,” eventually training routers and building continual-learning loops. Base10 uses heterogeneous compute across 20 clouds/90 regions to provide reliability, data residency, and lower latency.

Written by AI. May contain mistakes.

Chapters

2:05

Base10's Background and Vision

Discussion on Base10's mission and the early challenges they faced.

“Tuin, maybe just to start, we can start super simply.”
4:05

Pivoting Towards Inference

Tuhin describes the strategic pivot towards focusing on inference.

“I think we'd always built serving what we now call inference as part of that thing.”
6:35

Growth and Market Realities

Insight into the unexpected growth and market dynamics of AI and inference.

“When you say completely off, are you off by like an order of magnitude, one, like a two X, three X, like, do you get better at forecasting that or do you get surprised each time a little bit?”
9:45

Cultural Reflections: Australia vs. America

Tuhin shares cultural insights from growing up in Australia and living in America.

“Like, where did you see that in yourself?”
12:00

The Case for Inference in AI

Exploration of why inference remains an underserved area in AI and its complexities.

14:00

Understanding Inference Challenges

Explore the complexities and challenges hyperscalers face in solving inference problems.

“Negotiating around these commitments can take months.”
17:47

Market Dynamics and AI Spend

Discuss the projected growth of AI spend and its implications for inference markets.

“So let's assume these application layers, AI application spend runs that 50 % gross margin.”
20:36

Customer Journey in AI Model Adoption

Learn about the customer journey from using closed models to adopting open-weight models.

“is the demand for specialized models and yeah, the different ways that people are using that.”
26:35

Future of AI Model Diversity

Anticipate the proliferation of specialized AI models across various companies.

“the sort of, you know, your own owned model piece, the, you know, customized model.”
28:00

Open Models and Portfolio Approaches

Discussion on the future of AI companies and model strategies.

“Instead, I think everyone starts to think about things as a portfolio of models.”
Show all 18 chapters
29:15

Heterogeneous Compute and Inference Layers

Exploration of Base10's approach to computing and inference layers.

“And, you know, those benefits for our customers are reliability and the fact that, hey, if one region is down or one cluster is down, we can always route you elsewhere.”
31:05

Capacity Challenges in Compute Supply

Insights on the tight supply of compute resources and its implications.

“I mean, I think it's very clear from the outside that to everyone that is very tight.”
33:53

Financial Complexity of Inference

Discussing the financial structures needed for scalable inference operations.

“Because people think you just take VLM, which is an open source framework, you slap it on some GPUs and you call it a day.”
35:06

Building a Company with Customer-Centric Mindset

How to foster a customer-focused culture in a tech company.

“Like, there's a lot of complexity into how you finance these things, how do you think about them?”
36:37

The Role of Luck and Skill in Success

Examining the balance between luck and skill in achieving business success.

“Yeah, not everything's equally challenging, right?”
41:13

Vision for Future Experiments in AI

Imagining the potential of large model training without constraints.

42:04

Thought Experiments on AI Model Training

Exploring hypothetical scenarios for AI model development without constraints.

43:10

Book Recommendation: Breakneck

Discussing the significance of the book 'Breakneck' in understanding technological innovation.

“I think Breakneck is a very important book in the context of today and like how the world is evolving.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On The Generalist, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00Even in a world where there is AGI, the only thing left to do will be inference.

0:04Tuhin Srivastava: AI now has to think. In order to think, it has to inference. Inference in a lot of ways, I think it's maybe one of the most misunderstood markets in the world. GPUs fail all the time. Chips fail all the time. In a market where the demand is growing exponentially, there's so much complexity. BaseTan provides performant, reliable infrastructure for training and inference for the fastest growing AI companies in the world. We are a critical path. Like at BaseTan now, a lot of our businesses aren't working. Why doesn't every company just figure out how to wire inference better and better themselves and optimize that?

0:35It's hard to get capacity at a big scale. It's kind of impossible to get capacity at a small scale. And I think the complexity of the software and the actual running this at scale is quite high. There's opportunities to compete with hyperscalers and beat hyperscalers when you are bridging multiple of their different services. If you had unlimited resources and no operational constraints, what is an experiment you'd like to run? I'd love to run.

1:06Every industry has a cost of goods. Retail has materials, shelves, and trucks. Software has servers. For AI, it's inference, the work of running a model every time someone asks a question. As AI takes over more of the economy, that line item grows with it. Tuhin Suravastava thinks inference may prove to be the final market, measured in the tens of trillions. He also thinks it's badly misunderstood because most people have no idea how complicated it is to serve a model quickly, reliably, and at scale. Tuhin is the co-founder and CEO of Base10, which does exactly that for companies like Abridge, Cursor, and OpenEvidence across 20 different clouds.

1:49In June, a$1.5 billion round valued the company at$13 billion. Today, we discuss Base 10's insane growth, why companies may end up running dozens of specialized models, and the brutal supply crunch in compute. I'm Mario, and this is The Generalist. All right, let's get into it. I'm super excited about this. Tuin, maybe just to start, we can start super simply. For folks that haven't heard of Base 10, how would you describe what you're building? Yeah. Base 10 provides performant, reliable infrastructure for training and inference for the fastest growing companies in the world. What I love about your story is that you're sort of on this absolute burner of a trajectory at the moment.

2:31But it wasn't always that way. Sort of the first few years of the company, you had to endure a slower start. When you started back in, I think it was 2019. Yeah. How would you have described what you were trying to do at that point? at the time yeah so when we started the company it was pretty straightforward we we were thinking about me and my two co-founders amir and phil we were hanging out in you know all kind of without a job at the time we'd done a bunch of stuff around machine learning and we're like oh okay where can machine learning be used in x y and z way and then we realized that we didn't really have any idea about how you could like what is like the the right way to use machine learning or how machine learning will work in any one application, but we should build a tools company to kind of, as a picks and shovels business, to kind of enable the holistic machine learning trend.

3:22And so that's kind of what we thought about. The thesis back then was a bunch of people were training models and these models look very different now than they do now. Let us give you all the scaffolding you need to turn those models into value. To look back at those moments, you have to think about a few different things. So one, it was data scientists, not engineers back then, which is interesting. It was internal tools, primarily not production, because these were small models being used for stuff like fraud and classification recommendations. And so there were three things. It's data science, not engineers, small models, not big models, internal tools, not production.

4:01And we were building for that different world to some extent. Were there sort of changes to that product vision that you had to make to sort of skate to where the world took you? Yes and no. I think we'd always built serving what we now call inference as part of that thing. When you were serving models and CPUs and non-GPUs, inference was kind of trivial. And so even though we had this inference engine, we kind of gave it away for free, being like, oh, this is a commodity, no one cares. And so what actually ended up happening was, turns out when you have big models and big GPUs at a high scale in production, serving is actually incredibly difficult.

4:38built and so we had all this other scaffolding that we'd built besides this and then we just kind of cut that out and focus all on this one piece which was the serving infrastructure during those years of sort of uh you know more of a grind how did you maintain your equanimity as a founder like were were you tempted to sort of i don't know make a big pivot at some point or shut it down or do something else you know you were going through the the covid boom i'm sure of seeing tons of companies just ripping and raising. Yeah. And it took longer for your wave to arrive. Yeah, look, I don't think, you know, we had, you know, unbounded confidence in this idea that we were pursuing.

5:22It wasn't that challenging either to kind of focus because it felt early and things were changing enough and we were never out of ideas. And then more importantly, like the team was small. It was really fun. I was just, you know, we're just hanging out with our mates and having a good time to some extent. Yeah. Well, one of your investors told me you're a big runner. Is that right? No, not really. I run, who told you that? Nikki from BlackBerry. Nikki, no, no, that's not true. Yeah, I run here and there. But it's not any core part of my identity. It's not? It's not that, no. Okay, yeah. What is the way that you like find the place for your brain to like take a rest or reset or come up with new ideas?

6:01You know, some people, maybe that is running. I mean, look, I think maybe something that's interesting is that I actually live in LA and not San Francisco. And that provides like a really good, you know, breakdown of work and, you know, still, you know, there's always work. But it's also just, you know, it's something different and takes you out of kind of this environment. Yes. To some extent. I think besides that, I don't know, cooking, you know, hanging out with friends, you know. i would like to think that i have you know enough of a um enough balance beyond work but work is predominantly where i think it's also just very challenging i'd say i'm in this environment when things are changing so fast to really find balance yes yeah has it just been i imagine starting in late 2022 it's as if like you know a starter pistol went off and you've just been in a flat out sprint since then yeah i did like probably like q2 of 23 like that's probably like when like things really started changing and just it just started becoming like you know it's interesting looking at revenue forecast for the end of the year and like halfway through the year and like 23 we were completely off 24 we were completely off 25 we're completely off 26 it looks like we're going to be completely off no we were definitely completely off and so it's like everything just kind of like, oh, could it go any faster?

7:22Could it go any faster? When you say completely off, are you off by like an order of magnitude, one, like a two X, three X, like, do you get better at forecasting that or do you get surprised each time a little bit? Until now, I think we've been pretty surprised each time. That's like, hey, this market is bigger than ever. Inference is, you know, the final market to some extent. Yeah, no, it's off by a lot. Like I think if I compare our Jan forecast in 25 to our end of to what we actually finished we're off we're off by 4x wow yeah when you say inference is the final market I've heard you say that before I really like love the framing of it but I don't know what it means it sounds very cool yeah but what does that mean yeah well it means nothing no I'm just joking the um all we are somewhat alluding to there is that if AI is going to eat the world you know it all that's going to matter is where that you know how that ai is run i see yeah and you know like and even in a world where there is you know agi yeah the only thing left to do will be to will be inference to yeah to optimize to run the way this works to run the intelligence itself i see yeah okay that makes sense actually i think that maybe yeah i think that's pretty good yeah at what point did you realize hey this like you know all of this stuff we've built this one little piece here is the part that is the company and you know needs to be the whole focus and we can we can get rid of all of these other you know primitives that we built for data scientists or whatever else was that like the i was really quick it was like you know i'd say like december 22 actually i was just like oh already yeah it was like the i think the story we had there was that we had this early customer and they came to us and like oh we have this fine-tuned stable diffusion model that we want to run and we're like cool um and they're like oh we're gonna need a lot of gpus like yeah sure you will um and then it ended up like you know getting posted on hacking news and taking off and we kind of scaled with it we're like oh this is probably the thing that will be necessary and i think we probably made those changes like really in like six weeks where we kind of like took three years of stuff we've been building through it away and we're like let's let's let's do this thing only wow and this was refusion right yeah it's refusion okay um very cool which has since been acquired by it became producer then it's been since been acquired by google yes right uh taking taking a step back like you know when you look back at your your upbringing your background like were you were you building things from an early age was this like you know always in the dna to be an entrepreneur or something you came to later no i honestly like really like how i i we we got to startups was i was working in finance i ended up somehow working technology like i hated working in finance at the time and i i moved to boston doing some research stuff using machine learning through um some neuromuscular disease research which which led to a startup that was coming out of that research lab i got into tech and then in 2012 i moved to san francisco got involved in early stage technology started working to come to gumroad which like an e-commerce platform at the time i just really liked i really liked early stage and it wasn't even like oh like i need to be a founder i need to be an entrepreneur it wasn't that at all it was very much just like oh this sounds fun i can go work with a small group of people that i really like and yeah so like the first company we started was actually in 2014 i want to say so phil and i is my co-founder of base 10 as well one of my co-founders of base 10 we started a company we worked on at those other companies for about five years yeah and then we got to base 10 you know if entrepreneurship maybe wasn't sort of something that you had as a mental model in those early days, when you look back at yourself as, you know, a kid and adolescent, whatever, whatever, are there sort of traits or, or things you were doing that you're like, that was maybe my early way of, I don't know, building conviction or being sort of someone who brings things into the world.

11:15Like, where did you see that in yourself? Maybe sports? I don't know. I wish there was a romantic answer here, but there really isn't. Like, you know, I worked in like, I think most of my life was pretty boring. Like, you know, I grew up in a really um normal household i grew up in australia um i went to school i studied electrical engineering i jumped around a bit you know i went to work in finance because everyone else did it was 2009 when i graduated and you you know you that's where you worked in financial consulting yeah i went to work in finance and then like i kind of just it was i'd say it was not it was non-linear but not extraordinary yes i see by by any means yeah you had a clearly you know as you narrate the different jumps you made clearly you were willing to make some some unusual jumps you know like you know two years in finance to then like you know disease and all of these sorts of things you know you mentioned obviously growing up in australia now that you've spent so much time in america what do you think the best parts of the australian personality are versus the american personality like what's the best parts of those cultures yeah i don't know australia is pretty great have you been to australia i never have no yeah australia is awesome um what's the best part of australia i think part of the best part of australia is probably a few things which is one um we don't as a culture you're taught to not take yourself too seriously and in fact if you do take yourself seriously people kind of just a point in you and say shut up mate like yeah yeah no yeah um that's also probably there's a downside to this as well yes that's the better part of it the second part of it is i don't know if you go to sydney it's just fundamentally a lovely place um it is it's warm people seem happy people play a lot of sport and so i think just there is a sense of balance in people's lifestyle lifestyle i don't have that balance myself right now and i think the third one is probably just around like there's this idea of not self-optimizing i don't want to say it's like the opposite of capitalism but it's it's it's different at least where you're kind of like you know you're not like this it's like there's a name for it it's called top opportunity you're not exceptional yes right um don't optimize for outliers again like i think but i think there's like the downside of that which is that probably to some extent it cuts off the ends of the curve and so you don't get outliers but the upside of that is like everyone's kind of contributing back to the median and the mean which i think is a really nice thing because it just means that like life in sydney is very good industry that's very good for a lot of people well let's talk a bit more about inference i think for folks that maybe haven't you know spent a lot of time in this space they might think why isn't this already very well served by the big cloud providers yeah like you know it's not obvious uh if you haven't thought about it like what why is there this opportunity for base 10 so clearly so i i think what's interesting is that you got you got like a good way i think a good lens to think about this is to think about it through who are the obvious people who would solve this problem and i think i think you're right which is like it's like hyperscalers and i think there's like two or three different reasons why they haven't solved it i think the first one is it's a fundamentally new workload and you know and the ai workload and the ai workload actually broken into both training an inference and these are fundamentally different types of the same workload i think that took people a while to understand um that's one i think the second one why they didn't serve as like is like the market's just moving very very fast so you need to serve the needs of customers with really really short turnaround times and that's kind of just antithetical to how some of these large clouds work so a good example of this is like if you work with hyperscalers you know they have a commitment model where you have to make reservations and commitments over a long period of time.

15:05Yes. Negotiating around these commitments can take months. Wow. Especially at some sort of high scale. Yeah. Now think about, go back to like the world that we live in where there's new models every single week. Our customers are usually thinking like they work on the order of days, if not hours. And now when you put those two things together, there's a fundamental misalignment between the two. I think the third one from a hyperscalers perspective is like, I think, this is not my thought. This is Chardot and Index's thought, actually. There's opportunities to compete with hyperscalers and beat hyperscalers when you are kind of bridging multiple of their different services.

15:52And so, and what that means is that if you think about what does inference pertain to, well there's a lot going on there where you need obviously compute so you need compute you need like all sorts of network optimizations um you need a ton of observability and then you need a bunch of workflow and all four of those things to some extent the flavor of them that applies to inference is somewhat unique somewhat unique so you can't just like you know the observability for inference yeah is not just the same observability you need for the rest of your software stack so the sort of primitives for these other hyperscalers they don't necessarily have all of this pre-built in the way that you need exactly and so but inference as we sell it as we as we serve it to our customers is actually bridges all four of these and if not more and that becomes and within hyperscalers usually these things usually these things are served by different teams in different orgs even and so now for the customer to solve that they're now dealing with four different organizations at the hyperscale not just one um and that's where the opportunity compete with hyperscalers i think inference fits exactly into this category yes it's not just a feature that they can add to ec to and be like that's that's what it is i think the other set of competitors are like neoclouts yeah i say which is you know you know and we partner with a lot of these folks but i think one thing that was mistakenly assumed was that whoever you know because you did training, you could do inference.

17:25Or how hard could inference be? And I think that has been dispelled since then. It's just not, you can't just apply the same stack. It needs a fundamentally new stack. In the long term, though, do you expect that the best NeoClouds will compete here? And frankly, like the best inference players will start to, you know, make sense. I think it all converges. I think you, why wouldn't you, right? Yeah. Like if you're a NeoCloud, you're like, okay, well, this seems like the stickier part of the stack. Yes. um this seems to think that that grows somewhat linearly um with usage and if you are us you're probably thinking or if you're an inference company you're probably thinking well i need control of my own destiny to some extent yes um and there is obviously points of margins to explore yeah there as well in the fullness of time yeah so yeah i think so probably i think but i think like i think the other way which is maybe the less zero-sum way to think about it in which we we think about it, which is, look, AI spend is probably going to be, you take some, you know, random consulting report, but let's call it like$5 trillion of spend in 2030.

18:31Made up number, but what is that? A lot of trillions of dollars. Okay. What is the inference? Well, inference is the cogs of AI. So it's like, that's good. So let's assume these application layers, AI application spend runs that 50 % gross margin. Let's call it 60 % gross margin. So let's be, let's be a Let's be conservative. Okay. So that means that there's like$2 trillion of inference spend. Inference spend. Yeah. Now then think about like AWS. So AWS, from what I understand, and we can double check this. Maybe they can be a little footnote. I mean, I think it's like a few hundred billion dollars of revenue.

19:13Might be a bit more now. and if you think about them and that's probably going to i think if i've read if that if it was to go public by itself as a spin-off company it'd probably be two to three trillion dollars market cap so we're seeing about 10x that opportunity of market cap yeah in the fullness of time and so in my mind yes there's a like you know everyone's going to want a piece of this pie and i think you know obviously we are a market leader i'm in inference um at this point but the pie is also really big and i think what you're going to end up with is if you if you think about that that you know um 20 trillion dollars of market cap that that's going to be in print and you know maybe you can argue that 80 of that will be the labs the labs so let's say 20 labs so that's four trillion of market cap yeah that's a lot that could probably support you know three trillion dollar companies yeah totally i mean definitely can and and you know another 10 you know 50 billion company yeah and so i think what you're going to end up having is just a lot of companies supporting different flavors of inference and i think um different flavors of inference and then i think um yeah and i think that's kind of like that's the favorable way that i like to think about it which is like hey this pie is just going to be so massive yes totally yeah it doesn't feel like this is going to be zero sum at all it's more just yeah how how the game maybe plays out at these different levels.

20:35Clearly a big part of sort of the demand here is the demand for specialized models and yeah, the different ways that people are using that. Maybe to get even a layer more detailed, can you sort of give me maybe a customer story of how someone is using Base 10 and what that process looks like a little bit? I think all customers look like they're on the same type of journey. But really like what it looks like is everyone starts with closed models. and using closed models kind of was easy. You know, you had two, you kind of had like three options, probably two at any given time. They've all kind of converged upon the same API.

21:14And so you choose one, maybe you choose the other and you probably fail over between the two. Yes. Like when you get really limited one, you know, use the other one. Cool. That's how most people were using it like a year ago. Yeah, totally. I think about some messages I got from our customers being like, do you have anything that would like support like the routing layer a year ago between closed models. Oh, interesting. About 12 months ago, but about 12 months, okay, so you're using closed models. Great. Yes. You suddenly discover open-wage models. And I think this has obviously changed over the last 18 months.

21:48Yes. The last 18 months. But, you know, 18 months ago, there was DeepSeek V3, which was the first time people were like, are open-wage models actually really good? And you're like, yeah, they're actually fantastic. But over the last, like, six months, what you have seen is some, like, you know, week after week just launch after launch of these models coming from six or seven different players yes um all kind of progressively better and also just getting better at the same time and maybe just to like cap off that point if you go look back at last week and you know i don't want to date this but let's go last week glm 5.3 release glm glm 5.3 it's frontier like you know you you know it's not it's not a the best open source model yep it is competing at the frontier you know it is indistinguishable and even better than the closed frontier you really think so i haven't i haven't used it myself but i mean you can go look at the benchmarks but it's hard to sometimes totally tell just the benchmarks but it's the way our customers are using it they're seeing the same they're like it's close enough it's close enough call it the frontier yeah so like okay so you everyone's using closed models about a year ago everyone starts thinking about open-weights models like oh maybe we can augment it and they start basically augmenting some of their usage between closed and open.

23:00That same routing software they were using before, they're using it here. It's pretty rudimentary still. You go to your favorite inference provider, which is obviously Base 10. You route between them now. So that's the second step of the journey. Now these customers are using closed open way models. Not in any fancy way. Just like they're like, hey, I have another option now. Yes. What's cheaper? Yeah. You can use that in a couple of ways. You can either use it as like a multi-tenant API, which is, you know, it's kind of a common endpoint. It's hard to guarantee SLAs on multi-tenant APIs because there's the noisy neighbors problem.

23:29Maybe you run into that or maybe you start to think, hey, actually, I want to run this model in a very, very specific way. That's not an option you used to have with closed model. But open models, you might say, hey, actually, I've got Kimi K3. I want to use it half of the time. I want to use it for like a throughput optimized version. And the other time they want to use it for like a latency sensitive version. It's like I need custom deployments. And so that's where we have like dedicated deployments. So we have like model APIs, which is the shared APIs. Then you have like dedicated deployments.

23:59Cool. So now you have all the different ways of using these, but now these are starting to generate a lot of data. And now you're like, hey, I've actually got this perfect combination of what are the customer's inputs and therefore the prompt to the model. I've got the model's output. And as a customer, you probably have some user signal about what is good and what is bad. Yes. So then you start to think about post-training and you start to think about, I've got these open-waste models that I can post-train against some utility function, which represents the customer user signal in your model, and I can hill climb against that.

24:33Yes. So now, you know, you go through this post-training, we have a product called Loops for doing post-training. Basically, you end up with a model that is custom, that is yours, that is fitted to your data, and you start running that in production. So now you have kind of like four different types of models, right? So you want to close, to vanilla open, to dedicated open, to customized open, and then you really end up with this way where you're like okay now i have many models how do i wrap my traffic yes between them and that's why you know with a lot of customers we're even helping them train routers which is like how do i that seems like the next step right how do i do that and like you know i kind of painted this as a linear journey yeah but the extent to which there are loops between these all these things like that is the holy grail right yes which is that you have models that are pretty that are being used in production that are producing data that you are then using to retrain those models and then run them in production again and the extent to which you can set up those loops that is the wish to you have continual learning unlocked inside your customer like that's really like what we are selling to our customers which is like when you when you think and and if you go and look at like a lot of the narrative of last six months about owned intelligence about owned intelligence yeah a lot of it's been around hey what is the compounding advantage that um companies have yes and the compounding advantage that companies have is that they have proprietary data they have um the workflows that power um their user experiences they have customer feedback that is just so these are things that are just theirs and then you have the model improvements that all three of these things can power um and that is that that that circular thing continual learning piece that i'm talking about, which is the extent that you can plug that into that continual learning loop and have that accumulate inside your company is the extent to which you have owned intelligence.

26:22And that's really how our customers are thinking about this. Yes. Well, I really love the way you laid that out. I feel like that was one of the crispest explanations I've had of that loop in a way. When you think about the fourth category there, the sort of, you know, your own owned model piece, the, you know, customized model. What do you expect that to look like in a few years from now? Is it going to be the case that every company has 10 specialized models, 100, 1 ,000? Like, you know, it's, you can imagine. I think so. I don't know about every company, I think. But a reasonable size, you know?

26:58Yeah, but a bunch of companies that are running a lot of AI, which is going to, I think, be most companies. I can think of three data points right now just off the top of my head, which is like, if you go look at Sierra, which is Brett Taylor's, the CEO of the company that he founded with Clay that's working on agentic customer support experiences. Like I think he tweeted out that, hey, they use 15 open and close those models in production. that's one and the second one is brian armstrong brian armstrong um tweeted out recently um that they've been able to exponentially grow token usage while keeping cost flat which i think is is a function of you know some combination of intelligent routing yeah and um and caged inputs and outputs and like figuring out that there's many models can serve the purpose yes and then to to swing all the way back to like a large enterprise so you have an ai native company in Sierra you have a digital large digital native company in Coinbase um but is I think most people consider that enterprise but then think about a company like AT &T where they just said hey we're publicly publicly staking our future on open models to save costs and to protect the proprietary data oh interesting and I think like that is probably like that is today yeah and that's three different flavors of companies you know like and I think it encompasses most of most of the world like like companies that matters, which is very fast-forwarding AI companies, very large technology companies.

28:25Yes. And just the enterprise. Yes. And to me, like, if you think about that, like I think most of the companies, most companies like projecting forward a few years, start to think about not just frontier models and them equivocating that with closed source models and models from the three labs. Instead, I think everyone starts to think about things as a portfolio of models. you have closed models from the labs you have the open frontier and then you have post-trained models where you're matching your tasks to the like to a long tail of models and then you're getting really good at routing between them you know the other sort of piece of this puzzle that we we've alluded to maybe a little bit already is is the compute part of this question you have a sort of front row seat to a lot of this maybe to start like what is the sort of state of play for base 10 in terms of you know clusters and clouds that you have to sort of be across yeah yeah look that's that's like a fundamental thesis that we have is this idea of like heterogeneous compute like i we think about base 10 with three layers it's a heterogeneous compute layer then it's a core inference layer then it's all the primitives that add value and um kind of make base 10 stickier for our customers and so we can add more add value the heterogeneous compute layer assumes that we're to get compute from a bunch of different places today we work with 20 different clouds in 90 different regions and i call one of our core pieces of technology is being able to put mcm we've talked about a lot publicly the ability to scale across these clouds and that's a necessity for us in a compute constrained world which is like hey like we have you know we're obviously open for business and we have compute for our customers but you know um we need flexibility in terms of where that comes from where we can stand it up and how we can span across it and And like one thing that we can do really well, which has a lot of benefits like for the end user beyond just capacity is stitch it all together and kind of make it look like one virtual class.

30:21And, you know, those benefits for our customers are reliability and the fact that, hey, if one region is down or one cluster is down, we can always route you elsewhere. Yes. We give you geographical optionality if you have data residency requirements. And also we can just like serve inference from very, very close to where your end users are, which again cuts down. latency. In terms of the available supply right now, like how tight is that squeeze feeling? I think I heard you say in maybe the No Priors podcast that you were sort of constantly at mid-90s utilization. Yeah, that was maybe a few months ago.

30:58How does that feel? Oh, it's just as tight, if not less. You know, it's very tight. You have to be creative. What does that mean? Well, you know, you have to start to think about where things can come from. you need to start to think think about different margin dynamics like where can you get um like emergency capacity you know i think anthropics talked about this um really publicly as well um i think krishna on a podcast said this about how you know you're constantly thinking about what is the long-term forecast and we built towards a long-term forecast and then we bridge towards that with short-term capacity um and so is it constrained yeah 100 % And do we have like, and is it tight?

31:36I mean, I think it's very clear from the outside that to everyone that is very tight. And, you know, inference has kind of taken the world from by a storm, which is fantastic. But if we're not flexible, like that is our business of being flexible and helping our customers not have to deal with this. Is it to the point where your growth gets pulled back by it at all? Like, is there a version where if you had access to 10 times the compute? Yeah, this is an unlimited market. Yeah. Like, you know, it's like a truly infinite market. There is. So that's the biggest break, essentially, at the moment.

32:11Yeah. What did the arrangements look like in terms of buying compute at this point? Are you buying several years out for now? It's such a complex planning exercise. Yeah. Like, yes and no, right? Like, everything looks slightly different. But, you know, term is a big piece. like you know you're but you're committing usually months beforehand um but there are variables you play with like price length like like how do you want to finance it all these things you know we're starting to see these neo chip companies come to the fore you have you know folks like etched obviously cerebris a little further along how does that sort of play into this equation yeah it's very exciting right because like in with inference you want a bunch of different chip types and like you know um you see that with like lpus and you know some of the stuff that um nvidia um just announced um i think it's called lpx that i was just post croc acquisition inference is an interesting workload and i think there's like lots of optimizations to make and i think all these new technologies add new dimensions to it i think the reality is is that nvidia has such a um amazing position um and they're operating at such scale that you know that is the best that is the best supply in the world like right now and like it is your best chance of getting chips is is through nvidia yeah just because like the supply chain is so scaled up as you start to like you know continue to have to scale up that supply how do the different financial requirements change for your business, do you start to have to build up a big debt practice?

33:47Or, you know, I could imagine lots of different ways of getting, you know, the oxygen into the system that you need. Oh, totally. I don't know about a lot of debt practice, but, you know, one of the very interesting things about inference is like, inference in a lot of ways, I think it's maybe like one of the most like misunderstood markets in the world. Because people think you just take VLM, which is an open source framework, you slap it on some GPUs and you call it a day. And supply-contained market operating at large scale with what I would call quite flaky technology, which is, you know, GPUs fail all the time, chips fail all the time, and running at very high utilization in a market where the demand is growing exponentially.

34:26There's so much complexity at every layer of this. And then when you add the fact that, you know, there was capex involved, and whether that means, you know, through building data centers yourself, or whether that means through, you know, like if you are looking at a couple of billion dollars of compute expenditure over a given period. In today's market, that probably requires about, you know, a five-year term and a 30 % prepayment. I can say, you know, play that out pretty clearly. So let's say you wanted to do this$10 billion of spend over five years. It's called a five-year loan with 30 % prepay.

35:03That's$1.5 billion of prepay you have to do. Yeah. Yeah, like, yes. So the answer to your question is yes. Like, there's a lot of complexity into how you finance these things, how do you think about them? Yes. And I think that's like another thing that factors into like the complexity of building an inference business. This is where the two years at Macquarie doing infrastructure. Project financing. Ended up being really helpful in the end, right? That was probably painful two years, but it's come back around in a way, right? Yeah, no, like I think it's really interesting because a lot of these actually look very much like project finance deals.

35:31Yeah. And, you know, it very much, and frankly enough, a lot of my old bosses are now in the data center world. That totally makes sense. You mentioned that people don't appreciate the complexity here. Is that sort of the ultimate defense against the maybe too facile argument that one could make that, hey, software is being solved? Why doesn't every company just figure out how to wire inference better and better themselves and optimize that? Because it's just extremely hard. It's hard to get capacity at a big scale. It's kind of impossible to get capacity at a small scale. You know, and so it's like, you know, we, you know, to some extent, you're trying to make that job easier for our customers.

36:16And then I think the complexity of the software and the actual running this at scale, it's quite high. And I think one way to think about this is like, go look at the companies that are doing well in inference and think how incredibly technical those teams are. Yes. Not that SaaS companies aren't, you know, challenging in this way, but like, it's not like, hey, you can, anyone can pick it up and start building this next week. Yeah, not everything's equally challenging, right? Challenging, yeah. Yeah. I'd love to talk a little bit about sort of how you think about building your company. I think you've talked before about the fact that, you know, it's not helpful to say that, you know, we look for smart, hardworking people because, you know, everyone looks for that.

36:53Yeah. I want not smart and lazy people. And lazy people. That's the dream. That's the dream. Yeah. Well, what do you care about and how do you find it? yeah um look i think we care about like a few things a few um things which is like are you intellectually curious that matters a lot um can you move very fast can you work for you know like i think part of our job is to be very fast because that's the way the market involves i think the biggest one and like look if you're smart and hardworking and you're technically we obviously have things that we want experts in that's right but i think one thing that's really really hard to teach it's just this you know customer customer service mindset like we we think of ourselves as you know if we think about um danny meyer's book um about about union square cafe yeah you know a lot of it's just about like this full service mindset we are critical path to most of our companies businesses to some extent like in base tens now a lot of our customers like businesses aren't working i think you obviously need to be an amazing software vendor um but that goes without saying i think beyond that it's like we know we want our company the customers to think about us as extensions of their own teams and a lot of that's um how we show up how we respond to incidents how we build relationships so you know when we're not in a renewal period you know like we would never do a qbr with a customer because you know we are talking to them every day yeah like that that's unnecessary we're doing a business review with you every day we know what's what's hurting you yes we know what's where you need help and you know really start to build relationships over like over time and i think that's you know that part of hiring is like it's hard to screen for but it is a call way we do business and then i think beyond that the biggest thing it's like you know take you know at the end of the day like you know we're just building software yes we are powering amazing use cases and you know we've got to build the best software actually possible and like it this is we're going to build as many nines reliability as humanly possible but i think it's like hey and just understand i also just understand that you know we are you know the core part here is to have a good time as well and enjoy yourself and And like, I think that ends up with the best thing for the company, but probably ends up with the best thing for the customer as well.

39:22You mentioned the Danny Meyer book. There's another one that comes to mind. I wonder if you've read it called Unreasonable Hospitality. I thought that was like really interesting the way that they frame that. And it feels like the same philosophy in many ways. One of our teammates went to one of our customers' wedding like 10 days ago. Wow. Yeah. I don't think you can do that unless you're really, you know, building strong relationships. You've also said before, I think, that you think success is really 90 % luck and about 10 % skill. Do you think that's really about the right split? Some people get upset at me for saying that.

39:57I don't get upset. I think you might be, there's something that you're right about there. I mean, I think people, I think everyone, so many people work very hard. You know, actually, Nikki from Blackbird said this years ago, which is like, everything is hard. So you may as well pick as big a market or as ambitious project as you can. but it's just as hard to build that at times than it is. Yes, very hard to run a cafe extremely well, right? Exactly, and to me, the extent to which what you pick is the right thing at the right time, that probably is all of luck. And I think there's enough, I think most things are, opportunities come about because the market shows up at the right time.

40:41Like I couldn't manufacture inference becoming the largest market at this time. and then obviously execution is a big part of it so what i would say is like that 10 skill is necessary though like the whole thing falls like so maybe the thing that like the the part that is wrong it's like i think they equally both are necessary but the majority is you know one part of it's way more in your control which is this which is like the the skill and hard work piece yeah well you mentioned inference becoming a market is one example where else do you think you've gotten luckiest most of these people i work with to be honest like we we like be since you know such a fantastic place i think obviously my co-founder is amazing but you know entire team you know everyone on the team works so hard um and they care they care about the customers so much i think we're very lucky to have found a lot of those people i'm very lucky to work with um sarah from graylock oh sorry from conviction she used to be it yeah she used to be a graylock and now she's a conviction um sarah was a great luck when she invested in base 10 seven years ago and you know we work with i still chat with her you know multiple times a day oh wow um and she's very involved with the business and we've been very lucky there um and then i think a lot of the people we've hired that you know i think um some of the earliest people like working with punkage as our co-founder and you know emily's has been around we've had a relationship with these people 15 years and then even recently people like danny who's our president who's amazing i think it's just the people i always like to wrap with a you know a thought experiment or two if you had unlimited resources and no operational constraints what is an experiment you'd like to run i think we'd want to train a very very large model um you know i i think you know i think that's gonna it doesn't make sense for us at this moment to do that we will who knows but whether where the world is going but like i think that's one which is if we had no constraints and obviously we have a lot of constraints and the biggest constraints in our computers that our customers need that compute more than we should you know there's plenty of models right now yes what they don't what doesn't exist enough of it's like really amazing infrastructure to run those in production that's where our focus should be yeah i think we you know think about training a model is interesting just because like i think it is so challenging um and it's just a new it's a new skill that we do if you could assign a book to everyone on earth to read what would you want to give people i can't i can't remember the name of the book um but it's a really good book which really explains kind of like how the Chinese government has promoted technological innovation, breaking.

43:27Breakneck? Breakneck. Good book. Breakneck is a good book. Yeah. I think Breakneck is a very important book in the context of today and like how the world is evolving. Yes. I think everyone should read that. Well, Tuan, thank you so very much. This was a lot of fun. Yeah. Amazing. Good for having me. That's it. Thank you for listening to this episode of The Generalist Podcast. Please subscribe on Apple Podcasts, Spotify, or your preferred podcast app. Ratings and reviews help others discover these discussions, so if you enjoyed the conversation, I'd be grateful if you could take a moment to leave one.

43:59For all past episodes and more, visit us at thegeneralist.substack.com. See you next time as we continue to explore the future.

From the publisher

Tuhin Srivastava is the co-founder and CEO of Baseten, a company building infrastructure for training and inference. When an AI model runs in production (responding to a user, powering an app, making a decision), Baseten is one of the companies that makes that happen reliably, quickly, and at scale. The company counts Abridge, Cursor, and OpenEvidence among its customers, spans 20 clouds and 90 regions, and was most recently valued at $13 billion following a $1.5 billion funding round.


Tuhin sees inference as AI’s “final market”: as AI spreads through the economy, the cost of running intelligence grows with it. Baseten’s breakout began when it discarded three years of other product work in roughly six weeks to focus on an inference engine it had been giving away.


In this conversation, we explore why inference has become one of the most consequential and least understood markets in AI, what it actually takes to run a model in production at scale, and why the compute squeeze may be the single largest constraint on AI growth right now. We also examine the customer journey from closed models to custom fine-tuned deployments, the concept of owned intelligence, and why Tuhin believes luck accounts for roughly 90% of startup success, while insisting that the 10% skill is still indispensable.

—

Tuhin’s bookshelf

When I asked Tuhin what book he’d assign to everyone, he named Breakneck. He also pointed to Danny Meyer’s book about Union Square Cafe, Setting the Table.

—

Every timestamp

(00:00) Intro

(02:06) What Baseten is building and how it began

(04:47) Finding balance in a fast-growing startup

(06:45) Why inference is the “final market”

(08:34) Tuhin’s path to founding Baseten

(12:06) What Australia gets right

(13:43) The opportunity Baseten found in inference

(20:34) How customers use Baseten

(29:00) Managing compute across clouds

(30:40) The compute crunch

(32:11) Financing inference at scale

(35:40) Why inference is hard to build in-house

(36:41) Who Baseten wants to hire

(39:46) Hard work and skill vs. luck

(42:20) Final meditations

—

Resources

—

Where to find Tuhin

X: https://x.com/tuhinone

LinkedIn: https://www.linkedin.com/in/tuhin-srivastava

—

Production and marketing by penname.co. For inquiries about sponsoring the podcast, email [email protected].

More from The Generalist

All episodes