Speed will win the AI computing battle with Tuhin Srivastava from Baseten

0:05Hi, listeners. Welcome to another episode of No Priors. Today, Alad and I are catching up with Tuhan Srivastava, the CEO and co-founder of Base10, which gives teams fast scalable AI infrastructure, starting with inference. They're one of the players at the center of the battle heating up around AI computing. Welcome, Tuhan. Hi, thanks for having me. Good to see you guys. Let's start at the beginning. For any listeners who don't know, what is Base 10 and how did you start working on it? Base 10 is an infrastructure product, so we provide fast scalable AI infrastructure for engineering teams working with large models.

0:36Currently, we're focused on inference and we want to do a lot more after that. But for the past, I'd say, four and a half years, actually, oh, that's a long time. For the last four and a half years, we've been cutting our teeth and trying to build this thing. I think it's been pretty rewarding over the last 12 months seeing the market kind of show up and everyone get equally excited about AI infrastructure. We started this honestly because, firstly, we thought ML was pretty cool in 2019. We thought it was going somewhere. And we wanted to build a picks and shovel business and kind of solve the problems that we were running into.

1:11I think the side note here is that I wanted to start a company with my friends. You often say that base 10 isn't no code, it's efficient code. Like, why does that difference matter? That wasn't always the case, I'd say. I'd say like, you know, those times when we had elements which were definitely a bit no codey. I think what we've learned over the last three or four years is, you know, code is just incredibly powerful and engineers want to write code. Even in its best form, you know, you want to build really, really tight abstractions. But I think the ability to turn the knobs under the hood is very, very important.

1:45I think no code kind of makes that a lot harder. I don't think it removes it, but it makes it a lot harder. So what we do is just build very strong intuitive abstractions that try to make the easy things super easy and still make the hard things possible. So, you know, you can get a lot of value really quickly. But I say, unlike a lot of other infrastructure products that have been built over the last 10 years, we're trying to solve against the graduation problem, which is that, you know, that we're able to support teams as they grow on scale. And just to sort of make it a little bit more visceral for our listeners, like what are the types of applications that run on Base 10?

2:22Like what's the scale of the platform? Do you have a favorite application? Everything from, you know, tiny side projects on weekends, all the way to companies that are pretty AI native. We've supported foundation model companies. We work with companies like Descript, where AI is very, very core to the product experience. We power a lot of AI features that Patreon has shipped. But I'd say some of the more interesting use cases from our perspective, actually, from my perspective at least, are either the really small teams that we're giving a lot of leverage to so that they can ship things very quickly.

2:55So a really good example of that might be a company like Planned AI, which is basically building an SDK for call centers is how I describe it. But, you know, they're able to ship models and, you know, co-locate workloads so that they can get, you know, sub 300 millisecond or sub 200 millisecond responses without, you know, months and months of infrastructure effort. But I think, on the other hand, it's really exciting to see companies become AI-enabled. That's where we see a lot of the value is going to be over the next decade. If I look at a company like Picnic Health, which has actually been around for a decade, and it's starting to do a very, very interesting thing with this corpus of data that they've gathered over the last 10 years, and supporting those use cases, I think their model is called Picnic GPT, which extracts information from medical records.

3:44And to me, those are the really exciting use cases where you're giving leverage to companies that are good at the domain that they are working in. Their model might be proprietary, their data might be proprietary, but the infrastructure doesn't necessarily need to be proprietary. And we can give them just an easy way to deploy that stuff without many, many people months. It's become like in vogue to compare the size of your GPU cluster. Like people are spending a lot of money on GPUs. We hear about 600 ,000 H100 equivalents and lots of venture rounds being raised often to train, you know, large models in some domain or another or even, you know, more and more expensive post-training.

4:29Are training and inference workloads different? Yeah, I think so. I think they just have very different, almost like SLAs for the customer. Like, you know, things that matter for inference are things like, you know, your cluster is somewhat co-located with where you're doing your work, whereas training, you know, stuff like that matters a bit less. It doesn't really matter where that training is happening as long as all your GPUs are somewhat together. You know, even the GPU clusters themselves, like, you know, the full training networking is a very, very important piece to have networking on the racks themselves, where Inference matters a little less because you're doing a little bit more on individual GPUs and less so across GPUs.

5:13I think from a user perspective, there's a lot more workflow, I'd say, in Inference that's repeated across customers as opposed to, you know, you guys work with a bunch of companies training models. You know, the state of the art really is, give me some SSH keys and let me go at it. Whereas with Inference, there's definitely repeated workflow where people are trying to get similar things out of the inference infrastructure, whether that be version management, whether that be the way they deploy it, hooking it up into CICD, cold starts, and so on and so forth. So I'd say it seems a bit more repeatable today how the problem is being solved by customers.

5:53I'd say the hardware requirements are quite different. They're probably a little less to some degree. But I think resiliency and reliability matters a lot more. you know, downtime is unacceptable from an input perspective. Nodes get terminated all the time from a training perspective. You were quite early to this market, and I think you folks pioneered a lot of the sort of early ML sort of infrastructure for these sorts of use cases and applications. What has been the most surprising thing, or what did you least expect relative to how things evolved? Yeah, I think I can answer that question to two different abstinence.

6:28I think you can answer that question from like a market perspective, which is that, I think, Eli, you have some old writing, which is basically markets or all that matter. I think we've felt that very viscerally in some sense, which is that you can build all this cool stuff. And then when markets show up, you feel that. And that really pushes the customer forward and the needs for your product forward. So that's one thing, which is the acceleration we saw through the end of 2022 and in 2023. definitely took us by surprise. I can be really honest and say that from 2019 to 2022, it was pretty quiet.

7:10We had happy customers, but the demands weren't necessarily there. I think from a practitioner perspective, how fast some of these teams move has really shocked me, which is I think what is really clear in AI and early-stage AI in general, and I think the enterprises are waking up to realize this right now, is that speed is actually your number one advantage. Things are moving so fast that if you're not competing on speed, you're going to be left behind. And so there's actually a lot of propensity to buy versus build, I'd say, which is people are happy to buy technology. where I'd say in the past people were pretty hesitant to buy infrastructure.

8:01We talk with companies all the time where we think that, oh, they probably have something built out where they're a lot less sophisticated than you think and they're handling a lot more scale. So they need to be able to have the infrastructure to support that. So I think that's probably one of the larger things that we've been surprised by, which is how fast people need to move to be relevant. I think the other thing, which is just like how GPU needs have evolved going from, you know, at the end of 2022, most of our customers were using T4s and AT &Gs. You know, after that, it changed to A100s.

8:37Now it's gone to H100s. Like the compute needs aren't necessarily going down. They're only going up, especially as these services scale up. You guys were just highlighted for leading on the independent artificial analysis.ai benchmarks for highest throughput and lowest latency serving. Congrats. Can you give us some intuition for what is driving that? Like, you know, what makes inference hard to run fast? Inference is, you know, I think there's like multiple things which are quite difficult about inference. You know, I think there's a workflow headaches, which we've talked about a bit. I can talk more about this, like the scalability and reliability bottlenecks.

9:15I think the stuff you're talking about is performance optimization, which is really like how long does one generation take? I think, you know, there's a lot of work and research that's being done to run generations as fast as possible to get the maximal throughput and the minimal latency. I think historically, historically, well, I mean, it's funny when I say historically because I mean over the last six months. But over the last six months, a lot of work has been done in the research community to get basically these things to move faster, stuff like speculative decoding, which came out sometime in the last six months.

9:52It started really being used. For us, what it means is how well you can use the GPU, how you can scale across multiple GPUs, and how you can honestly be really, really up-to-date with the latest things that are happening in open source and research. We've partnered with a company, with NVIDIA, with a company. We've partnered with NVIDIA and really worked really closely with their LLM engine called TRT-LM. And that's actually driven a lot of the performance gains that we've worked with and we've contributed to that, we've forked that. But the hard thing there, a lot of the optimization you're doing is pretty low level and there's no real abstraction.

10:42So you either have to learn how to use OpenSource or rewrite some of these kernels by yourself. If you look at something like OpenAI, what do people complain about? A lot of the time it's speed. It's speed. And that's probably one of the core performance advantages of open source is that you can get these smaller models to run faster. And I think that will continue to be a massive focus for us going forward as well. On the benchmarks, I think it's pretty crazy how that's evolved as well. I think we've gone from state-of-the-art being 90 tokens a second, then it got over 100, now it's over 200.

11:22Now we're talking for some people up to 300, 400. And I think that's going to continue to be a very, very important place to innovate. And we think over time, it will get somewhat commoditized, the performance, especially for language models, to be honest. I think more and more of that stuff should run locally to some degree, I think. But being on top of it and making sure that we're kind of attached to the state of the art is, if we're not, existential risk to the business. And so we have to do it. How much optimization have you been seeing for other types of models? So diffusion models, some of the Texas speech models, other areas like that.

12:08I'm just sort of curious. There seems like there's different types of optimizations happening across different foundation model types as well. So I was curious, you know, what's state of the art there and how you're thinking about it. I don't have the metrics on hand, but we're seeing and, you know, also pushing limits there as well. So just yesterday, for example, you know, we were able to get Whisper running. So there's Whisper, there's faster Whisper, and there's Whisper on TRT, which again is an NVIDIA thing. I think what we are seeing is that there's more and more focus on bringing these experiences to real-time as possible.

12:41And so one of our customers is a company called Gamma, AI-powered storytelling software. They use stable diffusion image models to generate images. Getting that, not waiting four or five seconds there and still having high-quality images is, again, core to their business and making it very, very fast and easy to use. And I think, you know, we are seeing like definitely from a customer requirement perspective, we're seeing that. I think, you know, have we been able to juice as much there? Not yet, but I think we're getting there. Speaking of the applications, you know, driven by these models still being generally startlingly slow, including the like really amazing capable ones from ChatGPT to Cognition to, you know, things like Pika and MidJourney.

13:33I mean, like the, you know, in a way that consumers have not seen in many years, we are waiting, you know, seconds for interactions. Is your view of that like it'll change because the models are getting smaller? It'll change because more smaller models will get more powerful. People do distillation. People just get better at running these things. Like we'll get better hardware. What's the path to like not waiting 10 seconds a generation, two minutes a generation? If you look at so many of the gains of running this trial fast, let's take that as an example, the step function gains come from a few things.

14:14The first one is running on H100 and not running on A100. So I think as hardware gets better and better, you get this almost like leg up. And so as hopefully those prices go down and that gets more available, that'll be one thing. Hopefully the H200 comes up next and we do that. I think the next piece is around software optimization. So stuff like continuous batching and dynamic batching and speculative decoding, which basically makes it easier to either parallelize or batch process a bunch of things or makes each individual generation faster or offsets it, offloads it to another model. There's lots of stuff in that place.

14:52I think these models are also just going to get smaller, to be honest. I think that's the really, really powerful small model that does one thing, that's pretty exciting. I think stuff like Ollama is very interesting. But I think if you saw, Sourcegraph has this where they basically announced that Cody is now running locally for a lot of customers. I think that's a pretty exciting proposition. We have to figure out where we sit in the world like that. It's not necessarily amazing for cloud providers like ourselves because we want everything to run on the cloud. but I think things do get smaller.

15:31Things do get more efficient and we have more distilled, more powerful models, sharper models to do. Yeah, it's the unbundling of models. How and when do customers choose to deploy their own models or open source models or fine-tuned open source models on their own infra versus use public model endpoints? Or like what guidance would you give people? This is a general trend, right? Which is that you got to open out, you got to Anthropic, you have an API that works. Really quickly, you're like, that's too slow. or that's too expensive or I don't need something that powerful. So we often see, if customers have a lot of money, they end up going, paying for private deployments in Azure, they still have the sloniness issue, then they go to open source models.

16:13When you're starting out with open source models, you're going to go to a shared endpoint. There's lots of great shared endpoint providers. But there are things that might matter to you which that shared endpoint providers can't give you. Firstly, you want your own SLAs. So you don't want kind of this noisy neighbors problem. If me and Elad have two competing apps and Elad apps get slammed, I don't want my app to slow down. My model calls still need to be fast. So in that case, you might want dedicated compute. But there's also like data and privacy stuff. Maybe you don't want your data running on the same infrastructure, data going through the same infrastructure that other folks' infrastructure is going to.

16:56And honestly, maybe even at some scale, it's cheaper to run it yourself. And they're the three things. I think then when you go to larger companies, because of no brain, like shared endpoints don't really work for large companies. They're definitely not going to work for enterprises. And a lot of times you're going to want a, you might want Mistral 7B not coming off the shared endpoint provider, maybe not even on a dedicated endpoint. You might want it self-hosted within your own AWS and GCP. And that's kind of actually where we see a lot of the world going as well, which is that especially larger customers, they're going to have actually pretty good compute deals.

17:32And they're going to have their own credit system or spend commit with the marketplaces. And actually running it on the infrastructure solves a lot of problems and has a massive cost advantage to them as well. And so I think there's like three stages to it, which is like you started shared inference endpoint providers, you go to dedicated in the cloud. But I think for some customers, that's not enough either. And you want to go into your own cloud. One prediction on the enterprise side would be that, you know, if ChatGPT only launched 15, 16 months ago and GPT-4 just came out a year ago, then most enterprises are still in a planning cycle and they haven't really adopted AI at any real scale.

18:12Which means for infrastructure providers like Base10, it's like a huge opportunity that's about to come, right? Already you're cresting this giant wave and the wave is about to get 10 times bigger potentially. Ashley, what do you view as the timeline for really mass scale enterprise adoption of AI? And where do you think things will be in terms of order of magnitude usage overall a year from now, two years from now? I'm just sort of curious, like what your view of that future is. I think it's a good question. I think we've been so wrong with time, with time scales here. But I do think like what we see it right now is that when we go to talk to enterprises, I'd say a lot of them, you know, like for honestly, what we're seeing now is that like co-pilots, especially co-gen stuff, that actually has already made its way into the enterprise.

18:59Like most enterprises we talk to, like when you say, when you ask them how advanced are you on the AI strategy, they'll tell you, well, everyone uses co-pilot. That's like the first big foray. I think the next piece then is like using like OpenAI or Anthropic in some way. Well, I think that is going on right now. And I think people are starting to experiment with that. I think the fear, I would say, someone was just telling me that I think it was Pfizer, Ian Mark, tens of millions of dollars for ML investment or AI investment over the next 12 to 18 months. That's kind of frightening to me in some degree.

19:36I think it's great. It's great for us. I love to hear it as a business builder here. But it's also like, to me, that means that the pressure is actually coming from the above. And, you know, that's kind of like, I'd say, the ML trap we fell into in 2018 to 2020 when CIOs were buying software. And it wasn't really attached to real user value or product value. And so, like, I would actually say that we're probably overestimating how big enterprise will get in the next 12 to 18 months. But we're underestimating where we'll be in, you know, three to five years from now. And I think 10x is like a pretty massive underinvestment.

20:12underinvestment. I'd say what we see, we're working with a customer right now that has four engineers and has, by the end of this year, will have mid-hundreds of thousands of dollars annual spent. That's for one use case with sub-thousand users for them. And they're already cashflow positive as a business, which is insane to me in general. But I can't even imagine what these workloads are going to look like when we get to the enterprise. When you start to think about, you know, take a customer service and chatbots is probably like the number one place where people think about efficiency. The volume that some of these customers, you know, my brother is the head of AI at Sunrun, which is like a public company that does solar panels.

21:07And I think that's like another really good example of a company where there's so much opportunity for so much opportunity for AI just to eat away at processes. And the volume is just so much higher than what we're thinking from like a traditional business perspective with harder requirements, which even drive spending higher. Yeah, I think one reset in stance that people should have on spending here is that traditionally, people were looking at software companies. you got really concerned as a software investor if your cost of goods sold was affected by a lot of like data processing, basically.

21:49And so, you know, you have this like expectation that your average SaaS business at maturity might have 80 percent gross margins. And I think like, you know, now people understand that the training businesses have a big upfront CapEx investment. that, you know, may or may not pay off. But I think one of the things that you're pointing out is that you can actually spend a lot on the inference and the core intelligence and that actually, you know, end up with a very valuable business on the other end with perhaps fewer people. And so I think people have talked about that shape of company, but they don't really think of it as a norm yet.

22:29And, you know, at least in my portfolio, we are seeing more efficiency on headcount and like a lot more compute spend. And I know for some of the base 10 customers, that compute spend for inference is actually like the, you know, one of the largest items on the P &L. We were working with this customer and we basically, and this was like a challenge for us. So we asked for an upfront payment for the year of compute. And the CTO came back to us and said, hey, I appreciate what you're doing, but no, this is, after payroll, this is our second biggest expense for the year. We're not going to do that.

23:16And I think that's somewhat indicative of how much spend there is here. And it's probably also somewhat indicative of how big I personally think the market can be once we start seeing mass scale adoption. But I do think, I think that's a good point, Sarah, which is I think it's somewhat of a reset in terms of, I don't know if this looks like normal SaaS business. Actually, I know, sure, it does not. And I think even like traditional multiples, it's really, really hard to think about. And what's crazy about it though is I think the most efficient businesses through markups and through software optimization can actually drive pretty healthy margins and still have these really aggressive consumption contracts.

24:07And I think that's, I don't know, I think that's right. I can't think of, I feel like you guys see more businesses than I do, I hope. And then, but I hope, but you guys will be able to chime in on that more. Is that unique to this industry? Where else have you seen that? It's been a while since I've seen so many companies ramping so quickly. And sometimes they were fake ramps. So like, you know, in the internet wave of the 90s, it was kind of startups selling to each other and kind of bootstrapping off of venture capital. and then there was giant telecom buildouts on like a five-year cycle that caused huge revenue uplift and then suddenly there was a glut and things dropped dramatically here it feels like things are ramping really fast off of products that are a couple months old which sometimes suggests that there's not defensibility and so then the question starts to become okay how do you build defensibility and what does that mean and how do things get commoditized and do they and you know so there's a couple different markets where suddenly you see three companies all go from zero to five or zero to 10 million of revenue in a year.

25:16And then you're like, okay, there's three of these companies and they all ramped at the same time, the same amount. And so there's enormous demand. But what does that mean in terms of, do they cannibalize each other? Can three more entrants come in and do the same thing? Like what, what is the basis for competition in that market? And so I think there's a lot of that happening too, which is at least for me, pretty unexpected. And I think it's just because we have such a big technology capability shift that suddenly you can do things you literally couldn't do a year ago. You know, it's kind of amazing.

25:44I think it's particularly exciting when you go and apply that to, I think, Eli, you've cut your teeth on a bunch of different healthcare initiatives. Healthcare is like, you know, a really interesting place where you look at, you know, like Nuance, if you remember Nuance Technologies, like, you know, they had the stranglehold over this market for years. And honestly, it always looked like they were kind of struggling all along the way as well. and then Whisper comes along and you see that market now of note-taking for the medical thing. It's insane how fast it's going and there clearly is real value there.

26:22And then I think the question actually goes, maybe this does look like a SaaS business again when you're like, okay, what's the workflow? And is the power and defensibility of the workflow you're powering? Yeah, it's a really good point. And then the other thing I think that people often forget is that many markets are not monopolies. Many of them are oligopolies. You know, that's payments with Stripe and AdGen and PayPal and all these things, right? And so it's also possible some of these market structures are oligopoly markets. And then it's possible it actually ends up being win or take all.

26:50And there's some network effect or data effect. But if you look at some of these types of companies like healthcare, to your point, is a great example where it's deal driven, right? You have large deployments with big customers and you lock them in for multi-year deals. and if you're actually able to lock down customer bases and effectively you you can fragment in an oligopoly market more easily than if you have renewables every year right so i think also part of it is just like what's the contractual structure of a market and people really don't talk about that kind of stuff but i think it's really fascinating to think about through the lens of what actually is a sustainable business in each each one of these categories you know beyond health care it's like where are these businesses going to get disrupted um i think like financial services is an obvious one.