How to Engineer AI Inference Systems with Philip Kiely - #766

0:00I mean, it might be the fastest timeline in the world. If you think about medicine, for example, it can take decades for research to reach a pharmacy. If you think about, you know, physics or engineering, it can take years to apply a new concept. Even within AI, if you want to train a model off of a new technique, it can still take weeks or months to find the exact right way to sort of express that technique. But with influence, the timeline is often ours. A new model architecture comes out. You have to figure out how to support it day zero.

0:46Sam Charrington:All right, everyone. Welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. Today, I'm joined by Philip Kiley. Philip is head of AI education at Base 10. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Philip, great to see you again and welcome to the podcast. Hey, Sam. Thanks for having me on. Absolutely. Absolutely. We met at the recent GTC conference where you were signing copies of that book that you have there over your, I guess, your left shoulder. Yeah. And I've got one here on my desk. Hey, there you go.

1:25There you go. That was a really fun conference. I felt like a rock star, you know, just like handing out. We also had, honestly, I think part of it was we had free ice cream at the booth as well. And I gave away a lot of books, but I did run the numbers at the end of the show. The ice cream was more popular.

1:45Sam Charrington:you know i i thought uh it'd be good to get you on and talk through like you know a little bit about you know inference engineering you know what what you're writing about in the book but also kind of what you're seeing beyond the book and um what is really kind of making a difference from you know inference perspective nowadays so before we get to that though i'd love to have you you talk a little bit about, you know, your journey, like what brought you to, you know, base 10 and inference engineering? So back in 2022, this was 10 months before chat GPT was launched to the public. I had a really strong and novel thesis about the AI industry.

2:31That strong and novel thesis was I need a job. and so i i applied for a bunch of jobs and i ended up choosing between base 10 and a laundry delivery startup um and it was like a really tough decision uh because hey it's it's a good the other one's a good company they're still around um but ultimately i i decided to go work on some some really interesting problems in the ml space with a tiny startup at the time called Base 10. And I've been at it for more than four years now. I've had a really fortunate position to be able to learn on the job and experience the rise of the AI industry firsthand, going from predictive to generative modeling, going from modeling being something mostly done for internal workflows to modeling, you know, being part of the cost of goods sold, modeling being in the revenue path for AI native companies and going from like very, very small use cases because of that to very, very large and complex ones.

3:40Sam Charrington:You know, one thing that's interesting about your experience, and I'm thinking about Base10 in particular, like Base10, as your timing indicates, predated ChatGPT. And they were, you know, one of, you know, a great number of companies at the time kind of competing around this MLOps, you know, set of ideas. And with the rise of generative AI, a lot of those companies, you know, got acquired. A lot of those companies went away. But of those companies that are, you know, still around and doing well, like a lot of them are in inference. Why is that? So inference is, I believe, the most important workload in AI.

4:29And it's definitely the stickiest. That's been one of the reasons we've treated it as our wedge into the AI infrastructure market. Inference is very, very complicated. And doing it well is really critical to any company that actually relies on generative models to power their product. so we started as an inference company but we were just influencing very different models i was you know taking cpus and i was taking some you know t4 gpus or a10gs when i was feeling real fancy and running you know xg boost models on them running very early like wave 2 vec and gptj type models and honestly just like having a lot of fun with it uh we were building simple classifiers dashboards that kind of stuff and found over time that like as the models were getting a little bit bigger and more complicated like i remember when whisper came out that was huge because oh man now the model is a billion parameters like can you imagine one billion parameters in just one model like i can't believe i'm going to spend all my parameters in one place but this model now it's like it's really capable but now it needs gpus you need a lot of them all of a sudden and it's hard to get it running it's hard to make it fast enough there's a lot of interesting problems all of a sudden in in running models and so i think it it just kind of naturally occurred that like the most complicated piece of the broader ml serving stack became inference as these models got larger.

6:08And then as, you know, true generative models came around, we were already, you know, thinking about an inference first view of the world. And it felt natural to just extend that to larger and larger and more capable models.

6:21Sam Charrington:You referenced both model serving and inference as a component of model serving. Like, where do you draw the line, you know, between the beginning and end of inference and the broader model surveying set of requirements? I probably use those terms a little too loosely and a little too interchangeably because the technical definition would be that like model serving is everything that happens end to end from a user request to a user getting the response and inference is the piece that happens on the GPU. But when I think about inference as a discipline, I am thinking about end to end. I think there's, as an inference system, you have to own the entire experience.

7:09And so to do that, you have to think not only about what's happening on the GPU, but what's happening every step of the way on the user journey.

7:17Sam Charrington:Yeah, so let's talk a little bit more deeply about like what makes inference difficult. You know, clearly there, you know, we know the models are getting bigger. We know we don't have enough compute in the form of CPU, GPU, etc. You know, talk a little bit about some of the other challenges that you see with regards to doing inference well. Yeah, so inference is a really fun and difficult topic. And I'm going to go into metaphor territory for a second here, which is that I'm a martial artist. I have been my entire life. Are you familiar with like UFC and MMA and all those kind of things? so you know there you can't just be an expert in one thing and expect to become a champion you can't just be a great wrestler you can't just be a great boxer the idea is that you have a lot of different skills each one of which can take a lifetime to master and somehow you have to be excellent at all of them in order to be a well-rounded mixed martial artist and you know in my background, I'm certainly no UFC champion, but I've done striking of various disciplines.

8:25I've done wrestling in various disciplines, and I've started to understand how those things blend together. I think inference is very similar. You need to have a wide range of expertise on a lot of very different complicated topics to build a truly effective inference system. There's what happens on the GPU, right? There's understanding CUDA level programming and PyTorch on top of that, and then the inference engines on top of that. There's being able to take and apply the research, the understanding of different quantization techniques, different speculation algorithms, different KV cache reuse mechanisms.

9:01How do you parallelize models across GPUs? How do you disaggregate across different types of hardware? So there's all that applied research. And then on top of that, there are all the traditional problems of running very large scale distributed systems. Anything that you're familiar with from the last 20 years of cloud architecture, as well as understanding how to take very, very large volume workloads that are coming from around the world and handle them appropriately. So every piece of backend web infrastructure, every piece of GPU programming, it all comes together in inference in Windows and packages that are very demanding.

9:43You often have a couple hundred milliseconds latency SLAs that you're dealing with. So it's the orchestration of a large number of very complex systems together that makes it such a difficult problem.

9:57Sam Charrington:In AI broadly, there's this very tight relationship between research and implementation. but an inference in particular like I've noted on several occasions like you know hear about the speculative coding work one day and like I hear about it all over in two weeks and everybody's already using it like do you find that that research to production timeline and inference is you know particularly rapid relative to other aspects of AI? I mean it might be the fastest timeline in the world. If you think about medicine, for example, it can take decades for research to reach a pharmacy. If you think about, you know, physics or engineering, it can take years to apply a new concept, material science.

10:50Even within AI, what moves faster? Training ones, for example, if you want to train a model off of a new technique, it can still take weeks or months to fine-tune the hyperparameters and find the exact right way to sort of express that technique. But with inference, the timeline is often hours. A new model architecture comes out, you have to figure out how to support it day zero. We had PoloQuant come out, that research paper, and an engineer on our model performance team had it implemented 31 hours later as a CUDA kernel. So the pace of applied research is very, very fast because it is a highly competitive industry and everyone is sort of searching for that next edge.

11:37Maybe the only industry where research goes into production faster than inference might still be like trading. To be clear, trading with a D, not training.

11:47Sam Charrington:Yeah, this might be a little bit too in the weeds, but with regards to PolarQuant, like the paper was a year old and then all of a sudden it got really popular last week or a couple of weeks ago? Like, what's the backstory there? Yeah, so it was sort of republished as a blog post and sort of caught everyone's attention. I think, like, as a little detour, you know, I grew up in Iowa, so I was very influenced by sort of the best Midwestern school of thought in economics, which is the Chicago School of Economics, famed creator of the Efficient Hypothesis. I think the most dangerous falsehood that you can expose an impressionable young person to is the efficient market hypothesis.

12:35And the idea that like everything is already figured out and everyone already knows everything because that's just not true. You know, with the volume of research techniques that are coming out, I think that it's easy to lose track of things. But it's also a case where there's a lot of low-hanging fruit in the inference world that has already been picked. And so now, as we try to, you know, figure out what's next to push the envelope, sometimes techniques that might not have seemed, you know, either feasible or worthwhile, even as recently as a year ago, all of a sudden become like, oh, wait, now we're ready for this.

13:15Now it's time to add this to the sort of industry standard and move the inference stack forward.

13:21Sam Charrington:It was just that, you know, the bump in the email that brought it back to everyone's attention and the timing was just right. It worked. You know, if the great thing about a launch that doesn't go viral is that no one knows about it if you want to launch it again. Exactly. You know, let's talk a little bit about who should care or who does care about inference engineering. Like for many, you know, that's somebody else's problem, you know, and it's, you know, the problem of engineers at like a handful of really large companies. But I suspect that there are a number of reasons why a broader set of folks should care about it.

14:03Sam Charrington:Like, what's your take on that? Yeah. Well, one reason to care about it is that it's a really interesting field that you might consider working in. There was, you know, three years ago, there were maybe a couple hundred inference engineers in the world. They probably didn't even call themselves inference engineers back then. And now there are thousands, tens of thousands, probably, depending on, you know, where you make the cutoff. and software engineering of course undergoing a massive change in the way it works and the best engineers are becoming substantially more productive there's a lot of questions around how many engineers we're going to need in the future and what our jobs are going to look like I believe that even with the advances in AI assisted code generation even with the increase in productivity on an individual engineering basis, there's still going to be demand for 10 to 100 times more inference engineers in a couple of years than we have today.

15:05I mean, I know at Base 10, we can't hire people who are knowledgeable about inference fast enough. And even if you're not wanting to work at an inference platform, the other reason to care about it is that every vertical AI application company is eventually going to have to figure out what their inference strategy is. And it's going to need knowledgeable members of technical staff who understand the actual technologies and trade-offs involved in inference to sort of develop and execute a competent strategy. Because great inference can be the difference between a really fast product that feels magical to users and a slow buggy experience that causes constant shown.

15:49Sam Charrington:You know, so let's talk about like when and why and how like knowing about inference impacts, you know, the decisions I'm making and the strategy or approach that I'm taking. Like a lot of people will be delegating inference to a provider, whether it's a, you know, first party model provider or a third party, you know, inference provider like Base 10 or Firefly or someone like, you know, what are the situations where I'm going to want to, you know, have deep knowledge about inference? Yeah. So I think the first thing is understanding what's possible, because in a world where most AI engineers are sort of trained on GPT and cloud models, you understand the world in a certain way.

16:39I have access to N models, and there's maybe three or five or ten. And each one of them has a sort of fixed level of intelligence, a fixed level of speed in terms of tokens per second, time to first token, maybe a range of that. A fixed cost in terms of input output tokens with maybe a couple variables around something like a cash token price or a discount for long running batch jobs. And then also a reliability. You know, you have an endpoint where you might get a couple nines of uptime and maybe need to engineer the rest of your product around that reality. And the first sort of lesson of inference engineering is going from taking all of these as sort of immutable discrete points and understanding them more as a spectrum.

17:31You can trade off between, for example, latency and throughput in a given inference engine by adjusting things as simple as batch size or things like, you know, adding or removing a speculation algorithm. When you do that, when you create sort of a spectrum of outcomes, an efficient frontier of high performance inference, then you start to understand like, wait, I can choose to change the way that I consume these systems. So, for example, you might start having priority queues in your product where paid user traffic gets prioritized over free user traffic. And maybe that's something that you couldn't have built previously.

18:18Or maybe you add a concept of, you know, maybe you understand, like, I have the ability to quantize models. And because I have my own very sophisticated and product-specific evals, I can do that with complete confidence that I'm not degrading the quality of the service my users are experiencing. And then when you quantize your own model and start to serve it faster and less expensive and with a great deal of confidence that you've calibrated it properly, then you are not subject to sort of random intelligence degradations from a third party provider who might not understand the exact nature of your workload.

19:01Instead, you've achieved the sort of price and performance goals that they're going for with that, but actually, you know, kept things appropriate for your task. So, you know, if you don't know a lot about inference, you might think like, oh, all quantized models are bad, when in fact you can calibrate them appropriately to your workload. There's all sorts of, you know, examples like this, even things as simple as, you know, understanding that the way KVCache reuse works means that you have to think about only your prefix and even one token difference early in a sequence can throw off the entire rest.

19:38And maybe that's how you structure your chat template or how you structure your inputs to your models. There's all kinds of ways where understanding what's possible with inference helps you build a better product and helps you deliver a better user experience.

19:55Sam Charrington:The last example you gave about the KV cache, KV caching, you know, is a clear example of it doesn't matter whether you're, you know, building out your own inference system, you know, you benefit from having that knowledge. A lot of the other examples, do they, you know, to take advantage of, you know, things like things that you mentioned, like, you know, changing batch sizes or, you know, different tiers of service? Like, do you have to host your own inference service and, you know, do inference on your own models in order to take advantage of those? or are there, you know, providers that like allow you, I imagine like everything, there's like a spectrum of the number of knobs you get when you're doing inference.

20:48Our philosophy at Base 10, not to, you know, cut a promo too much, is that like you as the end user should have access to every conceivable knob. And of course, along with like guidance on, you know, which ones are hot to touch and which ones aren't and what some sensible defaults are. But if you are making the transition from working with a closed model where you don't have access to the weights to working with either an open model or a model that you trained yourself and doing dedicated deployments, which basically everyone at a reasonable scale starts to need to do, where instead of paying on a per token basis, you're paying for the underlying GPU hardware, then you have access to these knobs.

21:35Whether or not you as the engineer have the sort of knowledge and confidence to tone them is why this discipline of inference engineering exists. But as companies of all kinds, especially like vertical AI native companies are achieving the massive scale that they are, everyone is thinking about taking control over their AI, owning their intelligence. and part of that means owning their influence outcomes, taking these knobs and figuring out the right positioning for their product and for their use case.

22:08Sam Charrington:Let's talk a little bit more deeply about the levels of control that one might pursue and it may or may not be a strict sequence, but what are the things that are happening in a business that, you know, drive the transition from, you know, one state to another. Like, so you talked about paying per token to paying per GPU time. You know, that spectrum goes to, you know, I guess I'm trying to get at like, you know, you could use a provider like you guys, you could, you know, stand up a Kubernetes cluster on like an infrastructure provider. You could, you know, host it all, you know, in your own building.

22:58Sam Charrington:Like, what are the dominant models or the dominant setups that you're seeing? And like, what drives a company to go from one to the next? The journey I sort of see is, in many cases, aligned with a product maturity cycle. And I want to specify this is really a product maturity cycle. This is not a company maturity cycle, because oftentimes you'll see relatively small companies further along in the cycle versus relatively large companies, just because they're sort of either more AGI pilled as a company, or just because the AI component of their product is more critical to what they're doing than the AI component of maybe the larger and more established product is.

23:50So as we think about the maturity cycle, there's a few options. Option number one is to rely entirely on pro-token closed model providers. And everyone starts here just about. Everyone should start here. It's really easy. You get frontier intelligence with an API key. And that's a hard thing to beat when you're starting out. I think the next level often looks like running into one of two problems, either a cost problem or a capacity problem and saying like either my token costs are just getting out of control and or I just can't get enough tokens to do the thing I want to do. And for that, oftentimes people start turning to hyperscalers, AWS, GCP, et cetera, and doing things like provision throughput purchases, doing things like spinning up models on like a bedrock or a vortex or an Azure AI Foundry or something like that.

24:51And this kicks the can down the road a little bit because now you have like a large scale commit with a provider who has, of course, massive scale and the ability to deliver you some capacity. And then the sort of next step that I see a lot of times is going on to more of a dedicated inference provider like a Base 10 where companies have, you know, the models with the weights that they own, and they start to set up, you know, a specialized deployment. I think the sort of other fork in the road there is to go for something that's a little bit more in-house, either, you know, building an in-house platform, staffing that team up and trying to deliver a sort of best-in-class inference platform internally.

25:38Or, you know, in some cases, like going really deep into edge inference, especially if you want to do a lot of sort of field work, you replace all of the distributed systems problems with Internet of Things problems, but you have a very similar challenge in scaling inference for sort of a large distributed edge network. I see a lot less of like companies going out and making enormous capital purchases of large numbers of GPUs, sticking them in the basement of their office and doing all that stuff internally. I do see that, especially in the medical field and medical research and healthcare companies do that.

26:17I've definitely seen that in the finance world as well and some of those other traditional enterprises. I guess the alternative to that is also like going out and making a very large spend commit. You saw like Jane Street, for example, announced today with CoreWeave. That's sort of a financialization difference between, you know, going out and buying a bunch of GPUs and sticking them in your basement. You know, I think that all of these approaches are valid in the right circumstance. I certainly wouldn't say that, like, all inference in the world needs to be run through a dedicated provider. But I will also say that, like, these technologies, like we talked about earlier, are, like, very challenging to do well and to keep up with as the industry continues to scale.

27:03So it's definitely becoming like more and more of a trend for folks to buy instead of build on the inference side, which is something that I'm like very grateful for. Yeah, yeah.

27:16Sam Charrington:One of the critiques that you hear, like about the businesses that are being set up to do inference for other companies as like the, you know, AI grows is that, you know, these GPUs are a rapidly depreciating asset and the companies are financing the purchases on the public markets and financing them through debt and all these other mechanisms. I'm just wondering if you have like a strong feeling about like you know the GPU lifespan as it relates to as it relates to inference and inference maturity and the way the the field is going so I've been through now three full GPU cycles ampere hopper blackwell and and I actually like haven't really been all the way through blackwell because blackwell isn't fully rolled out yet.