Teach the primitives or watch your competitor define them | Baseten’s Philip Kiely

0:04Today, I'm joined by Philip Kiely, and Philip is the head of AI education at Base 10 and the author of the new book Inference Engineering, which we're going to talk about some today. And Philip and I, you know, we share a background in developer advocacy and around education and developer advocacy itself has always been about empowering developers. And we've talked a lot on our show about how right now this is something that we all need to rise to the occasion for in terms of teaching ourselves, but also teaching others about the new norms of engineering and the way that we all get our work done.

0:36And especially now today, we're certainly starting to see a lot of folks in titles that are more AI native and focused on the AI front that's unfolding in front of all of us. And so we're going to talk about what it means to be in an AI education role, what that means to uplevel your game as a developer advocate and enter this new world of agentic engineering and the engineers that are here building. And we're also going to explore a little bit about the connections between all of this education, engineering and product and how it creates that sweet, sweet motion of go to market that we're also obsessed with.

1:07And it's frankly roles as well that, you know, head of AI or at AI education or AI platform, like these are brand new roles, but now they're in many cases becoming so essential. So Philip, welcome to the show. We're really excited to dig in with you today. Hey, thank you so much for having me. I'm really excited to be here. Amazing. Well, I'm going to dive right into talking about your role in developer advocacy. see. I'm curious, like when you talk about your role as head of AI education, how do you describe it? What is what is it that is most important to you? It's a great question. You know, I've always been very dismissive of titles.

1:47I've not been super into labels and definitions when it comes to my work, because my ultimate goal is just to be fill up at base 10, where I have a good understanding of the things I like to do and the things that are valuable to the company. And I just do those. But of course, you have to communicate to the rest of the organization as well as to the rest of the world, like what you're working on in a sort of succinct way. Just like maybe the title of a book is the most efficient version of the thesis of the book, the title of a person should be the most efficient thesis of the value of their work.

2:20And what we found is that there are a couple different types of go-to-market versions right now in the AI industry. One is a very traditional sort of market capture where you're trying to go after existing workloads. You're going after people who know exactly what they want and trying to move them over to your platform. the other opportunity though the the potentially more exciting motion i think is more of a terraforming market development type motion where you're going out and introducing people to new concepts that they're going to need to learn in the next months and years to effectively do their job and you're making sure that from the first moment they experience these concepts they're thinking about them in a way that's conducive to the way you do business so that's really my mission right now is to introduce a bunch of developers to AI concepts in a way that gets them thinking about the problems of inference and the problems of building reliable AI native tools in the same way that we do.

3:24Because if they think about the problems the same way we do, they're going to naturally come to us as the solution. Yeah, you're actually hitting on something that's really exciting, I think, right now, working in a role where you're kind of educating, informing, and empowering people about new stuff, that you're just putting new ideas in their head that are going to sit there for a while. They're going to kind of like you're planting the seed of thought, right? With the idea that over time, this demand, this knowledge, and what people can leverage from it is only going to compound and grow. So the earlier you're there in their mindshare and you're able to kind of set them on the right path, then the more they're going to like attribute so much of that back to you you're kind of part of like the nucleus of the thought which is like like what you hit on like what's something that's really exciting right now because it's like you know you're putting this new ideas of engineering in their head and then you're seeing how people are able to run with it yeah and you can do that from a sort of content and marketing perspective you can also just do that on technologies like hey, everybody is still using the OpenAI SDK, even when they're not using OpenAI models, and they're still using the same OpenAI output format, just because that was the original industry standard from a few years ago.

4:38That's developer mindshare, as an example. There's a lot of advantages to being a sort of first mover when it comes to capturing that mindshare. Of course, you've got to do a lot of other things to turn that mindshare into business value, like first movers don't always win, but you get the mind show. And that's a, that's a very valuable thing. Exactly. So how do you think about the intersection of like the things that, you know, a developer advocate might do with more traditionally engineering or product world, especially because now there's so much new everywhere on the education front, but also in the product engineering world, how do you partner internally as well as part of your role?

5:19what's cool is that i now have a lot of leverage to do engineering work because of coding agents so i'm able to ship like more complicated demos than otherwise i would have been able to build i'm able to ship you know programmatic seo on top of just ordinary sort of blog content i'm able to you know write custom software to make the process of creating and publishing a book easier there's a lot of different things I can do on the engineering side. But at the same time, I don't really think of my role as a developer advocate in terms of creating alpha and a lot more around distributing it, discovering it within the company, and bringing it to the market.

6:01And with that, that takes the close partnership with engineering to know who everyone is on the engineering team. That's a big challenge actually now because we're going through so much hyper growth. You've got to know who everyone is, know what they're working on, know their communication style, what they're going to be excited to talk about. Some people are very loud when they come up with a breakthrough. Some people just kind of ship it as a PR and don't talk about it. And so you have to be able to surface all of these different styles from across the organization and say, like, all right, of every awesome thing that got shipped this week, like, what is going to make the biggest impact?

6:39The other piece is helping enable the engineers themselves to become content people because a piece of content or an idea is always going to be best received coming as directly from the source as possible. So if that's ghostwriting, if that's co-speaking, if that's just like getting people a Twitter account and saying, go have fun, there's a lot that you can do to turn your army of engineers into an army of content people in a way that helps your company's brand be very authentic and very technically driven. Yeah, absolutely. I mean, engineers are classically very allergic to marketing. And the more that you can meet them on their level and also to empower engineers to tell their own story instead of telling it on their behalf, then the more flow of information that we have around all of these kinds of new ideas.

7:32And it makes everything that we do higher leverage as well. Yeah, I found that my engineering team is not allergic to marketing, in fact. I mean, I think of myself as an engineer. I'm an engineer by training and I love attention. It's all I do. Relatable. And so I guess that's why I say classically engineers are allergic to marketing. I do feel like that perception is changing. But the thing is like a lot of it comes down to just helping them understand the value of what they're doing on the go-to-market side. Showing, you know, engineers love numbers. Like I love seeing numbers go up. So just you can sort of show impact that way.

8:13but also showing in terms of, hey, we were able to bring in this customer because of your work, or we were able to bring in these candidates to join your team because they saw the interesting work you're doing and they wanted to come work with you. Just being able to demonstrate an actual outcome, a really concrete and tangible one, I think goes a really long way to getting just about any engineer hyped about the idea of creating content. I work with people who don't even want to, you know, get up on stage, don't like attention at all. And once they understand, like, okay, you know, if I do this, I'm going to get recruiting, I'm going to get customers, then they're willing to get up there and get a little spotlight on themselves.

8:56Yeah, exactly. And I think as well, like, you made a good point about how they don't typically maybe get as much exposure to the customer and their problems. And so once they get a taste of that, it becomes really exciting to be engineering and delivering things that where it's like, then you go meet with the customer or able to get closer to like how somebody used something that you shipped. Because at the end of the day, we all became engineers because we wanted to build cool stuff that people used and it gave them value. And so I think the closer that we all, you know, get to the reality of that conversation, the more productive and the happier and more fulfilling our jobs even feel, which is what you kind of called out there, you know, about like one's engineers kind of start to figure it out.

9:41And I like how you said too about distributing, like the distributing, like the alpha distributing, the gains, you're more like a bridge. I think that's classically like really when the developer advocate role has always traditionally kind of been like this marketing to product bridge or marketing to engineering bridge. And now it's a bridge between a lot of different places. And frankly, I actually think that's really a fundamental trait of the new AI leadership kinds of roles that are popping up at companies is that they're ultimately brought in to be a bridge because rolling out AI across a company is a very cross-team, cross-department initiative.

10:20You need somebody who can meet a lot of different problems at a lot of different levels and inspire people to come together to work on that. And so like, you know, at Dev Interrupted and at our parent company, Linear B, we talk a lot with folks that are in a developer of AI enablement role. That's somebody who's tasked with making AI a reality specifically like for the engineering team. So they can use it and they can ship with it and deliver it safely and the company can understand the impact of that. So understanding the impact of how we're, you know, adopting this new technology is really important for a lot of companies.

10:56How do you think about like measuring or understanding the enablement of the work that you do? One sort of difference is that my work is much more externally facing. That said, I do a lot of internal enablement as well, especially within the go-to-market team. What I like to measure is almost like a language learning process where the first step is trying to get people conversational, understanding the nouns and the verbs and how all these concepts work together. And then eventually we want to get them fluent, where we feel really good about putting a seller on the phone with a technical point of contact at a customer and having them be able to go through a discovery process and go through a genuinely consultative cell without every time they hear a word like VLM having to run back to the engineering team and ask what's going on.

11:52And then the ultimate goal is to sort of become AI native in whatever role they're doing. So like an AI native seller who really understands the space and has a strong intuition for what individual customers need, you know, an AI native engineer who is able to seamlessly use these tools to accelerate their work. One thing that we've done, which is really exciting, is we have a, from our infrastructure team, a mechanism for deploying Vibe-coded sites in a sort of secure, centralized, and standardized way. You know, it's behind our single sign-in gates. It's limited to only base 10 people. And with this, I've been able to ship cool sites, but also people from outside of engineering teams, from our people team, from our sales team, from our operations team, have been able to create custom tools and then host them internally through this sort of centralized mechanism.

12:56This is the sort of tooling that I think is really effective for these kinds of enablement pieces, because if you have everyone just kind of doing their own thing, it can really spiral out of control very quickly. But you also don't want everyone to have to go through a big procurement cycle every time they want to just ship a little web app. So having a strong set of tooling, technical tooling in place to be able to just let people ship is, I think, one of the biggest unlocks you can have for delivering AI across the organization. Yeah, I couldn't agree with that more. We've talked about that a lot on the show from leaders who are working on or building in spaces or otherwise have adopted these kind of closed ecosystems within not just their engineering work, but their whole company where anybody can go to this one platform and make an app or make an agent or deploy or share a workflow.

13:53And it's one about centralizing and standardizing all of them in one place where they can be, you know, controlled together. But then it's also too about sharing your domain expertise, getting other people involved in what you're working on and building, finding unexpected like users and use cases of things that you're building. And by having like a centralized place where all of that comes together, I completely agree is really critical. like we had James Everingham, the CEO of Guild AI, and he talked extensively about how he had built exactly that when I think he was at Meta. And so when he was at Meta, he had built a platform where folks could build and ship like internal apps and agents and tools.

14:34And the adoption off of that was, you know, it was, it skyrocketed and it got the whole company to become AI enabled in a really structured way. I'm curious to speaking about the whole company, how do you think about making it so that this is, we talk a lot about the show about the broadening of skills and the rise of the generalist. And you get these deep specializers and you get these broad generalists, like a designer who can ship or an engineer who can go to those customer meetings. I'm curious how you think about the shape of skills that engineers are developing and what's most important now.

15:06I think that what I'm seeing a lot is people with two deep skills. So if we've previously talked about like t-shaped engineers maybe this is a upside down u or something yeah need to need to brush up on my alphabet shapes it's a double it's a double t double t yeah generally it's you know for me it's engineering and writing engineering and and sort of content creation in general speaking all that kind of stuff for a lot of our engineers it is that sort of customer facing communication and accountability piece. One thing that we do is we have a forward deployed engineering team. This team reports up to our head of engineering.

15:52So at a lot of companies, you have a sort of sales architect, sales engineer motion where these roles are sort of within the go-to-market team. And at the moment, we don't have something like that. We have highly technical sellers who are able to go out and sort of speak to the customer and understand their problem. And then we have a team of forward-deployed engineers who, again, sit within the engineering org, are in the Slack channels with the customers, are deploying into customer accounts, are doing sort of hands-on development, co-engineering with the customers, helping them figure out what model's going to be best and figure out which parameters are going to result in the fastest inference.

16:35Because of that forward-deployed engineering function, And especially, I think, because that function exists within the engineering org. There is a culture throughout the entire engineering org of being customer facing, where it's very common for engineers from other teams, like infrastructure and model performance and core product, to go directly to the customer, ask them questions, get feedback on the product, help them get unblocked on tricky issues, and then take that feedback and work it into the product. So I think that's the first sort of big second skill for a lot of engineers is being great at both the building piece and the sort of one-on-one customer interaction, which especially when you're working with highly technical customers can definitely be a lot more comfortable than say like speaking on stage or doing a podcast, writing a tweet thread.

17:33The other way to look at potentially a two parts of depth, two legs of depth would be to say, okay, so you need to be an expert in AI technologies and the models and the harnesses and their capabilities and in one piece of traditional software engineering. So it's AI and CUDA kernel optimization, AI and keeping the Kubernetes cluster alive, AI and billing infrastructure. There's got to be some additional depth beyond just the AI stack in order to sort of match that domain expertise with the acceleration of the tools rather than just kind of everyone being a slop cannon. everyone being a slop can it's like obviously about doubling down on your specializations like if content or something else is not your particular strategy that doesn't have to be part of your double t strategy um and i also like to how you you cut out the importance of like the before you get into the t's when you're talking about the generalist is stretching across um the importance of like really um creating that that solid highway across your whole organization so So that those generalists that that designer can ship can really easily go through the pathways to do it.

18:48That's like actually key because it's like just because you have the skill doesn't mean you have the organizational throughput to execute on it. They're like those are two different problems. Also, too, like when you're picking your specialization, I like how you called out like it's usually like AI and something else. AI and this AI and that. It's oftentimes these folks who are deep domain experts in something in a pre-existing way or have accumulated that knowledge really quickly through working with other tools or methods. And then they pair that with, obviously, the ability to be agentic and to distribute those gains to other people.

19:25I think that's really the key unlock is being able to then distribute, like you said, the gains, which is what then AI is enabling them to do. I'm curious from my own personal perspective, because, you know, you wrote a book, Inference Engineering, and it's during a time when the industry is changing so, so much. Like here on Dev Interrupted, we have a hard time even keeping our news segments fresh from week to week from when we record them and then when we publish them because the space moves so fast. So I want to learn more about like how you went into writing Inference Engineering, what folks can learn from it and how you how you went to set out to write a book during an era where like you know everything is changing so rapidly you know in marketing we have an idea of like evergreen content and you know publishing something on on trees is definitely putting your your flag in the ground and saying hey this is going to be some evergreen content i was in a big rush to get this thing written and published because i felt like okay maybe this is going to have a 12 to 18 month shelf life from when I put it out.

20:34I've seen a few changes. Obviously, you know, my book talks about Deep Seek 3 when we're now on Deep Seek 4. It talks about Kimi 2.5 instead of 2.7. There's, you know, things here and there that are updating. You know, my section on quantization doesn't have anything on polar quant. There's, of course, progress in this industry, but I've been working at Base 10 on inference for more than four years now. And I've seen, especially over the last year, the stack become a little bit more solid. The foundation is there in terms of CUDA through PyTorch, through your various options for inference engines like VLLM, SGLang, TensorRT, LLM, Dynamo, and some of the orchestration stuff on top of that.

21:26We're seeing some standardization in the cloud providers and in the neoclouds and sort of the way that they present GPU compute, the way that compute can be purchased, allocated, scheduled, controlled, spun up and down. So while there is a lot of change in this industry, we're getting to the point where many of the fundamentals are becoming settled, at least when it comes to inference. I think that training, the training stack is a little bit less settled right now. The sort of agent orchestration stack or the harness stack is very, very up in the air. There's still a lot of questions about what that's going to look like long term.

22:07Maybe that's kind of more like the JavaScript framework of the AI industry, where there's just kind of like a new framework every couple months that people are switching to. And eventually, you're sort of like angulars and views and reacts of the industry sort of come out of it. but for inference particularly there is a good deal of solidity in in the fundamentals of the space and i think that there's a lot of opportunity though still to sort of push the envelope on inference like when we publish blog posts we're talking about being 20 or 40 faster like two or four x faster like if all the low-hanging fruit was gone we'd be talking about our two percent and our 4 % optimizations.

22:49That suggests to me that we need a lot more people working in this space. And so if you can make the fundamentals sort of broadly accessible and easy to learn all in one place, then like engineers can come into this space. And I've seen people get up to speed in like weeks to months and start making sort of frontier contributions in inference. Because again, while like the fundamentals are pretty settled, there is a ton of work to do in the sort of optimization and the last mile delivery of inference to the vertical AI companies, to the new labs who are making their own custom models, and to sort of the world of developers at large in the form of like faster, cheaper, better tokens.

23:32Yes, you made two strategic bets. And going back to the original idea at the beginning of planting the seed, of getting it in the hands of folks. That way they can learn and they can have access to the information. And the second thing is, you know, you bet on primitives. Like what you said, there's some core underlying parts of it that are solid that you can talk about that aren't going to be rapidly shifting without otherwise changing the entire definition of the space anyways. In which case, having to go through and update the Kimi 2.5, the 2.7 is just a custodial task compared to that. So you captured the primitives and made them accessible more broadly.

24:09You broke them down into a language that wasn't going to immediately get outdated, going back to your thesis around it being evergreen and, you know, fighting to make it last as long as possible once it left your hands. I want to learn a little bit more, too, from you about how, within the book, how you think about equipping folks to work with inference as traditional software engineers, as people coming from other fields. What are some of the things, now we're kind of getting into maybe the content of the book a little bit. Like what's like your biggest takeaway that you really want engineers who pick it up and open it to really be able to extract and run with?

24:44My thinking on the book is that like any given engineer with a bit of experience who picks it up is probably going to know one or two chapters better than I do. If someone's an SRE and they read chapter seven about production, they're going to be like, well, what is this? Influence for babies? You know, if someone is an ML researcher and they read chapter two about the architecture of these models, they'll be like, yeah, bro, I get it. It's matrices and you multiply them together. Like, we've been doing this for a decade. What's up? But I think that the interesting part of inference is the absolute breadth of the stack, where you have to go from reasoning about the architecture of a set of neural networks to thinking about the specs of your GPU to thinking about an entire stack of technologies from CUDA to PyTorch to an inference engine to a bunch of new research optimizations that you're applying in each of these technologies to modifying that depending on your modality.

25:41and then shipping that on scalable infrastructure. There's distributed systems problems here. There's sort of competitive coding style algorithm optimization problems here. There's business and economics problems here. There's just a lot to think about. And so my goal with the book was to take one of those engineers who has the deep expertise in one part of the stack and expose them to the ideas of everything else that's going on and how all these pieces fit together. Because ultimately, there is no one silver bullet for inference. You can't just get some GPUs, take an inference engine, slap the two together, and be like, boom, tokens.

26:22And expect that to scale and to hit frontier performance, to hit three or four nines of uptime. So yeah, that's kind of the goal here, is to deliver the message that inference is a stack, it's a complex problem, it's not just one thing. And so it's like the surface area, like you're saying, it's very broad. There's a lot generally broadly that you can teach and you can share and that you need to. And you're going in writing a book where you know people that are reading it are going to be an expert on you and some of these really broad domains. Because it's impossible to be a better expert at somebody in all of them and to capture that.

26:59And, you know, that's really a really kind of like as a writer, that's like a sensitive and a hard place to put yourself into. but it's also really bold. And it's kind of like a, the kind of the direction that I think a lot of folks need to do in sharing their expertise and their content is about betting on the primitives or presenting it in a way that makes it as broadly accessible for people within like the same problem space. But then also to just really equipping them with the mindset reset that they need. I think we talked about that week after week within our show about how a lot of times within engineering right now, your biggest obstacle is is your assumptions from yesterday, not the technology you're holding in your hands.

27:39And being able to understand what is holding me back and what do I need to invent net new are two totally different questions to be asking. And folks are typically asking only the order why you need net new. And so it's about revisiting all of those fundamental disciplines and principles and going up to them and shaking that SRE on the arm and being like, you know, like, I know you know this better than me, but there are some parts of this that are a little fresh that are still going to probably be helpful for you to be reading. You know, just kind of like tying this back maybe a little bit more to Base 10 as well.

28:12We talked about how your role is educating folks within the space and about making AI more broadly accessible. And that's all part of Base 10's mission of providing an inference platform. I'm curious, like, what is it like working for a company that is working within like the inference platform space that's such like an on the edge kind of domain? like what are some unique things that you see or that y 'all work on as a business that are that are just really might be kind of like really insightful to our audience who's more coming from a traditional engineering org that's making a transformation instead of being a more ai uh centered one i think that one thing i've learned is everyone thinks about influence and the problem of influence differently you know one one thing that's been pretty cool is is actually hearing from developers at companies like NVIDIA and OpenAI and Google Gemini who are on influence teams reading and debating and engaging with this book.

29:08And I'm like, guys, why are you reading this? You're already experts on this topic. But what they tell me is, well, yeah, we know how we do it, but we want to see how everyone else is thinking about it too. We want to get those outside perspectives. I've been very fortunate to see the perspective of hundreds of different companies in terms of the way that they think about inference. And one thing that we're increasingly seeing is this idea of an AI native company where inference is mission critical. What mission critical means is that it's in the revenue path. If your model goes down, your product goes down, you're not making money, your users are churning, they're going over to something else.

29:49And that's a very different thing than a lot of companies maybe earlier in that transformation are seeing where AI products are just kind of a bolt-on to their existing platform rather than the actual foundation of the thing that they're building. I think that when companies make that transition from the add-on to the foundation, they start to experience a ton of pain. Like, wait, why is it so expensive now that I'm doing tokens for thousands or millions of people instead of tokens for a small group of beta users? And why is my app crashing and slow all the time? Because I'm sitting on top of some public model API that's got like one or two nines of uptime.