Pydantic AI with Samuel Colvin

27:48So let's say you're building a research agent that has access to go and query some big SQL database. Now, the obvious thing to do is to go and smash the whole of your schema into the main agent and it now has access to all of the information it might need and it can go and write SQL to query that data. well fine but you've now if you're combining that with like some other significant tasks you've got an awful lot of stuff in your context if you have a tool that is called like run aggregation let's say and it takes a natural language description of the data that it's trying to find and then within it it calls a separate agent which that has access to all of the schema and the database context and examples and stuff like that then one we have a system that's way easy to debug because we can go and run the SQL agent, see when it works and see when it doesn't, and write evals on that in particular.

28:35But two, the main agent that's got to do that, and also a bunch of other tasks, look up some RAG database, worry about memory, access some NoSQL database, blah, blah, blah, blah, blah. It doesn't have to think about that at all. It just gets this plain text tool, describe the data that you're looking for, bit of context on the kinds of attributes you like. and you've gone down from thousands of tokens of context for SQL down to a few hundred or tens even to describe that like get data endpoint. Okay. Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge and is simplifying car shopping using self-reflection and layered reasoning with live API checks.

29:18It doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade-in value. Advanced, intuitive, and deployed. That's how they stack. That's technology at Capital One. We're going to move on to graph theory, or graph theory meets AI, if you want to call it that. I think just did some sort of pre-research, obviously, as I do before all interviews. I think you've talked about the concept of graph theory meets AI, if you like. So I believe there's concept and whilst I'm not in this domain so audience forgive me on this one but DAGS directed acyclic graphs could you talk to us a bit about that and sort of how this is sort of looking at the process by which an agent might move along its kind of steps I think in theory one is kind of quite linear and the other is you can kind of have a cycle and I believe pedantically I kind of prefers one approach over the other like I say I think the jury is out to some extent on graphs and their use.

30:16I mean, I think most people, when they talk about DAGs, they mean effectively a graph with some dependencies that like relate by node. The graphs that you end up building, if you're using an LLM are quite often cyclic, as in there's no reason why you can't have cycles in there. We have Pydantic Graph, which is part of Pydantic AI. It's used under the hood by agents. I'm a bit torn on how valuable they are. Someone said to me recently that the most useful thing that graphs do is make people who want graphs happy. And that is a very clear definition of why a graph is useful. How much value are they after that?

30:49I think not that much. I think that to be rude for a moment about a capacitor, I think Langchain had taken a lot of heat for Langchain and how it didn't have any functionality. They chose to go and build Langgraph because that seemed like the right thing to do. And bluntly, they didn't understand how to do durable execution. So they had graphs as a way of snapshotting. And now they're stuck saying graphs are the right way of doing it because they can't go and build a third library that's their new way of doing it. So what's happened, I mentioned this earlier, that we were able to adopt the new way of doing things.

31:17One regard in which I have sympathy for them is that since they released LandGraph, everyone else, Anthropic, OpenAI, Google, us, and a bunch of other agent frameworks have all centered on this model of agents of LLMs calling tools in the loop, which are a very powerful primitive. Now, you can implement that with LandGraph, but it's a lot simpler just to go and use our agent or even OpenAI agent's agent implementation. There's a reason that one is so similar to ours, which is it's, I'll say, inspired by our agent implementation. So the other thing that LandGraph lets you do is snapshot at the end of each node.

31:49And so if something fails, you can go back to that point. Now that works, but if you want to have that and parallel node execution, so run multiple different nodes in parallel, you basically have to abandon type safety completely. You have to manually check that your data is consistent. Our approach is different. we support durable execution frameworks like temporal deboss and we have a bunch of others coming soon and they let you get the like durable execution this idea of an agent that can run for minutes or hours and resume from basically pick up from where it left off if it stops or if you get errors which i think is a much much more powerful way of getting the like longevity part of long-running agents but i was giving a talk yesterday on the temporal and durable execution obviously you can use durable execution with graphs to run your graph within a durable execution framework and now you get that same snapshotting effectively behavior restarting a graph from where you left off it's much more fine-grained it's literally snapshotting at every async call and the code should be much easier to write because you don't have to worry about this like snapshotting that gets in your way i mean just i guess talking sort of slight lame in terms here but if you know if we're talking about failure modes as such and okay something fails and there's this concept of snapshotting but is there a concept of being able to kind of move on even though some part of the chain failed so our graph implementation we have some basic snapshotting i think we will i don't think we'll retire it because people are using it but i think my approach would be durable execution is the way forward like these are solved problems people like temporal but they're not by any means the only one there's a whole space of those companies have done an amazing job of giving you ways of writing what feels like normal procedural Python code.

33:34But if you get a failure, it will automatically be retried. If you want to go and sleep for three weeks before the next task needs to be run, you just sleep for six weeks and it will take care of restarting the process from the right point. That stuff is, I think, really powerful. I think it is a far better solution to the same problems as graphs. So let's talk about the general DevEx. How did you think about that? I mean, I guess, were there any things that you've learned through, especially Pydantic v1.2 and maybe even sort of how you've been thinking about v3 like as to how the devx feels and looks for someone coming into Pydantic AI yeah I mean I think that we know we're battle scarred by introducing the wrong API and having to maintain it for a long time and so we're very careful about the surface area and not just adding in any old thing that someone wants to add I think others who are earlier on in the like arc of maintaining open source perhaps don't necessarily think that way.

34:29I think that the extraordinary powerful thing about code and about good open source libraries is people can go and use them in ways you never thought of, right? Pydantic is used for a myriad of things I never thought of when I started it. And lots of things I, you know, to this day, there are hedge funds, for example, but other, you know, large organizations who do stuff with Pydantic that I had never thought of. And that is, you know, if you build a really powerful tool, the point it is universal enough that people can go and do things you hadn't occurred to you. We want to build an agent framework where we build those fundamental things you don't want to have to go and repeat.

35:00So the very simple thing an agent will do is if you're doing structured data extraction and the model gets the response wrong and you get a validation error, it will return that validation error to the model and say, please try again. That is a very neat, very nice thing that will very often catch intermittent or intermittent bugs. You do not want to have to go and implement that again. It may as well exist in a library where you're sharing that implementation with everyone else but then how you go and implement rag for example is far more opinionated far more context specific far more room to go and like innovate and try doing unusual things and we don't want to get in your way of letting you do that so we're i think yeah strongly of the opinion that we're trying to build the right foundations rather than give you these like high level abstractions that like tell you what to do but constrain you in doing it well you touched on it with in reference to pure Pydantic but when we think about case studies or things you've seen Pydantic AI being used for I guess could you just like pull out some of those that come to top of mind right now in terms of either things that you've just been very impressed by or I would say maybe more interesting the things that even you hadn't thought of Pydantic AI going to be used for I mean I'm trying to think there's someone in our public Slack who's written a coding agent with Pylandic AI, which is a like neat working implementation that like works very nicely.

36:20I think, again, coming back to the log fire and the SQL thing, it's amazing how powerful agents can be if you just hook them up to a SQL connection and let them retry a bunch when they get the SQL wrong. We've done some stuff with Pylandic AI, but I've seen others do it where you're effectively doing data analysis with a SQL tool. The other interesting thing is how much has come and been implemented by the community or from pull from the community. So AGUI integration, AGUI is a protocol for talking to a UI, to a chat interface, basically, but you have rich components. That was implemented by someone else.

36:55But also some of the model implementations are either implemented or maintained or improved by the community and an awful lot of people coming and contributing to it. I think one of the unfortunate things about maintaining open source is you often don't see the most interesting things people are doing with it because those things end up being proprietary. but I hear quite often from people once I found PyDance AI suddenly I had found an agent framework I could actually bear and now that's the only one I will use and I hear that roughly that line from experienced engineers all over the place and that that gives me you know that feels great because that's where I come from right I'm not a cursor type developer you know and start developing with cursor I've spent many years learning it the hard way and there are lots of other people who come from that like deep engineering experience who for whom pydantic ai resonates yeah i've spoken to many especially through this podcast many open source maintainers and i'm not one and i always just assume that they kind of know all the projects that are using their libraries especially sort of the top ones and they're like no i actually have no idea not no idea but it's sort of it's very hard to keep on top of the web of of things that it's being used for state management do you have any sort of recommendations for then where that should be managed if using pydantic ai we We have some neat examples in our demo repo of managing memory with either tools.

38:14So you have a record memory tool and a retrieve memory tool. That works surprisingly well. Or just recording all messages and then doing a bit of work to cut off some older messages. If you have very long-running conversations going on, both of those work fairly well. I think the fundamental, again, I come back to it. I've said it before, but I'll keep saying it. Like we're not trying to give you the high level opinionated here. Is it like the, I know one of the other agent frameworks has like three different memory implementations for short-term memory, long-term memory, contextual memory. It's very unclear what they do under the hood.

38:48We let you do tool calls. We return you structured like Python objects as messages. You go implement the thing you want. If it's a simple demo you want to build, the simple thing will work. If it's a production application where you've got lots of nuance, those prebuilt ones aren't going to work for you anyway. I think we might move a little bit more in the direction of having a like, having support for storing messages easily in a database, basically an abstract base class and a few implementations for that because it's a common enough pattern, it makes sense. And I think we're thinking still about how we'll have embedding support soon.

39:23There's no embedding support in PyDance AI because it honestly hasn't come up. It's one of the most upvoted issues, but it hasn't been a burning need for the most part. is that as in like actually creating the embeddings yeah like generating the embeddings API and once we have that which is relatively simple API do we then go further and have concept of rag and hybrid search or vector search or do we just say yeah we can bet here's an API for generating embeddings how you go and implement the next phase is up to you yeah I mean the other place I mentioned AGUI already we also are about to have support for Vasell AI elements which is another protocol effectively for communicating with chat uis so the principle is you should be able to build a chat ui with really a few lines of javascript or no javascript at all just using a pre-built ui and then you can go and do your innovation within the agent however you like in python and then just to kind of i guess tie a bow on it observability i guess log fire that's the sort of maybe batteries included piece if you want to but again we work really hard to follow open standards so pydantic ai emits standard compliant open telemetry data a couple of us are reasonably involved in the hotel sig for gen ai so we like push hotel to work the right way for gen ai and then we support that and so at my last count there are about 13 different observability platforms that support pydantic ai one way or another we obviously think logfire is the best of those but unlike some of our competitors we're not trying to use a proprietary protocol to kind of lock you into using our one we think that like we'll win because we have the best agent framework and the best observability platform and we can kind of guarantee they work well together but we have we're not trying to stop bringing a different agent framework or bringing a different observability platform yeah and i definitely see that trend with a whole bunch of platforms where ultimately they're they are open source and they have maybe different sort of arms but the key i would say part of their success is just maintain that thing as an open standard and if somebody wants to swap out a bit that's completely fine but at the same time you've got you know three to four bits of the ecosystem that you still know will work well together if you just want to kind of default to something i think it's valuable if you're an enterprise that you can come and you have one company you can come and shout out when the two don't work well together that's the powerful bit of what we have is you know we control an awful lot of the stack right as in people on our team although they're not directly part of Pydantic, Marcelo maintains Starlit and UVicorn, which is basically the modern networking stack for Python.

41:52So from Pydantic to Starlit to UVicorn to Pydantic AI through Logfire SDK to Logfire, most of those bits are literally under the Pydantic umbrella. And even if they're not, we're pretty involved in the ecosystem. And that is, I think, where companies of all sizes with some engineering taste, but particularly enterprises, find the value in the one solution or the one point of contact for many different solutions. Yeah, absolutely. Being on the side shouted out on that basis, but that's a good place to be. So we're going to hear about something kind of exciting in a minute, I believe, as a new product in the wings.

42:25But just to kind of wrap up this, again, I'm just the voice of the audience here in terms of LLM gateways. Yes, so we're about to release Pynastic AI Gateway. I think by the time this goes out, it should be launched. It's in particular from Enterprise. It's a feature we just hear like immediate need for like it's the hair on fire problem right now. There seems to be no good go to solution. I mean, it makes sense, right? You're a financial services company. You're expecting to spend five million a month on OpenAI. Are you going to give everyone in the company an OpenAI key where technically they can spend five million dollars and then some internally something running overnight and you've spent a big chunk of that?

43:05Obviously, you're not. And you want observability into what's going on. And that's if you only have one model. Now, what if we're doing some research and we have Anthropic, Gemini, Mistrial, Grok going on? Like, it obviously makes sense to have a single platform to manage those things. And once you have that platform, it's a very useful place to do a number of things. So whether that be caching or security or fallback. And so we just kept speaking to enterprises who didn't have a good solution for this, but needed it. And so, yeah, we're about to release Panasonic AI Gateway. it will obviously have a very nice, easy integration with Pydantic AI.

43:42So you can basically set gateway and then the provider and the model and your one API key will then let you connect to all of those different models without having to go and put a credit card into each of them if you're getting started. And yeah, we'll let you use all the big models through one gateway. The initial launch, we won't have many of the like shinier features, but they will come very soon afterwards, the caching and the fallback and the security stuff. Very exciting. Yeah. And then the actual gateway itself is open source, but they're like console, the platform for managing it is closed source and that we will sell.

44:15And part of it comes back to like, it's all very well, but like we can make the process of getting going with building with LLMs incredibly simple. But at the moment there is this barrier that a lot of the LLM providers are just like, their platforms are not that easy to use and understand. And we want a really nice way of allowing developers as they get started to go from zero to I have an app running with an agent in it and I have observability very, very quickly. And that makes complete sense. And obviously one of the neat things we can do in the gateway is we can emit open telemetry from within the gateway.

44:47So if you're a large organization and you desperately want, you definitely want to record all prompts and you want to look for phishing or for prompt injection, you can do all of that stuff in the gateway with Logfire, which is why they kind of play well together. Yeah. And yeah, just to call it, so we're recording this the third week of October I believe it's sort of end of end of October is the release on that one I've got a big night ahead of me because we were hoping to get to the private beta tomorrow I think it's now going to be Monday and then yeah hopefully 31st of October we'll do the public announcement and let you sign up put a credit card in if you want to use the models we resell or bring your own key if you want to like put your own key in and that'll be free yeah so yeah just to call out thank you so much for coming on given that this is your evening over on the west coast and as you just said you've got a big night ahead of you i misheard you saying you've got a big night ahead of you at the pub but you're about to say public so so uh yeah thank you for making the time i thought it might be nice just to kind of round out with a little bit of hacker news feedback actually because i think it's kind of fun when like people have given andory there's no kind of gotchas here this is just i just want to i think it's kind of interesting you know six months ago someone said you know i found that pyda into ai framework strikes a perfect balance between control and abstraction?

45:59What would you say to that? I think that's exactly the kind of how we've tried to think about it. In particular, I mean, I remember speaking to people at the beginning of this year who would say, I don't need an agent framework. I don't want all those abstractions. The one thing that I want is the model agnosticism and be able to plug into any of the big models, not have to go and use the OpenAI or Anthropic or Google SDK and then be stuck to them. And so to some extent, we tried to build it as model agnosticism without too much on top of it. We've added a little bit, but it's all opt-in and fundamentally the agent is pretty simple some pretty minimal behavior on top of the standard LLM calls but with like with that nice unification so you can switch model in very quickly.

46:41One more yeah just I've been building an integration with Pydantic AI and the experience has been great questions usually get answered within a few hours and the team is super responsive and supportive for external contributors. Yeah I mean I think we take that very seriously we care about the kind of response rate on GitHub, even on the open source or replying on Slack. Like one of us will reply to your message on public Slack almost always within an hour or two in most time zones. We've been there, we're developers, right? Like I'm writing code most of my day still, luckily, maybe I should be doing more sales, but hey, like, and we care about that stuff and are like, because we are ourselves open source developers, that is like one of the things we care about most.

47:20We had a sales call earlier with someone from Enterprise and they were compelled by what we showed them in Logify but really the reason they came to us was the solution they were using before had 290 something open pull requests that no one ever responded and that was the thing that had actually driven them to be like what else can we use and came to us so I think that like responsiveness and engagement with the community is a big part of what makes us different Awesome, yeah and obviously I said no gotchas but also why not pull out something that maybe there's someone in the audience that is saying this in their head anyway and there's always going to be people that aren't just throwing all positive comments but someone saying i really wish pydantic invested in pydantic instead of some ai api wrapper fair and i think we probably one of the reasons we haven't been that much recently is when we made new releases of pydantic mostly people were annoyed that we broke things because we fixed someone else's thing and so pydantic is a big established library we are now david hewitt as i said is working an awful lot on new stuff within Pydantic.

48:19You will see big new features come out. The other thing, though, to say is go and look at the top 100 most downloaded Python packages. Look for eponymous companies in that list. You will see four companies. You will see Google, Amazon, Microsoft, and Pydantic. If you look in the top 25, you will see, I think, only Amazon, maybe Google, and us. Now, per head, per dollar of however you want to measure it, we are one of the most impactful companies in terms of how much open source we do. But we have to make money, right? We're a startup. We've been given money by some big VCs to go out there and make a profit.

48:54We're not a charity. I find there was someone who worked, hilariously worked in the CTO's office at Google, who said, oh, I'm not sure about Pydantic. Now they've raised money. I'm not really sure about these open source projects that are trying to make a profit. Saying the same thing about Astral and Ruff. He ended up deleting his comment because he obviously realized he was on the wrong side of history on that one. But like, there are a certain number of like, millionaire communists in particularly in California, who would love open source to be, you know, entirely benevolent, but they get paid an awful lot of money to buy big tech companies who don't particularly contribute back.

49:30So we subscribe to the pledge, open source pledge. So we give$2 ,000 per year per developer to yet to open source. That's above and beyond all the open source we do. We think that stuff really matters. I see very few of the other bigger companies doing that. And I suspect the person who made that comment doesn't work for a company who does that. So I think we do enough for open source. I'm pretty proud of what we do. And I'm pretty robust in rebutting anyone claiming that we don't do enough. Yeah, absolutely. I wanted to bring that out because I very much stand with you on that one, both seeing what Pydantic does as well as many other companies that get the same heat just because they switch focus a little bit to something that happens to have a commercial arm to it and somebody gets all rubbed up the wrong way and i'll tell you that if in the unlikely event pydantic doesn't work out and we end up shutting the company down or being acquired by someone that people don't like we'll get a lot more heat for it then well the reason we're trying to make money is so that those things don't happen right because maybe i'm not as nice as guido i'm not going to maintain and obviously pydantic is nothing like as big as python itself but like I am not going to go and maintain like Padantic on my own for the rest of my life being paid.

50:40Like I was probably getting towards$40 ,000 a year when I was maintaining it on my own. That's not me. I'm not going to do that for the rest of my life. So these projects, the company needs to make money to support both our open source and the wider ecosystem. Absolutely. Just kind of talking about, I guess, the company today, how many people do you have in the team now? And are you hiring? Is this a good place to plug any hiring or? we will be hiring a little bit over the next few months the blunt truth is that we like i think that it's got easier and easier to apply for every single job you can think of and the number of pretty poor applications we get now whenever we put a job ad up we're being more and more like targeted and looking for particular people so and unfortunately we're a small team we don't have capacity for like junior people or interns in general and so i'm afraid if you email me being like I'm a big fan I'd love to do an internship with you when I finish university unfortunately I'm I'll try and get back to you but the answer will be no we're always looking for like really bright experienced engineers who have a proven track record in open source in Python Rust or TypeScript but if you haven't got a like pretty impressive record in that direction we probably aren't gonna it's probably not gonna work with us yeah but yeah we will be hiring we put everything on social media and would love you to apply if you match the conditions on the application.

51:59But please don't email me. Our first rule of anyone that we hire is they need to follow the hiring rules, which start with email careers at, not Samuel. Yep, I've been there as both a past life owner of a company where developers are applying as well as them being a CTO and having people sort of try and get around the process. And I just politely say, please follow the process. You're not going to get anywhere just by emailing me. Unfortunately, all these stories from back in the day oh i guess the email address is just nonsense now or it should be there's a process usually for a reason so i just add one thing though i mean a lot of the great people we've hired have done significant stuff in open source now don't think that as a three weeks into your python career you can go and like start using cursor to generate pull requests on pyantic and we'll go and hire you for loads of money but like we have found some incredibly talented people who probably would have been overlooked by bigger companies by finding people who have like been working away maintaining awesome python libraries for a long time and so i do think that if you can get into open source if you can go and do the hard yards of building up a reputation in open source both us and many other companies will hire you so i do think but it's not a quick fix it's not like you can't just go and like buy the right crypto coin and suddenly be a millionaire right it takes you five years of learning how long does it take to get 10 years of experience it takes 10 years and you can't really accelerate that arguably that ais make that even harder because the discipline you need to learn it yourself when an LLM will probably get something approximately right instead is it's getting harder yeah it's something we've discussed a few times we have an SED news monthly and Sean and I have discussed this a few times just sort of what is the where's the tipping point between people coming in now as developers and not to in any way dissuade people or but yeah there is no point in just firing up cursor and then thinking that you're a programmer like it doesn't work that way and it's not you know we're not so we're the old guard saying you know you can't come into the club it's not that whatsoever it's just that it's engineering is a fundamental there's all these principles and concepts that just seem helps if you understand them before the code that's written that you're then reviewing or editing you know it's a bit like you know whether you're flying a plane or driving a truck like sure you can put it into cruise control when you're going down the motorway but when you get to that narrow lane where you need to reverse around a corner like you still need those expertise and in the the end sure i use code a whole lot and it does lots of things for me that i don't want to have to go and write all of those react components but like when it runs into some weird bug when you need to set up exactly how you're gonna share type safety between the front end and the back end that still requires me and i don't think there was a and maybe we're about to agi we're all redundant but like until that point however little time it is you still fundamentally need a truck driver in that truck to reverse around the corner even if they spend a bunch of their time in cruise control.

54:48And the same is true with code. And in some ways, it's a multiplier, right? We all have the resources of a team lead, but you still need the knowledge of a team lead to be able to deploy that team, whether it is human or AI, effectively. Yeah, absolutely. So yeah, just kind of final closing question. And this is more like a personal one to you, I guess, which is more just inspiration. Like, I guess, have you got any people, whether it's in the community right now, or even, you know, living or dead, who's kind of like inspired you and does inspire you, I guess. I have to say Guido von Rossum, the creator of Python.

55:23I've met Guido a number of times now. He can be reasonably blunt, but he's always friendly and fun to talk to. And he is so humble. I mean, I have seen him walk into rooms where no one knows who he is and just stand there and like listen to what's going on. He so often describes himself as the author of Pep482, not the like BDFL and the creator of Python. And I'm so impressed by what he has built in Python, both as a community and as a language. He got so much stuff right long before his time in terms of realizing that it was something that programming languages are for humans to use, not for computers to use.

55:56And prioritizing the human bit. And I think the success of Python, you look at every other successful language, with the possible exception of Rust, which I'm also a big fan of, they've all been anointed somehow. They've all had a reason why they're going to win. JavaScript had the browser. Go had Google. C Sharp had Microsoft, et cetera, et cetera. python had one random dutch guy and an amazing community and i ever impressed by what python has become and you look now right like in ai sure some people say typescript might take over but like it's typescript and python it's a two horse race and python's doing amazingly and i i'm like proud to be a part of that community and i've had a little bit of impact on it over the years yeah i mean i have to have to say i am more on the typescript side but that's not because i dislike python it's just that's just the route i took in life and i'm really impressed by how Python has, as you say, it's the leader clearly in AI programming.

56:46And it's been fascinating to kind of watch that. Well, look, it's been absolutely a pleasure to have you and feel very lucky to have you on SE Daily, especially as you've called out in an episode, this is sort of towards the end of your night, and it's not finished yet by any means. So yeah, thank you so much for coming on. And obviously, we look forward to the PyDatic AI Gateway release, which is probably going to be out by the time that this is airing. So yeah, thanks so much. No problem. Thanks so much for having me. It's been a pleasure.