The best AI agents are simpler than you think | Ben Tannyhill, LangChain

21:08They're not seeing these, but it gives us kind of like a second pass or like another quick check on the issues that are being created. Do they look right? Are they written appropriately? Are they high quality? In a way that is a little bit more difficult to grasp from these benchmark evals that we're running on Harbor.

21:27Ben Tannyhill:When we have agent running in production in real life, what metrics do we track to get a sense of how it's doing? We track a ton and we have so many cool tools that make this really easy for us. We have like our actual database tables that can inform us like what customers have Engine actively running, who's turned it on, how frequently is it running, how many traces is it ingesting each one of those runs. Those actual like databases are very helpful for us to just have a high level view of who's using the tool. Engine actually reports back on a lot of its metrics as well. Like I just mentioned that we'll be able to see the number of traces that are ingested by Engine.

22:08There are a variety of things that Engine will kind of spin back to us and inform us. Like X amount of traces were analyzed. It gives us insight into the size of those traces that it's ingesting, the latency of its own runs. Like we start to understand a little bit more about how Engine is running for the different customers that we work with. So that's kind of like the Engine internal stats we get to see as well. We also do a lot of user tracking to understand how users are engaging with the actual issues. We've talked a ton about the agent and like the engine, behind engine, right? Like how it's all investigating and servicing these issues.

22:42But there's also this entire product side where users are interacting with those issues and making adjustments to the fixes that it proposes. And so we have lots of user analytics there as well. But I mentioned that we have like so many awesome tools. We use the Hex agent a ton to spin up like dashboard, new dashboards every day for the things that we're looking at. And it is, in my mind, like one of the agents I use most frequently.

23:05Ben Tannyhill:One of the most interesting things about Engine, I think, is the fact that it's like an ambient style agent. It just runs in the background, runs on a schedule. I think that also makes for really interesting UI, UX considerations. When do you bring the human in? How does the human interact with this agent that's running the background? Could you speak a little bit about how you think about UI, UX for Engine? any evolutions that have taken place and what that like human in the loop looks like for these ambient style agents? The very first version of engine, like the simplest form was that it would just spin up PRs and it would just create PRs for different issues that it found.

23:41And so many of those PRs were bad and it suddenly was just super noisy, right? Like no one wants to deal with a huge list of PRs that they have to go sort through and understand the commits. No matter how good your description of that PR is, like it's just too much. And so very quickly, this concept of like an inbox became like a good option and one that we started kind of socializing with our early testers and customers. And it just made sense because there are there are these different clustered problems that engine will be able to identify that have their own history. And so this inbox gives you a very clear diagnosis of what problem has been encountered, what exactly is going on, as well as a look into how frequently has this been going on?

24:22Is this a long-term thing that you've seen for a month? Is it happening to every one of your traces or some small subset of that? But all of that is kind of like a new alert that a user can interact with. But with each one of those alerts, rather than just something to be made aware of, there's a series of actions that can be taken. And so this is something that we're like every day making tweaks to is what are the optimal steps for a user to take? What are the easiest ways for a user to take a set of problematic traces and come out with a better agent. And like, yeah, how can we optimize that flow from seeing and understanding this is going wrong in my agent, this is the exact fix I need to make, and this is how to deploy it or test it most effectively.

25:05So that process has been a lot. It's honestly been a challenge to go from like just surfacing a problem or a PR to informing user in the right way and encouraging them to fix it in the right way. The last thing I'll note on that as well, that has been also difficult to kind of bake into the equation here. A user has so many unique preferences and so many insights into the issues and into the agent itself. We often find that things that are real issues that we can say are objectively problems with the agent are just unimportant to an issue. They're just unimportant to a human. And so that team might not care about times when the context has exploded or they might not care about these minor hallucinations.

25:46And so that process of saying, yes, this is a real problem. I don't care enough to solve it. let's move past it has also been interesting as we've tried to like make engine learn more and

25:55Ben Tannyhill:more from a user one of the big things that we try to talk with customers about when they're building agents is try to try to figure out the the bodies of work where you can do a ton of work but there's still like a human involved at the end and i think we practice what we preach a little bit with engine because i think it does do all this work but there's still human involved before it opens up that pr before it adds an evaluate or before it adds a data set but it but but but i think we can have it do a lot of this work. You mentioned at the end kind of like there's all these issues that people might not care about for whatever reason.

26:26Ben Tannyhill:I imagine it would be pretty annoying if we kept on bringing up the same issues over and over to users. How do you think about memory in Engine? Within Engine specifically, I mentioned at the beginning that there's an opportunity for a user to kind of describe things that they're interested in right up front before you know engine even does its first run that might be like we've talked about you know specific categories that are important to the user that might also be like important nodes of information that engine should have about my agent like hey it calls this separate sub-agent it's important for you to understand the distinction between the two all sorts of these different preferences or understandings that the user can express to our agent the way that we handle memory is through what we call this agent overview document.

27:12It basically is like a Claude MD or an agent's MD file that Engine is able to reference, and it does reference on every single run to understand, has the user's preferences changed? Are their interests different? Has the structure of the agent been adjusted? And so that's referenced on each run. It's updated on each run. It's updated as a user interacts with these different issues. The process of creating a memory storage and updating it is super simple. it's very hard to do so in a context-efficient way. It's also very hard to, like the user interface, to encourage the kind of extraction of important information from a user to add to that memory.

27:51Like, obviously, Engine will not work very well for a certain subset of customers right out of the box, and it will work really poorly with, you know, like a muddled memory that has all sorts of things they don't care about. But trying to really understand what are the things that a user cares about is super, super challenging.

28:08Ben Tannyhill:Do we give people the ability to leave natural language feedback on issues? Yeah, we do. Every time they ignore or they resolve or they say that this is not a high priority, this is a low priority, they have the option to add in like, this is low priority, don't tell me about this in the future. Yeah, I think that's great. And I think capturing all of that and then passing it. The thing that I think is really interesting about memory with engine actually is generally we see that there's kind of like two different ways that you could do memory for agents. One, you could have the agent as it's running and interacting with the user basically update its own memory.

28:42Ben Tannyhill:And then the other way you could do it is you could have a process that runs in the background and looks at all the interactions it's had recently and then updates its memory kind of like in the background. Kind of combine them here because there's never a place where the human's directly interacting with engine. They're leaving feedback, but then that gets picked up on a background run. And so it's kind of something that's twisted my mind a little bit where it's this weird hybrid of like, yes, the engine is doing it. Engine can update its agent overview whenever it does a run and it looks at all this feedback and it has that.

29:14Ben Tannyhill:But it is like a background process of sorts. So I thought that was kind of like an interesting middle ground for some of these things. We talked about cost a little bit earlier on. And yeah, I mean, I imagine running an engine over a ton of traces can get expensive. How are we making sure we don't make LangChan go bankrupt? Yeah, it is very hard. And I often get messages from you or from our head of engineering letting me know that our bills for our inferences are going crazy. It's something that we're always working on improving. It has become one of the things that has been fun to experiment on.

29:53in the process of improving the agent, there are all sorts of these different obstacles or different bottlenecks that we are able to identify and then work on improving. If we were to run like a state-of-the-art Opus model to do everything that the agent does, a screener is using Opus or a state-of-the-art model from OpenAI, if it's running on one of these high-powered models, it's gonna run up a huge bill for us. And so the process of determining, again, going back to like model selection, determining which models can be used for different tasks has been very interesting. Where can we use a less competent, much cheaper model to run this process?

30:28And that all goes back to this kind of like process of running evals where we have a hypothesis, we might do investigation to understand that this portion of the run is responsible for 33 % of our total cost. How can we reduce that specific part of the run? Can we switch out elements of that to a different model? That kind of investigation is super interesting. And then it's a ton of experimenting and hill climbing against our evals with the different changes that we've made.

30:52Ben Tannyhill:practically speaking what what models are we using today for different parts of the agent right now we use honestly a cocktail of different models we use anthropic models we use opus as a main agent we also use uh models from open ai like 5.5 we use haiku models for a lot of our our screeners or our verifiers we've swapped in gemini models at different times we've also explored a lot of different open source models for especially for these less challenging tasks like for the screener sub agents that are doing just like a very quick analysis of a trace. But it is it is really changing frequently.

31:30Ben Tannyhill:We launched engine I think publicly about a month ago at this point. What did what did the rollout of engine look like from kind of like initial idea to to now I guess and like who have we launched it to how have we how have we incorporated feedback What does that kind of like rollout of an agent look like? It's really just been like a slow expansion of customers that have been onboarded to this new feature. And, you know, we've had really awesome design partners early on that were excited about the vision and very trusting in a very early, very bad version of what the agent was. the earliest version of the agent we call it we called it forge back then this kind of uh early prototype was running with some of our customers like credit genie or unify um and i i mentioned before that it was putting up these these prs and it was very easy to get feedback because these teams are super familiar with their agents and the the kinds of errors that they should be looking out for these teams are agent engineers that are used to looking through langsmith traces and identifying problems and then putting out fixes for those exact problems that they've identified.

32:39And so the feedback loop was really strong, especially when it comes to the agent's ability to find traces that were meaningful and to put up good fixes for them. But it really has just been this process of expanding the group of users that are using it. Sometime in, I believe it was March, we started rolling out the earliest version of Engine to maybe five to 10 customers. We had kind of like an expanded beta, private beta, where we had more customers using it. We had suddenly an interface within LangSmin that they could interact with. Again, low quality, but something that was able to get us a lot of feedback.

33:13And then, just at Interrupt, we launched our more available version of Engine. And then we've had this deluge of feedback that has come in.

33:21Ben Tannyhill:I think one of the things that we emphasize as a company is shipping quickly and then iterating rapidly after that. And I think we did a pretty good job with that here i remember uh you and and palash and some of the team members shipped like a version of this so fast and it was incredible to see as you said we basically grew the blasters we tried it out on internal agents and then we tried it out with like two design partners and then i think by the time we launched it we had like 20 or so design partners using it and then and then it went to you so i think like yeah we i think we definitely practiced that like launch quickly iterate rapidly uh philosophy that we have and i think that what has made that so possible is like our ability to to write and ship code is extremely fast with coding agents now like we really could be making meaningful changes to the agent in a matter of hours so we hear feedback in the morning from one of our design partners make a quick change run engine again for them and get additional feedback by the afternoon and so the the pace has been kind of frightening let's maybe talk about that for a little bit like what does the team building engine look like it's very different from what I'm used to as a product manager.

34:27I'm used to like a team of product engineers and some kind of, you know, architect really that's doing more of like the infra on the team. We still obviously have a lot of product engineering to be done. We have an interface within Langsmith. There's obviously a lot of components of infra as we use our Langsmith deployments product. We have these sandboxes that that engine utilizes, but there's also this kind of newer branch where we have our agent engineers on the team. There's a very different process for the two teams right we have it's not quite agile anymore with the speed that we're working with with new coding agents like it's extremely fast but it involves us saying here's a very specific product outcome that we want we're going to do some kind of like early design doc or or spike on that and then we'll implement that very quickly now with coding agents the process for like the second part of our team this applied agent engineering team is very different where as we talking about it's more of like this experimentation flow where we are sitting down together and coming up with you know wow the agent is taking a ton of tokens to be to running this specific process or this sub-agent is taking forever what are some potential ways that we could reduce that and coming up with these different hypotheses and then throughout the course of a week usually we'll say like these are the hypotheses we want to implement let's test them out we run our evals against them and then usually the latter half of the week we're implementing something like that so the flows and kind of like the the style of engineering is so different across the two teams and then there's obviously places where they interlap where they overlap so we have other

35:54Ben Tannyhill:ai products in langsmiths as well um and i think in some ways engine is an evolution of them but then also in other ways we still need to figure out how to make them work well together maybe i can talk a little bit about how i view some of the evolution bits and then i'd be really curious to hear your take on A, anything that I missed, but then B, what the future looks like. So I think two of the previous AI experiences we had and still have in LinkSmith are Insights and Poly. And so Insights, it basically clusters, it runs over traces similar to Engine, it clusters them, it does two levels of hierarchical clustering, and basically shows you insights, different trends of what's happening in your trace data.

36:37Ben Tannyhill:Poly is a chatbot that sits inside links with them. I'm actually not sure which one was launched. Do you know which one was launched first? I have no idea. I think it was before my time here, actually. I think Insights was launched first, even though the chat's the more basic thing. But the chat, we really didn't want to do kind of just a general chatbot. And so we tried to focus on places where it would provide value. And two of those places were within the tracing project. It was within a trace and within a thread, and you could basically ask it what was going wrong. And then the other place was in a playground, and you could ask it to fix the prompt that was there.

37:07Ben Tannyhill:And in some ways, I kind of view Engine as combining the best parts of both of them. So Insights ran over all your traces. Interesting, but not actionable. Like in order to take an action, you'd have to think about what to do and then go do it. And then Insights also was kind of broad. It gave you these clusters. Polly was very focused on specific things. And in Playground, you could actually ask it to fix things. And so Engine, I think, kind of combines the best of both worlds. Because one downside of Polly is we never had a mode where you could chat with it and ask it to do like massive trace analysis because that's a hard problem.

37:41Ben Tannyhill:And so Engine, A, it lets you run over all your traces just like insights and solve the pain point of Polly. But then it produces these really actionable things that you can do, which was a pain point of insights, but something that Polly did well. So I kind of view it as bringing the best of both worlds together. I'm curious from your perspective, building out Engine and interacting with these others, will they all just be part of Engine in the future? Are there differences? How do you think about that? there are differences between the three nodes that we've talked about between insights engine and poly i think that there is something really exciting to me about the actionability of engine right now i'm going not just from like understanding my traces but i can now use those to make a change which i think is really cool and i think is like the right direction i still think that there are like components of insights and poly that engine does not do very well for example engine is really good at giving you, you know, very specific issues that I'm going to go fix.

38:37But I actually don't understand that much about the status of my agent from engine. I might look at a list of issues and determine I have 15 issues here. Something is majorly broken. When in reality, they might be smaller issues. And that isn't a great sense of like the health of my agent or its general metrics. And so insights maybe doesn't do that perfectly today, right? It gives us nice clusters of what's, how are users interacting with my agent? But there is this zoomed out view of insights that I think is really important. And I think Engine is lacking that. And so maybe there's like a combination of the two where I can see both a very actionable set of issues that I'm going to resolve, as well as a high-level view of, is everything okay on my agent?

39:16How are people using my agent? What are they asking it? You know, what are general problems that we're seeing or categories of issues that we're seeing? Moving on to poly, like we've talked about how Engine is running on a schedule right now and it doesn't have the interface to necessarily interact with the user in an easy way. But we've also seen times where a user doesn't want to just be fed this list of alerts or issues, and they want to ask questions to Engine like, you know, across my agent, is my latency getting worse? This is something that Engine, with its ability to look through traces, could very easily do with some minor tweaks to it.

39:50But right now, it doesn't allow for it. So I think there's kind of like elements of these other components that we've built in Langsmith today that I think Engine should probably adopt. Or maybe there is a world where they totally all just meld into one agent that runs on top of Langsmith. But I think they all could mesh together nicely.

40:06Ben Tannyhill:Kind of continuing this, you know, we talk about Engine as an agent for agent engineering. What other parts of agent engineering could we help automate or help do with agents in the future, whether it's part of Engine or some other thing? The biggest one that I feel really excited about, and there are a lot of unknowns too, and a lot of challenges that we've talked about and outlined, but right now, Engine, I think, sits more squarely on this monitoring function of an agent engineer's work. It is looking at traces and investigating and clustering them into things that are more easily digestible for an agent engineer.

40:42And so it does that functionality really well. It could probably improve a lot on its ability to make fixes and to build for these agents and to make those agents better, but it does that today. The component that I think Engine is missing today is this testing ability. Right now, Engine will find an issue, service it to a user, alongside a proposed fix. The user, if they're kind of like following standard agent engineering practices, before they implement or merge that fix into production, they're going to ensure that it passes their evals and they're going to run these regression tests to ensure that this is an appropriate fix.

41:17It's not going to, you know, nuke all these other use cases that we've seen previously. With agents, that's super easy to do if you're making a tweak to a prompt. And so right now, Engine will create dataset examples based on those production traces. If a user said, you know, something inappropriate and your agent handled that poorly, it will create a golden dataset example of that input so that your evals can include that in the future. That's really as far as it goes. Right now, it doesn't actually run that experiment. That process of running your agent, not really your agent, but a new version of your agent with this proposed fix against these new evals is very challenging, but something I think is really exciting.

41:58What that would effectively mean is Engine would find issues from your production traces, propose fixes that are tested and kind of like hardened because they have been run against your evals so that we can say with some degree of confidence, Since this phase is a good fix, it will not regress other inputs that you've seen with your agent in the past. So that's a direction I'm super excited about. There are a ton of challenges to that.

42:22Ben Tannyhill:We've talked about this a bit. What are some of the challenges? The first is this question of running your branched version of your agent, right? If engine is going to make a proposed fix, it is going to make an adjustment to your agent. And how do we run that agent? We have to have all of the environment variables. we have to have like the API keys available to us. It's easy to make the change and to make a pull request against a specific agent. It's very difficult to run that agent in some environment. The components that are necessary to run that agent, I think are kind of difficult to scope out.

42:54That's one of the components. The other element that I think is very challenging is that there are a subset of agents that are making read-only tool calls. Like we have an agent internally called ChatLangChain that is like our docs agent and it will ask or it will answer questions about our documentation or about what is Langsmith capable of? What are deep agents? An agent like that is just reading from different data sources and it's not making any updates to those data sources. To run a branched version of that agent is very easy because it can make these tool calls with no real world effect.

43:30It doesn't matter if it calls my production tool call to my documentation. It's not going to make an adjustment. If I'm running a branched version of my agent that makes real world right access tool calls, suddenly I might be modifying my database. I might be sending out emails to a customer and interacting with the real world in a way that is still in a development stage. It's really a process of creating an eval environment for these right access agents. I think it's super challenging. And I actually don't think that a lot of teams that are building agents know exactly the right way to build those environments.

44:04Ben Tannyhill:Yeah, I completely agree. I mean, I think the running evals is really interesting, but you have to have evals first, kind of, unless we help people create evals as well, which I think is also a really interesting direction. You spoke about the process of like recreating these environments to run evals for engine, right? Where we have like these stubs for Langsmith. Maybe you can speak to that. Like, what is that? That feels like the process that customers would have to undergo if they want to run evals as well. Open research question, unfortunately. I think it's really, really hard. And that part of the reason is because there is a lot of domain expertise involved.

44:41Ben Tannyhill:So let's take the chat link chain example. If we wanted to create, let's say this is a real thing. Like we use Mintlify for our docs and we use it in chat link chain, but we didn't want to use it for our evals because we were potentially thinking about doing an RL and we didn't want to do a ton of rollouts and slam the Mintlify servers. So we thought about creating a synthetic environment. And one way to do it would be to, okay, let's point it at the traces. we can get a good sense of like what questions are asked. That's definitely feasible. We can have some sense of what like, we can get a good sense of the tools that are used and the input schemas and the output schemas, totally feasible.

45:15Ben Tannyhill:We can get some sense of the documents and the underlying data, not all of them, but we can sum up it there. And you could imagine pointing a coding agent at a bunch of traces and being like, hey, go create like some synthetic mock server. That's just not the right way to create it. The easiest way to create it is like, hey, we have all the docs in Markdown format in our repo. Like, let's just pull those down, put those on disk, create some synthetic questions by basically taking a document, finding an answer, and then creating a question for that. And then basically use like grep or something as a fake search and use that to mock out the server.

45:48Ben Tannyhill:It's far more reliable. It's just better to do. And so that's like, I don't think a coding agent that was tasked with looking at a bunch of traces could ever come up with that. I mean, and so maybe you just say, hey, like in those scenarios, like, yeah, humans should be involved and do that. But for other scenarios, like we can automate that. And that's more of an open research question. Really hard problem, but I think would be super, super interesting. It's funny because like it's very applicable to engine and making engine better. But the reason that it's applicable to engine is because it's so applicable just to these agent engineers.

46:17Like that process of defining that environment is so, so challenging. And so many of the customers I'll talk to will ask this question. I'm like, wow, cool. That engine proposes a fix, but how does it test the fix? Yeah. And I'll explain like, well, you would have to test it. No, we can't. We don't have a really obvious way to create this environment to run evals in.

46:35Ben Tannyhill:Right now, Engine's mostly used for first-party agents that teams are building. And it suggests a bunch of code fixes and adds evals and adds data sets. I think it's also, but this process of running agent over traces is, I think, a general process. And I think there's other things you can use it for. I have one idea that's top of mind, and then I'd be curious what other ones you have. So one I'd say is for memory. So we released a video earlier today of using Engine to power long-term memory for an agent, specifically using it as like sleep time compute. So we traced all the interactions to Langsmith with just normal tracing.

47:13Ben Tannyhill:And then we ran Engine over all those traces and we pointed it at our context hub. And in our context hub, we had the agent's memory. We had it's like agent.md files and some skills. and then this background process, engine, basically suggested changes to the skill files and to the agents.md and then once those were merged, those could be pulled back in for future runs for the agent. And so in that way, I think you can use engine as memory. What else could you use engine for? Something that we've gotten a lot of requests for as well as for running engine on top of coding agents. And that is, it sounds very similar to like what we do for these bespoke agents that our customers build and then trace to Langsmith.

47:52the difference is that they aren't making modifications to the middleware to the prompting of the agent instead they're making adjustments to skills or uh you know to these these agents md files it is very similar right like at its core it's still the same thing of ingesting and analyzing for issues across different traces clustering those into things that make sense as a group and then proposing a specific fix in this case the fix is maybe not a prompt change or a code adjustment it is just like the creation of a new skill or the adjustment of an existing skill so it's it's i mean it's very very similar but there is a difference to that in the sense that like it needs to be tuned for cloud code traces or for codex traces right it needs to be modified slightly and then it has this uh in a sense it is it is very related to your point on memory right like it's a different kind of file that is that is being output of this it actually brings up another point that i wanted to mention which was that we currently run engine on top of engine traces.

48:51Does it find a lot of stuff? It does. It's super valuable. So, Engine is running on top of traces from an existing agent. Sometimes that's a customer agent, but for this use case that I'm describing, it's our internal agents. Engine, as an agent, produces its own traces. And so we have another Engine that runs on top of those traces. It's super meta. When we first thought about this, it was very early on that we kind of joked about this, and we assumed that it would be crap, and that it wouldn't actually help in any way to find real traces but it's been extremely helpful and it's actually one of it's becoming more and more like one of the primary ways that we find improvements to be made on engine like my my dream is that our team just uses engine for any of the agent engineering so that we can find problems in our production traces and make fixes so it is it is an interesting process but what you're describing of like using memory and and uh using engine for these different things has just you know, brought this to mind that like, engine is kind of just this improvement loop for whatever kind of agent, whether it is a coding agent, whether it is something that relies on memory, whether it is engine itself, or like, it just is this improvement driver.

49:59Ben Tannyhill:Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback or questions to maxagency at langchain.dev. We want to hear from you.