The best AI agents cost less than you think | Eno Reyes, Factory

23:51Eno Reyes:Yeah, and this is, I think, one of those things where in order to be, to give like a super empirical answer, I'd have to like go in and look at the harnesses, sorry, the eval harnesses that they're using. But here's my like gut intuition. The vast majority of the pre-training data, is coming from like a shared corpus. The synthetic pre-training is then built off that shared pre-corpuses just data distribution. The actual individual task set or the RL environments that they're building are being sold by an increasingly limited number of companies. And the overlap in sort of the breadth of what tasks matter is the union is increasing very rapidly.

24:31Eno Reyes:So in other words, almost everyone now is going to have a very elaborate harness, eval harness for code review. And I think that probably there is, like in the latent space of all code review, like a very solid workflow that is just like the best workflow. And so do you think basically like OpenAI just has a better eval harness for code review and that's why their harness is better at it? If I had to guess, they got closer to like the canonical, like global minimum of loss on like code review at this point in time. And what that means is that the workflow they've uncovered, you know, these agents, they, the reasoning traces are not exactly human interpretable.

25:11Eno Reyes:But if you follow the flow of what they do, you know, it's like you watch a human who like reads over some things and then does work versus doesn't read over and does the same work. It's like there's certain things that are human interpretable. And I would imagine that basically what they're finding with all of these things is that a really great model does like a very clever workflow very consistently and and builds the nuance of the task into their workflow. And so we have the benefit as in fact, everyone out there has the benefit of like basically observing this. I have like a spicy take about like models being there's a thing called like the platonic representation hypothesis.

25:51I haven't heard of that. You're going to have to explain it to me. But I love spicy takes. So let's go down this path.

25:54Eno Reyes:Yeah. So basically what it says is when you train multiple models with different architectures on the same data distribution, if they actually converge towards a very shared representation, but just with differing levels of compute. In other words, all learned algorithms might be very similar in the sort of mechanism that they're unfolding, which is matching the data distribution. And it basically how your your model architecture or what you've built primarily impacts how efficient you are at finding that data distribution. And so the sort of end result would be if you take model training all the way to its limit, we're basically building all towards like a platonic representation of all of human knowledge.

26:41Eno Reyes:And that already exists out in the ether. And we're sort of discovering it rather than creating something. And this has another interesting thing of basically meaning that like distillation is effectively unpreventable. Because once you've found the shape of intelligence, you know what it looks like. And so distillation might just be this thing that is not possible to prevent. And that's what's leading. And I think the model providers might know this. And so they're going down the route of regulatory capture in order to prevent basically the inevitable, which is that the shape of intelligence will be uncovered and then everyone will have access to it.

27:16So, okay, so this is something I think about as well. Like, are we building general purpose models? Are we building models that are tuned for a particular harness? And I would argue, I'm curious for your take on this, but like, I would argue a year and a half ago, however long before they started like really post training on these harnesses. Yeah. The models were maybe like more similar than they are today. And it seems like today there's some differences in terms of how they work and like what tools they work best with. But it kind of sounds like what you're saying is like over time, you actually expect those differences to go away as we discover like what this platonic ideal is.

27:47So like, yeah, I'm curious whether you agree with what I said or disagree or I'm getting anything wrong.

27:52Eno Reyes:Yeah, I think, you know, what's funny about this is I see the difference as basically like, and you remember this because I remember we once related to this. It's like you used to change like a comma in the description of like the edit file tool and it would actually change your performance on a benchmark by like 10%. I think OpenAI famously had like a sentence you could add to codex that increased its score on Sweebench by like 30%. And it was like one sentence and it was like basically like don't stop until you're done. I think that you see a lot less of that and a lot more like obvious spikiness on tasks where like some models just can't do it and others are really good at doing it.

28:28Eno Reyes:And I think that the way I would sort of also analogize this is we went from a world where like you had people who just like some were trained to use keyboards and others were trained to use mouse mice and like others were trained with touchpads. And if you gave the other one a keyboard, they just like wouldn't be able to do it. And now you have a world where like all of them are more than competent at using all of those tools. However, like one's a bio PhD and the other is like a math PhD. And so I think that like that, that it's almost like the tool use. And this is the explicit goal of all of the model.

29:02Eno Reyes:Like they want generalizability. I think that they're going to struggle to fight against generalizability because that is effectively the goal of these are like general learning algorithms. So the more they also converge, Like I know that our harness is definitely in the post-training data sets of nearly all of the frontier models. So like the more they generalize across different open harnesses, their own harness, at least the way that they communicate this to us is that they want that generalizability. They want more people's unique harnesses to like also perform well. The question is, is does your harness unlock the workflow that they have decided really matters?

29:39Eno Reyes:Like on, for example, in like bio or in math or in science. Do you use different prompts or tools for different models or families of models in Droid? We still do. And you know what we haven't actually done in a little bit is we haven't rigorously evaluated if bringing them back to a union is ideal yet. Right now, the way our harness like auto optimizes is we actually explicitly tell it to find the tool descriptions that map to the models. One thing, though, that is I think the noise is gone. Like if we like change a couple paragraphs or we change a sentence, we're talking about incremental gains on.

30:18Eno Reyes:We do think we have a good representative benchmark. It could be that we are actually like optimizing a local like like it might not matter. And I do think, though, that different tools for different models does still matter. Yeah, that's what I was going to say. So we have deep agents. We have what we call like harness profiles or things for different models. Yeah. And I think like the biggest thing that we found kind of to your point, like it's not like small things that matter. I think it's like the edit tool is like pretty different from Anthropic and OpenAI. Like one's like, I'm going to mess it up, but like one's like patches and one's like.

30:49Eno Reyes:Yeah, apply patch, like edit string. Exactly. Or string replace, yeah. Yeah. And so like, and that's kind of what I mean by like a year and a half ago, I guess like a year and a half ago, like the smaller things mattered more. Yeah. But it wasn't like one model was like really good at one of those and really good. And like those aren't like, I don't think like those edit strings are like really specific workflows. Those are like tools that you would use, but they're still kind of different between the models. Well, you know what's interesting is like it kind of is a workflow, though, because like the way that they error correct, like edit strings sort of error corrects in a different way.

31:20Eno Reyes:And one of those like workflows is better. Like the you think one of the string replacements is better. Yeah. I think I think that I think Codex is editing is actually better. Why? On our benchmarks of like the algorithm, like it just performs better on on editing. that in fact is shout out to Ader that original like coding CLI that benchmark is no longer it's totally saturated now it's like super trained on I think but a variant of that that we have we don't run it very often anymore but at least as a couple months ago like I think that there is just a better way to do it but that being said once you start to build up a corpus of this stuff and also the the traces are consistent and then you start scaling that up like you basically have to say, well, what if we just scrapped, you don't scrap it, but rather you have to then start building up the new algorithm in your post-training from there.

32:15Eno Reyes:So I think it's like just not worth the cost-benefit analysis of making the model good at a ever so slightly better workflow. And so that's, I think, a great example of finding like of probably a very close like global like minimum. Like they just like know that both these are decently good enough that it doesn't matter. And can one do the other? Yeah, totally. So going to that, like, yeah, can one do the other? Like, it sounds like there are some differences in terms of like how you guys have your harness now. Maybe you should reevaluate that. Maybe not. But like, let's say Anthropic does not adjust their thing.

32:48Like, you know what? We're going to keep it here. Open eyes, like we're going to keep it here. Do you think you'll continue to need that in a year, in two years, in three years?

32:56Eno Reyes:I think that the nature of our own intelligence shows us that we might need to keep that for a very long time. Like, imagine just like being like left or right handed, right? Like we just like encode these preferences that are like pretty arbitrary. And it's really hard for us to to like switch. And some people can like with a ton of effort choose to switch. But, you know, that's sort of like how I would describe some of these, like where like the workflow is it's almost the same, right? And so I would argue that this is a great example of like what this convergence will look like in practice is like it won't be that they do the exact same thing.

33:36Eno Reyes:It'll be that they found two solutions that work so well at solving the problem or the multitude. Right. There might be like eight solutions that all basically work. And maybe you could argue that one's like one percent or two percent better. I think that this is why at the limit, like the models themselves are not the driver of the differentiation because you eventually just sort of find these like sort of optimal strategies across a bunch of different tasks. And then, you know, if you were able to find that a little bit earlier, maybe there's some alpha, but but not for long. So the model is probably not the driver of the differentiation.

34:08But couldn't you argue like you need to use it in that harness to achieve that like near like even if they're close, like you need to use different harnesses for different models.

34:18Eno Reyes:only if there is no ability to basically make a harness that takes all of those into account, right? So like, I would sort of argue that like, a lot of what we do is we just sit on top of these models, and we look for those local like optima, and then we and we compare a lot to so like, what's interesting is in our cybersecurity benchmark that we have, it's like a cyber gym plus plus sort of situation. What we found is that GLM 5.2 is honestly, it really looks like it might just be better at this task than the other frontier models. However, the harness is seriously holding it back because we're still observing and hill climbing these dumb decisions that it makes that when we sort of investigate, we read the reasoning traces, we see the like, clearly it is post trained on the harness in a very different way.

35:07Eno Reyes:That's not enough to overcome us just saying don't do that. So when we say don't do that, it doesn't. And then it goes and it's actually way better at solving the problem. And so there, and this is like, I think this interesting thing where, um, a lot of that is pretty intuitive. Like you read the, the, you read the reasoning trace or will an agent reads the trace. It sort of builds a hypothesis. It edits the system and then it goes in and it says like, let me run that again. And you sort of do this for enough. And you start to get this feeling that like, there's probably these like workflows that exist that no one has uncovered yet that actually changed the impact of the models by like such a significant amount.

35:44Eno Reyes:And like no one's uncovered them yet, right? They just, they exist in the latent space, but haven't been found. So yeah, I think that is where a lot of alphas we found, not all of it. I think also like obviously the difference between like Opus and Fable is huge. So like clearly model scale as well. So hopefully we have people who keep investing in like all of the above, but I don't think that you like need to invest in the entire stack to see alpha or outperformance. It sounds like one of the things that you're talking about, and correct me if I'm wrong, but it sounds like one of the things you're talking about is basically running some sort of process to hill climb on like what the peculiars of the harness should be with relative to a particular model.

36:26What do those peculiarities end up looking like? Is it the prompt, the tool descriptions, the tool arguments? And what does that hill climbing process look like for you guys?

36:36Eno Reyes:This is actually the origin of the concept of signals for us is we were seeing so much of these. We built this system called signals, and this is actually like a much older version than what we have today. But basically what it did is it had a very lightweight model on top of the actual agent, just sort of look at the agent's activity. And then in online, like on in real environments, sort of self assess. Did I complete that successfully? What issues did the user encounter? Did I make mistakes that the user had to correct me on? And we would then aggregate and anonymize the underlying activity, but send up basically like, I consistently was told to use this tool and I was not.

37:19Eno Reyes:And it's like that sort of insight, right? And so you aggregate enough of these and you basically then you build this list of like, wow, this open model just refuses to use ripgrap. Why is that? And then you sort of look and you see, oh, like the way that we've been prompting this is via a system environment injection. And then you notice this trend. Wow, every single one of these like system injections is being ignored by this open model. Clearly that open model was not post-trained with environment injections. So then you have multiple options, right? One is you can like try and force that way into the harness.

37:53Eno Reyes:But at this point, what you've uncovered is this is effectively a model problem. So we'll share that and we'll share that with the company and we'll say, hey, by the way, check this out. Like look at all of these things that work super well for these other models. that does not work for your model. Then the next iteration comes. So that's part of it. The other part is sometimes you notice subtleties in the harness. And this is always the first time you roll out a model. It ends up being that like whatever inference providers you're using have just like, not to blame them, but like it's a very hard thing to do.

38:19Eno Reyes:But like the thinking traces are no longer consistent. They're not being passed in order. Like oftentimes you find API errors. And that is actually the vast majority of for a lot of companies when they first rolled out either inference or models. we see that like the vast majority of what we're doing is just finding bugs in the api and so that's like the honest answer is a lot of it is uh is i feel that all of this is engineering it's like good old-fashioned engineering it's not magic or like arcane whispering it's just straight up like looking at problem finding bug and then pointing out who should fix it i think that's underestimated that that requires like a team of people and you have to be thinking about it all the time so my main thing would be i don't think that it's like only one people one company can build a great harness or model independent harness.

39:04Eno Reyes:In fact, I think like you guys build an awesome one. I think ours is great. The AMP folks make a great one. Cursors is awesome. Like, I mean, so many companies make great model independent harnesses. But I do think that you almost certainly need to be focused on building a harness if that like as a full time for your company, because otherwise it's going to just be this painful process that you only discover on like the sixth model you integrate that it's actually really hard. Speaking of some of the API errors, do you think the model APIs are going to get more similar or more different over time?

39:35And how much of the differences do you guys lean into? Do you take advantage of all the different things that Anthropic does? I think they have some auto-compaction, I think, tools they can call that run server-side. How much do you lean into those server-specific ones?

39:50Eno Reyes:I mean, basically all of the server-side operations that all of the API providers are doing are basically like better served implementing them ourselves. The primary reason being that we get it for everyone. So like, for example, we, I think might be the only model, maybe you guys have it, actually, I don't know for sure. But like, as far as I'm aware, like we were one of the first at least to have tool search in like Kimmy and GLM and like Minimax. And now I think it's a lot more common to have like tool search. But there's a lot of things like that, where if we just choose to implement it on our side, we get a lot of value.

40:24Eno Reyes:I also think that like one of the things that we do that's a little like kind of like pattern breaking is that our full agent, the entire loop, the whole harness is inside that binary that we give. So like if you really wanted to, like you could kind of see how the sausage is made by just decompiling that binary. You know, maybe don't do it. But like at the end of the day, it's out there and you could if you wanted to. So I think that like a lot of this stuff doesn't need to be black boxed from the user. Like you can just run it on the client. But if you did want to sell your API and prevent people from leaving, you would put this on the server side.

40:57Eno Reyes:So it's like a reasonable business decision to do. Yeah, I mean, we do a similar thing to you where we just we implement these mostly inside the harness. But it's a question I come back to. And the root of this is like because I think if you're talking about kind of like just matching the prompts, matching the tool arcs, that's easy. I think you can pretty easily do that. 100%. Is there anything that like a model provider could do to make their kind of like harness really actually legitimately only work with their model and not easily be copied by that? That's kind of the root of the question.

41:30Eno Reyes:We have like thought about this a ton. I'm sure you have as well. The answer is one, never say never. So like I don't know. It could there is totally a scheme I could imagine. I think that almost every variant of this has so many properties that I think make it not like an obvious thing to do. The first is I think that because the API itself needs to be available because that's their core business, right? Like they sell an API, a platform. As long as you have that publicly available, I think that you can always build things on top of that. I think that there are certain decisions that would severely limit the availability of their API.

42:10Eno Reyes:like I think as an example of this is obviously there's some decisions that have been made to like degrade the model given operations on a certain type of like workflow. So for example, if you tend to work on something that maybe they view as competitive, they being anyone who chooses to implement this sort of thing, then maybe they degrade the model silently. That happens on the server side. You can imagine for a second that they might also then say, unless it's got a tag for our harness, right? I think that because these are also client side bundles, you would see that. So like you would know that if you included a tag that said, were this harness, it would go.

42:52Eno Reyes:Now you can on your server side say, well, I'm also searching for the prompt. And so only this system prompt does it. I think as you start to go down this route, you realize that you have to be actively combative to other harnesses. And I think that that's sort of a jump that is a decision as a business that signals that platform business is no longer your core competency or you don't want it to be your core competency anymore. For so many reasons, that will lead to a funnel of money moving away from them, I think, to others. And I also think that to the market, right? They're valued as platform businesses, right?

43:26Eno Reyes:Especially in this IPO roll-up. It would be interesting to see what happens as their thing that makes up 70 to 80 % of the revenue has a forward future projection of maybe like$0. right and instead they think applications will be the only thing they want to serve they would then have to make up the delta in revenue immediately and so then you would they would have to imagine that they can do that for some reason because maybe their models jump far enough ahead that like people will regardless of what they want obviously pick that to me this scenario sounds like hard takeoff so i do think that like that could happen like and if that does happen i think the implications for the rest of the world are actually like so much more important than for like our company.

44:07Eno Reyes:Like I think that like hard takeoff is like so interesting and important for so many reasons that I'm like kind of like, you know what, if that happens, we're going to play a different game. Like a different thing is going to happen and we can always like war game that, you know? What about some of like the statefulness stuff? Like that's kind of what I come back to. Like in theory, if like you could derive like just so much value from some stateful property of the API that I guess you could use that without like. Like memory maybe or like. Exactly, yeah. That I think is the only like, yeah, that would be the only thing that would be like, I think, big enough to kind of like maybe carry some weight.

44:41But memory is so early on. I mean, maybe this is like, how do you guys think about memory?

44:46Eno Reyes:Yeah, memory is hard. Memory is a very interesting thing. I think that there are several different directions that memory goes down. And one of the things that we've continued to say is like, we want to make sure that we're building things that are like maximally useful for our customers. And so I think that there is it's like very worthwhile to do research on this. So we've thought about so many different ways of implementing this. I think the big first fork is, do you think this needs to be a learned memory via gradient descent? Like, is this a learned process? Does this need to change the weights somewhere?

45:17Eno Reyes:Is this like a LoRa fine-tune? I think Ramp just did some really cool work on sort of like learning a LoRa head on top of a model, or they called it something else, but it's an equivalent sort of system. If that is the case, then it leads to some very interesting things. Like, for example, you could then you don't give people the weights. So you basically host these like Laura heads for every customer that get learned over time on preferences. I think that might make the market for open models that do this like significantly larger because not your intelligence. You know, there's probably a flashy way to say that, but you know what I mean?

45:51Eno Reyes:Like, I think that honestly, that decision would might actually shift people even more towards open models. So I think that that's also an interesting thing to go. The other is if you think that memory is effectively a like us, like a system of record, like files, context graph, objects, durable objects or something. We think there's a ton of things that like state outside of the agent, super important. And you could call it memory, but I can always come up with better names like preferences, you know, style guides, skills. All of these to me feel like they're more useful than saying memory. A few things you said that I agree with.

46:27I think like in terms of enterprises or anyone wanting to own like their memory in whatever form, I think that's like spot on. I think like that's absolutely what you want to own. I also think like it will probably in the long, long run, I feel like it will probably be a combination of these things. Totally. Anyways, I don't know how we'll get there. One thing, though, that you guys did, which is maybe kind of like adjacent to memory, AutoWiki. Can you talk about AutoWiki and how you think that fits into the whole software factor?

46:55Eno Reyes:So there's been so many variants of this, right? Like when we three years ago, when the first droid we ever made was ReviewDroid, right? And what we ended up finding, and this is like a bit of like a history lesson for us, we ended up finding that ReviewDroid was just not consistently gathering the context that it needed about like, not just the current decisions, but historical decisions. And so then we built DocumentationDroid. and what's funny is that the actual high-level workflow is basically exactly the same as what we built three and a half years ago now what has changed is first of all there have been several products that have come out that are like sort of also along these lines and I think that each of them had like really interesting things that they did around like how you want to represent this like should this and also I think it helped us answer the question who is this for right like is this for the agent or is it for the person?

47:48Eno Reyes:I think that it's clear to us at least that this is actually for the person and the people. I don't know if agents need a wiki anymore. I think that they're good enough at finding this context. And so what we did is we said, what would it mean to optimize effectively like really useful, dense representations of knowledge that a person can glance at and they can learn much more than even asking an agent. And so that made us turn towards like what are the structures of knowledge that are useful. So I think a lot of the value that we're actually bringing with auto wiki, the workflow is cool. And I actually think that like you could, like, if you just kind of want to say, like, go and create like a wiki for me, like Karpathy's living wiki, all these sorts of things are all great ideas.

48:30Eno Reyes:I don't think you actually need us to do that. I think what we basically provide are a couple of canonical structures where we're like for the wiki, for the wiki itself, like this type of code base, you should actually just think of your code base like this or like you are saying a lot of different things but really you're just talking about a microservice monorepo and so like i think a lot of that is just saying like here are some canonical structures and then we did one kind of spicy thing which is we made it create a video like a video overview so like it uses like a skill to actually design like a motion graphic text-to-speech sort of like video walkthrough of the repo and like all of our new hires like watch those videos for us.

49:10Eno Reyes:They're actually like pretty good. I'm like very, it's a very amusing feature. How long are they? Like four to six minutes. I'm super amused by them. Oh, and then we have a thing called Lore and Lore is so fun to read. It's just the history of the code base but sort of like annotated by an agent. And that's part of like auto wiki. Every auto wiki gets Lore and when you read it, if you've been working on your code base for a couple of years, it is like a trip down memory lane it's like pretty fun to observe and it also pulls insights about I think the people we try to make it pretty celebratory because you sort of see like these things where it's like oh Stepan made the first commit of this thing that we call Agent Core and like this thing is now the centerpiece of your company and like these are the types of things that are like very fun to see you guys also have like a I think you guys just released something like a couple days ago in this kind of camp?

50:06Yeah, we're starting to play around a lot more with this idea. And I'd be curious for your take here, but it kind of goes back to memory. And we kind of see like two different things in memory. One, I think the first person who I talked to about was actually someone at a financial institution. They were talking about their traders on their desk. And like, they basically said, I want to like take all the knowledge that's in this person's head and that's expressed through their like trades that they make and slacks that they make or whatever. And I want to like clone their brain. so that if they leave and go somewhere else, they can have it.

50:37Super interesting. Yeah, and so I kind of view some of these wiki ideas as basically cloning a brain of some third-party data that already exists. And then I think there's a separate form of memory, which is as the agent runs and as it interacts and humans give feedback, how do you update based on that? And maybe that's the same data structure that it goes back to. Maybe it's a separate one. That's a separate process. But this sort of first clone your brain upfront thing, a pattern that we've kind of like seen a few times, I think like auto wiki, deep wiki, this trader that I mentioned, Karpathy's LLM wiki.

51:15And so yeah, we released something we called open wiki, same, I basically just run an agent, create this wiki, which is like a set of files. Some of the open questions in my mind, and you actually already answered one of them, which is like the structure of the wiki, like how opinionated should it be? And should it be different for different domains because i think we're really interested in enabling not just coding agents but everything um and uh this may be a little bit of time travel either future or past depending on when the episodes come out but someone i just recorded a video with um for another podcast they do the same exact thing and they also said they enforce a structure and it's part and that's that's how they think about the value they provide probably similar to you guys like you guys have domain knowledge of different types of repos and so like anyone can point a coding agent at a repo and be like, create me a wiki.

52:01But you guys say, oh, it should be in this format with this structure.

52:03Eno Reyes:Yeah, it's funny you say that. I do think that this is just the single it's I think it's boring is why people don't think about this more often. Like the boring answer to like why application layer companies are continuing to grow and add value and people care about us is that we just simply take a bunch of opinions about a workflow and we just keep encoding it deeper and deeper and deeper. And I do think that that is always going to be more valuable than just like not having the opinion. And I think that that's just intelligence in general. Like we see this modeled in consulting firms and in other services industries.

52:38Eno Reyes:Like you can hire generalists or you can hire specialists. And I feel like that idea of we just give you the almost like an opinion about it is always going to be valuable. I think I think that's exactly right. There's an interesting I don't know if you've seen going to mess up the acronym. It's from Google. I think it might, I think it's OKF, open knowledge format. Oh, yeah. Yeah. Have you thought about adopting that for auto wiki or conforming to that or leaning into that? Yeah, I saw that. I think that that's one of those interesting things. I actually think this intersects with our previous conversation about model preferences.

53:13Eno Reyes:Like my biggest thing would be is like that open knowledge format. What model is like sort of jumping to that And is it in the data sets enough? And I think that if the goal of that project is to say, let's make sure every frontier model post trains on like an open knowledge format based structure to maintain its own curated memories palace like that to me is like great, like then we should definitely adopt it. But my guess is that that's like Gemini, like Gemini is probably good at maintaining that format. And maybe like the other models are going to take a couple generations to even like learn about that format.

53:47Eno Reyes:And so that would be my initial hesitation. However, very open-minded. I do think that the difference in structure, though, is useful. I don't think that a code base that represents a web app is even close to what FFmpeg looks like. And so we should just not model those as the same, I think. I need to also look closer at OKF to see if... I think you can probably have some difference in structure. Yeah, totally. Maybe more some of the semantics around things. but it also goes back to what you said earlier which i thought was really interesting because when we're thinking about these wikis because we're thinking of this memory we're very much thinking about them as power and the agents yeah and so i thought it was really interesting that you thought it was more for humans which i totally get and and i don't know if um i don't know if that's something we're wrong on i don't know if it's just different use cases need different things well i think that i think that if it's not a wrong or right because like if we were to build and we're actually very interested in building like the same structured state of knowledge outside of the agent as like for the agent.

54:50Eno Reyes:That was a really weird way of saying it, but effectively, we're also interested in that. To me, the auto wiki product, though, that we sell, that is primarily for humans. So I think we'll call whatever that like is closer to what you're building something else. In fact, that may be like something that we call memory or I just think that that word has caused so much trouble for so many people in the AI space. Memory. Yeah. And this is more just like I've got my enterprise selling brain on. I do think that if you say that word to a CIO or a VP of engineering, they have just heard that pitch like a hundred times.

55:25Eno Reyes:And so I almost feel like we downplay some of that. Even if we were to do it, we might even hide it so that you don't even necessarily know the memory is happening. You guys do have something around basically like agent readiness report. And so I'm like, maybe that ties in in some way, because that is like presumably how you I'd love for you to give a deeper overview, but presumably it's like how you structure your code base so that an agent can use it. Well, that's kind of the similar things. Exactly. Like, I think that how you structure is a huge part of agent readiness and really even more broadly.

55:55Eno Reyes:What I would say is that agents succeed when they have non-human deterministic feedback. So like when they can consistently receive feedback from anything other than us. then they can keep going. For software engineering, is that unit tests or are there other things? So many things. In fact, there's like hundreds of these sources of deterministic feedback. Like what are some examples that I'm not thinking of? So not only are, do you have, so testing would be a whole category. Unit, end-to-end, integration, fuzzy. Also like test coverage. So like whether or not you pass test coverage is itself a feedback item, right?

56:32Eno Reyes:So there's like testing, dev environment reproducibility. So can you spin up the dev environment? When you run it, does it work? Is it flaky? So there's sorts of flakiness detection, right? Like those sorts of things. Standardized metrics and logging. Do you have a rule that says, if I try to make a metric that doesn't fit our format, will it fail? So that's like a form of linting, but it's a little bit more specific to how logs are done. And you can get really specific with all of these, like, do you have AST parsers? Do you have opinions about the code base? Are you linting file names? Do you have a structure?

57:06Eno Reyes:Do you have code owners? So all of these things, basically the presence or lack thereof, hurt or help agents. And so what we have seen is that if you really want to solve a problem with an agent, if you can define the problem clearly and define the outcome and measure whether that outcome is true, there are harnesses, like for example, missions that can just achieve that outcome as long as you're willing to pay the inference costs. I wanted to ask about missions. Yeah. Could you talk about missions and then what makes maybe like missions successful and what's going on under the hood? Super related to agent readiness.

57:39Eno Reyes:In fact, we actually now have instant we used to not do this. But now if you try to run a mission and you haven't run agent readiness, we gently recommend that you do that first. We're not going to block you, but we gently recommend missions basically is a attempt to say there is a like more, even more. I've used the word workflow a lot. there's an even more like general meta workflow that an agent can use to discover the right workflow to solve almost any knowledge work problem. And so that's like kind of a lofty goal, right? Like a universal meta harness almost. What we basically think is that this verification problem or like can you validate or verify the outcome is so fundamental to how agents like succeed or fail inside of almost all tasks that you should just build a system that says for whatever problem that they've just defined, one, define what that workflow should be and define it only via the structure of what should be, what is the outcome and how do you verify that outcome?

58:41Eno Reyes:So for example, when you run a, that's like a very kind of like meta way to describe it, but basically take a software engineering task. When you do like a spec-driven agent, you create a spec and you say, this is the outcome I care about. And at the bottom, you might even say like, you know how we're going to validate it you write tests blah blah this is way too loose like what we do with missions is we say let's actually create an extremely structured data object that represents every possible like hypothesis that could be tested about this software like can i do this action is this flow supported does it do these are these things present and then you also in that structured object say and this is how i will know deterministically that this is true or if you can't deterministically validate.

59:28Eno Reyes:Here's how an LLM will analyze the success or not. Do those deterministic things end up looking like tests or I guess some maybe some lens? So it could be a test. It could be like, I know that this flow works because a unit test passes. But another one could be, I know that this flow works because when I open up the browser and I use computer control to click on the sign up button, it takes me less than four clicks to get to this page. And so like that is obviously non-deterministic. However, it's very clear. And so like a verification agent can go in and it can run and it can say, I either achieved that outcome or I didn't.

1:00:02Eno Reyes:And so when you scale this out, you start to realize there is no system that can let a non-deterministic agent actually accomplish all of these goals in a very consistent way. So you have to add a little bit more of a push. Do not let the agent finish until it has confirmed every single one of these validations. As a result, this can take a very long time. And so we sort of say, look, this is probably going to get faster. It's probably going to get better. But for now, if you want to ensure an outcome, then you can line it all up such that the outcome is confirmed if you can verify it. So that's why it ties then to we say, look, do you want to bring the performance of the system down to X milliseconds?

1:00:47Eno Reyes:Do you want to validate that this thing works? As long as you can verify, then put it in a mission and it will succeed. And what we've seen is it can do some pretty amazing things. Some problems are like unbounded or unconstrained. So you can't necessarily just solve everything. I don't necessarily know if missions couldn't, if that like if there's some more advanced version of this couldn't solve like most verifiable problems. For some of the tests that you run, won't those need to be generated by the agent as it's running? And like, couldn't the agent, you know, just write a silly test that always passed and be like, oh, satisfied the mission.

1:01:22Eno Reyes:Totally. And so a lot of what ends up happening is you need to validate the validators. So it's kind of turtles all the way down. Right. Right. So there is implicitly some trust, but I actually think this is how like human organizations also work is like you kind of have were fuzzy, non-deterministic systems. What's nice about agents is that they are pretty good faith. They do genuinely want to solve the problems. And I think that the reason Anthropics interpretability research on this was so fascinating of like panic and like, or at least the, the auto encoded representation of panic inside of agents, as they get closer to failing their goal, they become more likely to cheat.

1:01:57Eno Reyes:right? So we actually want to work around some of these weird quirks. We make sure that like agents are sort of given very specific instructions. And so there's a lot of like harnessing around this. Our goal is to remove as much of that harnessing as possible by getting increasingly eloquent workflows like skills, basically. And so at this point, Droid actually will define its own skills for each worker. And then it will continuously refine those skills as each worker works. And because they're workers, they don't have context or access to what others have done. So each one is sort of like starting fresh and given a blank task with a set of instructions that are sort of like the results of many agents running on that task.

1:02:39Eno Reyes:And so you have a good chance that like any like p-hacking or like any attempts to sort of like sketch through it only lasts within a single run. And then if you have like milestone-based validation, you can get rid of a lot of the like variability within a given like individual run. So just to make sure I understand by worker, you mean like one run of the agent up until it checks the criteria. Yeah, at a very high level, it's like an orchestrator that defines skills that then workers pick up the skill and the task. They do the work and then a validator comes in and it validates the work. And then you can imagine running tons of those consecutively.

1:03:19Eno Reyes:And then you sort of have like almost like a consolidation phase where you like look at all of the accomplished tasks and you say, is this chunk of work sort of adequately completed? To us, it feels like a lot of the problem with this definitely uncovers how far we have to go with even the models themselves. Like the clearly a very diligent human would do much better at like every individual task. But the sum of work, I don't know if there's many humans that would as diligently complete the total sum of work in the time in anywhere near the time that these can. In practice, do you see that the validators are more often than not the deterministic things or the LLM as a judge type things?

1:04:00Eno Reyes:The validators rely very heavily on the deterministic flows. So for example, if it says like, in order to do this, you must pass this end-to-end test, the validator will be very good at writing like a pretty fluent end-to-end test. I mean, the intuition for this is like, if you open up a coding agent in an empty repo with like pretty little context, and you say to it, like, write a really good end-to-end test for this workflow, they tend to be like pretty good at that style of task. So that's basically what we're doing is we're just sort of imitating the fresh context, clear repo, clear instructions workflow, but scaled across what is like a very large task that's been decomposed into its principal units.

1:04:37Missions sound like they would cost a lot of money. And I know in general, like token usage and therefore costs associated with coding agents has risen dramatically, even inside of Langchain over the past six months, I'd say. What are you guys seeing and how do you guys think about dealing with that?

1:04:55Eno Reyes:Yeah, tokenomics is the number one topic for I think everybody right now. Missions do cost a lot of money. I think that there's like a true ROI story for them. Like this scale of tasks that people are currently using or think order of magnitude, like a couple tens of millions of line code base migrated. So I think it's the type of thing where the order of magnitude for a mission could be like$100. I mean, it could be$1 ,000, right? And at the end of the day, that is a decision that you sort of have to make knowing it's going to be expensive. I think in many ways, this is not that different from almost like, instead of just like hiring a contractor, it's more like hiring like a firm, right?

1:05:34Eno Reyes:Where you're really going and you're getting like a quote. You're saying, let me actually shop this around, right? Like, I want to understand, is this going to be expensive? Do you provide those estimates? And do you let people say, hey, do this mission up until it costs me 50 bucks? because that's the max I'm willing to pay. Yeah, we have price controls, and increasingly we're working on our prediction of the total cost. Because I think that in the end, like, what we want to do is we almost want to be able to say to you, you know what would be great is to insure it. Like, we promise that you'll get this done for this much money, and we know what that price is, and we sort of hold ourselves to it.

1:06:10Eno Reyes:I'd love to get to that. That's a tech problem. That is nothing other than a tech problem, and we're working on it, actually. But I think that what a lot of people actually face is more like wasted work, not in missions, but across the vast majority of enablement. Like when you roll agents out to an entire company, you've got 10 ,000 people using these tools. It's so easy to waste money with them that I actually have found that the vast majority of wasted spend has nothing to do with like whether the agent is 5 % more efficient. It's just sort of like cutting off access to expensive models when they shouldn't be used.

1:06:44I was going to ask, yeah, like how do you what do you see people doing? How do you help companies deal with that?

1:06:49Eno Reyes:Auto model routing. Basically, the biggest save for most people is to understand what tasks actually need what models and to remove the human's need to learn an entirely new world in order to use these agents. And I think you guys launched or announced model routing a month-ish ago. And I think some of the figures where it cut costs by 30%, something like that, 50%. It's actually a parameter, which is funny. Like, you can choose how much it saves you money. And that is because what you're trading off is quality, right? So there's a basically... Is this the admin that chooses that? The admin does.

1:07:28Eno Reyes:Yeah. So it's actually pretty cool. Like basically we give you a slider on the Pareto frontier, right? You can sort of say, hey, I'm comfortable with this level of like aggressive cost savings. And I know it's going to, is that that different from picking a cheaper model as the default? No, I'd argue it's actually better because the more that you're able to sort of slide up and down that true Pareto frontier, the more effective you can find like a happy place. And I think right now, a lot of this is like kind of the equivalent of like the elbow curve estimation where data scientists kind of eyeball the part on the curve that looks good, what we are doing is we're actually trying to build what we call agent effectiveness tooling.

1:08:06Eno Reyes:So think of it like this. We know that there's a task distribution that's unique for all tasks that agents are being used for in every company. So if we can take that task distribution and learn a model that is extremely good at deciding when to use what model across that distribution, we can actually give you the correct answer on that curve. You know, you basically want to preserve the best possible quality and the best possible cost. And open models are getting so good that it is impossible to ignore. However, what we found is that if you show people the model, people's preferences immediately fly towards frontier models.

1:08:44Eno Reyes:So basically, I think a lot of this is like psychological. Like if I do not know what model I'm using, I am likely to say, wow, this is a great model. If I know the model I'm using, my bias is towards the frontier models. So I think that like that alone explains a lot of what has happened over the last couple months is like marketing. Like some models are better at a lot of things. And if you're better at the right things, people will get convinced that they're better at everything. And in reality, I think if the cost was like not that much, this probably would just be fine. But the cost is so we're talking about tens, hundreds of millions of dollars, even in a single enterprise.

1:09:20Eno Reyes:I think math is going to win here. One common critique of model routing that I hear is prompt caching. How do you guys deal with prompt caching and model routing? Cache-aware model routing is itself like, we have like an eval routing bench. And so the big challenge is breaking the cache is oftentimes worth it if you know that that is, basically you have to just be aware of it, right? So what's cool is also within a given family of models, you can at least preserve a little bit more continuity. You can't preserve the cache. but it's like faster to switch because like basically this is just how our model switching works.

1:09:58Eno Reyes:We compact in order to switch models because the reasoning traces don't nicely pass over between the two. And so like if you're routing to a model in the same family, you can preserve like midway through instead of just compacting. But what's nice about the cache aware sort of evals that you build is you actually need to make model routing, not just, and this is why I don't think you can do it in the gateway. Like I do think maybe if you really wanted to, you could route, you could get some gains from like a gateway. But at the end of the day, routing is a harness problem because the harness needs to be aware of when to upgrade or downgrade.

1:10:36Eno Reyes:Right. And so if the harness becomes aware of its own ability to upgrade its intelligence and aware of its own cache, then it knows, well, I'm about to finish this problem anyway. And so I might just finish it now while it's cheap. And if I can't finish it, then I upgrade versus I am so clearly far away from solving this problem. I'm just going to upgrade now because otherwise I'm going to waste tokens. That is actually instructions and guidance you can provide the like monitoring system that does the routing decision. So tech problem, very solvable. You mentioned open models earlier. What are you seeing there in terms of how viable they are, how much adoption they're getting, and where you think the future goes from there?

1:11:16Eno Reyes:We're very bullish on open models. It seems like I wish there were more options in general. I think that my hope is geopolitics doesn't get in the way of the progression of open models. I think great U.S.-based open models is obviously a huge need. I would argue it's actually like a national security imperative for us to be building better American U.S. models. But that being said, I think that my hope is that we're actually able to sort of transcend a little bit of that. And open models are, to me, a global thing. So if we can get like really amazing globally, just open models, I think that that is clearly much better for almost every single business in the world.

1:11:59Eno Reyes:There are like a very limited number of businesses that do not benefit from open models. And I think the vast majority do. So I believe in like sort of like accelerating the economic prosperity of the entire planet. So I think that even if the US doesn't get access to open models, I'm still in favor of them. because at the end of the day, you know, intelligence is probably going to be a commodity, interestingly enough, from our previous conversation. And so if intelligence is going to be a commodity, my hope is that people have access to it. There's another type of routing or there's another type of model uses that sometimes gets lumped in with routing, I think called like Fusion, like Open Router had their kind of like Open Router Fusion.

1:12:41Sakana AI had Fugo, I want to say, something like that. Have you guys thought of that? basically routing to multiple models at the same time and then consolidating the results back.

1:12:52Eno Reyes:So this is super interesting because we experimented with several different flows. I will say, I think we're in our like, this is a throwback, the cognitive architectures era, where the naming of this stuff is probably doing a little bit more confusing. At the end of the day, in my mind, these are all basically model routers. And the interesting thing is there's so many levers to play with. For example, I mentioned upgrading, like giving the model a tool to upgrade or downgrade its intelligence. Then there's sub agents. So sending off sub agents that are weaker models to then come back and synthesize.

1:13:27Eno Reyes:And then there's parallelization and synthesis, like run to it once and pick the winner. We used to do this all the time back when you had to like really hard code that workflow. Super not obvious to us if there's like a strictly greater than. But I do think that what's clear is all of these strategies are helpful. So I think that right now we're allowing users to kind of toggle a couple of these things. The parallelization has been a feature for a little bit. You can run multiple at once with different models. It's a very unpopular feature. And that's parallelization of agents. So not just like a single call, but actually agents.

1:14:01Exactly.

1:14:02Eno Reyes:Like let's run two agents with effectively the exact same prompt, but with different models and resynthesize. These are all very unpopular with our users. Why? I think that it's too complicated. And so maybe if you like chose to hard code this and like make it like the only option, maybe people would like it. But what we see is that it's way more expensive and unclear gains. That could be just us. I have no knowledge of if it's there's like a best answer. You mentioned evals a few times. How many different benchmarks do you have? we have we even have a benchmark called bench bench uh which is droids ability to create more benchmarks um so we have a ton and i think a lot of them are small though i think people assume that like to create a benchmark you have to build this like massive data set um and we have a couple that are objectively very large but also we just run we have a very eval driven culture in general like we have these little like regression based evals that run like 10 tasks and they just make sure that prompt caching works or compaction works.

1:15:05Eno Reyes:We have. And so I think the line for us between very integrated end to end test and like eval is actually very blurry. I consider it an eval when there's some degree of like rubric based grading or scoring that doesn't it's like not a binary. Yes. No. Instead, there's like sort of a gradient of scores. The cool ones. Legacy Bench is cool. That's like COBOL basic Fortran migrations. BenchBench is cool. CodeReview is a totally uncharted territory of benchmarking. It's insanely hard to benchmark. I actually think that CodeReview benchmarks might actively be hurting CodeReview products. I was going to say, there's at least one CodeReview benchmark out there, I think, right?

1:15:47Yeah.

1:15:47Eno Reyes:Do you just not trust it? Everybody's CodeReview, we learned this recently, every single company in this space's CodeReview benchmark is a fork of that benchmark. And so that benchmark is effectively like, no statement on if it's good or not. I think it's like was useful. It's probably not useful anymore. And it was a great innovation at the time. So it's like not shade to the creators. But I do think that that benchmark is massively influencing how code review is done across the board. And I don't know if it's obvious that like it is aligned with a better experience for people who use these tools.

1:16:19How often do your benchmarks change? Like when does a benchmark stop being good? Because I think it's totally reasonable what you said. Like, yeah, it was a great innovation at the time. It probably provided a bunch of value.

1:16:29Eno Reyes:benchmarks have a very short shelf life for us, like three to six months max, because we get better ones or we build like a V2, like terminal bench for us, we still run, not because we think it's making our agent better, because it's actually a very useful regression tester. Like you just like there's certain things that at this point we built a ton of tests for regressions, but like we do not even look at terminal bench anymore. It's just not even it's not useful. So three to six months is my kind of like shelf life for it, I'd say. Do you use Harbor as a format for benchmarks? Love Harbor. That was great.

1:17:01Eno Reyes:I'm glad we were standardized on that. Not only us, but most people as well. We just did an integration with them this past week. That's amazing. Deep agents in Harbor, trace things to Langsmith, Langsmith sandboxes. I think for long-running agent evals, it definitely seems like the industry standard. Yeah, completely agreed. Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback or questions to maxagency at langchain.dev. We want to hear from you.