How to Find the Agent Failures Your Evals Miss with Scott Clark - #767

25:22They're not necessarily just the weirdest things we've ever seen, which would be at the very tail edge of this distribution, but they might represent just these pockets. And so we can use stratified sampling to try to find these pockets and then try to do clustering on it. And this is where we can apply some techniques like the Clio paper from Anthropic to try to do topic modeling. and once we have these individual sub pockets, then we can do this direct comparison. We can create a taxonomy of this looks like this, which is different than this and that's good or that's bad and then just recursively throw it at an LLM basically to like further refine these pockets, get a better taxonomy, a better human readable representation of it and then also have it suggest fixes and that fix could be as easy as add to your system prompt that you have to always call the tool, or it could be use a caching layer so you don't hit as many 429s or whatever it may be.

26:23And over time, this system gets smarter and smarter and fixes some of this low-hanging fruit, and you're able to get a better and better system that finds weirder and weirder patterns. But you could also use it to write evals, which will then start pointing it in a direction, And it'll kind of like, this is the adaptive part of adaptive analytics. Like it'll learn the types of signals that you care about and start pulling those out of this distribution more and more. So relatively simple, but there are a lot of hand waving. Well, it sounds like also a lot of moving parts. Yeah. Like any of these systems.

27:01Yeah.

27:02Sam Charrington:Yeah. What is the Clio paper? The Clio paper was a paper published by Anthropic a year or two ago now, CLIO. And I'm going to forget what the acronym stands for. But it basically learns this topic representation on LLM data. And it's a great way for being able to say like, oh, some people are talking about this. Some people are talking about this and pulling these patterns out. So evals has come up a couple of times in this conversation. From the perspective of analytics, the analytics become kind of a source for evals. But I'm wondering if you can talk a little bit more broadly about what you're seeing in terms of, you know, how folks are approaching evals and like what works, what's not working and like that kind of loop.

27:57I think people are rediscovering what's been true for a long time in machine learning that like, yeah, the loss function matters, the reward function matters, and it needs to be very specific to what you care about. I talked about the fraud example where, again, naively, if you were just like solving an entry Kaggle competition, you're like, oh, just over optimize accuracy, whatever it may be. But when you go and you talk to a real bank, they're like, well, here's the tradeoff. And like these customers, it's like the false positive is like way higher pain than a false negative and all this sort of things like it.

28:30It becomes extremely complex. I remember back in grad school, an example of this was I was trying to come up with what you would call an eval now for metagenomic assemblies. So like when you're trying to assemble a whole bunch of genomes from like a gut biome where everything's mixed together simultaneously. And every single time I'd come up with a great idea, I'd go and I'd talk to the actual biologists in the wet lab and they're like, oh, yeah, that's really clever about like how insertion and deletion happens or something like that. But they're like, you know, every so often when we put this thing into an aluminum machine, there's just like a bubble and it just warps everything.

29:07I'm like, oh, my gosh, I have to add that to my objective function. And like it's this incredibly recursive loop where you have to like talk to an expert or like fundamentally understand what's happening before you can have this larger world model of like what actually matters. It's true in fraud. It's true in metagenomic assembly. But it's also true in these agentic systems. And so basically what I'm getting at here is the best way to come up with an eval or set of evals that actually represents what you want and what you can optimize towards or automatically optimize towards is to have this recursive loop of continual refinement.

29:45And I think analytics is the way to take these larger patterns and signals and then use that to map down into a smaller latent space or manifold of like, these are the numbers I really care about. And I want this one to go up. I want this one to go down. I want this one to never cross this threshold or whatever it may be. But that process of taking this like massively high signal production data and mapping it down to this harness or whatever it may be is this incredibly difficult problem. And it really needs to be task specific. Like you can't just use the rouge or the blue score or be like, is the toxicity high?

30:28Like that can help and you can bootstrap from that. But in reality, every single one of these agents, especially as they're becoming longer running and doing more complex tasks, there's a million gotchas. And that might be in someone's head or might be institutional knowledge or might be whatever it may be. Or it might be the type of thing where it's like, I'll know it when I see it. If this is bad or good, you got to show me examples and show me when it messes up. And then I can help you quickly, like, say if it's good or bad. And this is kind of where like some reinforcement learning is going with like DPO and things like that.

31:04But you need to have the example first before you can say A versus B.

31:09Sam Charrington:You alluded to this earlier, but like as you're, I guess I'm trying to get at like the interconnects in terms of, you know, mental models, like the way you're thinking about building between kind of evals and, you know, some of the higher level of observability. is it something that you should be thinking about early? Is it something that you can be thinking about early? Is it something that you can't be thinking about early? You just have to log everything. And like, you know, once you have the data, then you have to, then you can figure it out. Yeah. So I think more of a roadmap is, yeah, log everything.

31:45And like, it's worth spending the time up front to like instrumenting open telemetry. Like everybody is consolidating around that. It's an open framework for like logging traces. It's nice if you can pick like one of the semantic conventions that everybody's coalescing around. Gen AI has become really popular and that's like an open semantic convention.

32:10Sam Charrington:And just to click into that for a second. So OpenTelemetry, OTEL is like kind of standard logging for anything. like been using this for you know web servers and whatever else like feeding it into your data dogs and those kind of systems and then but because it's you know for everything and anything is just text like you need a way to structure that and that's you know the gen ai is um a way to structure yeah you can think of it like a like a schema so instead of just like here's how json works like throw whatever you want in there it's like here's a specific schema that knows what like LLM responses look like, knows what tool calls look like.

32:56And increasingly they're adding things for like evals and stuff like that as well too. So like it has a general idea of like what the structure looks like. And then that makes it way easier for these other tools to ingest it because instead of trying to like look at this giant JSON blob and be like, hmm, I wonder what, which one of these was the user. Like it's very easy to see what was going on. um and then from that going back so like use open telemetry use the gen ai open uh if i'm going to be like really uh prescriptive here use the gen ai um semantic convention and then some things are just going to pop out at you even if you just like tail 20 and like look at some of these like some of it's going to be like oh wow like some of these call 40 tools i thought it should only call one or something like that maybe i should have that in my monitoring system to like look at the distribution of tools or whatever it might be and so there's going to be some low hanging fruit um and again similar to classic ml where you're like wow sometimes the fraud detection system like it's really aggressive or is extremely biased towards this one subgroup or whatever it may be like that'll jump out to you um and that'll give you that start and it's all about bootstrapping and really everybody's embraced this kind of like learn as you go mentality which is so different than I remember 10 years ago when people were like terrified of random forests and logistic regression and like, how do I understand?

34:21And I would need to look at every part of the decision tree or something like that. Like collectively we've decided that like, let's just YOLO a bunch of stuff out there. Like three nines of uptime doesn't matter. One nine of uptime doesn't matter. Dangerously ignore permissions.

34:37Sam Charrington:Yeah, exactly. Dangerously ship slop. But at the end of the day, especially at and that's fine in this extremely high temperature and kneeling state of like we're trying to like figure out what works and get a bunch of stuff out there but as you want to like lower the temperature on your product and like start to like really make money on it and make things improve and i think obviously some of the foundational labs are here and there's there's companies that are making billions of dollars whether it be uh decagon or harvey or whatever it may be now you actually want to start to learn and adapt and like iteratively refine these systems And I think that's where it can become incredibly valuable to say, okay, obviously I'm doing my telemetry at the very base.

35:19I'm doing some monitoring. But now how do I continue to learn to make the monitoring better? Maybe start logging some new things in these spans within these traces that might actually then help the whole system. And just everything ends up getting stronger, more refined and aligned to your individual use case, but also catching these issues over time. And as you go, and I'm a sucker for premature optimization. So, like, of course, you should be, like, thinking about how you're going to do analytics at the end of the day. But, like, as you go, just know, just keep moving forward. And then eventually, once the low-hanging fruit is gone, there are tools out there to get to that next layer.

36:03You don't need to look through 100 ,000 traces to try to mentally come up with a pattern. There are tools for that. And the only way to make that flywheel actually automatic and put it into a Carpathia auto-research loop or something like that is to come up with these new signals and ideas and inject that into the system.

36:23Sam Charrington:You mentioned goblins earlier, which if anyone missed that reference, there's a blog post from OpenAI about why the GPT-55 is like talking about goblins so much recently. Another kind of contemporary recent articles from, you know, Anthropic once again, like breaking down, like why the models felt like it's been sucking more recently. Um, and one of the issues with like building agents is that like, you know, for the most part, like they're dependent on these black boxes that you don't control, you don't know. And like the more, you know, we see disclosure from Anthropic and the like, the more we understand that like, there are all kinds of knobs that they're twiddling with, even though like the model version number is like the same and they impact our you know the experience you know of you know our experience as builders of things and the experience of the people that are using the things that we build and I'm wondering if like this kind of analytics is part of the answer to that like if you could have you know predicted in a live system that the underlying model, I guess, distribution, again, is, you know, shifting under you.

37:51Yeah. And I think this non-stationarity, which is what you're pointing at here, is actually one of the strongest arguments for keeping this in an online loop. Because whatever evals and guardrails or whatever it was that worked before might not work tomorrow, because the model may have shifted around it or whatever it may be. It's like you're trying to box something into something, some high dimensional box, and it's going to find some dimension that maybe you didn't realize, or it's going to come up with some new skill, or it's going to come up with something. And they were using reinforcement learning to teach it to stop talking about goblins, but it ended up learning that it should be mean to your users or something like that.

38:37Sam Charrington:And maybe it's an important point that this is not an anomaly. Like this is by design. This is what the thing does and why it works so well. Yeah, yeah, exactly. And it's important to think about these dynamic systems that do effectively evolve over time. And yes, Anthropic and OpenAI get to choose the stimulus and the reward that guides that evolution. But that evolution is going to have some interesting side effects to it. And so I go back to the, there's this fun quote from Jurassic Park from 25 years ago or whatever about like the raptors like testing the fence and like constantly trying to like find its way out.

39:15And it's like a fence that works in one situation isn't necessarily going to work in the other or like eventually there's going to be ways to get out of these systems. And this isn't even with like an adversarial system that's like intentionally trying to escape the box. Like these things are just going to naturally grow in a way where it's like, yeah, it turns out your chicken wire fence doesn't work for an elephant.

39:35Sam Charrington:And so like I've often, you know, when I when I come across these reports like the anthropic report, like I've often wondered where is like the the Internet like or the model weather report? Like, how's my model doing today? Like, does this type of approach to analytics give that to me for the types of, you know, for the models that I'm building on? Yeah. Well, and again, I think it's, to use the weather report analogy, I think that there's like two ways to look at this. Like, one is like, what's the temperature today? And that's your monitoring system. That's like a known signal that you can like plot in a time series.

40:15And if it spikes, like, you know what that means.

40:18Sam Charrington:And so that might be, for example, rejections. Yeah, rejections or number of tool calls or yeah, whatever. Again, it's hard to come up with good generic e-bails because they, by necessity, need to be constructed very specifically. And there's been some great work out there. Hamil Hussain has great blogs about why you need more specific e-bails. You should have him on the show, by the way, if you haven't already. He's been on and we can drop a link to that podcast in the show notes. Wonderful, wonderful. Not that we're not overdue with, you know, likely for a new convo, but we will include that.

41:00So, yeah, temperature, humidity, things like this. These are known knowns that you can track looking at the time series. You can immediately interpret that. But something like, oh, there's a heat dome or an atmospheric river or something that you didn't know to think about ahead of time, but could be used to explain some underlying intrinsic phenomenon within your eval system could actually be really interesting. And then once you know about this phenomenon, then you might actually put it into your monitoring or weather app. You're like, oh, I actually want to know when the atmosphere has this sort of like barometric pressure differential or whatever it may be.

41:34And that could be your trigger for like probability for atmospheric river or heat dump or whatever it may be. But you need to like find it from this way more abstract, like higher dimensional data first. And that's what analytics can help with. So it can help explain issues that happened in your lower dimensional eval space, but it can also help you construct once this event does occur and you do see it, ways to measure it more directly and more timely. But you need to have that richer analysis in order to do so.

42:05Sam Charrington:And so in the context of this, you know, LLM weather notion, what's the what's a concrete example of that kind of distributional shift that I might be able to see? So it could just be, yeah, the proclivity for tool calls, as I was saying before, like if you and it's all these like interesting unintended consequences that we see where it's like you tell it to be more efficient or you tell it to like use resources more because maybe your boss told you, hey, we're using too many tokens per request or whatever. And you add something to your system prompt to say like, don't call tools when you don't need to or something like that, something obvious.

42:48So it's like, don't don't waste resources. Um, and you look at your monitoring dashboard, everything looks great. Like all of the evals you have look the same and, oh, your costs drop at like 20%. That's great. But what you don't realize is in some small fraction of your sessions, like it's cheating and like trying to use a pre-cached version or if it's like, oh, like I, I probably memorized a pretty good answer for this. So I'm just going to like use it from no knowledge instead of doing a web search or whatever it may be. And you don't catch it. because what you were looking at before actually looks good.

43:23And it's doing what you told it to do, but it had this second order effect.

43:27Sam Charrington:The thing that I'm poking at though, or want to poke at is like, we said that the number of tool calls is like the temperature. Like that's a monitoring thing. You know, we're talking about in terms of analytics is not, you know, so we could see what you're describing, a scenario you're describing by looking at the temperature and that the temperature is lower, you know, or has a tendency to be lower. Is what you're talking about like, is it, you know, analogous to like the first derivative, like the change of, you know, n over time? Or is that also a metric as opposed to analytics? You could also, to know that you want to look at that as a metric is something you could discover from analytics.

44:12Maybe to go back to the weather analogy, it's like, I didn't remember, like, So I lived in the Pacific Northwest growing up and lived there a little bit recently during the pandemic. And this concept of a heat dome was not something that I knew growing up. Like it was a relatively temperate place. And it turns out it's like this interesting atmospheric phenomena that like just recently started happening because of climate change. and that's the type of thing where you wouldn't know to even look for it before but because of the non-stationarity of global climate like now it's something that can happen and it can actually have these incredibly adverse effects and can blow up your your simple metrics but you wouldn't necessarily know why those metrics were going crazy without having this other concept i think

44:58Sam Charrington:that's closing the loop for me. It's not that you can't see the important things as these kind of, you know, independent numbers, you know, temperature or number of tool calls. It's that, you know, starting from a blank slate, you might not realize that the tool call is the thing that you need to be tracking. You may have thought it was something else. You know, maybe you're, you know, just your token cost or something and you're tracking that. But then you see, you notice some change in something that you are tracking, or you maybe even don't even notice the change. But you can be alerted to about, you know, this key correlative metric, I guess, through analytics.

45:51And it might be something completely orthogonal to like, go back to a fraud detection, super simple binary classification. You could look at the confusion matrix of true and false positives and things like that, and maybe have a pretty good idea of like, okay, my classifier is doing pretty good. But that's very different than looking at that versus looking at that weighted by average transaction cost within each bucket. And now all of a sudden it could jump out to you that like false negatives were all the massive transactions And that has a way different business impact to you than, oh, it was only 2%.

46:24If you're weighting it by transaction volume, which is this completely orthogonal signal to correct or incorrect, now all of a sudden it jumps out that's like, okay, this part of the confusion matrix is actually worth billions of dollars, whereas the rest of it's kind of whatever. Got it. Got it. Got it. But analytics is how you learn that you should even look for that. And then you put it in. Yeah, eventually everything is measurable and then you can put it into the monitoring system or whatever it may be. But finding those unknown unknowns, especially in a non-stationary environment, is a really difficult problem that needs to be automated with analytics is my hypothesis.

47:01Sam Charrington:And distributionals tool is open source? Not open source, but open distribution. So completely free to use, deploys on-prem, or you can use our SaaS offering for free. But a lot of our clients like to keep their logs because there's a lot of sensitive information in the logs. They like to keep that to themselves. So it deploys on-prem, uses whatever local or third-party LLM that you already have in place, and runs these analytics alongside your monitoring system or your nightly ETL jobs or whatever it may be. if I'm using, you know, analytics broadly or, you know, distributionals tool, like what is the, what's the key metric that I'm tracking?

47:49Sam Charrington:Is it like improvement in, you know, whatever metric that I care about, like over time, or is there some, you know, how do folks think about, you know metrics for applying optimization or or for analytics metrics for a metric discovery system exactly yeah right right right right no i love it let's go full meta so i would say uh is it is it finding things that you either wouldn't have found or would have taken you a long time to find and then you could do some sort of calculation for like how much quicker am i finding it or what's the value of finding this thing ahead of time and so it ends up looking closer to like how many like net new features or pull requests or new metrics discovered as opposed to this specific metric dropped or raised by this or not it could have that type of global effect because if you find these like implicit signals that are impacting a very specific business metric like it could have that type of effect as well too but again similar to data science when you're doing like more descriptive data science as opposed to more predictive data science it's about like what were the insights discovered and then did that materially affect the business and what would have happened in the absence of that insight and that's ultimately like the artifact that we have in our system at the very end of this it like goes through this whole data pipeline and it pops it up as an insight it's like two percent of your data had this pattern Here's some examples.

49:23It thinks it's bad for this reason. Here's how you could fix it. Do you want to track it? Do you want to run an experiment? And then loop. And does there exist like an academic benchmark

49:36Sam Charrington:for this kind of problem? It strikes me that it doesn't necessarily, it doesn't defy benchmarking. I could have some benchmark of some logs for some commerce problem, or maybe it sounds a little bit like cybersecurity, like some cybersecurity dataset, and you apply this, you know, analytical tool and, you know, other approaches to analytics and try to benchmark how many insights, you know, are identified. Is that a thing or is it a thing that could be a thing? I think it's a thing that could be a thing. Like, yeah, having some sort of hidden signal that needs to be learned in order to like fix some some known signal and then trying to find a system that can do that automatically security is actually a really interesting use case here because weird behavior or like these these sub distributions of behavior could represent like literal issues with your agent or it could represent malicious activity as well.

50:43And so that's a really interesting use case as well. You've given me something to think about. We should write that benchmark.

50:53Sam Charrington:Have you done much or been pulled into security scenarios? When we pitched this to companies, we can typically focus on enterprise customers. They definitely are very attuned, and especially with Mythos and things like that coming out, that security is going to become a really big thing. Everybody's gotten superpowers, including the black hat hackers. And so, yeah, this is an interesting side effect. It's really difficult for us, though, to get good security data in order to train our models. And because we're on-prem and data providence focused first, we can't get a lot of signals from our customers as well.

51:34So I think this is going to be something that's going to require a much deeper partnership or acquisition or something like that for us to be able to like really sink our teeth into. Got it, got it, got it. So what's next? We're just going to continue to work on the system. Again, we pivoted somewhat recently from the pre-production testing into this post-production analytics and have found a lot of value there. The system's completely free to use, free to download, free to deploy the whole system. It packages a Helm chart and like you can just run it on your infrastructure, or air gapped if you want and see where it takes us.

52:09I think the larger market is really interesting right now. There's obviously a lot of consolidation happening. There's a lot of different tools and open source and things like that. And we're just looking to have the largest impact we can and try to help move the field forward as much as we can.

52:29Sam Charrington:Well, Scott, it is always great chatting with you and particularly great catching up about, you know, this idea because it's an important one, you know, in the grand scheme of things, we want to put systems out in production and ensure that they work. And this is an important idea on ensuring that these statistical systems work in the ways that we want them to. And continue to work over time as the world changes underneath you. Right, right. Underneath us and them. Yeah. Awesome. Awesome. Well, thanks so much. Yeah, thanks so much, Sam. Always a pleasure. Cheers.

53:32you