AI Agent Failure Modes (The Agents Season, Episode 6)

0:01Hi, welcome to Linear Digressions. Today we are going to talk about the many, the varied, and the prolific ways in which AI agents can fail. If you're paying attention to marketing hype, then you could be forgiven for thinking that AI agents are perfection itself and they never make a mistake. But if you have ever used an AI agent, in all likelihood, this is not something that's shocking you, that AI agents can make mistakes. But nonetheless, it's, as you might expect, a pretty rich topic. It's one that we're going to be exploring in some detail today. This is the sixth episode in an entire season arc here about AI agents.

0:45Very excited to talk about failure modes today. You are listening to Linear Digressions.

0:53As I mentioned, we've been talking for several episodes in a row here about what agents can do. They can use tools. They have memory that they can manage. They can plan. And this episode is going to be, let's call it a counterweight to all of that. It's going to be a little bit of a wet blanket, a little bit of a reality check. Of course, if you want to actually use agents for anything, then studying what their failure modes are and measuring the failure rates relative to the success rates is going to be really important for engineering robust and reliable agentic systems. and reflecting that there's plenty of interesting research that we're going to go through today.

1:35So in this episode, we're going to talk about two distinct things going on when agents fail. The first is mathematical. It's a property of any multi-step system. And it's something that in my opinion, doesn't always get enough attention, but there's a second type of failure or aspect of failure that we're going to talk about, which is taxonomic, which is to say, what are some specific named failure modes that show up repeatedly across different agents, different frameworks, different tasks, all of that. So we'll cover all of that today. And at the end, bring it back to some real benchmark numbers and maybe your intuition for whether agents are getting better or they're still kind of garbage or somewhere in between.

2:17First, let's talk about math for a little while though. I want to start with something simple, a mental model. Go with me for a minute. Imagine you have an agent and it is 90 % reliable at each individual step it has to execute. And 90%, by the way, is quite good. Maybe even a little bit optimistic for most real tasks. So nine out of 10 times it's doing the right thing. One out of 10 times it makes a mistake. What kind of success rates are we looking at for a 10 step process? You have a task that's going to take 10 steps to execute. How's that going to look? Well, if you make the simplifying assumption that each of those steps are independent, then you can do some simple math.

2:56You say that the probability of getting all 10 right is 0.9, 90%, and you raise that to the power of 10. So it's 90 % on the first one. It's going to be 90 % of the success rate of the first one on the second one, 90 % of the first and second on the third. you do that math and you work out to about 35 % overall success rate for a 10-step process. So you start with 90 % per step reliability. By the time you get to the end, you're at 35 % end-to-end reliability. So you have a task that's mostly reliable at each step, but it's still unreliable overall. And as you might imagine, you push that out further, you start to get into more complex tasks that might have more steps.

3:38If you want a 100-step task, which is in the realm of what a coding agent might run for a complex feature implementation. If you have a 90 % per step reliability, you're essentially at zero end to end. It rounds to zero. And 99 % per step, which is extremely reliable, better than most real systems, you're at 36.6 % for 100 steps. So put another way, you need four nines of reliability, 99.99 % per step to get above 90 % end to end on a 100 step task. But wait, you might be wondering if it makes an error in one direction at one point, could it make an error back in the other direction at some later step and those errors cancel out?

4:24Like if you make a mistake and then correct it, doesn't that help? And the answer is sometimes that can happen. But the more important dynamic is that when the agent starts to make mistakes, it's changing the state of the world that subsequent steps operate on. In other words, if you have an agent that makes a mistake at step three, it's not just wrong at step three, it's now reasoning from a mistaken or a corrupted state at step four, at step five, at step six. So we might call this like a propagation of the error that it passes through to the later steps in the process. And by the way, the agent often doesn't know that it's propagating some mistake that it made earlier on.

5:07It doesn't have like a ground truth to compare against, it just keeps going with this mistaken perception about what the state of the world is, in all likelihood, creating further errors as a result of that mistaken perception. And this is something that if you've only worked with chatbots, it's going to be pretty different when you start to work with agents. Because when you're interacting with a chatbot, if it starts to wander off, you can catch that right away and kind of pull it or guide it or nudge it back onto the path you want it to be taking. But if an agent starts to take a wrong action and changes something in the world and then takes another action based on that changed state, then in all likelihood the problem might actually stay under the surface.

5:51It doesn't actually become visible until you're 10 steps down the road and it's going to be really difficult to trace back to the original mistake. So you take all this math and we can make it concrete with a benchmark. As you know, I love benchmarks. And there's one here that I am delighted to name drop, which is Tau Bench. Tau is, and they use the Greek letter Tau, but it's kind of an acronym. T-A-U, Tool, Agent, User, Bench. This is designed by who else but Shen Yu Yao, our hero for the season. I think this is the third or fourth time we've mentioned his research. So drinking game. Every time we say Shen Yu Yao, take a drink.

6:32So anyway, TauBench, back to regularly scheduled programming. So this is a benchmark that's aimed at agentic tasks that are simulations of what real agentic tasks might be. In particular, they give it a set of tasks related to customer service in retail and other tasks are related to customer service in airlines. So things like returning a purchase or changing an airline reservation. The agents have access to tools like APIs and policy documents. And the measurement is with a simulated user task, what is the success rate of the agent using those tools to successfully complete the task that has been requested.

7:21And there's something really interesting about tau bench that gets back to the math that we were just talking about, which is that tau bench isn't just about whether the agent is able to successfully complete that task once. But instead, they introduced this formalism called pass at k. This is a metric that they define in Taubenge. If you see this in the literature, it's like pass with a little carrot sign, like an exponential, and then the letter k. According to Claude, you pronounce this pass at k. So anyway, pass at k is where they do repeated runs of the same task multiple times. And k is the number of times that they repeat the task.

8:01So if you pass at six, the question is not just can the agent complete this task successfully one time, but can it pass that task successfully six times, succeeding every time? And once you start to introduce that sort of reliability measure into the metric itself, you see the performance of the agents really start to show that even when they can do some of these tasks once or twice or three times, when you start to ask them to do it robustly, get it right every single time, the performance drops off pretty precipitously. The benchmark was launched in 2024. At the time, the best models were GPT-40 class models that even on the base benchmark itself, just can you successfully complete the task, was successful less than 50 % of the time.

8:54And pass an eight score for the retail domain, for the questions that related to simulating retail interactions, those were below 25%. In other words, you'd ask the same agent to do the same task eight times, and it would succeed all eight times less than 25 % of the time. It's not 2024 anymore. The current leaderboard has improved, where the most performant models are now at 60-70 % or so. So we have made a lot of progress. but if you want to look at it in kind of a glass half empty way that we're still seeing that the most performant models are failing a third or even more of the time at realistic customer service tasks and that's even before you account for consistency the pass at k numbers are even lower so we've got compounding math all over the place impacting how well this is working So those tau bench numbers are related to a couple of representative classes of customer service tasks.

9:59But another type of AI agent that's very common, probably the most common type of AI agent, is the coding agent. So I want to zoom out now and do kind of a broad picture of what the numbers are actually showing in terms of the performance landscape right now for coding agents. And I think this is really important because any one benchmark, it's really easy to find cherry-picked numbers or things that give you just a little slice of an idea of how performance is evolving. And I think one of the best things you can do to understand the overall performance of these systems is try to keep sort of a broad view of a number of different benchmarks.

10:40And you'll get different stories depending on kind of how you look at these numbers and interpret them. So on coding tasks, the headline number that you'll see most often is from Sweebench. In particular, there's a version of Sweebench called Sweebench Verified. We've talked about this before on some of our previous episodes, but the idea is these are software engineering tasks. They're taken from real GitHub issues, and they have sort of correct or working solutions that are used to benchmark the AI responses. When this benchmark was launched in 2023, the state-of-the-art model from our friends at Anthropic was CLAWD2.

11:23It resolved less than 2 % of the issues. We've come a long way since 2023. And by late 2024, we had the best system starting to cross the 50 % threshold. Now where we are in the early part of 2026, the leading reported scores are above 80%. And as of the most recent benchmark result that I could find when I was recording this was about 94%. So that's a pretty remarkable trajectory. Let's take a step back and say what we just said. We just said that we went from 2 % to 94 % on this particular benchmark in terms of the success rate in just a few years. In fact, you may even be wondering if we've solved agentic coding, like 93, 94%, like it sounds like we've solved this problem.

12:17But this is where it's important to keep in mind that benchmarks, any one benchmark isn't going to give you the whole picture. A few things to keep in mind. So the first is that SWE Bench Verified is pretty targeted at these relatively narrow and well-defined tasks. So there's a specific task, it's like a ticket, fix this bug, resolve this issue. And they've also selected problems for SweetBenchVerified that have verified answers. So it's, which is really great for a benchmark, because that helps you know if the AI has gotten it correct, there's sort of a ground two or three to compare it to. But what it doesn't do is capture the full breadth of what engineers actually do.

13:00There's another benchmark that we could talk about in that context, a SWE bench mobile. The aim of this benchmark is to evaluate coding agents on more challenging software engineering tasks, ones that are representative of industry level mobile app development, not just bug fixes, isolated bug fixes. So we have some tasks here with SweetBenchMobile that are more representative and finding that even the best configurations still achieve only about 12 % success rate. You're using the same frontier models on this harder and more realistic task. You're getting a dramatically different result, 12 % versus 94%.

13:44The other thing to keep in mind when you're comparing some of these numbers or benchmarks is that it's really hard and a little bit misleading maybe to directly compare them across the different vendors and tools and systems. So you have the LLMs, the models themselves. That's sort of most often where you're reading about these benchmarks. And new model comes out and they have a table of benchmark numbers showing that the new model is doing better at all of them or whatever. But in the context of actually making a working AI agent, there's all this other stuff, all the stuff that we have been talking about over the last three episodes, that makes a huge difference.

14:22In some ways, that's as important or maybe even more important in some cases than the LLM itself. So what tools does the LLM have? How do they do memory management? How does the LLM connect to different systems? Like all of these are going to vary and they can significantly affect the scores that you get, significantly affect the actual performance of the model. In fact, in some of the research I was reading, you actually can see as they work through different permutations of models and scaffolds. So, for example, using clawed models versus OpenAI, but then also swapping them in and out, combining them in different combinations with, say, codex or cursor.

15:06what they show is that through those different combinations of model plus scaffold or harness, you can actually have very different results. The same model, the same LLM can show up to a six-fold performance difference depending on the age and scaffolding around it. So when you see a benchmark number, like there's a new model, it achieved 80 % on Sweebench or whatever, you're not seeing a raw model capability number. You are seeing the model plus the specific agent design around it, plus a specific set of tools, plus a specific evaluation setup. And it's really difficult to untangle those and just say in isolation, what is the contribution from the model itself.

15:52Last and not least, the last thing that I will point out here is that even for Sweebench Mobile or Sweebench Verified, these benchmarks that are aiming to get as close as they can to something that's representative of quote unquote real engineering tasks, they're still falling pretty far short of what an engineer's actual job is. So when you're comparing agent results to say human programmers, still a pretty narrow comparison. most human developers do much more than resolve github issues with a fixed context window they they're navigating ambiguous requirements they're talking to other people they're making decisions about the architectures they're going to use and different trade-offs of implementation choices they could make they're you know noticing when something feels weird and fixing it before becomes a bug.

16:47So with something like Sweebench or Taubench or Sweebench Mobile, you're each capturing kind of a little piece of that, sometimes complementary pieces, but there's nothing here that's telling that whole story in one benchmark. So you put all this together, and the picture that you get is that for narrow, well-defined, verifiable tasks, especially in the context of coding or software engineering, agent performance has improved dramatically, and it's continuing to approve. But for broader and more realistic tasks, where there's a requirement of sustained judgment, coordination, following policies, consistency across many interactions, the research suggests that agents are still failing on a significant fraction of attempts.

17:34So there's the story that you might get if you're just paying attention to the headlines of the best benchmarks. But if you're thinking about real deployment, you're still going to be seeing a significant gap. All right, so now we've talked about the mathematics of agent failures and how we measure this. Another interesting research vein is trying to taxonomize what's actually going wrong. This is where we're going to dig into a 2025 paper from UC Berkeley, where the researchers systematically cataloged the failure modes of multi-agent systems from the ground up. A brief digression in the methodology, because I'm a big nerd for these things and I love it, really enjoy understanding how they measure some of this stuff.

18:22So these researchers, they got their hands on and analyzed more than 1 ,600 execution traces across seven popular agent frameworks. So these are the logs of the actual agent runs step-by-step, what the agents did, observed, tried, you know, their reasoning, their planning. They gave these to six expert humans who annotated them and identified the failures. They had kind of this iterative process until the humans were generally in agreement about what sorts of failures they were seeing in these traces. And they put this together into a taxonomy with the acronym MAST, Multi-Agent System Failure Taxonomy.

19:07I guess the F didn't make it into MAST. I guess MAST is a little bit better than MAST. Or MAST. Yeah. MAST. Multi-Agent System Failure Taxonomy. And they found 14 distinct failure modes, and they organized them into three categories. So let's spend a second on those three categories because I think they actually help you think about the different types of agent failures that you should be on the lookout for. First of these is specification and system design failures. So these are ones that in a sense happen before the agent even gets started or they can happen very early. So this is where things happen like the agent disobeys the task specification, like you tell it to do one thing and it ignores those instructions or it violates them or something.

19:54This also includes things where agents are repeating steps unnecessarily, steps they might have already completed, or they get stuck in these loops that they can't break out of. You can also get failure modes like losing conversation history, manifesting some of the things that we've talked about where context can get compressed or compacted, or you lose detail or stuff gets lost in the middle, that sort of thing. So in general, what you have here are design and configuration failures. So the agents were set up wrong, the task was specified ambiguously, context management was inadequate, something like this.

20:31The second category of failures are inter-agent misalignment. So that's when you have more than one agent and the failures arise when there's coordination that is supposed to happen that doesn't get successfully executed. So these are things like there's one agent and it produces an output that it needs to hand off to another agent to take the next step in the process. And the second agent can't parse it for some reason. Or it could be cases where there's two agents that reach contradictory conclusions and there's no way to resolve the conflict. Or maybe there's an orchestrator that's giving instructions to multiple different sub-agents.

21:15and those sub-agents get confused or there's maybe some ambiguity in the instruction and they start to wander off in different directions. So obviously this is a set of failures that's pretty specific to multi-agent systems. You're not going to get these sorts of failures when you just have the single agent working alone, but this can actually be a pretty significant contributor to the failures of the more complex agentic systems. Third and final type of agent failure that they see in MAST is related to task verification and termination. So basically knowing when you're done. The agent doesn't know when it's done or it thinks it's done when it isn't, produces an output that looks like completion, but it doesn't actually satisfy the task requirements, or it can overshoot, It keeps going after it should have stopped.

22:08And it's like over generating, do way more that you need producing work that wasn't asked for. I want to come back to what I said briefly a second ago about multi-agent failures. This is actually one of the things that I think is most interesting, not necessarily intuitive, but probably actually really important for anybody who's designing agentic systems, which is that you might think that multi-agent systems, which is where you have multiple specialized agents that are collaborating and coordinating and checking each other's work, you might think that those in general do better than single agent systems.

22:45But the research in this paper suggests that at least for what they were looking at, you don't necessarily get better performance from those more complex systems. In fact, sometimes the multi-agent systems are doing worse than the single agent systems that are working alone on the same tasks. This is sort of counterintuitive because there's sort of a common wisdom that having multiple specialized agents, each of which is really good at a single task, and they can work in parallel and check each other's work and do all of this nice, complicated stuff, that that should add up to a more effective system overall.

23:26And sometimes this does help. But because when you have multiple agents having to work together, that introduces this entire second category of failure, the interagent misalignment. And none of those failure modes exist when you have one agent working alone. Like you just don't have the same kinds of handoff errors or miscommunications that you get when there's just one agent that has all the contacts themselves. This is kind of intuitive if you think about two people working on a problem together versus one person working alone. You've just introduce the possibility of communication failures in addition to all the other failure modes that you might have for whatever the problem is that you're working on.

24:05And so the point here is that as much as you can introduce some nice resiliency and specialization with multi-Asian systems, you've also introduced this new failure mode, and that can significantly mitigate some of the gains that you might be getting from the more complex system. So it doesn't necessarily mean that multi-agent systems are always worse. It just means that the benefits of coordination, they don't come for free. And many multi-agent systems in practice are paying coordination costs and they are not necessarily all getting coordination benefits. So beware if you are building agentic systems, really think carefully about when you want to branch them out into multi-agent systems because you will start to pay some overhead for that design decision.

24:56With all of that said now, let's tie this back to probably the intuitive experience that you may have had working with AI agents. So the first thing to take away from all this research is that AI agents are getting dramatically better than they used to be. Like that signal is unambiguous from the benchmarks that almost regardless of the benchmark that you look at, AI agents are better now than they used to be. But at the same time, the pass at K metric, the idea of being able to complete a task successfully once is very different than being able to successfully complete it every time. That pass at K is still significantly lower for most complex agentic tasks.

25:45It's just a reflection to some extent of the math of multi-step tasks. It's also in all likelihood reflecting maybe some coordination overhead that some of these agentic systems are paying. But the point here is that as a user of these systems, even if the signal is quite clear that they're getting better, that's a really different thing from they get it right every time. And we're still very much not in a regime where they are getting it right every single time. second as i was sitting and reflecting on this a little bit there's kind of two narratives that i see in a lot of the online discourse about agents i was thinking about them a little bit how can these both be true which is you can see in literally the same thread you'll you'll be on hacker news or reddit or whatever and there's one camp of people that are saying like ai agents have changed my life.

26:45They are doing all this complicated stuff for me. I log in in the morning, get my agents going, go to get a cup of coffee, you know, come back and they've, they've coded up my new feature meeting, like, you know, living, living in the future in the best possible sense. That's one camp. And then there's another camp, which is like, these, I don't know what you guys are talking about, because these AI agents are garbage. Every time I try to ask them to do anything of any complexity. They just create this AI slop, and I have to clean it all up. And it's even worse than if I did it myself, and, you know, so on and so forth.

27:21So sort of like, okay, besides just people having personalities and preferences, and, you know, liking to fight on the internet, like what's going on here. And so one other thing that I think that's, I was reflecting on is, yeah, okay, in addition to the story about the agents getting better but not being perfect yet. I think there's a story here about also us as users getting more adept at identifying the types of tasks that agents are more capable at executing consistently or sizing our tasks or scoping our tasks so the AI agents are more likely to be successful in them. In other words, if you throw any random task at an AI agent without any particular consideration for like how to present it or how to prompt it or how to guide the agent through the process and where to check in on its results, yeah, chances are based on these results, you know, the odds are not in your favor that they're necessarily going to get it right.

28:22However, we are all learning how to use agents better. And so I strongly suspect that some of these folks, and I'll kind of put myself in this camp, When I'm having a really good experience with using AI agents, it's largely a reflection, I think, on what I have learned about how to prompt it, how to pick out the assignment in the first place, how to guide it through the conversation so that it's making the most of where it is capable and generally avoiding some of the pitfalls. So as much as the AI agents themselves, in combination with their harnesses and their tools and their scaffolds kind of have their own story of development and improvement, we as their users are evolving and learning alongside them.

29:08And so I think that's the combination that really can make a big difference when you put both of those pieces together and might be the separation between some of the users who are getting really great results because they've really settled into a workflow that works for them versus other folks who maybe in some cases are a little bit more frustrated because they're expecting the agent to be able to do the sorts of things that agents still are consistently not always able to do. All right, so with all of that, we have covered a lot of failure modes today. I hope you learned something. I certainly learned a lot when I was putting this together.