The good, the bad, and the future of AI agents

0:28Hey there, and welcome to Decoder. I wanted to have David on because earlier this week, Anthropic released a brand new AI model called Claude Sonnet 4.5 that's been making waves. For reference, Claude is to Anthropic what ChatGPT is to open AI. This new model, Sonnet 4.5, is being built as a big breakthrough in autonomous agentic AI, especially for coding purposes, which is a big battleground in the AI market right now. All these companies want to get a slice. These types of AI products can, in theory, be given complex tasks and then go off and complete them over the course of many hours, or in some cases, even multiple days.

1:07And Anthropik says this particular model, Sonnet 4.5, can run for up to 30 hours straight without any human intervention, all while working on a singular task like building a software application from scratch. For the last year or so, companies like Anthropic, Microsoft, OpenAI, and others have been promising that this agentic technology would be the next phase of AI, the next big hype-filled thing that comes after general-purpose chatbots. They say it could really unlock generative AI's potential, and it's true they've made some strides. But as we've seen so far, agents aren't quite there yet, and they have a ways to go.

1:46Most of us are not, in fact, sending agents off on the internet to do our bidding, and we're certainly not giving them tasks that might take 12 or 24 or even 30 plus hours of autonomous work without human handholding, at least not yet. At the same time, many companies are looking at agents as the breakthrough that's supposed to unlock huge productivity gains from AI models, including the opportunity to use them to replace or augment human labor. So I wanted to sit down with David, who spends a lot of time testing out what models like Claude Sonnet 4.5 can and can't do to ask him where we are on this promise of AI agents.

2:22I wanted to talk about what these types of products are good at from a consumer standpoint beyond just programming purposes and also what the path forward looks like as AI agents progress. Okay, here's Anthropics David Hershey on the state of AI agents. Here we go.

2:52I wanted to ask you about your view of the current state of play for AI agents. We hear all the time that agents are the next big thing for generative AI, but are we still in the prototype stage, the testing phase, or what? How would you characterize what AI agents do today, right now, relative to what AI companies actually want to offer in the end, which I hear from a lot of execs is basically Jarvis from the Marvel movies. I am less confident in our ability to project the end, but I'm happy to talk about it now. I've seen agents come a long ways in the last year working with customers. And my view is there are places where we're starting to see what it looks like when agents work really well.

3:36and there are still a lot of places where they don't work really well. And I think that kind of makes it confusing. I think for some people, code is a great example. When you're writing code, and especially with Sonnet 4.5, when you watch the model spend a lot of time as an agent developing software itself, it's incredible. It can do a ton. It's gotten much better literally this week. As we release models, it's really visible and obvious how much better they're getting sort of if you're plugged into that at being an agent doing really long-running complex tasks. When you look at other sections of the economy or other like jobs that people want agents to do large pieces of, sometimes there's stuff they're still not good at.

4:16In some cases, they're not good enough at deciphering what's on a computer screen yet to be able to navigate complex UIs or whatever it is. And so they fall over and stumble over themselves on something sort of silly. And it's easy to point at that and say, what are agents all about? Like, is this kind of fluff or hype or whatever it is? And I think the way that I see this sort of generally happening is we've like slowly been ironing out the kinks of the stuff that models fall over themselves on. And we've done that the best so far in coding. I think the industry has done that the best so far in coding where we're making really fast progress on how much an agent can accomplish when writing code.

4:57And we are, I think, starting to make that progress in a lot of other domains. For example, like I talked about clicking on UIs in a computer. This model, Sine 4.5, is way better at that. I don't know if it's exactly the point we're going to tip into people trusting it to automate a whole bunch of stuff they do when they're clicking around a browser yet, but we're getting there. And so I think my general view is we're making really fast progress. It's just not necessarily visible in every part of the economy yet. And then every job and every person and every individual, I have a feeling sort of each model that comes out will get one bit closer to being something that everybody can sort of interact with and see.

5:34So it's great at developing software. It's great at coding. It kind of reminds me of the robot hand moment, you know, how robots can do things that are really, really complex and hard for humans, but actually grasping something has always been a real headache. What about consumer facing stuff? You talked a little bit about UIs, but what do you think agents right now are the absolute worst at? What's the simplest thing that they truly just cannot do? I honestly sometimes struggle to put my finger on this because I think it's surprising little funky things in a lot of different agents. So for example, if you're trying to do something related to finance, maybe there's a bit of manipulating a spreadsheet that is really hard and it falls over.

6:18And so it's like 99 % of the stuff it can do, it can do the math and it kind of understands how a finance model works, but it will stumble over a spreadsheet. This is the thing you talked about. I don't know if there's like, I think it's hard to boil down all of the jobs that we want to help people do. And one tiny little thing that the models aren't there. If it was that easy, I guess we would probably be on top of it already. And I think like maybe a better model is for each of these different things that we want models to help us with. There's like a million little components it breaks up into.

6:47You need to be able to see the right cell on a spreadsheet and know how a formula works and know the macroeconomic model. And thing by thing, you can find in each different task the little thing that's not quite there yet. And so as we think about it, coming from Anthropic, when we think about it, it's like we just have to sort of be able to iron out, like fix each one of the little gaps in each of these to help everybody use it. But I honestly, this is probably an unsatisfying answer, but I can't quite put my finger on like there's just like this one thing. I think this is why this field is hard.

7:17We're just sort of like constantly working on the whole universe of the stuff people do in a remote computer and trying to help them out. And that just means it's a really wide scope. And there's just a lot, it's like from my side, it's a lot, it's fun. There's a lot of hills to climb to try to make the models better at all of the things that we wish they were good at. So we're working on all of them at once. Being someone that works with a lot of different startups and a lot of different industries on how they're actually applying AI, what are the industries that have surprised you the most? What sectors are clients in that you just wouldn't really expect?

7:50Or maybe the ones that you've seen the most growth in in the past six months, the past year. What are the trends you're seeing in terms of which sectors are really adopting agents at scale? I have two versions of this. but I'll give you the fun one first. I think like one of the domains that surprised me the most in my personal customer work is the legal domain, which is at some value, at face value, it's apparent why that can be really useful. There's all this information you need to know about case law and studies. And it's just fusing information is something that's like pretty obvious models are good at.

8:26But actually, there's so much depth and complexity to the legal field, which I didn't appreciate. I'm not a lawyer. I didn't appreciate it until I started working with people in the legal domain. I used to write a lot about legal AI and how the law sector was kind of the last to adopt a lot of AI tools because they were super old fashioned. I did a couple of trend pieces on that. And then when they started to adopt it, I was really surprised at how quickly it came. Yeah, that's exactly what caught me off guard. Honestly, there's so many things that make it hard. Writing a really good legal system, you typically need a lawyer to tell you if it's good or not.

9:00Having to have lawyers in the feedback loop of how to build is challenging. and so I've been really impressed with a lot of the companies I've worked with and their ability to sort of like work around that where they like have this interesting build of companies that have like lawyers on staff to help with product building like like like big amounts of lawyers on staff to help them build products they build agents and cool things like be able to like comb over case law and look for the right details and and pieces and then obviously use the stuff that AI is obviously good at of synthesizing answers at the end but how quickly that field I think it's just like a it's probably like the scale of the upside there of how much just the pure volume of work that needs to be done and how much that it can help that has driven the speed that people go but that domain has surprised me i promise you a two-part answer my second part of this answer is i think the funny thing about this space is it's really hard to guess where the next agent is going to take off because of that thing we talked about before or i talked about before where it's like sometimes there's just this like one little tiny thing that's not super obvious to any of us that's blocking an agent from working so like if you wanted a model to do your taxes for you i think that'd be very nice i don't look forward to doing my taxes every year and you just find out that like actually what it's really bad at is you upload your w4 and it can't quite see the difference between the two boxes on your w4 and that's why the whole thing doesn't work you know this is a toy example i don't think that's actually where models are but yeah that It would be pretty complex.

10:32I remember when I worked in two states or I had a job in one state and I lived in another, it was also really hard for me to figure that out. So not surprising that AI can't either. Yeah, it is complicated. But I think we can get there. It feels in the domain of what the models can do. I've seen them do much more complicated. I don't know. They're better than me at math by a decent amount. So I would expect them to be able to do this. But sometimes it's just like this one little thing. And so I'm kind of constantly surprised. And it mostly is around when we release new models. Like, it turns out there's some set of things that the model can do now.

11:04And then you see new agents crop up that we're doing things we couldn't do before. And so there's like nice little micro surprises that come out. I don't spend a lot of time thinking about accounting normally in my day job, but then you run into an accounting startup that can suddenly do something new and interesting. And that's pretty cool. Do you think that data annotation is going to be a big part of that? Do these models get tripped up because there's some data that they don't have enough of, especially with a niche industry? Is that kind of where you see some of the obstacles come into play?

11:32I think we need to learn from specialists, and we need great ways to learn from specialists. And I don't know if it's necessarily data in the classical sense that there's some pile of data that we're going to learn from that makes it better, or there are just better ways we need to incorporate the intelligence of accountants and lawyers and other people into our models. Yeah, I think that could come from data. Some of it certainly will. I think that can come from talking to and interfacing and working with people in those domains. And I think a lot about learning directly from our customers. I think there's a future where more companies can contribute more directly to making the models do the stuff that they care about.

12:14It would be really nice if someone saw this big, important thing that they'd like to achieve instead of having to sit around and wait for a lab to hopefully build a model that helps them do it. They can sort of like directly work with the lab. And I think that's probably somewhere that, well, I know that's probably somewhere that we'll head where we can have more direct mechanisms of working with sort of experts. And yeah, it's partially a data thing, but I also think it's just like, I don't think it's surprising that one of the things that models are great at today is software engineering. When the building that I'm in, in Anthropic headquarters is filled with software engineers who know how to make models great at software engineering because they write software all day, you know?

12:52Yeah, that's true. And yeah, exactly. Like it's, it's deeply unsurprising that that's true. And I think as we grow up as a company that's only a few years old and has spent most of our time hiring software engineers and figure out how to consult and work with more people in more diverse places and doing more diverse jobs, we'll get better at building models that are great at all of the other things that people wish models were great at. We need to take a quick break. We'll be right back.

13:27Support for this show comes from LinkedIn. Imagine if any of the movies that included the line, I need the right person for the job, settled for, I'll just take about anyone. How many heists would have failed? How many deals would have fallen through? How many secret spy missions would have ended in disaster? So why would you accept just anyone when hiring for your business? When you need the right person for the job, you can turn to LinkedIn Jobs. And now LinkedIn Jobs is stepping things up with their new AI assistant. so you can feel confident you're finding top talent that you can't find anywhere else.

14:01With LinkedIn Jobs' AI Assistant, you can skip the confusing steps and recruiting jargon. It filters through applicants based on criteria you've set for your role and surfaces only the best matches so you're not stuck sorting through a mountain of resumes. Hire right the first time. Post your job for free at linkedin.com slash partner. Then promote it to use LinkedIn Jobs' new AI system, making it easier and faster to find top candidates. That's LinkedIn.com slash partner to post your job for free. Terms and conditions apply.

14:43We're back with Anthropics' David Hershey discussing the landscape for AI agents. Before the break, you heard David explaining some of the trends he's seeing, both from his work internally at Anthropic testing new models and from clients who are now using this tech in their industries. But now I want to ask David about Anthropic's big announcement this week, Claude Sonnet 4.5, and why it's being billed as such a step forward for AI agents. Just released Claude Sonnet 4.5. And to me, that was a big deal. I'm really in the weeds on this stuff. So I was really into all the specs. But for the average person, that's just a word and a number with a decimal and another number.

15:26So why is this a big deal? And how does it differ from your latest models before that? What's the meaningful difference and the meaningful step change here? Yeah, I'm very excited about SONI 4.52. And I'm also very cognizant that when my mom texts me asking about it, it just is a model with a decimal and a number after until I talked to her too. So there are a lot of things that I think are exciting. One thing that I want to say up front before I get into a lot of details that like have jumped off the page to me about the model is it's sometimes hard to predict like what the big impact is going to be on everyone.

16:04These models get generally smarter. They get generally more capable. and not to keep harping on the same point, but sometimes we just don't know the blind spots that they had until we get over them. And so when we released a model last year, Sonnet 3.5, and suddenly all of these vibe coding startups happened, I don't think we actually knew that that was true. We didn't know that there was going to be, the moment we released that model, I don't think we could have predicted that so many companies would crop up and be able to help people start writing code with agents in this amazing way that we've seen happen.

16:34in. So part of what is exciting about Sonic 4.5 is the unknown to me. I'm really confident this model is the smartest model we ever created. I'm really, I've seen, and I'll get into some of the things I've seen it do that I've never seen another model do before. And that's really cool. And part of it's concrete stuff. We'll talk about, I'll tell you, I'm excited to talk about some of the software engineering stuff I've seen it do, but some of it's stuff that we really don't know until our customers and the people we work with try to build cool new things that they couldn't make work before and then they suddenly make a big work before and so when i if i had to guess how my mom might see sonnet 4.5 and it might make a difference to her is that there's some product that she couldn't use before or didn't exist before in the way that perplexity has happened in the past for consumers or or cursor exists now for developers that it's going to happen that's going to impact your life because this model is capable of a new thing that we didn't know before And I have a hard time predicting it.

17:28It's part of the fun part and the hard part about being someone who works for customers now is it's really hard to predict this stuff the day we launch a model. But it tends to happen. And that's cool. Have you seen any trends start, even though it's been like one day? Anyone that's starting to use it, you know, in a new way, one of your clients or even in beta testing? Too early for customers, I think, to see anything brand new. It typically is like, it's a very fast field, but I have like a give it a month rule to find out what the new companies are going to be. My testing, I have certainly seen some new stuff.

18:02And with the testing some of the team has done, that is really exciting. One of my favorites to talk about is my team has been working on seeing how much of a software engineering task a model can take on at a time. And so I think we're all aware that models can be used to write code. like that's sort of obvious to a lot of people at least in this industry and probably people listening to this podcast at this point but it's often in pair with a human really directly like going back and forth using quad code or an ide like write one thing at a time check and review it and the longer you let a model go trying to implement something big and complex the more likely it is to like do a whole bunch of stuff you didn't want to happen and so we were like really curious, what is the most we could stretch that?

18:45If I give Quad a really huge task, what can I accomplish? And one thing we've seen with this model is in a way that is really not true of models I've tested and seen in the past, if you give a sufficiently good overview of what you want a model to accomplish, I haven't really seen a ceiling on how much it can keep Costa consistently working on making something better. And so my favorite actual example of this, We released a video of this yesterday, but I'm going to talk about this until people get bored with me. We asked Claude to recreate Claude.ai, our consumer chat application, from scratch.

19:24Sonic 4.5 just worked overnight. We woke up and it just did it. This beautiful clone of Claude.ai that works incredibly well. in my favorite moment from it there's a feature that people like of ours called artifacts where when you ask quad to make a document or make a web page it will make it and render it next to the chat so you can like play with an app that you built live or whatever it is and i was saying to the person on my team who was working on this demo like work on this thing it'd be really like cool if quad could like build artifacts itself that would be amazing it's like a complex feature it's pretty hard to figure out.

20:02It's like, we should, we should try and see if that happens. And then he messaged me like two hours later and he didn't do anything. He didn't intervene. And he's just like, Hey, quad just built artifacts on its own. And it's like currently testing it live. It's just like trying it out itself. And I really, it's just like, we released artifacts a year ago. It was like artisanally created by a whole bunch of like wonderful engineers around me at Anthropic who were doing all of the software engineering to build this really complicated things themselves. And then a year later, we released a model and we asked it to build Quad.ai and let it go for 12 hours and just did it itself.

20:41Instead of this giant team of people at Anthropic that work so hard on this thing, the model is just really capable of biting off really meaty, complicated tasks. This is something that would take me months to do if I did not have quad. And overnight, we sort of looked at it and watched it happen. And that progress of just like, like, going from a point where it was neat that a model could write a snippet of code for me a year ago to like, oh, it can do like a big chunk of the stuff of the complex developer work that I need to do. It's just like, I don't know, it blows my mind. It was like a pretty big, Wow.

21:21So that took about 12 hours, you said? That specific one was like a 12-hour thing. We've seen up to 30 hours of continuous dev work. Okay, this is what I was going to ask you about, because something that unexpectedly went viral from my own article about Sonnet 4.5 was the 30-hour bit, the fact that Sonnet 4.5 could code autonomously for up to 30 hours with no interruption. So I heard one engineer at Anthropic used it to code a chat app that the company compared when they talked to me to Slack or Teams. Obviously, it was only 11 ,000 lines of code, so much smaller than Slack or Teams. And it seemed like just an example project.

21:58But people online are really excited about that detail and calling for Anthropic to release it, and they want to know more about it. So can you give me any details? Was that someone on your team that was testing it? And how impressive was it really? Or was it just kind of rudimentary? Give us the deeds. Yeah, yeah, yeah. This is the thing that also, I don't know, like all of my excitement and the reason I'm years because I've been working on this exact thing. So it was someone on my team. His name's Justin. Shout out Justin. You should give praise to him. He's amazing. Recently joined the team on a side note.

22:26He's pretty good. Nice. This was born out of this thing of a lot of people have built demos or proofs of concept. And there's this sort of vibe coding trope that you can, yeah, you can build it, use it to quickly mock something up, but can it really like build a real application? Justin really wanted to like test that and so he was experimenting using our quad agents SDK which is just sort of like a more programmatic version of quad code to some extent he was testing like can I give a full spec of like a complex thing like an app similar to Slack to the model and just like watch it build and we did some takering and experimentation around it to get it right but But yeah, you asked, was it impressive?

23:09It's impressive. It has DMs and threads and channels and a slick search functionality. And you can upload images and GIFs and render them. And multi-user authentication. And Claude, we didn't ask it to, but implemented a whole bunch of AI users for testing. So if you log in, you can send a message. And there's Alice, the PM, is in there that you can send a message to who will respond to back stuff about PM work. it is it is remarkable it is like it is not by any means slack but like if you didn't spend a lot of time thinking about it you'd like look at it and think that was a pretty reasonable productivity app that you would use to chat with your co-workers wow what did you guys name it i don't think we have a name i need to ask justin to give it a name we need i actually no i lied he has a work chat very boring we're engineers we're clearly not the product folks in the org so So I love that.

24:06Yeah, someone who worked at Slack, I think, messaged me and said, you know, our code base is a thousand times that. And I was like, yeah, it's just an example. But I mean, it happened in 30 hours. So it's a big deal. I was also going to ask you what surprised you the most during your own testing of it, of Sonnet 4.5. I'm just going to continue double clicking on this thing. The way that it built the app was surprising and really interesting. I'm not surprised that the model is getting better at building complex apps. the thing that was really interesting for the way it does this is it just has a tendency to like bite off little pieces that it can handle really well and do that continuously so a lot of times like models in the past would get really eager and ambitious they would say like i'm going to build this whole thing and i have like these grand ambitions and it would kind of just like meander everywhere trying to do this miraculous piece of work and the thing that was really cool about outside of 405 is just like pragmatic kind of like it it's like okay right now I'm going to test like does image upload work and then it's going to do that it's going to spend a little while doing that but it just like bites off one little chunk at a time and that feels a lot more like what I want a co-worker to work like or a collaborator work like it's like if I ask you like hey I need you to go build workshop I don't want you to go off on this like crazy escapade trying to make everything magical I want you to just like bite off a piece at a time show it to me, like committed to get like that kind of thing.

25:32And it's just a little bit more natural and collaborating with feels with, it feels more natural. And funny enough, like, I think this is unrelated, but we've been chatting in our internal company Slack with Claude a lot lately and just like chatting with it. And it's been really natural. Like it feels just a little bit more like working with a coworker does. In terms of the tone? Yeah. Like the, the tone, how it, how it responds, like how it acts in Slack, how it, how it participates in a conversation, what it tries to do. It's a little less like over the top and eager that has jumped out a little bit, which was surprising.

26:07Like that's not something that I normally expect, but yeah, it's also been funny, I guess, which is a funny thing. Like it like cracks better jokes. Like it's a little bit more witty, that kind of thing. Do you think it also dialed back the sycophancy a bit? You know, I know that's been a big word in this space lately in terms of, you know, being over eager, over validating? Did you feel like it was more real and that it was less like that a little bit? I have felt that. And I think it's certainly something we're pressing on. I, it's like one of my least favorite thing about every model is when there's a phantic like that.

26:37I think it just gets in the way of doing good work. And it's been a big focus as far as like, I think nobody ananthropic likes that. And, and also just like, I think it's, it's bad for the, it's just bad. So I do think we have made meaningful progress and this model seems to be a little bit more willing to push back. And that's part of that thing. Like being a natural coworker is someone who actually can tell you when you're wrong. When it comes to rebuilding claw.ai, that's pretty big. Did that worry you at all? Because, you know, it's kind of doing, like you said, you know, organic work that you worked on for months.