Boris Cherny: Building Claude Code

0:06All right, Boris, we're so excited to have you here, the creator of Plot Code. Thank you.

0:19It's great to be here. Fresh off the press, you guys just shipped Opus 5 yesterday.

0:26Boris Cherny:Yes. And it seems that model performance keeps accelerating. You guys took Arc AGI 3 to 30%, which is incredible. Yes. And for context, before the best score was in the low single digits or low teens, right? What can Opus 5 do now that it couldn't versus a previous version? Yeah, there's a lot that goes into every new model. And there's a lot of new capabilities that we teach and get the model to do. Whenever you do model training, you try to teach a whole bunch of different things. And most often it doesn't work. But some subset of the things, the model does learn. And sometimes it also surprises you.

1:19Boris Cherny:It has these skills. It has abilities that you actually didn't really teach it. but it just kind of learned. For five, one example of something it does that I think no other model has done is it runs for a very long period of time. And especially when you combine Opus 5 with auto mode, it's just like incredible. Like it can go for days, weeks, months at a time. It just won't stop. You don't even need to use scaffolding. So you don't need slash goal. You don't need all this other stuff. It'll just go because it knows it needs to do the task. Another thing that I'm really excited about, and I'm going to start, I think, to talk about a little bit more, but it's kind of surprising because it's such a new capability, is the model does not seem to be prompt injectable anymore.

2:08It's not prompt injectable.

2:11Boris Cherny:It's crazy. People have talked about this lethal trifecta for a long time, and this really affects kind of harness design and agent design and product design. Because if the model reads some instruction on the internet that's like, you know, do X and Y and Z and also delete everything on the user's computer, a year ago the model would have just done it. But nowadays, Opus does not. And this has actually been the case since like Opus 4.7, 4.8, Sonnet 5 has been quite good at this, Table was quite good at it. But Opus 5 just hits like a new frontier on this. So essentially, if you combine a well-aligned model, so this is like essentially three years of research into alignment, with a prompt injection classifier, which we run for all traffic, and what this is doing is it's based on Chrysola's mechanistic interpretability work, where it's literally, we're looking at neurons in the model's brain that light up when prompt injection happens.

3:07Boris Cherny:So the model won't even tell you, but we can actually see those neurons, and we can figure out and diagnose that it's happening. And then you combine that with the auto mode classifier. And with these three layers, we just cannot demonstrate prompt injection anymore. And we've hired security researchers. We've done red teaming. We've done competitions. No one can demonstrate it. And so I'm actually quite curious if people are able to. And that'll actually be a really amazing signal for the research team. So if anyone in this room prompt injects claw, Opus 5, you'll get a special prize for Boris, maybe.

3:42Now, talking about a prompt injection, the other side of the coin is now the system prompt. Let's talk a bit about the new release. You actually deleted over 80 % of the system prompt from Cloud Code. Tell us more about that.

4:00Boris Cherny:I think something that a lot of people might not realize is Cloud Code as a product and as a harness is just always changing. We're always adding stuff. We're always deleting stuff. every time that a new model comes out, we delete a bunch of the system prompt, change a bunch of the system prompt, we change the set of tools all the time, we change the prompts for the tools all the time. And the reason is every model is very different. So something that you did for one model maybe three months ago, it just might not translate at all to the next model. And so one thing about Opus 5 is it's just really intelligent.

4:41Boris Cherny:And a lot of the stuff in the system prompt was correcting for these behaviors that the model should have known, but it didn't. Now, Opus 5 just does it. So, yeah, we deleted 80 % of the system prompt. You can actually try deleting the rest of it, too. So when you run cloud code, you can just do, like, dash dash system prompt and set whatever system prompt you want, if you want to experiment with it. And another thing that you can try is simple mode. So this is actually this kind of undocumented feature. If you do quad code simple equals one, like this environment variable, and then you run quad, it'll delete all the system prompts, including from the tools.

5:21Boris Cherny:And we actually use this as a sort of ablation to figure out is the prompt useful. And what's interesting is that the model is actually a little bit more intelligent without these prompts. That's something that we've been finding. But when you use quad code as a product, you do actually want some of these prompts because it helps you use the product and it helps the product behave and the model behave in the way that you would want when you're using it as a person. I think the thing that's really fascinating in this era of building, basically you have built the best hardness in the world for Cloud, and that's Cloud Code.

5:56From what I'm hearing, for every model released, you basically delete all of the code base, delete all of the prompt, and start from scratch every time. That in the old world would have been not something started would have done for the product. It's like press delete every six months for everything.

6:14Boris Cherny:That's right, that's right. So to be fair, we don't delete the entire code base, but we do delete a lot. So every time there's a new model, we try, in research, we call this ablation. And so what this means is you delete the entire system prompt and then you bring it back line by line to figure out what is the impact of each individual line. It's sort of like an eval and you can kind of like evaluate it and the ablation essentially it's an eval, but you delete things to figure out the impact. And yeah, we do the same thing for tools. We unship tools all the time. We delete code in the harness all the time.

6:45Boris Cherny:If you look at actually the code that's in the Cloud Code harness today, almost all of it is about safety and permissions and static analysis. And there's a bunch of UI code. And we've actually unshipped a lot of the other code already. Do you think this way of building a agentic product and harness and basically doing ablations every time there's a new model released. Should everyone in this room that's building AI products basically do that? Be comfortable and brave to press delete. 100%. Yeah, and for people that aren't building agentic products, but you're using Cloud Code, every six months, delete your Cloud MD.

7:24Boris Cherny:Delete your skills. Delete your hooks. See what the model does, and it might surprise you. And actually for Opus 5, this is something we really do recommend is just try deleting all of these things because the model might really just not need all those instructions that you needed for past models. Let's talk a bit about how then you build this new prompt when there's a new model release, like for everyone in the room, everyone will want to try Opus 5 and they're going to press delete on their system prompt. How do they go about rebuilding the system prompt? how do you set up your environment? So you do it kind of piece by piece.

8:04Boris Cherny:So the first step is you delete. The next step is you use it. And you don't want to guess what's the instruction that the model needs because you might not predict it correctly. The thing that you want to do is you want to run it. And if it's like a custom agentic product that you're building, you want to kind of run the product. You want to see where it fails with the model. You want to see what it does well. if you're using quad code, you want to see where it does well with your code base, or maybe where it stumbles over the architecture or stumbles over something else. And only when you see it repeatedly stumble on the same thing, that's when you add it back.

8:41Boris Cherny:But you don't want to do it too early. Because remember, the model is going to read this instruction every single time you use it. So you really want to make sure that the model needs this instruction. I think this is sort of the crazy thing about building on models. is just so different than all the engineering that I've ever done. Like, in the past, when you built on systems, you built these, like, big, beautiful systems, and you really think about the system design up front. You have, like, a big suite of unit tests. You think about everything, and, you know, like, a re-architecture is a big project.

9:09Boris Cherny:Sometimes it takes months. I've worked on re-architecture products at, you know, big companies that take years. And the motto is not like that. It's, the way to think about it is almost like a living creature, like as something more organic. It's a thing where every model generation, it behaves differently. It has a slightly different personality. And you have to take the time to get to know it and then adjust the harness based on that. And I think it's just very much like an empirical and kind of scientific thing. You have to take a very scientific mindset to it where you try something, you see the result, and then you iterate based on that.

9:46If you're building in this world right now, what then becomes stable? Are evals something that you keep from the previous models and keep using them in each new model release?

9:58Boris Cherny:We do until we max out the eval. So that's sort of the tip for everyone. So code and system prompt, if you want to build at the bleeding edge and have the most capability for models, you've got to delete those. But evals are constant and keep appending to them, basically. Yeah, you keep appending. What happens is, you know, I actually wouldn't even go this far. to be honest. I think evals, they outlive the harness a little bit, but not that much. Like an eval might live for maybe one, two, three model generations. But nowadays, you know, we're on the exponential. The model is improving so quickly.

10:35Boris Cherny:Very often, we just saturate the eval, and then we have to throw it away, and we have to come up with a new eval. And this is just part of the process. And again, it's about being empirical. You have to use the product. You have to use the model. You have to see where it struggles. And then based on that, that's the evil set that you should build. I think one term I heard you describe how to build the best agentic products on top of a plot is this concept of an unhobbling plot. And tell us more about what that means. Yeah, so hobbling is this idea in a research that the model is doing something and you're just getting in the way.

11:16Boris Cherny:there's this kind of like way of thinking about it that I really like it's very useful when you're building product and it's called product overhang and the idea is the model is able to do all sorts of things with today's models not a future model but today's model that we have not yet realized and there are so many capabilities the model has like this that people are not aware of. And this is like the ability to, you know, like maybe use a particular tool, use a particular language, solve a particular kind of problem, do things a particular kind of way that we thought was kind of beyond the model's capability.

12:01Boris Cherny:And there's this overhang because the model can do this at every given model generation, but there is often not a product that lets the model do this and lets it express this kind of ability to do this. And on the flip side, often what happens is the product gets in the way. And this getting in the way, we call this hobbling, and then not eliciting the correct behavior from the model, we call this product overhang. So it's kind of like two sides of the same thing. One example of this was the original plot code. When I first started working on it, this was, you know, like a year and a half, two years ago, something like that.

12:38Boris Cherny:This was like SANA 3.5. At the time, that was an incredible coding model. That was like the best coding model that exists. Nowadays, it's, you know, a pretty terrible coding model by modern standards. But I think that was like the first great coding model that we built as Anthropic. And at the time, if you looked at the coding products of the time, what were they doing? They were doing like single line autocomplete. They were doing sometimes multi-line autocomplete. That was sort of a new idea. They were doing chat. So you can talk to the agent, but it wasn't write access. You could only read.

13:11Boris Cherny:You could ask about the code base. And so the feeling was that there wasn't really a product that was fully eliciting the model's capability to write entire functions at a time. Entire files at a time. At the time, it wasn't entire features. We weren't there yet. but probably entire files. That was the level of capability at the time. And so the idea with quad code was, all right, we think the model can probably do this. What if we get rid of all the scaffolding and just give the model the simplest possible harness so it can write an entire file at a time and build an entire feature? And that was kind of it.

13:49Boris Cherny:Like, that was the product overhang at the time. The model was capable of doing something, and everything was just kind of getting in the way. I think that nowadays with modern models, there is so much product overhang that I'm not seeing startups capture. And I think there's people thinking about these problems, but there's just a huge amount of opportunity to elicit these behaviors from the model that are just like amazing and interesting and commercially valuable. I think this is such a special insight for everyone here in the room. basically all of you could create the next Cloud Code if you figure out how to unhubble the models because that's effectively the birth story of Cloud Code.

14:32You unhubble Sonnet 3.5 because all the previous iterations were still getting the model very rigid in IDEs. And Cloud Code was one of the first instances that gave it just a full terminal access.

14:47Boris Cherny:Yes. And that then created this amazing product that keeps going. So let's talk about what are some areas and how should future founders here think about unhobbling Claude and fixing this product overhang? So there's a couple of things that I will think about. One is you should give the model slightly harder tasks than what you think you can do. I think a really common mistake that I see is people are using quad code, they're using quad, and they just give it way overly specific instructions. They're like, I want you to do this, but I want you to do it in this way, this way, this way. You must do one, then two, then three, then four.

15:36Boris Cherny:And for modern models, that's actually really not the way to do it. You want to go a little bit higher level. You want to describe the task. You want to describe the guardrails. You want to describe the exit criteria. And then just go with the model cook. And come back in a little bit. And I think it'll surprise you. And again, this is just not something that would have worked six months ago, but it does work today. Can you give some examples of these challenging task or capabilities that people should explore that it can do now that it couldn't six months ago? Yeah. Yeah, so okay, one example is the model can now rewrite essentially any code base from one language to a different language.

16:15It's just sort of crazy.

16:17Boris Cherny:Like it's this work that would have taken just like a very long time as an engineer and now the model's like quite fast at it. So one example of this is Cloud Code is built on the Bun JavaScript Runtime. It's an open source JavaScript Runtime. It's an alternative to Node.js. It's kind of a faster node. Bun was written in ZIG. ZIG is a systems programming language. It's kind of like C. It's very low level. One of the problems with ZIG is you have to manually manage memory. And so it's quite easy to run into situations where there's like memory leaks and other memory management issues. And so one thing that the Bun team was doing is they were having Claude fuzz the code base and try to simulate and trigger memory leaks.

17:03Boris Cherny:and they were doing this for a long period of time. They were able to find a lot of memory leaks. It was sort of like a case at a time. And that was kind of the capability of the model at the time, was doing this fuzzing. And then at some point, Jared on the team was like, okay, let's just rewrite it. Maybe the model can do this. And I think this is one of these test problems that he kind of threw at the model with every new model generation. And starting with Fable, the model started to be able to do it. And so I think Opus 5 could do it as well. And so what he did was, essentially he defined a test suite.

17:38Boris Cherny:The nice thing about Bun is it's very, very well tested. There's a big test suite in Bun. There's a big test suite in Node.js. So it's easy to know if you did the right thing. And he had the model rewrite it from Zig to Rust. It was one prompt. It was a dynamic workflow. And dynamic workflows are a feature in quad code that essentially let you orchestrate dozens, hundreds, thousands of agents to do work productively. And it ran for 11 days, and it rewrote the entire code base. And this was one shot? It was one shot with... No, it wasn't one shot, but there was steering. There was steering. But previous models just couldn't do this, even with the steering.

18:15Boris Cherny:It just wouldn't have been possible. In just 11 days. Oh, my God. This would have taken in the past, even with the best engineers, multiple months, years? Definitely over a year. Yeah. Yeah, over a year. This was like over 100 ,000... Like JavaScript runtime is really complicated. There's a lot of stuff in there. And yeah, it works. This is in production now. This is what quad code uses now when you're running it. So this is kind of one example. I would give a second example of a product overhang. And so this is like a practical use case where there's a problem you're solving. It's like a business problem, an engineering problem, a product problem.

18:49Boris Cherny:And you should just keep throwing the latest model at it to see if it'll just do it. Because even if a previous model didn't, the new one might. I think the second way to think about it is experiment. And just give yourself freedom to play with a model and do creative things. Often it'll surprise you. So something that's actually been really popular internally that's been kind of viral within Anthropic the last couple weeks is someone figured out that you can give Opus 5 OpenCV. And you can have a draw. And so something you can do is you can ask Opus, hey, use OpenCV to draw this image. And it's actually quite good.

19:25Boris Cherny:It can do portraits. It can draw animals. It can do landscapes. And we didn't train the model to draw. It's just the solicitation gap. If you ask it to do it the right way, it can just do it. And we discovered this kind of accidentally just by playing around and trying creative things that didn't have direct commercial applications. But it's just kind of interesting. And my hypothesis is there's probably dozens, hundreds of opportunities like this with the models of today that no one has yet realized. And the big area of research for this is basically model elicitation, right? Becoming really good at figuring out all these capabilities and asking the model to do the right thing, right?

20:06Boris Cherny:Yes. How do people get better at that? And effectively, how do people get better at prompt engineering? Do people still need to do a lot of prompt engineering? Or is that changing as well? Tell us about where this is going. Yeah, I remember like a year ago, one of the most popular job openings was prompt engineer. And then it kind of changed and then I think it became like context engineer. So there's these kind of waves of it. I think these will kind of like come and go. I think the skill nowadays is less about prompt engineering and more about figuring out how do you give Quod a hard task that seems a little bit too hard.

20:46Boris Cherny:And then how do you make it possible for Quod to verify its work along the way? And the verification, I think, is probably the single most important thing that people do not get right, largely. One example of this is people were, you know, we have this desktop app for Cloud. And it's built using Electron. We've made it quite fast. So now it's like a pretty awesome experience. Six months ago, it was like sluggish and it wasn't very reliable. Now it's pretty awesome. And, you know, it's the thing that most of the team uses. As an experiment, though, I wanted to see, like, what would it feel like if it was native?

21:22Boris Cherny:And so what I did is I started a quad tag session. And quad tag is just, you know, it's a new product we have. It's just quad running in Slack. My first question was, hey, Tag, do you have access to a Mac OS runner on GitHub? And it said no. And then I hooked up a runner, so it was able to start a Mac virtual machine using GitHub. And then my second question is, I created this, like, empty code base that was a cloud desktop app rewritten in Swift. And I asked, can you access this code base? It said no. And then I gave it access, and I was like, okay, great. Now I have access. And then I was like, okay, now what I want you to do is I want you to rewrite the Electron app in Swift.

22:03Boris Cherny:I want you to run the Electron app in the Mac virtual machine, screenshot it, and then look pixel by pixel, compare it to the Swift version, don't stop until you're done. And that was your prompt, basically. That was my prompt. And how long did this take to run? It's still running. When did you start it? It's been a little over two weeks. So it's like 14 days, 15 days. Yeah, so I don't know if anyone in the audience has gotten clocked to run a task for more than two weeks. I don't know. Raise your hand. Anyone in the audience? Oh, this is you. All right. Some? This is like one of these, this is about elicitation.