Claude Code: Anthropic's CLI Agent

24:50We invoke it with slash project colon lint, which just invokes that local command. It'll run the linter. It'll identify any mistakes. It'll make the code changes. And then it'll use the GitHub MCP server in order to commit the changes back to the pure. And so you can kind of compose these tools together. And I think that's a lot of the way we think about code is just one tool in an ecosystem that composes nicely without being opinionated about any particular piece. It's interesting. I have a weird chapter in my CV that makes me, I was the CLI maintainer for Nellify. And so I have a little bit of a dive.

25:26There's a decompilation of cloud code out there that has since been taken down. But it seems like you use Commander JS and React Inc. is like the public info about this. And I'm just kind of curious, at some point you're not even building cloud code, you're kind of just building a general purpose CLI framework that any developer can hack to their purposes. You ever think about this? This level of configurability is more of like a CLI framework or some new form factor that doesn't exist before. Yeah, it's definitely been fun to hack on a really awesome CLI because there's not that many of them.

26:03Yeah. But yeah, we're big fans of Inc. Yeah. Vadim Demetis, we actually used React Inc. for a lot of our projects. Oh, cool. Yeah. Yeah, Inc. is amazing. It's sort of hacky and janky in a lot of ways. It's like you have React, and then the renderer is just translating the React code to ANSI escape codes. That's the way to render. And there's all sorts of stuff that just doesn't work at all because ANSI escape codes are like, you know, this thing that started to be written in the 1970s. And there's no really great spec about it. Every terminal is a little different. so building in this way it feels to me a little bit like a building for the browser back in the day where you had to think about like internet explorer 6 versus oprah versus like firefox and whatever like you have to think about these cross-terminal differences a lot but yeah big fans of ink because it helps abstract over that we're also uh we use bun um so big fans of bun um that's been it makes writing our tests and running tests much faster we don't use it in the runtime yet it's not just for speed but you tell me i don't want to put words in your mouth.

27:06But my impression is they help you ship the compilation, the executable. Yeah, exactly. So we use Bunn to compile the code together. Yeah. Any other pluses of Bunn? I just want to track Bunn versus Deno conversations. Yeah. Use Deno's in there, you know. I actually haven't used Deno back. It's been a while. That's what a lot of people say. Ryan made it back in the day and it was like, there were some ideas that I think were very cool in it, but yeah, it just never took off to that same degree. Still a lot of cool ideas, like being able to NPM just to import from any URL, I think is pretty amazing.

27:40That's the dream of ESM. Yeah, very cool. Okay, I was going to ask you one other feature, then we can get to the thinking tool of auto-accept. I have this little thing I'm trying to develop thinking around for trust in agents, right? When do you say, all right, go autonomous? When do you pull the developer in? and sometimes you let the model decide. Sometimes you're like, this is a destructive action, always ask me. And I'm just curious if you have any internal heuristics around when to auto accept and where all this is going. We're spending a lot of time building out the permission system. So Robert on our team is leading out this work.

28:21We think it's really important to give developers the control to say, hey, these are like the allowed permissions. Generally, this includes stuff like the model's always allowed to read files or read anything. And then it's up to the user to say, hey, is it about to edit files? Is it about to run tests? These are like probably the safest three actions. And then there's like a long list of other actions that users can either allow list or deny list based on regex matches with the action. How can writing a file ever be unsafe if you have version control? I think that's... Yeah, I think there's like a few different probably like aspects of safety to think about.

29:01So it could be useful just to break that out a little bit. So for file editing, it's actually less, I think, about safety, although there is still a safety risk because what might happen is, let's say the model fetches a URL and then there's a prompt injection attack in the URL and then the model writes malicious code to disk and you don't realize it. Although, you know, there is code review as like a separate kind of layer there as protection. But I think generally for file writes, the model might just do the wrong thing. That's the biggest thing. And what we find is that if the model is doing something wrong, it's better to identify that earlier and correct it earlier.

29:33And then you're going to have a better time. If you wait for the model to just go down this like totally wrong path and then correct it 10 minutes later, you're going to have a bad time. So it's better to usually identify failures early. But at the same time, there's some cases where you just want to let the model go. So for example, if Cloud Code is writing tests for me, I'll just hit Shift-Tab, enter auto-accept mode, and just let it run the tests and iterate on the tests until they pass. Because I know that's a pretty safe thing to do. And then for some other tools like Bash Tool, it's pretty different.

30:07Because Cloud Code could run rm, rf, slash. And that would suck. That's not a good thing. So we definitely want people to be in the loop to catch stuff like that. But the model is trained and aligned to not do that. But these are non-deterministic systems. So you still want a human in the loop. I think that generally the way that things are trending is kind of less time between human input. Did you see the meter paper? No. They established a Moore's law for time between human input, basically. And it's basically doubling every three to seven months is the idea. and Enthopic is currently doing super well on that benchmark.

30:47It's roughly autonomous for 50 minutes at the 50th percentile of human effort, which is kind of cool. Highly recommend that. I put cursor in YOLO mode all the time and just run it. Which is vibe coding, right? Yeah, and there's a couple things that are interesting when you talked about alignment and the model being trained. So I always put it in a Docker container and I have a prefix every command with the Docker composed. and yesterday my Docker server was not started and I was like, oh, Docker is not running. Let me just run it outside of Docker. And I'm like, whoa, whoa, whoa, whoa, whoa. You should start Docker and run it in Docker.

31:23You cannot go outside. So that is like a very good example of like, you know, sometimes you think it's doing something and then it's doing something else. And for the review side itself, I would love to just chat about that more. I think the linter part that you mentioned, I think maybe people skipped it over. It doesn't register the first time, but like going from like rule-based linting to like semantic linting, I think it's like great and super important. And I think a lot of companies are trying to do, how do you do autonomous PR review, which I've not seen one that I use so far. They're all kind of like mid.

31:55So I'm curious how you think about closing the loop or making that better and figuring out, especially like what are you supposed to review? Because these PRs get pretty big when you buy code, you know, sometimes I'm like, oh, wow. Oh, GTM. You know, it's like, am I really supposed to read all of this? It kind of seems, most of it seems pretty standard, but like I'm sure there are parts in there that the model would understand that are like kind of out of distribution, so to speak, to really look at. So, yeah, I know it's a very open-ended question, but any thoughts you have would be great. The way we're thinking about it is Cloud Code is, like I said before, it's a primitive.

32:29So if you want to use it to build a code review tool, you can do this. If you want to, you know, build like a security scanning vulnerability scanning tool, you can do that. If you want to build a semantic linter, you can do that. And hopefully with code, it makes it so if you want to do this, it's just a few lines of code. And you can just have Cloud write that code also because Cloud is really great at writing GitHub actions. Yeah, one thing to mention is we do have a non-interactive mode, which is like what Cloud uses or how we use Cloud in these situations to automate Cloud code. And also a lot of the companies using Cloud code actually use this non-interactive mode.

33:06So they'll, for example, say, hey, I have like hundreds of thousands of tests in my repo. Some of them are out of date. Some of them are flaky. And they'll send cloud code to look at each of these tests and decide, okay, how can I update any of them? Like, should I deprecate some of them? How do I like increase our code coverage? So that's been a really cool way that people are non-interactively using cloud code. What are the best practices here? Because when it's non-interactive, it could run forever. and you're not necessarily reviewing the output of everything. So I'm just kind of curious, how is it different in non-interactive mode?

33:43What are the most important hyperparameters or arguments to set? Yeah, and for folks that haven't used those, so non-interactive mode is just cloud-p, and then you pass in the prompting quotes, and that's all it is. It's just the dash-p flag. Generally, it's best for tasks that are read-only. That's the place where it works really well, and you don't super have to think about permissions and running forever and things like that. So for example, a linter that runs and doesn't fix any issues. Or for example, we're working on a thing where we use Cloud with dash P to generate the change log for Cloud.

34:14So every PR is just looking over the commit history and being like, okay, this makes it into the change log. This doesn't. Because we know people have been requesting change logs, so we're just getting Cloud to build it. So generate non-interactive mode, really good for read-only tasks. for tasks where you want to write, the thing we usually recommend is pass in a very specific set of permissions on the command line. So what you can do is pass in dash dash allowed tools. And then you can allow a specific tool. So for example, not just bash, but for example, git status or git diff. So you just give it a set of tools that it can use or, you know, edit tool.

34:49It still has default tools, file read, grep, system tools like bash NLS and memory tools, right? All those are. So it still has all these tools, but a lot of tools just lets you, instead of the permission prompt, because you don't have that in the non-interactive mode, it's just kind of pre-accepting. And we'd also definitely recommend that you start small. So test it on one test, make sure that it has reasonable behavior, iterate on your prompt, then scale it up to 10, make sure that it succeeds. Or if it fails, just analyze what the patterns of failures are and gradually scale it from there.

35:22So definitely don't kick off a run to fix 100 ,000 tests. I think the, so at this point, I just, you know, I want to, this tagline is in my head that basically at Anthropic, there's cloud code generating code and cloud code also reviewing its own code. Like at some point, right? Like different people are setting all this up. You don't really govern that. But it's happening. The point of the thing I was thinking about was we have, you know, VPs of N, CTOs listening. This is all well and good for the individual developer, but the people who are responsible for the tech, the entire code base, the engineering decisions, all this is going on.

35:59My developers, I manage like 100 developers. Any of them could be doing any of this at this point. What do I do to manage this? How does my code review process change? How does my change management change? I don't know. We've talked to a lot of VPs and CTOs about it. They actually tend to be quite excited because they experiment with the tool, they download it, they ask it a few questions. And like quad code, when it gives them sensible answers, they're really excited because they're like, oh, I can understand like this nuance in the code base. And sometimes they even ship small features with quad code.

36:32And I think through that process of like interacting with the tool, they build a lot of trust in it. And a lot of folks actually come to us and they ask us like, How can I roll it out more broadly? And then we'll often have sessions with VPs of DevProd and talk about these concerns around how to make sure people are writing high-quality code. I think in general, it's still very much up to the individual developer to hold themselves up to a very high standard for the quality of code that they merge. Even if we use Cloud Code to write a lot of our code, it's still up to the individual who merges it to be responsible for this being well-maintained, well-documented code that has reasonable abstractions.

37:17And so I think that's something that will continue to happen where Cloud Code isn't its own engineer that's committing code by itself. It's still very much up to the ICs to be responsible for the code that's produced. Yeah. I think quad code also makes a lot of this stuff. A lot of quality work becomes a lot easier. So for example, like I have not manually written a unit test in many months. We have a lot of units. We have a lot of units. And it's because quad writes all the tests. And, you know, before I felt like a jerk if on someone's PR, I'm like, hey, can you write a test? Because, you know, they kind of know.

37:50For code coverage? Is that still relevant? Yeah. Okay. And, you know, they kind of know they should probably write a test and that's probably the right thing to do. And somewhere in their head, they make that tradeoff where they just want to ship faster. and so you always kind of feel like a jerk for asking but now i always ask because quad can just write the test right and you know there's no human work you just ask quad to do it and it writes it and i think with writing tests becoming easier and with writing lint rules becoming easier it's actually much easier to have high quality code than it was before what are the metrics that you believe in like is it a lot of people actually don't believe in 100 code coverage Because sometimes that is kind of optimizing for the wrong thing.

38:29Arguably, I don't know. But obviously you have a lot of experience in different code quality metrics. But what still makes sense? I think it's very engineering team dependent, honestly. I wish there was a one-size-fits-all answer. Yeah, just give me the one solution. For some teams, test coverage is extremely important. For other teams, type coverage is very important. Especially if you're working in a very strictly typed language. and, you know, for example, avoiding like Ennies and JavaScript and Python. Yep. I think psychomatic complexity kind of gets a lot of flack, but it's still honestly a pretty good metric just because there isn't anything better in terms of ways to measure code quality.

39:07Okay. And then productivity is obviously not lines of code, but do you care about measuring productivity? I'm sure you do. Yeah, you know, lines of code honestly isn't terrible. Oh, God. It has downsides. Yeah, it's terrible. Lines of code is terrible for a lot of reasons. Yes. But it's really hard to make anything better. It's the least terrible. It's the least terrible. There's lines of code, maybe number of PRs. How green your GitHub is. Yeah. The two that we're really trying to nail down are one, decrease in cycle time. So how much faster are your features shipping because you're using these tools?

39:44So that might be something like the time between first commit and when your PR is merged. It's very tricky to get right, but one of the ones that we're targeting. The other one that we want to measure more rigorously is the number of features that you wouldn't have otherwise built. We have a lot of channels where we get customer feedback. And one of the patterns that we've seen with Cloud Code is that sometimes customer support or customer success will post, hey, this app has this bug. And then sometimes 10 minutes later, one of the engineers on that team will be like, Cloud Code made a fix for it.

40:20And a lot of those situations when you ping them and you're like, hey, that was really cool. They were like, yeah, without Cloud Code, I probably wouldn't have done that because it would have been too much of a divergence from what I was otherwise going to do. It would have just ended up in this long backlog. So this is the kind of stuff that we really want to measure more rigorously. That was the other AGI-appealed moment for me. There was a really early version of QuadCode many, many months ago. And this one engineer at Anthropic Jeremy built a bot that looked through a particular feedback channel on Slack.

40:52And he hooked it up to code to have code automatically put up PRs with just fixes to all this stuff. And some of this stuff, you know, it can fix every issue, but it fixed a lot of the issues. Was it like 10 %? 50? You know, this was like early on, so I don't remember. But it was surprisingly high to the point where I became a believer in this kind of workflow. And I wasn't before. SAPM, isn't that scary too, in a way? Where you can build too many things, it's almost like maybe you shouldn't build that many things. I think that's what I'm struggling with the most. It's like it gives you the ability to create, create, create.

41:24But then at some point you got to support, support, support. This is the Jurassic Park. Your scientist is so preoccupied with whether you could. Yeah, yeah, exactly. But now we should. Yeah, how do you make decisions? Now that the cost of actually implementing the thing is going down, as a PM, how do you decide what is actually worth doing? Yeah, we definitely still hold a very high bar for net new features. Most of the fixes were like, hey, this functionality is broken or there's a weird edge case that we hadn't addressed yet. So it was very much like smoothing out the rough edges as opposed to building something completely net new.

41:56For net new features, I think we hold a pretty high bar that it's very intuitive to use. The new user experience is like minimal. It's just like obvious that it works. We sometimes actually use Cloud Code to prototype instead of using docs. Yeah, so you'll have like prototypes that you can play around with. And that often gives us a faster feel for, hey, is this feature ready yet? Or like, is this the right abstraction? Is this the right interaction pattern? So it gets us faster to feeling really confident about a feature. but it doesn't circumvent the process of us making sure that the feature definitely fits in the product vision.

42:36It's interesting how as it gets easier to build stuff, it changes the way that I write software. Like Kat's saying, before I would write a big design doc and I would think about a problem for a long time before I would build it sometimes for some set of problems. And now I'll just ask quad code to prototype three versions of it and I'll try the feature and see which one I like better. And then that informs me much better and much faster than a doc would have. And I think we haven't totally internalized that transition yet in the industry. Yeah, I feel the same way for some tools I build internally.

43:10People ask me, could we do this? And I'm like, I'll just, yeah, just build it. It's like, well, it feels pretty good. We should like polish it, you know, or sometimes it's like, no, that's not. It's comforting that, you know, like that your max cost is, I mean, you're even at Anthropic where it's theoretically unlimited. The cost is roughly$6 a day. That gives people peace of mind because I'm like,$6 a day? Fine. $600 a day, we have to talk. I pay$200 a month to make Studio Ghibli photos. So it's all good. That is totally worth it. You mentioned internal tools and that's actually a really big use case that we're seeing emerge.

43:48Because a lot of times, if you're working on something operationally intensive, if you can spin up an internal dashboard for it or like an operational tool where you can, for example, grant access to a thousand emails at once. A lot of these things, you don't really need to have like a super polished design. You kind of just need something that works. And QuadCode's really good at those kinds of zero to one tasks. Like we use Streamlit internally and there's been like a proliferation of how much we're able to visualize. And because we're able to visualize it, we're able to see patterns that we wouldn't have otherwise if we were just looking at like raw data.

44:28Yeah, like I was working on also this like side website last week and I just showed CloudCode the mock. So I just took the, you know, the screenshot I had dragged and dropped it into the terminal and I was like, hey, Cloud, here's the mock. Can you implement it? And it implemented and it looked like, you know, it sort of worked. It was a little bit crummy and I was like, all right, now look at it in Puppeteer and like iterate on it until it looks like the mock. And then it did that three or four times and then the thing looked like the mock. Yeah, this was just all manual work before. I think we're going to ask about two other features of, I guess, the overall agent pieces that we mentioned.

45:02So I'm interested in memory as well. So we talked about autocompact and memory using hashtags and stuff. My impression is that your, like you say, simplest approach works, but I'm curious if you've seen any other requests that are interesting to you or internal hacks of memory that people have explored that you might want to surface to others? There's a bunch of different approaches to memory. Most of them use external stores of various sorts. Like Chroma? Yeah, there's a lot of projects like that. It's either a K-value or graph stores. That's the two big shapes for this. Do you believe in knowledge graphs for this stuff?

45:45If you talked to me before I joined Anthropic and this team, I would have said, yeah, definitely. But now, actually, I feel everything is the model. That's the thing that wins in the end. As the model gets better, it subsumes everything else. At some point, the model will encode its own knowledge graph. It'll encode its own KV story if you just give it the right tools. I think the specific tools, there's still a lot of room for experimentation. We don't know yet. In some ways, are we just coping for lack of context length? Are we doing things for memory now that if we had a 100 million token context window, we don't care about?

46:20I would love to have 100 million token context. Some people have claimed to have done it. We don't know if it's true or not. I guess here's the question for you. Sean, if you took all the world's knowledge and you put it in your brain, and let's say there was some treatment that you could get to make it so your brain can have any amount of context, you have infinite neurons, is that something that you would want to do? Or would you still want to record knowledge externally? Putting it in my head is different from me trying to use an agent tool to do it because I'm trying to control the agent. And I'm trying to make myself unlimited, but I want to make the tools that I use limited because then I know how to control them.

46:56And it's not even like a safety argument. It's just more like, I want to know what you know. And if you don't know a thing, sometimes that's good. Like the ability to audit what's in the time. And I don't know if this is the small brain thinking because this is not very bitter lesson, which is like, actually, sometimes you just want to control every part of what goes in there in the context. and the more you just, you know, Jesus take the wheel, trust the model, then you have no idea what it's paying attention to. Yeah, I don't know. Did you see the Mech interpretability stuff from Chris Ola and the team that was published?

47:28Like last week? Yeah, last week. Yes. What about it? I wonder if something like this is the future. So there's an easier way to audit the model itself. And so if you want to see like what is stored, you can just audit the model. Yeah. The main salient thing is that they know what features activate per token. And they can tune it up, suppress it, whatever. But I don't know if it goes down to the individual item of knowledge from context. Not yet. But I wonder, maybe that's the Bitter Western version of it. Any other comments from memory? Otherwise, we can move on to planning and thinking. We've been seeing people play around with memory in quite interesting ways.

48:06Like having Claude write a logbook of all the actions that it's done. so that over time, Cod develops this understanding of what your team does, what you do within your team, what your goals are, how you like to approach work. We would love to figure out what the most generalized version of this is so that we can share broadly. I think with things like CodCode, I think when we're developing things like CodCode, it's actually less work to implement the feature and a lot of work to tune these features to make sure that they work well for general audiences across a broad range of use cases. So there's a lot of interesting stuff within memory, and we just want to make sure that it works well out of the box before we share it broadly.

48:54Agree with that. I think there's a lot more to be developed here. I guess a related problem to memory is how do you get stuff into context? Knowledge base. Knowledge base, yeah. And originally we tried very, very early versions of Cloud actually used RAG. So we indexed the code base, And I think we were just using Voyage. So, you know, just off-the-shelf RAG. And that worked pretty well. And we tried a few different versions of it. There was RAG, and then we tried a few different kinds of search tools. And eventually we landed on just agentic search as the way to do stuff. And there were two big reasons, maybe three big reasons.

49:26So one is it outperformed everything. By a lot. By a lot. And this was surprising. In what benchmark? This was just vibes. So internal vibes. There's some internal benchmarks also, but mostly vibes. It just felt better. An agentic rag, meaning you just let it look up in however many cycles it needs. Yeah, just using regular code searching, you know, glob, grab, just regular code search. Regular code search, yeah. Yeah. So there was like one, and then the second one was, there was this whole like indexing step that you have to do for rag. And there's a lot of complexity that comes with that because the code drifts out of sync and then there's security issues because this index has to live somewhere.

50:02And then, you know, what if that provider gets hacked? And so it's just a lot of liability for a company to do that. Even for our code base, it's very sensitive. So we don't want to upload it to a third-party thing. It could be a first-party thing, but then we still have this out-of-sync issue. And agentic search just sidesteps all of that. So essentially, at the cost of latency and tokens, you now have really awesome search without security downsides. With memory is like planning, right? There's kind of like memories, like what I like to do. And then planning is like, now use those memories to come over to plan to do these things.

50:34um there was one or maybe put it as like memory sort of the past like what we what we already did and if planning is kind of what we will do yeah it just crosses over at some point yeah i think maybe slightly confusing thing from the outside is what you define as thinking so there's like uh extensive thinking there's the think tool and it's kind of like thinking as in planning which is like thinking before execution and then there's like thinking what you're doing which is like the the think tool. Can you maybe just run people through the difference? I'm already confused listening to you. Well, it's one tool.

51:10So Quad can think if you ask it to think. Generally, the usage pattern that works best is you ask Quad to do a little bit of research, like use some tools, pull some code into context, and then ask it to think about it. And then it can make a plan, do a planning step before you execute. There's some tools that have explicit planning modes, like RueCode has this and Klein has this and other tools have it. Like you can shift between, you know, plan and act mode or maybe a few different modes. We've sort of thought about this approach, but I think our approach to product is similar to our approach to the model, which is bitter lesson.

51:48So just freeform, keep it really simple, keep it close to the metal. And so if you want Claude to think, just tell it to think, be like, you know, make a plan, think hard, don't write any code yet. and it should generally follow that. And you can do that also as you go. So maybe there's a planning stage and then Claude writes some code or whatever and then you can ask it to think and plan a little bit more. You can do that anytime. Yeah, I was reading through the Think Tool blog posts and I said, while it sounds similar to extended thinking, it's a different concept. Extended thinking is what Claude does before it starts generating.

52:20And then think it, once it starts generating, how do you have to stop and think? Is this all done by the Claude code harness? so people don't really have to think about the difference between the two, basically, is the idea? Yeah, you don't have to think about it. Okay. That is helpful. That is helpful. Because sometimes I'm like, man, am I not thinking right?

52:42Yeah, and it's all chain of thought, actually, in QuadCode. So we don't use the Think tool. Anytime that QuadCode does thinking, it's all chain of thought. I had an insight. This is, again, something we had, a discussion we had before recording, which is in the CloudPlace Pokemon hackathon, we had access to more sort of branching environments feature, which meant that we could take any VM state, branch it, play it forward a little bit and use that in the planning. And then I realized the TLDR of yesterday was basically that it's too expensive to just always do that at every point in time. But if you give it as a tool to Claude and prompt it in certain cases to use that tool, seems to make sense.

53:23I'm just kind of curious, like your takes on overall like sandboxing environment, branching, rewindability maybe, which is something that you immediately brought up, which I didn't think about. Is that useful for Claude? Claude has no opinions about it. Yeah, I could talk for hours about this. Claude probably can too. Yeah? Let's get original tokens from you and then we can train Claude on that. By the way, that's like explicitly what this podcast is or just generating tokens for people. Is this the pre-training or the post-training? It's a pre-trained dataset. You got to get in there. Oh man.

53:52Yeah, how do I buy? How do I get some tokens? Starting with sandboxing, ideally the thing that we want is to always run code in a Docker container and then it has freedom and you can kind of snapshot, you know, with other kind of tools where you're on top, you can snapshot, rewind, do all this stuff. Unfortunately, working with a Docker container for everything is just like a lot of work and most people aren't going to do it. And so we want some way to simulate some of these things without having to go full container. There's some stuff you can do today. So for example, something I'll do sometimes is if I have a planning question or a research type question, I'll ask Claude to investigate a few paths in parallel.

54:29And you can do this today if you just ask it. So say, you know, I want to refactor X to do Y. Can you research three separate ideas for how to do it? Do it in parallel. Use three agents to do it. And so in the UI, when you see a task that's actually like a sub-Claude, it's a sub-agent that does this. And usually when I do something hairy, I'll ask it to just investigate, you know, three times or five times or however many times in parallel. and then Claude will kind of pick the best option and then summarize that for you. But how does Claude pick the best option? Don't you want to choose? What's your handoff between you should pick versus I should be the final decider?

55:08I think it depends on the problem. You can also ask Claude to present the options to you. Probably, you know, it exists at a different part of the stack than Claude Code specifically. Claude Code as a CLI, like you could use it in any environment. So it's up to you to compose it together. Should we talk about how and when models fail? Because I think that was another hot topic for you. I'll just leave it open. How do you observe cloud code failing? There's definitely a lot of room for improvement in the models, which I think is very exciting. Most of our research team actually uses cloud code day to day.

55:40And so it's been a great way for them to be very hands-on and experience the model failures, which makes it a lot easier for us to target these in model training and to actually provide better models, not just for cloud code, but for all of our coding customers. I think one of the things about the latest Sonnet 3.7 is it's a very persistent model. It's like very, very motivated to accomplish the user's goal, but it sometimes takes the user's goal very literally. And so it doesn't always fulfill what like the implied parts of the request are because it's just so narrowed in on like, I must get X done.

56:20And so we're trying to figure out, okay, how do we give it a bit more common sense so that it knows the line between trying very hard and like, no, the user definitely doesn't want that. Yeah. Like the classic example is like, hey, go on, get this test to pass. And then, you know, like five minutes later, it's like, all right, well, I hard coded everything. The test passes. I'm like, no, that's not what I wanted. Hard coded answer. Yeah. But that's the thing. Like it only gets better from here. Like these use cases work sometimes today, not, you time and the model sometimes tries too hard, but it only gets better.

56:52Yeah. Context, for example, is a big one where a lot of times, if you have a very long conversation and you compact a few times, maybe some of your original intent isn't as strongly present as it was when you first started. And so maybe the model forgets some of what you originally told it to do. And so we're really excited about things like larger effective context windows so that you can have these gnarly, like really long hundreds of thousands of tokens long tasks and make sure that quad code is on track the whole way through like that would be a huge lift i think not just for quad code but for every coding company fun story from david hershey's uh keynote yesterday uh he actually misses the common sense of 3.5 because 3.7 being so persistent 3.5 actually had some entertaining stories where apparently it gave up on TAS and just 3.7 doesn't.

57:46And when Cloud 3.5 gives up, it started writing formal requests to the developers of the game to fix the game. And he has some screenshots of it, which is excellent. So if you're listening to this, you can find it on YouTube because we'll post it. Very, very cool. One form of failing, which I kind of wanted to capture, was something that you mentioned while we're getting coffee, which is that Cloud Code doesn't have that much between-session memory or caching or whatever you call that, right? So it reforms the whole state from whole cloud every single time. So it has to make the minimum assumptions on the changes that can happen in between.

58:23So how consistent can it stay, right? So I think that one of the failures is that it forgets what it was doing in the past unless you explicitly opt in via Cloud.md or whatever. Is that something you worry about? It's definitely something we're working on. I think our best advice now for people who want to resume across sessions is to tell Claude to, hey, write down the state of this session into this text doc, probably not the Claude.md, but in a different doc. And in your new session, tell Claude to read from that doc. But we plan to build in more native ways to handle this specific workflow.

59:00There's a lot of different cases of this, right? Sometimes you don't want Claude to have the context. and it's sort of like Git. Sometimes I just want a fresh branch that doesn't have any history, but sometimes I've been working on a PR for a while and I need all that historical context. So we kind of want to support all these cases and it's tricky to do a one-size-fits-all, but generally our approach to code is to make sure it works out of the box for people without extra configuration. So once we get there, we'll have something. Do you see a future in which the commits play a bigger part of like in a pull request?

59:33Like, how do we get here? You know, there's kind of like a lot of history in how the code has changed within the PR that informs the model. But today, the models are mostly looking at the current state of the branch. Yeah, so Claude, for some things, it'll actually look at the whole history. So for example, if it's writing, if you tell Claude, hey, make a PR for me, it'll look at all the changes since your branch diverged from main. And then, you know, take all of those into account when generating the pull request message. You might notice it running git diff as you're using it. I think it's pretty good about just tracking, hey, what changes have happened on this branch or so far and just make sure that it's like understands that before continuing on with the task.

1:00:16One thing other people have done is ask Quad to commit after every change. You can just put that in the QuadMD. There's some of these like power user workflows that I think are super interesting. Like some people are asking Quad to commit after every change so that they can rewind really easily. Other people are asking Claude to create a work tree every time so that they could have a few Claude's running in parallel in the same repo. I think from our point of view, we want to support all of this. So again, Claude code is like a primitive, and it doesn't matter what your workflow is. It should just fit in.

1:00:47I know that 3.5 Haiku was the number four model on Ader when it came out. Do you see Claude code have a world in which you have like a commit hook that uses maybe Haiku to do some like the linter stuff and things like that continuously. And then you have 3.7, that's the more. Yeah, you could actually do this if you want. So you're saying like through like a pre-commit hook or like a GitHub action or? Yeah, yeah, yeah. Say, well, kind of like run cloud code, like the length example that you had. I want to run it at each commit locally, like before it goes to the PR. Yeah, so you could do this today if you want.

1:01:21So in the, you know, if you're using like Husky or like whatever pre-commit hook system you're using, or just like get pre-commit hooks, just add a line cloud-p and then, you know, whatever instruction you have and that'll run every time. Nice. And you just specify haiku. It's really no difference, right? It's like maybe it'll work a little worse, but like it's still supported. Yeah, you can override the model if you want. Generally we use Sonnet. We default to Sonnet for almost everything just because we find that it outperforms. Yeah. But yeah, you can override the model if you want. Yeah. I don't have that much money to run commit hook on through one set.

1:01:54Just as a side on pre-commit hooks, I have worked in places where they insisted on having pre-commit hooks. I've worked at places where they insist that they'll never do pre-commit hooks because they get in the way of committing and moving quickly. I'm just kind of curious, do you have a stance or recommendation? Oh God, that's like asking about tabs versus spaces. A little bit. But I think it is easier in some ways to, if you have a breaking test, go fix the test with clock code. In other ways, it's more expensive to run this at every point. So there's trade-offs. I think for me, the biggest trade-off is you want the pre-commit hook to run pretty quickly so that either if you're a human or if you're a quad, you don't have to wait like a minute for all the checks to run.

1:02:36Yeah, so you want the fast version. So generally, you know, pre-commit for our code base should run... Just types. Yeah, it's like less than, you know, five seconds or so, like just types and lint maybe. And then more expensive stuff you can put in the GitHub action or GitLab or whatever you're using. Agreed. I don't know. I like putting prescriptive recommendations out there so that people can take this and go like, this guy said it, we should do it in our team. And like, that's a basis for decisions. Cool. Any other technical stories to tell? I wanted to zoom out into more product-y stuff, but you can get as technical as you want.

1:03:11I don't know. One anecdote that might be interesting is the night before the code launch, we were going through to burn down the last few issues, and the team was up pretty late trying to do this. And one thing that was bugging me for a while is we had this markdown rendering that we were using. and it was just, you know, it's like the Markdown rendering in Quad today is beautiful and it's just like really nice rendering in the terminal and it does bold and, you know, headings and spacing and stuff very nicely. But we tried a bunch of these off-the-shelf libraries to do it and I think we tried like two or three or four different libraries and just nothing was quite perfect.

1:03:44Like sometimes the spacing was a little bit off between a paragraph and like a list or sometimes the text wrapping wasn't quite correct or sometimes the colors weren't perfect. So each one had all these issues and all these markdown renderers are very popular and they have you know thousands of stars on github and have been maintained for many years but you know they're not really built for a terminal and so the night before the release at like 10 p.m i'm like all right i'm gonna do this so i just asked quad to write a markdown uh parser for me and they wrote it zero shot yeah it wasn't quite zero shot but after you know like maybe like one or two prompts they got it and um yeah that's the markdown parser that's in code today and the reason that markdown looks so beautiful.

1:04:23That's a fun one. It's interesting what the new bar is, I guess, for implementing features, like this exact example where there's libraries out there that you normally reach for that you find some dissatisfaction with for literally whatever reason, you could just spin up an alternative and go off of that. Yeah, I feel like AI has changed so much and literally in the last year, but a lot of these problems are, like the example we had before, a feature you might not have built before, or you might have used a library. Now you can just do it yourself. The cost of writing code is going down and productivity is going up.

1:04:58And we just have not internalized what that really means yet. But yeah, I expect that a lot more people are going to start doing things like this, like writing your own libraries or just shipping every feature. Just to zoom out, you obviously do not have a separate cloud code subscription. I'm curious what the roadmap is. Is this just going to be a research preview for much longer? Are you going to turn it into an actual product? I know you were talking to a lot of CTOs and VPs. Is there going to be a CloudCode enterprise? What's the vision? Yeah, so we have a permanent team on CloudCode. We're growing the team.

1:05:33We're really excited to support CloudCode in the long run. And so, yeah, well, we plan to be around for a while. In terms of subscription itself, itself it's something that we've talked about it depends a lot on whether or not most users would prefer that over pay as you go so far pay as you go has made it really easy for people to start experiencing the product because there's no upfront commitment and it also makes a lot more sense with a more autonomous world in which people are scripting cloud code a lot more but we also hear the concern around, hey, I want more price predictability if this is going to be my go-to tool.

1:06:13So we're very much still in the stages of figuring that out. I think for enterprises, given that Cloud Code is very much like a productivity multiplier for ICs and most ICs can adopt it directly, we've been just like supporting enterprises as they have questions around security and productivity monitoring. And so, yeah, we've found that a lot of folks see the announcement and they want to learn more. And so we've been just engaging in those. Do you have a credible number for the productivity improvement? Like for people who nodded in topic that you've talked to, like, you know, are we talking 30 %?

1:06:50Some number would help justify things. We're working on getting this. We should, yeah. It's something we're actively working on, but anecdotally for me, it's probably 2x my productivity. Oh my God. So I'm just like, I'm an engineer that codes all day, every day. For me, it's probably 2x. I think there's some engineers at Anthropic where it's probably 10x their productivity. And then there's some people that haven't really figured out how to use it yet. And they just use it to generate commit messages or something. That's maybe like 10%. So I think there's probably a big range. And I think we need to study more.

1:07:23For reference, sometimes we're in meetings together and sales or compliance or someone is like, hey, we really need X feature. and then Forrest will ask a few questions to like understand the specs. And then like 10 minutes later, he's like, all right, well, it's built. I'm going to merge it later. Anything else? So it definitely feels definitely far different than any other PM role I've had. Do you see yourself opening that channel of the non-technical people talking to ClockCode and then the instance coming to you, which like they already define and talk to it and explain what they want. And then you're doing kind of the review side and implementation.

1:08:02Yeah, we've actually done a fair bit of that. Like Megan, the designer on our team, she is not a coder, but she's landing pull requests. She uses code to do it. She designs the UI? Yeah. And she's landing PRs to our console product. So it's not even just like building on quad code. It's building like across our product suite in our monorepo. Right. Yeah, yeah. And similarly, our data scientist uses quad code, right? like, you know, like BigQuery queries. And there was like some finance person that went up to me the other day and was like, hey, I've been using quad code. And I'm like, what? Like, how did you even get it installed?

1:08:37Like, you don't use Git. And they're like, yeah, yeah, I figured it out. And yeah, they're using it. They're like, so quad code you can pipe in because it's a Unix utility. And so what they do is they take their data, put it in a CSV, and then they take the, they cat the CSV, pipe it into code. And then they ask it code questions about the CSV. and they've been using it for that. Yeah. That would be really useful to me because really what I do a lot of the times, like somebody gets me a feature request, I kind of like rewrite the prompt, I put it in agent mode and then I review the code. It would be great to have the PR wait for me.

1:09:12I'm kind of useless in the first step. Like, you know, taking the feature request and prompting the agent to write it, I'm not really doing anything. Like my work really starts after the first run is done. I was going to say, I can see it both ways. So maybe I'll simplify this to, in the workflow of non-technical people in loop, should the technical person come in at the start or come in at the end? Or come in at the end and the start, obviously that's the highest leverage thing. Because sometimes you just need the technical person to ask the right question that the non-technical person wouldn't know to ask.

1:09:45And that really affects the implementation. But isn't that the bitter lesson of the model, that the model will also be good at asking the follow-up question? Like, you know, if you're like telling the model, hey, you are... That's what it trusts the model to do the least, right? Sorry, go ahead. Yeah, no, no. If you're like the model, hey, you are the person that needs to translate this non-technical person request into the best prompt for cloud code to do a first implementation. Yeah. Like, I don't know how good the model would be today. I don't have an eval for that, But that seems like a promising direction for me.

1:10:19Like it's easier for me to review 10 PRs than it is for me to take 10 requests, then run the agent 10 times, and then wait for all of those runs to be done and review. I think the reality is somewhere in between. We spend a lot of time shadowing users and watching people at kind of different levels of seniority and kind of technical depth use code. And one thing we find is that people that are really good at prompting models from whatever context, maybe they're not even technical, but they're just really good at prompting. They're really effective at using code. And if you're not very good at prompting, then code tends to go off the rails more and do the wrong thing.

1:10:52So I think in this stage of where models are at today, it's definitely worth taking the time to learn how to prompt models as well. But I also agree that, you know, maybe in a month or two months or three months, you won't need this anymore because, you know, the bitter lesson always wins. Please, please do it. Please do it, Anthropic. I think there's a broad interest in people forking or customizing cloud code. So we have to ask, why is it not open source? We are investigating. Ah, okay. So it's not yet. There's a lot of trade-offs that go into it. On one side, our team is really small and we're really excited for open source contributions if it was open source, but it's a lot of work to kind of maintain everything and like look at it.

1:11:35Like I maintain a lot of open source stuff and a lot of other people on the team do too. And it's just a lot of work. Like it's a full-time job managing contributions and all this stuff. Yeah. I'll just point out that you can do source available and that solves a lot of individual use cases without going through the legal hurdles of a full open source. Yeah, exactly. I mean, I would say there's nothing that secret in the source. And obviously it's all JavaScript, so you can just decompile it. Compilations out there. It's very interesting. Yeah. And generally our approach is, all the secret sauce, it's all in the model.

1:12:07And this is the thinnest possible wrapper over the model. We literally could not build anything more minimal. This is the most minimal thing. Yeah. So there's just not that much in it. If there was another architecture that you would be interested in that is not the simplest, what would you have picked as an alternative? You know, like we're just talking about agentic architectures here, right? Like there's a loop here and it goes through and you sort of pull in the models and tools in a relatively intuitive way. If you were to rewrite it from scratch and like choose the generationally harder path, like what would that look like?

1:12:39Well, Boris has rewritten this. Boris and the team have rewritten this like five times. Oh, that's a story. Yeah, like quad code. It is very much the simplest thing, I think, by design. Okay, so it just got simpler. It got simpler. It doesn't go more complex. We've rewritten it from scratch, yeah, probably every three weeks, four weeks or something. And it's like a ship of Theseus, right? Like every piece keeps getting swapped out. And just because quad is so good at writing its own code. Yeah. I mean, at the end of the thing, the thing that's breaking changes is the interface. The quad, FCP, blah, blah, blah.

1:13:10all that has to kind of stay the same unless you really have a strong reason to change it. Yeah. I think most of the changes are to make things more simple, like to share interfaces across different components. Because ultimately, we just want to make sure that the context that's given to the model is in like the purest form and that the harness doesn't intervene with the user's intent. And so very much a lot of that is just like removing things that could get in the way or that could confuse the model. On the UX side, something that's been pretty tricky, and the reason that we have a designer working on a terminal app, is it's actually really hard to design for a terminal.

1:13:50There's not a lot of literature on this. I've been doing product for a while, so I kind of know how to build for apps and for web and for engineers in terms of tools that have DevEx. But terminal is sort of new. There's a lot of these really old terminal UIs that use curses and things like this for very sophisticated UI systems. but they all feel really antiquated by the UI standards of today. And so it's taken a lot of work to figure out how exactly do you make the app feel fresh and modern and intuitive in a terminal. And we've had to come up with a lot of that design language ourselves. Yeah.

1:14:26I mean, I'm sure you'll be developing over time. Cool. Closing question. This is just more general. I think a lot of people are wondering, and Topic has, I think it's easy to say, the best brand for AI engineering, like you know developers and coding models and now with like the coding tool attached to it is it just has the whole product suite of model and tool and protocol right so and then i don't think this is obvious one year ago today like when cloud 3 launched it was just it was just more like this is general purpose models and all that but like cloud sonnet really took the scene as like the sort of coding tool of choice and i think built anthropic's brand and you guys are now extending So why is Anthropic doing so well with developers?

1:15:09It seems like there's just no centralized... Every time I talk to Anthropic people, they're like, oh yeah, we just had this idea and we pushed it and it did well. And I'm just like, there's no centralized strategy here? Or is there an overarching strategy? Sounds like a PM question to me. I don't know. I would say Dario's not breathing down your necks going build the best dev tools. He's just letting you do your thing. Everyone just wants to build awesome stuff. It's like, I feel like the model just wants to write code. Yeah, I think a lot of this trickles down from like the model itself being very good at code generation.

1:15:44Like we're very much building off the backs of an incredible model. Like that's the only reason why quad code is possible. I think there's a lot of answers to why the model itself is good at code. But I think like one high level thing would be so much of the world is run via software. and there's like immense demand for great software engineers. And it's also something that like you can do almost entirely with just a laptop or like just a dev box or like some hardware. And so it just like is an environment that's very suitable for LLMs. It's an area where we feel like you can unlock a lot of economic value by being very good at it.

1:16:29There's like a very direct ROI there. We do care a lot about other areas too, but I think this is just one in which the models tend to be quite good and the team's really excited to build products on top of it. And you're growing the team you mentioned? Who do you want to hire? Yeah, we are. Who's like a good fit for your team? We don't have a particular profile. So if you feel really passionate about coding and about the space, if you're interested in learning how models work and how terminals work and how like, you know, all these technologies that are involved. Yeah, hit us up. Always happy to chat.

1:17:07Awesome. Well, thank you for coming on. This was fun. Thank you. Thanks for having us. This was fun.

1:17:20You