The harness is the showdown, the humans are the tool calls, and have you seen my Claude Code buddy?

0:05Ben:Oh, so you saw my message about Trixel and sadly what happened. You'll remember a few weeks ago when Claude rolled out a whole bunch of features that got, you know, quote unquote leaked. And one of them was a fun buddy system where you can like hatch a buddy. It had like a name, a description and like some fun traits and whatnot.

0:22Andrew:It would literally hang out with you in the terminal.

0:25Ben:But sadly, and what none of us knew, is that when you upgrade your clod version, which I mean, God, come on, we're all doing pretty regularly now, it gets rid of your buddy. So session by session, as I was finishing up what I was doing and closing it out and then whatnot, I lost all my sessions where Trixel existed except for one. so now Trixel is just in this one little pain on this one cloud code version in my terminal that I never use because he's been fully upgraded and it got rid of him everywhere else

0:58Andrew:it's almost like a friend that leaves without ever saying goodbye you just didn't get that chance to say goodbye I know, I'm so sad, it's like I didn't even mean to

1:06Ben:kill you Trixel, I just was trying to upgrade my cloud code version but I guess that's you know, shame on me for getting to attach to my little terminal buddy, but there'll be more.

1:17Andrew:Yeah, well, we're some friends here that aren't saying goodbye today. Welcome to the Friday Deploy brought to you by Linear B. I'm your host, Ben Lloyd Pearson. And I'm your host, Andrew Ziegler. And this week, we are covering AI pricing chaos, cloud code patterns that have been exposed, and Vercel's third-party AI breach. You want to just start right at the top, Andrew, with this AI pricing chaos?

1:43Ben:Sure. So let's talk about all the different providers right now adjusting the pricing around their harnesses and our access to their models. This is a really a pricing revolution that's happening among the biggest names in the field. One of them that's like a big story that comes to mind first is at GitHub Copilot, they made changes to their individual plans. In fact, what they did was they even paused signups for new subscriptions to GitHub Copilot Pro, ProPress, and also their student plan. And that's a pretty shocking kind of like discovery to find that they're going to stop all growth to address what is clearly a pricing problem within the model usage.

2:23Ben:So what are you seeing when you see this kind of thing happen in the news?

2:28Andrew:Yeah. Yeah. And they also mentioned, you know, they're tightening like the usage limits for individual plans. So, you know, you're potentially going to start getting less usage out of it. And they're also deprecating some Opus models. Like, you know, they did say that Opus 4.7 will remain, but 4.5 and 4.6 are going to be removed, at least from the Pro Plus tier. Which I actually thought that last bit was pretty interesting because, you know, supporting 4.7, but while dropping the other two, it makes me think that it must be to do with promotional pricing for the 4.7 launch. So it kind of feels like it's a temporary state.

3:05Andrew:Like I can't imagine that they're really getting like long-term cost savings out of that. But yeah, I mean, suspending signups, I think is the most telling aspect of this, you know, like that really says that they're concerned that they aren't going to be able to fulfill the promises that they're making to their new customers. But, you know, I, I kind of think in some ways this is a pretty simple problem to solve if you really think about it. But, you know, like most of these tools that we're using right now, they just encourage you to default to the most expensive model for everything. You know, just why not pick the best model at all times?

3:44Andrew:Because, you know, I'm being charged a fixed limit and a fixed cost. And as long as I'm not hitting my session limits, you know, that pricing model encourages me to consume as many tokens as possible until I hit my limit. So that's like that's like what I'll do. Like yesterday at the end of the, at the end of my day, I was like sitting at like 89 % clawed to session use it, uh, like out of my session limit. And I was like, I should push this to a hundred before I leave for the today. Right. Like it just wants you, makes you want to consume more. Um, and you know, a lot, and these companies really aren't thinking about efficiency yet.

4:20Andrew:Like we have covered some stories like Shopify. we recently covered how they're switching or they've been using quen to sort of use sub-agents to do things locally and i think that you know there's a there's a lot of space for companies to come in with systems that get better at delegating work to less expensive or even local models so that i can do as much as possible like on my local laptop or you know in your case you'd like to do everything on a vps like why not just do your ai processing there rather than having to rely on an external service. So yeah, if there's anyone out there that works at GitHub listening to this, I think that's your answer.

5:01Andrew:So you're welcome, I guess.

5:03Ben:Yeah, you heard it here first. You make a lot of great observations here. One of them being around having a model router that understands what level of intelligence needs to serve the requests that you're asking and not defaulting to using the most default and intensive model that you can. I mean, a lot of these providers, they do shove the strongest and the best in front of you because that's their best shot at guaranteeing you're going to have that great user experience and they're going to keep you. And so it makes sense from like a growth play of why they do it. But ultimately, like in the world I've been in of taking things that I typically did, like in a chat in CloudCoder, I would typically like or hop over to Cloud or OpenAI and use it in ChatGPT, like turning those into workflows and agents and systems with determinism and that live somewhere else, like in another machine, is that you have to understand all of like the seams around those requests.

6:01Ben:And also like understanding when you do this, you start to realize that not everything needs to get sent to the most expensive API call that you can make now that you're making just raw API calls. You start to think like, oh, this stage of the process is just a little bit of pre-processing or we're picking something out or we're cleaning something up. And like, let's throw that the haiku. Let's do something on machine, which I've done now multiple times. And understanding that model choice is a big gradient. And we have to, as consumers, get smart about choosing where on that gradient we need to be for what we're doing.

6:38Ben:But in the meantime, I feel like we're all going to just be in this environment where we're just constantly encouraged to use more of the tokens. Like even when Opus 4.7 came out and I upgraded and got access to that, immediately it's on the extra high effort, high intensity mode to the point where like I think if I ask it to ultra think it's going to dumb itself down. Like I've kind of like it's trying to operate on such a high level. Maybe I don't always need it that way. So a lot of really good observations about how it kind of like feeds the way that we use the tool. Yeah.

7:11Andrew:I mean, it seems like all these companies that are building like wrappers around the foundation models are just in a really tough place because I think what's really going to probably happen or what will be one of the many results from this trend that we're seeing, because we'll get into another story here that is very similar, is that when you're paying for like a seat based unlimited usage plan, you're probably going to like I feel like being able to select your model is a luxury that we have right now. that when you're on a seat-based model will probably get taken away from you at some point.

7:44Andrew:The provider will decide which models are going to be used for you. And that's where then you go to an API where it's more usage-based, you know, because we have some workflows, like my personal workflows are all on my seat-based consumption model. But when we deploy a workflow, it gets moved over to an API consumption-based model. Then we actually do get really conscious about which model we're picking because we have to pay for it. Exactly.

8:09Ben:You're describing like my exact trajectory. Like my cursor, when you use cursor, cursor is a model router. It defaults you to using auto mode, which then routes you to a lot of different models, including their own, to do a lot of different requests, which they know they can serve at various rates of cost. And that's how cursor has been able to just scale to such a magnitude. So you're right that these providers, they become the router just as much as they're the harness.

8:33Andrew:Yeah. All right. Well, let's talk about one of these foundation model companies. is also playing around with their pricing. So Claude Code. Andrew, do you think this is going to cost$100 a month moving forward? What do you think?

8:45Ben:A Claude Code? Yeah, I think it's probably going to cost more eventually. I'm actually very much of the mind that the costs for all of these tools are going to start ratcheting up and up and up. And you're going to start seeing equivalents that are more closer to salaries just because they're going to be able to measure the value of the output that way. and I feel like a lot of those tools might slowly get out of a lot of people's grass

9:11Andrew:yeah yeah and of course I bring it up because Anthropic got called out this week for quietly testing moving Claude code to their pro or excuse me to their max account it's currently offered at their entry-level pro account so you know a cost difference of five to ten x depending on which max account you sign up for social media was abuzz with all of the news about this and people wondering like, oh, are we going to lose access to Cloud Code? You know, all these people that are on these$20 a month accounts. There's a representative from Anthropic that, you know, said it was just a small test on like 2 % of their new signups.

9:48Andrew:However, the pricing change was visible to all users, which, you know, just added a little bit of confusion to the market. It seems like Anthropic's doing some pricing experiments on the back end, trying to see if people are willing to pay more for something that they're currently getting a lot of value out of at a very low cost. So yeah, I don't know, Andrew, what do you think about this story? Like, do you think it's something, a sign of an imminent change? I mean, you seem pretty convinced that it's coming either way.

10:14Ben:Oh yeah. My Cassandra complex is on fire with this one. Absolutely. I just, as, as, as someone who's paying for just one of the max tier, the highest level max tier accounts just one i have to caveat that because there are folks who are adjacent to this podcast who use multiple and there are a lot of folks who bought them and use them and use them heavily for their open claw subscriptions which you'll remember a few weeks ago we talked about anthropic clamped down on for the same exact reason right of people not utilizing the hardest in the tool and the allotments correctly and really messing with their pricing so as someone who's on the highest level max tier it's like i know it's not getting it taken away from me yet but i do expect the cost for my tier to go up even when i got my vps immediately after getting the vps it was i was told that like you know when it comes time for renewal it's going to cost a lot more just because like the door closed behind me and it shows how high the demand is and when we talk here and our listeners here and like we're all listening it's like we're a little bit ahead of that curve because we're all early adopters and early users.

11:25Ben:But there's a huge surging wave of demand behind us that is hard for us to comprehend. And this pricing experiment, you know, companies of this size are going to do pricing experiments all the time. I don't think that is necessarily the biggest, scariest thing in the world. If this freaked you out or if this was an existential crisis for you, then I invite you to step back and think about the places where you You can source your inference and not be so dependent upon one provider. That might mean distributing your workload across multiple tools. It might mean starting to explore some local language models or self-hosting.

12:00Ben:You might find that GPU hosting and fine-tuning costs are just much, much, much lower than this. And if your biggest concern is the harness, just remember that a coding agent is a very small loop with about three tools. And there's a lot of them available. and that's not going to get taken away anytime soon. Yeah.

12:21Andrew:Yeah. You know, the pricing experiment, I actually want to touch on that a little bit because I think it is, I think it is a little telling about the culture and the way that Anthropic works. You know, I'm, I'm using Claude, like basically every hour of the working day at this point, the platform almost feels like this almost amorphous blob that just sort of like constantly shifts in real time. Like things are always changing, adding new features, like I'll open a new thread and Claude will respond in a way that I'd never seen before and will do something that surprises me.

12:50Ben:And I feel like this is just a constant state of change.

12:53Andrew:And I don't know, like Andrew, if you've seen this, but like recently I just see random errors popping up that is like Claude couldn't do this or something. And it's like this obscure error message, but then it just keeps chugging along and solves my problem anyways. Like it just looks to me like they're constantly iterating on the platform. And I think this actually parallels like what's happening to a lot of organizations, particularly those that are starting to operate more in this AI forward or agentic way. AI tooling, we always have to remember, it's part of this stochastic system. So if you ask AI to build something that has relative complexity 100 different times, it's going to do it 100 different ways.

13:31Andrew:And not to mention, sometimes it's going to succeed, sometimes it will fail. So you have to have checks in place to capture all of these things. And there are clearly an emerging group of companies who are operating in this highly agentic manner. And I'm not just referring to engineering, like we see it a lot in engineering, but there are places where it's happening across the entire company. And they're operating in what I would call like an agent first manner. So when I say that, it's like what I mean is when you have a challenge or a problem that's presented to you, your default approach is to go into an agentic system to solve that problem.

14:07Andrew:You know, it's not just like chatting with an AI tool of your choice. It's like having an agent that you communicate with, convey meaning to it and have it go out and solve that problem completely for you all the way to production. and you can see how quickly Anthropic is iterating on their core platform it's like one day you have this cool little buddy and then the next day it's gone forever but you know I'm at these these tests like to your point these tests are probably going on non-stop on an Anthropic like on an hourly basis you know I think some companies that maybe get to a point where they can do it maybe on a daily basis or a weekly basis but It could even be happening faster with a company like Anthropic.

14:53Andrew:And I suspect that they have a good sense of how different companies use their platform, potentially even all the way down to the individual level, like which individuals are using our platform in a certain way. And based on that data, they can serve as capabilities to them to see whether or not the capability works. So I'm not really reading a whole lot into this specific case. you know you might be right that these capabilities are going to get more expensive either that or you know maybe maybe like co-work becomes the$20 a month version and code becomes the power user you know 100 or$200 a month version something like that you know and co-work maybe they can do what i was saying where they implement more control that takes the decisions out of your hands so you don't get to pick the models and and whatnot but anthropic is also struggling with the same problem that GitHub has.

15:44Andrew:Their current pricing model encourages me to consume as much as possible, as frequently as possible, as long as I don't exceed my limit because that stops me from being able to work. But yeah, it's just further proof. We need more efficiency in this space. Whoever can solve this problem of giving us these powerful systems, but in a much more efficient manner, that in my books is going to be the next winner in this space.

16:10Ben:Yeah, really well said. And I think that leads us pretty well to our next story as well, where we talk about how these major AI providers, like we've been talking about, Anthropic, but also OpenAI, Google, and Microsoft are converging around the idea that, you know, AI agents are harnesses. And that infrastructure layer that manages all of that execution and memories and tools has a staggering price tag involved. Understanding its consumption is a huge task. And they're all taking radically different pricing approaches to try to understand what the winner is going to be. So some of these I want to call out is like, we've been talking about Anthropic.

16:48Ben:They also launched this managed agents system. And this operates at$0.08 an hour. And Ben, this might be a little bit of an answer to what you were asking for of you put it in their hands and you let them decide the model and how it should run. That's kind of how managed agents work and allow people to kind of push that up into their Anthropic system and let Anthropic handle all of the nitty gritty details. It's probably easier for them and more predictable for them to price that kind of work, especially because it's more batch based. Whereas like OpenAI then immediately countered with an open source SDK because that's OpenAI's play is put the harness and its tools for building in everyone's hands.

17:27Ben:And so they provided this open infrastructure representing kind of like a fundamental split. You know, you see one go fully open source, you see one make it this closed proprietary system. I think it shows a lot about like the diverging ideas. What do you think about some of the differences in which these organizations are pricing their harnesses?

17:48Andrew:Yeah, I mean, you know, cloud service pricing has always been somewhat opaque, but you can generally estimate what your costs are going to be if you know the level of resources that you need to implement. Like you can decide that a certain service is big enough for what we need. So it has an ongoing fixed cost or, you know, maybe a cost that scales based on our estimated usage. But I feel like AI tooling pricing is still just all over the place. We've been hitting on it a lot here. And I think the competition is great, but I'm really hoping that we can get more consistency and expectations around how these things are priced.

18:26Andrew:Like it's kind of obvious that that most many of these companies are operating in unsustainable pricing models today. So it would be very, I guess, comforting to know where the industry as a whole is going to standardize over, you know, over the next year or two so that we could make better decisions about like where we need to be investing our AI usage. But yeah, and then the article specifically calls out that it's really frameworks like LangChain, CrewAI, VoltAgent, like these are the most likely that are likely to be disrupted by all of these agent harnesses that are emerging. Like that seems to be like the thing that everyone wants to productize right now is a harness on your agents.

19:12Andrew:And yeah, this article that we'll link to, it does a really wonderful job at just illustrating the current state of what I think are the two biggest trends happening right now. And that is harnesses and orchestration. You know, everyone on the leading edge is thinking about those two terms. And there's a lot of products now that are starting to emerge in that space.

19:34Ben:Yeah, and things around too about how you operate these tools. Like we've been talking with a lot of folks on this show that have been pioneering a lot of that stuff. Like we've talked about the RALF loop and research plan implement RPI. We've talked with both Jeffrey Huntley and Dex Horsey behind those ideas. From this, it's like another thing we've really learned, for example, is when we talked with OpenAI's Codex team, We had Thibaut Sotio on the show to talk about exactly this, about their play, to make it open, to take a different stance in this conversation, kind of like what you're calling out, Ben, of like, you want them to come together.