BONUS: LIVE: Claude Opus 4.7 Just Dropped. Here's What Actually Changed.

0:08Okay, sweet, we're live.

0:14All right, what's up everyone? I am here with my friend Kyle, who is a developer, and we are going to test out Opus 4.7, which just came out as well as on Kyle's side we just saw that there was a new open source release of Quen 3.6 so we're also going to be playing around with that so uh shout out in the comments uh if you have anything you'd like us to to test out but uh but we're going we're going live right now and we're gonna we're gonna try this um okay

1:05Okay. So, first and foremost, what I'm going to do is I'm going to pull up the Opus Details. I'm going to go ahead and grab that trail.

1:28Let's look at it while people load in.

1:37Okay. So first of all, for anyone joining us, welcome, welcome. So today you are graciously being hosted by myself and my good friend Kyle. Kyle, you're now live, by the way, if you want to say hi. All right. Hi, all you people. What's up? What's up? And we're going to be talking about Opus 4.7. Now, if you don't know what Opus 4.7 is, it's the latest version of Claude. Claude is made by Anthropic. and Anthropix, you could call it their like frontier models or their smartest model state of the art is their Opus series. And so the latest one before today was 4.6 and now we have 4.7, which just came out.

2:21And so we're looking at it right here and we're going to go through the benchmarks really quickly, but I watched a great video this morning that came out maybe like, I don't know, 20 minutes after they made this release that I'm also going to recommend people check out and I'll include a link to that in the comments where Nick, I don't know if I'm pronouncing his last name correctly, but Sarave goes through all of these benchmarks in more detail and kind of talks about them. So we'll let you watch that if you want to dive too deep into it. But the key thing to keep in mind here, so when you look at my screen, we have Opus 4.7 on the left hand side and right next to it is Opus 4.6.

3:03And then you also have GPT 5.4, which for For people who don't know, that's ChatGPT's best new model, at least as of this writing. And then Gemini 3.1 Pro, which is the same for Google. And then over here on the right-hand side, we have something called Mythos Preview. Now, if you haven't heard about Mythos, this is the model that basically is the one that scared the pants off of everybody who's used it. it can apparently find zero-day exploits in any piece of software on the internet, and Anthropic is withholding it from the general public while they test it and basically give banks and all the major tech companies access to it so they can sure up their defenses before they release it to people they don't trust.

3:51So fun stuff there. Yeah, we'll see. We'll see what happens there. Kyle, do you have any thoughts on Mythos? or just what you heard about it i mean it's you know it's always hard to say when you can't really use the thing the benchmarks look insane and i you know i have reason to believe that anthropic wouldn't really pull our leg on that so right i'm i'm treating it as a potential game changer but I don't hold my breath at this point ever. Yeah, I think that's fair. I think that's fair. And so the thing that's interesting about the video that I watched earlier that I mentioned from Nick is Nick basically thinks that Mythos is a distilled, or sorry, Nick thinks that Opus 4.7 is a distilled version of Mythos.

4:45And I think there's some credibility to that. And basically the reason he gives is that all of these benchmarks on the left-hand side here for opus 4.7 are higher than opus 4.6 um with a couple exceptions one of them is the uh the agentic search was one that stood out to me yeah agentic search is lower that's right yeah agentic search is lower and so potentially that could be something that they artificially nerfed, basically. And one of the reasons that Nick gave for this, which I think makes a lot of sense, is essentially that could be one of the capabilities that makes it easier to find those exploits that he was talking about, right?

5:35And the other one is agentic terminal coding. Now, agentic terminal coding is higher on 4.7 than 4.6, but not by much. it's like a little bit better. But he was saying that that's kind of like where a lot of the, like the agent's ability to use tools and bash commands and things like that help it hack systems. So that's why he thinks that that was also nerfed, which I think is a good call. But pretty much everything else here, I believe, is higher. Actually, cybersecurity vulnerability reproduction is lower. So that's another sign that doesn't... That's an interesting one. That feels too on the nose to be an accident.

6:19Yeah, I think that's why. And then also, there was something that was flagged here. I think it might be in the larger report by Lizanne Al-Gaivon X. And basically, the long context is actually worse on 4.7. It looks like the chart that he shared isn't on here. So let me see if I can pull it up.

6:48I'm on my personal computer right now, so that's what you're seeing. Well, let's see what notifications we get. I will say on their other benchmarks, one of the more impressive improvements is the visual reasoning. Seems like it got a huge jump up. So I have reason to believe they focused a lot on vision, which is something I've sort of seen them get a little dragged about recently. Oh, yeah. Yeah, it was actually kind of terrible. Like, I can't tell you how many times I'm using it, it being Opus 4.6 to code, you know, like a video game or something like that. And I tell it to fix something with the UI, and then it's like, looks clean, looks good, you know?

7:31And I look at it, and it's like pixel slop. Like, it looks terrible. I'll have to show you an example later. But it's just hilarious. Like, its vision capabilities were, in my opinion, very lacking. and I have to tell it multiple times. That's something I'm definitely looking forward to testing out a little bit. Yeah, the thing that Kyle's talking about, let's see, where is that? Vision, vision, vision, visual reasoning right here. So this is a huge jump up. So with no tools, it's a massive jump up and with tools, it's significantly better as well. So yeah, so and then this page goes on to talk about Project Glasswing, which Project Glasswing is what I hinted at earlier with Mythos, where basically they're working with a bunch of industry partners, sharing Mythos with them internally to shore up their systems.

8:22And so they're kind of talking about how they're working with their team there here. And then they've got some pull quotes from people who they've let test it out. And here's some of their highlights that they flag on their blog. So instruction following, They say Opus 4.7 is substantially better at following instructions. This means prompts written for earlier models can sometimes now produce unexpected results. So where previous models interpreted instructions loosely or skipped parts, 4.7 takes instructions literally. I kind of hate when this happens. I don't know if you feel the same way, Kyle.

9:02but I when when they all of a sudden become more literal it's like technically a good thing but then you have to be a lot more precise with what you're saying and especially if it struggles with long context tasks which I will pull that chart up in a minute you know that means you have to be a little more careful with what you write yeah it is an interesting thing that you could either you could interpret it as an improvement in the controllability of the model or you could view it as like a slight regression on its ability to pick up nuance yeah i think that's probably true especially if it if it is struggling with long context like that does feel like it's a bit of a regression to me um let me pull that shirt up because i think that was i flagged it immediately i was like oh this is yeah here it is so basically hold on we've got a we've got brazil mentioned in the chat what up hey brazil what's up brazil welcome to the show i can't let one of those go by yeah no i i appreciate that i appreciate that yeah thanks for um thanks for shouting out lol mao mythos preview destroys basically everything yeah pretty much at some point hackers are going to get aversions or someone else is going to make something like it then all the banks are screwed well hopefully not because anthropic's giving them all access to it now so they better use that thing like uh there's no tomorrow and sure it up um i don't know kyle what What do you think as a developer, is it possible to fix every bug?

10:42I don't think so. Okay, so there will always be an exploit somewhere in the code, do you think? I mean, we might get to some new paradigm someday, but it's not going to be before Opus 4.8 or 9. what what what the heck do we do then in that scenario like that's just wild to me that's it's gonna be an extremely active area of research i think it's it's just everything's gonna accelerate you're gonna have i don't know maybe you have to put an open claw at every network endpoint oh wow we'll see yeah that's that's wild i could i could see it though i could see it um thanks for the overview yeah what's up middleton middletown ohio is it middleton or middletown let me know um what up what up harmonious crow yeah appreciate it you're welcome you're welcome so here we go this is the one thing i wanted to flag because this concerned me and the reason this concerned me is that um i use a lot of long context like kyle was talking to you about this earlier this week.

11:53Like I'm, I'm a little bit, uh, undisciplined in terms of just throwing a bunch of stuff into my agent and being like, okay, now read this, now read this, now use this, um, and try to give it a lot of information for it to parse through. So for it to go from, you know, Opus 4.6 level, which here you can see, this is their 64k extended thinking version, um, has 78.3 % on this eight needle test. And I don't know exactly what the eight needle test is, but basically the needle test in general is how well can you pick out an important detail from a large, long context window. So the classic thing is like loss in the middle problem, right?

12:38Where it just loses details in the middle. It's good at keeping track of things that you say at the beginning and things that you say at the end, but in the middle, it just loses focus. Um, so to go. I still have the habit of, um, prompt at the start context in the middle reprompt at the end, which was totally necessary for the first long context models. The newer ones probably shouldn't need that, but it's just kind of baked into my brain now. No, I think that's a good, that's good policy. And I should probably do that as well because I became very undisciplined with 4.6. As you can see, like the, the best jump between 30, like, like this was like a, this was like a twofold, like this is double as good as like the next.

13:16ones and so to see it drop back down for three points or 4.7 uh is a little concerning so that makes me wonder i don't know should i switch to 4.7 maybe for some things but not for others yeah it could be a thing where you want where you want 4.6 to take your million context and bring it down to half a million so 4.7 can handle it better it'll yeah take some testing to see that honestly makes a lot of sense um what up sf bay area oh hey nice nice to nice to see you john peters let's see what's all who else is here chris taylor oh wow you got another uh person from middletown in the chat that's awesome um san diego land of enchantment don't know where that is but right on um all right so should we should we mess with this stuff should we should we put it to work I think we should yeah I think we should let's see what it can do all right so what I'm gonna do is so I was working on a project on my cloud code over here and I'll give you guys a little preview of what I've been working on um so this was a game that I've totally vibe coded with claude 4.6 opus so i have written no code on this whatsoever and it is basically i i gave it a couple prompts to start off with and then i've just been iterating on it as it goes so this whole thing i've been working with is opus 4.6 right um and so i've got it working on something right now if this is the first time you've ever seen claude code um in the desktop app um when it's working on with its like preview mode here this is what it looks like and this is nice because Claude can actually see the screen but this is what I was talking about earlier when um basically like it will like for example let me see I'm gonna hit new game here I'm gonna write um test and I'm gonna go open world so originally what so okay let me give you the pitch of this game actually So this is basically like a Final Fantasy Tactics style game set in Renaissance Italy, right?

15:33So I'm really trying to go for the Final Fantasy Tactics vibe and aesthetic. That's a turn-based strategy for anyone who hasn't played that. I can't think of a bigger example. Yeah, yeah. No, I think that's probably the best way to put it. And I'll show people what it looks like right now. But I was trying to get it to set up this map, right? and originally it was making a map of Italy it was like the most complete garbage thing I've ever seen and it's like this looks great it looks just like Italy and basically now what I'm gonna do is I'm gonna try and test it out and see how well it does with the visual the visual reasoning now that it knows well how to basically how to see better okay so we got people from New Mexico We've got people from Newcastle, Australia.

16:26What's up? So basically, as the models improve, they anticipate a need for prompts and sort of build them right in. Yeah, I think that's true. So basically what you're saying is as the models improve, they themselves anticipate the need for prompts and build them into the system. Do you think that's what they're doing, Kyle? I would say there's something close to this where they're generally taking the ways that people are getting the most out of the models, and then they're building that into the subsequent models. Like before we got internal reasoning in models, the big trend was that you would just tell the model, think out loud, write a plan, think it through, and then respond.

17:12And so then the new models, that just became an internal functionality. And then they're also, you know, focusing on what fields people are using them the most in, and they are trying to expand their capabilities in those specific fields. As for like specific prompts, that's less baked in. Yeah, yeah, I think that's right. So what's interesting about this is I don't have in this window, I don't have 4.7. available to me. So what I'm going to do is I'm going to start a new session. I wonder if the app needs an update. Yeah, let me relaunch it real fast. Probably, that's my guess.

18:06Used up the limit on a deep research and it session limited you. Yeah, well, tell them what happened to you the second you tried 4.7 this morning. yeah i opened it up i gave it one prompt to make a svg of a of a rising phoenix and it got perfectly through it but then it just died immediately at the end because i had already been cranking 4.6 so much this morning so i i have heard from a lot of feedback on x and maybe i'll pull up some of those posts later that it does burn tokens like a lot more interesting yeah it's the the pricing is the same point as 4.6 that's right um but there's always you know different models will prefer to reason a different amount so you can sometimes get a model that costs the same amount but you ask it a question it thinks twice as long and it costs you sort of twice as much yeah oh cool and we do have 4.7 now i see somebody asking to ask questions just ask the questions yeah yeah you can ask questions go for it um does it use usage quickly i think we kind of just covered that um yeah and uh so if you guys have a prompt or something you want to see let's let's test it out uh right now what i am going to do is i'm going to test it on this renaissance bomb fantasy tactics game so um i'm going to say uh what were we last working on per your last plan.

19:37Also, did you notice that they changed the desktop UI recently? Do you use it or are you using it mostly in the terminal? I'm pretty much either quad code or I'm in DS code using one of the various integrations into that. Yeah, yeah. I actually switched between the two. For this one, I pretty much only worked on it in the desktop version just because it has that nice side view where you can actually preview what you're working on. but for other stuff I'll use a terminal emulator like Ghosty somebody asking if ChadGPT 5.5 is coming I have heard no sign of it but I will confidently say yes so there was rumors that Spud was coming soon Spud is like the internal name for whatever 5.5 will be and in the case of if they're launching it today the only thing i've seen is that tibo or tibu uh from the codex team said i'm feeling codex-y like 40 minutes ago so that could mean that something is launching but um all i know is these guys don't like to let any other company have the spotlight for too long so exactly exactly um great let's keep working on that

20:57All right, so I'm going to let that crank. Let's see. What does it excel at? We just did a quick overview of that, but while I'm waiting for this to kick off, I'll go back to Claude Opus 4.7. This is what the Anthropic team says it excels at. So instruction following, which we covered earlier, improved multimodal support, which is the vision thing that Kyle mentioned earlier. It has better vision for high resolution images. It can accept images up to 2500 pixels, more than three times as many as prior cloud models, and this opens up a wealth of multimodal uses, so that's good to know. For real world work, as well as its state of the art score on the finance agent evaluation, which we did not talk about, but it's good at finance.

21:46Let's see. The internal tests show that Opus 4.7 to be a more effective, more professional, sorry, more effective finance analyst than 4.6, producing rigorous analysis and models, more professional presentations, and tighter integration across tasks. 4.7 is also state-of-the-art on GDPVal, which is a third-party evaluation of economically valuable knowledge and does doing work across finance, legal, and other domains. Also, it's better at using file system-based memories. So it remembers important notes across long multi-session work. So maybe it's worse at long context, but it's better at like pulling in context strategically.

22:27You think there's a thing to that?

22:32That could be the case. Yeah, you might be able to have it work with your file system and then more selectively pull in the context and be able to get the correct information without bloating it. But it does seem like that context fall off is pretty bad. That's one thing I might expect them to change in a later push. Yeah. Yeah. Especially if people blow the whistle on it like I'm doing. Cool. So let's see what's going on in the chat. Sweet, thanks for the overview. What do you guys see using 4.7 for though? Oh, yeah. So I'm going to test it on this coding use case right now. effectively complete, no audio assets, no separate main menu, procedural art only for phase 7.5.

23:22I'm going to say read the whole code base and confirm those are true. Because sometimes what will happen is when it looks at an old plan, it will make conclusions, but then it will go read the code base. It'll be like, oh, you're actually done with that. So it's good to tell it to just like read, like read the whole plan. Yeah, I see some mention of haiku up at the top there too. It sounds like it might have farmed some of that out to a sub-agent. Yeah, that's one thing that this one does a lot of, it seems like.

23:58Which I don't really like that. I'd rather you know if you're going to read something and you're going to make conclusions about it, use the smartest version available. Why are you... I think it's configurable. okay um i do something a bit similar to that where i try to use a local model to summarize short bits wherever i can to kind of save my expensive tokens yeah but yeah there's there are some times where you just it's you would rather have it right than have to correct it yeah no we were talking about this earlier where basically i can uh be a bit of a token waster at the moment because I'm on the max plan whereas you're much more efficient because you are not on the max plan and I think that there's good practices to that that everyone should use.

24:54Somebody's asking what we see ourselves using 4.74. Yeah you want to answer that? Like it sounds like for you the answer is almost everything. Oh yeah well I yes so as I just said with the max plan, I do tend to use the highest intelligence version for as much as possible, because if I'm not doing it myself, then I want the system I'm offloading it to, to be as close to like, like doing it perfectly as possible. Right. Um, but that's, I guess, a privileged position. Yeah, I personally, I'm a little more shifting my workloads. Kind of a common workflow I've been following lately is that I'll start with a state-of-the-art model, maybe 4.7 or Gemini 3.1 Pro.

25:47And I'll talk with that for a bit to kind of form a plan and then ask it to make a really thorough plan. And then I'll pass that to a, like a coding agent model. Usually it'll be like a medium one, maybe a Gemini Flash or a Sonnet, and I'll see how far that can go. And then usually the 3.1 Pro or the 4.7 now will come in to kind of clean up the last bits. But that way you use the smartest model to get the plan. You build a lot of the context, you know, doing the cheap, just look up this file, write the simple functions. And then now that all that context is there, it only takes one quick blow from the from the state-of-the-art model to clean it up yeah i like that i i think that's great um i should be doing that honestly uh but not if you have plenty of max yeah yeah well someone just said you guys should give out max plans on the newsletter totally i'm gonna reach out to anthropic and see if they'll do that the problem is i feel like they don't want people to buy max plans anymore because when you max them out it can cost them a lot of money so i don't know if they're gonna try and uh i don't know if they'd be down to give them out but i like that idea we should try it um when do you think air will start to be funny not like forced funny but integrated into how it communicates um i think i think it's funny like one percent of the time right now like every once in a while it'll squeeze in a little zinger that gets a real like like a heh out of me yeah I think that's accurate I will say I do use Claude when I'm writing the newsletter and sometimes it comes up with pretty good jokes so it's not it's not uh it's not not funny but it has to be in the right context I think

27:41let's see here another YouTube channel right now is about to have six instances open at the same time. That's wild. Oh, look at this! SubAgent was wrong! Audio Assets do exist! Okay. All right, we like self-correction. Let's say we're gonna... This is the last message I had queued. And I'm gonna send that to it. Let's see.

28:23Meanwhile, so I have my terminal agent over here. Let me see if the model actually shows up. Not yet. I might have to restart it.

28:43So this is another thing I was working on. For people who don't know, so a couple of weeks ago, we interviewed a company called LTX. And LTX created this tool that was called LTX Desktop. And it's like a desktop app where you can essentially generate your own images or videos and then send them to a timeline and edit them. And what I've been doing is I've been expanding it because it's open source. so i've been adding like i added a director tab where you can enforce a style across all of the shots that you generate you can give it a mood board you can give it a cast list so anytime this character's name shows up this is the guy that you see a location registry so it's the same for a location and then a shot list card and the main thing i wanted to do was create a version where you could essentially as you're writing the script start generating shots and you could have those shots kind of load in as you're writing the script so you can sort of see what it looks like I think this is the way that generative AI in film should work where you're basically like writing it from the screenplay standpoint and you set up all of this system around it to enforce it so that it essentially like enforces your style across the entire script.

30:09And then, you know, there's all these sort of add-on things I've added since then, like an outline mode with a lot of different flows. I added a shot list where you can see all of your shots. In this case, I didn't generate them yet. Storyboard mode, so you can just see just the storyboard as you go. A whiteboard where you can just do ideas. And what I'm working on now is a production mode where you can kind of see the whole script and watch it over here, but that's still in progress. And then I also added a little agent mode here where you can actually talk to the agent and have it do some of the some of the things you want it to do for you.

30:52So I'm going to try and work on that over here with 4.7 if I can pull that up.

31:05It's got you on 4.6 right now, I think. Yeah, so I'm going to have it. So it kind of laid out a basic plan for me here. So I'm going to make it do a plan in plan mode. Or you know what I could do? What I could do is I could copy this. Like, hey, this is what we're working on. Let's make a plan for it. So let me do that. While that's cooking, I'm going to open a new window.

31:41Do you ever run it with skip permissions? Not full YOLO mode. Do you do that?

31:54Oh, yeah. I made an alias on my bash to where I just type YOLO and it comes up in YOLO mode. That's awesome. I do it mostly on my on my Linux laptop that doesn't have all of my life on it yeah good call good call yeah I have a I have a PC where I can do that let's see what's going on in the chat oh yeah I I actually I've been wanting to um share the link to that um but I just wanted to get it to a point where it was like actually ready to ready to share with people um but I'll do that once I do the blog version of this I'll share a link to it let's see someone says I feel like we all just a bunch of idiots trying to create stuff from every format release that's true cloud is a public company cannot have millions of people writing their own I mean that's an interesting angle basically yeah like when when will they pull the plug on solo developers being able to use cloud models to build competitors to like multi-billion dollar SaaS companies I don't know I don't know if it was ever is ever in their interest to to do that but potentially they will just create products where you just run them and it will do exactly what they designed it to do that's a concern for sure where basically they just are saying okay you don't get access to this anymore we're gonna we're gonna tell you what you get access to do Do you think they would ever do that, Kyle?