Reading model benchmarks like a pro, Mythos is looming, and Claude talk caveman, save big token

0:05Andrew:Andrew, what opinion have Claude Mythos? Andrew have many opinion Claude Mythos.

0:13Ben:Claude Mythos, big model.

0:18Andrew:Yeah, all right. We'll get into why we're talking like a caveman. But yeah, Andrew, Claude Mythos, it seems like everyone's talking about it. Is this what you're hearing out there right now?

0:27Ben:Oh, yes, indeed. And welcome to the Friday Deploy, y 'all. It is true that people are talking about Mythos out in the wilds here at HumanX. I've definitely heard it on people's lips here on the expo floor and definitely the security minded companies have been top of mind for them. I think Mythos is a really fascinating kind of sea change event in model capabilities, especially in a realm where there's already a huge amount of like disparate abilities between attackers and defenders in the cybersecurity space. Anything you're going to put out there that defenders can leverage, an attacker can leverage 10 times better and faster and more aggressively.

1:06Ben:The really unbalanced world. So Mythos entering it is, you know, I think really trepidatious for some people. What have you been hearing about Mythos?

1:16Andrew:Yeah, I mean, the cybersecurity, we'll get into that here in a moment. You know, I've just kind of come to expect that every, I don't know, every month now, maybe every two months, the timelines seem to be condensing where some new major incremental improvement comes out. So, you know, the current Frontier models are pretty amazing. So I only kind of expect the next ones to sort of take it a level up. So, but yeah, so let's get into it. Yeah, as you mentioned, this is the Friday deploy. I'm your host, Ben Lloyd Pearson. And I'm your host, Andrew Ziegler. So this week we are covering this AI cybersecurity arms race that we were just discussing.

1:52Andrew:We'll also go over how to read AI scorecards and benchmarks. We'll talk about some open source frontier breakthroughs. And then we're going to get into why Andrew and I started this episode talking like caveman, because it's actually a really cool story.

2:06Ben:Andrew, excite him.

2:07Andrew:Yeah, exactly. But let's kick it off with Project Glasswing. So this is a new thing that has been announced from Anthropic. They're partnering with a bunch of major tech companies to use advanced AI models for finding software vulnerabilities before attackers do. So we were getting into this a little bit, but Claude Mythos has been previewed, or Anthropic released a preview of it. And along with it, it seems to be becoming a lot of just like concerns and warnings about how it could be used maliciously. One of those ways is that it's finding a lot of security vulnerabilities in very commonly used libraries.

2:42Andrew:Like, for example, it found very old bugs within FFmpeg that have been undetected, you know, despite being scanned thousands of times over the years. And it really does, you know, highlight that there is this urgent arm race between AI-powered defenders and attackers, you know. So in this, Anthropik is partnering with like the Linux Foundation and a whole bunch of other like big logos. I don't remember the whole list, but there was a bunch of companies on that list. And they're committing 100 million in usage tokens to help organizations scan their systems. So it's really great to see them being proactive.

3:21Andrew:And, you know, I think the mythos has like some doom and gloom around like how security is about to become a nightmare. But at the same time, I think it's also really good to focus on the good things that AI can do. Like longstanding security vulnerabilities in FFMPEG should be fixed, whether or not we have AI. So we can also use AI to do those things. So yeah, Andrew, what are you thinking? Because I know you've been following some of this stuff pretty closely.

3:46Ben:The mythos rollout, or rather development, and then them partnering with organizations in this Project Glasswing initiative, I think is a really great play to see from Anthropic, who sees itself as a partner with the software ecosystem. The organizations that they reached out to maintain the software and operating systems that power the entire world. And by just partnering with that small, very concentrated, large group of organizations, you get like a really widespread over all of the tools we use every day. So Anthropic was definitely really smart in partnering with those people first because they acknowledge this inequality between attackers and defenders in the cybersecurity space.

4:26Ben:and I definitely think that this is it speaks to the ethos I think of Anthropic as well. I don't know necessarily OpenAI would have made that same decision. I think they probably would have more welcomed the disruption as opposed to trying to gently roll it out. However, of course, Anthropic 2 I think also has ulterior motives here. We're living at a time where serving Opus to their customers is extremely difficult for them at scale right now. There's been a lot of talks about folks having different experiences with Cloud Code and with Opus in particular, getting good usage out of it, getting the limits and the mileage out of it that they used to.

5:07Ben:And there have been strains around delivering that compute. So when you're talking about something like Mythos, which is just an order magnitude larger model, you're talking something that's more expensive. So at the same time that Anthropic has to partner with these organizations to improve their security, they probably also got to beef up their data centers to actually deliver mythos. So I do think that there's like a dual play going on here. But Project Glasswing, I think, is a great initiative to see from the foundation model provider.

5:37Andrew:Yeah, I mean, it feels like an arms race. You know, we have people that want to do really big, good things with AI and they have to race to stay ahead of these malicious actors that are getting their hands on the same tools. But, you know, like I said, these issues would persist even if we didn't have AI. So they would be out there. They have been out there in the wild, allegedly. So fixing these vulnerabilities is going to be super important. I think it's actually a great thing for society at large to have something this powerful that can solve security challenges. but yeah I'm going to be watching this project pretty closely because you know there's so many big companies about it I really like the partnership approach as you mentioned and you know just generally having having AI agents you know and all these organizations coordinated on using AI to solve really big challenges is you know that's a significant development I think just for the industry at large I agree all right Andrew let's talk about benchmarking 101 AI tool school.

6:37Andrew:So what's this article?

6:38Ben:Yes, this is an article on Substack that I really loved. It's from a blog called In the Weeds, where they do exactly what the title says. They actually get really in the weeds on some of these more technical and nuanced topics, particularly around AI. They have a lot of materials around learning different things. And one of the most recent ones is benchmarking 101, understanding how to read a model card, understanding the relevance of a benchmark and how to understand that model score on it and what it means. There's a lot of fluency involved in understanding the competency of a model and scorecards have only gotten more complex.

7:13Ben:And if you don't already have experience and background in understanding them, they can be really hard when a new model comes out and you see the benchmark results to actually think for yourself. And, you know, should I try this instead of what I'm currently using? What might I expect? What should I be looking for? and at the same time when people use models that improve like you move from Opus 4.5 to 4.6 you can feel it get better or you can feel it perform better but maybe you lack the verbiage to explain or to really pinpoint what is different and what this article aims to do is to make you more capable and fluent in understanding those things as they come out so that you can better evaluate those for your team as they happen a really great shout out about this too is that they make it so easy to learn.

8:00Ben:Like you can just go clone a GitHub repo where they put all of the resources and then you go ask that repo your follow-up questions and dig more into it. And I think this is like the future of learning. This is already how I package up most things for other people to consume outside of this podcast. It's like put it in a repo or put it somewhere where they can just ask questions of it. So really smart teaching approach from them. And I highly recommend you go check it out. Like the biggest takeaway is is that in a bunch of models are scoring 90 % or higher on a benchmark, then the benchmark doesn't matter anymore.

8:31Ben:And the older a benchmark is, the less it matters too because the more likely it is to be in someone's training data. And the best benchmarks are ones that models universally struggle to solve, but humans universally don't struggle to solve. And a great example of this is the ARC AGI-3 benchmark, which the best foundation model out there can't even get 1 % on, but any human can score 100 % on. These kinds of benchmarks are really important for actually delineating the capabilities of the model. So strongly recommend you check this one out.

9:02Andrew:Yeah, first of all, call out the GitHub repo approach

9:05Ben:to their content.

9:06Andrew:You know, I love seeing all these new content strategies emerging in the AI era, like new ways of packaging up information for both humans and agents to consume. So it's a really cool approach that we might learn some stuff from. But yeah, this is a dense article and in a good way. You know, it's very information dense. And it is probably the best breakdown I've seen on how benchmarks themselves work. Because, you know, whenever I've seen like these ratings on like it scores a certain model scores a certain percentage on a benchmark, I kind of my eyes just kind of glaze over. I've never really understood.

9:43Andrew:I'm like, cool, I guess that sounds awesome that they can solve tests, you know. But it does a really good job at like breaking down the success rates for various tests, you know. And if you have a benchmark that, for example, has low average scores, those are the best ones for differentiating models. So, you know, if everyone's getting an A all the time, you really can't trust that test anymore because it's been, for a variety of reasons, it's no longer a good way to judge model performance. you know and and the part of the challenge that exists in this space is that like models are improving at this like accelerating rate and solving tests practically as quickly as they can be built it seems like so you know we're talking about mythos and how all these new and we're actually going to have another story here in a minute how like there's just so much innovation happening on the models themselves that the tests are really struggling to keep up and then not to mentioned you have good heart's law like that still applies in the ai world right like if you tell your model it's really important that you pass this test it's going to it's going to figure out how to pass the test but yeah and you mentioned arc agi 3 uh this really does seem to be like a new bent a new way of benchmarking like it's a new it's a new benchmark of benchmarks in a sense like because it has dramatically shifted uh the performance you know and i'm like they're saying the the best frontier models are scoring under one percent success rate whereas a human is getting a hundred percent like pretty easily um and it's and it's like just based on like these random puzzle games it's actually really cool like i played around with it a little bit and was like wow this is actually really neat that like like for a human i just intuitively figure it out you know without a whole lot of thought um but you know i i actually am wondering if like we're going to start to see this like pattern of behavior emerge or or a pattern emerge where we see patterns of behavior that humans instinctively follow, but it's very difficult to replicate that with an LLM.

11:42Andrew:Solving a game that has no instructions on how to do it, just you have to intuitively figure it out by playing the game, is so far a way to differentiate a human from an AI actor. But I also think there's probably design, for example, I think is particularly for game design. I think that's one of those skills that humans have that is really difficult for an LLM to replicate. And there might be other things like human humor, beauty, like being able to judge if something is funny or looks, you know, looks interesting or, or attractive. Uh, you know, those may be other tastes that like humans are uniquely positioned to understand, but I do want to make one final point on this.

12:23Andrew:So I feel like arc AGI three is doing exactly what I've always wanted from this era. And that is bringing back the web arcade. now imagining like wouldn't it be cool if we had like online communities that use puzzle games as the gatekeeping mechanism to participate in them like oh like you can't hang out with us unless

12:44Ben:your world score and like this good or whatever you know you have to solve you have to solve like

12:49Andrew:these four puzzles with 100 accuracy before you're allowed to like comment in our community You know, like stuff like that.

12:57Ben:I love the idea of a puzzle-gated community as a puzzle person myself. So I welcome it.

13:03Andrew:Yeah, yeah, exactly. All right. So let's talk about some of these other innovations that are happening in the frontier model space. So a whole bunch of updates coming out in a short time, Angie. What do we have going on here?

13:13Ben:So in the last week, it's kind of like a blink and you miss it scenario. Four open models have hit the scene that have totally transformed the ability of an average user to own the capabilities of a frontier model. There have been four major AI models released in the last week. One of them being Gemma 4, already a really well-established model library of on-the-edge and small language model for devices and stuff. You also have models like Bonsai, which is a much thinner parameterized model that gets great performance on small machines. You have Trinity, which is a performance at around like an Opus 4.5 level and less than 95 % of the compute and run cost.

13:58Ben:And you have Holo3, which is a state-of-the-art desktop automation agent. And all of these are released under an Apache 2.0 open source license. And this is a dramatic shift in how these models have been created and put into the environment. Gemma and Google, they've never put these models out under this kind of license. And Apache 2.0 is one of the most permissive open source licenses, allows you to fork it, modify it, and then build a business and sell a product on top of it without owing anything back to the original provider. All four of these are offered under that same kind of license. And so what you're going to start seeing are companies that have up until now been pretty beholden and dependent on using OpenAI or Anthropic to get the large scale compute they We need likely doing what we actually saw Shopify did last week.

14:49Ben:We covered a story where they used Quinn to fine tune a local model. And in the process, unlocked a multi-agent architecture that was profoundly more cheap and profoundly more effective for them because it was trained on their data. And it was a model that they owned because it was a Quinn model. This is kind of like a similar kind of trend where this is opening the door for other software leaders like those at Shopify to look at their compute costs and look at the problems that compute is solving. And then look at these models that are now out there and say, how can we fine tune and serve these ourselves in order to reduce the costs?

15:26Ben:And obviously there's boundaries and things around this like getting the machines and the GPUs that you need. But when you compare that to a monthly bill from something like OpenAI or Anthropic, it's likely still an order of magnitude cheaper. I think it totally shifts the economics around models and the accessibility for them. And I think you're going to start seeing a lot of specialized fine-tuned models in large and small companies. What do you think, Ben?

15:53Andrew:Yeah, well, I don't want to say I told you so too early, but it really does feel like my theory of the commoditization of model capabilities being like one of the prevailing trends of the near term future. You know, things are going to get cheaper and easier to run at a higher level of quality. But yeah, I like that you brought up Shopify because the reason we covered this or one of the reasons we covered that story is that when we're speaking to friends of the show out there who are engineers who are building agentically now and have adopted this agent orchestrator mindset, so often this is just the mind of an engineer.

16:32Andrew:they immediately wonder like, well, can I run this locally? Can I, can I like own the infrastructure that, that can I like break it apart and actually like own the component parts of what I'm building here in this orchestrator? Uh, and a lot of that ends up in like trying to look into stuff like Quinn to see if they can run, um, these tasks, uh, locally. Um, but you know, I, at the high level, I just think we're in an incredibly interesting space or place right now with frontier model development. You know, we have stories like Anthropic with Claude Mythos coming out where the the highest tier of capabilities are continuing to increase um but then we also have all these stories here about ai is getting more and more efficient and easier to run on cheaper hardware at a lower cost sometimes locally you know um i think it was gemma they were saying like one of the models you can run on a raspberry pi now so like everyone go dust those off and make it your your already dusted off then we already we already got a rack of them but yeah Yeah, you're right.

17:28Andrew:Yeah. So, you know, it only makes sense to me to do more and more of this work on your local machine or on infrastructure that you own. You know, particularly as we're seeing more agentic systems emerging like OpenClaw. And one of the stories that was in this article about the Halo 3, you know, which has, you know, they have that computer use agents that you just point at your desktop and can navigate around and do things for you, use your web browser. I actually want to call out just some like big issues with this before, you know, just sort of separately. You know, first of all, is that like, you know, our desktop experience was built for humans.

18:06Andrew:It wasn't built for agents, you know, and I feel like instead of trying to force fit our agents onto an experience that makes sense for us, we should be building the experience that makes more sense for our agents. you know so to that end like the user space of of a typical operating system just isn't set up with things like a basic permission schema for your ai agent and you know and that specifically is a recurring theme we're seeing with ai like the software experiences we've built have the assumption that there will be a human at the center of it operating operating it all but yeah i mean to that end you know there's just there's so much happening and even if i have some issues with the specifics of where this technology is today.

18:48Andrew:Like, I still see the promise of it and I'm looking forward to like seeing additional iterations. But like right now, it just feels like we're kind of in a peak AI moment because of it's, you know, it's the slope on a slope thing. Like things are getting better.

19:01Ben:It always feels like we're in a peak AI moment right now.