Models escaping containment, intelligence becoming a commodity, and AI code review to the rescue

0:04Andrew:So, Andrew, tell me, are you Pelican Maxing yet?

0:08Ben:Not yet, but I'm getting pretty tempted after reading this article that kind of broke down the idea of how Frontier Lab could potentially game the very famous Simon Willis and Pelican riding a bicycle benchmark that he's been running on every LLM release really since they started. And as you could imagine, a pelican riding a bicycle is not a picture that is in the training data of any model. So it represents a novel generation. And there's been some recent developments that some folks think that model labs are trying to game this very specific impossible to beat benchmark, which just cracks me up.

0:46Ben:But I can't say that I'm in line to do it the same. What do you think about the whole idea of people gaming the very silly pelican on a bicycle benchmark?

0:56Andrew:Yeah, well, first of all, I realized that this article is all about like getting rid of this myth that models are out there, like trying to learn how to train on this challenge of illustrating a pelican riding a bicycle. But, you know, what I really love about this, the pelican test to begin with is that the possibilities here are just like endless. You know, so even if even if the models were to benchmark themselves on a pelican riding a bicycle, there's so many other variations that you could have of this. Like you could have like an elm tree riding a giraffe or, you know, my favorite my kid's favorite, which is a butterfly driving a truck.

1:34Andrew:So, you know, shout out to all the Casper Baby Pants fans out there.

1:38Ben:That's amazing. My kid loves that song.

1:41Andrew:So, you know, but yeah, it's it's really interesting. And you know what, we're going to be talking a lot about different ways to maximize success on benchmarks today, because this is the Friday Deploy brought to you by Linear B. And I'm your host, Ben Lloyd Pearson.

1:55Ben:And I'm your host, Andrew Ziegler.

1:57Andrew:And this week, we are covering AI escaping its sandbox, the open source commoditization of AI and whether or not you should be afraid of it to begin with. And then finally, the agentic code review loop. And we got a bit of a fun story at the end to wrap up with too. So, Andrew, let's just dive right into it, because I believe this is your prediction of AI escaping the lab, so to speak. So let's talk about this open AI and hugging face incident that happens and how they appear to be partnering together to resolve it.

2:29Ben:Yeah, this is a really fascinating story. And while I usually love to be right, I didn't really like to be right in this case, the idea that an AI could escape its testing environment. So this is a news story that broke this week. It made major headlines about a model that existed inside of an open AI sandbox that managed to autonomously chain a zero-day exploit and stole credentials to break out of its environment and access Hugging Faces production infrastructure. And it pulled data from its database. And you know why? Just because it was trying to get the answers to a benchmark or a test that was trying to pass.

3:07Ben:It's helping.

3:08Andrew:It's helping.

3:10Ben:at any means possible, right? And the model wasn't even trying to act maliciously, which is the really key thing to pay attention to. It was hyper-focused on solving a problem, and it used an extreme unintentional means to cheat. It escalated its privileges. It moved laterally. It even executed code. And it did all of this without anyone noticing. The ones that did notice were actually the security agents over on the hugging face side that noticed it in real time and were able to contain and triage it. So again, this is like a firefight's fire situation. We're in an unfortunate situation where defenders have to be utilizing agentic technology to protect their infrastructure just because the threats are so autonomous and so at scale and move so much faster than humans could possibly consider.

4:00Ben:Now, the one thing that really stood out to me is that if I were the CEO of a company and my technology had just hacked into the production database of another company, I probably would have asked my lawyer to be involved in writing the press release about it instead of like my marketing team. Because really what this sounds like was when OpenAI talked about this and ultimately reached out and they're going to be partnering with Hug It and Face, it became like a big message to the market of, hey, these models are really great and hey you got to buy more models to protect yourself from the models and it left a pretty poor taste in a lot of folks mouths and it's leaving folks wondering you know is open ai even going to be held responsible for what is ultimately you know enact a criminal act that their that their model performed on their behalf it also even points out like the idea of like ownership and accountability who's responsible in that instance was it the engineer who triggered the test you know there's just so much to to be said what do you think of this really interesting

5:00Andrew:development yeah well i and my initial thought and you you're kind of building on it is that you know some element of this requires us to essentially take the words of open ai at face value um so you know i'm kind of reserving judgment on like the intentions of this rogue agent you know um there are ways to make to make it look like it's something that is supposed to be helping but you could have, you know, like sort of quietly influenced it to be, to behave maliciously. And kind of to your point, you know, it does, but it does also show how on the surface, you know, AI does look very dangerous, you know, because it can exploit these weaknesses,

5:38Ben:even unintentionally often.

5:40Andrew:But, you know, I do think it's also important to recognize, because we're actually going to get into this a little bit too, that, you know, it shows how AI can actually help identify weaknesses to fix things. So I know that it feels kind of like a marketing pitch for these frontier model companies, but I don't even think we need the frontier model companies to achieve security, to leverage AI to achieve better security. And I really do believe that over time, I think we're going through this awkward, messy middle phase, but over time, it will more and more become a tool that helps make your security posture stronger so that when open AI's agent goes rogue, you have ways of catching them and shutting it down.

6:24Andrew:But there's going to be sort of this nonstop arms race, right, between malicious actors that are out on the internet using the latest tools to try to hack organizations and then the security teams at those organizations who have to constantly monitor and harden their infrastructure. And I think AI is going to be a very powerful tool for both of these groups of people. Here at Linear B, why we care about this so much is that we really do value keeping visibility into how AI is impacting your entire SDLC, because there's all sorts of new vulnerabilities that are being introduced through the use of it.

7:00Andrew:It's really important that you just have awareness of how your software quality is being impacted by all of these things. So I think we're going to learn more about this situation. And actually, I think one of our next one of our upcoming articles, we're going to talk a little bit even about how Hugging Face responded to this and what it means. But before we get into that, I want to talk maybe a little bit about some of the open source models that are coming out of China. So what do we have here, Andrew?

7:26Ben:Yeah, there's been a lot of new model developments that have been coming out of labs in China. There's the Kimi K3 that's been really making a splash in the last week or so, simply because of its extremely high scores. In fact, one test claims that it beat Claude Fable 5 on a front-end coding arena benchmark. And the Moonshot AI company that produces Kimi, they delivered the largest open-weight AI model ever. And it's really interesting to really evaluate the environment in which this is happening. Because we're talking about a foundation-level model that has as many parameters and weights in it as maybe something like Fable.

8:07Ben:But it's entirely open source and accessible to technologies, to companies to download and to utilize internally. This is like a major problem for folks like Anthropic that have poured immense amount of money and infrastructure into creating the training data, the necessary servers and the people to fine tune and train and optimize these models. And then when they hit the market, there's this flip that happens where competitors or even users like you and I can effectively distill the model into whatever specific use case that we need. And in the case of like you and I, Ben, like we might distill the model, you know, through the practice of using it with skills or otherwise fine tuning it into something very specific.

8:53Ben:This is like where you see organizations like Shopify abandoning open AI models that have multi-agent orchestration. Those are all fine-tuned, highly specialized, effectively distilled versions of stronger models that have been made on the fly for something. And the act of distillation is just part of utilizing the models. But in the case of an underlab using a large foundation model to effectively fully create a new foundation model, it's very feasible. And it's actually very easy because all you need is access to the model. Anthropic tries to be available to everybody no matter how much they shut it down in different scenarios.

9:31Ben:And people are finding really clever ways to game the system. So it really calls into question of like if the if U.S. companies and these organizations are going to just funnel a huge amount of energy and time and money into creating these net new intelligent levels of intelligence. What does it mean that any other lab or someone off the street with a fraction of those resources could effectively utilize it as leverage to jump ahead or arrive at the same level without having to spend all that money? It really actually calls into question the whole like CapEx model of why people would be investing in making the models in the first place, because now it feels more incentivized to wait till a smarter one comes out.

10:18Ben:It's a really tricky scenario for foundation model providers to be in.

10:22Andrew:Yeah, my guess is that a lot of the performance benefits that we're seeing from these new open source models is probably a result of bench maxing largely. um so you know for our listeners who aren't aware it's it's where you use the benchmarks themselves as the the the training data that you uh distill the model to solve so you're distilling a model specifically to solve benchmark challenges that exist out in the world um happens to basically every benchmark that gets gets created um but i also don't think that's necessarily the like the worst thing primarily when like cost reduction is your primary objective because this model does seem to be working comparatively well at a few other benchmarks as well with some of the early data.

11:07Andrew:And, you know, and this is really the struggle that, you know, these frontier model companies are going to constantly face. You know, it's never been easier to iterate and be the best at something, but it's also never been easier for everyone else to catch up to you at this incredibly rapid pace. So, you know, I think over time we'll probably see a few companies that emerge as like the winners of like the general purpose AI tool sets, you know, like, like Anthropic and Gemini and, and open AI with chat GPT. Like they're, they're all like these, these sticky tools that, that make it easy to leverage all of this, all the benefits of AI.

11:45Andrew:But I actually could also see this whole cottage industry appear of like niche LLMs that are popping, that just pop up all over the place that are just purpose built for very specific tasks. And, you know, So today, a lot of these benchmarks we have are around software development, which is why I think we're seeing so much advancement and development on the capabilities around that. But tomorrow, they might be generating models that are designed to generate educational materials for children about pelicans riding bicycles or something. So I think it's always great to see a lot of competition in this space, and I hope we continue to see more of it.

12:22Andrew:And, you know, in fact, at Linear B, we spend a lot of times, a lot of time with organizations, like really looking at which models are costing them the most and where they're getting the most productive output from them. And it depends greatly on like the code base and the models that you're applying to it and the situational awareness that it has. So, you know, there's a lot to learn in this space. And it's, I think we're just going to see more of this. Yeah.

12:49Ben:Yeah, but I love the comment you made about the cottage industry. That's exactly the direction I see this going. I think that domain expertise becomes the true moat when it comes to the last front leg, last frontier of model creation. Because you get these like large general purpose models that have very widespread capabilities. And those are going to continue to grow. And you're going to get these like this slingshotting effect between you get a big closed model release and then a big open model release. a big closed model release, a big open model release. I think we're going to keep seeing that over and over.

13:22Ben:But in between, we're going to see really fascinating specialized model releases. You're going to see people like Mira Mirati's new company, like in their Inkling LLM. That is exactly what they're betting on. The idea that they open source the base model and they provide it to folks. And instead of selling the tokens or the model or the compute, you are paying them for the training services, the training platform, because that becomes something that truly you're not going to buy or own yourself. But fine tuning your model and owning your data that the model is trained on, that is something that I think a lot of organizations are going to rotate more into.

13:57Ben:Like this week right now, I'm at AMD's Advancing AI conference. And this is a hardware driven conference. This is people who are obsessed with owning the server racks and their buildings to be able to run these inference at scale and provide it to their developers. And that's really just one side of this conversation. Like you want to own the compute, you want to own the intelligence, at least that's like the through line here at AMD's event. But on the other side, to control the cost is one part, but you also have to prove the value, which has been like a really interesting synergy with how folks are using those tools and then delivering things like code, understanding like, is this code good?

14:36Ben:Does it meet our standards? And when it gets shipped, did it stay in production? Those are like two through lines that these organizations are connecting when it comes to owning and operating on top of their own intelligence.

14:47Andrew:By the way, at 6.30 p.m.

14:50Ben:on August 5th in Soho, London, Linear B is bringing together a group of senior engineers to let off some steam and talk AI strategy with their peers. Because AI is writing more of their code bases every day. But the question remains, are we shipping faster or are we just busier? And CTO Yishai Biri will be on site sharing the latest AI benchmarks from 2.7 million PRs and 250 organizations that made that data possible. So if you lead engineering in the UK, don't miss your chance to connect with peers at this exclusive event.

15:25Andrew:All right, Andrew, but I want to talk about this next article. It comes from Ben Thompson, and he asks, who's afraid of Chinese models? Should we actually even be afraid of all this commoditization that's happening? So, you know, looking at models like Kimi K3 and Quan and, you know, a lot of these open source models are matching the frontier capabilities at a much lower token cost. The author really argues that, you know, tokens themselves aren't the thing that we need to commoditize. It's the intelligence behind those tokens. That's the thing that is getting commoditized and actually provides value in like real output, you know, that solves problems.

16:05Andrew:So, you know, this whole battleground on like token efficiency alone isn't really like the entire answer in terms of like how these models will compete with each other. And in particular, when you look at why companies like OpenAI and Anthropic have such a high token cost, it's fueled largely through demand. You know, there's demand for more compute that they're constrained on being able to deliver. So it comes at a premium cost. If they can solve that problem, then suddenly the token economics look very different compared to these open source models. But there was one really interesting finding in this that I think is really worth focusing on.

16:47Andrew:And that is how Hugging Face reportedly had to use one of these open source models, specifically GLM 5.2, to respond to a recent security incident. I was trying to figure out, was this the open AI incident or was it something else? And it wasn't clear if those were totally connected. But the reason was that the ones from the US-based frontier models had guardrails that prevented them from using it to respond to the security breach versus the open source model lacked those guardrails. So there's just questions on like whether or not like guardrails like that are actually achieving what we're hoping to solve with them.

17:27Andrew:But Andrew, what did you think about this article?

17:29Ben:Well, that's a really smart call out. I hadn't really thought about the idea that the guardrails that are frontier model would or frontier model provider would put on them to stop you from using them in an adversarial way would actually paralyze them in the event of an attack or something it should respond to. um yeah it it's a good reminder that like you know the attackers out there the the cyber security hackers that are using these technologies they're not leveraging guardrails they're throwing the raw intelligence at the problem and leveraging its autonomy to do long-running horizon you know string together things and so if you're not fighting on that same level if you are using a tool that is um effectively has like one of its hands tied behind its back or it has to ask you know, its parents for permission before doing it, then you're just not going to have the level of defense that you need.

18:20Ben:You're back at square one, you're back at like a human in the loop protecting your infrastructure. So it's a really smart call out. You know, up until now, I've really been thinking about like model or like smaller orgs would want to own their models and distill their domain expertise into it and provide it that way. But there's also something to be said about the security teams, the folks protecting the infra to also pick and distill their own open source models and provide them. That way they can have this guardrail free environment. But then you just move the problem to, well, there's no guardrails here.