Srsly Risky Biz: “Rogue AI” isn’t going anywhere

0:04Amberleigh Jack: Hey everyone and welcome along to Seriously Risky Biz. This is our podcast here where we talk about the kind of big picture stuff, policy and intelligence. And we're running things a little bit differently this week. So regular viewers and listeners of the pod will be used to seeing my colleague James Wilson at the helm here. I've been a bit unwell. James has been taken over. And usually we will bring in Tom Uren, our policy and intelligence editor, and have a bit of a chat about Tom's seriously risky business newsletter that he writes every week. But Tom is not well today. So Tom, we are thinking of you.

0:40Amberleigh Jack: We hope you get better very soon. And James and I have kind of scowled over what Tom's written. We figured we'd get together and we'll have a bit of a chat about the newsletter. So we'll do that very shortly. First, though, I would like to thank our sponsor for this week, which is Portswigger, makers of Burp Suite, and you can find them at portswigger.net. James, it is so good to see you. Hello.

1:00James Wilson: Hello, Amberley. It is great to see you as well.

1:03Amberleigh Jack: I'm pretty sure this is the first time you and I have done a pod together. So it's a bit exciting. Yeah, yeah, yeah.

1:08James Wilson: I'm looking forward to it.

1:09Amberleigh Jack: Now, as I said, James, you and I have both poured over Tom's newsletter. If you haven't read and subscribed to Tom's newsletter, make sure you head along to our website, risky.biz, read it, subscribe, it lands in your inbox every week. You will feel smarter every Thursday for having read it. But James, you and I have had a look through and there's a couple of things that we'll talk about today. A little bit later, we're going to jump into the Shiny Hunters group who look like they're in the, I guess you could say, finding out phase of a very well-known thing, having played some rather silly games with the FBI.

1:44Amberleigh Jack: But first, a little bit closer to home for you, James, Australian Prime Minister Anthony Albanese has kind of kicked off this whole models misbehaving discourse again because he is not happy that OpenAI agents went meddling through an Australian government website. And Tom's taken a look at that plus a few kind of incidents. But I guess to start with, James, what have these sneaky agents been up to since we last talked about them?

2:14James Wilson: Are they sneaky though? That's the thing, right? At the end of the day, these models are still doing essentially exactly what they've been told to do, right? They're in a testing mode. They're being told, don't take no for an answer, complete your task. And unlike virtually every other user of this technology, they are given virtually limitless access to compute, which allows them to just relentlessly and prolifically go out there and achieve their aim. And they do it by hacking into things if that's what they need to ultimately do. Nice.

2:46Amberleigh Jack: And there's kind of a spectrum as well, right? Like you look at some of them and you're like, oh, that's a bit naughty and cute. And then you look at others and you go, yeah, that's a little bit illegal and less cute.

2:56James Wilson: Yeah. Yeah. I mean, the Australian one in particular, I mean, like, well, what's the saying, right? No such thing as bad publicity. But in this case, the Medicare issue was not hacking, right? This was just files that were available. There might have been some bypassing of bot protection or other sort of very, very basic rules. But like this is, you know, moderately skilled human with mediocre amount of motivation could have achieved the same thing. But nonetheless, it's, you know, it's brought it into the spotlight. And as Tom talks about in the newsletter, OpenAI is certainly getting very good at rolling out an apology tour, but we're going to see a lot more of this, I think.

3:37Amberleigh Jack: So James, I'm just kind of thinking out loud here. So I may be way off the mark, but from how I see it, these frontier models go off, they break into some stuff, some companies get upset, they might get some support from the frontier companies, they might get a few tokens. Isn't it kind of better that AI hacks your stuff if it's insecure than maybe like shiny hunters who will demand money rather than give support and fix stuff.

4:08James Wilson: Well, I mean the other flip side of that is shiny hunters might not be around for long enough to be hacking into things. Which we'll get into shortly. We'll get into that shortly. But look, I think this is, you know, and there's some nuances in Tom's newsletter that I would push back on a bit, but I think I fundamentally agree on the premise that this is just the new norm. um and i and i agree also with you that that it is better in a sense that these underlying problems and weaknesses and vulnerabilities are being found by um you know relatively well-intended good actors despite you know some of the carnage that can happen with these frontier models but at the end of the day like if we were to catalog where is the actual problem in this whole situation the problem is that these vulnerabilities exist the problem is not that a model can go and find them And the problem is not that OpenAI doesn't watch what they're doing and they, you know, go rogue.

5:00James Wilson: But don't get me started on how much I disagree with that framing. But the problem here is these vulnerabilities exist and they're just going to be found. They're going to be found at greater scale. And, you know, to your point there about, hey, you know, OpenAI comes out and says they're going to do better for Australia. This, I think, is the part that I do really strongly disagree with. They are not doing better for Australia. they are not doing better for the cybersecurity community because they are still operating in this mode of heavily protected, super, super guard railed, opaque processes for getting into their validation programs that give you access to models.

5:33James Wilson: And even then, you don't get access to the latest models. And so they're not doing better. They need to do much better, but it doesn't come in the form of, here's a couple of tokens and sorry, we hacked into your stuff.

5:43Amberleigh Jack: So it comes in the form of what?

5:45James Wilson: It comes in the form of, you know, really, I think, as difficult as it is getting to a point where we're comfortable with embracing the fact that this is the new norm these models will find the vulnerabilities they will exploit the vulnerabilities the question will be just whose hands do we put these models in with the with the requisite amount of compute to find the vulnerabilities and create these exploits that we're seeing and how do we make sure that we put that compute and these capabilities in the hands of the defenders before it becomes commoditized and easily accessible to attackers. The only sort of thing I'd pull Tom up on if he was here, and we do hope you get better, Tom, is that...

6:23Amberleigh Jack: Sorry, Tom.

6:24James Wilson: Yeah, sorry, Tom. You're not here to defend yourself. So I'm going to take issue with one thing, which is, you know, he says that Tom's take is that the models will, the open weight models will achieve these same capabilities. This is where I disagree. And it comes back to my framing of what needs to be done here. I think that the open weight or closed weight models both have equal capabilities to mount the attacks that we have seen, both in the scale of Hugging Face, in Medicare, in all these wikis that get hacked. The issue is not the model capabilities at the moment. It is the compute that is made available to those models.

6:59James Wilson: Because we're not seeing these models sort of just start off with a one-shot prompt where they then say, oh, you want me to go and do that thing? Cool, I know how to hack that. I'm going to go and create the remote code exploit. I'm going to go and do this. No, no, no. This is the result of thousands, if not tens of thousands of agent sessions working across hundreds again, if not thousands of turns over and over and relentlessly getting to the point where they've done all the work to find how to exploit these vulnerabilities to achieve their aim. But that's only possible for a frontier lab that literally has access to hundreds of thousands of these, you know, NBL 72, whatever they are, colossal racks of GPUs.

7:39James Wilson: So what do they need to do to make this better is actually to put that level of computing capabilities in the hands of defenders, not wait for that to become a commodity that is accessible to anyone, attacker or defender, you know, because that then levels the playing field in the favor of the attackers. And that's the thing that will be inevitable and will happen sooner rather than later.

8:00Amberleigh Jack: That sounds really nice, but it sounds like a bit of a pipe dream. But here's you saying that's what's going to happen. So I'll take your word for it. Well, it happens either way, right?

8:08James Wilson: We constantly see models get faster, they get cheaper, and we see the gap between closed weight and open weight models closing rapidly. It was 12 months, it was six months now, standing at about four months capability gap. But what's not changing fast enough is access to that frontier capability with the required compute fast enough such that a four-month lag of an open weight model catching up that could be used by anyone on a basement full of potatoes having the same capabilities. But anyway, we'll see. It's largely out of our hands. We'll see how this plays out.

8:42Amberleigh Jack: In the meantime, we'll just watch some more hacking and sit back with our popcorn.

8:47James Wilson: Exactly.

8:47Amberleigh Jack: Speaking of kind of sitting back with our popcorn and watching the chaos, shiny hunters, James, have found themselves inside the, well, deep inside the kind of adage, play stupid games, win stupid prizes. This is the second piece that Tom has written about this week. So this all started with the group announcing they'd breached FBI systems. They had details on pretty much all the agents and people that had applied for jobs with the Bureau. And the group went and shared some sample data to some media outlets. And now the heat's kind of coming on them and sort of backpedaling and holding up their hands and saying, hey, hey, hey, it's not our fault if this leaks.

9:32Amberleigh Jack: It's all the media's fault. We're not extorting, we're coercing, was it?

9:37James Wilson: I think their line was something like, think of this not so much as extortion but more of coercion. And I would say, I mean, look, all of this boils down to just a stellar display of a lack of emotional intelligence, common sense. Like, you know, these groups, we know they're relatively young kids, right, and I feel like old men, you know, shaking my fist at clouds when I say young kids, but, you know, clearly this is a group of people that lack emotional maturity and understanding of the world and their, I guess, technical capabilities to pull off a hack like this far outstrips their ability to fully step back and think about the ramifications of this.

10:20James Wilson: Because how on earth could you not have thought this was going to go horribly wrong for you? Regardless of what data you took from the FBI, that agency is not going to stand for, you know, this sort of publicity, because I think they started off as like, initially my sense was here that they were talking about that there was a breach to a jobs portal.

10:41Amberleigh Jack: Yeah.

10:41James Wilson: If it was just that, even then, FBI is not going to tolerate that. They're not going to see that as anything other than a complete affront to their, you know, dignity and capability. And they would have stomped them out just for that. But the fact that it's now turned into such a massive breach of data that far extends just jobs is, yeah, I mean, you know, how many adages can you throw at this? Dog that caught the car, F about and find out. It is the find out phase.

11:09Amberleigh Jack: It's quite funny. As a journalist sort of seeing their – in Tom's newsletter, he sort of quoted from them saying, you know, if this leaks, it's the media's fault. It's not ours.

11:22James Wilson: Damn irresponsible media. Yeah.

11:24Amberleigh Jack: If I was sitting in a newsroom and a data extortion group hit me up and was like, hey, here's a sample of a whole lot of stuff that we stole from the FBI. do with it what you may uh i'd for sure be putting it through the legal team but i'd be running with it and i wouldn't expect to be the one to blame yeah no i mean the blame is going to sit squarely on uh these folks and um look and already we're seeing as tom points out in the newsletter some

11:51James Wilson: very both unique and strong language coming out of the fbi yeah tell me about that video i'm not us familiar with how this normally goes, but I know enough to know that this is different in terms of Leatherman, who's one of the senior directors, I think it is, at the FBI, put out a video saying, you know, it's a pretty strong language and quite threatening. I would say the quote here, I think, was the longer you stay in this, the more we learn about you. You know how to find us and we know how to find you. I suggest you reach out first while the choice is still yours. You know, that's really laying down the gauntlet.

12:26James Wilson: I think Tom does a good job of pointing out also that while the language is strong, the facts don't actually quite line up to the FBI's sort of the way they want you to perceive this. They want you to perceive this as, aha, they found the guy responsible for this. We've got this, no problem. But the timeline doesn't work. They arrested this guy on the 16th of September, I think it was. The hack didn't come to light until the 21st, 24th, somewhere around.

12:50Amberleigh Jack: 21st, yeah.

12:51James Wilson: Yeah. Yeah. So look, it's obviously they're doing something. This is something so great. Very fascinating to see the super strong language. Now we wait to see, does this translate into some sort of broad spectrum sweeping up of these guys? Do they scurry away into the darkness and we never hear from them again? Do they rebrand? Who knows?

13:13Amberleigh Jack: And, you know, you sort of pointed out the, and as Tom and his newsletter pointed out, the timeline doesn't quite add up, but the FBI must be stoked with the timing.

13:23James Wilson: Yes. Oh, yeah.

13:26Amberleigh Jack: And I know we spoke about it very briefly, but it was definitely the line that caught my eye in Tom's newsletter. What we plan to do, this is a quote from Shiny Hunters, what we plan to do is not something I'd call extortion, maybe coercion. And James, can you explain it to me like I'm five?

13:51James Wilson: Well, Amberley, I think only a five-year-old would really apply this logic. You know, I can't fathom this, nor can I fathom their later response, which was, it's not extortion. This was when a spokesperson reached out to various journalists even more recently. Look, it's not, in capitals, extortion. It was merely a marketing campaign. Now, no one in the FBI is going to turn around and say, oh, coercion, marketing. Oh, that's a different thing. Off you go then. Yes, yes, yes. Good job. You know, I think Tom wraps it up so well when he says, you know, whether it was a marketing campaign or coercion, congratulations, shiny hunters on the most brilliant marketing campaign because you have single handedly transported and transformed yourselves from being a middling concern at best to the FBI to now being their seemingly top priority.

14:44James Wilson: And, well, in terms of what a marketing campaign can do, you do got to hand it to them. That is some top-notch marketing.

14:51Amberleigh Jack: Yeah, nice work, boys. Here is your stupid prize for your stupid game.

14:55James Wilson: Well, listen, I'm actually really excited to see what marketing industry awards Shine Hunters are nominated for. And does anyone show up to the awards ceremony? Maybe that's where the next arrest happens at a marketing event. Look, the world we live in is so strange. Anything's possible at this point.

15:13Amberleigh Jack: Yeah, I'd give money for a ticket to see a Shiny Hunters member in his Sunday best being carted off by the FBI. 100%, yes. I don't know. I'm with you there, James. But hey, look, we're going to wrap it up there. Thank you so much for joining me to talk through Tom's newsletter that he has written. You can find that newsletter, of course, on our website, risky.biz. and Tom, all the best and hope you get better very soon. And just a quick note as well that Risky Business Media will be going on hiatus next week, so we won't be publishing podcasts or newsletters on any of our feeds. Everything will return from the week starting Monday, October the 12th.

15:51Amberleigh Jack: So James, enjoy your week off next week and thank you so much for joining me. It's been a blast.

15:56James Wilson: Thanks, Anvoli. This has been a lot of fun and Tom, get well soon, mate.

16:03Thank you.