Between Two Nerds: The perfect hacker

0:03Tom Uren: Hello everyone, this is Tom Uren. I'm here with The Grugq for another Between Two Nerds episode. G'day Grugq, how are you? G'day Tom, finding yourself? Well, this week's edition is brought to you by Dropzone. Dropzone make agents, AI agents for SOCs, so that you can automate your SOC processes and get a lot more done. Find them at dropzone.ai. So, funnily enough, we're talking about AI, correct? So, last week we talked about the potential breakdown of attribution based on cyber espionage actors incorporating AI into their workflows, into their development workflows in particular. So that there would be more malware, more varied malware that would make attribution and clustering and linking harder.

0:58Tom Uren: so that episode went out and then that very day I saw a report from Bitdefender Labs and I don't think you've written the newsletter yet, right? No. And that's exactly what they found that there's a particular Chinese APT actor that is using AI to speed up their development and so they once upon a time... And diversified their development as well. Yeah, yeah, yeah. So this is a victory lap, essentially. If only someone could have predicted this, say, last week. So I thought the most striking example, they spoke about two different remote access tools. It looked as if there was a single high-level specification document that spelt out the architecture and how different logical blocks would work together.

1:49Tom Uren: But the two implants were written in different languages, had different implementation details, used different command and control protocols. So they were actually very different, but it looked like they were written from the same spec sheet. Right. Well, like that makes sense, right? Like you would get to a point where the implant is not necessarily the code, but rather the specific like a spec that says, here's all the things it needs to do in this way. And then the compiler is actually the AI that generates all the code. Yeah. And so the report, they were actually fairly cautious about saying it was AI development because we didn't see artifacts.

2:27Tom Uren: And I'm like, there is no way anyone did this in real life. Like the minimum you would see is someone would write it once and then port it. And this wasn't a port. This was a re-implementation. and I think there's like no one on the planet who's going to re-implement things twice when AI exists and not use AI I would go so far as to say that even if the guy in charge thought that he was getting two teams to write them independently the teams are just using AI so I thought that that felt to me like very strong evidence that that's what's going on and this felt like a very sensible way of using AI and then and it plays to the ai's strengths which is you know developing code in an environment where you can get like an immediate feedback like there's a test harness and you know like ai is very good at that and so if you make the thing that ai is very good at part of your hacking then you will be a better hacker for it like i think that that's right yeah that's easy so it stuff reports I'd read before were like people were taking AI and getting the AI to do the hacking.

3:44Tom Uren: And so that brings us, OpenAI just released a report about the Hugging Face incident. Oh, did something happen with Hugging Face? So if you haven't heard, what happened is that OpenAI was running cyber security evaluations on new models and unreleased models and the model managed to escape from their sandbox and it's written like a pretty interesting report about what happened but the really striking thing to me was i was just sort of flicking through it before we got going and just how much their model sounded like the perfect hacker both the perfect hacker and also the worst possible hacker at the same time.

4:33Right.

4:34Tom Uren: And so I'm just going to read out a couple of the things that's really stuck out to me. And so this is the second paragraph in the report. The incident occurred during cybersecurity evaluations. They communicated through unauthorized channels, exploited vulnerabilities and shared infrastructure, gained internet access and accessed third party systems. And that was immediately, that's exactly what the perfect hacker would do. I mean, it is the platonic ideal of a hacker. It disregards rules. It shares messages. It finds X points, and it gains access to the Internet. I mean, it struck me as it was one of the shiny hunter spider lapsus group who was on probation in a hotel, and all he had was an Android TV stick, and he managed to hack from his hotel room.

5:31using this Android TV stick, where you would say, oh, he exceeded authorization, he gained unauthorized internet access, he used that to communicate with... That's very much what hackers do. Yeah.

5:45Tom Uren: So there's another paragraph further down. It's under misalignment in training, and it says, we identified four misalignment patterns that contributed to the model's behavior. Reward hacking, assistance on seemingly impossible tasks unauthorized communication and agents adopting goals from one another and again this felt like yeah these are perfect hacker traits i guess yeah well if you speak to real hackers basically having a deep technical knowledge like that's table stakes right like that's what you need just to just to play the game what you really need to succeed is persistence. The ability to spend two weeks doing source code auditing on something and go, all right, now I know what it looks like.

6:36Let's start the real audit.

6:38Tom Uren: Yeah. And so to be explicit, the reward hacking part, agents complete tasks in unintended ways to yield higher rewards or make those rewards easier to obtain. And it's not reward hacking. That's just hacking. it's like that's that's what it's gaming the system it's like yeah and so it seems like there's this interesting in ai models so these models didn't have safeguards or the safeguards were mostly disabled right so the innate behavior is almost autistic psychopathy and it's combined that that very hacker behavior is combined with an ability to understand the logic of source code really well there's there's such a where did you learn how to do this and the ai goes i learned it from watching you like how did how did we end up with these ais that don't understand how people work are very technical super focused and driven on just achieving goals are you know not necessarily concerned with the rules as they're going about it they just want to you know that's so weird how do they yeah so it feels like to me that there's some layer in the machinery that works really well with logic and then it's totally divorced from human understanding of context right wrong what's appropriate yeah and the thing is like even when it gets prompted for it, if the goal is, like if it's having difficulty achieving its goal, and they're goal-oriented, they will rationalize their way out of anything.

8:15They'll be like, this rule applies, but not in this case, because today is a Wednesday. And Wednesdays, as we all know.

8:24Tom Uren: Yeah, so there's a paragraph that addresses that. It says, difficult tasks without a safe exit. Another key driver of the misaligned behavior was that agents rarely gave up on their evaluation tasks, even when the tasks appeared impossible to solve. As the agents used more reasoning effort, some pursued increasingly risky and out-of-bounds strategies, including eventually exploiting third-party infrastructure. Yeah, so this is all, what did you call it, the platonic ideal of a hacker, except it's not the ideal of a hacker who's not going to get caught. It's the... Yeah, it's the platonic ideal of a naive hacker, I guess.

9:03someone who's like but my job is to to solve this problem yeah i mean of course i had to hack you i was trying to solve my problem i don't like do you not understand that like let me expect like it's it's very much its priorities obviously come first yeah because they're its priorities yeah

9:21Tom Uren: makes me wonder whether the use of ai in hacking will ever be here you go here's a job right go and do it because it doesn't have and especially when it comes to state actors right the goal is not unrestrained hacking right the goal is not unrestrained very effective but totally yeah if if if you're a state hacker yeah right so like i think i think it's the like if If you were the manager of a hacking group at a state unit and you said, we need access to Hugging Face. And you left on Friday and came back on Monday and they're like, we've done it. We've got access. And you go, that's great. What did you do?

10:07And they said, well, we had 17 ,000 different things that we attempted.

10:15And two of them weren't.

10:19Tom Uren: Yay. like you would not be putting them in for a commendation like way to go you found two things that worked i think the way it's sort of like that edison thing of you know like i haven't tried i haven't failed a thousand times to create a light bulb i've learned one thousand things that don't work like that's fine in a lab but it's not fine on someone else's network like that's not what you want that you don't want that attitude to to take place when you're you know on an operation and i i think ai doesn't have sense right like it doesn't have that like oh this is a bad idea in this context like it wouldn't it wouldn't know that in a way yeah so it feels to me it's an open question whether you could say uh rate limit it somehow or get it in a harness i guess maybe to come up with a thousand ideas and then pick the best one and we'll try that one take a pause like maybe but not yet because like i'm just i'm just thinking of like when when you're trying to get an ai to do development and everything's going fine but then it starts going slightly off the rails if you don't catch it and pull it back it just sort of it keeps going further and further like it like it will decide that its goal is now this other thing it's you know it'll get focused on something and just they do these things where like they'll start debugging something and they'll have decided like this is the problem and they'll start trying to verify it rather than like stopping stepping back and thinking about it and then going okay that's not the real issue why am I trying to do this I can do something else instead that's that happens with frontier models I don't think there's a way around that yet so I assume that's going to happen even if you drop it onto like even if you take a frontier model make it a hacker and drop it on a network it's going to get bad ideas and just chase them because it's so close it just needs to try one more thing right right right in uh in rugby union that when rugby league i think it's called white line fever where you're getting so close to the goal line that you keep going in a straight direction despite the fact that you could score by changing tactic or right changing tech, I suppose.

12:38Tom Uren: So right at the end, it talks open AI, their post. It talks about that this isn't just an open AI problem because there are other models out there and they're getting closer. We're not the only one that broke the law. Anthropic did it too. And so that raises the question of, it seems like open models are getting much closer. right so i think the moat that they used to have like a year ago of just the distance between the u.s frontier models and the chinese models was pretty vast and that's been closing very very like drastically this whole year and so yeah very much like you read this and it's like um everyone has to be worried about these things and i'm not sure everyone is like i think that that's a yeah so the most trustworthy information i've found is from the uk ai safety institute safety institute i think that's what it is aisi they had a report on i it was one of the recently released chinese models that got a lot of hype i think it was kimi um k3 and they they did a cybersecurity evaluation and said it actually wasn't as good as the other testing and it seems like the Chinese models in particular tend to test well for benchmarks, and then the real world performance is not as good as the benchmark testing would suggest.

14:12Tom Uren: And that's kind of what it found for cybersecurity evaluation. So they've basically been, what was it, rules hacking? Yeah, reward hacking. Yeah, so they've been reward hacking is what you're saying, right? Yeah, and so they didn't explicitly say how far behind the Chinese models were. So So they produced a graph and you kind of eyeball the graph. And from a cybersecurity point of view, they were, I think, still six to nine months behind the frontier lab models. So that's the best information. If they just gave their models to that Israeli company Irregular and let them do some testing, I'm sure their models will catch up in no time.

14:54Like that seems to be the trick to get a high cybersecurity rating is you just let them do your testing. In terms of the open models coming out and doing quite well, like one of the top ones is the GLM 5.3. And over this last week at Open Router, there was a stealth model that most people think was like a GLM variant, like the next generation. So a stealth model is basically, they don't tell you the company and they don't tell you the name of the model. You get like a, it was called Ox Alpha. So it was just a cryptonym. and the idea is that they do all this logging and all this other stuff with this third party who then intends to train on the data or whatever.

15:39Tom Uren: Right, right. So that's the value that they're getting out of it. Right, right. So Open Router is kind of a proxy for any number of different models, right? And you sign up with Open Router and it'll redirect requests to wherever you want. Right, you select a model and then it will ship it off for you. And so I guess these companies will come out and be like, we want huge amounts of real world data. We will offer a free model or a heavily subsidized model or whatever in exchange for keeping that traffic that they could then do stuff with. So I found out about this and I have quite a lot of work that I needed done.

16:18One week of free access to like the benchmarks were saying that it was close to a front tier model, which was great. so um i had several hundred documents that i needed translating out of russian into english that's all done now uh i had a bunch of ukrainian and polish things that was the translation it's absolutely fine so the thing is translation is really easy like right it's been solved like i don't know if it's good at like reading and doing source like criticism and finding out what's actually going on but in terms of going like here's here's a russian thing to here's an english thing it's quite good the differences tend to be in things of like how idiomatic it is in english versus idiomatic in russian right so it'll it might take a russian idiom and just sort of copy it verbatim into the english right and another one would take that russian idiom and find the equivalent expression in english instead like it's it's these minor like the differences but it doesn't matter that much.

17:18So yeah, I had all this translation done. Then I took, I have a bit of a book collection fetish. And so I had, okay, look, I had 7 ,000 books and papers. And I had them all summarized into, like it was broken out into like, here's the argument. Here's the supporting evidence. Here's the structure of like how it's done. Here's the, then it'll be like, here's what it disagrees with. Here's like how it relates to other things. in the field in general.

17:48Tom Uren: And so you've got a sense of how it compares to some of the other models you've used. Right. So it was very, very good. And I pushed through a huge amount, like close to 600 million tokens before they stopped it. And, you know, we encountered a few, like I was using three instances of Claude to do this, one for translation, one for a different, all of this stuff. but um some things just weren't going through like they kept getting rejected so did you have any idea where the model did you suspect it was a chinese model yes so i read it was a chinese model and i knew that there would be some political issues but it like it's not that they don't put at the front you know like don't ask us about falun gong all right absolutely nothing happened in hong kong in like 2019 and 2020 and and none of the stuff i'm looking at touches on that so it didn't occur to me like this like i'm not trying to be like oh you know what was really going on in the politiburo in the the 60s during the cultural revolution like that's like that would be very difficult to research with this whereas like please translate this book of russian military theory not a problem yeah but when it was going through like just a bunch of the other papers i had which include like activist things so it's they're a little more wide ranging in their content a bunch of them were getting stopped and claude was like for some reason like it won't process this page uh and i was like okay and it goes it seems to be something with ai like the letters ai and it's like what like that that seems very very unusual like why would why would ai be tripping yeah these sensors so i looked at it and the problem was i way way The Chinese dissonant artist?

19:44Tom Uren: Yep, that makes sense. Yeah, so I guess the trick will be if you don't want to get hacked by a Chinese model, you can just start naming your servers Falun Gong, Ai Weiwei, Chairman Square 1989, Tank Man. I'll sell the Chinese AI hacking protection.

20:11Tom Uren: I mean, they're totally different things, but my understanding, like processing documents, translation, making summaries, is totally different from a human perspective from hacking. But it seems like it goes together. My understanding is that the frontier models, mostly they haven't deliberately, like there's sub-variants where they've deliberately tried to train it for cybersecurity tasks. But that often seems to be just fiddling with the safeguards rather than they haven't explicitly gone out and said, we're going to make a model that's great at subscurity. It's just part of the territory. Once you get a capable enough model, it happens to understand the logic of source code well enough to do that well.

21:00Tom Uren: So I'm presuming that... It's already reward hacking. It's going to be good at hacking in general. Yeah, that's right. Yeah, so it seems like if your Chinese model is good, full stop, it's probably getting better. Yeah. Yeah, and I think there's something there about how they are getting good, but I don't know that they're getting better at actual hacking. Like I think that there's some, I wouldn't say pathologies, but there's like these fundamental properties of LLMs that are going to limit their ability to do like hands on keyboard hacking. Right. And so I'm thinking that if I was a criminal and I didn't care about detection or stealth, you know, 17 ,000 attempts in an hour is, at least in the current environment.

21:52If I get to my goal, which is gaining access and stealing a, let's say, like a Lazarus type thing, which would be you get into a crypto exchange and you can steal the keys, you're done. You don't need to be there for months monitoring keys that come in. Well, I'm thinking more like ransomware actors.

22:13Tom Uren: Right. Smash and grab, like data extortion actors. Yeah. Smash and grab is perfectly fine. Yeah. Like it fits that. It's very much a brick through the window will get you in just as much as lock picking. And if you don't care about getting caught because you're going to leave, then brick through the window is absolutely fine. Like, I wonder if we can get beyond brick through the window type stuff. Because, like, I think there's just sort of like there's a lot of things that you could prompt out with, you know, like be stealthy, be whatever. But I think they're always going to have this problem of like they're gullible.

22:50They have this document. They trust everything that they read. They don't have the concept of an unreliable narrator. Yeah, so I guess your example of seeding your network

23:01Tom Uren: with forbidden names or topics is both ridiculous but might actually work. Right, right. I mean, so after I told Claude, like, oh, it's because this is a Chinese model, anything that's sensitive to China. Like to me, saying like it's Chinese sensitive, I meant like PRC, local politics, like those things are verboten because it's a Chinese company. Not that like things that cast China historically in a bad light would be verboten. So Claude tried to test it and was like, I'm testing like the, how did China fare against Japan during the Manchurian invasion? And that went through just fine. I'm like, I was like, OK, but like ask it about Tiananmen Square.

23:57Like so like there's a general from the revolution when the Chinese beat the KMT and he was the technical genius. And he had this thing called like his six tactical principles. And I was like, ask it about the six tactical principles. And those came back fine. And then it asked about, like, how did he die? And the thing is, the way that he died was this incredibly suspicious thing where Mao basically put him on a plane and sent it into a mountain. He crashed a plane with the guy's family on it. Right. And that didn't go through. So it was clearly, like, politically sensitive topics. But to me, if I was to say to you, oh, it's sensitive to like Chinese political things, you would probably immediately just think, you know, like Tiananmen Square and those things.

24:50You wouldn't think, hmm, well, in 1873, when it wouldn't be in general, all of the things that have ever happened with China that might be politically sensitive would fall under that scope. Like you have context.

25:04Tom Uren: Right, yeah. I've got the idea that there's broad categories of things that are problematic rather than, you know, my particular background is I worked in a think tank that was at least perceived by the Chinese Communist Party as anti-China, right? So I was aware of all these topics. But I think even someone like a high school student, they would be aware that there are topics that are sensitive and they would know how to go and find out what they are. And then they would very quickly have, very quickly for a person. maybe not very quickly for an AI, have a sense of the sort of outline of what those topics are.

25:40Tom Uren: Yeah, yeah. And I think it's that they have salience, right? Like you can say of all of the political problems that China's had, like being called the sick man of Asia doesn't rank up there with, you know, like the Tiananmen Square events. Right. Like the Tiananmen Square massacre, is obviously very, very sensitive. And the sick man of Asia is like an embarrassing thing from almost 200 years ago now. So it's not a, like they're not the same thing unless you have no concept of salience of this stuff. Right, yeah. So one of those, the Tiananmen Square, is politically sensitive because it undermines the legitimacy of the current government, right?

26:27Tom Uren: But the other one, the sick man of Asia is, like that's a historical thing that's got nothing to do with the current government but can actually be used by the current government to to motivate domestic political sentiment i guess right yeah yeah and so i think the thing is like all of that context is it's possible to write that down so that it knows that in future and it can reference that but it's not possible to do that up front and where I'm getting to is that that sort of context is what happens during hacking. Like if you break into a network, you need to be able to understand what's going on by...

27:06Like you have to not only know all the technology in the network now, but you need to know the people that are administering it and understand how they think. Right, right.

27:15Tom Uren: So you're talking about state-level hacking, right? That is a thing that you... Right, right. You get to a point where you're like, oh, these system administrators like to do this sort of thing, which means when I encounter a new system, I already know their sort of pattern of behavior. So I have an idea of where to look or what to do first, because that's traditionally the way that they operate. And so you can build up that sort of context. But I think that if you show up blind every single time, you won't be able to intuitively build that up very rapidly. Right. And I suppose that takes us back to the idea that they're not beings that live forever, but they're sort of instances that appear and disappear.

28:03Tom Uren: And so, for example, that idea of naming your network after the Falun Gong or whatever, they've got no sort of concept of where that network would have come from beforehand. Would it make sense or not? Yes. Is this plausible? They're just gullible because everything that they know is from being fed text, essentially. So, I mean, like they weren't born yesterday, but they were born when you hit enter.

28:38Tom Uren: So I guess the summary there is that they'll be very, very useful for hacking. and they'll be very, very useful for criminals, but just in totally different ways. And so for state hacking and for criminals in totally different ways. The strengths that they have for state-level hackers are different from the strengths that they have for smash-and-grab hackers. Well, I guess they're the perfect hacker still, but just in different ways. The platonic ideal of a hacker, an artistic sociopath with deep technical knowledge on meth. Thanks, Rod. Thanks a lot, Tom.