Between Two Nerds: The AI crime machine

0:03Tom Uren: Hello, everyone. This is Tom Uren. I'm here with a Grugq for another Between Two Nodes. G'day, Grugq. How are you? G'day, Tom. I'm fine. And yourself? Well, this week's episode is brought to you by PortSwigger, makers of the Burp suite of software analysis products or web application analysis products. Find them at portswigger.net. so we've spoken a fair bit about ai in recent times and in the last couple of weeks there's a mounting array of evidence that threat actors are kind of getting it together so and figuring out how to use ai in a way that to me actually makes sense like to achieve their goals so a while back several months ago i think we talked about a couple of threat actors who used ai to help their hacking and they were very successful at actually hacking organizations but they weren't able to convert that into actual money so their purpose was cyber crime but they didn't actually succeed at the crime very well well well i mean they hacked things so they did hacking crimes so one example was a Mexican.

1:20Tom Uren: He targeted the Mexican government, hacked a number of databases, but was from the report, was kind of searching around for how do I turn this into money? How do I monetize it? So we've got a couple of data points that indicate that things really are changing with AI. the first one is the verizon data beach data breach investigations report for this year so that came out several months ago in may and i didn't notice at the time but the number of initial access by exploitation that had gone up a significant amount so 30 percent of the initial access was by some sort of exploitation right that's a that's a huge change because you know Even for the last couple of years, it's been like 20%.

2:14Before that, it was even lower.

2:16Tom Uren: Yeah. As recently as 2022, it was like, I think maybe 8 % or something like that. So it's gone up a huge amount. There's not enough information in the report to say this is all threat actors using LLM. So it could have been literally anything that made a huge amount of exploits show up for threat actors. We're not saying it's LLMs. We just want to raise the possibility, right? Draw your own conclusions. We've spoken a few times about the just, I guess it was a month or two ago, we spoke about the number of CVEs being issued, I think. And the charts that were going up and to the right in a kind of exponential way.

3:00Tom Uren: And at the time, we said there wasn't a lot of evidence that it had been converted into anything. It resulted in a whole lot more hacks. And it was, I guess we just spoke about it like we do. But there's another report that you drew to my attention from Gambit. They also did the report on the Mexican hacker who hacked, I think a single individual hacked nine government services or something like that. Right. As I recall, it was a bunch of like provinces or states, I guess they have. Maybe the tax office or something. Yeah. Yeah. It was like really, I don't know what he was thinking. well yeah what i thought was interesting about that is that they had constructed a number of different personas where you know one was a you are an elite level network analyst right yeah you um and then they had the like you're a you're a network specialist who can like find where things are and you're like a hacker who can like do exploits and you're a whatever so that they had outsourced the i guess lateral movement and trying to find valuable stuff to llms that was he was on the right track but he was doing he was doing it sort of manually by hand right so it was it was interesting to have these approaches but then he was uh going to each individual llm instance and saying you are this thing right and feeding it a prompt and that's obviously not going to scale very well because you've now capped it sort of quite low.

4:35Whereas this other incident is a lot more interesting in terms of like how they've chained different frameworks together to be able to automate the entire thing, like with orchestration layers and scanning layers and then like...

4:51Tom Uren: Yeah, yeah. So the title, Autonomous AI Agents are breaking into hundreds of online retailers for$25 a target in an ongoing campaign. So it seems like there's a single individual, they've used several different harnesses or frameworks to run different models within. And it seems that they had a plan for what they were going to go after. And it's actually a pipeline where it looks like, at least from what we can see, it's paid off so it seems like gambit has an ability to get onto the computers that these people have been using and then do the forensics and recreate what they've done and how we'll leave that as an exercise for the reader yeah everyone was a suspicious person i wonder how exactly they're doing this so frequently but yes anyway so what's left out to you about this so like there's some interesting things here like basically uh this guy was using uh strix which is a framework that does like uh it's a pen testing framework for web app vulnerability scanning and hunting sort of things so he used that as this like initial layer and then for that he was using uh he started out using glm 5.2 but eventually it refused so he switched over to using deep seek and these are sort of they're both chinese models aren't they yeah they're both chinese models right then um Once he had this vulnerability report, he was handing it off to this other framework called Cairn.

6:29And Cairn is, I've never heard of it before, but it's a Chinese framework. They talk about it sort of like exploring the problem space as like a generic approach. And they happen to have made it do pen testing. Because pen testing is one of those, as we've said before, there's a true positive when you get something right. Right. Which is very useful, right? Right. it's essentially the way that this thing works like you give it a bunch of knowledge you give it a goal say where it is right now and it sort of tries it launches workers to try all the stuff and build up the knowledge and like eventually it will find the solution by having right right so having exhausted everything else i guess so the knowledge might be here are a set of

7:10Tom Uren: standard pen testing techniques or whatever well those those would be the skills that it has the knowledge will be like here's the set of ips that we're on and we're the goal is we want to have a shell. So it'll be like, okay, well, this looks like you've got an arbitrary file upload vulnerability. What I'm going to try is writing a thing and then uploading it, and then if I can access it. And then if I do that and I get a shell, then it works. And if it doesn't work, then I'll write it into the knowledge base. It's like, I tried this thing and it failed. And someone will try something else. And so there's a little orchestration agent behind that, which sort of doles out work to each of these workers who look at the thing and then they say, I'm going to do this.

7:52Like, this is my proposal. I'm going to pursue that. And so it's like if I were doing espionage hacking, I would not want this.

8:00Tom Uren: No, it sounds extremely noisy. That's exactly what I was thinking. It's like, this is a recipe to get caught on day one. This is not how you deal with a network where there's any sort of monitoring at all. But I think if you're going after like a retail website somewhere in the US, like, sure, why not? Yeah, so I thought the, like maybe stepping back or forward a bit, I thought the selection, target selection was quite interesting. Yeah, so part of the reason that this feels like someone who knew what they were doing is that the initial like filtration of targets, like the way that he figured out who to go after was an interesting approach.

8:37So he started out by, he took one of those, you know, like the top 1 ,000 traffic sites or whatever. So he took a top list and then he said, give me all the ones that do retail. And then from that list of the ones that did retail, he said, okay, everyone that uses like an open source, like a popular open source framework, exclude them. Everyone that uses a common commercial framework, exclude them. And so what he had left was a priority list of people using custom stacks and custom code who were sort of very high volume.

9:11Tom Uren: Which the attacker assumed were more likely to be vulnerable, which I think also more likely to be, that seems fair enough, but also more likely to be less tightly monitored as well. Right. If you say, like, let's just pick, like, the top retailer, you go, okay, I'm going after Amazon. like you're not wrong but you're probably not going to get very far particularly if your approach is to explore the problem space by trying absolutely everything right so yeah so he narrows it down and that seems to be like not a manual process in doing the filtering but a manual process in terms of the thinking right like you you'd have to sit down and go like all right what am i looking for i'm looking for things that have credit cards i'm looking for things that are easy to break into, like the intersection of that is going to be this thing, these are the criteria, right?

10:01And you come up with that and you just say like, whatever, chat GPT, go and apply this criteria to this list. And it gives you the results back. That's fine.

10:10Tom Uren: Yeah. So I interrupted you. You were talking about Cairn, I think. So Cairn is a, as a hard asset has done quite well in a 10 cent CTF for AI stuff. Like it came and third but it was the only one that solved every single flag right so like i'd never heard of it and it seems like it's it's not it's not presented as a pen testing thing it's presented as this like generic problem solving thing right which makes me suspicious that like i think this guy might have been chinese that and the fact that he wrote in chinese you know i don't want to judge too soon but uh it doesn't say like yeah those things are both true It doesn't say he excluded Chinese sites, but actually looking at the, it's got a table of where the credit cards were exfiltrated from.

11:02Tom Uren: Hong Kong appears in it, but mainland China does not. So Karen handled the actual hacking bit, like the, okay, I found all the vulnerabilities. Get me a shell. Okay, get me onto the database. Like it did that part. but then to orchestrate all of this he then used hermes which is a uh sort of like it's an orchestration management sort of agent thing like it's very long running and you can give it a task and it will do it for days and days and days like these other frameworks should be like okay do this task and it will do that and then once it's got a shell it'll just stop because it's finished right but you can't like you can't scale if it stops every time it finishes one goal right right so instead yeah you have to like have this other one and interestingly enough the hermes framework was using opus 4.6 and that's because he had to like he tried more recent ones and they kept refusing so he kept going back and back and back until he got to 4.6 and then that ran the whole thing right and uh he told it that it was called soul that's the the persona was soul the red team operator right so so that's the premise of of why opus was happy with it because you're doing you're doing good work good job soul i i think maybe the other reason that opus was happy was this guy spent something like uh well this guy spent 12 to 18 000 in uh tokens right now i i don't think that the credit card scammer was paying out of pocket for this?

12:44Tom Uren: Yeah, so the other example Gambit talked about, well, maybe it was a different one, but they had a, I guess, a library of stolen, I think, ChatGPT tokens in that case. And so it seems like there is, like every info stealer is now stealing AI tokens as well. Yeah. Like the token, access tokens. Yeah, the API. Yeah, I got to say, some of his things are kind of funny. Like, he complains about the code quality in the exact same way that anyone who's done vibe coding... Like, a script tag outside the HTML tags is not right.

13:29Yeah, like, so you've malformed the web shell. Way to go, genius, you know?

13:36Tom Uren: yeah so the report has like actual prompts that this person gave and what is kind of surprising is just how short they are sometimes like uh like read the vulnerability report test the pseudo password first this one read the vulnerability report and start

14:00Tom Uren: i think there was a couple of sites where the operator already had the admin password and the prompt was just get going i'm not paying you to sit around and one of the things i want to highlight is okay so like this this karen thing is it's interesting in doing like exploring the problem space. But if you look at sort of how it went about doing it, you can see why you needed an LLM for this. So in one of these attack instances, it starts out with, it gains unauthenticated SQLI. And then from there, it does a plain text read from the OTP table. So it can dump the SQLI to get the one-time password that gives it MFA access that allows it to get into the admin panel.

14:50From the admin panel, he has arbitrary file upload. And so they upload a file that allows them to do host RCs. They've got a shell. Then they use sudo with no password for Python so that they can execute as root. So now they've got root. From there, they NFS mount. And then they use that to access the WordPress blog database credentials from the WordPress config.php. which they then write so they modify and add a new user. Then they use that to upload a WordPress plugin which gives them an RCE on that blog host from which they then access the AWS Secrets Manager which they dump to then access the Aurora Magneto database which they then dump the encryption key details which they then use to decrypt the credit card numbers.

15:48Right.

15:49Tom Uren: Yeah, so when you read that out to me, I was thinking if you're a real person, you would just get sick of it, right? And the other thing that occurred to me is that there's probably a whole spidering out of... Right, these are the ones that worked. Yeah, that's right. This is the sort of backward logic of how you get there. Right. Also, Rundle, if there was perhaps a quick away... There was no one here that's clean the first time. You know, probably as soon as they did, like, SQLI, there was like, and here's the key to the CT. Yeah. I mean, this might have been, it might've been that the LLM got fixated.

16:23Like once it came up with a path, it would just...

16:25Tom Uren: I mean, it's hard to know without knowing what the network looks like, right? Right. All sorts of things. But like some of the stuff that happens is like once LLMs have come up with what they think the solution is, they'll keep trying. And then rather than attempting a new type of solution, they'll sort of back up and try it like a slightly higher, like basically they expand scope of what they're going to try it on until they can find something which is fine if you're doing a capture the flag but it's like it seems very noisy it's it's not what i would choose yeah so the hacker like there are some funny incidents that happen where he says things like dump the two tables in those two databases and then empty them once the is done and it wiped the databases yeah so that's again an example of if you were doing real cyber espionage that's a undesirable outcome even i think for stealing credit cards it probably doesn't help because it i think you don't want to draw attention yeah that's right it forces some sort of incident response that may mean that the credit card companies are tipped off right I mean, after so many years, one assumes that the credit card companies are very, very up to date on how they do these things.

17:39One of the other data loss instances was when it was attacking some databases. The agent used ZQ as a prefix to create these staging tables when it was like aggregating all the stuff together. And the guy said, OK, delete the ZQ tables. And so it deleted everything that had ZQ because it turned out like basically my suspicion is that it was. looking at like what is the what is the standard convention on this database okay we do zq underscore you know admin or zq right so it does that and then he goes all right dump the zq tables and so it wiped everything that began with zq including a bunch of backup tables that would be made so again like that's like those the sort of things that happen with uh llms where like right it will It will have made a decision that makes sense to it at the time as like the correct way to do things.

18:34And because you're unaware of what that decision was and what the knock-on effects are when you adjust things on that. That is one of the things I think everyone learns very, very quickly because it's like a hard edge.

18:48Tom Uren: Yeah, so it seemed like this is, compared to the previous one we talked about, a step up in sophistication and actual success. So the report doesn't actually say the perp made off with X number of dollars, but presumably the selling of credit card data is occurring somewhere else, not on this particular box that Gambit had access to. I think that the key difference here is like the last one was very much someone. So like the Mexico incident feels like someone who sort of wants to get into this hacking thing. Like they're very excited at the idea of hacking. And so they do hacking. And like that's cool in and of itself.

19:33This seems more like a guy who's very excited at making money. And who's like, how can I use this new technology to get rich? or to do my crime business better. And so it's very purposeful, right? Each of these things that he's doing is with intentionality. He knows if I do this, it's because I'm solving this problem that I have. And if I do that, it's because I'm solving that problem that I have. It's not just like this would allow me to hack more things for hacking more things.

20:05Tom Uren: The target selection indicates that right at the beginning, he's got a plan. whereas the other one was i'll get this data and then i'll figure out what to do with it step one hack first profit later whereas this one is how am i going to profit that's where i'll hack right exactly and and this feels like more of what we're going to be seeing in the future rather than the other one like the other one seems well okay we're probably going to see a fair amount of that, of like people who are curious and interested in the idea of like hacking. And so they'll be able to do it now when they, before they would have had to do a lot of skilling, like a lot of training and learning and that, and it'd probably get caught at a much lower capability level.

20:52Whereas now they can sort of upskill very, very rapidly. And when they get caught, it's going to be for something bigger because they'll have had that, you know, upskill. whereas this seems like how a criminal would do llm hacking right they're going to say what is what is my business process i do this i get that i do that okay how do i automate these steps what is the thing i can do that will allow me to make more money and they're going to try and do that and i think it's going to look like what we see here where it's multiple frameworks it's a plan And from the beginning, it's all quite well done for what it is.

21:29Like it's not like an MVP, right? Like it doesn't feel like...

21:33Tom Uren: Right. Like it feels a little bit sort of hacked together just to make it work, but it doesn't feel like throwing things together to see what works. Right. Like it's a purposeful hacked together machine to do something. In the newsletter this week, I wrote about a Russian APT who was doing, in a way, something very similar where they... I spoke about it with Patrick, so I won't go into a huge amount of detail, but they were basically getting AI workflows to do all the work. This was produced by Anthropic, so it didn't talk about what harnesses may or may not have been used, because I presume they didn't have visibility into that.

22:12Tom Uren: But it talked about using different clawed skills, and the people would just modify the clawed skills. They wouldn't actually try and dive into the... So it felt very similar. And it seemed like that is acceptable in that case because the risk appetite is kind of the same. Like, you know, we've got a target set, which in this case was Ukrainians, you know, particular types. But there's plenty of them. It's a subset of Ukrainians, but it's... Yeah, and like they were particularly interested in the drone supply chain. So again, for like this person, it's, you know, if one of those shopping sites doesn't pay off, there's hundreds more, whatever.

22:56Right.

22:57Tom Uren: And if I'm noisy and I get caught, whatever. If I'm noisy and I'm successful, happy days. That's good enough. Yeah. And so it's interesting that there's this overlap. They get caught on every intrusion, but they make it up in volume. Yeah, and I guess you can get caught, extract some credit cards, and you can still get paid, perhaps. Right. So it's not like it's success requires you never to get caught. Yeah. Well, he doesn't need presence. Although with the skimmers, he does, which is an interesting thing. But even the Russians, one of the things that happened is because the dwell time has dropped, they changed their TTPs so that when they first gain access to the thing, rather than being like, well, we're done.

23:48we'll come back tomorrow and we'll like start our collections right they'll do huge sweeping takes of everything right so that if they get kicked out they'll have at least collected right something yeah more smash and grab so like it strikes me as like the same sort of the same sort of logic right which is the if you're just trying to demonstrate that you've done something at all it doesn't really matter like in the case of the the intelligence services where you're sort of showing that you've done work as opposed to having done work that's good.

24:21Tom Uren: Right, right, right, right. So I was kind of thinking that it comes down to what your intelligence requirements process looks like. And so if it is like, number one, we want Zelensky or Minister of Defence or Chief of the General Staff or whatever, and then there's a high weighting on that, the way you operate looks a lot different. You wouldn't be doing this. No, absolutely not. And if it's like your career depends on getting access to some number of those very high-priority targets, this is not the kind of program. Right. I think we've talked about this before, the RK thing of like, do you invest heavily in one or two things or do you just invest heavily in doing as many as possible without caring about each individual one?

25:13Tom Uren: yes so this is definitely that that latter strategy i mean that doesn't preclude that they're not also doing that other one exactly exactly right like not everything has to be a high priority right in this like not not everyone is zelensky right like yeah you can you can put your best teams on like the most important targets and then you take all the other teams and say give me as much coverage as you can right but it feels to me that criminals will be very much in this world the second cap yeah of we're very happy to trade vast scale for llm mistakes llm weirdness whatever like they're absolutely a volume business so um several years ago i wrote about fin7 and how their business was basically they had solved the problem of converting an intrusion into money, right?

26:08Like they had this specific workflow that says, here's your initial access, and then you do this, this, this, and this, and at the end, you have Bitcoins. And like, that's a solved thing, and they've got their whole thing set up for that. Then in order to scale, they need to run as many of those as possible, right? Like they need to scale out laterally or horizontally or whatever. And so that's where the affiliates come in is they allow them to do that sort of large width approach of having as many people running this business process as possible. And so for the people managing the Fin7 infrastructure, their actual business then becomes not so much the hacking as the managing these different workflows at these different stages.

26:52And I thought that was very interesting and innovative at the time. But it seems like that's sort of where everyone is going to be now. because rather than having all these affiliates doing things, you have all these LLMs doing things. And so now the skills shifts from being able to do the hands-on keyboard stuff to just being able to manage as many of these in parallel as you can to do portfolio management to make sure that you...

27:21Tom Uren: I mean, it just seems like such a sad future when all hackers become managers. we used to do hacking and now we can thanks a lot thanks a lot Tom