Srsly Risky Biz: OpenAI's Kazoo-Grade Spying Incident

0:04Amberleigh Jack: Hey everyone and welcome along to Seriously Rescue Biz. This is our podcast here all about cybersecurity policy and intelligence. My name is Amberleigh Jack and very shortly I'll bring in Tom Uren, our policy and intelligence editor and we will have a chat about Tom's Seriously Rescue Biz newsletter that he has written and published this week. You can of course read and subscribe to that over at our website risky.biz but before we get into today's show I'd like to I think our sponsor is Sondera. And you can also head along to our risky bulletin feed and check out an interview that our colleague James Wilson did with Sondera co-founder Josh Devon this week, all about how AI agents are sort of highly capable but maybe not the best rule followers or decision makers.

0:49Amberleigh Jack: Speaking of AI, Tom, we've got a bit of that in today's show. It is great to see you, my friend.

0:55Tom Uren: Hello.

0:55Amberleigh Jack: Tom, I want to jump straight into this first story because reading it is a crazy little windy road to be driving down. The Saudi Arabian OpenAI employee was arrested, charged with bugging a conference room in one of OpenAI's San Francisco offices. Sounds, okay, newsy, but all right. But the story, Tom, is the story about how this guy allegedly went about his spying ways is crazy. So tell me, what's the Yakuza doing?

1:32Tom Uren: Yeah, so very occasionally you come across stories of very serious and competent spying that somehow has been caught after 20 years. This is not that story. So the person who's been arrested, Abdulrahman Al-Fazan. So as far as I can tell from the reporting and from the court documents what triggered it was that people at open ai noticed that something had been plugged into the back of a tv in a conference room and that something was wrapped in toilet paper and was a usb device that you can just buy and so they did some work on that usb device and it looked like it'd been set up to be encrypted so they couldn't get into it.

2:24Tom Uren: You can buy these things for like less than 15 bucks. And so OpenAI set up basically a trap. They started to look at their logs of who had been in the conference room. They set up a camera behind the TV and eventually the story is that they caught him on that camera mucking around with the USB device. and so they're onto AlphaZan, they look at who's been into the conference room, he didn't have permissions to go in the conference room, yet he'd been hanging around and it gets worse. So he used OpenAI's own chat GPT where they would keep logs and he was asking about what is the best conference room for sensitive meetings, I guess.

3:21Tom Uren: So he did his research on the company's own equipment.

3:25Amberleigh Jack: And logged in as well in his wife's own account.

3:29Tom Uren: Yep. Smart. And he used his wife's ChatGPT account to research. This is one step away. It's not him directly. It's his wife doing it. But how do you set up a USB device to be an audio recorder? How do you encrypt it?

3:49Tom Uren: and then to add insult to injury he has a number of these devices and he leaves one in his bag with his wallet and id unattended in a printer room and so you know unattended bag in a room someone comes and searches for it they confiscate the device open ai is onto him at this point so they confiscate the device um but it's just you know comedy hour it kind of feels like espionage

4:14Amberleigh Jack: 101 is you do not leave your ID in your spy backpack.

4:20Tom Uren: It's espionage minus 101. So some of the US court documents have been, we think that there's a genuine flight risk because he's supported by the Saudi Arabian government. My reading of all this, all these allegations is that there is no way he is working at the behest of the Saudi government because there's no way anyone would be that incompetent if you're risking that kind of thing. So the court documents play up the fact that the Saudi government paid for his wife and child to fly back to Saudi Arabia and that a member of the Saudi military paid for a business class flight for his wife and child.

5:10Tom Uren: I think probably what's happening is that I think it appears like he's a competent engineer. He's done well. He's working at OpenAI. He was talking to members of the Saudi royal family. But I suspect that he was just well connected and they're like, we're going to look after this guy. Probably the Saudi military person. I wouldn't be surprised if they were his relative, brother, something like that. Who knows? Yeah, right.

5:38Amberleigh Jack: I'm not privy to Frontier Labs kind of vetting processes and employment processes, but I would kind of assume that if you are smart enough to have a decent role at a company like this. These are the kind of mistakes I just don't imagine you'd be making.

5:59Tom Uren: I think there is a big difference between what people think spying is and what good spying actually is. And so I think that if you're in your own world, you can be a very technical person that doesn't really understand OPSEC or operational security all that well and how to do things properly. Like you see this in the cybercrime community all the time. People will commit crimes, very serious crimes, and the way that they're discovered is comical.

6:31Amberleigh Jack: I think we've spoken about a couple of them.

6:34Tom Uren: The story is always, almost always. And so that's why when you see something of like real spy tradecraft, that is, it's exceptional because they basically don't get caught. Anyway, so there's this story of how he, there's some sort of relationship with the Saudi government. It seems to me probably if these allegations are proven that he was just, I'll get some extra information. I want to move back to Saudi Arabia. he was about to take leave parental leave for a couple of months and travel back to Saudi Arabia and there's a lot going on in AI everywhere and so there's an overlap between Saudi Arabian interests the royal family member was saying we need you back in Saudi Arabia you can do so much for AI here I think there's an overlap between what he wanted to do which was kind of bolster his own intellectual capital Yeah, right.

7:34Tom Uren: And so that conference room, it was meant to have high-level, sensitive technical discussions, commercial discussions, strategic decisions. So, you know, interesting stuff.

7:45Amberleigh Jack: On a bit of a more serious note, Tom, I mean, we're kind of, you know, laughing about this, and as you said earlier, there's good spying and there's bad spying, and this wasn't really good spying. But, I mean, you'd be pretty naive to think that there isn't good spying going on at these frontier labs, right?

8:05Tom Uren: Yes, you would imagine they must be a high priority target for state intelligence services. So those are the espionage capers that are getting away undetected.

8:16Amberleigh Jack: Because they don't wrap things in toilet paper and stuff on TV.

8:19Tom Uren: That's right. That may work in a bathroom, not in a conference room so much. You've got to have a plausible cover story is one of the first rules. But there is some good news for OpenAI here in that they did find the bug wrapped in toilet paper. So Wired says it was during a counterintelligence sweep or a surveillance sweep. I could not find any information elsewhere that corroborated that. If that is true, that's great news. They're actually doing something. my problem with the whole situation is that i think the incentives for these companies are not to invest heavily in security and so there's this tension between moving very rapidly which is what the frontier labs are all doing and having strict security which maybe not by definition but almost always slows you down so i kind of think of it if you're an intelligence agency, they take really very rigorous steps against insider threat and espionage, you know, very strong personal vetting, personnel vetting, monitoring, behavioral monitoring, really strict access control of who and what you can get to and when.

9:35Tom Uren: If it was a really sensitive location, it wouldn't be possible for someone like AlphaZan to just walk in. There'd be an access control on the door. And so those are the kind of steps that an intelligence agency would take. Now, you might say, well, OpenAI and Anthropic, they're not intelligence agencies. And I would say, yes, that's true. But I think that they're targeted probably just as highly nowadays as those agencies are. I think there's a real tension there. And we can see it in the fact that there've been all these hacks that are extremely embarrassing for OpenAI and Anthropic. And I think they're downstream of having a weak security culture.

10:13Tom Uren: So the security culture does not override the need to move fast. And so I think they're at risk.

10:19Amberleigh Jack: Do you think there's a line where the need for speed and being secure can actually balance out or do they just not work together?

10:29Tom Uren: I think you can choose one or the other. For all organizations, I think there is a point where you arrive at a sensible balance. And I think it is difficult to find where that point is. If you're in a really secure place, you're always frustrated at how difficult everything is to do. And if you're at a really fast moving place, things are getting stolen all the time. So I think the lesson for OpenAI and for Anthropic is that what are our crown jewels? We really need to focus on that and do a good job at least there. and in this case for example he did have the bug in that room for a number of days there's probably some intelligence value there is it their crown jewels no i don't think so so that's kind of the good news but i think it just is worth making sure that the crown jewels are actually protected like crown jewels like uh you know you can ignore security in some places and that's probably acceptable um not great if you're a security focused person but acceptable i think given the broader context when it's crown jewels no not acceptable yeah fair enough and i guess the lesson

11:45Amberleigh Jack: for spies is um if you want to spy on your company don't ask your company's ai agent how to spy on your company all right tom hey i want to i want to move on to your to your next piece here which is a you've written about a Norfolk report this week about this autonomous pen testing agent Ken that malicious actors seem to be using to help them sort of automate the process of making money. Tell me what you found here.

12:20Tom Uren: Yeah, so the reason I thought this was interesting is that there've been a number of reports that have been, oh, OMG, hacker uses AI to hack things. and one, for example, a bloke in, I think it was Addis Ababa, hacked a number of organizations, was quite successful hacking, but then he was asking the AI, how do I turn this into money? What do I do next? And they'd basically done the hacking part but couldn't do the making money part. And so I thought this was an interesting counter example where the threat actors had gone all the way from selecting targets that made sense to were in a position to make money in a straight line with AI assistance.

13:10Tom Uren: So they, I guess the key part was that they were just looking for credit cards or cryptocurrency or payment details, something like that. And so the Norforge discovered a number of, it looks like a number of different threat actors who are running separate CAN instances. Now, can is just a, I guess I'd call it a harness, where you go, I'm starting here, and I want to end up in this place over there, and go figure it out. Nice. And it works particularly well for pen testing because you've got a concrete starting point, which is like, well, you're on this IP address or you're on this system, and you can issue commands.

13:53Tom Uren: And you've got a concrete destination, which is I want to get on this system and issue specific commands. And so you can tell whether you've succeeded or failed. There's a very strong or measurable win condition. And it'll just go out and try a whole lot of things. So it also works well for mathematical proofs and vulnerability research, which have those same properties of, yes, it is either proven or not proven. The vulnerability, you can exploit it or you cannot exploit it. And so when you've got that sort of structure, that problem structure, apparently it'll just keep on working at that. Now, whether it's any good, Northforge found that in about 10 % of cases, it seems like it was able to hack a target.

14:39Amberleigh Jack: And pretty quickly as well, right?

14:42Tom Uren: So, yeah, it found that it did vary, but mostly it was within 24 hours. So some of the, I guess they called them projects, lasted for many days and it kind of spiders out trying different things. So it sounds like a very noisy exercise from a defender's point of view. They're trying all sorts of different things. But because the threat actors basically selected places where you would be likely to find credit card details or cryptocurrency, that lends itself to go figure out how to hack this, go figure out how to get those details that I need. And that's easy to convert into money. So these threat actors had the, I guess, the business model or the theory of victory or their thought process straight from the beginning.

15:35Tom Uren: It's not, I want to hack something and then I'll figure out what to do. It's, I want to make money and this is how I use AI to hack to make money. So that's, I thought, an interesting development in the, is it the sophistication or the, it's just the cyber criminal use of AI evolving and maturing, I would say.

15:54Amberleigh Jack: You know, things are evolving and these criminals are sort of, you know, figuring out how to use AI to do more quickly, I guess. What, Tom, do you reckon is a key kind of takeaway or option or whatever for defenders here?

16:12Tom Uren: I think in these cases, North Forge found different threat actors seem to be using different, I guess, strategies. One was more focused on cryptocurrency. One was more focused on small retailers. I think in all of these cases, the way CAN works is noisy because it tries things. And it knows if those things fail, and then it can move on to a different spot. So I expect that from a defender's point of view, it is like these attempts are noise, but we need to be monitoring them and trying to figure out what to do next. There's one particular example that is in a Reuters report. So Norforge spoke to Reuters.

16:59Tom Uren: And it says that in one case, after failing to find any weakness in a big company's public-facing application, the AI agents poured through digital records to find associated infrastructure, located a public-facing control panel, discovered a subtle weakness in the way it processed data, and after hours of trial and error, reported leveraging it to trick the server into executing malicious commands. So I think that's a, you can sort of see how something like Cairn would spider out, try all these different avenues, reach roadblocks, try different approaches, eventually figure out a pathway. It's kind of like rain dripping through cracks in rocks, right?

17:45Tom Uren: You get there eventually. But in that case, I think that would be a noisy example. That control panel should not have been, probably should not have been public facing, I suspect. And it says after hours of trial and error. So you have a public facing control panel that is being hammered by, I think you would be able to see that they're malicious requests. And somehow that did not get escalated for anyone to do something about. So I think that's the kind of message there.

18:15Amberleigh Jack: So many lessons to be learned in today's newsletter. but hey look we we are gonna leave it there but tom it is always a pleasure to chat to you and um really enjoyed this week's news today you can of course read that um over at our website rescue.biz but tom have a wonderful week and i'll see you again same time next week thanks angeli