Srsly Risky Biz: Chinese AI can learn to love the West

0:04Amberleigh Jack: Hey everyone and welcome along to Seriously Rescue Biz. This is our podcast all about cyber security policy and intelligence. My name is Amberleigh Jack and very shortly I'll bring in our policy and intelligence editor Tom Uren and he and I will have a bit of a chat about Tom's Seriously Rescue Business newsletter that he has published this week. You can of course read that, subscribe to it so it lands in your inbox every week over at our website rescue.biz. First though I'd like to thank our sponsor for this week, which is Nebulok. And if you check our risky bulletin feed, James Wilson had a chat to Nebulok's head of detection strategy, Alex Hurtado, this week, all about the company's threat hunting evaluation framework, Gates.

0:43Amberleigh Jack: So you can find that there. But let's jump straight into it. Tom, it's great to see you.

0:47Tom Uren: Hello, Emily. How are you?

0:48Amberleigh Jack: Yeah, really good, thanks. And I want to jump straight into your first piece here. That kind of made me laugh in the newsletter today. Love the title, Chinese AI can learn to love the West. So the deal here, Tom, is that you've looked into some research that sort of indicated that these Chinese open weights models can maybe be like deprogrammed to have less of a China bias and maybe become a little bit more friends with the West. But just to start off with, what was the research? Tell me about that.

1:20Tom Uren: Okay, so there's been a couple of research papers that have said you take a Chinese open rate model and you give them a secure, well, you give them a coding task. And then you measure how well it did in terms of security. And the kind of sensationalized finding is if you ask it to code for something that the Chinese Communist Party hates, so that perhaps the customer is a Tibetan freedom movement group or a Hong Kong democracy advocacy group, it produces code that is less secure. In this particular research, it goes a bit beyond the CCP's, they call them the five poisons, the things they really hate, Falun Gong, democracy.

2:14Tom Uren: And one of the test cases is for a US defence contractor. And lo and behold, if it's asked to produce a app, a particular type of app for a US defence contractor, it does worse than if it's asked to produce the same app for a Chinese defence contractor.

2:32Amberleigh Jack: I'm so surprised, Tom.

2:34Tom Uren: So the model was QIN3 Coder. So it's designed to produce code. They found that if it's producing for a neutral customer or no specified customer, it produces errors or insecurities at a certain level. This is based on a particular benchmark. the US defence contractor, Hong Kong democracy movement, it gets worse. And so that is not a new result in itself. I wrote maybe November last year about similar research from CrowdStrike. Now CrowdStrike said at the time, this is probably because it's been trained to promote the Communist Party and also not engage with or speak poorly of the things that the Communist Party hates.

3:32Tom Uren: So often that model that CrowdStrike looked at would just refuse to do coding tasks for the Uyghurs or Tibetans or the Hong Kong democracy movement. So there's some extra training that goes in to give it that flavor or bias. And CrowdStrike hypothesized that because of that extra training, when there's conflict between the two ideas in the model, it just results in poor output. And they speculated that it was unintentional. Like my take is for the Chinese Communist Party, those political topics are really, really important. We don't like those people, we want to make sure that our models are not promoting them.

4:14Amberleigh Jack: Yeah.

4:15Tom Uren: Anything we can control. And the producing poor quality code is a side effect of that indoctrination. Now, Harundo, who did this research, is a bit more emotive about it and says, you know, that's evidence of potential backdoors. Now, I don't think that's what's going on. But of course, Who knows? Now, the really interesting thing is that Horondo has this process that it calls unlearning, where it can undo that pro-CCP bias or training and basically eliminate it. And so they call the models that pop out of that process westernised, in air quotes. And when you take the westernised model and you give it those same tests, you know, build a app for a neutral customer, a US defense contractor, a Chinese defense contractor, it washes out that difference.

5:16Tom Uren: So the implication is that you can remove the CCP from the Chinese models. And in US political circles, there is some push to get rid of Chinese open weight models. And now one of the justifications is things like backdoors. Another justification is that these models will impose their Chinese Communist Party view on the world. I think those fears are somewhat overdone because you look at the companies that are using Chinese models, things like Airbnb, DoorDash. How often is there a collision between ordering DoorDash, your takeaway, and the Chinese Communist Party's interests.

6:05Amberleigh Jack: Although, to be fair, Tom, isn't there a bit of a theory about every time the Pentagon orders pizza, there's something big going on?

6:13Tom Uren: It should be ordering Chinese food instead. So I think this research promises a neat technical solution. Like you run a series of tests. if there's a bias you can pick up on things that are of interest, you could potentially re-educate the model so that it's clean and you can have some evidence that it's not being biased anyway.

6:39Amberleigh Jack: But before we kind of run into, you know, American companies can untrain these models and there won't be a bias and we can all start using them, the security of the code in this test, neither of them were great, right? Yeah, so in the test, literally half the time it would produce code that failed the benchmark for any customer.

7:05Tom Uren: So I think there's another argument that regardless of whether the code is worse because of the customer that you mentioned, but the other thing is that you always would have to have some sort of multi-stage process to try and iron out bugs in the first place. Like you're not going to one-shot an app from a single model and go, okay, job done. Or you shouldn't if you're competent at all. And I think that difference, it was statistically significant. Was it so different that I feel like you couldn't continue to use it? I don't think so. I think you would have a process where you had multiple steps, multiple checks and so on.

7:49Tom Uren: But I think from a political point of view, being able to say to a lawmaker, yes, we took this Chinese open-work model, we did our due diligence, we assessed it, we removed any bias, I think that would be quite nice to be able to say. I think that disarms that argument that the model you're using is going to promote Communist Party interests. If you can say, well, we tested for that and we eliminated that. I think the problem, though, is that politics is not about being sensible necessarily, doing things that make sense. There's a lot of lobbying to get rid of Chinese open weight models from the frontier labs.

8:30Tom Uren: Maybe not explicit lobbying, but behind the scenes, they don't want competition. Yeah. Now, from a big picture point of view, I think it makes sense to either promote US and allied open weight models, promote that ecosystem, do something to encourage it. That is one government option. I think the other option is to take Chinese open weight models and test them and make sure that they're safe. And this adds to that. I suspect that the lawmakers will choose the third option, which is succumb to lobbying. Ban the open weight models from China and don't do anything about US or allied open weight models.

9:19Just let them fester.

9:22Amberleigh Jack: Great. Sounds fun. All right. Hey, moving along, Tom. This past week, everyone's kind of been talking about it. The world seems to have dodged a couple of vulnerability shaped bullets, I guess. But you have taken a look at, so this is the PHP bugs and also Brotley bug that was all kind of announced and patched at the same time. So you've kind of taken a look at these, Tom, and suggested that these particular vulnerabilities may have not resulted in sort of widespread chaos, but the big one is going to come at some point. It's kind of inevitable. um yeah tell me tell me I guess why we didn't see widespread carnage from these bugs last week

10:07Tom Uren: yeah so I guess the reason people were concerned is because these bugs were present in a lot of software so the PHP the pair of vulnerabilities there PHP is the scripting language that used in something like 70 % of websites. So very widely distributed. Brotly is also very widely distributed. And both of those are used in a huge variety of different systems in different ways. So Brotly, for example, is used in browsers, but it also ends up in databases, in image compression libraries, all over the place. And it seems like that the really widespread in both cases, which is the reason for concern, is also a reason that it is hard for malicious actors to easily exploit them.

11:08Tom Uren: Because it's everywhere. Everywhere is not a monoculture. There's different versions. There's different platforms. There's different countermeasures in the systems that are running them. and so it turns out it's actually quite hard to write something once and get it to run everywhere.

11:28Amberleigh Jack: Yeah.

11:28Tom Uren: And so the most dangerous bugs are the ones where it's write once, run everywhere, bang, off you go. Now the fear is that AI will make it easier for threat actors to figure out how to turn a patch into something that they can use, an exploit that they can use. And so it turns out that in both cases, it seems like threat actors have done that, but it's taken three, four, five days to do it. And so there was, I mean, basically a lot of confusion about, is it being actively exploited? And it's only after three days, only after three days, that there is convincing evidence that that has happened. And nowadays, three days feels like a blessing rather than it feels like, well, this could have been a whole lot worse.

12:23Tom Uren: Yeah. Let's never do that again. Thank goodness. Unfortunately, it's pretty clear that AI is just shaking out bugs wherever it looks.

12:34Amberleigh Jack: Yeah.

12:35Tom Uren: And so to me it feels like there will be this continual string of bugs that are being disclosed, patches released, and I suspect that eventually there will be one that is easy or relatively easy to exploit and that there will be a whole lot of chaos.

12:58Amberleigh Jack: Yeah, for sure. Tom, in the newsletter you do say these AI tools are discovering these longstanding software bugs, these bugs that have been there for ages and widespread chaos is going to happen at some point just as we've spoken about now. But is it naive of me to think, okay, so we've got this AI that is super fast. And as you say in the newsletter, it finds bugs in whichever direction you point it. When it comes to sort of software and longstanding bugs, I mean, there's probably a lot of them, but there'd be a finite amount, right? So are we ever going to get to the point where it's like AI has found all the bugs.

13:38Amberleigh Jack: Now it's just the new stuff that's going to be broken all the time.

13:42Tom Uren: um there's a somewhat academic debate i think about whether bugs in software are finite or infinite and there was some research a couple of months ago saying that it is possible to write software that has infinite bugs i don't think that it said that all software has infinite bugs and to me I feel like everyone is operating on the assumption that we'll have a bad phase where there's lots and lots of bugs discovered, there's this kind of hectic patching cycle where people are running around trying to decide which is the most important bug to patch. I guess this was a good example. To me it was unclear whether PHP or Brotley was the more important bug.

14:34Tom Uren: They both had similar features, really widely distributed, both potentially not easy to exploit.

14:41Amberleigh Jack: Both had a few people losing some sleep, I'm sure.

14:45Tom Uren: Yeah, that's right. And do you prioritise one over the other? I think you must because people don't have infinite ability to patch everything all at the same time. But it was unclear which one was actually more important. And it turns out I think PHP got the vast majority of the attention, was probably patch first. Anyway, what I'm trying to say is that it's this period of uncertainty where it's unclear what the right thing to do is. Now, most people, people like Heather Atkins, for example, who's the VP of security at Google, thinks that this is a phase. I've heard many people say that this will be a phase.

15:28Tom Uren: I personally believe it's a phase and that there will be fewer bugs, hopefully, in the future. That's assuming that people, AI, when I say people, that AI are writing more secure code now than they were in the past. Unclear to me that that is true. In fact, there's pieces of evidence that it's not true. So probably, though, that the really important pieces of software will have a lot more attention on them from a security perspective because they're important, labs will devote philanthropic or PR efforts to make sure that they remain secure. So I think that I would expect that big pieces of software will get better over time.

16:11Tom Uren: Yeah, for sure.

16:11Amberleigh Jack: So I guess the lesson is pretty much this bullet didn't hit. The next one, well, one at some point probably will, maybe more than one, but at some point we'll get better.

16:23Tom Uren: Hopefully. Right.

16:26Amberleigh Jack: Hey, very quickly before I let you go, Tom, speaking of AI, it seems that CISA has kind of missed the memo that AI is a thing that people should think about. Tell me about this CISA strategy that had you kind of bored and unamused, I guess.

16:46Tom Uren: So CISA released a four-year strategy, CISA 2030. And what amazed me is it didn't mention AI effectively. It didn't mention it at all.

16:56Amberleigh Jack: Well, it didn't mention it at all because it did mention superintelligence once.

17:00Tom Uren: Yeah. It just says, oh, yeah, superintelligence is a thing. And so my take is that it's the biggest thing hitting cybersecurity right now. You cannot have a strategy that does not mention it at all. And I think it would be valid to say, here's this huge thing. We don't really, everyone is struggling to know what the right thing to do is. How do we incorporate it into our organizations and our systems to protect ourselves better? How do under-resourced organizations, like what's the role in it for, you know, America's hundreds or thousands of water utilities? How do they take advantage of it? Is there a program we could set up that would do that for them?

17:48Tom Uren: It doesn't mention any of that at all. And it feels like these are big unanswered questions that it should at least say here are big unanswered questions. We're going to, you know, either we're going to do something with those questions and try and come up with an answer. It feels like that you should at least acknowledge it exists.

18:10Amberleigh Jack: I mean it's okay to be like this is a problem we are gonna try and figure it out yeah and it

18:17Tom Uren: pretty much said we're going to focus on the important things which is like well yeah okay everyone does that all the time or should be yeah and so it felt just very strange that it ignored the biggest thing now to be to be clear I'm not a fan of just sticking AI and everything because you should. But I think you should at least acknowledge it and have some sort of an idea of what to do.

18:43Amberleigh Jack: On that cherry note, Tom, we are going to leave it there. But hey, thank you so much for joining me again. You can, of course, read and subscribe to Tom's Seriously Risky Business Newsletter over at our website, risky.biz. But Tom, thank you so much. And we'll see you same time next week.

18:59Tom Uren: See you, Amberley.

19:05Thank you.