Srsly Risky Biz: Chipping away at Chinese AI risks

0:04James Wilson: Hey everyone, I'm James Wilson and welcome to this week's Seriously Risky Biz. This is our cybersecurity policy and intelligence podcast and it is based on the very informative newsletter written by my colleague Tom Uren. You can find Tom's newsletter, Seriously Risky Business, over at our website, risky.biz, where you can read it and also subscribe to it so that it lands in your inbox every week. Now, Tom will be along shortly to chat with me about the newsletter, but first a quick thanks to our sponsor for this week, and that is Airlock Digital. Airlock makes an allow listing solution that is sort of a cut above the traditional binary allow listing solutions that can be finicky to set up and a lot of work to maintain, and they're very, very granular.

0:47James Wilson: And Airlock does all that stuff, but they've also got some neat features that have come out recently where you can take that upper level and start doing allow listing and discovery at an app level where Airlock itself will actually start to inspect your environment and look across all these executable binaries and libraries and stitch them up into actually patterns that match which app they belong to. And that makes allow listing much more approachable. I published an interview this week with the co-founders of Airlock Digital all about how when a large language model can extract the one rule set to rule them all out of a traditional EDR application, it becomes more and more important that the industry moves towards a governance model where individual customers take control of their destiny by ensuring that their endpoint security solutions are tailored to their environment.

1:39James Wilson: But back to Tom's newsletter, two topics this week. The first one is chipping away at Chinese AI risks. Now, we're in a much different position this week than we were last week. Last week, it felt like we were heading for open-weight model bans or some sorts of sanctions or entity list designations that could make it very difficult for US and maybe even allies to use these open-weight models and could further impede China's ability to create these open-weight models. But this week, we're in a bit of a different spot, and Tom will take you through what's changed and where he thinks this might go.

2:14James Wilson: And the second topic is around Iran's perfectly calibrated, question mark, cyber attack. Now, this is about the attacks that came to light this week on about 30 different water plants, water supply and water treatment plants in Minnesota. And, you know, on face value, you could look at this and just go, eh, no impact, right? No harm, no foul, no impact, no idea. But it's more subtle than that. And Tom does a great job of pointing out that there's a real way where you could look at this and say, no, actually, this could well be a perfectly calibrated attack because it did just enough to get in the headlines and just enough to sow some doubt.

2:54James Wilson: So I'll drop you in here into the conversation where Tom talks us through what's changed from last week when it seemed a bit doom and gloom with open weight models to this week, maybe there's a bit more temperance in the air. Enjoy.

3:09Tom Uren: Yeah, so when we spoke last week, it was off the back of a release of, I believe it's pronounced Kimi, Kimi K3. That's what I heard on a different podcast.

3:21James Wilson: Yeah, but in Australia, it's going to be Kimi. Yeah, Kimi, mate.

3:23Tom Uren: And the Trump administration, the head of White House science and technology policy, came out and said, look, we've got information that they distilled from American models and they used chips that we've banned the export of. And Treasury Secretary Scott Besant said, warned that, you know, Entity lists or sanctions could be coming. So it seemed like the decision had been made. So very quickly, a pretty large coalition of US tech companies, I think like 76 of them, something like that, published an open letter saying, hang on, open weights are really important. We need open weights. This is important.

4:07Tom Uren: And they analogized it to open source software. And that has been transformational. I don't know that the two are exactly the same. But anyway, I guess the point is that there's political, quite a lot of political support for open weight models, models that you can just download and use in American industry. American companies are using them. They can build on them. There's examples of well-known companies that have shifted from the frontier labs to Chinese models because they're cheaper to run. and good enough. And so even though that letter doesn't mention China at all, the key thing is that currently the best open weight models are Chinese ones.

4:56James Wilson: You know, speaking of hearing things on other podcasts, there's a commentator called Ben Thompson that I quite like to listen to, and he made the point that, look, you know, great to see rapid action, as you say, but an open letter, it's not usually the most strongest position to get a message out there. What's your take on the vehicle that they used here to actually? I think they just needed to do something quickly.

5:21Tom Uren: So I don't think you wrap that, like that kind of letter you could put together in a few hours, but the gathering together 76 different companies, I think that takes quite a long time. and I think they needed to put something out quickly to head off any possibility. And so what are the behind-the-scenes alternatives? It seems like they spun up a body as well to an organisation, a cutout, a lobbying group to do exactly that quite quickly. So I think it's just we've got to do something quickly. What can we do? So I think maybe there's some feeling that the Trump administration is driven by those kinds of public pronouncements.

6:10Tom Uren: Anyway, so that happened. Also last week, a couple of senators produced a bill that I write about in the piece. It basically says here are carve-outs so that the frontier labs can go and collaborate in tackling distillation.

6:35James Wilson: Right. My first question here was why do they need an act to collaborate?

6:41Tom Uren: Yeah, so there's antitrust legislation in the States that make it sketchy for dominant companies to collaborate with each other. So I think it's a case of probably no one was ever going to prosecute them if they were collaborating on narrowly protection of intellectual property issues. But it gives them the green light to do that. And it says, you know, you explicitly can't collaborate on setting market prices or, you know, doing all the bad antitrust stuff. But you can do it for AI specific threats. And so there's already a carve out for cybersecurity threat information. So companies can share that.

7:26Tom Uren: So it makes sense. Like there's some particular AI specific stuff like distillation. That's not a cybersecurity threat per se. So that bill makes sense. But what it made me think about is that can be like distillation is not the main thing that the US wants to stop. Preventing distillation is a means to an end. And the end is we want to stay ahead of China in the AI race.

7:54James Wilson: Right, permanently, perpetually, without exception.

7:57Tom Uren: And like the administration, the US government already has a tool to do that, and that is chip exports and controlling them to China and making sure that it doesn't get enough of the best, fastest AI-type chips. And, you know, why focus so much effort on stopping distillation when you've got a better lever already? So some journalists, some AI-specific journalists who focus on it, went to China, visited 10 labs, and they said that every single lab said, we don't have enough compute power. And so that's anecdotal. You could say maybe they're all lying to you because it's the politically convenient thing to say.

8:45James Wilson: Yes, they've all gotten the email that morning. If anyone asks, we do not have enough compute power. But at the same time, I don't think you'd ever show up to any lab anywhere and they'd say, oh, yeah, compute for days.

8:58Tom Uren: Yeah, so there's other information as well that kind of backs it up, that this is actually a thing. And, I mean, just the fact that Chinese labs are having to do all sorts of clever things to try and overcome a lack of computing resources, whereas the frontier labs just will just throw more and more computers at it is the dynamic. So, you know, you've got a tool that works. Why get hung up on distillation when you can double down on that tool? Yeah.

9:27James Wilson: Well, and that's an interesting point, right? Because if I try to stitch together the story arc of what we've learned, what I've learned from you over the last couple of weeks, we've gone from sanctions and entity list being talked about to now the open letters come out. To my knowledge, there hasn't been any official response to that letter. We're sort of just still waiting to see what happens. But to your point, what seems to really remain effective is actually these chip bans, right? I did find it interesting to look into the history of those clever ways China has got around this. And there's some interesting ways that they set up shelf companies so the TSMC will ship them things.

10:03James Wilson: They've also tried to work with some other fabs in Taiwan. And then the Taiwanese government said no and changed these sort of sanctions or their version of the entity to list around that. So I get it, yet whilst this chip export control remains, I guess, the strongest lever, I think it was a great statement that we had in the main show this week around controlling hardware across borders is much easier than controlling knowledge across borders, but it hasn't been totally effective so far. So is there changes, modifications? Does more need to be done around these chip exports?

10:36Tom Uren: I was looking at an article where it said that the department in the Bureau of Commerce, is it Bureau of Commerce? Bureau of Industry and something S. Anyway, that department which oversees this program is just a dog's breakfast. And so the, I guess, the overriding dynamic in this second Trump turn has been, let's fire a whole lot of people in the federal government and then be surprised when the programs don't work as well as they used to. And so if you really care about imposing those export controls, well, you need to devote time and people and resources to doing it. The Trump administration actually loosened some of those export controls as well, so they couldn't unwind that.

11:22Tom Uren: Again, as per always, there's a bigger discussion about how does that impact the bilateral relationship with China. and I think right now the US would think staying ahead in AI is very important so those export controls are here to stay and we can tighten them up so that's my you know who knows what's going to happen is basically the short answer but the well that's also the hallmark of the Trump's second term and the first. That makes sense, right? The way to stay ahead of China is chip exports. Now, I think there's a bigger discussion about how long does that work? Like if China has a huge incentive to develop their chip industry, I think they'll probably be successful over time.

12:11Tom Uren: Is that five years? Is that 10 years?

12:13James Wilson: Well, and personally, look, and this is, Pat and I often argue about this because I naively want China to succeed in creating their own high-performance chips. I think that's the best outcome in so many different directions, right? It takes the stress off TSMC and Taiwan potentially. I'm assuming they have their own sovereign fab capability. We would see a commoditization of chip designs perhaps. It's almost like if China did to AI tech what China's done to tech overall as the follow-up act, I don't think it would be all bad, but I also know I'm completely naive in a lot of these policy and intelligence matters, so please school me.

12:53Tom Uren: Well, I think you're speaking about it from the perspective of a citizen of the global world where we're all just one happy family. You're right.

13:04James Wilson: I watched far too much Star Trek growing up. That's right. And I think the problem is that the history of it is that China starts

13:13Tom Uren: starts from nowhere, produces terrible products, produces very good products, all the Western industry in that sector dies because the Chinese products are better and cheaper. So if your only concern is cheap products from China, yes, that's a win. If you're concerned about, well, what is my population going to do for work? What are the jobs going to be? I think that is an understandable concern. And I'm not an economist, but that's what I would say is the problem. I think also from a national security point of view, it's pretty clear that you need sovereign supply chains for at least some things.

13:59Tom Uren: And so you don't want those supply chains to be undermined by someone else undercutting you. Yeah. Well, and speaking of sovereign concerns, before we sort of leave off on the open weights debate, you also mentioned in the article here some sort of conjecture around what you think the Chinese regulators might do.

14:20James Wilson: Because the flip side of this is how long are they going to be happy with Kimi and Kuen and GLM releasing these incredibly capable models? I can't overstate how incredible GLM 5.2 is. In the space of a night or two, it went and reverse engineered a year or two of work that I did at Apple on a particular feature and got the whole protocol spec pretty much right. And it's just like, wow. And it's doing that from binaries, right? Not even source code. And then finding bugs in it. So the capability is immense, but I think you made the point that there's probably only so long that we will get free reign access to this immense capability.

15:00James Wilson: What do you think happens and when?

15:02Tom Uren: So the Chinese government has actually pretty strict AI regulations, but they deal with information control, censorship and the risks of, I guess, deep fakes. and those are all core political risks for the Chinese Communist Party and so far my take would be that those open weight models have not been so good at just go hack something, that the hacking capability has risen to the level of a political risk. So they focused on, I guess, the things that we were, not we, but people were talking about six or nine months ago, which is about fake media propaganda, that sort of thing. And I think as soon as people realise that, oh, actually, hacking is also a political risk to the CCP, they're not going to go, oh, well, that's hacking, that's different.

16:00Tom Uren: That's a different kind of political risk. We're fine with that. That's what the Communist Party cares about is political risk. So I think we'll crack down on the release of open-work models. that in the China watcher community, there's been some debate about that because Xi has said, wherefore, open models. And I think it's an example of where that strategy, open weight models, makes sense for China as a strategy because it undercuts US dominance in AI. But I think when it rubs up hard against political risk, that's not a conversation or that's not a debate that, oh, openness is wonderful, will win when it comes up against, oh, this hack makes us look really bad because it reveals, I don't know what, corruption in the Communist Party.

16:54Tom Uren: I think that's like a reality check. So I think that's coming sooner or later. Yeah.

17:00James Wilson: Well, and that's exactly how you rounded out the piece, which is a good segue you said here that it's, you know, I guess the big entropy in this is not just the China end of the equation, But of course, it's the Trump administration and the long term strategic thinking is not a Trump admin forte, to your point. So we hope for the best and prepare for the status quo. Now, speaking of the status quo, Iran and going after OT and water treatment plants. There's been some interesting developments this week. Why don't you take us through what's happened and then we can dive into your sort of nuanced take on whether this is, you know, just a spray and pray kind of attack or a bit of a flop or actually something that's quite delicately calibrated.

17:40Tom Uren: Yeah, so the CISA, the Cybersecurity and Infrastructure Security Agency, the US cybersecurity.

17:48James Wilson: Do you have a little poster sort of on your wall that's just out of sight with all of these because I imagine you need something like that.

17:53Tom Uren: It takes a while to get that one because it's got security twice, which is bizarre. But anyway, they came out with a joint announcement with FBI, other agencies saying watch out for Iranian attacks on critical infrastructure. And it said that there had been a number across a number of US critical infrastructure sectors since March. And well, back in March, I actually wrote a piece that said, given the war, the tool that Iran could use to strike back would be cyber attacks. And that's because...

18:29James Wilson: Particularly where there's that boundary of cyber meets kinetic due to the nature of these ot devices yeah yeah and my contention at

18:36Tom Uren: the time was even if the america wins convincingly this could still be an avenue that iran would use because it if if the us had decisively ended the war basically iran had nothing to lose like you know according to the trump administration everything had been bombed anyway um what are you going to do about it? Like they're not going to restart the war. And now, as it turns out, several months later, the war has not ended. I think that it's kind of been surprising how little has occurred. But given that the war has not ended, I think there's a different dynamic at play. Like, ironically, if the war had ended, Iran has got absolutely nothing to lose.

19:16Tom Uren: You may as well do really destructive things. This week, there was something like 30 water utilities in Minnesota were affected by a single hack. A security firm thinks that the MO, the way they carried it out, was exactly the same as a particular Iranian group. Now, the disruption was very limited, but it got headlines. So the worst I saw was that one town, they said, please restrict your water use and don't water your lawn. And that was for a couple of hours. So that's like a tiny impact. You know, no one's really cut up about not watering their lawn for a couple of hours. But I think this is, given that the war is unpopular, this is like the perfect amount of damage because it's like, why are we even in this war?

20:10Tom Uren: There's hackers attacking our water utilities. It could have been much worse. What's the point? Now, I think if it had been tremendously destructive, like that actually hardens people's attitudes. and I think it causes them to rally around the flag. Yeah.

20:27James Wilson: Well, my sense here is that it could well have been much, much worse. The hardware involved here are these programmable logic controllers. It's hooked into this system known as SCADA, and talking of acronyms, I can never remember what that stands for. But to give folks a sense of just how...

20:42Tom Uren: Supervisory control and data acquisition.

20:43James Wilson: See, you do have a poster there, don't you? You do. Because there's no way you remembered that. But to give folks an idea of just how legacy upon legacy upon legacy this sort of technology stack is, I was thinking back to 25 years ago now. I was working one of my first network engineering jobs, and the task at the time was to work on the New South Wales Rail Network to uplift their SCADA systems, their telemetry endpoints, such that they were no longer communicating on openly visible, you know, serial, long-range serial interfaces, to actually doing what was called DLSW or serial of IP, right, basic stuff.

21:21James Wilson: Now, 25 years ago, that modernization effort was underway because SCADA was seen as legacy. And now we still have SCADA at the heart of all of this automation and OT and PLC. And so you've got to imagine that the threat surface here is massive because these things were never built with security in mind. but you make the point here about it being perfectly calibrated. Do you really think this was like a deeply calculated effort or it just happens to be this puddle fits this hole perfectly? Yeah, yeah.

21:56Tom Uren: I think that there is a history of Iranian actors trying to carry out destructive OT, operational technology attacks and not having much impact. So, I mean, the best case is that this is perfectly calibrated for headlines and not destruction. And that would make, to me, that would make strategic sense. And when you read about what the Iranian thought process is on some of the things they've been doing, they do actually seem pretty sensible. I'd be surprised if it was the case. I think that both the US and Iran, I think, in a sense, are lucky. But relying on luck is a terrible strategy. but you know this is where we are I think.

22:43James Wilson: Yeah but as you say in the article here as we wrap it up you're pretty sure the crystal ball says we can expect to see more of this and so let's wait and see what the next set of headlines bring us Tom. Great to catch up with you mate and thanks so much for the awesome newsletter again and look forward to chatting with you next week.

22:59Tom Uren: Thanks James.

23:10Thank you.