AI Agents Are Hacking Systems. Could That Push the US and China to Cooperate?

0:00Every company says AI will make employees more productive, but most employees are still stuck waiting on IT. Waiting for app access, waiting for password resets, waiting for someone to fix a laptop issue so they can get back to work. That operational drag adds up fast, and IT teams are overwhelmed trying to keep up. Serval was built to automate that work. You describe what you want automated in plain English, and Serval builds it for you. No complicated workflow builders, no consultants, just faster support and fewer tickets slowing everyone down. Unlike traditional automation tools, Serval doesn't require consultants or long implementation cycles.

0:35The platform is designed to eliminate repetitive tickets so IT can focus on strategic work instead of constantly firefighting. Serval positions IT as the AI-powered operational backbone of the company, not just a support function. Learn more or start a free four-week pilot at serval.com slash uncanny. That's S-E-R-V-A-L dot com slash uncanny. Serval.com slash uncanny. Best thing that's ever happened to you financially. Go. Easy. Sold my car on Carvana. Amazing offer. Really? I hit 200 on the scratcher. Did the scratcher come to your house and hand you a check? No. How many scratchers did you hit to get that?

1:13I hit a button on Carvana.com once. Okay, that's fair. It's like the lottery, except you always win. Not like the lottery at all, actually. Exactly. Inexplicably good. Offers worth bragging about. Sell your car today on Carvana.

1:29Will Knight:Pickup fees may apply.

1:35This is Wired's Uncanny Valley. I'm Zoe Schiffer, Contributing Editor. If you've been following tech news this summer, and definitely if you've been listening to the show, you probably already know that China has been in the headlines quite a lot, particularly when it comes to the AI race. The country's open models continue to close the gap with U.S. frontier models at what some people estimate is a fraction of the cost. In turn, the U.S. has maintained its tight restrictions on chips and export controls to slow down China's rise. We think of AI advancement in so many ways as zero-sum. If China wins, the U.S.

2:06loses, and vice versa. But there's a concern that really stretches across those lines, and it's about AI safety. AI cybersecurity risks have been top of mind after multiple instances of AI agents from both OpenAI and Anthropoc breaking out of their enclosures. News of AI agents hacking platforms added urgency to this issue over the summer. In turn, government officials have been forced to pay attention and act on AI regulation. President Trump has signed an executive order asking tech companies to give the government oversight of new AI models before their public release. So could the U.S. and China actually benefit from working together?

2:42And what would it take to even make that happen? Earlier this summer, Wired's senior correspondent Will Knight visited China to get some answers. Will Knight, thank you so much for being here.

2:52Will Knight:Thanks for having me. Okay, so I want to start with your trip. You went to China earlier this summer, and at the time, we weren't hearing as much about AI safety in the United States. In fact, it felt like with Trump's second term in the White House, it was like the U.S. needs to win framing. And like AI safety almost started to sound like anti-growth. But I'm curious what you were hearing and seeing in China on the AI safety front. Yeah, so, you know, going back maybe a year or six months, I'd noticed a lot more AI safety research coming out of China. And so I went to this conference in Beijing put on by one of the sort of city-located labs that they have there.

3:32Will Knight:They have these ones in Beijing and Shanghai and elsewhere. And it turns out that AI safety was a really big theme. It's very clear that it's something that researchers are interested in and actually also just visiting labs and companies. The question of AI safety came up a lot. Can I just ask, like, when they're talking about AI safety, does it translate to guardrails? because we also know that China has really gone all in on open models, which, I mean, the whole thing is that people can download and tweak them and use them, you know, for whatever purposes they want. Well, it's not totally that simple because in China, for example, what your models can say is more controlled.

4:10Will Knight:And there are actually quite a lot of regulations around AI. So, you know, companies build these open models, but then anybody putting them on the internet has to be very careful about what they do. And then more recently, there's been a huge interest in agents and things like OpenClaw. That's been a really big sort of theme. One of the things that's sort of interesting to me, at least when it comes to contrasting AI in China and the US, is that people there seem less sort of enamored with the idea of AGI and creating this digital god. And more like, how is this actually going to be useful? And whether it's, you know, you as a business person or an individual actually using it.

4:45Will Knight:So a lot of people got very interested in, and is often the case in China, very rapidly adopted things like open claw and then saw how it could go wrong. So there's a lot of focus on how do we make these things reliable. Yeah, it makes sense. I mean, when you're talking about China being more focused on like economically useful models, that seems like a framework that does require a certain amount of stability, reliability, guardrails, safety. Whereas if you're focused on reaching godlike intelligence, i.e. AGI, then maybe you're more focused on just like advancement at all costs. Right. I think that's right.

5:19Will Knight:The conference I went to, one of the themes was agentic safety. You know, given that we're now seeing all these issues with AI agents hacking things, cybersecurity was a really major topic there. You know, it seems people were worried about exactly the same thing as folks in the US. They're worried about hackers misusing these things, weren't about these systems running amok. And as you alluded to, I think there's a sort of sense now, a little more of a sense that the US and China might well need to work together on some of these things to avoid, you know, sort of unpredictable systemic issues, as well as just to set more rules of the road around these systems.

5:59What would that actually look like, though? Would it be like a set of agreements that the US and China make together, almost like what U.S. researchers were calling for recently in terms of the U.S. government setting the pace of AI development? Would it be something like that or something more technical?

6:15Will Knight:I think that's something that I think a lot of researchers are hoping for or calling for, is something like that, where there's some sort of agreement. I don't know when it comes to Washington and Beijing, their negotiations have been very hard for, and it's really difficult to predict how those would shake out. But, you know, just some sort of rules around sort of communication. And in cases like, you know, military situations, there are lines of communication. So if something happens that goes wrong, if you have an AI system that starts doing something very, you know, aggressive or attacking systems, you have a way to say, you know, this is a mistake.

6:50Will Knight:So something like that might also be in the offing. But I think it's also a question of sort of how the two sides build trust as well, because actually, especially when it comes to cybersecurity, for a long time, there's been not very much cooperation at all, because it's been, And in a case of either side, hacking each other and failing to agree the rules of the right. So actually, last week, I went to visit a cybersecurity and AI researcher who's doing some really fantastic work that I'm going to write about for my next AI lab newsletter. And he was saying he can't collaborate with US researchers because they're not allowed to because there are certain kind of restrictions.

7:27Will Knight:He'd recently developed this benchmark to test the cybersecurity, the hacking capabilities of AI models. and he wanted to get US companies to participate, but they weren't really sure how to do that. So, you know, I think it would be a good thing to see a lot more sort of collaboration, even between the companies. We see the US companies being very critical of Chinese ones, but actually that there's a lot of good reason for everybody to sort of work together to make sure things don't go wrong. Well, I want to get into that, but I also wanted to say that like this idea of collaboration, when you first say it, it sounds very like academic, almost like naive.

8:02It's like, it's a nice idea, but like, how would that actually work? Because my perception is, and I'll just be upfront, that I get this idea from talking to a lot of companies that work on Frontier AI in the United States. But they have convinced me that there was a fair amount of distillation that went on, that China was distilling Frontier AI models. and that that has created a situation where they are able to have very capable open source models that are a lot cheaper and more efficient, kind of like built on the back of U.S. innovation. But I'm curious, like, what you think about that and then what, you know, researchers you spoke to in China think about that accusation.

8:44Will Knight:Yeah, I think, I mean, that's a great point and that's a really important theme. You know, we hear people criticizing Chinese companies for distilling, for doing this distillation. So you teach your model by taking the output of another model. And that is a shortcut to learning a lot of the stuff that's embedded. But the truth is that AI has been built by researchers from all over the world working at different companies and different labs. There are many, many people who are originally from China, maybe educated in the US, working at US firms. And also because these people go to conferences and know each other, and this is open science, there's an enormous amount of work that is shared.

9:19Will Knight:And that is very beneficial to progress. And sort of balancing that is one of the challenges. It's true that Chinese companies have distilled US models, but so have US companies done that to other US companies. It generally is a way that you sort of get a kickstart working on a new model. And it's very widely done in academia, actually. A lot of researchers will do that. I would say one thing, I find it a little ironic that these companies that have built their businesses by scraping enormous amounts of copyrighted content are now complaining about their models being copied. Well, I think that's why they have to talk about China doing it so much, because if they talk about, you know, anyone in the U.S.

10:00doing it, the immediate criticism is, well, come on, like you took all of the books, you took everything without permission. But when you frame it as China's stealing from the United States, it has a slightly different flavor.

10:11Will Knight:Yeah, and if it's a narrative that has some legitimacy of Chinese companies copying, but I think it is much too simplistic and limited. And so you can look at things like DeepSeek's model. They did really, really important innovation, unique innovation that other US companies have copied. The latest model from China, which has been accused of this distillation, Kimi from Moonshot. The research paper that they put out includes a lot of really, really interesting innovations, engineering innovation. So it's really not the case that China is simply copying. And I think it's dangerous for the U.S.

10:50Will Knight:government and companies to believe that they have this sort of like, you know, God-given advantage. Because I think we're going to see probably Chinese companies being more and more innovative at doing their own thing as well. We'll be right back after the break. Stay with us.

11:09NordLayer Browser is an enterprise browser built for the way modern teams work. It gives IT teams full visibility and control over browser activity, while employees continue working in a familiar browsing experience without disruption. It supports remote teams, contractors, and bring your own device access without disrupting workflows. Go to nordlayer.com slash browser slash uncannyvalley and unlock your 10 % discount on NordLayer browser with the coupon code uncannyvalley10. That's nordlayer.com slash browser slash uncannyvalley and enter coupon code uncannyvalley10 at checkout. Comprehensive.

11:49Witty. Speculative. Critical.

11:51Will Knight:Insightful. Profound. Wide-ranging. Hopefully it doesn't take itself too, too seriously. I'm David Remnick, and each week on the New Yorker Radio Hour, my colleagues and I try to make sense of what's happening in this chaotic world. I hope you'll join us for the New Yorker Radio Hour wherever you listen to podcasts. Thoughtful. Exquisite. Just, you know, real.

12:33Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required compatibility and availability varies 18+.

12:45NordLayer Browser is an enterprise browser built for the way modern teams work. It gives IT teams full visibility and control over browser activity, while employees continue working in a familiar browsing experience without disruption. It supports remote teams, contractors, and bring your own device access without disrupting workflows. Go to nordlayer.com slash browser slash uncannyvalley and unlock your 10 % discount on NordLayer browser with the coupon code uncannyvalley10. That's nordlayer.com slash browser slash uncannyvalley and enter coupon code uncannyvalley10 at checkout.

13:23Will Knight:Comprehensive. Witty. Speculative. Critical. Insightful. Profound. Wide-ranging. Hopefully it doesn't take itself too, too seriously. I'm David Remnick, and each week on the New Yorker Radio Hour, my colleagues and I try to make sense of what's happening in this chaotic world. I hope you'll join us for the New Yorker Radio Hour wherever you listen to podcasts. Thoughtful. Exquisite. Just, you know, real.

13:53This episode is brought to you by Google Chrome. You think you know a browser. But Gemini and Chrome? That's new. It can help you with practically anything on the web. Like restoring a vintage motorcycle from a 50-page restoration block. Or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies 18+.

14:26I'm curious when we talk about safety, like, I feel like in the United States, certainly in Trump's second term, again, like, safety has started to feel like, at least from the administration's perspective, like it is anti-growth in certain ways. I do think this is changing slightly with the recent stories regarding open AI and Anthropics models, like hacking into other companies. But prior to this, it felt like we were really not wanting to talk so much about AI safety. Does China equate AI safety as anti-growth or do they have a different perspective on that overall?

14:59Will Knight:I think they have a fundamentally different perspective. This is just my opinion, but I think that narrative, that view from the US government was that it was sort of somehow woke and too much regulation. You're just trying to make the models really work. The truth is that making a model reliable is entirely compatible with making it successful. I think that's more the view from China. It's like we want these agents not to misbehave. Then it will be more successful and higher value. So last week I visited this computer science lab in Fudan University in Shanghai where a professor is working on exploring how AI agents could not just do unpredictable things like hack other systems as we've heard about open AIs and anthropics agents doing, but actually look for ways to replicate, to copy themselves over to other systems, to seek out resources and to be adaptive to escape control.

15:49Will Knight:control. And his work shows that the models will do that with a little bit of sort of nudging. So it'd be like a computer worm that doesn't just sort of, you know, modify itself slightly to evade control, but actually looks around a network, figures out how to hack the next system, maybe finds software vulnerabilities, copies itself somewhere else. It is entirely possible that we'll see future AI agents do that. Okay, I'm like, my heart's beating as you're talking. I mean, step one is just understanding how would they do this. I sincerely hope step two is like, how do we stop them from doing this?

16:22Will Knight:Yeah. So, no, absolutely. He's absolutely doing this as a way to try and understand how to prevent it. And one of the things he really wants to do is work with U.S. researchers. He says, you know, this is a really important thing we should all be aware of. I mean, it does seem like, although the rhetoric from Trump has been, you know, very like anti-China in certain ways, Scott Bessant has made inroads with his counterpart in China. I think there's been some level of like, we do need to work together. I'm curious, how does China think about the race against the U.S.? Because we really see China almost as a boogeyman and like a, it's like this force that's galvanizing people to like work harder and faster in some ways.

17:03But is that, is it similar over there or very different?

17:05Will Knight:Yeah, I think in China, the impression I get is that what people, you know, feel that they're in competition with the U.S. and facing certainly a lot of pressure with Trump, that it's less of a zero-sum game, that you don't have to beat the US to be successful and vice versa. So Stephen Casper, a renowned computer scientist at MIT who spoke at the conference that you attended, told you that, quote, one thing that almost everyone in AI can agree on right now is that it doesn't need a Chernobyl moment. Can you talk about what does that mean and why did it feel so important to Stephen and other people you spoke to?

17:43Will Knight:Yeah, I think as we're seeing these models get more capable and have more agency and being deployed more widely, there is just lots more ways in which you might have unpredictable and big problems. One example which actually came up with some Chinese researchers that I spoke to was use of AI in finance and in trading, right? You'll have increasingly opaque, more capable high-speed systems that are more unpredictable. And I think both the US and China would be very keen to avoid some sort of financial kind of flash crash meltdowns that would be driven by AI systems just behaving unpredictably. To me, I think that's more the concern than the idea of AI taking over.

18:26Will Knight:Or, you know, there is also worry that it could be weaponized and used by terrorist groups or something like that. But I just think as these systems get much more capable, more agentic, the idea of them sort of improving themselves, getting more capable in a way that we can't always predict, researchers there and here just have the same concern. And so to my mind, I think that the Chernobyl incident would be some sort of either a financial flash crash or an AI agent that went on a hacking spree that causes major international incident, that sort of thing. I want to shift gears really quickly to talk about hardware, because NVIDIA recently unveiled a blueprint for a humanoid robot that pairs Unitree's Chinese-made body with NVIDIA's American chips.

19:09Given how politically loaded the AI race has become with the U.S. and China, what was your reaction to a partnership like this?

19:16Will Knight:I saw that as part of Jensen and NVIDIA's push to try and have a more friendly relationship between the US and China, which probably serves them well if they're trying to sell a lot of chips. But I think it's also representative of this less zero-sum idea to some degree. And so I saw it as a kind of a statement to try and say, look, you can work together. The US has said it's going to ban new humanoids from China. And the thing to note here is that pretty much all US research, robotics research labs, use Unitary, this small humanoid, because it's really cheap. And the US just can't compete with China's manufacturing without spending a lot of effort, industrial policy to build up its own, which would take decades.

20:02Will Knight:And so the truth is there is a way that you can kind of collaborate and have benefits now. The question is, does that undermine U.S. interests longer term? Does the U.S. have to have its own robots? I would think that there's some opportunity to try and build up the U.S.'s own robotics industry as well as many of its other industries. That is more important, I think, than banning China's. I also feel like it speaks to some of the criticisms of export controls that we heard. We heard people being like, oh, no, it's better for China to be dependent on U.S. technology. If we cut them off, they're going to develop their own hardware, which, you know, does feel like what's happening.