China vs USA: Should Trump ban open-source AI?

0:00Danny Fortson: This episode of the Times Tech Podcast is sponsored by health and life insurer Vitality, your health's best friend. Most of us want to be healthier, yet life so often gets in the way. Vitality's health and life insurance is built around that reality. Get active, look after yourself, and you can unlock rewards from some of the UK's top brands and help keep your insurance premiums low. It's insurance that works for you. By using tech and insight to understand your health, they can incentivize you to live better. The healthier you get, the more you are rewarded. Find out more at vitality.co.uk. This episode of the Times Tech Podcast is sponsored by IBM.

0:40Wimbledon is one of those rare events that feels steeped in tradition, but behind the scenes, it's also a huge data operation.

0:47Danny Fortson: Millions of fans follow the tournament across time zones, devices, and platforms, and increasingly they want highlights, stats, and personalized updates in real time. IBM has worked with Wimbledon for more than 30 years. using data and AI to help create digital experiences for fans, including match chat, which can answer questions during play, and likelihood to win, which uses live match data to offer near real-time predictions. It's a useful example of AI being applied in a way people can actually see and understand. To learn more about how IBM helps create smarter business, visit ibm.com slash Wimbledon.

1:26Danny Fortson: There's a huge debate that is raging as we speak. Right now, Sam Altman is in Washington, D.C., meeting with Trump's cabinet to talk about this question. It is a fundamental question when it comes to AI as to what to do with open source. Presto changer, we have superintelligence. Why are you going to die and why am I going to die? Because we get out-competed. Stop. Do not build superintelligence. If we build superintelligence, it's too late. If there is a technology that is so dangerous, we cannot allow this technology to exist. Hello and welcome to the Times Tech Podcast. I'm Danny Fortson out here in Silicon Valley.

2:06Danny Fortson: Katie, she's not here this week, but stepping in ably as he often does, is Mark Selman, the tech correspondent for the Times. Hello, Mark. Hello. I'm so happy to join you. Thank you for stepping in to Katie's shoes once again. How's it going? All right. It's a little crazy over here. A little crazy. I'll give you the quick rundown. August 1st, the White House is going to come out with its guidance. It's kind of soft regulations around how frontier AI should be regulated, how it should be rolled out. Should there be like a little kind of period where governments can look at and see and test the latest models?

2:47Danny Fortson: there's also going to be theoretically some rules of the road around open source i.e. free anybody can download it for free versus closed ai models i.e. anthropic open ai etc there's a huge debate that is raging as we speak right now sam altman is in washington dc meeting with trump's cabinet to talk about this question because we have Kimi K3 you know this new model the latest out of China it's free it's open source anyone can download it and it's as good or better than the top U.S. models and that has prompted a complete freak out over here because people are like look this is effectively a super weapon people can use these models to do bad things whether that's like hack into people's systems, disrupt your utilities, your water supply, create bioweapons, whatever.

3:45Danny Fortson: We need to be really careful about this. And on the other side, people are like, look, this is how technology goes. Open source is the way we've always done it. So another quiet week. Another quiet week, exactly. And of course, in the balance hangs trillions of dollars and maybe the future of the economy and more broadly, the future of humanity itself. So yeah, as you say, super quiet out here. We're going to talk about kind of give this context of what's happening a little bit more granular detail. But now we have a guest who's fantastic, who's going to really help us understand the context of all of this.

4:22His name is Connor Leahy.

4:24Danny Fortson: He's the executive director of the campaign group Control AI. He's an AI researcher. He co-founded a Luther AI, which was like an early open source project, founded the AI safety research company Conjecture. And I spoke with him actually earlier about everything that is happening here with a real focus on the hack that that OpenAI that happened with a couple of the OpenAI's models, how it broke out of its, you know, supposedly safe container. committed some crimes, stole some data, and I think kind of drove home the point that when we're talking about control or the illusion of control is what, you know, Connor will talk about before we get to Connor.

5:11Danny Fortson: Yeah. Can we talk about just a few of the top lines of what is happening this week and it really starts with Kimi K3 which I don't know made waves over there but it certainly made waves here I think it did again within the tech community it does this has dominated for close to two weeks now I think obviously because of what happens in the US and this is a US-led discussion but it is a fundamental question when it comes to AI as to what to do with open source and how will that affect the industry and security at large? I know we talked about it a bit last week as well, but Kimi K3, it was created by this company, Moonshot AI, out of China.

5:58Danny Fortson: They put this model out and it was as good or better as the top models from OpenAI and Anthropic across a number of benchmarks. And the reason why that created this freak out is because you have Anthropic, you have open AI, both sprinting, well, Anthropic in particular, sprinting toward an IPO, trillion dollars plus. Thousands of people are going to get very, very, very rich. At least that's what they hope. And then you have the arrival of these open source models that are, depending on how you slice it, 50 100 times cheaper than what an anthropic will charge now anthropic has a much more kind of slick product but the actual core technology what we're talking about here if it's that much cheaper and all of a sudden you have this thing that is a fraction of the cost that's a problem for these ai companies and so there's been a lot of effort and it's really kind of leading to all of these talks this week in Washington because by August 1st, just a few days from now, the White House has said they're going to come out with like basically a volunteer framework for how companies should roll out new frontier models.

7:24Danny Fortson: And it's kind of like a voluntold type thing or voluntold. Like you can, this isn't kind of hard and fast rules, but this is how we think this should go. You AI companies, when you come out with a new model and in the midst of all of that is how do we deal with open source? Because a lot of people out here in Silicon Valley are like, well, open source is like, that is the ethos, you know, let a thousand flowers bloom. and if you try to stop that even if it's coming from china what you're really doing is just protecting the business model and the potential riches of a few thousand people inside a few leading companies that is kind of what is happening right now as we speak as everybody's trying to run around and trying to you know basically make their points do you think they're serious the trump administration in putting any restrictions on i think yes to a degree because we had things like this open ai hack where it was like oh there's this thing from one of our leading companies that completely shocked them as to the capabilities the lengths that went to to reach a goal and oh my goodness they don't really have control over this But that was a closed source model, ironically, run by a closed source company.

8:48Exactly.

8:48Danny Fortson: And they stopped it with an open source model. I know. An open source Chinese model. So, you know, that's the irony of it. Is it a security issue, really? Or is it really an economic issue? Is this stopping essentially Chinese dumping of a cheap product on the market? Right. And I think you even had Mark Zuckerberg. He, you know we're part of the msm the mainstream media or as our opponents call us the lamestream media he gave an interview this week amidst all of this because everybody's trying to kind of set out their stall basically supporting open source and really bashing open ai and anthropic whose kind of pitch is like look this is effectively like a nuclear weapon we have to be really careful about how we rolled this out.

9:40Danny Fortson: And he said, quote, I think a lot of people have this notion that if you build some kind of singular AI, you can through some idealized form of alignment, make sure it is benevolent to humanity. I'm personally skeptical of this path. In other words, we need open source because the thing about open source is that you put it out there and then everybody can tinker with it, test it, push it to its limits, see what, you know, fix it, whatever it may be. The whole point is, you know, you have the collection of everybody's brains on top of it. And he said, the other quote, which is really interesting, he said, I think it's literally impossible to have a single benevolent superintelligence that is simultaneously aligned with everyone at once, i.e.

10:25Danny Fortson: don't kill the open source baby with the bathwater or throw it out, you know, whatever the analogy is. So I just thought it was really interesting because, of course, They're also building their own superintelligence. They've, at least initially, were very pro-open source, and now he's coming out, kind of setting out his stall. Yeah, and obviously, MuseSpark is a closed source model, so he's not exactly voting with his feet there. That company probably did more than any other to accelerate the Chinese open source movement by releasing the very, you know, early on, very powerful open source models, and that accelerated the open source movement, definitely.

11:00Danny Fortson: Yeah, and then just the other thing that OpenAI has come out, With since, you know, I feel like my eyes are glazing over a little bit every week. It feels like there's a letter that everybody signs that says we are for X or Y or whatever when it comes to AI. Open AI organized this letter that was signed by over a thousand employees for a bunch of the leading AI companies saying, quote, So we request that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development. So deliberately pace is an interesting choice of words.

11:42Used the word slow, did they? Stop. No slow or stop there. But yeah, I mean, I think people interpreting it as a slowing. and obviously you've got the co-founders of OpenAI signing it and lots of Anthropic people, Meta, Google, you know, including Dario from Anthropic. From Anthropic, Brenda the Pod. Yeah, exactly, and Jack Clark. So, yeah, who's who, another one of these letters. I mean, no one's going to pay any attention to it, I don't think, because, you know, we had a similar one. I think I remember Demis Hassabis, right, you know, signing one a few years ago. So it's very, very hard to square, you know, these companies, you know, employees signing these letters yet, you know, participating in the industry that's pushing us forward.

12:29And I think I can understand ordinary people going, why are you building it if, you know, if you're so concerned? And that always does come back to that.

12:37Danny Fortson: Well, exactly. And I think that's kind of leads perfectly to this week's interview. Because the question is, why are you building it? Right. And the answer for the companies is very clear. OpenAI and Anthropoc have each raised more money than any other private company in the history of capitalism. They have a huge amount riding financially on these impending IPOs on a kind of winning the race, quote unquote. And they've both said very clearly, we want to build a super intelligence, which by definition is something that is more powerful than kind of the sum of human knowledge. All of our collective wisdom put together.

13:17Danny Fortson: But it does lead to the question of like, so what's the plan? And there's that tension between the financial imperative and is this okay for us? Is this going to lead to a dark place? And one of the people who is kind of very clear on the stakes is Connor Leahy. Lee, he actually spoke with him initially last week about kind of the open AI hack. And I just thought he was a really clear eyed thinker on all of this. And as I said, he's an AI researcher, he co founded a Luther, this open, early, early open source project, founded Conjecture, which is an AI safety research company. And we started off by chatting about the open AI agent that managed to hack the startup hugging face last week and what it tells us about this current moment.

14:17Danny Fortson: Let's have a listen. Based on your work and what you have been seeing, why is this a time where we should stop and be like, oh no, this is really, really scary? Yeah, because we're really seeing this transition from just chatbots to agents, to systems that can act in the world, that can make decisions and unilaterally pursue dangerous, complex plans that are clearly against the user's intentions. If you're familiar with AI from two years ago, you have no familiarity with AI that exists today. It's so different because we've had this change from just chatbots, things to go back and forth with, to agents, systems that could take actions in the world.

15:04And it's also important to understand that the underlying technology has changed quite a bit. Where in the past, these were what were called LLMs, large language models. So they were trained to predict the next words. That is no longer true. This is very important to understand. There's LLM components to modern AI systems. It's a very important part of modern AI. So what are they now? These systems now also use many other techniques and in particular, what is called reinforcement learning. Reinforcement learning is kind of like how you would like, you know, maybe like train a dog or something, You tell it to go fetch, and when it fetches, you give it a treat.

15:38Where instead of training the AI to predict the next word, you train it to solve problems or to win it games. You give it a game or a test or something, and you just have it kind of run and see if it can solve it. And if it can solve it, you give it a reward, so to speak. You have it learn from that result. And this has been a larger and larger and larger percentage. We still do a lot of the predict the next word thing. That's still an important component. But we now do a lot of this RL. What's important to understand about reinforcement learning or RL is that we've known that RL leads to agents that hack and cheat and steal at every opportunity since the 1980s.

16:14This is a thing that we've always known is that if you just train the thing to get the reward at any cost, you create these crazy power seeking things that will take every possible cheat they can find. This is a thing that we've known for decades. And now we're really seeing this applied at these very, very large scales. So this was always going to happen in a sense, but now it's happened. Now we have the example. We have the fire alarm. We have the warning shot where, thank God in this instance, no one was really hurt. The companies weren't really harmed. But imagine if this had gone after something else like a power grid or it had gone after your bank or whatever.

16:54Recently, the director of the NSA said that Mythos, one of those powerful AI systems, was capable of breaking into almost all of their classified systems in hours, not weeks. These are not words that a military wants to hear. This is not words you want to hear about genital safety. And this is still before we're even talking about superintelligence. So systems that are smarter than humans and more capable than humans at all kinds of business activity, political activity, military activity, for that matter. If we have systems like this not aligned with our values, doing these crazy things that we can't know or predict, it's really hard to imagine that going well.

17:35Danny Fortson: right so i want to talk about the alignment question and then round out with what is happening this week around this debate over open versus closed if we're moving to a place where this notion of super intelligence is real where these systems get stronger and stronger is there a world where we can train them to be like we just want you to act in our interest ultimately. Is that possible? Because we've had Dario Amadei on this pod, and they talk about, you know, installing this kind of like moral code inside their AIs. And then they'll come out with a report and be like, oh, our AI is lied, cheated and steal in our, you know, evaluations.

18:21Danny Fortson: So I'm just wondering, like, on a technical level, is that just fanciful? Is it possible today? No. Categorically, no. Anyone who says anything else is selling you snake oil. Anyone who claims in any degree that we can control even the AIs that exist today at the level that we would need for this little thing is lying. Lying through their teeth. I can tell you this with complete confidence. Now, could it be possible to figure this out, to develop this technology? I think it's imaginable. But let's take a moment here to just think about what that would actually look like. say you want to develop an aligned superintelligence.

19:01What you're basically saying is I'm going to build something that's so powerful. It can replace the government. It can replace the economy. It can replace all people by force if necessary. It will basically be a one world government and it will solve all moral problems. It will solve all economical problems. It will solve all medical problems. It will solve all interpersonal problems, all military problems, all political problems. And I'll do all of that using software on the first try with no bugs. And I'm sorry, that is not how anything in reality works. I'm an engineer. This is not how anything in the real world works.

19:40So there's the technical aspect of just like, obviously, this is the hardest scientific problem humanity has ever faced. The idea that you could build a one world government out of software that is somehow good and okay is crazy on many levels, not just on a technical level, but also on the moral level. Who gets to decide these things? Well, we should also say the stated goal of Anthropic, of an open AI at all, is we want

20:07Danny Fortson: to make superintelligence. We want to make the thing that is smarter than the sum of humanity's wisdom and knowledge. That's our North Star. Yes. They're blatant about this. And if you go to the parties in San Francisco, they'll be even more blatant about what they intend to do with that superintelligence. Of course, this is false. They won't actually do anything with the super intelligence because the superintelligence will do things with them. AI is very different than traditional software even today. You know, most software is written line by line with code. The engineer is telling the computer exactly what to do in what order.

20:43This is not how AI works. AI is more like grown rather than written. You have these huge piles of data and using what's called neural networks. You kind of have a program to self-assemble using your data. You just train it. It learns from your data. And what comes out the other side is kind of like billions and billions and billions of numbers called the model. And if you multiply and add all those numbers in the right order, you get chat GPT. But importantly, we do not understand why. We do not understand what's going on inside those numbers. It's the same way as like a neurosurgeon. You know, they You can open up your head.

21:19You can look at all the neurons in your head. They're right there. You can go look at them all you want. That doesn't mean you know what this person is thinking or what they will do in the future. We have no idea. You know, we've made some progress. Dario Amadei, as you say, himself has said that he thinks we understand maybe 3 % of what goes on inside of our neural networks. So before we're talking about, you know, the moral aspect of alignment or the technical aspect, we can't even read the code of our own AIs. So it is so scientifically laughable and like unbelievable to claim that we're in any dimension anywhere close to approaching a problem of this severity with anything that approaches like due diligence.

22:02We are not. We're completely flying by the seat of our pants. Could we develop this technology? I believe in the human spirit. I believe, you know, that we could do a lot of great things if we spend, you know, three generations of our greatest mathematicians, scientists and philosophers worked on this problem. I'm sure they can make great progress, but that's not what's happening right now.

22:21Danny Fortson: It's 200 folks in San Francisco. Yeah, optimistically. Which brings me to what's happening this week and this debate, this open vote versus closed debate. It's raging right now from where you sit. what is the best outcome between where it is right now where it's like open ai and anthropic at least by their actions they've kind of come out and said yeah we're for open weights and open source etc but they have these two very large fast growing businesses they're both barreling toward multi-trillion dollar ipos those are closed models that we all have to pay for on the other side is like a bunch of open source stuff coming from China and elsewhere, where it's like this stuff is free, but then you can't control it.

23:07And oh my god, it's like putting like the recipe to

23:11Danny Fortson: create a nuclear bomb in the hands of anybody who has a computer and can download it. What do you think actually, again, going back to the technical side of it, what makes the most sense in terms of regulation and not being squashed like little bugs by this thing that we're building? So yeah, as someone who's built open source AI, you know, was very much, you know, one of the first people kind of like really working very hard in this area. The way I see thing is, is that there's a reoccurring problem or state that a few people make in these debates, especially when it comes to like free market versus regulation, open versus closed, which is they pretend that you can only use one tool.

23:44Is that we have to solve everything using one tool. And this is just not true. Free markets are great. There's a lot of good things to be said about free markets. You know, should there be a free and liquid market for nuclear weapons? Probably not.

23:55Danny Fortson: I'm gonna say no. I'm gonna say no. Yeah, controversial opinion. Really controversial political statements we're making on this podcast here today. But I think probably not. Probably we should not have anyone in the whole world allowed to buy a nuclear weapon if they feel like it. So when we look at, like, say, open source, should Linux be open source? Yes, I think absolutely. I think Linux being open source is a great benefit to mankind. Should the blueprints for the F-16 fighter jet be open sourced? Probably not. Probably not. AI is a very, very broad set of technologies. If you ask me, should a dog detector be open source?

24:31I'm like, this is a very different question from should superintelligence or cyber weapons be open source? These are just very, very different questions. And I think it's a very misleading thing that a lot of industry lobbyists and so on try to do. It's quite equivocate between these things. Whenever someone like I says, hey, superintelligence is super dangerous, they'll say, oh, are you against AI that helps with cancer detection? And those are just not the same technology. The same way that, you know, the F-16 fighter jet and Linux are just not the same technology. They're different things. And we should have different regulations for these different things.

25:03So similar here, should, I don't know, some like, you know, image detection stuff be open source? Should some chatbot software be open source? Yeah, probably, probably. But should super intelligence or, you know, massive cyber weapons be open source? I mean, if you do, the consequences are predictable.

25:19Danny Fortson: Is this an all or nothing thing, though? Because I can ask Claude to help me write some emails. But if I ask it like, hey, can you, you know, help me figure out how to make like a biological weapon in my garage, it's just gonna say no. You know, like, is there a way to kind of put guardrails on it that actually are real? No, absolutely not. They can always be circumvented. If it's on your enemy's computer, it can be hacked. So what's the answer? What do we do? Fundamentally, we have to be realistic here, is that if there is a technology that is so dangerous that it threatens our lives, such as superintelligence, we cannot allow this technology to exist.

26:04We have to prohibit its development. So just stop. We have to, when it comes to superintelligence. Now, for other AI, there are many other regulations we need. We need child safety regulation. We need all kinds of form of privacy regulation. There's a bunch of other kind of regulation we need. I don't work on any of that kind of regulation. I don't know what the answers to those questions are. But when it comes to superintelligence, I can tell you that there is only one stable solution, which is don't build it. If we build superintelligence, it's too late. Superintelligence is not something you wield.

26:35It's not a tool. It's an adversary. It's a system that can act independently of you and that you cannot control. You don't use superintelligence to do things. Super intelligence uses you to do things. And it doesn't matter who builds it, whether it's the US, China, someone else.

26:50Danny Fortson: And how do we know when super intelligence is like when it is too late? That's the problem. We don't. We don't really know when the line is. The way I often think of that is not in terms of exactly super intelligence, but point of no return. The way most people expect this point of no return to happen is with what's called recursive self-improvement. what this means is is that we talked about this on the pod like last month because anthropic put out a whole post around the self because they're trying to do this and they're talking about it openly yes and this is just where intervention is necessary so hold on let's pause for a sec because this is where people are like this is just science fiction craziness so let's say tomorrow ta-da presto changer we have super intelligence why are you gonna die and why am i gonna die Because we get out-competed.

27:39It's not that we all fall over dead instantly or there's terminators in the street. That's how it happens. What happens is that now you can't trade on any markets anymore because AI out-competes you. You lose all your money the moment you trade any stocks. You can't do any businesses anymore because anytime you do a business, an AI-run business immediately out-competes you. You can't compete. You have no more political power because all political campaigns that are not run by AIs get destroyed by campaigns that are run by AIs. AI does mass persuasion at levels, you know, never seen before. You know, people have their votes, their things are completely, you know, manipulated by super intelligence at like unbelievable levels.

28:18New technologies and weapons get invented that you don't even understand. You don't even know how they work or where they came from. You know, all of our, you know, economic and infrastructure and cyber systems, everything gets run by AI and you don't even understand what's going on anymore. We don't even see necessarily what something is wrong. We just don't even understand. things get very confusing. We just don't really know what's going on anymore. Like you look at social media and you see a bunch of stuff, but is any of this real? Is it just the super intelligence is manipulating you? Like who knows?

28:50And then what I think happens is the super intelligences also start fighting each other because it won't just be one. There's not going to be like one super intelligence. There'll be millions or billions of them and they'll be fighting. They'll be competing. There'll be an evolutionary arms race. They'll be competing for resources, for money, for space, for energy, for everything. And the most ruthless of them that are most willing to fight dirty will be the ones, you know, that, you know, win those fights or that fight those fights.

29:16Danny Fortson: And in that kind of future, we're like ants. We're just collateral damage. Like we don't matter, basically. Exactly. AI will not hate us. You know, it's not the same way we don't hate ants, you know, if we're building a highway. But if we're building a highway and there's some ants there. Well, sex did the ants. You know, tough luck. I think that's how AI will treat us as well. Tough luck. So how do you view what's happening this week around this question, the competition with China? How do we quote unquote win or not lose? And this whole kind of regulation around like, who controls it? Do you think like, if we continue on this trajectory, we'll just look back at this moment and kind of laugh like, you guys thought you were controlling this thing, ha ha ha ha, isn't that so funny?

29:59I think we'll think that very briefly before the highway comes. But fundamentally, the thing here is that I think AI has many valid commercial and military implications. AI, for example, and Control AI are not anti-use of AI in general, even for military uses. We think this is very important for our competitiveness. We are rivals with China. This is a very, very serious thing. We do have to rise to this challenge to build a future that we endorse, what are our democratic values, the life that we think is worth living. And this is an important component of this. The thing to understand, again, is that superintelligence is different.

30:37You don't control superintelligence. Superintelligence controls you. The only winner of an AI race is AI. When it comes to superintelligence, it's no longer a commercial question. It's a national security question. It's a mutually asserted destruction question. What we need is a trust but verify regime where we can mutually technologically verify that others are not pursuing superintelligence the same way as with nuclear weapons. If you don't have a backed by credible deterrence, all the UN declarations in the world don't help anything. There has to be an understanding that there is an escalatory ladder that we are willing to use because fundamentally, if a superintelligence is built by anyone on earth, that is a grave threat to our lives and we should reserve the right to self-defense.

31:24Well, I've been having

31:25Danny Fortson: some work done in my backyard. I'm going to go check on the ants. I'm going to make sure they're they're okay. I'll make a little highway for them and tell the guys like, do not step there. This could be us.

31:42Danny Fortson: This episode of the Times Tech Podcast is sponsored by KPMG. We often talk about the shiny object problem with AI, where things look impressive in isolation, but are much harder to roll out across a large organization. Issues like governance, workforce readiness and trust tend to become more important as things scale. KPMG's approach is to look beyond AI as just another technology rollout and instead how it could redesign work across the organization and fit into wider processes and decision making. So it's less about individual use cases and more about how organizations can transform and deliver outcomes powered by AI.

32:22Danny Fortson: To find out more about how it can help your organization make the difference, you can visit kpmg.co.uk slash AI. Today's episode of the Times Tech Podcast is sponsored by IBM. Wimbledon may look timeless, grass courts, white kits, strawberries and cream, but the way people follow sport has changed. Fans now expect fast updates, highlights and useful stats wherever they are. IBM works with Wimbledon to help turn live match data into digital experiences, including match chat and the likelihood to win. It's AI in action in a setting everyone understands. To find out more, visit ibm.com slash Wimbledon.

33:03This episode of the Times Tech Podcast is sponsored by health and life insurer Vitality, your health's best friend. Vitality works differently. Get active, build healthy habits, and you can unlock rewards from some of the UK's top brands while helping to keep your premiums low. It's award-winning health and life insurance that helps you live healthier. Find out more at vitality.co.uk.

33:33So, what do you think, Mark? I mean, generally speaking, people like Connor, You know, three or four years ago, when they were talking, as they consistently have, about the dangers, you know, they went through a phase of being listened to and then they were considered cranks. Yet, slowly but surely, like a tick box exercise, a lot of the early predictions have come true. You know, there's this very famous AI 2027 kind of doomsday scenario that has an AI agent breaking out of its sandbox and hacking another, going into an autonomous hack. And that was predicted to happen early next year. So we're ahead of schedule.

34:18The end of that story, by the way, is the superintelligence building robots and wiping us out. But I don't quite know whether we'll get there. but you know it is quite interesting that the credibility of people like connor etc is growing every time we have something like an autonomous ai hack that everyone thought was science fiction and you've got sam altman saying that on a podcast i think today or yesterday saying you know this is

34:45Danny Fortson: sci-fi you know it's not anymore obviously yeah he famously said once a couple years ago that there's more regulation on making a hand sandwich than there is at building super intelligence which that's on its face is insane but that is effectively the reality right i mean there's also a lot more regulation in china on ai than there is you know probably anywhere else in the world even in europe obviously for different reasons than other regulators do but you know this idea of obviously we don't want to regulate because it will slow us down in the race against China. Well, they don't seem to be concerned about that, obviously.

35:25They have a very state-driven system, so it's understandable. But that's also, I always find that very interesting. Not a lot is paid attention to that, but they have things like compulsory AI watermarking, which everyone's been talking about here with regards to deepfakes. Right, right, right. It's never happened.

35:43Danny Fortson: Well, I think that's what's really interesting is just this notion of just stop. the notion that we can control a thing that is so much more powerful than us, that we are the ants to this, this other system is interesting, especially in the context of the stakes and everybody, you know, just it's like everybody's running this race as fast as they can, but they don't know where they're going. Right. Or where the finish line is, because that's the other thing is, I think there's an interesting like nuclear disarmament type of analogy here where everybody gets around the table and been like, okay, can we all agree that we should stop short of X or that?

36:23Danny Fortson: And I don't know how you define that point before the point, but it also feels like, sadly, that there's going to have to be something truly terrible that happens before we got to that point. You're right. There will be a point at which people are forced to the table, whether or not we will be under the control of the super intelligent ai by then i don't know but if you listen to connor it may be too late but but the notion that we should stop pause completely regulate is not so fanciful if you continue the line from an autonomous ai hacking to something a lot worse yeah no exactly to even say this stuff out loud feels a little crazy you know it feels like oh you're one of these you know you've got your tinfoil hat type of thing but to your point it's like the more things like these hacks happen kind of like checkpoints you cross you're kind of like well maybe they do have a point because for every sir jeffrey hinton where it's like guys this is a really really bad dangerous thing for humanity you have somebody like jan lacun who's like everybody set take a nice big glass of settled down juice this is just math and i think connor sort of addresses that and he's sort of saying this is these aren't next word predictors they're not large language models anymore they're a different being and i think that is a useful framing in when you approach it you know we've moved on all the industry has moved on and developed the technology into something slightly different now.

38:06Autonomous agents are a different ballgame and I think you have to approach them differently in terms of the way that you think about it and obviously take them very seriously. But certainly I can speak in the UK. This isn't even the 75th priority of this government or the general public. There are many, many more things in the queue. so it's very possible that this stuff can just happen almost under the radar and you can be much

38:38Danny Fortson: too slow well as i said maybe we should think twice about the ants in your in your back garden or whatever it may be because that's who knows soon we could be the ants to our ai overlords you're never going to be able to garden and no one's ever going to be able to garden in the In the same way again. Yes, indeed. Indeed. Well, that is it for this week's episode of the Times Tech Podcast. If you're enjoying the show, please drop us a line to let us know. Email us at techpodatthetimes.co.uk. And we will see you back here next week. Yes, indeed. We'll see you again because Katie, just total layabout, is off next week as well.

39:19Danny Fortson: So we have the pleasure of your company once again. So I look forward to seeing you then. Look forward to it. Bye-bye.

39:29Danny Fortson: This episode of the Times Tech Podcast is sponsored by health and life insurer Vitality, your health's best friend. Most of us want to be healthier, yet life so often gets in the way. Vitality's health and life insurance is built around that reality. Get active, look after yourself, and you can unlock rewards from some of the UK's top brands and help keep your insurance premiums low. It's insurance that works for you. By using tech and insight to understand your health, they can incentivize you to live better. The healthier you get, the more you are rewarded. Find out more at vitality.co.uk.

40:09Shipping, billing, admin, payroll, marketing. You're managing all the things. So why waste time sending important documents the old-fashioned way? Mail and ship when you want, how you want with Stamps.com. Print postage on demand 24-7 and schedule pickups from your office or home. Save up to 90 % with automated rate shopping. That's why over 1 million small businesses trust Stamps.com. Go to Stamps.com and use code PODCAST to try Stamps.com risk-free for 60 days.