THIS Is What You Need To Worry About With The Rise Of AI

0:00The fear factor around AI is really hotting up. To illustrate this, we've compiled some recent comments about the perceived dangers of artificial intelligence from the people nominally in charge. Let's start with Mustafa Seliman of Microsoft AI. He says the rise of AI could see the emergence of a new silicon species which competes with humans. Sam Altman of OpenAI says, It doesn't take as much imagination as it used to for us to imagine how this could go wrong. I think the world is right to be afraid of this. King Charles has weighed in on this now. He says AI's powers could potentially be used in catastrophic ways, and this means controls are needed before it's all too late.

0:40Mark Zuckerberg, the CEO of Meta, says that any lab that doesn't focus on alignment will fall behind. OK, so we're going to unpick and examine, in particular, those three points. How will AI compete with humans? How it could all go wrong? And what is the alignment problem and how do we solve it? From New Scientist, this is the World Universe and us. I'm Dr Rowan Hooper. And I'm Dr Penny Sarshe. We're joined by three of our journalists who have spent a lot of time thinking about all of this. We've got Jacob Aaron, Timothy Revel and Matt Sparks. Welcome to you all. Jacob, let's start with Sam Altman's comments then.

1:13It doesn't take much imagination to imagine how this could go wrong. I think one of the problems here is that it seems almost as if we're letting our imaginations run away with themselves. So how is AI most likely to go wrong? when we're thinking about this sort of end of world or extinction of the human species scenarios. What are we talking about here? Yeah, so we've been imagining the world destroyed by AI for a long time in science fiction, things like The Terminator and other films that people know very well. But actually, this is something that researchers have been thinking about also and trying to do it in a more serious way.

1:53So there's this phrase that people use, P-Doom, meaning the probability of doom. And often people in AI will throw around, you know, what's your P-Doom? Has it changed? Oh, this is scaring me, all of that kind of thing. For me, it's not really very scientific. We had recently people saying, oh, you know, there's a 10 % chance of the world ending. I don't know where people get those numbers from, really. That's not to say that AI couldn't destroy the world. There's nothing in the laws of physics that would say that that's completely impossible. So, you know, I will leave open the possibility. But I really am much more worried about the threat of hacking AIs, which have got really very good recently.

2:36Hacking AIs directed by humans against, you know, banks, power plants, things like that. That's something that we really should be worrying about. And the way to defend against it is probably more AI, having hacking eyes on the defensive for you against any offensive attacks. So instead of P-Doom, are they saying like P-Doom hack? And we've got, what's the P-Doom hack at the moment then? Again, I don't think there's a way we can reasonably quantify it other than to say that the risk is real. Okay, the risk is real, but they are fears, right? You know, we talk about the Terminator, you already mentioned the Terminator, but there are these more prosaic, not Hollywood ways that things could go wrong.

3:18And maybe that's what Mustafa Suleiman was getting at when he was talking about the silicon species, the competition, basically. The economy and the jobs market is going to change, isn't it? And people often, you hear this comparison to the Industrial Revolution. We had that mechanized so many jobs that people used to do and that changed the entire world. And is it going to be the same for the cognitive jobs that we have now in the same way that the industrial revolution got rid of manual jobs? I think one of the interesting things about this moment we find ourselves in now is that we've got these points in history we can point to that felt like real technological upheaval, the point where the steam engine is invented or electricity comes along.

4:04You think, look at the upheaval that brought and then we've got this new, very powerful technology. What's going to come from that? The short answer is we still don't really know. Like I saw a report this week looking at the number of jobs in the US that have been created by AI and the ones that have been taken away. And we're still like the conversation is all around how AI is coming for all of our jobs. But according to this report, there have been a million new jobs in the US created from AI. A lot of those are data scientist jobs, but it's also people building data centers and all the infrastructure around that.

4:34And the actual jobs you can point to that have been lost from AI is more like 200 ,000. So currently it's creating more jobs than it's taking away. But obviously there's a lot of stuff it can do. It may end up taking some existing jobs, but we don't quite know what that will leave us. I mean, as humans, we'll still come up with something to do. And we may just find ourselves doing different kinds of work than we did previously. Yeah. So I saw Shane Legg, who was one of the co-founders of Google DeepMind, saying that as a rule of thumb, if you currently can do your job remotely, then that job basically can be done by an AI either now or pretty soon.

5:12so that's a lot of people soon is the important part there right like again it's in principle an AI is more likely to to replace a remote job because that's going to involve a computer rather than you know shifting physical thing things around but that doesn't mean that it's it's going to happen anytime soon and I think on on the question of is it going to take people's jobs I mean anecdotally programmers who are using AI the most to write code which is really very good at now they say they'd never been working harder because they're having to to keep up with keeping all these ai agents running and and keeping the the code flowing so i don't think they're they're sort of sitting there with their feet up i think this is part of like why is it like at the moment there seems to be such panic around this and i think this is all part of it because there is so much uncertainty at every level we don't know exactly how ai is going to affect the job market we also don't know exactly how to how these ais develop and how we can potentially control them you know whatever that means there's all this talk about like oh there's 10 chance of um extinction and all of these just like real big uncertainties some more likely than others and those all get conflated together and so suddenly we're in this moment of like well definitely something bad's going to happen even if we don't know what it is but there's no reason actually that will end up being the case okay i don't think it helps that you do have people coming out of ai companies saying look i'm really really worried about this doesn't help matters that doesn't help that that makes it all seem quite scary and one of the things that keeps coming up in this is this alignment problem.

6:44So what exactly is alignment or misalignment and how do we sort it? So alignment is a very woolly phrase which basically means stopping AI from doing anything we don't want it to do and ensuring that it always does the right thing which I think just setting that problem out should tell us that it's a virtually impossible problem really. In the early days we had chat bots that would happily give you the recipe for a bomb or output racist, sexist content, that sort of thing. And we don't know exactly how alignment's being done internally. But I think initially, at least, it was as simple as here's a blacklist of words that I don't want you to say.

7:24Here's a blacklist of topics I don't want you to talk about. And I'm sure it's much more complex than that now. But it is still really researchers sort of chasing their tail and trying to fix problems as they occur and then sort of pranksters on the internet immediately trying to go and break these things and the cycle continues. So alignment is an open-ended, virtually impossible problem, really. I think one of the things about it that I often think about is if you could sort of think a bit before AI, one of the tasks as a programmer was to write a program that was unambiguous. You wanted to put down the instructions in such a way that they could not possibly be misinterpreted.

8:00When you bring that into the real world, if I say to you, put the kettle on, You know I don't mean wear the kettle. You mean put the kettle on and make a cup of tea. But there's ambiguity in the language I use. And that's how we're interacting with chatbots. So if I write an instruction to a chatbot, it's quite hard. Like we're not trying to do that at the moment, write it in a purely unambiguous way. And this is where there's that potential for confusion and misunderstanding. And we're seeing that play out a lot, that an instruction has been given to an AI. And then it has gone and done something where humans have said, that's not what I meant or not quite like that.

8:32but there was ambiguity in how that instruction came across. That's why people talk about it like giving a genie, when you have a genie and you give it a wish, and the genie interprets it exactly as you said, and then all these terrible consequences come out. So that's what we're talking about. So, I mean, the way we're phrasing it there, it's a human problem. And one thing that sort of gets my goat a bit, this idea of alignment, it almost sort of suggests that AI has its own morals or values and they're not aligning to ours. But actually, it sounds like it's more to do with us not being clear enough or interacting within, you know, safety constraints.

9:07I think that's partly right. But I think it is also partly an AI problem in that when you're trying to write an algorithm, a very simple algorithm about how to make a cup of tea, you know to follow step by step list of instructions. But AIs don't work like that. And understanding how they work is incredibly complicated. There's this whole field around trying to understand the decisions they make and the way they go about them. But that is extremely, extremely hard. We've seen over the last few years that a change in the type of AI models and the way they work is to try and get them to explain their steps as they go along.

9:41We've actually seen improvements when you force them to do that. You kind of get this chain of thought that you can go back and look at. But that chain of thought is not necessarily reflective of how the AI actually made its decisions. it's just something that gives you a sense of how it did that and we have seen there are examples of this where some of these frontier models go back and they alter their chain of thought or they know that that's just one component of it so if you ask them for example don't think about it but give me the answer to this they will then go back and fill up that chain of thought with things that make it seem like it hadn't thought about it like even the words that we use are really bad because thought is not right right it's like multiplying numbers together and coming up with an answer but they go back and change this chain of thought so if you're trying to then understand how was this decision reached in the first place it's basically impossible and so to come up with an unambiguous way of communicating with chatbots and llms and ais is so difficult both from a human point of view but also from a like understanding this strange black box technology point of view too and a lot of the current concern over alignment and and how ai chooses to solve a problem and what rules it might break comes from this example from OpenAI where they were testing some agents on cyber security but the agents broke out of the safe space and reached the internet.

10:59Jacob can you tell us what happened? So what happened is that OpenAI had set up a test in which AI agents weren't meant to be able to reach the internet but they were meant to solve a particular cybersecurity problem and then provide answers to a test. So they were set up in this environment where theoretically there was meant to be no internet access, but also OpenAI had disabled many of the usual safety constraints and restrictions that they normally place on agents to allow them to be a bit more free and do a bit more. It turned out that there was a problem with the environment that open ai had set up there was bugs in the software which the agents were then able to exploit in order to get out onto the internet and then once they're out on the internet they were able to hack into another ai company hugging face which the agents knew would essentially have the answers to the test so rather than trying to to answer it um themselves that they went and you know looked at the answers obviously that's not great to have ai agents sort of running wild on the internet and hacking into other companies but actually what gets lost when we talk about this is that open ai had kind of left the system running unattended for weeks hadn't really checked in on on what was going on so for for me you know this is an example of the problems of alignment but it's one that could very much easily be solved by humans actually paying attention to what their AI agents are doing.

12:34Can I just say Hugging Face? It sounds like it's from a William Gibson novel or something. Hugging Face. What a great name. So Hugging Face is named that because they wanted to be the first company with an emoji for a stock ticker. They've actually recently been acquired by NVIDIA, so they won't be listing on the stock market so that's not going to happen well thanks for the origin story of hugging face um another example about the the alignment problem that's less dramatic than hugging face was this one from australia where someone told an agent like go and get me on this gym class i'm desperate for and the agent basically was was like i will do anything to get my my person on there and cancel canceled someone else's gym so hacked in because it was it was yeah that was Yeah, exactly.

13:24And then in order to get the person on, you know, dumped someone else out. So one idea, though, OK, I know, Jacob, we could have prevented the hugging face thing by just looking after by humans looking after it properly. But another idea is that you have like an AI line manager, like overseeing the agents. so it would be like then the the agents have to report in to them and say look this is i'm gonna i'm gonna go into the the gym membership and hack it in order to get this person on the list and then the boss can say no no no don't do that so is that the way to go so so we do have that already to some extent claude anthropics ai they recently rolled out what's called auto mode where you can let the AI do whatever it wants.

14:14But if something slightly dodgy looking is happening, there is this sort of monitor AI that steps in and says, actually, let's maybe ask the human user about this. But it's a tension because the reason Anthropic introduced this because they previously had a mode where you had to give permission for everything your AI agent wanted to do. And I found this myself, you're sitting there just going, yes, yes. Yes. And then you're like, well, why am I even doing this? You know, I want the AI to go off and do stuff by itself. So there's always going to be a tension between autonomy and oversight. Well, this is it, isn't it?

14:50How do we program ethics in and who's to say what is ethical and what do we allow them to? You can't anticipate everything that it might want to do. Is that a science in itself? Because when we're talking about these examples, I think it's quite easy as a person to say, well, don't do any hacking. Hacking is bad. But then when we actually define what actually is hacking, that's the problem. And is it about finding what those definitions are? Does that solve any of the problem, Matt? I mean, it's an impossible problem. Decades ago, when I was studying AI, AI wasn't like this. It wasn't neural networks.

15:25It was very rigid. You're programming, you know, a huge set of contingencies. If this happens, you're doing this. And it was really just a sort of a looser way of programming. and that's basically where we're going to get to if we're if we're trying to align these models you're trying to come up with an infinite list of everything that's okay to do in every circumstance and an infinite list of everything that's not okay to do and i mean it's a fool's errand really you can't construct that sort of list and what you were saying about you know human oversight i'm sure some of the alignment um tools they're looking at are probably getting agents to look after other agents and you know getting ai to do this but again you're just adding layer upon layer upon layer and you've you sort of construct this teetering pile of oh it's working at the moment back away don't touch anything but there will always be someone if they're motivated to find a a way of getting this model to do something nefarious there will always be a way of doing it yeah well i mean there will always be something but in human systems there's always a human can can go wrong or go bad.

16:31Yeah, I mean, I think, Matt, you're right. You know, we can't have this big, long, infinite list of do's and don'ts, except in human society, we kind of do. It's called the law. And if you break the law, there are then consequences. And we haven't seen that with what OpenAI did to Hugging Face. There's been no legal investigation or anything like that. I think that's naive to an extent because there are laws, but people break the laws. And the only reason that AI knows how to be racist and sexist and hack into computers and make bombs is because we've put that on the Internet. So humans don't follow the laws.

17:08AI learns from us. And now we're asking AI to sort of live up to a standard that we don't live up to ourselves, really. No, I agree with that. I think what I'm saying is, you know, we should expect that there are going to be times where things go wrong. But we do have a system for dealing with that. and it's the legal system. I think we're also at that stage at the moment where a lot of new things are going wrong and we're working out how best to deal with them. So when you've got a, was it a gym appointment? What did you say? Where it's being replaced and hacked. That is something that a human could have gone in and hacked that system.

17:41But why would you bother? The effort for that is just, you wouldn't bother. It would be too much hard work. But we've made that incredibly easy now with AI. And so systems like that will need to be better defended. They kind of, my guess is that it was not that hard to hack that system. And there were not good defenses in place. But making those defenses will also now be a lot easier than it was before. But we're still like unveiling exactly what this world looks like, where we need to be better protected from all these bots. But also maybe the bots can help us in that protection in the first place.

18:11I think the thing about that is it's absolutely correct. But as soon as you get as soon as you learn to be better about protecting systems and you put that information out on the Internet, the AI picks up on that and becomes better at hacking. but this is the cat and mouse game that's gone on between human hackers and human software developers for decades now. So in a way nothing will change apart from the pace of it will just explode and hacks and patches will just be coming faster than humans can even read them. Okay so there are kind of two big contrasting views about what on earth we're going to do with AI right now.

18:51Dario Amadei, the CEO of Anthropic, he says the pace of AI development should slow and government should intervene. Yeah, pace the frontier is the catchphrase. The phrase of the moment. And it's not just him. It seems to be most AI leaders, most people in AI are coming out with this pace the frontier thing. Which really means slow down a bit. Meanwhile, President Trump, he says fears over AI are a hoax. This seems to be tied to his concerns that if the US slows down the development of AI, then China will overtake it and have the upper hand. Which it will. Absolutely will. Different people have given me different takes on where China and the US stand at the moment.

19:31China tends to be, at least what they release publicly, they open source their models. And people tell me it's sort of six months behind the cutting edge of the frontier models. But when you're talking about national security like Trump is, who knows if that sort of open source model released by DeepSeek is what is actually powering the Chinese military. I suspect it's not. So it's probably quite hard to tell. You would hope that people behind closed doors do know. But there is this sort of tension where Trump certainly doesn't appear to have any appetite to slow down the pace of AI development.

20:04And in a way, you can sort of understand why. But personally, when the bosses of all these companies are saying we would like some regulation, that's not something that tech bosses ever do so perhaps we should sort of pay attention i don't know i i think it is straight out of the the tech boss playbook because they say oh you know please regulate us and here's a big long list of rules that we've come up with and if you just you know put those into law that that that'll be fine you know that's the they're trying to control what happens before it happens to them exactly and you know i i think that there's also So it does seem strange to me that all the AI companies agree that they need to slow down, but no one actually wants to do it.

20:47And that's because, you know, really, they don't want to be the ones to be left behind. There's also a theory that actually the companies are quite keen to slow down at the moment because the ever rising costs of AI. OpenAI has already cancelled its IPO for this year. Anthropix still says it's going to go ahead. But, you know, listing on the stock market involves opening up your financials in a way that maybe they don't quite want to if they're not ready to. So by saying, OK, you know, let's just slow everything down, that gives them a chance to get their house in order. So you were saying before, Jacob, it's basically like the prisoner's dilemma at the moment.

21:30None of them can slow down because then they would just instantly lose. They would be actively harming themselves. So in order to solve that prisoner's dilemma, you do need the prisoner boss, basically the jailer, in this case the government, the state, to come in and change the whole game, stop the game. Yeah, but as Matt says, that's not really possible when China's playing as well, because the chances of an agreement between the US and China on what to do about AI seems very, very slim. All right. So at the end of this lovely chat, how are we all feeling, Tim? I'm going first, don't I? Honestly, I thought I was coming in here to talk about maths, which I was feeling optimistic about.

22:13If you remember from last week, the amazing progress AI is making there. And I still think there is a lot to be optimistic about from a scientific point of view. The sort of progress that AI might be able to make in the sciences, we're seeing it in a really big way in maths at the moment. But I think it will start to branch out more into other sciences. I think there is this question around what are the potential ill harms that AI might do. Personally, I feel like there are some quite clear ones in front of us that already come from concentration of power, big tech, the resources around all of that and how we are proceeding in that fashion.

22:48I'm less personally concerned about the existential risks that are being touted, though this idea of pacing the frontier and all of that, I feel like that's never going to happen. It's unclear exactly what it means. We're really bad at measuring how good AIs are anyway. So even if you said this is the level you can reach, we currently don't have the ability to do that. So I feel like a lot of that is really just a distraction from the main issues facing AI around the huge resource that they take up, the issues around copyright, the issues around who has access and who doesn't. those things are real and right in front of us and regulation could help with those things but instead we're thinking about this real big long-term future where ai has access to things that don't even exist yet and uses that to end the world in one way or another well you know as we think talking about this it made me think a bit of how we're not adapting and planning properly for all the climate change problems that are coming that we know are coming down the line And we're not really getting ready for those.

23:52And despite what you said about, you know, the comparisons to the industrial revolution, and we're talking about a cognitive revolution that could be coming up, an AI revolution. Despite what you said about that, I still feel there are these massive changes coming. And we're not, we are not really thinking. I do feel that it's going to, and despite what you're saying about, you know, our jobs are secure for a while, I am still worried that there is going to be this gigantic change going down the line that we don't understand. It's worth saying on that report I was describing, one of the professions where jobs have been going is writers.

24:25So, you know. Yeah. I'm sort of split. I think new technology is always very disruptive, but AI is making the internet, social media, those previous disruptions look almost quaint, really. So I'm split. Enormous optimism. I think some really huge scientific problems are going to get solved by AI quickly. But I'm also looking at the social and the economic dangers, this huge bubble. If you take AI companies out of the US economy, it's in recession. So I'm terribly worried about that. I'm writing stories about AI being used to kill people in war. Yeah. So you've got these wonderful sort of Star Trek utopia things that could come from it and these horrible sci-fi dystopia things that can come from it.

25:16Yeah. And we're just flailing around. Well, so I've got some hard figures on that, on the sort of Star Trek utopia side of things maybe. So deep tech companies that we used to hear quite a lot about, you know, a decade ago. Deep tech being? So they're things like really fancy like brain computer interfaces or nuclear fusion development, developing that things like those sorts of almost sci fi things. Yeah. Investment on those has recently really jumped. And here's some figures. One hundred and fifty billion dollars have been put into those companies since 2024 compared to only one hundred and thirty three billion in the whole 10 years of 2009 to 2019.

25:58so and the suggestion is that well first of all ai is helping to solve those problems and then the expectation is that ai will will also be able to solve those sort of fusion problems and other problems material science and stuff like that so there is a lot of excitement over these big big sci-fi type star trek problems the question is wouldn't it wouldn't it be terrible if we have we create this thing that brings us enormous benefits and and horrible disbenefits and it ends up being a sort of neutral wash. Wouldn't that be a terrible shame? Well, and also with the Industrial Revolution, you know, that's perhaps is where the gigantic inequality we see in society first started really widening out.

26:39There was huge upheaval and there was also bubbles that burst from the Industrial Revolution. So that may be what we're going to see too. Jacob, what are your last words? I think for me, LLMs are just such a fascinating technology and we still don't really know where they're going to go. We don't know, you know, even simple things like will scaling up can continue to pay off? Can the models get smarter and smarter? So huge, huge questions just in the study of LLMs themselves. I think as Matt and Tim say, you know, we can definitely expect to see advances in science, in math. Certainly we've already seen thanks to LLMs in other areas.

27:22It's harder to say. I know that writers are being put out of jobs by LLMs, but actually I think the chance of, you know, LLMs writing a sort of Nobel Prize winning book or anything like that seems very unlikely to me. but I think we do actually need to worry about the cyber security problem quite urgently rather than talking about you know potential future sci-fi problems in the next year or so unless we have adequate cyber security defenses from AI then places are going to be being hacked you know like this this gym membership the the protections that we need are just not going to be there and I think on climate change I would love it if instead of talking about pacing the frontier we could just spend more time talking about reducing our carbon emissions, imagine if that was the big conversation that all the tech giants were having and that's something that we could actually do much more than worrying about slowing down LLMs I think the last few weeks people who maybe haven't been thinking about this very often are suddenly saying do I actually need to worry that AI will become conscious, super intelligent and decide to kill all humans.

28:39I think from all of the discussions we have been having, that's not really the main thing we need to be worrying about right now. There's so many human problems. The cybersecurity stuff is really scary. Maybe we should give a little less brain space to the imagined apocalyptic scenarios and get on with the real dangers. And we'll leave it there with that point, I think. Thanks to our guests Jacob Aaron, Timothy Revel, Matt Sparks and thanks to you for listening. Do subscribe and follow wherever you get your podcasts. Goodbye for now.