272. Is Anthropic holding businesses to ransom?

0:00The prospect that AI could exterminate us, you take that very seriously, don't you? People over the millennia have understood technology is both exciting and frightening. Look at OpenAI. People knew he was a pathological liar. This was a fairly common line of analysis in Silicon Valley. OpenAI is losing money hand over fist, hundreds of billions on research. So it's losing money. I actually think it's going to go bust.

0:24Robert Peston:It's just quite a big claim which I can't let go by. Why do you think they're actually going to go bust? because...

0:39Robert Peston:Hello and welcome to The Rest is Money with me, Robert Peston, and I'm delighted today to be joined by Sebastian Malaby, who's written a huge number of books, often on finance, but his latest book is absolutely of the moment. It's called The Infinity Machine. Now, it says that it is about Demis Hassabis, who was the founder of DeepMind and arguably, if not the most influential player in the artificial intelligence world, certainly one of the two or three jockeying for that title. but actually it's about much more than that as a book. And the reason, you know, I might as well get my praise out of the way that I loved it is because it's filled with really important ideas about how this wave of AI technology is changing our economy, changing society, something that on this podcast we're completely obsessed with.

1:45Robert Peston:But it's also got an element of a thriller. because it's filled with, I mean, weirdos, let's be completely honest, the enormous brains of the digital technology AI world. These are extraordinary characters. I do want to drill down into many of the ideas, but I thought I'd sort of start, actually, with the biggest question of all, reading between the lines of your, as I say, gripping book. It was clear to me that you take this idea of what those in the industry call P-doom, which is the prospect that AI could exterminate us. You take that very seriously, don't you? I do take it seriously. I mean, for the first year or two of my research, I consoled myself with the idea that, look, machines will be more intelligent than humans, but they don't have an incentive to kill humans.

2:46They don't want to compete with humans. We have evolved over centuries. We want to pass on our DNA. We want to survive. Survival is our top objective. Machines, on the other hand, they don't evolve, right? And they don't particularly need to survive. So why would we feel threatened? And so I went to see Geoffrey Hinton, the professor in Toronto, who is one of the big P. Dumas. And unfortunately, he upset my equilibrium.

3:10Robert Peston:And we should just remind people, Geoffrey Hinton was the scientist who came up with the sort of set of algorithms that underpin so much AI. And he also has a Nobel Prize. So he's somebody we have to sort of take fairly seriously. Right, right. So there I am in Toronto, sitting in his kitchen for a couple of hours and debating him on whether he's being too gloomy. And I said, look, you know, Geoff, you've been working on this since the 80s. But I have to say, I don't see why they're going to attack us, these machines. And he says, okay, imagine you have a good AI. Imagine that there are rival AIs controlled by your enemies.

3:42Those rival AIs are going to try and destroy yours, hack yours, get into it. And so you're going to have to tell your own AI to defend itself. And the moment you tell it to defend itself, it's got a survival instinct. It's got a sense of pain. It's got a sense of winning, defending itself. And so don't tell me, Sebastian, that these machines won't have a survival instinct because they're going to.

4:02Robert Peston:Sebastian, we've just got to break off for a couple of minutes for a break but tons more to talk to you about we're proud to say that the rest is money is powered by octopus energy this year uh greg jackson is back to answer another question now this is something that we see a lot that i wanted to ask you about when you're building a business from scratch like what's the most important thing early on and what do people get wrong yeah look early on the most important thing is to start building a customer base until you've got customers you have no idea whether the products and services you're offering are going to be the ones that people want so a lot of people spend far too much time you know building out pages and pages of business plans thinking too hard about what they're going to do rather than just doing it what surprised you most when you were launching kraken you know the great fun of launching kraken whereas it was the engine that powered Octopus to grow ultimately to the biggest energy supplier in the UK with I think the best service but we were able to license it to our rivals as well.

5:04Robert Peston:So the interesting thing is the world is more complicated than you think and your competitors will work with you if it's in their interests. Cheers Greg. Right well you'll be hearing more from Greg throughout the year but now let's get on with this episode.

5:21Robert Peston:I'm assuming that actually, you know, as you progressed in the book, your view also changed about whether these machines would evolve because they are evolving. Yeah, well, they're improving. Is that not the same thing as evolving? It's a good point, particularly recently, because what's happening recently is the machines are coding new code into their own code. if you can get your mind around that. So we're almost at that point, what people call recursion, where basically we've got machines that know how to improve themselves. And that would feel to me like evolution. I agree. Now, even if we conceded evolution, we might say optimistically that why do they need to survive?

6:06I mean, I think there's a, you know, Kazuo Ishiguro's novel about machine intelligence ends with the robot, I'm forgetting her name, but she's sort of quietly dying perfectly happily in a car park when she's been abandoned at the end and seems totally relaxed about the whole thing and that vision of a machine not minding about its own sort of life extension was that was what was comforting me and that's what i slightly gave

6:33Robert Peston:up now one of the striking things about demis hasabis and he is the main character in your book, even if there are lots of other characters in it too, is he does take the risk of human extinction seriously. He doesn't dismiss it. And yet, you know, he goes on and on and on building a better version of AI. How does he justify that to himself? Well, I think there's two answers. One is, look, if he quit his job at running Google DeepMind, the whole engine room for Google, and said, I don't want to build this stuff anymore because it's dangerous. I'm going to take a professorship at Princeton. It wouldn't make the world safer because there'd be a bunch of other labs racing ahead.

7:21Somebody else would take his position. How is that helping? So that's one reason to continue. And then the other reason is that, and I thought about this a lot, but people over the millennia have understood that technology is both exciting and frightening. And yet we go ahead with it. We embrace it. And we do this because otherwise we would still be living in caves. So to some extent, Demis' continuation of his quest for superintelligence feels irresponsible because he admits it's dangerous. On the other hand, he's an enlarged version of you and me. What do you mean by that?

7:58Robert Peston:Sorry, I'm being a bit demon. We all have this impulse, right? That we would like to, you know, we're frightened of technology, but we're excited by it and we move ahead, we're worried, and we nonetheless move ahead. And that's what I'm saying, that he's doing the same. Yes, I mean, I guess, so yeah, look, of course that's right. I guess one of the things that is just, I mean, look, if I'm honest with you, I sort of get Demis Hussabis. There are certain things that he's able to do that I'm incredibly jealous of, But he is definitely very different from me in the sense that, you know, here is this individual who I think you say, you know, he comes from a pretty, I mean, certainly not a wealthy North London background.

8:48Robert Peston:Correct. His mom, I think, grew up in absolutely, you know, pretty extreme poverty in Singapore. Is that right? Right. And his dad is a Greek Cypriot sort of hippie-ish musician. Is that correct? Yeah, that's correct. And at the age of four, I think you say, he's watching his dad play chess and just sort of picks up the rules sort of by osmosis and within an incredibly short space of time is on the chess circuit, the chess competitive circuit. I mean, I'm not getting this wrong, am I? No, that's correct. And, you know, he was the best youth chess player in Britain, second best in the world. We're talking seriously competitive.

9:33Robert Peston:And I mean, one of my favourite early stories is how he gets identified really when he's still really little by a sort of chess guru. He's so young, he falls asleep at six o 'clock over the chessboard during the competition. So he doesn't manage to win that particular. But he gets identified by this guru as this great prodigy. And that's why his father goes from being sort of laid back music hippie to, you know, the great sponsor of this prodigy's chess career. 1980s version of a helicopter parent. so what so at what point and why does he move out of you know wanting to be chess world champion into I mean in his own words wanting to understand absolutely everything as it were he is he is a really paradoxical and intriguing character because on the one hand there's so much about him which is attractive and then he comes up with these statements like his great ambition in life is to understand everything and you just think my god you're a most incredible megalomaniac so which which is he well both i mean you know he so i i had this experience right from the beginning of spending time with him i spent more than 30 hours and it was just the most amazing ride and right at the beginning he said look before we have dinner i want you to read this book called ender's game which is a science fiction story about a teenage boy who is trained in a battle station in space and then he defends all of humanity against the invading space aliens and essentially saves humanity.

11:09So I'm reading this, I think, does he mean to suggest that he's also going to save all of humanity? Because that's quite a lot of humanity to be saving.

11:16Robert Peston:It's quite a big ambition. Yeah. So I go along, I have supper with him and he says, yeah, you have to understand this because this is how I see myself. And, you know, it's partly also just the sacrifice of this child who gives up everything to be excellent at space battles. And I'm doing the same with AI and that's my mission. And that's why I stay up to four o 'clock in the morning every day, because I have to solve the nature of reality. And by the way, Einstein, maybe not Einstein, but I think he said Newton and Richard Feynman, the physicist, they were failures. Failures, because they only understand some of physics, not the whole thing.

11:52So I need to do better than that. And I'm building AI, because then it gives me a tool, a new source of cognition with which to understand all of

12:00Robert Peston:reality and that's what we need it is extraordinary because he also talks about which you wouldn't think is being consistent with that and i don't know if this is true but he says that he puts a very high way on empathy and understanding people do you do you think he does understand people or do you think this is a slight illusion on his part well no so this is i mean look he's such a fascinating person but he is somebody who was brought up by this mother who had been an orphan in Singapore as you said who was quite religious who took him to church who made him pray in the evenings who made him do church-based philanthropy and to care about other people and it was a big rule you shouldn't control or dominate other people and he's very empathetic and you know in my experience with him he comes across as a normal person he relates unlike most computer geniuses.

12:54Robert Peston:So when you have these, even though he comes out, because the quotes in the book are some of them are jaw-dropping, even though he comes out with these really extraordinary, almost messianic statements, nonetheless, when you meet him, he comes across as a sort of normal bloke. Yeah, he comes across as a normal bloke. And also you sit back and you think, wait a minute, he just said he wants to be better than Isaac Newton. But actually, you know what? With AI, he probably will discover new stuff that Isaac Newton didn't know. So maybe he's just being objective. I mean, he's so extreme in his attainment that he's allowed to be a bit extreme in his self-description.

13:26Robert Peston:I mean, there are so many sort of amusing anecdotes related to his competitiveness. I mean, I find it absolutely hilarious that on the one hand, you know, I can't remember, there's some competition which I only actually ever came across through your book, which is like the international mega, you know, brainy games contest. You know, there's the World Cup of Brainy Games or something, which he's won. You'll have to tell us what the actual title of this thing is, but he's won it four times. Yeah, for actually the Mind Sports Olympiad five times in a row and he will remind you. But he also insists that he is the world's best table footballer, which is so bizarre as a combination.

14:06Robert Peston:Again, I mean, it might be true. It probably is true. But to insist on it, that's the interesting bit. And I said to him, yeah, whatever, you know, table football, who cares? And yeah, no, no, this is quite serious. I went and watched, I looked up on YouTube, Sebastian. You can now see the world championships and table football, the American players, because they have a professional league, and they don't know the snake shot. They don't know how to hook their arm underneath the lever and twist it over and get double speed on it. And I did. And that's why I was the best player in Cambridge. Anyway, we should, I suppose, get ourselves back to the stuff that is transforming our society.

14:45Robert Peston:Now got a bit of a sort of sense of the sort of anthropology of all these people who've been contributing to this extraordinary revolution. In terms of where we are with artificial intelligence, when for you, I mean, obviously, we could talk about Turing, we can talk about Ada Lovelace or whoever, but I'm just talking about this generation. When does it all start and why does it all start? Yeah, so in the late 2000s, around 2008, 2009, I think was really the foundation when two things began to work. The first was the early deep learning, and that's the Geoffrey Hinton stuff, where basically you train a model on lots of data and it can make sense of some data.

15:40And then another thing was the early experiments in reinforcement learning. And this was learning through experience, trial and error, playing a game, for example, trying a random move and seeing that. This is the David Silver stuff. Yeah, exactly. David Silver being the scientist at DeepMind who was a specialist in this. And both these fields showed early, early, early signs of waking up in the late 2000s. And then around 2012, both of them had a breakthrough. The first one was the ability to recognize pictures. and there was a competition called ImageNet and there was a paper colloquially known as the Cat's Paper and the AI could recognize a picture of a cat for the first time.

16:18It's amazing. I mean, you think about how short of a time it's been since 2012, right? 14 years. And we've come from, oh, wow, it can recognize a cat. Oh my goodness, it can like break every cyber code in the world. So that was the deep learning side. And then in the game playing reinforcement learning side, DeepMind, Demis' company, produced a system called DQN, which played Atari games. And if you remember Pong or Breakout or Sequest or one of these ones. And essentially the computer could play at a human or better than human level at all these different Atari games.

16:56Robert Peston:And so that's the initial breakthrough. through, are we already then, because if we look at it today, there is definitely a race on, right? You know, we've obviously got DeepMind and Google, and we've got OpenAI, and we've got Anthropic, and, you know, in China, we've got, you know, a bunch of competitors as well, and DeepSeek is the one that most people have heard of. Is it, does the race begin there? So I think round one of the race you can identify as being 2013, going into early 2014, because there was a bidding war to buy DeepMind. You know, Google was the one who bought it, but actually Meta, which was then Facebook, wanted to buy it.

17:47And then Elon Musk wanted to buy it. And so three of the biggies wanted to get Demis and Demis chose one of them. And then, you know. Can I just ask you on that?

17:56Robert Peston:Because it's something we've obviously talked a lot about on this podcast, which is, in a way, what a sort of national tragedy it is for the UK that DeepMind and Demis felt that they could not finance their ambitions in the UK. Was he right that in order to achieve what he felt he had to achieve, there was no route to remaining British. I think that's true. There wasn't an alternative because you needed to be acquired by a company with A, deep pockets, and B, a ton of computing power. And I would just - Can I ask you, so I'm sure that's right, that that wasn't available then. So the obvious follow-up question is, if the equivalent of DeepMind were facing the same conundrum, the same quandary today, would it be possible for them to remain British today?

19:03Robert Peston:Because I'm not sure they necessarily could. Well, I'll make two points. So one of them is, yes, today, American venture capital, or we should just say global venture capital, A, has more money than it used to, and B, is quite happy to show up in Britain and back a company like, say, Wise, formerly Transverse, or other British startups. And so you can raise a lot of capital and stay in the UK. So let's just on that because we might come back to the significance of this later. So the fact that, so for example, we've already talked about reinforcement learning and David Silva. He's just created this business called Ineffable Thinking.

19:46Robert Peston:And we might talk about actually the significance of his technology versus the sort of Hassabis Gemini approach, because I think there's a very interesting sort of new battle going on for what the future of AI will be. But he seems to be raising an enormous amount of money, but it's all coming out of Sequoia and America. You don't think that matters where the money comes from. That doesn't make his business, in a sense, an American business if it's venture money. You're not worried about the source of the funding. Well, he'll still be the CEO. He'll be in control. So it is a bit different. But I would make a bigger point, which is that I think, from a British patriotic point of view, what we should care about is having great science companies headquartered in Britain.

20:36Robert Peston:Don't you want the profits staying here and the tax being paid here? Let's put it like this. When Demis sold his company in 2014 to Google, over the next 10 years, Google poured in nearly a billion dollars per year of research and development funding. Into London? Into London, yeah, because all of DeepMind's operations were in London. And so we could regard this as a victory for Britain that it persuaded this American group, Google, to just rain down research funding onto Britain. It's great, right? And in fact, they were quite hands-off about the management of DeepMind. They didn't intervene in what Demis wanted to do very much.

21:17And so I think that's great because a lab of brilliant people remained in London. I suppose so.

21:24Robert Peston:There's still an opportunity cost in the sense that you don't get the profit flows that generate the tax revenues and all the rest of it that would have been British had there been a way of it. I mean, look, I absolutely take the view I think it was perfectly rational for him to do what he did when he sold more than 10 years ago. But I still think there was a cost to the UK. But there were no profits. That's the thing. You see, there was just money coming in for research. There are today. Massive. No, actually, look at OpenAI as your thought experiment. OpenAI is losing money hand over fist, right?

21:59It's spending hundreds of billions on research and the revenues are tiny compared to the expenditures. So it's losing money. I actually think it's going to go bust. So even in Silicon Valley, it may not be very easy to create an independent, freestanding AI companies. So had DeepMind tried to do that, would the taxpayer in Britain have had any relief because DeepMind was paying taxes on its profits? I'm not sure.

22:25Robert Peston:You just made quite a big claim, which I can't let go by. Why do you think OpenAI is going to go? I mean, obviously, they've had to scale back some of their financing recently, but why do you think they're actually going to go bust? Because they produce forecasts for their clients, for their investors. and these are always leaked. And if you look at the leaked forecasts, their own estimates have been that they're going to burn$660 billion before they start making a profit. So 600, let's just say that number again. Yeah, it's a good point. $660 billion with a B. I mean, this is insanely big. They're going to lose that amount of money.

23:03That's how much they're going to burn. Yeah, lose that amount of money. Negative cash flow. Negative cash flow because their revenues are paltry. I mean, they're growing, but from a very low base, they've got almost a billion users, but only 5 % pay them money. Nearly all of them pay nothing. And so, you know, they can probably scale back some of that 600 billion of intended investment. And that's why they've cut their project in Britain, the Stargate project. That's why they've cancelled their whole video generation thing. So they're going to be having to tighten their belts.

23:35Robert Peston:You will have read that very long New Yorker character destruction of Sam Altman where, putting it in crude terms, over thousands of words, it basically says he's a pathological liar. Correct. There's not a lot more to be said about it. Actually, no, they would say a power mad pathological liar is I think what they would say. And a lot of the testimony came from this bloke, Ilya Suskova, who worked with him for years, again, a very senior person within this industry. I mean, do you think that will have an impact on the confidence of investors? And to go back to your question about the challenges that OpenAI faces, what did you make of that article?

24:26Well, look, I mean, it was in some ways a very good assembly of stuff that people knew already. I mean, people knew he was a pathological liar. This was a fairly common line of analysis in Silicon Valley. I would say that, you know, what's going on with OpenAI is that they've raised a record amount of money, the most in the history of capitalism, as a private company. that game is now up and you can see this in the secondary market for their stock you know when when you have private markets it's not like there's no stock exchange you can't trade it you can't see the price but people do try and trade stock you know kind of one-to-one and there are these you know exchanges that do that and if you look on those you see that the secondary market price of open ai shares is a lot below the announced official price so in other words so

25:16Robert Peston:So the implied funding round price is way above where it's actually trading in shadow form. Correct. Whereas with Anthropic, its rival is the other way around. Anthropic secondary shares are above the official valuation. So that means that the insiders in Silicon Valley, the kind of private investors, the venture capitalists and so forth, know perfectly well that OpenAI is in deep trouble. So OpenAI's escape hatch was going to be to go public and start tapping the retail investors who don't know so much. And if you want to tap the retail investors, it's not helpful if the New Yorker publishes thousands of words on how the CEO is a liar, because that will percolate out into the public consciousness and make it harder to go public.