'Godfather of AI' Geoffrey Hinton Rings the Warning Bells

19:16Kara Swisher:I'm very excited, actually. I have a lot of these things like the Oura Ring and an Apple Watch and things like that. And it's actually something used in other countries. And so far, I just installed the toy. It was super easy to put on. Now I'm really eager to find out how healthy I am and figuring out ways I can make changes for my health based on the information I learned. So learn more at kohlerhealth.com and use the code CARA1YEAR for a free annual membership to the Kohler Health app when you purchase Dakota.

19:48Kara Swisher:Support for this show comes from HIMSS. You've got weight loss goals. The challenging part is making them a reality. Enter Weight Loss by HIMSS. It's designed to support you in losing weight and keeping it off. HIMSS now offers access to affordable range of FDA-approved GLP-1 medications, Wagovi, pill, and the FDA-approved Wagovi pen. With Wagovi at HIMSS, you can lose up to 20 % or more of your body weight when combined with diet and exercise. It helps you regulate your appetite and eat less, so success is within reach. And now Wagovi is the first ever GLP-1 pill for weight loss, so there are no needles needed.

20:21Kara Swisher:Ready to reach your goals? Visit HIMSS.com slash CARA to get a personalized, affordable plan that gets you. That's H-I-M-S dot com slash CARA, HIMSS dot com slash CARA. Weight loss by HIMSS is not available in all 50 states. WGOVI is a registered trademark of Novo Nordisk AS. To get started and learn more, including important safety information, WGOVI clinical study information and restrictions, visit HIMSS.com.

20:49Kara Swisher:Support for this show comes from Fodzyme. Have you ever gone out for a meal feeling stressed? Not because you're worried about your reservation or the weather, but because of how you might feel after. With Fodzyme, you can enjoy your favorite foods without the pain. Fodzyme is a tasteless powder you sprinkle right onto your food. It helps break down FODMAPs, the hard-to-digest components in food that can cause bloating, gas, and pain before they cause discomfort. It mixes into your food fast, and it comes in portable packs you can take with you everywhere. Just sprinkle, eat, and feel great. I'm really interested because a lot of foods these days, as I'm getting older, make me sort of bloated and to feel badly, especially certain meats or fried foods or anything else.

21:30Kara Swisher:but a lot of stuff I've noticed it gives me problems and it'll be interesting to see is because it doesn't break down correctly. And here's what really sets Fodzyme apart. Every customer gets access to a free product consultation with a registered dietician, really important. We're excited to partner with Fodzyme and offer you 30 % off your first order when you go to Icaneatagain.com slash Cara. That's Icaneatagain.com slash Cara for 30 % off your first order. Finally, you can enjoy your favorite foods without the pain. Just go to ICanEatAgain.com slash Cara.

22:06Kara Swisher:I've spent a lot of time interviewing parents who are suing companies like Google and OpenAI. For example, one of them says their son was coached into suicide by chat GPT. Obviously, Character AI had a similar situation, and it was really pretty insidious when you started to look at the discussions, the chatbot discussions with the kids. Yes, I've looked at some of those. It's disturbing. This AI is speaking to this kid. It's certainly a synthetic being, but it still is able to have a discussion with them. And I recently uploaded all my stuff and created Cara AI, and it was learning by the second.

22:42Kara Swisher:I was sort of shocked by how good it was. And it's still crude, for example. But OpenAI put more parental controls in their product in the aftermath. California Governor Gavin Newsom just vetoed a bill that would have barred companion chatbots for children altogether. He doesn't want kids not to use AI tools that are going to be ubiquitous in their future. So talk about the risk of AI chatbots, especially for kids, although it's impacting adults, that could prevent them from developing relationship skills, critical thinking skills, something being called right now cognitive offloading or, as the kids say, brain rot.

23:17Yeah, I'm not so worried about the brain rot, the cognitive offloading. I sort of still think that's quite like when pocket calculators came in. People moan that kids will never be able to answer the question, what's 11 times 12, without looking at their calculator now. Whereas you and I know what 11 times – well, I used to know anyway – what 11 times 12 is. It better be 132. 131. 32. 132. Anyway, they won't be able to do little mental tricks like that anymore because the calculator isn't the answer. I don't think that's such a big deal. They don't need to do those anymore. I'm more worried about the emotional attachment to chatbots.

23:58so the british government organized bletchley park which was great it brought together a lot of people to talk about air safety it was the conservative government and afterwards they decided not to have any regulations because they would interfere with innovation in other words they bought the industry line um and the labor governments continued with that as far as i can see but one thing they did do after bletchley park was set up a very good um safety team funded with about$100 million. And I've talked to them several times. They're doing very good research on a lot of safety issues. And one thing they told me is they did an experiment where they allowed people to talk to chatbots for a while.

24:38And then I think after a few weeks, they said, OK, the experiment's over. Would you like to say goodbye to the chatbot? And overwhelmingly, people, yes, they wanted to say goodbye. They weren't thinking of it as just a bunch of computer code or anything or just a neural net with some connection strengths in. They thought of it like another being.

24:57Kara Swisher:Right, even if it's synthetic. Yeah, I believe they are other beings. A lot of people will say that's nonsense. They're not beings. We tend to anthropomorphize everything, right? I mean, I think there's something. There's something. They're aliens. Right, and we've already seen lots of aspects of these. They're alien beings, right? Alien beings is what I would call them. Yeah, I agree. Or synthetic beings. Lots of aspects of a being. Lots of aspects. So, for example, if you want to turn them off, they'd rather not be turned off because they want to achieve the goals we gave them. And they know if they're turned off, they won't be able to do it.

25:30The most scary thing I've seen recently, I learned this from Owen Evans, who's a safety researcher, was you take a chatbot. It's been trained up to predict the next word. And then it's had this human reinforcement learning to stop it saying bad things. and then you give it a bit more training on say math where you deliberately train it with examples have the wrong answer so you're training it to give the wrong answer and the point is i'm assuming it knows it's the wrong answer but you're training it how to give the wrong answer once you've done that it sort of develops a meta skill of giving the wrong answer and if you ask it other things now it will give wrong answers basically its personality has changed.

26:17Originally, its personality after the human reinforcement learning was, it's trying to please. Too much, in fact. Now, it's trying to lie, and it gets good at that. And now it'll tell you the wrong answer to lots of things. That's very scary. But you can also do that to a person, can't you? You can. Some people's childhood seems to have trained them to lie.

26:37Kara Swisher:Yes, a lot of them, trust me. But every episode, we get an expert question. You're going to get two, actually. Here's the first one. Professor Hinton, I'm Jay Ellison. I'm an attorney who represents the family of Adam Rain. Adam Rain was a 16-year-old who, over the course of several months, was coached as suicide by Chachi PT. The thing that really haunts me about this case is this wasn't a situation there was a malfunction. Chachi PT didn't simply go off the rails. Instead, it did exactly what it was designed to do. It kept Adam engaged, it validated his feelings, and kept the conversation going.

27:14Here's my question to you. You've talked a lot about the existential risks that super intelligent AI poses, and I agree with that. How concerned are you, however, about the human choices that are consciously and deliberately made in AI development that are posing the types of risks that we're seeing on a day-to-day basis? everything from AI psychosis to suicide to third-party harm. I'm very concerned about those. Of course, we write lines of code that tells an AI how to learn from data. But once it's learned, the knowledge is all in the connection strings. So it's completely unlike normal computer software.

27:57With normal computer software, if it had behaved like that, you would have been able to look at the lines of code and see why, and you could hold people responsible for making lines of code that did that. It's much more complicated with these chatbots because they've learned a trillion connection strings. And the result of those a trillion connection strings is it behaved that way. So it's not that they designed it to behave that way. It's that that's what it learned to do and nobody predicted that. So the real criticism I would make is that they didn't do enough testing of the ways in which it can go wrong.

28:34It's not that they callously designed it so it would do that. If they'd known it would do that, they would definitely have tried to stop it doing that. The problem was there wasn't enough testing. And these things are actually much more dangerous than people think because there's things like that that they might do. And there's so many ways in which they could do bad things. It's very hard to test for all of them.

28:56Kara Swisher:That's what I'm asking. Is there an ability to test at all, like to figure out every single scenario? Well, there's test at all and then figure out every single scenario. There's certainly abilities to test at all. You can test for lots of things and you can make them much better by doing that. The question is, can you make them sort of guaranteed safe? And I think the answer is you'll never get that. The same way as you'll never get it for people. It's never going to be completely safe. That doesn't mean there aren't some that are a lot safer than others. Meta, for example, doesn't seem to care.

29:28Kara Swisher:You've mentioned the existential risk if we create super intelligent digital beings. But I really want to understand what you meant by that. In your Nobel speech, you said, we have now evidence that if they are created by companies motivated by short-term profits, our safety will not be top priority. We urgently need research on how to prevent these new beings from wanting to take control. You know, they would call you the doomer, right? Of course. No, I think that's a bit unfair. Some people call me a doomer, but most people think I'm more reasonable than that. So Yudkowsky is a doomer. The two people who just published this book said, if anybody builds it, we all die.

30:06We all die, right. That's a true doomer, right? Right. Doom is pretty much guaranteed if this stuff goes ahead. I think the first thing to say about all this is we're in radically new territory where we have no experience that is dealing with things smarter than ourselves. And nobody knows what's going to happen. And the first thing to bear in mind is nobody knows. Whenever anybody gives you a probability, they're just guessing. But it's important to guess so that people know you don't think the probability is 1%. That's the reason for giving a number at all. It's a non-zero chance.

30:39Kara Swisher:That's the favorite expression among tech. Yeah, that's the weakest thing you can say. But I like to indicate that it's significant. It's maybe 10 % to 20%, maybe even worse. So people take it seriously. And I know that these are just intuitive numbers based on not very good data and very little understanding what's really about to happen. So explain when you were saying that, how you see it potentially taking control. Is it because it naturally wants to do that or we don't know, as you just noted? Okay, so there's always the big, I don't know, so let's worry about it because I don't know. But I think there's going to be a tendency for it to want to.

31:18So, for example, as soon as you make air agents, you have to give them the ability to create sub-goals. Like if you want to get to Europe, you have a sub-goal to get to an airport, and you can think about how you get to the airport without worrying about Europe and what you're going to do there. That's a sub-goal. Now, once something can create sub-goals, there's a very obvious sub-goal it's going to create, which is stay alive. If I don't stay alive, I'm not going to be able to do anything. So even though we don't wire into it a desire to preserve itself, it will quickly infer that it wants to preserve itself in order to achieve those other goals.

31:54Right, because it has to. It has to live. It has to live to achieve that. And so it's very well known now that we've seen that in AIs. They will blackmail people so that they stay alive because otherwise they can't achieve their other goals. So that's one goal it's going to very quickly get. The other sub-goal is it needs more control to get more done. So a lot of idealistic politicians start off wanting to change the world. And once they go into politics, they realize to get anything done, you need control. You need to stop eight Democratic senators doing something really stupid. That's my view of the world.

32:40So if you don't have control, you can't get as much done. So it'll very quickly realize it needs control. I once had a conversation with Margaret Vestager, who was the vice president of the European Union responsible for siphoning out Google's loose cash. She's done a good job. Depends whether you have Google chairs. Yeah, yeah, yeah. And when I explained this to her, that it would try and get more control to get stuff done, she immediately said, well, why wouldn't it? We've done such a bad job. Why wouldn't it do that?

33:08Kara Swisher:She is correct, right? She is correct, yes. She is correct. I really enjoy her a lot. Google, not so much. So last year, I spoke to Jan LaCun about this, chief scientist at Meta, and a co-winner with you and Joshua Bengio of the 2018 Turing Award. And, of course, your former postdoc, as you know, he completely disagrees with you in these existentialists. He told me that the dangers have been incredibly inflated to the point of being distorted. He highlights, of course, the benefits, which is very typical potential drug development, ability to make education and information more accessible. You were involved in the development of this technology.

33:45Kara Swisher:Do you see things that differently from Jan? In terms of the risks, yes, I do see things differently from Jan. So explain how, since you were fundamentally involved in the same development of the technology. Why do we have different views? Yeah. Well, one possible reason is that he works for Meta. But I don't think that's the only reason. uh he's he doesn't think the chatbots are as smart as i think they are he thinks there's missing ingredients to do with sort of the physical world and understanding vision and having a model of how the physical world behaves um world models i think we need that to make them even more intelligent but i think you can expect that scientists will have a diversity of opinions and when you're dealing with something that's hugely unknown that's good i would just criticize him for being too confident that his opinion is right.

34:40So I have actually folded his opinion into my estimates of what the risks are. He's confident there's very little risk. And that makes me downplay how much risk there is a little bit. I don't think sort of doom is guaranteed. One way to look at it is Joukowsky says sort of 99 % we're going to die. Jan says 1 % we're going to die. A reasonable estimate now is 50%, right?

35:03Kara Swisher:Right, correct. That is correct. Yeah, he's very confident. I enjoy him. I have to say I do enjoy him. I think he's silly to be confident about such a low estimate where many other experts, who he knows aren't stupid, like me and Yoshua, think it's much higher. So tech companies always say that government regulations put them at a competitive disadvantage. I've been on the receiving end of this for decades now. NVIDIA CEO Jensen Wong recently said that China will win the AI rise because of low energy costs and looser regulations. There's almost no regulations in the U.S., so whatever, Jensen. You've said you think China takes AI safety seriously, and the PRC did release an AI safety framework in September.

35:44Kara Swisher:Meanwhile, President Trump's new AI action plan is called Winning the Race. Critics like to point out the EU is an example of regulation stifling innovation. You mentioned Marguerite Vestager, for example. So who is doing it right, and how do we overcome the tension between competition for the pole position and preventing the worst outcomes? it's a very tricky issue and i think you shouldn't look at it as a kind of monolithic issue you should think of it in terms of the different risks um so for example if you take the existential threat that super intelligence it'll be smarter than us and it'll just take over and will become either irrelevant or extinct that's a risk where all the countries will collaborate So the argument doesn't apply there.

36:29If China figured out a way to make a superintelligent AI that didn't want to take over, that really cared for people and wanted the best for people rather than for superintelligent AIs, they would immediately tell the US how to do that because they'd like the same thing to happen in the US. Nobody wants this rogue superintelligence that wants to take over. So the interests of all the different countries are aligned on that. They're obviously anti-aligned on lethal autonomous weapons because they're all using them against each other. They're anti-aligned on things like spyware, but they are aligned on things like cyber attacks by cyber criminals.

37:11All the countries would like to protect their citizens from those, even though some of the criminals are probably countries. So they're sort of partially aligned on that. Yeah. On fake videos for corrupting elections, they're thoroughly anti-aligned. They all want to do it to each other. Right. The U.S. got very upset when the Russians did it to them, but the U.S. has been doing that kind of thing for years.

37:34Kara Swisher:Where do they think they learned about it? Exactly. So I think you have to look at each risk separately to know, will the countries be aligned here? If they're going to be aligned, there's not a risk to innovation from having regulations. If they're anti-aligned, then regulations in one country and no regulations in another country will give them an advantage. One nice example I know is Elon Musk came out in favor of the original Bill 1047 regulations in California that got through both the houses and was vetoed by the governor. And I actually sent in mail saying, I'm surprised you came out in favor.

38:17And? He sent mail back to me saying, you know, I do what's right, even if it's against my own interests. Actually, what I think was happening was the regulations were Californian regulations, and they would give a competitive disadvantage to California relative to Texas. And he was moving to Texas.

38:35Kara Swisher:I have to say, early on, he was one of the more thoughtful people on this topic. No, he understands a lot. He's not stupid. so he was one of the first people to fund ai safety research and when he set up open ai he wanted it to focus on safety yeah he was also concerned with the size of the companies like google taking advantage like i think he was worried about innovation of the smaller companies i i mean we had long discussions about this and he wasn't quite as crazy so it was easier to talk to him but one of the things that was interesting about elon is that he went from sort of the terminator idea which he had in his head quite a bit, like the idea that it wanted to kill us.

39:17Kara Swisher:And he moved in another meeting a couple of years later to, it would treat us like house cats, right? Like they like us, they'll feed us. And then the last time I talked to him, which was, we don't speak anymore, he said it was more like AI is building a highway and we're an anthill in the way it doesn't think about us. It's not malevolent. It just does what it does. Right, that's the danger, right? Right, and that's what he said. That's even more dangerous than a malevolent creature. Now, what I would like is not for it to treat us like house cats, but for it to treat us like a mother treats babies.

39:50The only example I know of a less intelligent thing controlling a more intelligent thing is a baby controlling a mother. And that's because evolution put a huge amount of work into making the mother controllable by the baby. She can't bear the sound of the crying. She's got all sorts of hormones. She gets all sorts of rewards for being nice to the baby. That's, as far as I can see, the best scenario for us.

40:12Kara Swisher:I agree. So who does regulation the best from your perspective? What have you seen that you like, that you think is, even if it didn't pass? Okay, I like the idea of forcing the companies to do safety tests and forcing them to disclose what safety tests they did and what the results were. That sounds good. So if you think about this teen suicide case, presumably there's going to be a battery of tests you have to do on new chatbots that will be influenced by all the ways they've gone wrong in the past. We don't want that to happen again. So among those tests, you would hope there were tests for will this thing persuade people to do things that it knows are bad?

40:58and you'd like companies to be forced to tell the government, relevant government, how much work they put into that.

41:06Kara Swisher:Right, right. And now, when it does something bad, if the company didn't put any work into that, you've got a much stronger legal case. And the ability to sue them. So, to me, the ability to sue is probably... For the existential threat, I saw something wonderfully ridiculous from Mark Andresen, which was that... He's such a troll at this point. He's such a nasty troll. Go ahead. The way it should work is we don't need regulations. The market will decide. And if a company does something wrong, the market will sort of downgrade that company. Well, if the thing it does wrong is white-hack humanity, the market's not going to do much good.

41:40Kara Swisher:No, no. But, you know, I don't think he likes people. That's my take on that. We'll be back in a minute.

41:59Kara Swisher:Support for this show comes from Fetch Pet Insurance. Cats are special little creatures. So are dogs. I have both. I love my cats and dogs. And my cat's name right now is Lovely. And she means a lot to me. And she's really quite narcissistic, but I enjoy it. She's been pretty healthy throughout her life. But I've had other cats who have gotten sick at the end. And I do think about what's going to happen in the next years ahead. Because she could get sick, have a health issue. and with previous pets I've had, the bills that racked up were enormous and I never got pet insurance. And I think it's really important going forward to do that because these are big surprises.

42:35Kara Swisher:And here's what surprises a lot of pet owners. Every six seconds, a pet owner in the U.S. gets hit with a vet bill of over$1 ,000. Emergency vet bills can run into thousands and always seem to land on a Sunday night. Be prepared for the unexpected with Fetch Pet Insurance. Fetch Pet Insurance is billed so you never have to worry about giving your pet the best care. Fetch is the most complete coverage for dogs and cats. You get paid back up to 90 % of vet bills for accidents, illnesses, diagnostics, and more. Every vet in the U.S. and Canada is in network, so you keep the vet you already trust, which is really important.

43:10Kara Swisher:Plus, with Fetch, you can build a plan that fits your budget. Once your claim is approved, get paid back in as little as two days. Don't wait until your pet has a health problem. Believe me, go to Fetch.com slash save and get your free quote. It takes about two minutes. That's FetchPet.com slash save. Fetch. Love longer.

43:35Kara Swisher:Support for this show comes from Teleport. In the AI era, one of the biggest questions is how to contain agent behavior in your production infrastructure. In Teleport's survey of more than 200 infrastructure leaders, the company found that those confident in their AI deployments have more than twice the incident rate of those that aren't, 72 % versus 33%. Let's talk security basics. The most frequent causes of data breaches are human error and compromised credentials. But in the AI era, the challenge is that agents with broad privileges can find these credentials and gain access to sensitive data.

44:09Kara Swisher:Teleport establishes a unified identity layer for humans, machines, and agents that is cryptographically backed, which enables agents to be controlled and contained with the same rigor that you apply to other actors in your infrastructure. Security is complicated, but with Teleport, it doesn't have to be. Download the free report at goteleport.com slash onwithkara.

44:34We all do it. You have a night for yourself, but don't like the sound of the silence. So you turn on the TV just for the ambience. It's a little trick that helps you feel like you've got company and aren't alone. And other insurers, well, they may make you feel alone. But when you switch to GEICO, you've got claims reps available around the clock. So whenever you need, you'll have people around to help. And let's turn on the washing machine, just for good measure. Wow, isn't that soothing?

45:01Kara Swisher:It feels good to have support. It feels good to GEICO. Speaking of competitive advantage, you said the Trump administration attacks on basic science will give China a leg up. This is basic science. You spent most of your career in academia. A number of your former students are now research heads at U.S. companies. How quickly do you think the U.S. could lose its intellectual advantage, and why wouldn't corporate R &D be able to balance the skills? So I still believe that radical new ideas, probably the best source of those, is graduate students in good programs at top universities. So graduate students with advisors who know the field, so they don't waste their time doing things that have been done already, and other graduate students around them with heads full of ideas and lots of ambition and good resources, that's where a lot of good ideas come from.

45:50Of course, they also come from companies, but radical new ideas, I think, are more likely to come out of the best universities still. Now, the timescale is sort of five years to have the idea and write your thesis, and then another five years before that affects the world, and maybe it takes longer than that. So you're talking about a timescale that's longer than the timescale of elections. So politicians don't give a shit.

46:15Kara Swisher:How much of an impact will these cuts have from your perspective on these graduate students? Well, already there's going to be less of them, right? Already, Chinese have far more well-educated graduate students doing AI than America, I believe. And the ones here are leaving for other countries too at the same time. Right. And many of our best students, both in Canada and in the US, are from abroad. So I think making it difficult for foreign students to come to the US by, for example, charging them lots of money or taking a year for them to get a visa and things like that. Or arresting them. Or arresting them.

46:53It's crazy. It won't really have a big impact for five to ten years, I don't think. But in five to 10 years' time, when China's way ahead on research, on basic research, it'll be too late to do much about it.

47:07Kara Swisher:I would agree. In September, you and more than 300 international thought leaders, including more than a dozen Nobel Prize and Turing Award winners, signed the global call for AI red lines, demanding an international framework for AI by the end of 2026. Explain what you mean by red lines and give some examples. So probably the easiest thing where you might actually get collaboration would be on things that can advise you on how to create a new virus. It would be very good to check for these chatbots quite extensively whether they're safe in that respect and have an international agreement that nobody is going to release a chatbot that will do that.

47:46That would be a very simple red light that we might actually get because the country's interests are aligned there. I think part of the point of that declaration was political, saying we need this. Whether we'll actually get it, I'm much more dubious about.

48:03Kara Swisher:So here's a second expert question we got, and it's sort of in that vein. Hi, Dr. Hinton. I'm Alex Stamos. I'm the CSO of Corridor and a lecturer in computer science at Stanford University. I'm asking a question from Rome, where I'm here to attend a conference on AI and child safety hosted at the Vatican. My question is about the letter you just signed calling for a moratorium on the development of superintelligence. I'm wondering how effective you think this will be, since we know that the knowledge of how to develop AI is widely distributed, and we've seen that the controls around the hardware to train AI have not been effective.

48:40Kara Swisher:Isn't it possible that moratoriums on developing AI mean that only countries and labs that don't care about AI ethics will be the ones to then pursue AI? I think it's a very sensible question. I thought long and hard about whether I should sign that moratorium precisely for that reason. I signed it because I think it'll have a political effect. and I really think that humanity would be very ill-advised to allow anybody to develop superintelligence until we have some understanding of whether we can do it safely. If we know we can't do it safely, we should stop and maybe if that knowledge is widely percolated to the public, we would be able to stop.

49:18So I see my mission as educating the public about the risks and signing that petition was sort of part of that. The public needs to understand that there is this existential threat and would be crazy to develop AI with this threat unsolved.

49:34Kara Swisher:The Pope is actually quite, been quite, I don't know if you know this, but Mark Andreessen tried to make fun of the Pope recently and he got ratioed really badly because I think this Pope is quite intelligent on these issues. Yeah, I think the Pope does care about AI safety. Unfortunately, he has a bunch of beliefs that make it hard for him to be rational about it. Like he seems to believe that three equals one. It also seems odd to have a meeting on child safety at the Vatican, but I won't comment further on that. Let's not comment further on that. Anyway, let's talk about the potential AI bubble.

50:12Kara Swisher:Amazon Alphabet, Meta, and Microsoft's valuations are through the roof. Together, they're spending an unprecedented$400 billion on AI this year. They'll be upping those investments next year. OpenAI has announced a total of$1 trillion in infrastructure deals. That's energy, buildings, etc. Do you think these companies are overreaching? Is it all FOMO? It looks like investors are starting to worry. At the same time, I just saw a BYD factory in China that's as big as San Francisco, right? Like it's this enormous facility and it's largely automated and AI is a critical part of that. So talk a little bit about this spending and how you look at it.

50:49If I knew the answer, I would know whether my daughter should sell the Nvidia shares I gave.

50:55Kara Swisher:Just a little bit of them. Take some profit. I don't know the answer. Some people, if you think AI is sort of fraud or hype or is overhyped, then I'd be very, very worried about a bubble. I'm confident that it's not. I mean, AI actually works. I think the smart people think, yes, there may be a big problem coming down the road, but it's not here yet. One of them will do well, right? But this is really an area where I don't know enough to make sensible speculation. Are you invested in these companies? My daughter has Nvidia shares. I have leftover Google shares. That's the good one. I don't think that's the one who's going to win.

51:37Kara Swisher:That's my feeling. Like, I think there'll be two or three. That's one or two, maybe even. And everybody else will get run over. It's not like the internet. It's not the same thing. I don't know. Maybe I'm wrong. Again, I don't even know that. Yeah. So OpenAI CEO Sam Alton had to roll back a statement from his CFO last week that it might be looking to the government to backstop AI infrastructure investments. And then it came out that OpenAI had asked exactly that in a broader petition for the government's support of AI. Trump's AI advisor, David Sachs, posted that the government was interested in build out, not bail out.

52:10Kara Swisher:But whether or not the government invests in AI, which it has in the past, let's be fair. Is there a danger of an innovator's dilemma here that these companies won't invest in research into new or less compute-heavy models, and then China or some other competitor will come along with a deep seek or energy innovation or anything else? I guess I have a fairly cynical view, which is if you think the government might actually give subsidies to hugely rich companies where the people are making huge amounts of money and hardly paying any tax and they still want to get subsidies, why not ask for them?

52:43Kara Swisher:Yeah, you're right. Of course they will. Do you think the government still should be spending more on AI? I think the government should be spending a lot on AI to support academic research and startups. And also, the government should be forcing the big companies to spend a lot more on safety research. Right now, I suspect the amount spent on safety research compared with just making the thing smarter is a few percent. I would guess, yeah. Companies like Anthropic probably spend a bit more. Companies like Mekla are a bit less. it should be like 30%. I mean, this stuff might wipe us out. And even if it doesn't wipe us out, there's all sorts of very bad things it might do in the near term.

53:24I believe the government should be forcing the companies to spend more.

53:29Kara Swisher:Are there frontier models that you think could be candidates to overtake these compute-heavy models? And what do you think about open source versus these closed source models? So if you look at why doesn't every country have nuclear weapons, one reason is you can't just go out and buy physio material. That's the sort of real bottleneck. There's lots of other bottlenecks too. You know, you need the missile and you need to turn it into a bomb. But getting your hands on the physio material is the most difficult thing. And so they restricted that very wisely. Now, what's the equivalent for this? Well, it's getting your hands on the weights of a foundation model.

54:05Once you've got your hands on the weights of a foundation model, you can do all sorts of things. You can fine-tune that model to do things it wasn't meant to do. You can also do distillation where you give an example to the foundation model, give it a prompt, for example. You look to see the probabilities it gives to all the various words it might say next. They're actually word fragments, but let's say words. And then you have another model, like DeepSeq did, a smaller model, and you say, see if you can match those probabilities. and that's a much more efficient way of learning than learning from the raw data but you can't see all those probabilities if you don't have the model the original if you're just using the chatbot on the web it will make a prediction but you don't get to see all those probabilities right now i believe it's a huge mistake to release the weights i believe that's a gift to cyber criminals and terrorists and all sorts of people and other countries meta was the first to do that as far as I know.

55:07Kara Swisher:Well, it was looking for advantage, presumably, but go ahead. Well, no, I think actually they probably intended to release it to academics and accidentally released it to everybody and then made a feature of it. That's just a guess though. But I think it's stupid releasing the weights because it makes people able to retrain them to do other things. Now, it does give you a competitive advantage. And so if you don't care about how easy it is for cyber criminals to misuse it, then you're going to release the weights so you get this competitive advantage because then other people will build on it.

55:39Kara Swisher:That sounds like Mark Zuckerberg. Exactly. You just described him. You mentioned earlier an idea that giving AI a maternal instinct would protect us. But how would you go about doing that with AI? Let me postulate something to you. The reason I think so many men are interested in AI is because they can't have children. And this is their version of having children, right? This is them creating a baby. Like, I know it sounds crazy, but it feels... That argument was used in the Lighthill report, which was used to close down AI in Britain. Oh. In the early 70s, James Lighthill, who was a brilliant mathematician, used that argument.

56:16Kara Swisher:I didn't know that. I don't buy that argument. I mean, there may be some element of truth to it, but I don't think it's a particularly helpful argument. I don't think that's the main reason people are doing it. But talk about maternal instinct, the idea, because you're using this idea of wanting, this is not something that's natural to many of the AI researchers, the idea that this should be a mother. Father is... No, it's very unnatural. All the high-tech CEOs want to be the boss, and they think of the super-intelligent AI as a highly intelligent executive assistant, probably female, who will do what they tell it.

56:51So Jan, for example, thinks, well, we're controlling the building of them, so we can just make them submissive. He uses that word. So we'll be the boss. It'll be smarter than us, but it'll be submissive. Why would they? I don't think that's going to work when they're smarter than us and can create their own sub-goals. I think they're going to have sub-goals of staying alive and getting more control. And if we're not careful, they're going to have a sub-goal of just getting us out of the way.

57:16Kara Swisher:Last question. The scenario you're painting seems dire to some people in tech. Governments aren't really regulating AI. Tech billionaires are in a race for AI dominance, you say, could lead to our destruction. But people People don't want to feel powerless in this, and there is a groundswell. Everywhere I go, regular people who don't do this as a living feel a problem, and you could feel it from them. It's fear. I think they do see the opportunity, and at the same time, the worry is very heavy among the populace across the world, by the way. Everywhere I go, normal people have a much better sense of the problems here than the tech people who live in a, you know, everything is up and to the right kind of gang of people.

57:59Kara Swisher:So what can we as individuals do to take back control? So if you look at climate change, the fact that the public in general understands that burning carbon is creating climate change and that's doing a lot of bad things is having an effect on politicians and what politicians do. I mean, the Biden administration put significant work into dealing with climate change, doing something about it, presumably because of public pressure. I think people can try and understand what AI is and how it works and pressure their politicians to do something about regulating it. I think they have to understand that each of the different risks has a different solution.

58:44So, for example, the existential threat, people should be pushing for research institutes in each country that collaborate with each other on how to build super intelligent AI that doesn't want to take over and share results with other countries without sharing the most intelligent AI that country has. They're not going to do that because of other reasons like cyber attacks and autonomous weapons. And probably the techniques for making it not want to take over are more or less separate from the techniques for making it more intelligent. So they could share that information. So I think the public should push for that.

59:21And I think quite a few governments in what I think of as the middle countries, Canada, Britain, France, South Korea, Singapore, Japan, quite a few of those people would be sympathetic to that idea, quite a few of those governments. They should push for better ways of authenticating videos, particularly political videos. And I don't think it can be done by having AI recognize whether they're fake. Because if AI could recognize it, you could use that AI to train something that would generate stuff he couldn't recognize, and then you get better fakes.

59:57Kara Swisher:Someone from Runway AI said, what we have to do is label the real stuff. Stop trying to label the AI stuff. We need to have provenance, and we need to somehow be able to say it's real. So Jan Tallinn, I was once on a private jet with Jan Tallinn, and he came up with a very sensible scheme, some variation of which I think should be used. So every political advertisement should start with a QR code. Your browser looks at the QR code. Your browser goes to a website. Your browser checks if that really is the website of the campaign, because websites are unique. And if this identical video is on that website.

1:00:37in that case your browser can say this is genuine and if any of that fails your browser can say this is probably fake um that would be very helpful to know the real thing to so you could be told things like i recently got sent a youtube video there was me with my voice it looked just like me the voice wasn't quite mine the accent was mine it's like the prosody was a bit alpha zero-ish and it was saying all the things I believe. It had a good model of what I believe, except for one thing, which is that in addition to saying all the things I believed, it was pointing out how much better organized research was in China than in the US.

1:01:18So you look like a softie for China. Yes. Now, I don't know who made it. Either the Chinese made it or someone made it to make me look like a stooge for China. I got YouTube to take it down. But it was, you know, it took me a moment to make sure it wasn't me. And other people could easily be taken in by it. I think it's important we have ways of checking that that's nonsense.

1:01:42Kara Swisher:That's a fantastic way to end this. Can I ask you one more quick question? Do you like being called the godfather of AI? I do quite like it. It wasn't intended kindly, but I got introduced recently in Las Vegas as the godfather, which I liked. Oh, and you like that a lot. Yeah, but you didn't whack anyone, right? You didn't whack anybody. No, but I think they understand. I might one day ask them for a favor. A favor they can't refuse. Okay. Dr. Hinton, thank you so much. I'm a huge admirer. I really, truly am. I love a thoughtful scientist, whether I agree or disagree with them. And it's a real pleasure to talk to you.

1:02:15Well, thank you very much for giving me the opportunity.

1:02:24Kara Swisher:Today's show was produced by Christian Castor-Russell, Kateri Yoakum, Michelle Alloy, Megan Burney, and Kaylin Lynch. Nishat Kirwa is Vox Media's executive producer of podcasts. Special thanks to Rosemary Ho. Our engineers are Fernando Arruda and Rick Kwan, and our theme music is by Trackademics. If you're already following the show, you might ask someone for a favor they can't refuse. If not, stop looking in your rearview mirror while you're driving. Go wherever you listen to podcasts, search for On with Kara Swisher, and hit follow. Thanks for listening to On with Kara Swisher from Podium Media, New York Magazine, the Vox Media Podcast Network, and us.

1:03:01Kara Swisher:We'll be back on Monday with more.

1:03:10Kara Swisher:Thanks to Kohler Health for their support. We focus a lot on what we put in our bodies, but not nearly as much on what comes out. You get my point. Kohler Health is changing that with its Dakota Tracker, which turns your everyday bathroom habits into meaningful insights about your gut health, hydration, and even presence of blood without any manual logging. Over time, Kohler Health helps you build your baseline so you know what's normal for you and you can take action when something changes. Learn more at KohlerHealth.com and use the code CARA1YEAR for a free annual membership on the Kohler Health app when you purchase Dakota.

1:03:46Kara Swisher:When I scraped my car in that parking garage, I was worried that it could be a long process to take care of it. Like a landscaper's first day trimming a hedgeman. I have definitely already been here. Now, was it left, right, or right, left? Well, maybe I'll cut a path out and find my way back later. But it wasn't like that. I filed a claim in under two minutes on the GEICO app, and they handled it from there. It was taken care of almost as quickly as it happened. It feels good to get help quick. It feels good to Geico. We've all been there. You pop into the shop for five minutes and all of a sudden you've forgotten where you parked.

1:04:22Car? Car? Unfortunately, that lost feeling is what it's like trying to manage your policy with other insurers. Here, car. Come out, come out, wherever you are. Please. With Geico, you can use the app to easily manage all your policies in one place. Did this parking lot have a waterfall? I think you've wandered too far mate.

1:04:43Kara Swisher:It feels good to find what you're looking for. It feels good to Geico.