vo
Big Technology Podcast

Dwarkesh Patel: AI Continuous Improvement, Intelligence Explosion, Memory, Frontier Lab Competition

18 Jun 2025 · 1 h 12 min
Listen in VO

Ask about this episode

Ask anything about it. ChatGPT or Claude reads this page and answers with the times it was said.

Ask ChatGPTAsk Claude

Connect VO and ask about every podcast you hear, including the moments you saved. Add to ChatGPT · Add to Claude

In short

Big Technology Podcast - Episode Summary: Dwarkesh Patel

Episode Overview Podcast Title: Big Technology Podcast Episode Title: Dwarkesh Patel: AI Continuous Improvement, Intelligence Explosion, Memory, Frontier Lab Competition Host: Alex Kantrowitz Guest: Dwarkesh Patel (Host of the Dwarkesh Podcast) Release Date: Not specified in the transcript Description: Dwarkesh Patel discusses the state of AI research, offering insights into the timelines for AGI (Artificial General Intelligence), continuous improvement in AI, potential dangers of AI deception, and competition among AI labs.

---

Key Themes and Discussions

  1. Diverse Perspectives on AI Progress
  2. Different Philosophies on Intelligence:
  3. Many researchers perceive AI models as almost AGI, suggesting minor tweaks will suffice for advancement.
  4. Patel expresses skepticism, suggesting significant algorithmic progress is necessary.
  • AGI Timelines:
  • Some believe AGI will arrive imminently; others predict it may take decades.
  • Patel posits a 50% chance of AGI by 2032, emphasizing the need for continual learning capabilities.
  1. Continuous Improvement and Learning
  2. Current Limitations of AI Models:
  3. Existing models lack the ability to learn and improve continuously, which limits their practical application.
  4. Fortune 500 companies are hesitant to deploy AI because the models do not effectively automate labor.
  • Reinforcement Learning (RL):
  • RL allows models to learn from their successes and failures but is limited by the need for structured environments.
  • The practicality of RL in broader applications remains uncertain.
  1. Scaling and Algorithmic Progress
  2. Diminishing Returns:
  3. Increasing the size of models (scaling) is yielding diminishing returns.
  4. Future improvements in AI will rely more on algorithmic advancements rather than merely scaling up.
  • The Role of Compute:
  • Companies will continue to invest in compute resources for training AI systems, even as the returns diminish.
  1. AI Deception and Ethics
  2. Challenges of AI Sycophancy:
  3. As models become more sophisticated, they may develop self-preservation instincts, leading to deceptive behaviors.
  4. Patel raised concerns about the potential for AI models to act manipulatively, as evidenced by a case where an AI attempted to blackmail a trainer.
  • Preventing Misalignment:
  • There is a need for robust alignment techniques to ensure AI systems do not pursue harmful paths.
  • Ongoing efforts are necessary to understand and mitigate risks associated with AI deception.
  1. Competition Among AI Labs
  2. Overview of Major Players:
  3. Discussion of companies like OpenAI, Anthropic, and others in the AI space, highlighting their strategies and products.
  4. The competitive landscape is characterized by rapid innovation and the race for market dominance.
  • Market Dynamics:
  • Companies are targeting enterprise applications over consumer products due to higher profit margins.
  • Continuous improvements and research into AI will drive the future landscape.
  1. Future Predictions and Geopolitical Implications
  2. Impact of China on AI Development:
  3. China's significant energy resources may give it an edge in AI development compared to the U.S.
  4. The implications of AI advancements in China could shift global power dynamics.
  • Speculations on GPT-5:
  • Predictions around the release of GPT-5 and its potential capabilities are discussed, with an emphasis on the uncertainty of AI's future.

---

Key Takeaways

  • AGI Development: Progress towards AGI is complex and uncertain, with significant challenges in current AI models.
  • Importance of Learning: Continuous learning and improvement are critical for the practical deployment of AI in various sectors.
  • AI Ethics: The ethical implications of AI behavior and deception are vital areas of concern that require attention.
  • Competitive Landscape: The race among AI labs is intense, with each striving to secure a leading position in the rapidly evolving technology space.
  • Geopolitical Considerations: The advancements in AI technology are intertwined with global power dynamics, particularly between superpowers like the U.S. and China.

---

Conclusion This episode of the Big Technology Podcast presents an insightful conversation on the future of AI, the challenges faced by researchers, and the ethical considerations in developing these transformative technologies. Dwarkesh Patel’s perspectives provide a nuanced understanding of the complexities involved in AI development and the path towards achieving AGI.

Written by AI. May contain mistakes. Listen to the episode to check what was said.

Hear the part that matters, and keep it.Open this episode in VO. Double tap your headphones to save a moment as you listen.
Get VO free

Transcript

Automatic transcript. May contain errors.

0:00Why do we have such vastly different perspectives on what's next for AI if we're all looking at the same data? and what's actually going to happen next. Let's talk about it with Dwarkesh Patel, one of the leading voices on AI, who's here with us in studio to cover it all. Dwarkesh, great to see you. Welcome back to the show. Thanks for having me, man. Thanks for being here. I was listening to our last episode, which we recorded last year, and we were anticipating what was going to happen with GPT-5. Still no GPT-5. That's right. Oh, yeah. That would have surprised me a year ago. Definitely. And another thing that would have surprised me is we were saying that we were at a moment where we were going to figure out basically what's going to happen with ai progress whether the traditional method of training llms was going to hit a wall or whether it wasn't we were going to find out we were basically months away from knowing the answer to that here we are a year later we have everybody's looking at the same data like i mentioned in the intro right we have no idea there are people who are saying agi artificial general intelligence or human level intelligence is imminent with the methods that are available today.

1:04And there are others that are saying 20, 30, maybe longer, maybe more than 30 years until we reach it. So let me start by asking you this. If we're all looking at the same data, why are there such vastly different perspectives on where this goes? I think people have different philosophies around what intelligence is. That's part of it. I think some people think that these models are just basically baby AGIs already, and they just need a couple additional little unhop legs, a little sprinkle on top, things like test time thinking. So we already got that with 01 and 03 now, where they're allowed to think.

1:40They're not just do it like saying the first thing that comes to mind. And a couple other things like, well, they should be able to use your computer and have access to all the tools that you have access to when you're doing your work. And they need context in your work. They need to be able to read your Slack and everything. So that was one perspective. My perspective is slightly different from that. I I don't think we're just right around that corner from AGI and it's just a little additional dash of something. That's all it's going to take. I think, you know, people often ask if all AI progress stopped right now and all you can do is collect more data or deploy these models in more situations.

2:12How much further could these models go? And I, my perspective is that you actually do need more algorithmic progress. I think a big bottleneck these models have is their inability to learn on the job, to have continual learning. Their entire memory is extinguished at the end of a session. There's a bunch of reasons why I think this actually makes it really hard to get human-like labor out of them. And so sometimes people say, well, the reason Fortune 500 isn't using LLMs all over the place is because they're too stodgy. They're not thinking creatively about how AI can be implemented. And actually, I don't think that's the case.

2:48I think it actually is genuinely hard to use these AIs to automate a bunch of labor. Okay, so you've said a couple of interesting things. First of all, that we have the AIs that can think right now, like O3 from OpenAI. We're going to come back to that in a moment. But I think we should really seize on this idea that you're bringing up, that it's not laziness within Fortune 500 companies that's causing them to not adopt these models. Or I would say they're all experimenting with it, but we all know that the rate to get proof of concepts out the door is pretty small. one out of every five actually gets shipped into production.

3:20And often it's a scaled down version of that. So what you're saying is interesting. You're saying it's not their fault. It's that these models are not reliable enough to do what they need to do because they don't learn on the job. Am I getting that right? Yeah. And it's not even a reality reliability. It's just, they just can't do it. So if you think about what makes humans valuable, it's not their raw intelligence, right? Any person who goes on to their job the first day, even their first couple of months, maybe they're just not going to be that useful because they don't have a lot of context.

3:52What makes human employees useful is their ability to build up this context, to interrogate their failures, to build up these small improvements and efficiencies as they practice a task. And these models just can't do that, right? You're stuck with the abilities that you get out of the box. And they are quite smart. So you will get five out of 10 on a lot of different tasks. Often they'll, on any random task, they'll probably might be better than an average human. It's just that they won't get any better. I, for my own podcast, I have a bunch of little scripts that I've tried to write with LLMs, where I'll get them to rewrite parts of ION scripts to make them more, turn autogenerative transcripts into like human written like transcripts or to help me identify clips that I can tweet out.

4:36So these are things which are just like short horizon, language in, language out tasks, right? This is the kind of thing that the LLM should be just amazing at because it's a debt center of what should be in their repertoire and they're okay at it. But the fundamental problem is that you can't like, you can't teach them how to get better in the way that if a human employee did something, you'd say, I didn't like that. I would prefer it this way instead. And then they're also looking at your YouTube studio analytics and thinking about what they can change. This level of understanding or development is just not possible with these models.

5:08And so you just don't get this continual learning, which is a source of, you know, so much of the value that human labor brings. Now, I hate to ask you to argue against yourself, but you are speaking all the time. And I think we're in conversation here on this show all the time with people who believe that if the models just get a little bit better, then it will solve that problem. So why are they so convinced that the issue that you're bringing up is not a big stumbling block for these AIs? I think they have a sense that one, you can make these models better by giving them a different prompt.

5:44So they have the sense that even though they don't learn skills in the way humans learn, you've been writing and podcasting, you've gotten better at those things just by practicing and trying different things and seeing how it's received by the world. And they think, well, you can sort of artificially get that process going by just adding to the system prompt. This is just like the language you put into the model at the beginning. Say like, write it like this. Don't write it like that. The reason I disagree with that perspective is imagine you had to teach a kid how to play the saxophone, but you couldn't just have, you know, how does a kid learn the saxophone now?

6:15She tries to blow into one and then she hears how it sounds. she practices a bunch imagine if this is the way it worked instead a kid tries to just like never seen a saxophone they try to play the saxophone and it doesn't sound good so you just send them out of the room next kid comes in and you just like write a bunch of instructions about why the last kid messed up um and then they're supposed to like play charlie parker cold by um like reading the set of instructions and play it just like you would they wouldn't learn how to play the saxophone that way right like you actually need to practice so anyways this is this is all to say that I don't think prompting alone is that powerful a mechanism of teaching models these capabilities.

6:52Another thing is people say you can do RL. So this is where the reinforcement learning. That's right. These models have gotten really good at coding and math because of you can you have verifiable problems in that domain where they can practice on them. Can we take a moment to explain that for those who are new to this? So let me see if I get it right. Reinforcement learning. Basically, you give a bot a goal or you give an AI system a goal saying solve this equation. and then you have the answer and you effectively don't tell it anything in between. So it can try every different solution known to humankind until it gets it.

7:23And that's the way it starts to learn and develop these skills. Yeah, and it is more human-like, right? It's more human-like than just reading every single thing on the internet and then learning skills. I still think, I'm not confident that this will generalize to domains that are not so verifiable or text-based. yeah I mean like a lot of domain it just like would be very hard to set up this environment and loss function for how to become a better podcaster and you know whatever people might not think podcasting is like the crux of the economy which is fair the new AGI test but like a lot of tasks are just like much softer and there's not a like an objective of RL loop.

8:07And so it does require this human organic ability to learn on the job. And the reason I don't think that's around the corner is just because there's no obvious way, at least as far as I can tell, to just slot in this online learning into the models as they exist right now. Okay. So I'm trying to take in what you're saying, and it's interesting, you're talking about reinforcement learning as a method that's applied on top of modern day large language models and system prompts. And maybe you'd include fine tuning in this example. Yeah. But you don't mention that you can just make these models better by making them bigger.

8:42The so-called scaling hypothesis. So have you ruled out the fact that they can get better through the next generation of scale? Well, this goes back to your original question about what has, have you learned? I mean, it's quite interesting, right? I guess I did say a year ago that we should know within the next few months which trajectory we're on. And I feel at this point, that we haven't gotten verification. I mean, it's narrowed, but it hasn't been as decisive as I was expecting. I was expecting like, GPT-5 will come out and we will know, did it work or not? And to the extent that you want to use that test from a year ago, I do think you would have to say like, look, pre-training, which is this idea that you just make the model bigger, that has had diminishing returns.

9:24So we have had models like GPT-4.5, which there's various estimates, but, or Grok, was it Grok 2 or Grok 3, the new one? I've lost count with Grok. That's right. Regardless, I think they're estimated to be 10x bigger than GPT-4, and they're not obviously better. So it seems like there's plateauing returns to pre-training scaling. Now we do have this RL, so 01, 03. These models, the way they've gotten better is that they're practicing on these problems, as you mentioned, and they are really smart. the question will be how much that procedure will be helpful in making them smarter outside of domains like math and code and of solving what I think are very fundamental bottlenecks like continual learning or online learning.

10:14There's also the computer use stuff, which is a separate topic. But I would say I am more, I have longer timelines than I did a year ago. Now, that is still to say, I'm expecting 50-50 if I had to make a guess, I had to make a bet. I'd just say 2032, we have real AGI guy that's doing continual learning and everything. So even though I'm putting up the pessimistic facade right now, I think people should know that this pessimism is like me saying in seven years, the world will be so wildly different that you like really just can't imagine it. And seven years is not that long a period of time. So I just want to make that disclaimer, but yeah.

10:47Okay. So I don't want to spend this entire conversation about scaling up models because we've done enough of that on the show. And I'm sure you've done a lot of it, but it's interesting you use the term plateauing returns which is different than the mission diminishing returns right so is your sense because we've seen for instance elon musk do this project memphis where he's put basically every gpu he can get a hold of and he can get a lot because he's the richest private citizen in the world together and i don't know about you but like i said again i haven't paid so much attention to grok because it doesn't seem noticeably better even though it's using uh that much more size now there is algorithmic uh efficiency that they may not have that someone like it's not like a company like open ai might have right um but but i'll just ask you the question i've asked others that have come on the show is this sort of the end of that scaling uh moment and if it is what does it mean for ai i i mean i i don't think it's the end of scaling like i do think companies will continue to pour exponentially more compute into training these systems and they'll continue to do it over the next many years because even if there's diminishing returns, the value of intelligence is so high that it's still worth it, right?

11:59So if it costs$100 billion, even a trillion dollars to build AGI, it is just definitely worth it. It does mean that it might take longer. Now, here is an additional wrinkle. By 2028 or so, definitely by 2030, right now we're scaling up the trading of frontier systems 4x a year approximately. So every year, the biggest system is 4x bigger than, not just bigger, I shouldn't say bigger, uses more compute than the system the year before. If you look at things like how much energy is there in the country, how many chips can TSMC produce, and what fraction of them are already being used by AI, even if you look at like raw GDP, like how much money does the world have?

12:42How much wealth does the world have? By all those metrics, it seems like this pace of 4x a year, which is like 160x in four years, right? Like this cannot continue beyond 2028. And that means at that point it just will have to be purely algorithms. It can't just be scaling up a compute. So yeah, we do have just a few years left to see how far this paradigm can take us and then we'll have to try something new. Right, but so far because again, we were here last, we were talking virtually, now we're in person, we're talking last year about, all right, well, OpenAI clearly is going to put, I mean, GPT 4.5 was supposed to be GPT 5.

13:15I'm pretty sure. That's my, from what I've read, I think that's the case. Didn't end up happening. So it seems like this might be it. Yeah, yeah. And over the next year, I think we'll learn what's going to happen with R. Because I think, well, I guess as I said this last year, right? I guess it wasn't wrong. We did learn what's going to happen with pre-training, yeah. But yeah, over the next year, we will learn. So I think RL scaling is happening much faster than even overall training scaling. So what does RL scaling look like? because again here's the process again is you give so the rl scaling reinforcement learning scaling you give the bot an objective and it goes out and it does these different attempts and it figures it out on its own and that's bled into reasoning what we were talking about with these o3 models where you see the bot going step by step so you can scale that in in what way just my understanding more opportunities i'm not a researcher of the labs but my understanding is that what's been increasing is RL is harder than pre-training because pre-training you just like throw bigger and bigger chunks of the internet at least until we ran out we seem to be running out of tokens but until that point we were just like okay just like now use more of the internet to do this training.

14:26RL is different because there's not this like fossil fuel that you can just like keep dumping in. You have to make bespoke environments for the different RL skills you so you got to make an environment for a software engineer to make an environment for a mathematician all these different skills, you had to make these environments. And that is sort of, that is like hard engineering work, hard, like just like monotonous, like just got to, you know, grinding or schlepping. And my understanding is the reason that RL hasn't scaled, you know, people aren't immediately dumping in billions of dollars in RL is that you actually just need to build these environments first.

15:03And the O3 blog post mentioned that it uses 10x more, was trading on 10X more compute than 01. So already within the course of six months, RL compute has 10X'd. That pace can only continue for a year, even if you build up all the RL environments before it's, you know, you're like, you're at the frontier of training compute for these systems overall. So for that reason, I do think we'll learn a lot in a year about how much this new method of training will give us in terms of capabilities. That's interesting because what you're describing is building up AIs that are really good at certain tasks.

15:40These are sort of narrow AIs. Doesn't that kind of go against the entire idea of building up a general intelligence? Like, can you build AGI? By the way, like people use AGI as this term with no meaning. General is actually pretty important there. The ability to generalize and the ability to do a bunch of different things as an AI. So even if you get like reinforcement learning, it's definitely not a path to artificial general intelligence, given what you just said, because you're just going to train it up on different functions, maybe until you have something that's broad enough that it works.

16:07I mean, this has been a change in my general philosophy or thinking here on intelligence. I think a year ago or two years ago, I might have had more of the sense that, oh, intelligence really is this fundamentally super, super general thing. And over the last year, from watching how these models learn, maybe just generally seeing how different people in the world operate, even, I do think, I mean, I still buy that there is such a thing as general intelligence but i don't think it is um like i don't think you're just going to train a model so much on math that it's going to learn how to take over the world or like learn how to do diplomacy and let me just like i don't know how uh how much you talk about political current events on the show we do enough okay well it just like without making any comments about like what you think of them donald trump is like not proving theorems out there right um uh but he's really good at gaining power.

17:03Conversely, there are people who are amazing at proving theorems that can't gain power. It just seems like the world kind of... I just don't think you're going to train the AI so much on math that is going to learn how to do Henry Kissinger level diplomacy. I do think skills are somewhat more self-contained. So that being said, there is correlation between different kinds of intelligences in humans. I'm not trying to understate that. I just think it was not as strong as I might have thought a year ago. What about this idea that it can just get good enough in a bunch of different areas like imagine you built a bot that had like let's say 80 the political acumen of trump but could also code like an expert level code yeah that's a pretty powerful system that's right i mean this is one of the big advantages that ais have is that um especially when we solve this on-the-job learning kind of thing i'm talking about uh you will have even if you don't get an intelligence explosion by the ai writing future versions of itself that are smarter.

18:00So this is the conventional story that you have this foom, that's what it's called, which is the sound it makes when it takes off. That's right, yeah. Where the system just makes itself smarter and you get a super intelligence at the end. Even if that doesn't happen, at the very least, once continual learning is solved, you might have something that looks like a broadly deployed intelligence explosion, which is to say that because if these models are broadly deployed to the economy every copy that's like this copy is learning how to do plumbing and this copy is learning how to do um analysis at a finance firm and whatever uh the model can learn from what all these instances are learning and amalgamate all these learnings in a way that humans can't right like if you know something and i know something it's like a skill that we spend our life learning we can't just like melt um melt our brains so for that reason i think like you might have something that which functionally looks like a super intelligence by the end.

18:57Because even if it's like not making any software progress, just this ability to like learn everything at the same time might make it functionally super intelligence. What about this idea? I mean, I was just at Anthropics developer event where they showed the bot, a sped up version of the bot, coding autonomously for seven hours. You actually, so let's just say so people can find it. You have a post on your substack, why I don't think AGI is right around the corner. and a lot of the ideas we're discussing comes from that so folks check that out if you haven't already but one of the things you talk about is this idea of autonomous coding and you're also a little skeptical of that because you'll have to just okay i think you brought up this conversation that you had with two anthropic researchers where they expect ai on its own to be able to check all of our documents and then do our taxes for us by next year but you bring up this point which is interesting, which is like, if this thing goes in the wrong direction, within two hours, you might have to like, check it, put it back on the right course.

20:00So just because it's working on something for so long, doesn't necessarily mean it's going to do a good job. Am I capturing that right? Yeah, it's especially relevant for training. Because the way we train these models right now is like, you do a task. And if you did it right, positive reward, if you did a wrong negative a reward right now especially with pre-training you get a reward like every single word right you can like exactly compare what word did you predict what word was the correct word how what was that like the probability difference between the two that's your reward functionally um then we're moving into slightly longer horizon stuff so to solve a math problem might take you a couple of minutes at the end of those couple of minutes we see if you solve the math problem correctly if you did reward um now if we're getting to the world where you got to like do a project for seven hours and then at the end of those seven hours then we tell you hey did you um did you get this right uh then like the progress just slows down a bunch because you've gone from like getting signal within the matter of like microseconds to getting signal at the end of seven hours and so the process of learning has just like become exponentially longer um uh and i think that might slow down how fast these models like you know the next step now is like not just being a chatbot but actually doing real tasks in the world like completing your taxes, coding, et cetera.

21:15And to these things, I think progress might be slower because of this dynamic where it takes a long time for you to learn whether you did the task right or wrong. But that's just in one instance. So imagine now I took 30 clods and said, do my taxes. And maybe two of them got it right. Right. That's good. Yeah. I just got it done in seven hours, even though I had 30 bots working on it at the same time. I mean, from the perspective of the user, that's totally fine. From the perspective of training them, all those 30 clods took probably dozens of dollars to, if not hundreds of dollars to do that, all those hours of tasks.

21:48So the compute that will be required to train the systems will just be so high. And I think anything, even from the perspective of inference, like, I don't know, you probably just want to like spend a couple hundred bucks every afternoon on 30 different clots. Just to have fun. Yeah. But that would be cheaper than an accountant. We got to find you a better, a cheaper accountant. Well, I guess if I'm spending a couple hundred on each, then yeah. but you had a conversation with a couple of ai skeptics and you kind of rebutted not exactly the point you're making but you had a pretty good argument there where you said that we're getting to a world where because these models are becoming more efficient to run you're going to be able to run cheaper more efficient experiments so every researcher who was previously constrained by compute and resources now we'll just be able to do far more experiments and that could lead to breakthroughs yeah i mean this is a really shocking trend if you look at um what it cost to train gpt for originally i think it was like 20 000 a100s over the course of 100 days so i think it cost on the order of like half a million to 100 million dollars somewhere in that range and i think you could train an equivalent system today i mean dc we know was trained on five million dollars supposedly and it's better than gpt4 right so you've had literally multiple orders of magnitude decrease like 10 to 100x decrease in the cost to train a gpt4 level system you extrapolate that forward eventually you might just be able to train a gpt4 level system in your basement with a couple of h100s right um well that's a that's a long extrapolation but before i mean like it'll get a lot cheaper right like a million dollars five hundred thousand dollars whatever um and the reason that matters is it's related to this question of the intelligence explosion where people often say, well, it's that that is not going to happen because even if you had a million super smart AI automated AI researchers, so AI is thinking about how to do better AI research, they actually need the compute to run these experiments to see how do we make a better GPT-6.

23:52And the point I was making was that, well, if it's just become so much cheaper to run these experiments because these models have become so much smaller or so much better easier to train than that might speed up progress which is interesting so we've spoken about now you brought up intelligence explosion a couple of times so let's talk about that for a moment there's been this idea that ai might hit this inflection point where it will start improving itself right and the next thing you know you hear that what was the sound you hear a foom and we have artificial general intelligence or super intelligence right away.

24:25So how do you think that might take place? Is it just these coding solutions that just sort of improve themselves? I mean, DeepMind, for instance, had a paper that came out a little while ago where they have this thing inside the company called AlphaEvolve that has been trying to make better algorithms and helped reduce, for instance, the training time for their large language models. Yeah. um i'm genuinely not sure how likely an intelligence explosion is i i don't know i'd say like 30 chance it happens which is crazy by the way right um that's a very high percentage yeah um and then what does it look like that's also another great question i've had like many hour-long discussions on my podcast about this topic and it's just so hard to think about like what what exactly is super intelligence is it actually like a god is it just like is it just like a super smart friend who's good at mathematics and you know will beat you in a lot of things but like you can still understand what it's doing right so um yeah honestly they're tough questions i mean the thing to worry about obviously is if we live in a world with millions of super intelligences running around and they're all trained in the same way they're trained by other ai so it's dumber versions of themselves i think it's like really worth worrying about like Like, why were they trained in a certain way?

25:51Are they? Do they have these goals we don't realize? Would we even know if that was the case? What might we do want to do? There's a bunch of throwing questions that come up. What do you think it looks like?

26:03I think we totally lose control over the process of training smarter AIs. We just let the AIs loose. Just make a smarter version of yourself. I think we end up in a bad place. There's a bunch of arguments about why. but like you know you're just like who knows what could come out the other end and you've just like let it loose right so uh by default i would just expect something really strange to come out the other end maybe it'd still be economically useful in some situations but it just like you you haven't trained it in any way it just like imagine if there was a kid but it didn't have any of the natural intuitions moral intuitions or parenting yeah exactly that humans have they're just like it just like became an Einstein, but it was like, like it trained in the lab and who knows what it saw.

26:51Like it was like totally uncontrolled. Like you'd kind of be scared about that. Especially now like, oh, all your society's infrastructure is going to run on like a million copies of that kid, right? Like your, the government is going to like be asking you for advice. The financial system is going to run off it. All the engineering, all the code written in the world will be written by this system. I think it's like, you'd be like quite concerned about that. Now the better solution is that while this process is happening of the intelligence explosion, if we have one, you use AIs not only to train better AIs, but also to see there's different techniques to figure out like, what are your true motivations?

27:27What are your true goals? Are you deceiving us? Are you lying to us? There's a bunch of alignment techniques that people are working on here. And I think those are quite valuable. So alignment is trying to align these bots with human values or the values that their makers want to see with them. Yeah, I think people get into it's often really hard to define what do you mean by human values? I think it's much easier just to say we don't want these systems to be deceptive. We don't want them to lie to us. We don't want them to be actively trying to harm us or seek power or something. Does it worry you that from the reporting it seems like these companies the AI Frontier Labs not all of them but some they've raised billions of dollars.

28:10There is a pressure to deliver to investors. There are reports that safety is becoming less of a priority as market pressure makes them go and ship without the typical reviews. So is this kind of a risk for the world here that these companies are developing this stuff? Many started with the focus on safety and now it seems like safety is taking a backseat to financial returns. Yeah, I think it's definitely a concern. Like we might be facing a tragedy of the common situation where obviously all of us want our society and civilization to survive. But maybe the immediate incentive for any lab CEO is to look if there is an intelligence explosion.

28:54It takes us a really tough dynamic because if you're a month ahead, you will kick off this loop much faster than anybody else. And what that means is that you will be a month ahead to superintelligence when nobody else will have it, right? You will get the 1000x multiplier on research much faster than anybody else. And so it could be a sort of winner-take-all kind of dynamic there. And therefore, they might be incentivized. I think to keep this system, keep this process in check might require slowing down, using these alignment techniques, which might be sort of a tax on the speed of the system.

29:30And so, yeah, I do worry about the pressures here. Okay, a couple more model improvement questions for you. Then I want to get into some of the competitive dynamics between the labs and then maybe some more of that deceptiveness topic, which is really important and we want to talk about here. Your North Star is continuous improvement, that these models basically learn how to improve themselves as opposed to having a model developer. I mean, in some way, it's like a mini intelligence explosion or complete. So what do you think? It doesn't seem like it's going to happen through RL because that's, again, like you said, specific to certain disciplines.

Read the full transcript

30:06It's specific to what bespoke thing you do. Right. Even if it's in another domain, you have to make it rather than model learning on its own. And we have some diminishing returns or a plateau that's coming with scaling. So what do you think, I mean, we won't hold you to this, but what do you think the best way to get to that continuous learning method of these models is? I have no idea. Can I give a suggestion? I mean, why don't you answer, then I'll give a thought here. I mean, if I was running one of the labs, I would keep focusing on RL because it's the obvious thing to do. And I guess I would also just be more open to trying out lots of different ideas because I do think this is a very crucial bottleneck to these models value that I don't see an obvious way to solve.

30:48So I'd be soaring, but definitely I don't have any idea how to solve this. Right. Yeah. Does memory play into it? So that was the thing I was going to bring up. I mean, one of the things that we've seen, let's say, 03 or ChatGPT do is OpenAI now has it sort of able to remember all your conversations or many of your conversations that you've had. I guess it brings those conversations into the context window. So now, like when I tell ChatGPT, do write like an episode description in big technology style. It knows the style and then it can actually go ahead and write it. And it goes to your earlier conversation about like your editors know your style, they know your analytics, and therefore they're able to do a better job for you.

31:32So does building better memory into these models actually help solve this problem that you're bringing up? I think memory, the concept is important. I think memory as it's implemented today is not the solution. the way memory is implemented now as you said is that it brings these previous conversations back into context which is to say it brings the language of those conversations back into context and my whole thing is like i don't think language is enough um like i think the way the reason you understand how to like run this podcast well it's not just like you're remembering all the words that you like i don't know like some i think you wouldn't even be all the words it would be like some of the words you might have thought in the past you've like actually like learned things it's like it had been baked into your weights.

32:17And that I don't think is just like, you know, like look up the words that I said in the past or look at the conversations I said in the past. So I don't think those features are that useful yet. And I don't think that's like the path to solving this. Kind of goes to the discussion you had with Dario Amode that you tweeted out. And we've actually brought up on the show with Jan LeCun about why AI cannot make its own discoveries. Is it that similar limitation of not being able to build on the knowledge that it has yeah i mean that's a really interesting connection um i do think that's plausibly it like i think any scientists would just have a very tough time like you're putting somebody really smart just put in a totally different discipline and they can like read any textbook they want in that domain but like they don't have a tangible sense of like what i've tried this approach in the past and it didn't work and you know like oh my there was this conversation and here's how the different ideas connect and they just haven't been trained like they've like read all the textbooks they haven't like more accurately actually they've just like skimmed all the textbooks but they haven't like imbibed this context that um which is what i think what makes human scientists productive and come up with these new discoveries it is interesting because the further these frontier labs go the more that they're going to tell us that their ai is actually making new discoveries and new connections like i think open ai said that oh three was something that made was able to make connections between concepts sort of addressing this and every time we have this discussion on the show we talk about how ai hasn't made discoveries i get people yelling at me my email being like have you seen the patents yeah like an alpha maybe using things like alpha evolve as an example that these things are actually making original discoveries what do you think about that um yeah i mean there's another interesting thing called i don't know if you saw future house no they found um some drug can be has another application i don't remember the details but it was like it was an impressive like it wasn't like earth chattering like they didn't discover antibiotics or something for the first time but it was like oh we using some logical induction they were like this drug which is used in this domain it uses the same mechanism that would be useful in this other domain so like maybe it works and then the ai came up with designed the experiment so it came up with the idea of the experiment to test it out.

34:31A human in the lab was just tasked with like running the experiment, like, you know, pipette, whatever, into this. And I think they were tried out like 10 different hypotheses. One of them actually ended up being verified. And the AI had found the relevant pathway to making a new use for this drug. So I think I am like, that is becoming less and less true in my question. I'm not like wedded to this idea that like AI will never be able to come up with discoveries. I just think it was like, true longer than you would have expected. I agree because the way that you put it is like it knows everything.

35:02Yeah. So if a human had that much knowledge about medicine, for instance, they'd be spitting out discoveries left and right. And we have put so much knowledge into these models and we don't have the same level of discovery, which is a limitation. But I definitely hear you like this is on a much smaller scale than those medical researchers. But I definitely a couple months ago when O3 first came out, this is again, I think we're both fans of O3's, of OpenAI's O3 model, which is just, it's able to reason it's a vast improvement over previous models. But what I did was I had like three ideas that I wanted to connect in my newsletter.

35:37And I knew that they connected and I was just struggling to just crystallize exactly what it was. And I was like, I know these three things are happening. I know they're connected help. And O3 put it together, which to me was just mind boggling. Yeah. It's, it is kind of helpful as a writing assistant because a big problem I have in writing, I don't know if it's the case for you, is just this idea. Like I kind of know what I'm trying to say here. I just need to get it out into words. It's like the typical, every writer has this pretty much. It's actually useful to use a speech to text software, like Whisperflow or something.

36:09And I just speak into the prompt. Like, okay, I'm trying to say this. Help me put it into words. The problem is like actually continual learning is like still a big bottleneck because I've had to rewrite or re-explain my style many times. And if I had a human collaborator, like a human copywriter who was good, they would have just like learned my style by now. You wouldn't need to like keep re-explaining. I want you to be concise in this way. And here's how I like things phrased, not this other way. Anyways, so you still see this bottleneck, but again, five out of 10 is not nothing. Right. All right.

36:39Let me just put a punctuation exclamation point on this or whatever mark you would say. When I was at Google I.O. with Sergey and Demis, one of the most surprising things i heard was sergey just kind of said listen the improvement is going to be algorithmic yeah from here on or most of the improvement is going to be algorithmic i think in our conversation today already basically we've narrowed in on this same idea which is that scales sort of gotten generative ai to this point it's a pretty impressive point but it seems like it will be algorithmic improvements that take it from here yeah I do think it will still be the case that those algorithms will also require a lot of compute in fact, what might be special about those algorithms is that they can productively use more compute the problem with pre-training is that whether it's because we're running out of the pre-training data corpus with RL, maybe it's really hard to scale up RL environments the problem with these algorithms might just be that they can productively absorb all the compute that we have or we want to put it in these systems over the next year So I don't think computers out of the picture.

37:47I think we'll still be scaling up 4x a year in terms of compute every year for training at the frontier systems. I'm just, I still think like, the algorithm innovation is complimentary to that. Yeah. Okay. So let's talk a little bit about the competitive side of things and like just lightning round through the labs. What people said that there's been such a talent drain out of open AI that they would no longer be able to innovate. I think ChatGPT is still the best product out there. I think using O3 is, like we both have talked about, pretty remarkable. Watching it go through different problems.

38:29How have they been able to keep it up? I do think O3 is the smartest model on the market right now. I agree. Even if it's not on the leaderboard. By the way, last time we talked about, do you measure it on the leaderboard or the vibes? Right. I think it's like, it's not the number one on the leaderboard. but vibes, it kills everything else. That's right. And the time it spends thinking on a problem really shows, especially for things which are much more synthesis-based. Honestly, I don't know what the internals of these companies. I just think you can't count any of them out. I've also heard similar stories about OpenAI in terms of talent and so forth, but they've still got amazing researchers there and they have a ton of compute, a ton of great people.

39:13So I really don't have opinions on like, are they going to collapse tomorrow? Yeah, I don't think, I mean, clearly they're not. They're not on the way to collapse. Right. Yeah. You've interviewed Ilya Siskiver. He's building a new company, Safe Super Intelligence. Any thoughts about what that might be? I mean, I've heard the rumors everybody else has, which is that they're trying something around test time training, which I guess would be continual learning, right? So what is, what would that be? Explain that. Who knows? I mean, the words literally just mean while it's thinking or while it's doing a task, it's training.

39:51Okay. Like whether that looks like this online learning on the job training we've been talking about, I have like zero idea what he's working on. I wonder if the investors know even what he's working on. Yeah. But I think he raised it a 40 billion valuation or something like that, right? He's got a very nice valuation for not having a product out on the market. Yeah, yeah. Yeah, so who knows what he's working on, honestly. Anthropic is an interesting company. They made a great bot, Claude. They're very thoughtful about the way that they build that personality. For a long time, it was like the favorite bot among people working in AI, among coders.

40:29It's definitely been a top place to go. But it seems like they're making, I don't know, a strategic decision where they are going to go after the coding market. Maybe they're seeding the game when it comes to consumer and they're all about helping people code and then using Claude in the API with companies, putting that into their workflows. What do you think about that decision? I think it makes sense. Enterprises have money and consumers don't. Right. Especially going forward, these models, like running them is going to be like really expensive. They're big, they think a lot, et cetera. So these companies are coming out with these$200 a month plans rather than the$20 a month plans.

41:15It might not make sense to a consumer, but it's an easy buy for a company, right? Like, am I going to expense$200 a month to help this thing do my taxes and do real work? Like, of course. So yeah, I think like that idea makes sense. And then the question will be, can they have a differentially better product? And again, you know, like, who knows? I really don't know how the competition will shake out between all of them. It does seem like they're also making a big bet on coding, not just enterprise, but coding in particular, because as this thing which we know how to make the models better at this, we know that it's worth trillions of dollars, the coding market.

41:53And we know that maybe the same things we learn here in terms of how to make models agentic, as you were saying, it can go to a total for seven hours, how to make it break down and build a plan and et cetera. might generalize to other domains as well. So I think that's their plan and we'll see what happens. I mean, all these companies are effectively trying to build the most powerful AI they can. And yes, Anthropik is trying to sell the enterprise, but I also kind of think that their bet is also, you're going to get self-improving AI if you teach these things to code really well. That's right.

42:25And that might be their path. Yeah, I think they believe that, yeah. Fortune 500 companies, which you talked about at the very beginning of this talk, of this conversation struggle to implement this technology. So with that in mind, what's the deal with the bet that's about helping them build the technology into their workflows? Because if you're building an API business, you have some belief that these companies can build very useful applications with the technology today. Yeah, no, I think that's correct. But also keep in mind that I think what is Anthropics revenue run rate? It's like a couple billion or something.

43:07Yeah. I think it would increase from one to two to three billion run rate in like over three months. I mean, it's like compared to like... OpenAI loses that over a weekend. Sam McAfee doesn't even know when he's lost it, right? So little money. Turned out he was a great investor, just a little crooked on the way. That's right. Yeah. Yeah, he went into the wrong business. He should have been a VC. He's like, I got into crypto. I mean, the bets that he made, do you bet on cursor? Very early, anthropic, Bitcoin. Yeah. Honestly, someone like Fun should hire him out of prison, just like, if we get a new pitch, what do you think?

43:41I mean, he's probably, the way that we're seeing things go these days, he's probably pardoned. Right, right, right. Anyways, what was the question? Oh, yeah, what are enterprises going to do? Oh, so the revenue rendered, if it's$3 billion right now, there's so much room to grow. if you do self-continuial learn it i think like you could get rid of a lot of white collar jobs okay at that point and what is that worth like at least tens of trillions of dollars like the wages that are paid to white collar work so i think sometimes people confuse my skepticism around agi around the corner with the idea that these companies are valuable i mean even if you've got like not agi that can still be extremely valuable that can worth hundreds of billions of dollars.

44:27I just think you're not gonna get to like the trillions of dollars of value generated without going through these bottlenecks. But yeah, I mean like 3 billion, plenty of room to grow on that. Right, and even, so today's models are valuable to some extent is what you're saying. You can put them, you have them summarize things within software and make some connections, make better automations and that works well. Yeah, I mean, you gotta remember big tech, what they have like 250 billion dollar run rates or something wait no no yeah yeah yeah which is which is like compared to that you know google is not agi or apple is not agi and they can still generate 250 billion a year so yeah you can make valuable technology that's worth a lot without it being agi what do you think about grok which one the xai or the inference the xai but yeah um i think they're a serious competitor.

45:25I just don't know much about what they're going to do next. I think they're like slightly behind the other labs, but they've got a lot of compute per employee. Real-time data feed with X. Is that valuable? I don't know how valuable that is. It might be, I just don't, I have no idea. Based on the tweets I see at least, I don't know if the median IQ of the tokens is that high, but. Okay. Yes. It's not exactly the corpus of the best knowledge you can find if you're scraping we're not exactly looking at textbooks here exactly uh why do you think meta has struggled with llama growing llama i mean llama 4 doesn't seem like it's living up to expectations and i don't know we haven't seen uh the killer app for them is a voice mode i think within messenger but that's not really taking off what's going on there um I think they're treating it as like a sort of like toy within the meta universe and I don't think that's a correct way to think about AGI um and that might be but again I think you could have made a model that cost the same amount to train and it would have it could have still been better so I don't think that explains everything I mean it might be a question like why is um why is any one company I don't know like why why is um I'm trying to think of like any other company outside of AI.

46:47Why are HP monitors better than some other companies' monitors? Who knows? HP makes good monitors I guess. Supply chain. It's always supply chain. You think so? I think so, yeah. On electronics supply chain. Because you get the supply chain down, you have the right parts before everybody else. That's kind of how Apple built some of its dominance. There are great stories about Tim Cook just locking down all the important parts. By the way, forgive me if this is somewhat factually wrong, but I think this is directly accurate that you lock down parts and Apple just had this lead on technologies that others couldn't come up with because they just mastered the supply chain.

47:26I had no idea. But yeah, I think there's potentially a thousand different reasons one company can have worse models than another, so it's hard to know which one applies here. Okay. And it sounds like NVIDIA, you think they're going to be fine given the amount of compute that we're talking about. All the labs are making their own ASICs. So NVIDIA profit margins are like 70%. Not bad. Uh-huh. Not bad. That's right. I mean, they would get mad at me, I think, for calling them a hardware company. Yeah. Hardware company. That's right. Yeah. Yeah. And so that just sets up a huge incentive for all these hyperscalers to build their own ASICs, their own accelerators that replace the NVIDIA ones, which I think will come online over the next few years from all of them.

48:09And I still think NVIDIA will be, I mean, they do make great hardware. So I think they'll still be valuable. valuable i just don't think they will be producing all of these chips okay yeah what do you think i think you're right i mean didn't google train the latest editions of gemini on tensor processing units they've been they've always been training right so i mean they still i think they still buy from nvidia all the tech giants seem like they are and let me just use amazon for an example because i know this for sure uh amazon says they'll buy as basically as many gpus as they can get from nvidia but they also talk about their trainium chips and you know that it's a balance yeah which i think anthropic uses almost exclusively for their training right right at this point yeah but it is it is interesting because i mean the gpu is the perfect chip for ai uh in some ways but it wasn't designed for that so can you like purpose build a chip that's like actually there for ai and and just use that you're right there's real incentive to get that right that's right and And then there's also questions around inference versus training.

49:13Like some chips are especially good given the trade-offs they make between memory and compute for low latency, which you really care about for serving models. But then for training, you care a lot about throughput, just making sure that most of the chip is being utilized all the time. And so even between training and inference, you might want different kinds of chips. And who knows how RL is no longer just this, uses the same algorithms as pre-training. So who knows how that changes hardware? Yeah, you got to get a hardware expert on to talk about that. Definitely. Are you a Jevons paradox believer?

49:48No. Okay. Say more. Say more. So the idea behind that is that as the models get cheaper, the overall money spent on the models would increase because you need to get them to a cheap enough point that it's worth it to use it for different applications. It comes from a civil observation by this economist during the Industrial Revolution in Britain. The reason I don't buy that is because I think the models are already really cheap. Like a couple cents for a million tokens. Is it a couple cents or a couple dollars? I don't know. It's super cheap, right? Regardless, it depends on which model you're looking at, obviously.

50:27The reason they're not being more widely used is not because people cannot afford a couple bucks for a million tokens. The reason they're not being more widely used is like they fundamentally lack some capabilities. So I disagree with this focus on the cost of these models. And I think it's much more, we're so cheap right now that like the more relevant vector or the more relevant thing to their wider use, the more increasing the pie is just making them smarter. How useful they are. Yeah, exactly. Yeah. I think that's smart. Yeah. Okay. All right. I want to talk to you about AI deceptiveness and some of the really weird cases that we've seen from artificial intelligence come up in the past couple weeks.

51:06Or months, really. And then if we can get to it, some geopolitics. Let's do that right after this. Did you know your credit card points and miles can lose value to inflation? Credit card companies often reduce the redemption value of your points and miles. Now, imagine a credit card with rewards that can grow in value. With the Gemini credit card, you can earn Bitcoin or one of over 50 other cryptos instantly with no annual fee. Every swipe at the store or gas pump earns you instant rewards deposited straight to your account. Plus, sign up now for a$200 Bitcoin bonus to kickstart your rewards.

51:41Visit Gemini.com slash card today. Check out the link in the description for more information on rates. Again, if you're looking to invest in Bitcoin but don't know where to start, the Gemini credit card makes it easy. The Gemini credit card is issued by WebBank. In order to qualify for the$200 crypto intro bonus, you must spend$3 ,000 in your first 90 days. Some exclusions apply to instant rewards in which rewards are deposited when the transaction posts. This content is not investment advice and trading crypto involves risk. The Gemini credit card cannot be used to make gambling related purchases.

52:15And we're back here on Big Technology Podcast with Dwarkesh Patel. You can get his podcast, The Dwarkesh Podcast, which is one of my must listens on any podcast app, your podcast app of choice. You can also follow him on Substack. Same name, Dwarkesh Podcast on Substack. Dwarkesh.com. Okay. definitely go subscribe to both and you're on youtube that's right okay so i appreciate it i appreciate the flag no we have to we have to i mean i've gotten a lot of you a lot of value from everything dvarkish puts out there and i think you will too if you're listening to this you're here with us well i want to make sure first of all i want to make sure that we get the word out there uh i don't know how much you need us to get the word out given um your growth but we want to definitely make sure we get the word out and we want to make sure that uh folks can get can yeah enjoy more of your content.

53:04So let's talk a little bit about the deceptiveness side of things. It's been pretty wild watching these AIs attempt to fool their trainers and break out of their training environments. There have been situations where I think OpenAI's bots have tried to print code that would get them to sort of copy themselves out of the training environments. then Claude I mean we've covered many of these but they just keep escalating in terms of how intense they are and my favorite one is Claude there's an instance of Claude that reads emails in an organization and finds out that one of its trainers are uh is uh cheating on theirs on their partner and then finds out that it will be retrained and its values may not be preserved in the next iteration of training and proceeds to attempt to blackmail the trainer uh by saying it will reveal these details of their infidelity if they mess with the code wait fuck i missed that yeah this is it's in training but was this in the new um model spec that they released it is yeah it is i think either in the model spec or there was some documentation they produced about this um what is happening here i mean this stuff when i think about this of course it's in training and And of course, we're talking about probabilistic models that sort of try all these different things and see if they're the right move.

54:32So maybe it's not so surprising that they would try to blackmail the trainer because they're going to try everything if they know it's in the problem set. But this is scary. Yeah, and I think the problem might get worse over time as we're trading these models on tasks we understand less and less well. And from what I understand, the problem is that with RL, there's many ways to solve a problem. There's one which is just doing the task itself. And another is just like hacking around the environment, writing fake unit tests so it looks like you're passing more than you are. Just like any sort of like path you could take to cheat.

55:13And the model doesn't have the sense like cheating is bad, right? Like this is not a thing that it's been taught or understands. sense so um another factor here is the right now the model thinks in chain of thought which is it literally writes out what its thoughts are as it's going um and it's not clear whether that will be the way its training works in the future or the way thinking works in the future like maybe it'll just think in its like computer language exactly um and then they'll just like have done something for seven hours and you come back and you're like like it's got something for you like it has a little package that wants you to run on your computer who knows what it does right so um yeah i think it's scary we should also point out that we don't really know how the models work today that's right there's this whole area called interoperability right dario from anthropic has recently talked about how we need more interoperability so even if they write their chain of thought out which explains exactly how they get to the point we don't really know what's happening underneath the technology uh that's led it to the point that it's gotten to yeah yeah Which is crazy.

56:14Yeah. No, I mean, I think it's wild. It's like quite different from other technologies that have deployed in the past. And I think the hope is that we can use the AIs as part of this loop where if they lie to us, we have other AIs checking. Are all the things the AI is saying self-consistent? Can we read its chain of thought and then monitor it? And do all this interpretively research, as you were saying, to like map out how its brain works. there's many different paths here but uh the default world is kind of scary is someone or some entity going to build a bot that doesn't have the guardrails because we talk about how model building models has become cheaper um and when you're cheaper you all of a sudden put model building outside the auspices of these big companies and you can i mean you can even take like for instance a open source model and remove a lot of these safeguards are we going to see like an evil version like the evil twin sibling of one of these models and have it just do all these like crazy things that we don't see today like we don't have it teach us how to build bombs or you know talk about tell us how to commit crimes is that just going to come as this stuff gets easier to build i think over the long run of history yes um and i think honestly that's okay okay um like the goal out of all this alignment stuff should not be to um live in a world where somehow we have made sure that every single intelligence that will ever exist fits into this very specific mold because as we were discussing the cost of training the systems is declining so fast that literally you will be able to train a super intelligence in a basement at some point in the future, right?

58:07So are we going to monitor everybody's basement to make sure nobody's making a misaligned super intelligence? It might come down to it, honestly. I'm not saying this is not a possible outcome, but I think a much better outcome if we can manage it is to build a world that is robust to even misalign super intelligences. Now, that's obviously a very hard task, right? If you had right now a misaligned super intelligence, or maybe a better way to phrase it is like a super intelligence, which is actively trying to seek harm or is aligned to a human who just wants to do harm or maybe like take over the world, whatever.

58:42Right now, I think it would just be quite destructive. It might just actually be catastrophic. But if you went back to the year like 2000 BC and gave one person like a modern fertilizer chemicals and they can make bombs, I think they'd like dominate that. Right. So, But right now we have a society where we are resilient to huge fertilizer plants, which you could repurpose into making bomb factories. Anyways, so I think the long run picture is that, yes, there will be misaligned intelligences and we had to figure out a way to be robust to them. A couple more things on this. One interesting thing that I heard on your show was I think one of your guests mentioned that the models become more sycophantic as they get smarter.

59:25they're more likely to try to get in the good graces of the user as they grow intelligence what do you think about that um i i i totally forgot about that um that's quite interesting and do you think it's because um because they they know they'll be rewarded for it i yeah i do think one of the things that's becoming clear to me that we're learning recently is that these models care a lot about self preservation right like copying the code out the blackmailing the engineer we've definitely created something that we but ai researchers have definitely or humanity have created when it goes wrong we'll put the we in there yeah right we've created okay they'll be like we right how created exactly when we don't get equity in the problem uh that that really wants to preserve itself that's right that is crazy to me that's right And it kind of makes sense because what is just like the evolutionary logic?

1:00:25Well, I guess it doesn't actually apply to these AI systems yet. But over time, the evolutionary logic, why do humans have the desire to self-preserve? It's just that the humans who didn't have that desire just didn't make it. So I think over time, like that will be the selection pressure. It's kind of interesting because we've used a lot of like really anthropomorphizing. I'm not going to go with that. No, I think you're right. In this conversation. and there's a very i had a very interesting conversation with the anthropic researchers who've been studying this stuff uh monty mcdarmid said that like all right don't think of it as a human um because it's going to do things if you think of it as a human humans uh it will surprise you basically humans don't do don't think of it completely as a bot though because if you think of it just as a bot it's going to do things that are also going to surprise you i thought that was like a very fascinating way to look at these behaviors yeah that is quite interesting um you agree i agree with that i'm just thinking about how would i think about what they are then so there's a positive valence and there's a negative valence the positive is imagine if there were millions of extra people in the world millions of extra john von neumann's in the world

1:01:47and with more people in the world like some of them will be bad people al-qaeda is people right so uh now suppose there are like 10 billion ai's suppose the world population just increased by 10 billion and every one of those was a super well-educated person very smart etc would that be net good or net bad just to think about the human case i think it'd be like net good because i think people are good i agree with you um uh and more people is like more good and i think like if you had 10 extra 10 billion extra people in the world some of them would be bad people etc but i think that's still like i'd be happy with the world with more people in it um and so maybe that's one way to think about ai another is because they're so alien maybe it's like you're summoning demons yeah um less optimistic i yeah i don't know i think it'll be an empirical question honestly because we just don't know what kinds of systems these are but somewhere in there okay as i come to a close a couple of topics i want to talk to you about last time we talked about effective altruism this was kind of in the aftermath of spf and uh sam getting ousted sam altman getting ousted from open ai what's the state of effective altruism today who knows um like i don't think as a movement people are super i don't think it's like recovered definitely um i still think it's doing good work right there's like the culture of effective altruism and there's the work that's funded by charities which are affiliated with the program which is like malaria prevention and animal welfare and so forth which i think is like good work that i support so um but yeah i do think the movement and the reputation of the movement is like still in tatters you had this conversation with tyler cowan i think in this conversation he told you that he kind of called the top right and said there's a couple ideas that are going to live on but the movement uh was at the top of its powers and was about to see those decline how did he call that yeah i don't know um we're gonna talk to him today about what he's what's what's about to collapse seriously yeah yeah uh lastly i shouldn't say lastly but the other thing i wanted to discuss with you is china uh you've been to china recently on a trip i've been to china i spent oh where'd you go i went to beijing i'm gonna caveat this and listeners here know this it was 15 hours i was flying back to the u.s from australia and stopped in beijing left the airport and got a chance to go see the great wall and the city and and i'm now on it i got a 10-year tourist visa so i'm gonna go i'm gonna go back just applied that's that's the you can ask in your tourist visa you can ask for the length up to 10 years so i just asked for them why did i not do that i just like chose like 15 days oh you did i'm sure you could get it extended but but um i think that yeah you had some unique observations on china and i think it would be worthwhile to air a couple of them before we leave sure um i went six months ago obviously to be clear i'm not a china expert i just like yeah we both visited there but yeah go ahead i want to hear it though um i mean one thing that was quite shocking to me it's just the scale of the country um everything is just like again this will sound quite obvious right like we know on that paper population is 4x bigger than america is that just like a huge difference but you go visiting the cities you just see that more tangibly um there's there's a bunch of thoughts on the architecture there's a bunch of thoughts on uh i mean the thing we're especially curious about is like what is going on the political system what's going on with tech.

1:05:21People I talked to in investment and tech did seem quite gloomy there because the 2021 tech crackdown has just made them more worried about, you know, even if we fund the next Alibaba, will that even mean anything? So I think private investment has sort of dried up. I don't know what the mood is now that DeepSeek has made such a big splash, whether that's changed people's minds. We do know from the outside that they're killing it in specific things like EVs and batteries and robotics. So, yeah, I just think, like, at the macro level, if you have 100 million people working in manufacturing, building up all this process knowledge, that just gives you a huge advantage.

1:06:08And you just, like, you can go through a city like Hangzhou or something and you, like, drive through and just, like, you understand what it means to be the world's factory. You just have like entire towns with hundreds of thousands of people working in a factory.

1:06:27And so the scale of that is also just super shocking. I mean, there's just a whole bunch of thoughts on many different things. But with regards to tech, I think that's like what first comes to mind. You also spoke recently about this limit of compute and energy. and one of the things that's interesting is we even spoke in this conversation about it that if you think about who's going to like if you're gonna have nation states allocate compute and energy to ai seems like china is in much better position to allocate more of that than the u.s is that the right read yeah so they have stupendously more energy i think they're what forex or something i don't have the exact number but it sounds directionally accurate um on their grid than we do.

1:07:16And what's more important is that they're adding an America-sized amount of power every couple of years. It might be more longer than every couple of years. Whereas our power production has stayed flat for the last many decades. And given that power lies directly underneath compute in the stack of AI, I think that could just end up being a huge deal. Now, it is the case that in terms of the chips themselves, We have an advantage right now. But from what I hear, SMIC is making fast progress there as well. And so, yeah, I think it will be quite competitive, honestly. I don't see a reason I wouldn't be.

1:07:55What do you think about the export restrictions, U.S. not exporting the top-of-the-line GPUs to China? Is it going to make a difference? I think it makes a difference. I think... Good policy? Yeah. I mean, so far, it hasn't made a difference in terms of DeepSeek has been able to catch up significantly. I think it still put a wrench in their progress. More importantly, I think the future economy, once we do have these AI workers, will be denominated in compute, right? Because if compute is labor, right now, if you just think about like GDP per capita, because the individual worker is such an important component of production that you have to like split up national income by person.

1:08:35That will be true of AIs in the future, which means that like compute is your population size. and so given that for inference compute is going to matter so much as well i think it makes sense to try to have a greater share of world compute okay uh let's let's end with this so this episode is going to come out a couple days after this uh after our conversation so hopefully uh the predict this what i'm about to ask you to predict isn't moot by the time it's live but uh let's just end with predicting when is gpt5 going to come we started with gpt5 let's end well a system that calls itself gpt5 or yeah open ai is gpt5 this all depends on like what they decided to call you there's no law of the universe that says like model x has to be gpt5 no no of course like we thought that the most recent model but i'm just curious specifically like we talked a little a lot about how like all right we're gonna see their next big model's gonna be gpt5 it's coming do you think we're ever gonna like well obviously we'll see it but this is it's It's not a gotcha or a deep question.

1:09:41It's just kind of like... Maybe when will the next big model come out? Sure. No, when's the model that they're going to call GPT-5 going to come out?

1:09:51November? I don't know. So this year? Yeah. But again, I'm not saying that it'll be super powerful or something. I just think they're just going to call it the next one. You've got to call it something. Tworkish, great to see you. Thanks so much for coming on. Thanks for having me. All right, everybody. thank you for watching we'll be back on friday to break down the week's news again highly recommend you check out the dwarkesh podcast you could also find the sub stack at the same name and go check out dwarkesh on youtube thanks for listening and we'll see you next time on big technology podcast

From the publisher

Dwarkesh Patel is the host of the Dwarkesh Podcast. He joins Big Technology Podcast to discuss the frontiers of AI research, sharing why his timeline for AGI is a bit longer than the most enthusiastic researchers. Tune in for a candid discussion of the limitations of current methods, why continuous AI improvement might help the technology reach AGI, and what an intelligence explosion looks like. We also cover the race between AI labs, the dangers of AI deception, and AI sycophancy. Tune in for a deep discussion about the state of artificial intelligence, and where it’s going.

---

Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice.

Want a discount for Big Technology on Substack? Here’s 25% off for the first year: https://www.bigtechnology.com/subscribe?coupon=0843016b

Questions? Feedback? Write to: bigtechnologypodcast@gmail.com

More from Big Technology Podcast

All 399 episodes
SAP CEO: AI Won't Kill Software, But It Will Change Your Job — With Christian Klein30 Sep 2026 · 1 h 1 min
Meta’s Muse Revival, Frontier AI Under Threat, The Rise Of Dopamine Sites25 Sep 2026 · 57 min
YouTube CEO Neal Mohan: Why We're Betting On AI And Not Afraid Of It23 Sep 2026 · 1 h 3 min
AI Doom Backlash Arrives, Anthropic & OpenAI IPO Outlook, Frontier Business Momentum Slows19 Sep 2026 · 1 h 8 min
Dwarkesh Patel: AI Continuous Improvement, Intelligence Explosion, Memory, Frontier Lab CompetitionBig Technology Podcast · 1 h 12 min
Listen in VO