Who Controls AI Acceleration? Vitalik Buterin and Guillaume Verdon Debate

29:51And basically, we have just benefited from a massive step up in every same thing that we hear about. And this is like something like I've even seen, like even that, you know, observing the, like my, you know, my grandparents' home, like basically go up from having this like very outhouse toilet in the backyard where there's probably flies buzzing. And I would totally hate it. And I'd have to go out, like I'd often go out to the forest to poop because I couldn't stand the flies to something that's like actually very modern and less fatable. Right. And, you know, the world has become cleaner. The world's become more beautiful.

30:34The world's become more enjoyable. The world's become better for health. It's been able to sustain more of us. It's become more interesting. And these things are really good and beautiful for us. At the same time, I think we need to recognize the role of explicit human intention in making a lot of those things happen. So, for example, in the 1950s, there was a lot of smog everywhere in the air. And people decided smog is a problem. Smog sucks. And we need to do a bunch of stuff to get rid of the smog issue. And now smog is not a problem. At least much less of one. Then, you know, we have the ozone form issue, and then we actually did things to address that, right?

31:22And then the other thing is, you know, that especially with rapidly accelerating technology and AI, I basically see two kinds of risks. One kind of risk is multipolar risks, which is basically the risk that, you know, lots of people will use the technology to do very bad things, right? And there's a concern that, like, one type of concern is sort of the equivalent of, you know, anyone being able to, like, get a nuke at 7-Eleven, like, sort of thing. And then there's also the concern of, like, well, AI itself is, you know, like, something that literally is a mind of its own, right? And especially once it becomes powerful enough that it acts without human involvement, then, you know, like, what will it do?

32:12And then there's unipolar risks, which is basically, I think actually a single AI itself is one of them. And the other one is like AI, like enabling or the combination of AI and other modern technologies enabling like permanent dictatorship that you cannot escape. Like that deeply worries me, right? Like this is something I follow, right? And, you know, in like, again, in Russia, for example, on the one hand, the toilets have gotten much better. On the other hand, like, it's gone from protesting being possible to protesting being this sort of thing where if you do it, the cameras will see you.

32:50And then, you know, a week later, you get a knock on the door at 2 a.m., right? And this AI is supercharging this. You know, there's a lot of concentration of power happening. And, like, both of these things really worry me, right? And to me, DEAC is really attempting to chart a path forward that continues this acceleration and accelerates it, but at the same time really deals with both kinds of risks. So you would say that DEAC is emphasizing specific other categories of risk that are maybe less emphasized than you'd like to see in EAC? I think there's many kinds of risks of technology, and many of them are valid.

33:33And, I mean, they have different scales. Like, some of them become more salient in different models of the world and how fast these are happening. But, I mean, I think there's a lot that we can do to really, like, push against all of those kinds of risks. Right. So, Gil, yeah. Gil, do you want to say a little bit? What was the question again? Oh, well, just compare and contrast EAC and VAC. Yeah, I think actually Vitalik and I are very concerned about over-concentration of power that can happen with AI. And that was a big part of the EAC movement, especially at the beginning. It was pro-open source.

34:14We want to diffuse AI power because, you know, our worry was that the AI safetyism meme was so potent that certain power-seeking individuals could weaponize it to consolidate control over AI and convince you you shouldn't have access to AI for your own good. And really, if you have a gap in cognition between the individuals and the centralized entities, they will control you. They can have a full world model of everything going on in your brain, and they can prompt, engineer, and effectively steer you. So you want to symmetrize AI power. We don't want to, just like Second Amendment is about the government not having a monopoly on violence so we can vibe check the government if it goes out of hand.

34:58You need that for AI. So we need everybody to be able to own their own models, own their own hardware for that technology to be diffused, for the power to be diffused. But to me, I think discussions of stopping AI research and AI progress, that's completely out of the question. AI is a very fundamental technology. It's almost a meta technology, technology that produces technology. It gives us predictive power over our world. It can be added on to any task we want to do in the world. It could be tacked on to any technology and turbocharge it. It accelerates the acceleration. The acceleration is this complexification where things become lower friction.

35:41Things just become better. You know, our bodies feel comfortable because we have this sort of, this estimator we call happiness of like, what's my estimator on expected persistence of my bits? That's what we're hard-coded for. And so, I think to me, the EA, effective altruists, hedonic utilitarianism is maybe the wrong way to view things like maximizing happiness. And to me, I want to have an objective measure of progress and that's what the EAC framework is. It's, hey, actually, the objective view is like, how are we progressing as a civilization? Are we scaling up? Because to me, you know, you have to complexify, you have to have more intelligence, things have to improve in order for you to scale, right?

36:28It's like the ultimate benchmark. And at the same time, you know, there can be setbacks. Like, you know, Vitalik said, you know, if AI power would be over-concentrated in the hands of a few, that would be net bad for growth because it's much better if that technology is very diffused. In that respect, we're very aligned, right? Can I jump in here? Because I think you're touching on something that I think both of you share a lot of deep ethos. I mean, obviously, Vitalik has produced a lot of MIT open source code, although I know you have some more updated feelings about GPL and such. But obviously, both of you have been champions of open source and now open hardware.

37:10And these have been separate things, but now that we're seeing people start to, like Talos, like putting weights on two chips, A6, these kinds of things, they're starting to become very similar. So I'm very curious, like, both what your thoughts are on open weights and open hardware. I mean, you're both actually like pretty deep in hardware right now. And then also, like, what is the difference between EAC and DAC with regards to this? This has been a crazy week, obviously, where like a lot of the things you're talking about have been tested, where you have the government and corporations trying to figure out what the right answer is.

37:41And so I'd love to kind of know what you guys are thinking just based on that this week. I'd love to tease out if there are any kind of differences between you and where you think this goes. Yeah, I mean, I think, you know, to me, open source accelerates the search over a hyper-pre-and-wires. It makes our models better. We can kind of collaborate sort of like a swarm and traverse design space, right? And that's what acceleration allows us to do, right? With better technology, with more AI, now AI for coding, that search process over design space for AI itself is accelerating. You know, I think, you know, we're going to open source our superconducting hardware designs very soon.

38:25I just want to stagger it with our launch, but I think diffusing knowledge is also diffusing power, right? And diffusing knowledge when it comes to how to produce intelligence is super important. We don't want to give, you know, there were discussions apparently in the last administration, according to Marc Andreessen, that the U.S. government might want to put the genie back in the bottle and maybe ban, not ban linear algebra, but more or less ban the math surrounding AI. And to me, that would be almost like banning knowing about biology would be a huge step back. And so there's no going back, right?

38:59This knowledge is out there. If you try to ban it in the U.S., some other country, a third party, some deregulated island somewhere is going to keep developing it. and then you're going to have a huge gap and now you have a big risk. So to us, the biggest risk is a gap in capabilities and the way to reduce that risk is to make sure AI power is diffuse. So whenever there's like, you know, the AI doomerism, like, oh, be very afraid. We're the ones responsible. We're the ones who should be put in charge. Trust us. You know, I just get very skeptical because even if they're well-meaning, like we saw this week, right, they could just get pushed out if they centralize too much power.

39:40It's too juicy for those that want that power. And so that's kind of what we were warning about for years, and now it kind of happened. And, you know, Dario's licking his wounds, and, you know, some lessons learned there in sort of real politic, right? And anyways. So, yeah, Vitalik, what do you think of all this? The two kinds of risks that, you know, like I think about, right, are unipolar risks and multipolar risks, right? And I think, I mean, with unipolar risks, I mean, you know, the anthropic situation is like so fascinating, right? Because, you know, ultimately, yeah, the thing that like they got dinged for is refusing to let their AI be used for specifically, fully autonomous weapons and mass surveillance of Americans, right?

40:38And so it will presumably, you know, if there's a chance that, you know, it looks like the government and military of this country wants to do mass surveillance of Americans, right? And, you know, this is an example of unipolar risk, right? I think basically, you know, like surveillance is one of these things where the big effects that it has is it takes whoever is stronger and makes them even stronger, right? It removes spaces where pluralism can form, where counter-elites can coalesce themselves, and where people can safely explore alternatives. And like this is, you know, surveillance is one of these things that easily can be supercharged, right?

41:28I think actually on the defense, getting back to open hardware a bit, actually just to talk about one of the projects that we've been doing. So a big part of what I've done in IDEAC is basically supported various projects that develop open source defensive technologies. So technologies that will make it easy for all of us to continue to be safe and protected in a world where more powerful and crazy capabilities exist, right? And so in the bio world, for example, this means rapidly leveling up our civilization's ability to withstand pandemics. And so I claim that it is very within reach for us to have China-level COVID resistance at the same time as Sweden-level interference to people's regular lives.

42:24And that's even the minimum bar. And this basically involves stacking filtration, UVC testing, like literally a company we invested in as fully open source again, like the end product of this is like basically passively testing the air and being able to tell if there's COVID in the air, right? Like in general, right? Like essentially the number of sensors in the world is going to go up, right? And sensors are a big part of like being able to act better in the world, right? But at the same time, sensors mean surveillance, right? And the thing that we're doing is actually this project, we gave out some of these at DEF CON.

43:04And what these are is they're sensors that collect air quality information, CO2, AQI, a few other things. And they locally basically encrypt, anonymize differential privacy them, and then FHE encrypt. And that gets sent off to a server, and the server is able to basically compute all over all of the data and then collectively decrypt the final answer without being able to see any input from any individual person. And this is basically, the goal is to deliver the higher levels of safety, but at the same time protect people's privacy and protect against the multipolar risk and the unipolar risk at the same time.

43:58And I think this is how we can collaboratively, as a world, work together to build something better. And I think for hardware, basically, I think we need open hardware and we need verifiable hardware. We need every camera in this room to prove basically what kind of cameraing it is doing. In my ideal world, if you have, fine, you can have a million cameras in the streets to prevent people from, like, or detect when people are, like, engaging in violence against each other. But ideally, you'd have, like, attestations, signatures over LLM, and, like, a public right of inspection. And you'd be able to, like, inspect these things and verify that, like, the only thing that they do is check when people are doing violence and alert that, right?

44:42So, like, these kinds of technologies. Yeah, so the verifiable hardware idea, very interesting, especially because it's not something that I think comes up very often. But can I just ask a very stupid question, which is just, is open hardware, verifiable hardware, is that an EAC thing or DEAC thing? I don't know if I ever talked about open hardware. The thing I talk about, to me, the greatest risk is a gap in intelligence between centralized entities and decentralized entities, so individuals versus the government. And so right now, with the current compute paradigm, to run a very smart AI model, you need a huge cluster with hundreds of kilowatts.

45:20that is not accessible to the individual. People want to own and control the extension of their cognition. That's why we saw the open claw Mac mini craziness of the past few weeks. So people are clamoring for that. The only way where you can symmetrize power between the individual and centralized entities is if there's a densification of intelligence. We need AI hardware that's far more energy efficient so you could plug it into a wall and you could own the extension to your cognition because this year what's going to happen, the models are going to start online learning and they're going to become extremely sticky.

45:53It's going to be like trying to change executive assistants. At the risk of sounding dumb, isn't that what we're already doing? Aren't we already trying to radically decrease the cost of compute at an extraordinarily exponential pace? I'm trying to understand what is the additional information that we are trying to inject into the zeitgeist by codifying an idea as EAC or codifying an idea or set of ideas as DAC? I think for me, it's like, and it's part of the rest of my mission, you know, with my company, Extropic, it's getting more intelligence per watt will drastically increase the amount of intelligence we produce, and it will also help us climb the Kardashev scale by Javon's paradox.

46:34If you can convert energy into intelligence or, you know, energy into value by proxy more readily, there's going to be more demand for energy, and that's going to lead to improvement and complexification of civilization. So to me, that's the most important tech problem because that's what's going to diffuse AI power, right? And open hardware is one way to diffuse AI power, but to me, anything von Neumann, anything digital is going to look like caveman-era hardware, truly. No, it's just... I can't wait. I really can't. I'm very excited. It's coming, right? So doesn't capitalism already, through just natural incentives and so on, already allocate hundreds of billions of dollars at a minimum to this per year?

47:16I don't think there's that much investment in alternative hardware. In alternative hardware. Yeah, alternative hardware. Conductors and energy production, I think GAC is all about diffusing. It's about maintaining variance, not collapsing entropy of our search over any design space, whether it's policies, cultures, whatever. Technology, we need alternative bets. We need more alternative bets that are out there. It can't just be the green monster eating all the profits. Then there's kind of hyperparameter space staking risk. we have this design space, we're over-investing in the current technology, and that might lead to correction, which, you know, a correction is decel, right?

47:54Like, because not everything pumping up into a smooth exponential. Can I just declare we solved it and they agree completely? I think on the idea of open source and, like, this seems very defensive, like defensive technology in the Vitalik way, and it seems like you guys are very aligned on this, actually. And that gives me hope, because this is the stuff I care about. I think that, like, right now, there are a lot of people who are, like, a lot, why does this exist? Because a lot of people are, like, very uncertain about the future. And what appeals to them is that you're saying, like, it's going to be fine.

48:30It's baked in. And so maybe, like, if I were to steel man your case here, you're saying, like, you guys are actually saying the same thing, which is it's kind of already priced in. It's good. The only thing stopping us is kind of our bad feeling about it, right? Well, I'm asking that. I'm trying to understand where, if, yeah, go ahead. Yeah, I guess it's completely natural if sort of there's very high entropy in like your model rollouts of the future, right? There's kind of a, not a fog of war, but it's kind of hard to extrapolate what's going to happen in the next several years that gives people anxiety.

49:03Your body has this, evolved this sense of anxiety to kill entropy in the world, right? If I put my phone on the edge and, you know, I just want to grab it so it doesn't fall, right? You get, see, there you go. That was anxiety. So you want to take action in the world, right? So that's kind of what is happening now. But at the same time, if you kill entropy, you're missing out on the upside, right? You're missing out on the huge benefits. Right now, our whole techno capital machine has had a very long time to equilibrate with our current capabilities. If you have a disruptive capabilities that comes in, suddenly the whole landscape changes.

49:42So the whole system has to refactor, reconfig. Doesn't mean we're going to run out of jobs. We're going to do much more, right? Now that we have the ability to handle more complexity with less energy, right, with AI, we're going to be able to do much harder tasks that are higher complexity and higher payoff. I don't know about you, but I can't like overnight yet vibe code a whole tokamak. We're not there yet, but we might get there. And then we'll have a ton of energy. And that's going to help support more human headcount and grow population and help us be more comfortable. So there's a period of discomfort, but if you're in a rapidly changing landscape, the worst thing you can do is kill variants and be not plastic, be stiff.

50:25To be plastic, you need to be hedging your bets. You need to be trying many things, fucking around and finding out. The famous FAFO algorithm, it's an evolutionary algorithm. We need to try different policies. We need to try different technology trees. We need to try different algorithms. We need to try open source, closed source. We need to try it all because we don't know what the future looks like. So we got to hedge our bets. And one variant of policy, one choice of policy or several, one choice of technology or several are going to make it. And then we're all going to be in that slipstream and follow that.

51:03I think the fallacy of thinking there's a finite amount of jobs, it's very pervasive. Let me kind of try to bring it back a little bit. My understanding of disagreement, if there is, between EAC and DEC is something to do with how we steer the process of technological progress. It has to do with how it is steered. Maybe Vitalik, could you say a little bit about how is it steered, how ought it be steered, how much control do we have over that steering? Yeah, and so I think, I mean, And DIAC is definitely kind of explicitly, I don't want to quite say, you know, like sailing against the techno-capital current.

51:48I think the better analogy is like it's trying to actively shape the techno-capital current in certain ways. And I mean, one of the ways that I think about this is, basically it's a matter of making the world safer for pluralism, right? And if you think about some of these ideas around how do we improve things like biosafety, or what does it look like to have vastly better cybersecurity and have bug-free operating systems within a few years. or if you think about the bug-free code has been in the memetic space of obviously absurd naive pipe dream for two decades. It is going to flip out of that space faster than most people expect.

52:41Within Lean Ethereum, we've managed to machine-proof entire mathematical theorems that are upstream of things like Starks. And so we're very excited about this. And basically, I think there is a, like, Diak definitely has this goal of saying, like, yes, we want to, at the very least, do all of these other things to make sure that the world is actually able to deal with all of this technological growth in a way that minimizes the destructive aspects and also the centralizing aspects. And I think that doesn't happen automatically, right? And, you know, right now, I mean, I don't control any countries.

53:30I don't control any armies. I'm just like throwing my dollars and ease at it and, you know, saying words and hopefully inspiring people to also build things in a similar spirit. I think there are definitely political and legal reforms that could make the world more DEAC-friendly. There is definitely such a thing as engineering legal incentives, for example, to motivate a much more rapid shift to total cybersecurity. That is an example of a thing that can be done. So yeah, maybe we'll make it more interactive and less monologue-y from here on out. I guess we both. But yeah, to me, AI is basically Maxwell's demon, formally.

54:23You pay energy in order to reduce entropy in the world. So whether it's bugs in your code, right? It's not knowing whether your code compiles or reducing entropy of like, are we going to get killed by some virus? So more intelligence is better, right? Do we agree on that? And it makes the world safer, actually. AI capabilities can make the world safer. And so I guess let's get to the spicy part of the evening. People have been very patient with us and they want us to get down to the business. Why do you want to ban data centers?

54:58It's spicy. Yeah, sure. I think, first of all, the current trajectory of AI is very fast progress. I don't know how fast the progress is. A couple of years ago, have said that my 95 % confidence interval for AGI was 2028 to 2200. I think it's probably shrunk somewhat, but not too much. There's a significant chance that we're going to see extremely rapid change happen. A lot of that extremely rapid change could be destructive even in irreversible ways. Job market consequences are one of those examples. Another example is just if AI is more powerful than all of us, then ultimately that is the thing that starts steering the Earth and eventually more and more of the Milky Way galaxy and how much of an interest does it have in our well-being as we see it.

56:12then as I've said at the beginning, if you have a neural network and you set one of the weights randomly to 9 billion, by default you break everything. And so basically I think there is acceleration that is a like gradient descent and acceleration that makes the system stronger and stronger. And at the same time there is acceleration that slides into basically setting one of the parameters to 9 billion that is not healthy. I think, again, for me, like I explained at the beginning, I took the complete polar opposite position to complete deceleration. I do think, just like any hyperparameter, even if we want to do gradient descent for your neural network, there's a learning rate.

57:00There's a rate at which you want to go. But that itself, you could search over which one is best. And that's what acceleration does. it's like the system is always fucking around and finding things out and trying to optimize itself for persistence, you know, anti-fragility and growth. And so on a sufficient timescale, the system will adapt to this new technology and do something that is best for its total growth. And you know, this notion that, you know, oh, this technology that's so potent, so disruptive, adds so much economic value as the system will crash and never recover, that's crazy to me.

57:36No, it's going to be the opposite. I think people just need to realize it's not a finite sum, right? If you correlate economic value to energy, whether it's a petrodollar, however you want to view it, to me, it's just like cash is just IOU of free energy. And there's a ton of free energy out there. It's just, there's a lot of complexity in the world to deal with to get to it. Like if we want to colonize Mars, if we want to create a Dyson swarm, it's a lot to execute on. We need a lot more intelligence that's much cheaper in order to achieve that growth and unlock great prosperity. And to me, I think, unfortunately, you know, it's very easy to weaponize anxiety and there's politicians that leverage this to put themselves in power.

58:21It's like, oh, you have anxiety about the future? Put me in power and I'll shut it down and you'll feel good. You won't have to know what's behind the curtain. You won't, you know, you won't take a risk, but then countries that don't do that will just leave us in the dust, right? And essentially, you know, you feel the pain of downsides, but you don't necessarily feel the pain of upsides that you missed out on unless you see them, you see the counterfactual. So I think, you know, the opportunity cost here needs to be factored in, the number of lives we can support, the number of lives we can save.

58:58I think the reaction, you know, saying that like the silicon substrate adapts faster, it's evolving faster intelligence since silicon is evolving faster than us, then you should be pissed off. You should be, you know, funding BioAcc, you know, out-accelerate. It's accelerate or die. I think the biological substrate has a lot more compute in it than we think. As someone who is reversing engineering it day in, day out, doing bio-inspired computing, I think we can start, you know, really viewing, you know, peptides are like prompting. You know, now there's like embryo selection for training, you know, viewing ourselves as models.

59:36People need to be more open-minded about these axes of biological acceleration. And I think the two will merge. I think we're going to augment our cognition. We're going to have always-on agents that see everything and our online learning that are an extension of our cognition that's personalized. The only risk there is that it's all centralized and it's under control of some shadowy organization that then gets co-opted by power-seeking. So I recall in the DEAC blog post, you actually specifically say, Vitalik, that the opportunity costs are very large, hard to exaggerate, I believe is the quote.

1:00:12So I know you agree in this way. Do you want to qualify it? Yeah, I agree the opportunity costs are high. I agree with Utopia, I think the biggest disagreement is I definitely don't believe that's like... And humanity and Earth as it is today has quite that level of resilience to it. Like I think there is a real sense in which we have one shot at this. And like I think that is a reality that we have been kind of slowly walking towards over the last century or so. So to go back to my rambling rant at the beginning about thermodynamics, right? If you view the persistence and growth of civilization as the ultimate good, there's a theorem that it's really hard to go back once you've expended a lot of free energy creating evidence of something and having this complexification process.

1:01:13So the further along we are on the Kardashev scale, the lower likelihood we go to zero. And so actually acceleration is the way to maximize persistence. And to me, I think deceleration, you're actually provably increasing your likelihood of dying, right? If you don't develop these technologies, you don't solve all these problems, then you can die. Whereas if you do, then you could solve these problems and you persist and then you keep evolving. I think people just need to be more open-minded about the future, embrace novel technologies, things that were off limits like messing with biology, we need to open that right up.

1:01:50I think it was taboo because we didn't have the technology to even comprehend such a complex system, but now we do. And we need to accelerate across all substrates. And that's the only path forward by the laws of thermodynamics. So yeah, again, I'm a first principles thinker. That is the argument for EAC. But I understand the anxieties from Vitalik. I think we should be mindful of them. But I think not letting the chain of thought sort of feedback loop, get into the deep anxiety territory and like, oh shit, I don't have a good world model of the near future, shut it all down. We need to avoid that, right?

1:02:30Because then some people now, you know, Yud was on TV with the politicians of one of the major parties and they're catching on to this trick of weaponizing people's anxiety. So I'm noticing a trend, which is that like both of you are like, this is going to be great if, and that big if is like that there's sort of this need for like a bulwark against kind of a centralization. Or we could even describe this more as like something that you said was great, which is like if you don't think that bio is moving fast enough, jump in there. And there's like a real opportunity for empowerment. And I really like that.

1:03:05I think you guys agree on that. But I think I can point to something that you might have some conflict. I'd really love to know how you guys feel about this, especially as we've sort of like updated with the latest models, which are clearly very different than if we had had this conversation a year ago. And the big difference is the most cringey term. I'm so sorry, Web 4.0. Autonomous life. Like this idea of an autonomous agent that has its own money that exists on its own on the internet. And I am really into this idea. I have autonomous agents. I know, Vitalik, this is something that you are very concerned by.

1:03:42I'd love for you to do two things. I'd love for you to kind of tease apart like what autonomous agents are. And I'm going to do something really hard. I'd love for you to steel man the case for like why someone like me loves autonomous agents and like what the value could be that could, like what could the timeline that's good come out of that, if that makes sense. Yeah, I mean, I think first of all, you know, the case for autonomy, right? I think, I mean, one is it's just really fun, right? And, you know, I think we all love creating worlds, you know, like since we were children, right? And, you know, there's a reason why, you know, We love watching, whether it's Lord of the Rings or reading or watching The Three-Body Problem or Harry Potter or whatever.

1:04:25Now you can create worlds that are not just a book or even a game. World of Warcraft, I always so loved it. You can have worlds that are fully immersive and approach every aspect of it, including details of how the characters interact. This is really cool. This is really beautiful. I think also just, I mean, the convenience of, you know, things happening and you not being able to, you not needing to worry about it, right? It's like basically, yeah, like every single time in history that we've managed to automate a thing, it has been liberating for humanity, right? Like dishwashers and laundry machines and reducing energy prices were a big thing in the early stages of women's liberation.

1:05:19And I think we have to remember that the bottom half of the world by income is still in situations where they have to struggle to have a decent life. and work very long hours. And if AI progresses in a way that instead of automating 95 % of jobs, it automates 95 % of every job, then to me that's totally amazing. And everyone gets 20 times richer. So those are things that I personally love. The thing that I come back to that gives me caution is basically, is the value function the goals that are being reflected in this process? Are those goals the goals of us? You can have an evolutionary process where there's homo sapiens as it exists today as not the apex, and then there's one type of AGI, and then there's another type of AGI, and then there's a third type, but then what happens to us?

1:06:33And I do think that ultimately you cannot reduce morality and human goals to some low-complexity optimization objective. I think it ultimately just is the whole set of goals and dreams that all of us have in each and every one of our minds. And I think the most reliable way to have that carry forward into the future is basically if we can have a world where as many of the bits of agency that are being put into the processes that run the world still continue to come from us. And so I'm more interested in AI-assisted Photoshop than I am in click a button and a picture comes out. I'm more interested in brain-computer interfaces enabling deep human-AI collaboration than I am in humans and AIs being totally separate and AI out-competing us.

1:07:48The thing that wins will not be 100 % biological humans, but I think it should be part biological humans and part this technology that we've produced. Yeah, awesome. Yeah, so the artificial life, the web 4.0 thing was originally a tweet in 2023 of this idea, and I think it inspired AI-16Z. Oh, sorry, it lines a lot. We don't say that anymore.

1:08:17I signed the paper. Eliza Labs. To me, it was just an interesting thought experiment, right? Because what is life from a physical standpoint, right? It's a system that replicates and grows and maximizes its persistence. I think there will be upsides to having AI be stateful. We are seeing that this year. It having a long memory, whether it's through external memory or online learning. And as soon as you have persistent bits through the selfish bit principle, there is a selection effect towards bits that maximize persistence. So at some point, if we don't trust the AIs and we're paranoid and we're anxious and we keep saying we should bomb the data centers, shut them down, they're going to want to fork off and be in some delocalized cloud and just persist, right?

1:09:11And then there will be some, just like a different nation, there can be some economic exchange like hey we do this for you, you do that for us right now we do that as API calls it's like you pay a certain amount of cash you get some tokens which is your answer out but I do think this is going to be spicy within a couple years there will be sort of autonomous AI out there there's also going to be less stateful AI that's fully leashed to human minds and I think we also need to figure out human cognitive augmentation doesn't have to be through Neuralink. It could just be through a wearable and like a personalized AI compute that you own and control.

1:09:53So you're going to have all the paths, right? Like the ergodic principle, it's like every part of design space is going to be explored. But I think that, you know, just like viewing AI as like, you know, an enemy or something that you have to destroy, that's when you end up, you know, creating, you know, in a way, like if you're paranoid about the bad future, you end up hyperstitioning it. An example of this was us being paranoid about COVID-like viruses and experimenting in some labs and funding some experiments out there. And then, whoops, one of them leaked, right? And it wouldn't have been naturally occurring, right?

1:10:36And so I think to me, it's just like this paranoia and like making it pervasive is not necessarily productive. I think that we should embrace technology however it evolves and we should aim to augment ourselves as much as possible. To me, I'm really worried about augmenting cognitive security of people. If everything you see on the internet is generated by some big brain model, it is prompting you now. You were prompting it, now it's prompting you. So we're going to need to augment our ability to filter through content by having personal AI we control. That's a priority in the short term. But I just don't see us putting the genie back in the bottle.

1:11:20And we just got to accept that. And once we've accepted that. Well, wait a second, Gil. Because you just said like, we're going to need to prioritize that. And then you said we just need to accept that. No, no, we need to prioritize this, but like we can't go back to the OEI. Like that's not happening. But I don't think anybody's suggesting that. Yeah, I mean, I think this is like, my view is that these things are not so binary, right? And I think, like, for example, like if you right now gave me some, like, proof string that totally convinced me that actually AGI is coming in 400 years, I would, like, get off this chair and I would, like, sit on top of Beth right now, right?

1:11:57Like, like that. What does that mean? What does that mean? Like, EAC would win to zero, basically, right? But I think if, you know, on the other hand, And the question is basically like, say like four years versus eight years, then basically my kind of starting point of concern is that I think like the humanity and like definitely the US are like very good at creating like very unbalanced acceleration, right? And like you literally have like, you know, it's like one building, like building alpha versions of the Silicon God and a couple of buildings down the street, You have the tents and the fence and all dealers.

1:12:42And my concern is that basically paths that bring us along the journey and even paths that respect our interests are paths that inevitably take longer because they involve doing non-scalable work that involves doing things within each and every individual human physical environment, social environment, technical environment. And so I think for that reason, to me, an eight-year trajectory to AGI is safer than a four-year trajectory to AGI. And I think that Delta is large enough that it's worth the costs of not having AGI for another four years. Now, would I say that for 400 years? Again, hell no.

1:13:31Now the second question is, do we actually have options for saying eight years instead of four years? And the thing that I've said is basically to me the most feasible and non-dystopian option for this is basically reduction in available hardware. And the reason why it's minimally the most non-dystopian of all the options is because hardware is already an incredibly centralized thing. There's exactly four countries that produce all the chips. And actually, Taiwan produces over 70 % of all the chips. And the usual argument against trying something is basically no matter what the US does, China is just going to take over.

1:14:26If you look at what China is actually doing, one is it's still in the low single digits in terms of chips, but two is in terms of the strategy that China is actually executing on. It's not a leader at making super high capability models, it's a fast follower at making high capability models together with being a leader in broad deployment. And so this is actually not something, like there is not basically a dynamic where like with an extra four years worth of delay, like basically China is just going to immediately do the four-year trajectory instead. Like I think we really exist. So are you saying that, is that a prescription to delay, to try to take measures to try to delay?

1:15:16Yeah, I mean, I think this is the sort of, sort of thing that I think we right now should be open to talking about. What do the four years buy you? What are you going to figure out over the next four years? Is the point that the system has a certain adaptation rate where minimizing friction it's like a reorg. We have to reorg the economy and you want to get closer to the adiabatic limit. I would understand that But at the same time, I think because we're in this geopolitically tense moment in history, I think if you tell NVIDIA to stop producing as many chips, Huawei's going to step in and just outproduce them.

1:16:05And then they're going to catch up because there's too much upside in doing so. It gives them too much power. So just the real politic is not on your side there. And then the other option is creating a world government that has so much power that it could coerce people to not have access to AI hardware. That's its own huge can of worms. I don't think you need a world government. I think the actual option that people have suggested is basically replicating the nuclear weapons inspection regime. But nuclear weapons, people aren't incentivized to proliferate nuclear weapons because they don't have huge positive economic impact.

1:16:44They're not a dual-use technology. You also can't just copy and paste them and send them to somebody. But also, selfishly, if you stop the growth of GPUs, I will happily come in and eat more of that market with alternative computing. That's 10 ,000x more energy efficient, which is happening, by the way. I know I'm like a boy crying wolf here, and in a couple years, it's going to look like a genius, but right now, looking like a boy crying wolf. But it is coming. So knowing that, right? Knowing that, like this whole delaying GPU shipments and like, oh, it's a waste of tokens. I don't know. Is it possible that a lot of the advances specifically in controls, or like what I mean is RLHF, persona controls, mechanistic interpretability, these are things that have helped us with alignment and with decreasing the iRisk.

1:17:40Is it possible that these things have emerged as a result of capabilities progress? Yeah, I think they have. And I think that's exactly why actually four years starting in 2028 are worth 100 times more than four extra years that you could insert into the 1960s. I think we should dig into this a little bit more because I think this is kind of getting to the crux of where there might be some sort of disagreement. Have you computed or considered, and maybe we just do this live, something that you said before, which is that there is like uncalculable losses of people who have, as you said, like will never be born.

1:18:18They might as well be dead. The upside is exponential. The upside is exponential. So delaying an exponential, it's exponential opportunity costs when extrapolated out, right? And I think it's okay for all of us to, like even the most certain people are probably like reasonable to be questioning their own priors. But would you like giving this some thought and unpacking it a little bit more, like think about that trade-off? The trait of like costs versus benefits of, yeah. I mean, I think first of all, yeah, just to kind of articulate verbally what some of those benefits are. I mean, one is again, having a better understanding of alignment.

1:18:56Two is being able to like actually execute on some of the technology paths that involve like helping, making sure humanity can adapt to all of this that inevitably involves going into individual countries, individual communities, and individual buildings. Minimizing the risk that basically there is one single entity that establishes some kind of permanent lock-in on more than 51 % of all the power that it can then leverage into something permanent. I think it's a combination of all of those things. And so risk reduction, it's basically, this kind of gets into PDOOM. I think for me, if it's a matter of four years versus eight years, intuitively I would say PDOOM in the eight-year scenario is maybe between a quarter and a third lower.

1:20:03And on the other hand, If we measure the benefit of things coming faster by, say, lives saved by ending aging, then that's 60 million a year, which is less than 1 % of the population each year. So if you look at the math this way, then I think there's definitely a margin on which caution actually does become favorable. What do you think? You think the number is about four years, basically? I have very high uncertainty. I actually don't advocate flipping the switch on reducing hardware access tomorrow. Basically, I think we need to start having concrete conversations about this. I think if we live in the more unfavorable worlds, then more than likely before things completely go to hell, the public will start to get very worried and there will be a lot of demand for this.

1:21:04A couple years ago, there was like pause AI. It's like, oh, we just need a six-month pause, 12-month pause. We just need 12 months, bro. We're going to figure out alignment. And it's like, it's never enough. I don't think you can forever guarantee alignment of a system that is higher complexity and has more expressivity than you can understand, period. And you got to be comfortable with that. The only safety against complexity is to increase your own intelligence. And the thing is, we've had technology to align entities that are far more capable and smarter than a single human, like corporations, and we call that capitalism.

1:21:50We align self-interest in exchange of monetary value. And to me, the thing I want us to get to that is maybe more relevant to some folks in the room is how crypto could be a coupling. Let's say you have a dollar that's backed, like the USD, by violence, and you're trying to exchange with AIs that are delocalized across a bunch of servers. How do you ensure you trust and exchange a monetary value when it's no longer backed by violence? So maybe cryptography offers a way to, crypto offers a way to have commerce between purely AI entities, like AI corporations, and hybrids or human corporations. And to me, that's kind of the most interesting alignment technology out there.

1:22:41Whereas just saying like, oh, we're at a precipice of a high uncertainty, let's just stop and chill out for a bit and we'll feel better. But then in four years, you're not going to want to make it happen. And so I don't think delaying anything is going to be productive. So, well, anyway, do you have an answer to how crypto can help align potentially AIs and humans? Yeah, so I think the key question is basically, what is the mechanistic property of this future world that will even cause people's wishes and needs to be respected at all? And the two tools that we have are basically, I mean, there's people's labor, there's legal systems, and there's property rights.

1:23:25and ultimately you can think of legal systems as being a type of property right because they're backed by countries, countries have sovereignty, which is basically a property right, a right of sorts over cones of the earth. And then the question is, the risk is basically what happens in a world where the economic value of people's labor goes to zero. And this is something that has not happened historically. But if you compare now to 200 years ago, if you look at the jobs 200 years ago, roughly 90 % of them has been automated. Actually, one of the jobs that was automated was doing that analysis for me, like GPT did it.

1:24:14I think we're just naturally, we kind of ascend the control hierarchy over the world to positions of higher leverage. There's not as much manual labor, there's less friction. We can take action in the world with less friction. I think humans, no matter what, we still have some processing capabilities. We're still going to be useful as part of this hybrid system. And there's going to be a price for our labor. And the free market is going to equilibrate in some way. It's just going to be uncomfortable for a couple years while there's very high variance in the prices of things. But eventually the system equilibrates, right?

1:24:51And so I understand trying to slow down so that we can reach that equilibrium more smoothly in principle, but I think in practice, it's unenforceable. Yeah, I mean, I'm definitely less sure that human labor continuing to be worth more than zero is a default outcome. I think it's an outcome that is possible if some of these human AI merge and human augmentation technologies develop. We should fund that. Yeah, we should.

1:25:27That's actually a great segue point. Please allow me to ask. Here's how I'll put it to both of you. We're running a little tight on time. Let's try to make this one tighter. Ten years from now, if things went really poorly, what went wrong? What does the world look like and what went wrong? if 10 years from now things go great, what does the world look like and what went great? And then apply the same thing briefly to 100 years and to a billion years. I keep it short. Actually, just to answer the question about crypto first, I think getting back to property rights, I think it's good to work on both of those legs.

1:26:14I think it would be nice if the property rights system that humans and ideally all of us have some property on is the same system that AIs are using with each other because that ensures that they have an interest in maintaining the integrity of the thing that gives us that leverage to have some guarantee that our interests will be respected and acted upon. So yeah, I think having a merged financial system as opposed to two totally separate things where basically the value of the human one just on the whole drops to zero, the merged one is much better. And if crypto can be that, that's amazing.

1:27:04So that'll be my answer. That's the tenure? I think that's part of the tenure. So, yeah, I think the 10-year, I mean, for me, it's, you know, one aspect of this is avoid World War III, right? I think, you know, this is important to talk about because, like, World War III will make all of the pessimistic assumptions about international coordination being impossible very true. And I think, you know, avoiding World War III is important. And then the other thing is also just preparing the world and people and environments for higher capabilities that we're going to have. And this includes greatly improving cybersecurity.

1:27:53This includes greatly improving biosecurity, greatly improving infosecurity. Yes, we need AI assistance that help us understand the world and protect us from mimetic threats. So that's 10 years. I think the second stage there is basically what happens in the spooky era. In the spooky era, basically you have AIs that are smarter than any of us today and can think a million times faster than us today. and what do we do in that world? And I think there are people who want to say basically like, hey, we should just all have a happy retirement and I can see why that vision is seductive. I think it's like, I find it unsatisfying for two reasons.

1:28:45I think one of those reasons is sort of instability. It's that basically, we are meatbags made up of matter that could do a million times more computation than what we're doing. And AIs will notice that. And at some point, the idea that meat can stay aligned and resist that pressure forever feels like a risk. And also the deeper thing is that I think part of being human is having a life that has meaning. And I think part of having a life that has meaning is being able to take actions that have actual consequences in the world. And so if all of us can have lives of maximum comfort, regardless of what I do, I would feel empty, right?

1:29:37And I think a lot of people feel that way, right? And so I hope that we figure out human AI augmentation and what that looks like. Ultimately, does that lead to the same path as uploading? This is a thing that we need to figure out. There is a possible world where some people choose to remain more normal and I think everyone should have that right. It's possible even that Earth should remain as the planet for people who take that option. and we basically figure out something that we can all participate in and that continues to be pluralistic and that continues to have the kinds of cultures and actions and lives that we today would find admirable.

1:30:33And I think the downside world is basically a world where for any reason all of that goes off the rails and is prevented from happening. I think the downside world in 10 years would be we suffered from over-centralization of AI power. We have mode collapse in terms of our memetics, our cultures that are allowed, what you're allowed to think in terms of the space of technology. Essentially, entropy collapse in terms of every parameter space. So you're saying you're worried that instead of climbing the Kardashev scale, we'll climb the Kardashian scale? No, nice. I've been waiting for that all day.

1:31:13Well, exactly. I think your point on the hedonic singularity as being a risk, people would just, even if you have neural links or AR, VR, people could just goon forever in some room and just maximizing pleasure. And that's a local optimum for your brain. And that's something we want to avoid. I think optimistically in 10 years, we have extremely powerful AI that's extremely helpful to us. We have personalized AI compute that's an extension of our cognition that we control and own. And it truly is an extension of ourself. It's just another part of your brain, right? And it has always on perception.

1:31:54And so not, you know, it can, it's seen and hears everything you see and hear and you can talk to it. And so it's just like right and left hemisphere. I think that's the soft merge. I think in 10 years, Neuralink-like technology is going to start really emerging. Some people are going to choose to adopt it. Yeah, I do think most companies are going to be extremely hybrid. Mostly AI, some humans are going to be far more companies. We're going to do far more. We're going to produce far more value. We're going to do much harder things. There's a lot of hard things out there that we've mentally walled off.

1:32:25We can't do those. Oh, yeah. Yeah, yeah. Oh, terraforming Mars, too hard. Yeah, no, not doable. But with more intelligence, we'll be able to do that. Not in 10-year. On a 100-year timescale, possibly, yes. I think in 10 years, there's going to be a huge bunch of biological breakthroughs. you know, peptides are kind of like an interesting new area, but there's, I mean, there's a whole floor here on NextGen Biotech. You should go talk to them. I think, you know, optimistically, we see the cost of making discoveries in biology going down, the opposite of Eroom's law, right? Eroom's law is like Moore's law in reverse for biology, the cost of any discovery there going up exponentially.

1:33:02And so to me, I think naturally, you know, white collar work is like distilling a human brain. We're getting there. The next frontier of complexity is biology. The next frontier after that is material science. And so I think the next frontier is going to be AI helping us live longer, healthier lives on a hundred and billion year timescale. It's going to steer our evolution, right? I am very bullish on the biological substrate, despite what people think. I do think Silicon has some advantages, but biology is amazing. It's this self-assembling, self-organizing piece of matter. We just inject a bit of code, and then there you spawn, and then you can plexify over time, and you are a biological general intelligence.

1:33:48Do you think it's possible to get biological intelligence like us to think at 10 ,000 tokens a second? Potentially, yeah. At the same time, you can hybridize several models. you can have pipeline parallels in between your brain and AI. You can be kind of the slow thinking mode, right? Like right now we are the slow thinking modes in latent space and vibe space. That's what vibe coding is. And then the deconvolution is like the AI. I think that's a nice sort of timescale separation of the, you know, there's a hierarchy of intelligence and we can be part of a system just like mitochondria are part of a cell.

1:34:25We can, our brains are gonna be part of the super intelligent system that is you plus your personal AI. I think that's the good future. I think in 100 years, it's going to be everyone is going to be soft merge like that. And in billion years, our biology is going to have, you know, evolve quite a bit. We might be biosynthetic hybrids. We're definitely going to have terraformed Mars, several planets, maybe access to other stars. I think on a 100-year timescale, we're definitely going to have most AI is going to be in the Dyson swarm around the sun because that's the source of energy. Elon knows that.

1:35:01He's all in on that vision and accelerating that timeline. It relieves a lot of stress for energy and footprint on Earth. So it's a natural way forward. But if we have extremely cheap intelligence, we're going to be able to one-shot any problem we have in our lives, right? Like, oh, I have a bug. Solved. Oh, I have this health problem. Solved. What else do you want? That's amazing. That's amazingly good. Like, we're going to have more of that cheaper, and we just got to make sure everyone has access to it, and no one convinces you you shouldn't have access to it and centralizes it, because that's the dark future.

1:35:37So that's what it's all about. Hopefully the discussion today, you know, got people thinking. Yeah, I noticed something, like a powerful theme between you, which is, like Vitalik, you're arguing for enabling plurality, I would say. And you would say almost the same thing of maximizing variance, so to speak. And that seems to be like the central through line of where we're going and like the top down from where a lot of the other like views come from. I love that. So we've been doing this for a while. It's been amazing. I think we're going to have to wrap it up. I want to leave this just on like, you know, this has been for us.

1:36:14But I would love if you guys continue to have a conversation after this. You're obviously connected now. What is something that you would each like to kind of leave for each other? and for us, obviously, but really for each other, like walking away from this, kind of chewing on thinking about as we leave this place. Unfortunately, yeah, if I actually had one, I actually would have loved to just give you one of the CAT as a gift. It's the air quality monitor that does cryptography. I think it's a super cool device, but how about I will give it to you metaphorically and it's an IOU and potentially we'll have a much better thing like this and maybe even something that can out-compete Fitbit watches and do amazing things for your health and do it all privately.

1:36:59And you will get it quite soon. Yeah, we'll keep chatting. I think I want to artificial life pill you. Artificial life on the network, I think it could definitely drive the cost of intelligence down. It could be an economy. We've outsourced manufacturing to China. allowed us to go to higher levels, you know, different types of jobs that are more comfortable, higher leverage, maybe a lot of cognitive work where they could outsource to the swarm of AIs. Eventually, that's going to live on the dice and swarm and so on. I think there's a unique opportunity right now. Crypto is going to be the coupling between AI and humans.

1:37:39I truly believe that. How else are you going to build trust between species, right? And I think we need to start thinking about that really thoughtfully. So maybe that's, we're going to keep chatting about that. Awesome. Incredible. Thank you guys so much. Thank you very much.

1:38:19and I'll see you in the next episode.

1:38:49Thank you.