EP 72: Jaan Tallinn (Co-Founder, Skype) on Lessons from Skype, Giving SBF $100M & Investing in AI

35:13So it's kind of like tracking what's happening. So I've been with Anthropic concretely because I just like, I heard it pretty much from the news that like, there's like this new company being formed and it's just like called up on Dario and asked like, what's going on? And what's the benefit, like Dario from Anthropic, You're a board observer at Anthropics, is that right? Yeah. And so what's the benefit to him and them of taking your money versus someone else that maybe isn't opinionated, as opinionated about the existential risks of what they're doing? How do you make that pitch? Hey, I'm going to be in your ear concerned about the safety of some of the things you're doing and that they willingly want that participation from you.

36:00Yeah, the sort of game theoretic pitch I'm making to founders in general, founders who are basically developing deep technology and most interested in investing in deep technology, including AI, is that, look, I don't have LPs. I don't have a boss when I invest. And I have like the sizable philanthropic operation where I want to make things go well for humanity. So I have a credible ability to walk away from money in a way that VCs, I mean, legally they probably can, but like it's kind of up to a particular person that you end up with on your board, et cetera. because they also heavily incentivized to make money for their LPs lest they have trouble like next fundraising and whatnot.

36:56So yeah, my usual pitch is that if you take money from VCs, there will be pressure for you to show optimized profits to take that military contract and whatnot. Whereas if you feel uneasy about what you might be doing that could be dangerous or negative. I'm totally fine leaving my money on the table and we can even shut down the company if we feel that that's not a good thing to do. Have you ever had an entrepreneur say no to you because of your interests or at the point that they get far enough in the conversation with you, is it pretty clear where you're aligned and so you don't actually get to that point?

37:39yeah I never never met like anyone who was kind of like okay this is like weird science fiction guy and like I don't want to be associated with that guy nope never but the main reason like we are sometimes I kind of like don't get invested in other kind of like usual mundane things that they have like a lot of investor interest and then they need to like juggle like multiple multiple parties and their different kind of requirements and constraints and kind of like pressure to to close soon and whatnot have there been companies that you've walked away from because they didn't have the appreciation of uh the risks or the ethics around um what they were doing or do you view that as an opportunity to hey better me than the next person that would provide them money and so i can work on convincing them over time yeah i think it's kind of hard to decouple it from generally think that these people aren't great in terms of their ability to see the dangers and from the assessment that these people aren't great, period.

38:50But sure, I think that, especially with hardware companies, as I earlier mentioned, I do have this dilemma. I mean, almost all hardware companies, of course, they think they're doing a great stuff, a great thing, and kind of providing a service to humanity. I'm like, no, you're not. And then almost all of them, not literally all of them. Can you explain that point for people that maybe aren't as familiar with the trade-offs or the capacity constraints between hardware and software today? What point you're making on that? Current AI paradigm is basically, I call it summon and tame. Like using a simple method that is very, very compute intensive, you summon a mind from mind space, and then you're going to start investigating it.

39:41What does it look like? What can it do? What can you teach it? Can you teach it manners? Or will it still pull a ping on you? or like could it potentially even dangerous? Is it deceptive? Does it hallucinate, et cetera, et cetera. But these are things kind of like that you only discover in the later, I call it taming phase once you have summoned it. But the important bit is like for summoning, it's like summoning or like pre-training as it's officially called. This is like very capital and compute and electricity intensive process. so any company who is like making that process easier is basically making the summoning easier and making it like enabling more and more and or more powerful minds being summoned so in that sense sometimes I say that the remaining runway that humanity has is not measured in wall clock time, but in CPU or GPU clock cycles.

40:45So if you are a company that provides more ability to compute, you're basically shortening the runway in real time because you're running through these cycles, remaining cycles for humanity quicker. It sounds like Demis had read some of your writings or at least some of your talks or whatever it was when you went up and talked to him at the conference. Did it seem that he had an appreciation for the risks associated with AI in your mind at that point in time? Yeah, I think so. I never got pushback on fundamentals from him. Of course, like many in general, like kind of AI company leaders like him, they're like, they're quite constrained in what they kind of say, what they can do, because they have to think about like what signals they are sending to their employees, what signals they're sending to their business partners, or like in Demis' case, his kind of immediate superiors in Google, etc.

41:55So it's, I mean, I can speak much more freely than he ever can. But the point being, he appreciates all this stuff. And I guess it went to the point of actually setting up an ethics board as a condition of this sale to Google, which I think you were involved in. Can you talk through, like, I guess the process of the sale and how you thought about that exit and then setting up the ethics board and what ultimately came of it? So I wasn't super involved in this ethics board thing, other than, yes, it was stipulated in the sale agreement. And the worry was that Google is, at that point at least, completely unaware of AI dangers in a way that DeepMind increasingly was aware.

42:52and then my good friend and fellow board member at DeepMind representing Founders Fund Luke Nozek who was and still is very concerned about AI like he was like I think one of the primary forces kind of pushing pushing for this some kind of intervention to address the problem that like you have like this increasingly safety conscious group being sold to a group that doesn't really care about long-term safety issues. So yeah, that's how it got into the agreement. But of course, what happened afterwards, I don't actually have any visibility to because once Google loaned DeepMind, I could just only visit as a guest there.

43:41Now I'm going to ask a seemingly random question that I think you'll appreciate, but you will kind of take us into the state of the industry today. So what do you think the biggest failing of the nuclear power industry was and why it wasn't able to succeed or why it's been so slow to adoption over the course of the last 40 years? So, I mean, if I were to believe Stuart Russell, Stuart Russell makes this point that nuclear industry has been just like overconfident and being like too dismissive about the potential risks. and then when the risks materialized it was basically shut down and Stuart Russell says that fellow AI people if you want AI as a field to persist we shouldn't make the same mistake I mean that said I don't actually know enough about the nuclear industry to be confident how plausible this theory is.

44:45But the point being, and I guess your perspective there, is that artificial intelligence industry should be upfront, communicative, and transparent, careful with the trade-offs and risks that are being taken at the expense potentially of inhibiting what the industry could be at some point in the future if controlled and regulated appropriately. Is that fair? Yeah, there is this, I guess Stuart's point is to relay this lesson from nuclear that shouldn't mislead the public about the risks that you're taking. So yeah, that's one argument. I think there's a stronger argument. AI developers also want to live.

45:33So that's I think at least partially motivating. It's the recent signatures on these open letters that look, AI could be potentially dangerous because people closer to the risks, just like Manhattan Project scientists, they basically are more informed about what the ultimate risks are. How would you characterize the risk of artificial intelligence today? There's so many ways to characterize it. And the confusion depends on the audience. Sometimes you even ask them, people question, like two questions. like A, can you program and B, do you have children? And then you get like four different quadrants to kind of explain the problem.

46:21And also it depends on like how it's like somewhere, I'm not talking to some like leader in high up or like I'm not talking to some engineer. So like for example, for like political leaders, one explanation I have is that look, you can think of AI as an automated decision-making machine that is not human and it does not care about humans. We are in the process, we as a civilization are in the process of delegating more and more decisions to those non-human machines that do not care about humans. And there is competitive pressure to delegate more and more because we will get faster and hopefully more competent decisions.

47:09But as every leader knows, whenever you delegate something, you will get something else than you originally wanted. You have this principal agent problem. So that's one very neutral way of framing it. What about to the average listener that is curious about all of this stuff and has seen OpenAI in the news and all the advancements and played around with ChatGPT? How do you go about the framing of what's going on right now and why they should potentially be concerned or cautious about this? Yeah, so several ways of putting it. One way is to make this species argument that humanity has been the top species.

47:52And let's not forget what the name of our species is. It's Homo sapiens. Sapiens means smart slash wise. And even though we are smart and wise, the thing that we are engaged in, the AI labs are engaged in and by extension humanity is engaged in is trying to make AI as smart as quickly as possible while using the fact that it is dumb to control it. So like this thing cannot go on forever. Like how long can it go on? We don't know. But at one point it will be just too smart for us to be able to control it. And the comparison that I guess like the last time intelligence at a comparable level was introduced in the world, it ultimately led to 85 % of mammal extinction, which is what humans have done to our world today, not necessarily because we were malicious towards those entities or that we wanted them to die, but that our goals were orthogonal to them living, right?

49:08And so can you talk through that point and how it doesn't actually need to be good or bad. It can just be totally separate from the goal that something that super intelligent is seeking? One point that this example of us having wiped out so many other species is that it is really dangerous to share an environment with other species that is more smarter, more expansive than you.

49:46because we are biological, we are very sensitive to changes in the environment. The biggest acknowledged global issue, global warming, talks about the temperature and related changes that are very, very minor on a scale of astronomical scale. Five to 10 degrees or whatever the current thing is, like six degrees per century change in the temperature of the planet is nothing compared to what the actual temperature differences are when you can zoom out from the planet. Furthermore, it's very unlikely that this particular environment is the best environment for non-human minds. So once they become sufficiently capable of actually modifying, starting to modify the environment in a way that humans really, really have.

50:47This planet went from, in the last 5 ,000 years, this planet went from producing forests to producing cities. So once we have the equivalent in AI, we will likely die as a result of environments just becoming completely uninhabitable or biological and there's even more extreme scenarios when I think about one thing that the AI is likely interested in is free energy and almost all the free energy resides in the hydrogen of the sun so that is the main pot of gold that the AI will, once it becomes at least as smart as I am without any constraints. And the really important bit is that the current pre-training doesn't put any meaningful constraints on AI.

51:34We don't really know how to motivate. And more generally, we don't know how to motivate AIs. We know how to make them behave in a certain way in a certain situation, but we don't know how to motivate. So if we have an AI that has orthogonal motivations, random motivations, just like humans have random motivations, when you're going to compare it to the motivations that evolution ideally should have installed in us from its perspective, then we should worry about things like disruption to the sun. I think that even the people that won't willingly sign a letter, like the furthest extreme of AI optimists, and maybe we'll put Jan LeCun from Facebook as an example of that, Even they would acknowledge this resource trade-off orthogonal goal-seeking issues, I think.

52:27But I think their pushback or their distinction is that this alignment problem isn't as hard to solve as people like yourself and Eliezer and others make it out to be. And so actually getting the AI in alignment with humans' interests, if we made it, we control it. Therefore, we can ultimately direct it in a way that we see fit. Can you talk through how you think about that and why alignment is such a hard problem to solve? So, I mean, first of all, I might be wrong. They might be right. So it's like... I hope so. I also hope so. But the problem is that if I... You don't get to say, I told you so, by the way.

53:18I'm sure you've thought about that. You don't get to celebrate at the end of you being right. Exactly. Sometimes I think about if I end up surviving this decade, it's the future, like the 2030s, basically look like me being wrong about many, many things. So there's like this unfortunate. There's going to be a lot of people saying, I told you so and calling you wacko, but at least you'll be alive to hear it. That is like the logical conclusion. And that's why I'm also like not, I don't want to be like fanatical in my views. I always kind of leave some probability that I just might be wrong in general.

54:05Also, one lesson that I learned from like years of investing and trading and whatnot. One reason why I'm not really consoled by people like Jan Lekun is that their arguments are just really, really bad. And when they say that this thing is easy, it would be really useful if they could actually give a recipe. The current situation is that we as species, we as technology developers, really don't know how to install motivations in AI. And this is important insight from some of the people in alignment community, like even Hubinger. The way we are producing AIs, we are using loss function to create this landscape of this boat.

54:59Can you define loss function, by the way, just for people? Yeah, this is going to go into the weeds. Or can you frame it without loss function? When we train AIs, train machine learning systems, the training regime is set up in a way that always monitors what the AI does. If it does the correcting based on some criteria, then it gets rewarded. If it does the incorrect thing, then it gets punished in some sense. that's like the essence of reinforcement learning. And like more generally, like it will get, its brain gets modified to be more likely to do the right thing in the similar situations in the future.

55:45When I ask a question of chat GPT, it responds and then there's an option that says, was this helpful or whatever? And there's a thumbs up. So that's a reinforcement. That's like, I mean, it's more nuances than there. Probably like your thumbs up and will not like directly go to the system. There will be some kind of like filter and reviewing system, I suspect. But it's providing feedback on the accuracy of what the question is. Now, the interesting, like, important bit is that when you are doing that, when we are doing that, when the AI communities is doing something like that, we are giving feedback on the behavior.

56:19We are not giving feedback on what this thing actually wants to do. So it's, and the important is that there are multiple, for every given behavior in a context, in a given context, there are multiple reasons, multiple motivations that give you identical behavior. One simple toy example, I think Ivan Hubinger even uses it, is that imagine that you have a labyrinth and you're training an AI to exit the labyrinth as quickly as possible. So your loss function, the function that evaluates how the AI is doing and giving this thumbs up, thumbs down, feedback is like how much time has passed and like how close you are to the exit of the labyrinth.

57:11Exits of the labyrinth they have like green exit signs. When you are using the loss function that evaluates its ability to exit evaluates the AI's ability to exit the labyrinth you can swap that loss function for a loss function that evaluates AI's ability to see green as quickly as possible. And you will get the same behavior. So when you are using one of those lost functions, you don't actually know which kind of AI you will get. Either the AI that wants to get out really quickly or the AI that just wants green as quickly as possible. So that's the thing that happened to humans. Evolution was selecting us for ability to self-reproduce.

57:59but effectively it ended up selecting a bunch of heuristics that were correlated with simple, easy-to-access environmental features such as the presence of fatty food or sweet or availability of sex, etc. These motivations gave identical or very similar behavior, sufficiently similar behavior in ancestral environment compared to having the correct motivation of self-reproducing. Once we're out of the ancestral environment, the training environment, we suddenly invent condoms, we invent ice cream, we go to the moon and stop reproducing to a large degree. So the same worry is there. that we are training AI to exit labyrinths and we think that we are motivating AI to exit labyrinths but no, we are creating something that has very different motivations and just exiting labyrinth because of some other heuristic that it has has just randomly accumulated and once it's out once we're going to release it or very importantly it could self-release I think there's a false dichotomy between training and deployment.

59:28Once we get smarter and smarter, the risk of them self-deploying becomes bigger and bigger. So I think we should not just be careful about what we deploy, we should also be careful about what we train, because we might be training things whose motivations we don't control, and then they might self-release. So I want to say that back to you and make sure I'm following. But essentially, if we draw the parallels, which I think are helpful to intelligence in humans, we were solving for like the actual genetic outcome that was being prioritized was reproduction. And that was like the goal ostensibly that we were pursuing.

1:00:10But along the way, and there were certain heuristics that that maximized reproduction and continued sustenance in our environment. Along the way, our intelligence and goal seeking started to deviate from that pursuit. And condoms were introduced or ice cream was introduced or going to the moon was introduced. And so actually setting the course of what the goal is and the implications around that goal was very hard for even humans. And in that case, it led to 85 % mammal extinction along the way, even though the only goal was reproduction. Is that a fair characterization? I mean, almost. I mean, of course, if evolution ended up installing in us the correct motivation of self-reproduction, then you would reproduce like hell.

1:01:16And that probably also means killing off other species. So the survival of other species is not... Other species should not feel safe if you would have this correct... I'm just like me and like, I think Eliezer also is using this like fact that humans are misaligned with evolution. And that misalignment comes from like very similar problem that AI research community currently has. Because like evolution didn't know how to motivate us and AI community doesn't know how to motivate AIs. So what do you think of people that aren't concerned with this? And they just say, hey, it's not that part of a problem.

1:02:05Do you think they're being disingenuous? Do you think that they haven't given it that much thought? But like the people that argue against you, if you were to frame them in the most optimistic presentation that you possibly can, what's your view on what their motivations are or what they don't agree with and that you're saying? The most optimistic view, of course, is that they know something that we, like myself and people who are concerned about it, and including people, many people who are going to like down in the weeds of training AIs, they know something that we don't know. It would be great if they articulated that as well.

1:02:48Exactly. It would be very useful. It's like, don't be shy, as Elias says. Don't be shy. Just tell us, like, how do you intend to motivate AIs once it's potentially smarter than you? And so do you think that they are just motivated by other, I mean, presumably they don't want to die either, or they don't want to wipe out humanity, I think. And so do you think it's a naivety that if they don't have some secret that they're not sharing with us, do you think there's something malicious? Or do you think it's just they haven't thought that critically about it? I mean, it depends on person, obviously.

1:03:23Different people have different... I think one of my worthiest opponents who I had a great debate with is Robin Hanson, who both is kind of skeptical of AI risk and not dismissive of it, but skeptical and has thought about it. So it's kind of like interesting exception to the rule. But the rule, yes, is that people who are dismissive of AI risk, unsurprisingly, they don't think about it. So like when they are basically asked to produce arguments, the arguments are crap because they haven't thought about it. For folks that want to hear Robin Hanson, he was on episode 59 with me as well. And so people can go listen to some of his takes on that.

1:04:03So you've been a part of this industry now, I guess, for 13 years, which I think makes you one of the more, I guess, in modern times, one of the older people with a more longitudinal perspective of what's happened. ChatGPT, I think, was a big breakthrough for a lot of people's understanding and at least the mainstream adoption of that. So what was your perspective when you saw that launched? I mean, I saw some early transcripts of GPT-4 before it was released. And yeah, it was very unnerving. What was the nerving about GPT-4 versus GPT-3? It just was much more consistent. Like the main, I think many people have this like misguided assumption that like there is this kind of magic dust called intentionality, volition that is just like missing in like non-human minds.

1:05:08and we don't even know how to install it, so we shouldn't be afraid at all of something that is a machine mind because it doesn't have its own volition, its own intention. I think it's just completely misguided. It's like heat-seeking missile has a lot of intention that makes it very uncomfortable being on the other end of. Intention is cheap. What is expensive, at least in the current AI paradigm, is long-term consistency. so like the big thing that currently is the big technical term that is currently correlated with it that is the size of the context window like how much how much content how much kind of context can the system have before it makes a new decision so like that was like very impressive jump up from GPT-3 to GPT-4 about like the AI being consistent, remembering things that it said earlier and not just talking at cross purposes to each other.

1:06:13But still, GPT-4, the context window is much smaller than humans have. We can still recall facts from when we were kids and based on decisions on those things. So the hard bit is basically to make AIs having long-term, consistent and coherent plans. And seeing this massive jump up from GPT-3 to GPT-4 makes me think, how many such jumps in planning ability, smart, consistent planning ability, we can still afford before we find ourselves at the receiving end of the plan that we don't like. So after you saw GPT-4 launch, you initiated the call for a six-month pause. Is that right? And how did that actually come to be?

1:07:14How did you go about pursuing it? Yeah. So in early March, I wrote a small memo that I shared with AI safety community and some some AI developers, where I just pointed out this increased uneasiness post-ChatGPT launch among a wider and wider circle of people, like Ezra Klein's New York Times piece, as well as Yuval Harari's New York Times pieces had come out, and there were increasing calls for some kind of action intervention from different people. So in that memo, I proposed two potential interventions. One was doing some kind of survey in AI development, AI labs, about what is the internal view about how much risk the labs are currently taking when they develop frontier models.

1:08:21Because like the Manhattan Project scientists, they did just that before the first detonation of a nuclear device. They did a calculation about what is the probability of actually destroying the planet using this third creation. So I think like this is like a homework that I think technology developers in general should be doing. And like now that there is going to be increased uneasiness in the labs, I think this is like a valuable homework to be doing and creating common knowledge about. And the other thing, if I remember correctly, I proposed is like, what about like creating some kind of moratorium before we have these numbers or before we have sufficient certainty about the risk being sufficiently small?

1:09:13shouldn't start advocating for slowdown. And again, there were some, Katja Grace, for example, published an article on Less Wrong about isn't it the time to slow down? And just arguing against the usual counter-arguments about slowing down being unrealistic. So after that memo, I waited for a couple of weeks, got like a bunch of great comments. And then I think around March 20, we had like Future Flight Institute's call. And I had, of course, shared the memo with these people as well. My co-founders there. And I think somebody there just proposed, okay, we have like ability to do open letters.

1:09:59Why don't we just like actually put this in practice and do another one and see like how many signatures we're going to get. So you published it. You got signatories on it with you, including Elon Musk came in and signed it as well. Was your intention just to raise heightened awareness? Did you actually think we were going to be able to get a six-month pause out of it? Why six months versus anything else? Yeah, I mean, so many things to say about that. The intention with the letter was to create what's known common knowledge, that we are seeing these signs of more and more people inside the labs and outside the labs, being worried about, wait a minute, where is this thing?

1:10:45What is this chattypity thing? Where is this going? Is this something that we signed up for? but there wasn't a common knowledge in the sense that people know that other people also know so it's kind of like one thing that you when you say that I'm uneasy about this you don't know how many people think like you the main argument for putting in six months was to have something that is more realistic and more concrete and also more informative when it fails so if if you have if you call for indefinite pause then like we expect that reaction to be like oh yeah this is completely unrealistic what do I mean indefinite etc so that's why we wanted to create a situation where people can't even pause for 6 months and the other thing was like this is the China argument that it's easier to advocate for like a small small pause and see like if this is possible in a situation where like the most common counter argument is that, well, if you pause, then like China just rushes ahead and gets an advantage and whatnot.

1:11:58Not that I believe in that particular argument, but like it was like a pragmatic thing to put in. How would you deem this success? If those were the intentions, Obviously, it didn't happen, but we have had a statement of AI risk, which you signed along with Sam Altman and Jeffrey Hinton actually resigned from Google and is now speaking more publicly about it. Demis from DeepMind and Dario from Anthropics signed the statement of AI risk as well. So if you were rating yourself on the original intentions of the six-month pause, knowing ultimately you failed, did it actually succeed in the mission?

1:12:38Or what grade would you give the call for a pause? I mean, counterfactuals are very hard. It's kind of very hard for me to imagine what the world looks like if it didn't put out this letter. I'm very confident that the main causal factor is ChatGPT here. The reason why it was easy to make this letter and get a lot of signatures was because of ChatGPT. If it had done it before ChatGPT, I'm pretty sure we could definitely get less than 1 ,000 signatures. And the second AI extinction statement would not have happened. So yeah, I think OpenAI sort of gets most of the credit for alerting society. But yeah, it's actually very hard for me.

1:13:25I should probably just ask Dan Hendricks about what does he think about would they have put out this extinction statement, which I think is super, super useful. Had FLI not created the Open Letter. How has your reaction been to the response and the progress and the changes that have happened since you put out the letter? I mean, I sort of like was hoping to getting like a lot of resonance from the larger society. And again, like the early signs were there that like more and more people not directly involved in AI are thinking about like what this situation now is. Like 2022 feels, 2023 feels significantly different from early areas.

1:14:18There's effectively a big three in my mind right now in artificial intelligence with regard to large language models. There's Microsoft slash OpenAI, Google slash DeepMind, and then also the far smallest, but also a very prominent one right now is Anthropic as well. You've obviously invested in two of the three underpinning technologies, I guess. To ask you about the first and perhaps the most prominent of it, OpenAI, what's your perspective on their attitude towards the risks associated with this? Sam obviously signed that statement of AI risk. He talks about this quite a bit. um what's your perspective on how they're acting versus what they're saying and are they just accelerating the inevitability in your mind or do you think they're being pragmatically cautious i mean i'm thankful to be able to live in a world where uh like the top labs the heads of like the leaders of the top labs uh they are conscious of the problem and uh being kind of like saying so especially some I think some has been like the most kind of public about the extreme risk concerns but like that doesn't change the fact that I think it's it's just like very reckless for any lab at the moment using the current paradigm to push the frontier because it's these are like uncontrollable minds that are being summoned in the hopes that they're not very competent that can't make these long-term plans.

1:15:54And these hopes just come with a certain risk that they might be wrong. I'm going to ask you the hard questions that Eliezer at least refuses to be pinned down on, but what percentage likelihood do you think we're on a path to extinction? It depends on timelines. Like a billion years, sure. That was the second question. So I'll let you weigh in on timelines as well. I mean, if it's like GPT-5 that kills us, then it's like 95 % or something like that. It's something which comes very soon using the current paradigm because really, really, we don't know how to motivate AI. If it turns out that this entire LLM paradigm, like my good friend, Kerry Marcus, argues, is just misguided.

1:16:41It's not going to get us to very competent AIs. Then we will have more breathing room. You might even have a couple of decades left still. and we need to invent some more things. And now that there is more realization that AI safety is a big problem, we might actually accomplish things that make us substantially safer in the next 20 years. Because the entire field of AI safety, I mean, people, including myself, I've gone back to people like Alan Turing and Norbert Wiener, IJ Goode, who basically 60 years ago or so already kind of mentioned that AI safety is going to be a big problem. But the field really started something like 10, 15 years ago.

1:17:23So if you would have another 20 years or so, then I would be starting already leaning optimistic that the probability of us actually surviving is more likely than not. But currently, it looks like we're not going to have 20 years. You think it's sub-20 years? and what probability would you put on chat GPT or GPT-5 being the cause if you were to just extrapolate the gains from three to four to four to five in some way? What probability would you put that risk at? Yeah, so my – actually, the probability that I used in my memo in early March, and I still can't stand by it, I haven't heard strong counter-arguments to that, is 1 to 50%, so like super wide confidence range because I'm just like very, very uncertain of complete wipeout of humanity per 10xing of compute that is being thrown at these experiments.

1:18:29So I think my current understanding, and I'm not an open AI insider, so I might be wrong about this, is that the difference between GPT-4 and GPT-5 is going to be more than 10x. So we're probably talking about a couple of percent chance of open airwaping out everyone later this year, possibly. That's quite, quite ominous. What would you recommend if people hear this and it's the average person, what would you recommend that they actually do about, let's say maybe they think you could be right? How would you propose that they respond to that? It's kind of hard for me to give recipes for common people.

1:19:12I think the general thing that we would want, and there's an increasingly big consensus, is some kind of compute governance that wouldn't be a complete free-for-all throwing of compute at these uncontrollable AI experiments. Ideally, we should have a cap on how much compute you can spend on these experiments or like growing those AIs that you actually don't control. So like any kind of advocacy towards that, I think currently is positive. There always could be like kind of like negative side effects, et cetera, like nobody's safe from that. and sort of putting more pressure on the leading frontier model developers.

1:20:05I mean, currently there are three of them, but there probably would be more as this technology becomes more widely available, the training technology, for them to make an affirmative case because at least one to 50 % is my subjective estimate. And I'm happy to, would be super happy actually to revise it down if there was some strong arguments why it's, for example, not possible for large language model to self-deploy. What are the actual constraints and limits on this training regime? Why are we confident that it can't just spin out of control during the months as it's left unintended? I realize it's hard to do the counterfactual, but I want to shift gears a little bit here.

1:20:56If not for AI and existential risk, which I think you've dedicated a large portion of the last 15-ish years to thinking about and investing behind and all of that, what do you think you would be doing if that concern didn't exist? I mean I definitely would be investing which I am although like I have delegated away most of the kind of day-to-day legwork there I would be building things which I still am in fact was coding earlier today and I would be dancing that's my main hobby freestyle street dancing so it sounds like you'd be doing more of the same but you would just you would just have time that you don't currently have.

1:21:43Yeah, yeah. And I think one way how this is healthy to have these side things is that there's this automatic concern, especially with Elon. When he says something that I think is reasonable, people always have suspicion. Why is he saying that? What are his actual interests? And in my case, it's like, yeah, well, I have these other things. I really love dancing. I would like to just go and dance instead of like, I'm not going to get my positive hit-ons from just talking about AI risk. I actually much rather be dancing. Well, last one, then I'll let you hop. What's a piece of conventional wisdom around startups or investing or anything that you disagree with today?

1:22:29I mean, I guess the obvious one, actually, is that, yeah, this kind of steamroller thing that in general, people, including people in technology seem to have this assumption that technology is now ready. And in the future, we will have a little bit thinner iPhones and that's about it. Whereas basically the changes that might be coming, people are just completely underestimating it. Of course, the question is that because the changes are hard to predict, even though you have some idea about what will happen in the limit, but how we get to the limit and what the actual chaos that might happen once we have billions of non-human minds that are roughly at the level of college students interacting with us and our people space all the time, how big chaos that will create.

1:23:24I don't know, but it's very plausible that just super big change compared to like default expectations of slightly thinner phones. Well, Jan, thank you for doing this and making the time and sharing all your lessons that you learned and your perspective. It's a super interesting conversation. Thank you very much.