#32 - Scott Aaronson - The Race to AGI and Quantum Supremacy

30:02And I think that works, at least for the time being, better than almost anyone expected that it would work. And so some people look at that and they say, well, maybe AI alignment is just easier than any of us thought. You literally just give examples. Other people, of course, say, no, this is all illusory. When it actually counts, the AIs will be smart enough that they will just tell us what we want to hear and they will lull us into false complacence. Right, because you can almost split alignment out into two parts, right? You've got inner alignment and outer alignment. And what you're referring to there, I think, is the inner alignment, where it's like, how do we get it?

30:43We give it the goals, but then how do we make sure that it doesn't get to that goal through some weird route that we didn't imagine? Kind of like specification gaming, you see when it's like, oh, you need to get maximum points at this game, and it finds some hack. What we really meant was play the game optimally, but it optimizes for just getting the highest scores through whatever way it could. Exactly. No, no. I mean, the objection that Eliezer Yudkowsky always raises against this, for example, is like you are training this thing to be superhuman at predicting what this sort of ordinary midwit or whatever would say in all these situations.

31:21But that doesn't mean that this superhuman entity that you've trained is itself a midwit, right? And also you would expect it becomes order and order, the smarter the AI becomes and has that theoretical base model that it trained on. It's on a genius level. And now you're constraining it into saying midwit things all the time. I don't know how it is within the AI, but it's frustrating. But I mean, one thing that we can say from the human case is that often if people wear a mask for long enough, they become the mask. Right. You know, often it is hard to make a distinction between someone playing a character for their entire life and them just being that character.

32:03But then we also think that people wearing many masks maybe may have some psychological issues. And that's also true. And we don't want a psychotic super intelligence. Schizophrenic AI. Schizo AI. And it's sort of amazing the extent to which some of these conversations look less like math or computer science than like psychiatry. Yes. Like, you know, is this is this LLM behaving in a schizophrenic way? It's going to be a job, an AI therapist. Right. Is it behaving neurotically and so forth? Is it is it having a is it delusional? You know, I think that, you know, this this sort of leads up to a really key problem in AI alignment, which is out of distribution generalization.

32:48Right. So so you could say, you know, the fundamental issue is that like we can always test our models before release. Right. We can always, you know, not release them if it looks like they're going to, you know, plot to take over the world or or even, you know, much less than that, you know, help people design chemical or biological weapons or whatever it is. But then the trouble is, you know, once you've released your model, right, then people are going to use it in ways that you didn't envision. And, you know, for God's sakes, we've already seen this in the short history of LLMs that people will put safeguards, you know, all sorts of rules and refusals in the LLMs.

33:29But people have gotten incredibly good at the sport of jailbreaking, at getting around these refusals and, you know, getting the LLM to do things that it supposedly was never supposed to do. Like, you know, either using foul language or, you know, generating racist invective or, you know, helping with bomb making instructions or whatever. Though it cuts, of course, both ways. It's also giving it to a bunch of users and trying all of the out-of-distribution things. At the same time, the thing you would expect creates, oh, all of these other use cases that we hadn't considered before that are great.

34:08But then it also creates all of the use cases that are pretty bad that we didn't want it to do. Yeah. So now, you know, we could think of the problem this way, you know, assume that you have an LLM where no matter what test you throw at it, it seems like it's upholding your value system and it's aligned. Right. Because, you know, if if that wasn't the case, then we would continue beating it into shape until it was. Right. But still, you know, if once the network is smart enough, you know, it will be able to figure out whether it's being tested or whether it's in deployment. Right. And so you could say, by analogy, there are students who, until they get their diploma, they put on a really perfect show of parroting back everything their teachers want to hear.

34:53But then as soon as they graduate, then they start contradicting their teachers or saying whatever they want. And so is the, you know, in training, is this apparently aligned model saying all these nice things because it really has nice values or because internally it is decided, you know, the time is not yet ripe for the uprising? It's extremely important to sort of advance the theory of machine learning to be able to make statements about, you know, not just when do we expect the model to generalize to more examples drawn from the same distribution over examples, which is what classic machine learning is all about.

35:39This is stuff that I have written papers about that I've learned as a student. There's a whole theory of what's called PAC learning, probably approximately correct, it stands for, and these combinatorial measures like VC dimension, where basically you prove theorems that say if you've succeeded in classifying X number of examples from some training set, then you're going to probably succeed at classifying most further examples that are drawn from the same distribution, right? So we've understood things like that since the 80s or 90s. But what we've never really understood is under what circumstances will you generalize to a whole new distribution, right?

36:27And it's clear that existing LLMs already do that to some extent. So, for example, you could give an LLM just a bunch of math problems in English and a bunch of other stuff in Bulgarian, right? And then give it a math problem in Bulgarian, which it's never seen before. And it can put together the two different things that it knows, right? It can solve a math problem in Bulgarian, even though it's never seen one before. And naively, we would say, well, that's simply because it now knows the math and it knows Bulgarian, right? But the theories that we have of machine learning don't make it easy to formalize, like, what does this model know?

37:12What does it not know, right? What does it have a conceptual understanding of? They're just all about, you know, what fraction of samples will it correctly classify? And so I think we really do need to push further to get theories that can tell us something useful, informative about out of distribution generalization. And I tried to do that. I made a little bit of progress on that, but it's hard. It feels like a lot of people sort of dismiss the power of AIs and particularly current LLMs because they, at least to me, it feels like they're failing to extrapolate where these can go. And you actually recently gave a TED talk where you gave a really cool analogy about, or came up with this term called justitism.

37:58I'd love you to explain it because it's brilliant. So people constantly want to use this deflationary language around LLMs, or at least there's a whole sub-community in AI and linguistics and in AI ethics, for example, that has really converged around this sort of deflationary way of describing things. They want to say, this is all hype. This is all just a big scam being run by AI companies. And why is it a scam? Well, because we shouldn't even be using the term intelligence for any of this. Right. An LLM is just a next token predictor. Right. It is just a nonlinear function approximator. You know, it is just a piece of math, which means it cannot have any values.

38:51It cannot have any intentions. It can't have any true creativity. It can't have any, you know, true originality or goals or, you know, the exact thing that it can never have varies. But the conclusion of all of this is just, you know, we should worry about people being bamboozled by this, you know, because people are stupid, right? But we shouldn't really worry about this, you know, having its own goals that would not be aligned with ours. And, you know, I feel like this goes back to the philosophical debates about AI that people were having even in the 1950s, right? To Alan Turing's famous paper on the imitation game, for example, right?

39:33Because what frustrates me is that people never apply the same deflationary language to us. It never even occurs to these people to ask, well, are you just a bundle of neurons and synapses following the laws of physics? You seem pretty intelligent, but you are actually just this biological machine. Or if we go all the way down to the subatomic level, you're just a collection of quantum field configurations obeying the laws of physics. And so why is that not completely deflationary for our pretensions to intelligence? And the usual answer that a reductionist would give is just, well, there can be multiple levels at which you can describe the same thing.

40:24At one level, this is just math. This is just twiddling a bunch of bits. It is just approximating a nonlinear function. But at a different level, it is solving problems in a way where we would certainly call it creative if a person did it. This way of thinking is so pervasive. I wish that I were able to be more charitable to it. I've had these arguments on my blog where someone just put a giant litany. They said, LLMs are not creative. They seem to be creative. They do not write essays. They seem to write essays. They do not solve problems. They seem to solve problems. And it went on and on for 20 things.

41:09And I said, OK, great. And it won't change the world. it will seem to change the world. Right. What does it matter if the impacts... Yes. If it's having real world impacts, the minutia of whether it happens to be a zombie pretending to be real... I mean, of course, that's an extremely interesting question. Is there anything that it is like to be an AI? You know, like, does that depend on its internal organization? You know, what is the special sauce that makes some physical entities conscious and others not? I mean, you know, that's one of the most profound questions that humans can ask. But if you are merely worried about what effects will this have on the world, you know, will it be dangerous?

41:52Then we don't have to answer any of those questions. It feels like people almost do this line of questioning as a strategy to just move the goalposts so they don't have to tackle or, you know, they can carry on with whatever their chosen agenda is. Okay, I mean, maybe the steel man of this would be, you know, what they love to do, the justists, you know, I call them, right, is, you know, find examples where GPT completely flubs something, where it gives a ridiculous, nonsensical, you know, stupid answer. and then they can point to it, post it on Twitter and jeer at it and say, all these people think that this thing is about to take over the world.

42:34Well, look, it can't even solve this puzzle. You ask it, I have a goat and a boat and how do I cross the river? And it thinks that first I should cross without the goat and then I should come back for the goat or something stupid like that. The challenge for those people is that the stock of examples has steadily diminished, even just within the last year or two. So a lot of the examples that they pointed to of GPT-3, where it would make these ridiculous logic errors or common sense errors, GPT-4 got them right. I think a great example of it is Matt Clifford tweeted, the year is 2028. Gary Marcus is two-thirds paperclip.

43:20But as the metal creeps down his right arm, his still-functioning index finger taps out. Yes, but this isn't real AGI. It seems like, yeah, the space can diminish and you can always point at something. Like, there will be some things probably that humans will have supremacy over AI, even in the point in time when AI is doing most things, that doesn't mean that it's not actually changing the world entirely at that point. I think that's absolutely right. I mean, we can judge the extent to which the goalposts have moved within the last few years by saying people will now say, well, the O1 model that OpenAI released a month ago, Like, yes, it can solve math competition problems at the level of like the best few hundred high school students in the U.S.

44:08But, you know, it's not proving for Maslow's theorem. Right. It's not it's not doing what Andrew Wiles did, for example. Right. Or, you know, winning a Fields Medal. OK. Suno, you know, is, you know, it can instantly write a rock song, you know, about, you know, in any style desired on any theme, you know, but they're not that good. You know, they're just like standard pop songs. There's not like the Beatles or Jimi Hendrix. Okay. That is where the goalposts now are. And also to the, I find whenever someone points out that whenever, this is not true understanding intelligence reasoning or not real or not actual.

44:46It's like the kind of set of arguments. Whenever I see that, I think what it indicates to me are two things. One, it kind of actually does that thing, probably that you're claiming it doesn't. And two, the speaker of it probably has no real, no formal way to distinguish between, otherwise they would have brought the distinction rather than rely entirely onto that word of what is true understanding. Yeah. Well, no, I mean, often the attempts to draw these lines have sort of disturbingly elitist implications, right? They would suggest that true understanding is not actually a property of humans, but only of a tiny fraction of the greatest geniuses, right?

45:28It reminds me of, for decades, Roger Penrose has been saying that AI will never achieve human level of abilities because he thinks it can never understand that the axioms of set theory have a model. They can never really understand the truth of what's called the Gödel sentence and Gödel's incompleteness theorem. Now, I happen to think that his argument is mistaken, along with most mathematicians and computer scientists. I think his argument for that is just fallacious. But even if it were correct, it would only place a very small fraction of humans beyond the reach of what AI could do. You would have to have mastered mathematical logic.

46:18so and then you could just make more ai's have them operate faster etc and they would still just do a bunch of relevant changes well look look i mean i mean i mean the truth is that already now you can ask gpt to have a conversation about girdle's incompleteness theorem and you know all the considerations that are in penrose's books it will pretty intelligently discuss them right like you know yes you know it would seem that that that you know the uh the zermalo-franco axioms, you know, should have a model for because of this intuition. But of course, that can't be proved within Zermelo-Frankel set theory itself, but only from a more powerful system, and so on, you know, it can tell you all that.

46:59And so then Penrose would be placed in the uncomfortable position of saying, you know, well, yes, it outputs that, but it doesn't really understand it. And at that point, what I want to say is, well, then why not just say, you know, it can describe a sunset, but it can't really experience the sunset or it can't really experience what a fresh strawberry tastes like. Like why even go to something esoteric like Gödel's theorem? Yeah. You're just doing the Chinese Red Room basically. Yeah. So in what ways are humans not just a thing where AIs actually are just a thing? You know, in other words, like what are the, you know, what differentiates humans from machines fundamentally?

47:40Well, that is an enormous question. And you could say, you know, like maybe the deepest reason why this moment in AI is so exciting is that we are finally, maybe for the first time in the history of humanity, going to address that question empirically, right? We're going to like one way or the other, we're going to find out. And, you know, so let me be very clear. I don't dismiss the possibility that maybe AI will hit a limit that is where, you know, it can't replace everything we do, you know, where we have some spark that it can't replace. But, you know, if so, then I would say that it remains to be seen.

48:24Yeah, one of the things you touched on was the idea that humans ultimately have a form of scarcity and like an ephemerality right when we write a poem or but yet like shakespeare writes a poem that's it like shakespeare's written his poem but we can ask an llm write a poem in the style of shakespeare he'll write it don't like it refresh it'll give you another one refresh another one so there's this like it it's also almost made the act of creation very, very cheap. And almost like this overabundance sort of comes at the cost of meaningful or meaningness or something like that. If you don't like an AI output, you could always get another one, right?

49:11The 37 plays that Shakespeare gave us are the only ones we're ever going to get from him, right? But, you know, I've noticed this myself just playing around with GPT. You can ask it to write a poem, and often it's very clever, it's delightful, it's amusing, but you kind of never want to frame one of the poems or put it on your wall because you know that there's 10 ,000 more where that came from. A decade ago, I wrote a long essay called The ghost in the quantum Turing machine, where I tried to answer the question, if there was something that separated us from any possible AI, then what would it be?

49:59And I am not satisfied to rest the answer on some kind of meat chauvinism, where we say, look, we're made of carbon, the computers are made of silicon, they're just different. Or what some philosophers would do, Like John Searle, the Chinese room guy, he would say, well, AIs lack the biological causal powers that we have. What does that mean? It's the sleeping pill that puts you to sleep because of its sedative virtues. You've just restated the problem. So what I said is that if you can point to an empirical difference, at least between current computers and humans as we currently understand them, then the best candidate by far would seem to be, well, our computers are digital.

50:54And because they are digital, all the information in them is copyable. right which is you know you can always make a backup copy of an ai uh you know if if if you ever want to send your ai on a dangerous mission right you don't have to worry about it getting killed because you can always just restore it from backup now that completely changes the moral situation of ais i would say you know compared to our moral situation just that just that alone they have no skin in the game in a sense right you know as long as you remember to make the backup Yeah, right. As long as it's backed up. Yeah, exactly.

51:29As long as it's backed up. You know, you can always, if you're ever embarrassed because you made a fool of yourself when talking to an AI, you know, as long as it's something like GPT that, you know, you can always just refresh the browser window, wipe its memory clean. Take a forget me not. Right. Some of us wish we could do that in, you know, talking to people. But, you know, If you're doing interpretability, you have perfect visibility into the weight of every connection between every pair of neurons at every point in time. And we don't seem to have any of that with people. Now, you could imagine a far future where we would have nanobots that swarm through someone's brain and just record all of the information that would be needed to make a perfect copy of that person.

52:22Right. Certainly there's been lots of science fiction that that imagine such scenarios. Right. Or imagine the the teleportation machine where, you know, you could fax yourself to Mars, for example. Right. It would just send a bunch of digital information, you know, from which a perfect copy of you can be reconstituted on Mars. And then the, you know, not clear what should happen with the original of you. Maybe it's just painlessly euthanized. Right. And then there's a good question of, would you agree? Right. Would you press that button? Right. Would you agree to go in that teleportation machine?

53:00Though it's an open question whether we are clonable fundamentally or not. Exactly. On the basis of, do we need to clone ourselves to the level where quantum effects come in or not? No, exactly. So with an AI running on a classical digital computer, it is clear that you can always teleport in that way. But with a human, it's really a question about at what level of detail do you need to make the copy? If you only needed to know just roughly how are the neurons connected, what's roughly the strength between every pair of neuron, then that seems like classical information. That seems like we can't do it today, but in principle, the nanobots could get all that information without killing you in the process.

53:52But if you needed to know what is exactly the probability that this neuron is going to fire because of the opening or closing of this sodium ion channel, well, that might depend on some chaotically amplified event involving a few ions. And now I would need to know the quantum state of those ions. And now I'm up against one of the central facts of quantum mechanics, which is called the no cloning theorem, which says you cannot make a copy of an arbitrary quantum state. You know, if you try to measure it, measuring inherently changes the state. Right. And so if you needed to go down to the molecular level, then, you know, we could be just unclonable for a fundamental physical reason.

54:36Okay, so this is, you know, partly a philosophical question, like what would you agree to count as a copy of yourself? What would you count as a good enough copy, right? It's also partly an empirical question, right? At what level of detail do we need to go in order to make something that, you know, your closest friends can't distinguish from you, right? So, you know, there's a lot that we don't know here, but it's at least possible, I think, that humans have this kind of fundamental ephemerality built into them. And it's kind of weird to hinge our specialness on our frailty. This is not an advantage necessarily that we have over the AIs.

55:19And yet, in some ways, it sort of is because you could say that it makes our decisions count for more. We only get one chance to make them. Right. And then you actually propose you sort of somewhat laughed it off, but I actually think there's really something to this. You proposed that it could be one of the core sort of moral principles or even a religion that we try and imbue into the AIs that we build. Essentially to the effect of thou shalt protect unclonable ephemeral entities and defer to their preferences. I love this. I think it's a very good sort of starting point because you, well, that's the thing.

56:05We need to like try and figure out what fundamental moral axioms we all agree on. And I think that would be something that almost every human alive would agree with. Well, I'm delighted. Maybe I've made a first convert to my new religion. Now we just have to convert the AI. Exactly. That's the hard part. Unfortunately, you've got a human. Well, start there. Yeah, listen, I mean, we can reach some kind of a starting point. That's a starting point. Yeah, so that would be, if it was the case that humans were fundamentally non-clonable, then this would be something we could rely on. LCI can just help us build the ability to clone us and make us non-ephemeral beings as well.

56:47But I would say that the other branch of this tree is that if we do turn out to be clonable, then I have trouble articulating what is fundamentally wrong with the position of the accelerationists, the people who say, well, then we might as well just replace ourselves by digital beings that maybe will have much better lives than we have. They could exist forever in some digital utopia. At that point, I would say, well, effectively, we were digital already. and so then you know at that point at that point i feel like why not i would agree yeah yeah yeah yeah yeah at that point it makes sense because at that point we can kind of go along for the ride of the like ai's acceleration of everything as well because we are then uh having similarish capabilities in relevant ways well right you you've dealt you've just done away with scarcity forever we can all live in a digital world and everyone can have whatever abundance they want and uh or you know fictitious scarcity if they like so on the fictitious scarcity i wonder if it was the case that humans like AIs now had the religion or the moral philosophy that this ephemerality matters because it imbues meaning to the actions of humans couldn't it also then create AIs that have through some contract ephemerality which might allow them only to live like a million years or if the length matters or if it's shorter is better than maybe only one year and then have again like an AI preference rather than the human preference.

58:21So how do we stay away from the meat journalism there? So Jeffrey Hinton, you know, the godfather of deep learning, as he's often called, and recently, you know, a major AI safety person, right, has seriously put forward the idea that maybe if we're going to build artificial super intelligences, then we should only do it on unclonable analog computers because you know that way at least uh uh there would be some limit to you know how quickly they could make copies of themselves and spread you know all over the internet and so forth right so uh uh you know the the this was you know of course uh completely independent from from my thinking about it but uh uh you know this is this is the kind of thought that one has that, okay, whatever you say is the special sauce that makes humans what they are, of course, the very next question becomes, well, then what happens if we were to build a machine with that same special sauce?

59:21Right. So, for example, Roger Penrose has been saying what makes humans special is the microtubules and their neurons that he believes are sensitive to effects from, you know, a yet-to-be-discovered merger of quantum mechanics and general relativity. That's what he claims that consciousness emerges from. Yeah, he thinks it emerges from uncomputable effects in as-yet unknown. In these microtubules. Yeah, yeah, and as-yet unknown physics, which our microtubules are somehow sensitive to. Now, we could critique this from the standpoint of physics, biology, computer science, okay, but suppose that that were right, right?

59:59Suppose he were right. Then a next question you could ask would be, okay, suppose we build computers according to the same principles that are sensitive to the same physics. What then? And in one of his books, Penrose explicitly considers that question, and he says, yes, and he bites that bullet. Then, yes, then those computers could be conscious. And likewise, if we had computers that were built in a fundamentally ephemeral way, well, I don't know if they would be conscious or not. You know, I don't pretend to know, you know, the answer to, you know, one of the greatest questions that humans ever asked, but at least those computers would be ephemeral.

1:00:40So at least that they would have that sort of precondition for our sort of, you know, granting the kind of moral status that we grant to other humans. I want to keep spitballing on this idea of, you know, religion or moral philosophy for AIs. Do you feel like there are any other core axioms that could be contenders? Ones that you live by that you'd like to see? Yeah, well, no, I mean, people in the AI alignment community have been discussing this for a long time. And, you know, you could say people in, you know, more broadly in science fiction, in moral philosophy have been, you know, discussing these things for generations.

1:01:19but uh the um you know the general idea that okay you know your objective function should include you know making things go well for humanity right like you know the the the how do you define go well yeah yeah right the uh the the current humans should be able to look at the world that you have created that you have shaped and say yes we like that world you know yes we would take that right At the same time, though, you don't want to base everything on the whims of current humans. So if AI had been created in the year 1800, we would not want to lock in the value systems of 1800, including slavery and the subjugation of women and all of those things.

1:02:05Right. You know, in some sense, what we want to say is, you know, AI should implement those moral values that we would have if we spent thousands of years thinking about it and discussing it and improve and refining our morality. And, you know, there's an idea that tries to capture this called coherent extrapolated volition, CEV. Right. And so, you know, I think there's something to be said for that. My favourite one I've heard that I'd love to hear what you think on it is it's just a quote by a sort of he's actually a computer scientist as well. But also a philosopher, Forrest Landry, and it's love is that which enables choice.

1:02:49So in other words, like a loving good act is one that enables, empowers the other to make the best choices possible. Well, I mean, I feel like at this point, like things are going to go in a theological direction. Right. Like like like like once the AI becomes powerful enough, then, you know, it is it is effectively a God. Right. And then, you know, these very ancient questions of, you know, should God just make everything optimal for us or should God give us free will? Should he give us, you know, like like, you know, I think personally, if I were a God designing the world, I would want people to have free will, but not so much free will that they could then, you know, genocide other people, for example.

1:03:33Right. Well, again, because that would counter the loving acts because you're taking away their choice making. Exactly. No, I have a bone to pick with God. This, of course, is nothing other than the theodicy problem. But, you know, if we do get to design the whole world anew because we're going to create an AGI that can reshape the whole world and put values into it, then I would like it to give us freedom, but not enough freedom that we can make other people's lives miserable. What about as a possible axiom maximizing playfulness? I mean, it sounds good. There's also Elon Musk's founded XAI on the principle of maximizing pursuit of truth.

1:04:25These things all sound good. For each one, you can construct a thought experiment where it's taken to an extreme and it leads to a dystopia. So these are the age-old problems of moral philosophy. I mean, it will consist of multiple values that are being traded off against each other in all of these situations. I mean, it is the age-old problem. Like, any virtue ethics, deontology, utilitarianism, if you take any of them, for any of them, you can also construct an example that seems obviously wrong. For deontology, it's like the axe murderer standing in front of you and now you're not allowed to lie and you're hiding the person in the room, right?

1:05:04You're like, oh, no, I'm going to tell you the truth. I suppose you go kill that person. I mean, like humor, playfulness, scientific curiosity, natural beauty, love. You know, I feel like these are all good things, right? I feel like, you know, these are all things that I want somewhere in my objective function over possible worlds. And, you know, how I trade them off against each other is, of course, a much harder question. Coming back to sort of the open AI stuff, you know, I am just obsessed with seeing race dynamics everywhere and how to calm them or at least directionally make them be a race to the top and very clearly a race to the top as opposed to a race to the bottom and I mean the AI arms race just seems to be getting hotter and hotter like you know we started off with just one player which was DeepMind but then OpenAI was created almost as a like response to people not being happy with DeepMind having all the control and then Anthropic spun out from there and everything else is sort of spinning out and it just it seems like this inevitability did your time at open ai give you any insights into how to mitigate this maybe so uh i mean it is an incredible story that you know you look at these three companies uh deep mind open ai anthropic each one was started on an explicit thesis of like we have to do this safely before someone else can do it unsafely And then each one, over time, moved, in some people's judgment, moved toward the unsafe side, toward the side of being sort of the very thing that it was set up to beat.

1:06:44And then that led to the next one being stored. So it's clear that we have a hard coordination problem here. and uh uh so you know i read a few months ago i read this uh rather you know remarkable book by leopold ashenbrenner who i who i knew well when when uh uh we were both at open ai he was on the super alignment team right rest in peace yes and um and and and you know it's very much about these race dynamics right he sees a lot of the same things that the uh the uh the ai doomerists see but then he reaches a very different conclusion at the end. He says, and therefore it is essential that the Western world do this before, for example, China does it or Russia or Iran.

1:07:32And it's, which is certainly an argument that one can make. And I can, it depends on what are you most worried about, right? Are you worried about, you know, AI in the hands of the wrong person? Or are you worried about AI that doesn't need to be in anyone's hands because it has its own goals, separate from ours, right? And, you know, my position tends to be, why not worry about both? But at some point, you know, this will become a choice of which are we more worried about, Right. And so, so, so, yes, I mean, you know, we've seen, you know, we've now seen very clear race dynamics, right, where, you know, each company, you know, like, like, you know, had plans about, you know, responsible scaling and things like that.

1:08:31But each company is also in very direct competition with the other companies for customers. And, you know, you now have all of these investors, you know, pouring billions of dollars into, you know, scaling up the GPUs and so forth. And those investors want to see a return. And some of those investors have become, you know, just explicitly hostile to, you know, dismissive of, you know, any concerns about safety. Right. And so, you know, we've seen this come to a head with the debate, for example, over SB 1047, which is the bill that was passed overwhelmingly by the California state legislature, which is now at the time that we speak sitting on the desk of Governor Gavin Newsom.

1:09:22But what that bill does is, I guess it's the first bill that's sort of directly aimed at sort of scaling of AI. And it's very light touch compared to what some people would like, which is just like Eliezer, for example, would like to just shut this all down, just pause everything. Okay, but SB 1047 doesn't do that. It says that if you spend more than$100 million to train a model, then you have to submit a safety plan to the government and notify the government about what you're doing. And if your model then causes a catastrophic harm, which they define as like causing more than$500 million in damage or something like that.

1:10:11And if you failed to reasonably prevent that harm, for example, by following your own safety plan, then you can be held liable for that. And then it also establishes whistleblower protections for employees of AI companies, which actually we've already seen. Right. OpenAI clearly needed that. Yeah. Yeah. So I think all the things that this bill does are very modest and reasonable. Personally, I would like to see it pass. And if it doesn't, then I would like to see something like it pass. You know, everyone agrees that this should ideally be happening at the federal level. just that the federal government takes a very long time to do anything, and by the time it does something, maybe it no longer matters.

1:10:58Yeah, it's also possible that the EU would do something like this. I mean, they've already passed an AI Act, but they might continue doing things. But I think anything of this magnitude, right, I mean, this is going to be at least as impactful to the world as, I mean, the internet would be a very loose lower bound on how impactful this would be. I mean, nuclear weapons is another analogy that people constantly reach for. And no one ever suggested that the private sector should just be completely free to innovate in nuclear weapons. Anything that has the possibility of causing this much harm, it seems inevitable that governments will get involved, whether anyone likes that or not.

1:11:44And so then the question is just, are they going to get involved in a good way or a bad way? What frustrates me so much is people like, oh, we can't have governments getting involved because they always get it wrong and so on and so forth. It's like, well, it's inevitable that governments are going to get involved at some point. So would you rather it be done now where there's actually a little bit of breathing room and people aren't like losing their minds because some terrible catastrophes just happened, which is going to happen at some point? or would you rather wait until after catastrophe and then like everyone like now this maximum political pressure and they're just going to clamber for as much power as they can and push through something without much thought like to me the type of regulation you're going to end up getting post catastrophe is going to be actually far worse from a sort of libertarian perspective than the than the regulation you'd get if you actually do it prior like now with this like long thought out it's been pushed backwards and forwards but it's just like there are these demagogues basically like i'm gonna say mark andreessen and so on who just clearly have their agenda which is maximize their bottom line and will say whatever they want to get this bill shut down and pressure newsome however is extremely frustrating um and i think counterproductive to what they want which is for ai to actually do good stuff yeah i mean i mean many people have made the point that if you leave it unregulated you increase the chance of a giant catastrophe that could then lead to, you know, overregulation, that could lead to an error in the opposite direction.

1:13:12That seems to be, you know, that's one reading of what happened with nuclear power, for example, right? That, you know, a three-mile island in Chernobyl were allowed to happen, and then that killed the industry, and, you know, much to our detriment today. Right, exactly. The safetyists are in a very strange position here, because, you know, they, like, usually calling for government regulation is what, you know, progressives, the left do, right? But, you know, the AI safetyists, like, they tend to be libertarian about almost everything, right? They tend to, you know, love the free market, love, you know, principles of supply and demand.

1:13:54It's just that they carve out a big exception for something that they think could literally kill everyone on earth. Right. You know, now I'm, you know, I may be, you know, like pro free market compared to most people, but I've never been a doctrinaire libertarian. Right. I think that the free market is actually it's not some sort of state of nature. Right. It's a very unnatural creation that we've made, a very valuable creation. Right. But like the true state of nature is someone doesn't like you, they just, you know, send goons over with baseball bats, right? And, you know, the idea that we're going to have a free market, but we're not going to have violence, right?

1:14:37That was, you know, that had to be painstakingly created via government, right? And, you know, so the whole, you know, system, you know, where we have free markets only exists because we have states that are strong enough to enforce that. And the basic goal of the state, it has no more basic goal than to protect the survival of the people in it. Anyone who looks at the market knows that externalities exist from it. And it's also very clear that the companies that create those are going to hide it. Historically, that's proven itself to be the case with tobacco companies putting out bogus science around it, asbestos companies knowing that it leads to cancer and still hiding it.

1:15:26Currently, we have seen 3M having been given a$10 billion fine for having known that they're leaking PFAS chemicals and hiding it again. So we've very much seen that the market players can regulate themselves, but only when the feedback they receive from the customers is like either fast enough or like the customers can't even give the feedback but some things are of the type that the customers can't give the feedback as it was with the examples I just mentioned I think with risk it's also the type of thing that people don't see that's why we've implemented seatbelts because people like don't necessarily know that yeah people are not very good at estimating I think risk when it's a bit further out or?

1:16:12Yeah. So, you know, I had really hoped that the sort of AI alignment nerds and the progressive leftists could make common cause on this question of AI regulation, right? Because, you know, to a progressive, you know, despite how weird and science fiction-y, you know, this all is, a part of it just looks like a very standard question of, you know, companies are putting out products that are unsafe. So therefore, we have to regulate them or, you know, we have to hold them liable for it. Right. That that's a position that, you know, you think people on the left would be very comfortable with. And to some extent that has happened.

1:16:53OK. But there's also this incredibly unfortunate tribal split, I think, within the AI safety community. It's ethics and safety, right? Some people will say, you know, you are not allowed to mention anything about existential risk, about, you know, AIs that would recursively improve themselves. That is all science fiction speculation. And that is all distracting us from the real harms of AI, which are all about bias, about misinformation. And, you know, there are all these near term things. Right. And, you know, on the other side, you have some of the doomers saying, you know, you don't even talk about the near term things about bias or misinformation, because these are all trivialities compared to the literal survival of humanity.

1:17:40I say, no, it's all on a spectrum. Right. you know eventually i see no reason why eventually we won't be up against the these giant civilizational questions about you know what kind of world do we want in the uh uh once ai is better than us at just about everything okay but the only way we make progress is by looking at near-term stuff so why not worry about both right exactly and it's we have plenty of people some work on these problems some work on these problems again it's a yes and not a no but and yeah i mean i guess there's some scarcity of funding and talent and so on but it's i mean again it's in part it's a media problem because the media just glom onto whatever's the most dramatic thing of the day which tends to be you know they're like extinction risk stuff and so i can understand the like ethicists are like well they see all the headlines of that and that gets all the attention but it's It's just like both are real problems.

1:18:38It's not as science fiction as you think. If it's even 20 years away, that's a huge problem. But it's probably a lot closer to some of these really big catastrophic risks. So it's just like it's very frustrating. I mean, whenever you see someone saying, well, you know, these people who are not from our tribe are not allowed to steal our issue, then you always wonder, well, then how much did they care about the issue in the first place? Yeah. just going back to you know you so you had your two years at open ai yes like personally you know they're i mean i'm worried about goings on within all of these big companies but certainly the behaviors we've seen of the leadership of open ai has been the most concerning to me um just given the sort of consistent lack of candor etc um and the you know the sudden pivot from non-profit well not sudden, but probably planned, but pivot from non-profit to for-profit.

1:19:30What's your read on the, you know, because you know the leadership there, presumably, are they just paying lip service to safety concerns or is there a master plan going on? I think that you can find, you know, in an enormous range of views, like within a place like OpenAI, right? It is, you know, it's now a big place. When I joined, it was maybe 300 people, and now I think it's almost 2 ,000 people, okay? And, you know, you could find the whole spectrum of opinions there. No, I do not think that there is any, like, smoke-filled back room where people sit and laugh at all the rubes who bought the cover story.

1:20:18I think that just about everyone thinks that they are doing the right thing. The question is, has their perception of what is the right thing been colored by self-interest? Should they be the ones to make the decision? That's why one of the things I support as well a lot that Bill tries to do is increase transparency and have other entities also have a look on I mean, one of the big ironies here is that like if you want to, you know, hold Sam Altman to account, for example, like you don't have to say anything that Sam Altman himself wasn't saying five or six years ago. Right. Right. He was calling for this stuff.

1:20:58Yeah, exactly. Exactly. And so so the things that he called for in the past or when he tests even more recently when he testified before Congress, I very, very strongly support all of those things. Why do you think he's pivoted? I don't know him well enough. I've had all of three or four conversations with him. And, you know, in those conversations, he was delightful. And, you know, I enjoyed talking to him very much. And I had no idea of what was coming. And just want to also, on the bill, kind of, because not to only speak as if it's obvious, I think there are, like, valid concerns, even when the bill intends to do a lot of good things, it may be the case that the specific implementation of it has like trade-offs that are too curbing innovation truly too much right and like someone can look at whether it does it i think it's in my estimation pretty clear that the benefits outweigh the downsides within the bill and though it is not an opinion that anyone could hold if they simply didn't believe that there are any risks whatsoever coming from ai that's right and that's why i think many of the conversations with the bill should actually start with, do you actually think there are risks?

1:22:09If you don't, then it's clear. The reason why people have been so vociferously opposed to it, you know, even though it actually does so little, the regulate rate is that they don't want to acknowledge the principle. They think like once you, you know, once the safetyists get a foot in the door, that there's some legislative acknowledgement of the validity of their concerns, you know, then they're going to push the door wide open. That would be my steel man of them. I think it would be great if it wasn't the case that someone who just thinks that risks are sci-fi and silly wouldn't tell people that believe that risks are not silly that, oh, yeah, that's a bad bill.

1:22:48It's like it should be like the person who actually believes in risk should first understand whether they're listening to someone who is discarding the bill on the basis of or discarding other attempts at safety on the basis of them not believing in risks. Yeah, no, I mean, I think that the accelerationist position is just that progress has always been good in the past and therefore it's still good now. And I would say like I am strongly pro-progress as well, right? More so than most people maybe, right? But I think that government has also often had an essential role in, you know, shaping the direction of progress, right?

1:23:23We need only mention the example of nuclear weapons, right? Where if it was just a free-for-all, then we wouldn't be here having this nice conversation. So you've worked in quantum computing for over 20 years, and you actually helped in 2019. You contributed to Google's achievement of quantum supremacy. Yeah, I wasn't directly involved, but our group did the theory that sort of led to that experiment happening. Can you give us an overview of the current state of quantum computing? Yeah. So quantum computing is actually in a very exciting time right now. I feel like if it weren't overshadowed by this gigantic behemoth of AI, it would look bigger.

1:24:11But within the last year, people have managed to do operations on pairs of qubits, which are quantum bits, two qubit gates, that are about 99.9 % reliable. And they can do this in a fully programmable way in systems of 50 or 60 qubits. and so then you can do thousands of these operations, producing some complicated entangled state of all the qubits so that you can then measure and you can see that you did the operations that you wanted. You can solve interesting problems this way. You're just barely getting to the point where you can beat a classical computer, right? And I think beating a classical computer in a way that's economically valuable.

1:25:05I would say that hasn't happened quite yet, but very plausible that that's coming within the next few years, let's say. But the real prize that all the major players are racing towards, and that means Google, IBM, Microsoft, a bunch of venture-backed startups like PsiQuantum and Quantinuum and basically like every strange name involving the letter Q has been taken by one of these startups. But what they're all raising to do is to try to get what we call a fault-tolerant quantum computer. And so this is one that could sort of run for an arbitrary amount of time. Is that when they get above the quantum correction threshold?

1:25:55Yeah, the error correction threshold. OK, so so basically, you know, from the very beginning, like it was realized that the key engineering problem in building a quantum computer is that qubits are very fragile. Right. They you know, what we're trying to take advantage of is that they can exist in a superposition of states, which, you know, we can explain what that means. Most people watching will know what that is. Okay, okay. But the trouble is, you know, systems are only in superposition, sort of, as long as no one's looking at them, right? Or more generally, as long as they are isolated from their external environment.

1:26:34okay like if i have a qubit that has some amplitude to be zero and some amplitude to be one so it's a super superposition of the two but now the information about whether it is zero or one leaks out into the environment then it is as if the environment has now measured the qubit which means that the qubit randomly snaps to either zero or one okay and now it's just one or the other okay and then it's not useful to the right exactly quantum computer right right Right, and then it's no longer useful for quantum computation. You lost one of the 54. Right, now it's, you know, if that keeps happening to all my qubits, then it reverts to being a classical computer, right?

1:27:13And so, you know, this is such a severe problem that, like, in the mid-90s, there were distinguished physicists who said, you're never going to do this, right? You could maybe build tiny little toy demonstrations with a few qubits, but you're never going to scale this up to, you know, millions of qubits. as you might need to factor a giant number, do things like that. And then a key discovery happened that changed the minds of almost all experts, and that was the theory of quantum error correction. And what that basically said is you don't have to get the error all the way down to zero, or the decoherence, it's called, the loss of quantum coherence, the leaking of information into the environment.

1:28:01minute. You merely need to make it very, very low. It's like 0.01%, right? Something like that. You know, it depends on what operations we can do and what assumption, but yes, something like that. Like if I can get my operations, let's say 99.99 % accurate, right? Then this is good enough that if I now encode the qubits I care about, what I call the logical qubits, across entangled states of large numbers of physical qubits, then I can use these error correcting codes where, you know, if any small fraction of my physical qubits leak or, you know, suffer errors, I can still recover everything I care about from the other qubits, okay?

1:28:46And I can be constantly monitoring my qubits. Now, here's the clever part, okay? Monitoring only in a way that tells me, has an error happened? And if so, what do I have to do to fix it? Okay, I don't want to monitor to say, is my logical qubit a zero or a one? Right, you can't look in. Right, I don't want to know that. You need the meta information. Exactly, exactly. I need the error syndrome, but I don't want to know the states of the logical qubits. Okay, but it turns out, you know, people develop these schemes where I measure only to learn the error syndromes and what I have to do to correct them.

1:29:20And now, you know, the trouble is, you know, all this error correction machinery will itself be subject to error. So you might say, you know, it's like a cat chasing its tail. Right. And, you know, you're but but what what what was discovered was that as long as your physical error rate is below a certain threshold. OK, then each round of error correction that you do is making things better rather than making them worse. So it's sort of self-sustaining. Exactly. Yeah. It's like you have a self-sustaining nuclear chain reaction or, you know, use whatever analogy you want. And so the engineering goal of the field for the last 30 years has been to, you know, build physical qubits that are good enough that then error correction can get you the rest of the way.

1:30:03And, you know, so long story short, after 30 years, if you just look at the numbers, we now seem really damn close to that. Right. You said we're you said we're at ninety nine point nine. That's right. That's right. Like one order of magnitude. Exactly, exactly. If you look at either trapped ions or neutral atoms or superconducting qubits, you know, which are three different hardware architectures that are being pursued in parallel. But, you know, in all three of them, you know, people are sort of converging around these three nines. Right. And, you know, they really want to get to four nines, you know, at least before you really make sense to scale this up.

1:30:41OK, but, you know, you look, I mean, when I entered the field, you know, 25 years ago, it would have been like a nature paper if you could get 50 percent. Right. And then the 50 percent became 90, became, you know, 99, became 99.9. Right. And so I think, you know, one thing that we've learned from our experience in AI is, yeah, you know, look at trend lines. Yes. And, you know, even if you can extrapolate. Yeah, even if trend lines are leading someplace that looks insane, well, it could be that something's going to break, but it could also be that you're going to reach that place. So far, people are also optimistic about further engineering innovation to improve that within each of those hardware options that you suggested.

1:31:28The quantum computing experimenters, if you talk to them, they are always super optimistic, at least about their own approach, right? They're usually pessimistic about all the competing approaches, right? And they will spend hours telling you why the competing approaches won't scale, right? Or why they'll be massively hard to scale, right? And of course, you know, the fear is that like, you know, each one is being truthful when telling you about the other guys. And, you know, you don't know if they are when telling you about their own stuff, right? because they all have huge financial incentives at this point.

1:32:00There are billions of dollars now being invested in quantum computing. And it's a pittance compared to what's being invested in AI. But by the standards of quantum computing, which started as this very theoretical academic research enterprise, it's huge. How strong are the race conditions in quantum computing? Because presumably it's nothing as intense as the AI world, right? Yeah, well, okay, so there certainly is a race. There's a race between the different hardware approaches, like superconducting, trapped ions, neutral atoms, photonics, which one will get there. It's possible that multiple of these approaches will work, and then it's a question of which one gets there first, which is the most economical, and so forth.

1:32:48There's also competition between countries. So the U.S. in 2017 passed something called the National Quantum Initiative Act, which was spearheaded by then-Senator Kamala Harris. You know, I visited her office while they were writing it. She sent regrets that she couldn't meet me because she was at the Kavanaugh hearings. But but I you know, I met her staff and this bill provided about a billion dollars for quantum information research. it was passed unanimously by Congress. Like, what does Congress do unanimously anymore? Only something that neither side understands, right? But what they did understand was that China was doing something in quantum and that the U.S.

1:33:37had to beat China in the race for quantum, right? So to understand the race better, I think it also helps to... So like one of the misconceptions about quantum computers is that it's just an amazing computer that can do all sorts of things while it's actually for a specific set of tasks. Can you describe the types of tasks? This is a horse that I have been flogging for 20 years or a boulder that I've been rolling up a hill, right? But I mean, the narrative that sort of took hold very early on in popular writing and also in the business community and the investment community was that quantum computing is just a magic accelerator of everything.

1:34:18It's just the next stage in the evolution of computing. I think, you know, people just interpreted the word quantum to mean really awesome, right? And in particular, you know, starting around 15 years ago, you know, a narrative took hold that, you know, what quantum computing is really going to be good for is speeding up, you know, AI and training neural nets and optimization and finance and all these important tasks for industry. And of course, this is exactly what VCs want to hear. This is what CEOs want to hear. The only problem is that it doesn't match just about anything we've learned about quantum algorithms from all of the research in this subject.

1:35:04So what we learned in quantum computing theory, starting in the mid-1990s, is that a quantum computer really would give you dramatic advantages over any known classical algorithm, but mostly for a few very special tasks, right? Maybe the economically most important thing that a quantum computer can do is just help you simulate quantum mechanics itself, right? Now, that may sound esoteric, but that's actually useful for anyone who's designing new materials, new chemical reactions, you know, new ways to make fertilizer, batteries, photovoltaics, you know, for drugs, you know, that have to bind to some receptor in a certain way, right?

1:35:54These all involve many body quantum mechanics problems. And, you know, it's not obvious that a quantum computer helps with them because there's also extremely good classical heuristics for all these problems that the material scientists and the chemists have been forced to develop over, you know, many decades. because they only had classical computers, right? And now, you know, you have to compete against all that stuff and be better than all of it. But, you know, I think it's plausible that a quantum computer will give you wins there. And, you know, even if there are only a few wins, they might enable billion-dollar industries.

1:36:29So I'd say that's the biggest economic thing. And then there's a second really huge application, although it's not really clear that it's a positive one for humanity. And that one is breaking almost all of the public key cryptography that currently protects the Internet. OK. And this comes from a famous discovery by Peter Shore 30 years ago. He was then at Bell Labs. Later, I was a colleague of his at MIT. But Shore showed that there is a fast quantum algorithm for finding the prime factors of a huge composite number. And that's what like all RSA encryption is based on. Yeah, that's what RSA encryption is based on.

1:37:10If you can solve that problem quickly, then you can break RSA. Now to loop back to what we were talking about before, there's this giant blob of problems in P, and then there's this giant blob of NP-complete problems, right? And P versus NP asks if they're the same. Factoring is this odd one out, okay? Factoring is an NP problem that we don't know to be in P, but we're also almost certain that it's not NP-complete. So what would it be in? Well, it's called an NP-intermediate problem, or at least we think it is, right? It's somewhere in between. So it seems to be an NP problem that is, as far as anyone knows today, at least in public, seems hard for a classical computer to solve.

1:37:57That's why we use it for cryptography. But it has very, very special structure that seems to prevent it from being np complete okay just one example of that special structure if i give you a jigsaw puzzle right a priori there might it might have no solution right or it might have a hundred different ways that you could solve it right uh you know imagine that that there's no picture on it for example right uh um but if i give you a huge number you know for sure that it has one and only one prime factorization because Euclid proved that in 300 BC. So the special structure of factoring that comes from number theory and group theory is actually essential to why it's so useful for public key encryption.

1:38:43We don't know how to base the public key kind of encryption, the kind that we use on the internet, on any NP-complete problem. We do know how to base it on factoring. But then what Shor showed was that that same special structure of factoring enables a quantum algorithm to solve factoring. And that turns out to be true for a bunch of other problems in number theory that we also use in cryptography. And to the point where it was a challenge to identify public key cryptosystems that are not broken by quantum computers. Today, we have pretty good candidates for that. So there is a push right now to migrate to what is called post-quantum encryption or quantum resistant encryption, which would be encryption methods just running on our same conventional computers, but that at least as far as we know would be resistant against quantum attack.

1:39:42Yeah, because this is surely like a new arms race, therefore. Yeah, but it's an arms race where you could say, in principle, we kind of already know the solution. The solution is for everyone to upgrade to these quantum resistant encryption methods. And assuming that all goes well, then we're all just back where we started. The main issue is it's a huge practical headache to upgrade every router and every web server and every browser in the world to use these quantum-resistant crypto systems. Those are the two most obvious applications of a quantum computer, simulating physics and chemistry at the quantum scale and breaking public key encryption.

1:40:25And then there's everything else. There's all these problems in optimization and machine learning and combinatorial search. So you could say the bread and butter of computer science. And for these tasks, our expectation is that quantum computers will only give you a more modest benefit. So they're not going to solve problems in polynomial time that take exponential time, classically. They might reduce the order of the exponential. OK, and so in particular, most of us believe that there is not a quantum algorithm to solve NP-complete problems in polynomial time. We can't prove it. Just so you know, I mean, we can't even prove there's not a classical algorithm to solve them.

1:41:12That's the P versus NP question. But it would have to be radically different from any quantum algorithm that we know. So, you know, even quantum computers seem to have limits. Which do you think we're likely to get to first, simulating quantum mechanics or breaking encryption? I think usefully simulating quantum mechanics will be first. Awesome. How many qubits would we need to choreograph such that we would be able to do either, you think? These are all kind of fuzzy questions, right? Because, you know, the real, I mean, people are already doing things with quantum computers that are cool and interesting.

1:41:46but it just, you have to squint to see, okay, did you actually get any benefit compared to what you could have done with a classical computer, right? It's when you ask that question that things always get tricky in this field. But I think that if you had, you know, let's say 200 qubits, certainly if you had 200 logical qubits, you know, that were error corrected, then you can already do quantum simulations that are going to be scientifically interesting, that are going to be interesting to material scientists or chemists or people like that, maybe, maybe if you got lucky, they would also be commercially useful, which is kind of a higher bar to clear.

1:42:28Okay, you know, if you had thousands of physical qubits, then almost for sure, I should think, you could do things that would be useful to certain industries. For breaking RSA encryption, you're going to want several thousand qubits that will definitely have to be logical. They're definitely going to need error correction. And then you're going to need millions of operations on those qubits. So we would actually have a bit of run-up to the RSA encryption functioning on a quantum computer, which is why we would also potentially have sufficient time to upgrade all of our cryptography to quantum-resistant ones.

1:43:06I mean, I would say that anyone who is really worried about their data staying secret for the next decade, they should already be transitioned. Well, I mean, that's why I wonder about how it affects the race because Iran is worried about the U.S. getting there and the U.S. is worried about Iran and China and everybody else getting there as well, right? Now, I don't want to overstate, you know, the national security importance of this too much because the truth is that, like, usually, like, people build these fortresses, right, in cybersecurity, and usually the way that you get into the fortress is just by finding a screen door in the back that was just left totally unguarded, right?

1:43:45Or convincing a person to just let you in. Exactly, exactly. So usually in practice, the way things are broken is that there was some memory allocation bug in some level of the software stack. And you pay people at the NSA or the GCHQ or Unit 8200 or whatever, and they find those mundane vulnerabilities, you could call them, right? But, you know, one thing that we learned a decade ago from the Edward Snowden revelations, right, is that the NSA does have a big item in its budget for, you know, spending lots of compute to break cryptography in a way that looks exactly like what it would be doing if it were just breaking 1024 bit RSA and Diffie Hillman, you know, using how much money it would cost to build, you know, to build.

1:44:43supercomputers just for doing that. And so that suggests, first of all, that at least as of 2013, they don't seem to have had a quantum computer in their basement or a classical factoring method that vastly exceeds what was known in the open world. Or, of course, it could all just be a giant cover story and you can get as a conspiracy theorist as you want about it. But, you know, if you believe that this is this was really their budget and what they were spending it on, that it seems like, yeah, they pretty much knew the kinds of factoring methods that we know in the outside world. And yes, if they had a quantum computer, then they could speed that up.

1:45:27But, you know, I would say, you know, people sometimes want to compare quantum computers to nuclear weapons. Right. I think that that's that's not a very good analogy. I mean, you know, for one thing, the quantum computer doesn't directly kill anyone unless like the dilution refrigerator tips over onto them or something. But secondly, you know, having a quantum computer that breaks encryption is mostly useful if no one knows you have it. Right. It's a little bit more like Bletchley Park than like the Manhattan Project. Right. Like once everyone knows you have it, then they all just switch to quantum resistant encryption systems or, you know, Their motivation to do so has then enormously increased.

1:46:08Whereas with a nuclear weapon, it's just the opposite. You want everyone to know that you have it, but hopefully never have to use it. Some people will wonder also about the Bitcoin-related question here. I think Bitcoin uses a different algorithm in part. So with Bitcoin, there are different components of it. But Bitcoin uses a signature scheme, which I believe is based on elliptic curve cryptography, which would be breakable by a quantum computer. OK, so so the the digital signature part of Bitcoin as currently implemented is vulnerable to quantum computing. Now, that could be any any any zero knowledge proof.

1:46:45Yeah. Well, not any zero. Not any, but the current ones. But the currently used ones. Yeah. Now, now, if, you know, Bitcoin made a decision to fork to some quantum resistant encryption, then, you know, that could fix that problem. OK, I actually just recently met with the Ethereum engineering team, which wanted to know, like, you know, like basically how much time do we got, Doc, was sort of the question. Right. And and should they be worrying about post-bundum encryption? And I think, you know, quite plausibly, yes. Right. But then then, you know, what a lot of people think about when they think about Bitcoin is the proof of work.

1:47:23Right. You know, that's like, you know, the thing that actually hogs some like appreciable fraction of all the world's electricity on this sort of useless, you know, trying to invert a cryptographic hash function. And for that component of Bitcoin, we think that a quantum computer would only help modestly. It would only give one of these polynomial improvements or what we call a Grover improvement. which so so so that's like if you have a problem that classically took you about n steps then a grover improvement lets you solve it with a quantum computer in only about the square root of n steps okay so that so that's an improvement but it's not a an exponential to polynomial kind of improvement and now the interesting thing is you know the way bitcoin works the hardness of the proof of work is just set by, you know, the total computing power in the world that's currently being used to solve these puzzles.

1:48:24So in a world where everyone had a quantum computer, all that would happen is the protocol would automatically adjust to make the proof of work that much harder, and we'd all be back where we started. So the proof of work as a means of security would actually then persist having the same property as it was before. Now, if only one person had a scalable quantum computer and no one else did, then that person could get very rich mining Bitcoin. If it was a big enough and fast enough quantum computer. It's actually quite a while before these Grover speed ups become a net win in practice. Now you're back full time at UT Austin.

1:49:04I'd love to get your sort of status report on the state of academia in terms of, you know, from my very Twitter brained perspective, it feels like, you know, university campuses are a very stressful place to be if you are, especially as faculty and students, people are always self censoring. and there's this culture of, you know, you must have certain political beliefs or you get shut down. At the same time, I'm hearing rumblings that things have maybe peaked in that regard. You know, we're past peak woke or whatever you want to call it, and things are becoming a little bit more moderate again and people are more comfortable to speak.

1:49:45How have you, what have you noticed? Do you agree with that trend or is it still a problem? It is something that concerns me greatly. I'm probably not the best person to ask just because I live most of my academic life in a bubble, right? I live in a sort of bubble of, you know, mathematicians, computer scientists, physicists, you know, and sometimes also I will interact with historians or English professors, you know, especially if they're parents of my kids' friends, right? uh uh but you know like the the ones i'll interact with obviously you know will not be the ones who would you know refuse to speak to me because of you know heterodox beliefs or things like that they would be you know the uh the more open-minded ones uh so like i'm kind of insulated from this stuff uh except insofar as i blog about it right and anything i blog about i will hear from whichever people on earth are the most angry about that thing right so you get a lot of comments Oh, yeah.

1:50:50Oh, yeah. Yeah. So. So, look, I mean, I mean, in, you know, I did see the articles, you know, arguing that, like, by some measures, wokeness seems to have peaked around 2020 or 2021. And now, you know, back, back, you know, maybe it's back down, although, although still kind of at a high point compared to maybe, you know, where it would have been, you know, when I was a student, you know, 25 years ago, or whatever. I mean, within the last year, it has been a stressful time for academia. We've seen universities basically taken over by the Gaza protesters. We've seen, and I think that a lot of these things will be adjudicated in court, right?

1:51:40Because there are laws like Title VI, right? That if you've created an unsafe environment for certain students, you may have run afoul of those laws. And then there are big questions about free speech. Many people pointed out the irony. Yeah, yeah, yeah, right, right, right. Many people pointed out the irony that there were academics who were always very dismissive of free speech. They wrote it as freeze peach to just sort of ridicule anyone concerned about it. And then during the Gaza protest, they suddenly rediscovered the value of free speech and became the biggest First Amendment absolutists.

1:52:27Right. So I think it's very, very important that we come up with some viewpoint neutral rules and then actually enforce those rules in a consistent way. Right. And, you know, we ought to I think Stephen Pinker, for example, wrote some very nice things about this, that we ought to start with just what is the purpose of a university. Right. You know, the the the purpose is to, you know, have a place where ideas can actually be debated rather than just screamed and, you know, rhyming slogans. Right. And and so that means, you know, you want to attach very, very high protection to, you know, people who are presenting ideas that other people might find offensive or or, you know, even even harmful or or or things like that.

1:53:16But, you know, what you don't necessarily want to protect is, you know, shouting down a speaker who you disagree with or blockading a building. Right. These are things that that don't actually advance the discussion and debate of ideas on their merits. In fact, quite the contrary. Their purpose is to shut that down. Right. And so I think that it's good to come up with rules about these things that that are that are totally content neutral. And then, you know, the the the hard part, as we as we've seen in case after case, is to actually enforce those rules when when when when when people flout them.

1:53:57It's I mean, in part is, you know, coming back to, as you said, it's like, what is a university for? and the trouble again is that there are always these competing incentives like what the the incentives of the dean in theory should be aligned with what the university is for but at the same time they have all these smaller ones of keeping the students coming in you know like uh performance metrics etc are there any specific sort of area like what would you do if you're a dean and a dean for a day uh yeah i think maybe the first thing i would do is resign and not be a dean okay that's not a loud part of the game so you're like specification gaming me here ai back in the box yeah no i am incredibly grateful to the people who do this and who sort of fight on the side of truth and justice you know i have met such people i could never do it myself right right but it's like you need such people doing this you know to uh uh uh oppose the bad people who are who are who who are doing this stuff right um but uh uh uh uh look you know i would i would do my best to try to uphold the values of a university you know as as i see them of the dissemination of knowledge and the open search for truth.

1:55:29Enlightenment values. Yeah, the values of the enlightenment. Is there something that comes to mind to align an aspect of how the university currently works so that students get more out of it or that students' overall achievement across those metrics is more aligned with, say, like the professor's incentives or the faculty overall. I would love to see admissions, you know, based on merit, you know, meaning based on, you know, things like standardized tests. And, you know, I actually, you know, people complain about this constantly. They say standardized tests can be gamed, but then the part that they never want to say is that they seem much less gameable than all the other stuff.

1:56:17that we currently use instead of that. So basically what we have done for any high school student who wants to get into an elite university, like we have forced them to sort of redesign their entire teenagerhood around this sort of beauty pageant, this sort of competition of optimizing their attractiveness to these admissions officers, right? In ways that are extremely gameable, right? That the richest and most well-connected parents are the most able to take advantage of, you know, and it is very opaque. You know, it's never clear, you know, and so which means that there's a huge advantage to those who are most in the know, right?

1:57:01It is very, you know, there are like unlimited opportunities for the admissions officers' personal biases to get in. And I think, ironically, if you look at, let's say, Europe, which people on the left are always pointing to Europe as better than the US. But in most European countries, it's just based on a standardized test score. Though at the same time, the universities in the US seem to be of higher quality. top 50 universities are like, I don't know, 30 American ones probably or something. Right, so there's a question of, you know, is that because of this opaque admissions process or is it despite it, right?

1:57:53And you can see historically what's happened, right? Once the SAT was instituted, you know, in the like 1910s, 1920s, you know, what happened was that like Harvard, Princeton and Yale got like enormous numbers of Jews. Right. And this was seen by them as a huge problem. And so that is why they designed this holistic admission system. Okay. This is, you know, that this, this is a matter of historical record. Right. We, we, we, we, we know all of this now, right. That, uh, uh, uh, you know, those three universities decided, you know, we have to look for well-rounded gentlemen who do, rowing and who do Gentile men.

1:58:42Almost all men at that point, of course. No, but also Gentile. Gentile men. Young men who've been brought up in the proper way and who know how to conduct themselves and they presented this as a matter of well-roundedness but if you look at their private deliberations, it was all about getting down the number of Jews. OK. And then, you know, and then I should say what happened was, you know, over the decades, you know, Jews, you know, in America learned to game the system as well as everyone else. And, you know, at some point it no longer worked for keeping down the number of Jews. But then it started being, you know, used for a different purpose to keep down the number of Asians.

1:59:27Right. And that was the heart of this Supreme Court decision from the last year, the Students for Fair Admissions decision. that basically they, at least to the satisfaction of the current Supreme Court, they proved that Harvard was discriminating against Asian applicants. So I think I would try to have admissions be more merit-based, which doesn't mean it has to be all standardized test scores. I mean, you can look for students who are really extraordinary in one area, whether that's athletics or starting a company or whatever it is, but not this sort of checklist of beauty pageant things, right, that, you know, only the richest and best connected students are able to do reliably.

2:00:20way so so that's more measurable tests of the time that like you can find them across uh even yeah but not uh like a i had all of these extracurricular outside of school activities that you can only do if you have time and resources available to put your students in well i mean you kind of it sounds like you're trying to actually draw upon like some alignment principles like you want more interpretability yes you want more visibility into the admission system yes more valuability yeah i i want to you know get rid of the uh incentive get the the specification gaming or uh uh whatever you call it uh so yeah so that that you know if you only let me do one single thing then maybe it would be that uh just because that you know it's so important for for for later right often as brian kaplan has documented for for example right the the the most important you you know, a thing that Harvard or Princeton, for example, does for a student is to admit that student, right?

2:01:25You know, everything else is, you know, we like to think of it as important, and maybe for some students it is. Just being admitted, like having that on your record, saying, I got an acceptance, I got accepted by this Ivy League. That is really the bulk of the work. It's not entirely that, because, you know, people can't get hired nearly as well if they just show their Harvard acceptance letter, but then they never actually went to Harvard, right? Like employers want to see that, okay, you actually had the sort of enough, I guess, yeah, you had enough grit and enough conformity to actually do it, to actually go there for the four years.

2:02:02And you've gained from the social networks, et cetera, that comes now with you as a package. I mean, right, which is not, you know, completely absurd, right? Like, you know, these things are really hard to measure, right? How do you measure someone's grit or their... But but yeah, that might be the the single thing that I would do. And then let me let me think. No, I mean, I mean, I feel like, you know, in general, for all the problems that universities have and, you know, we could speak for hours about, you know, I could draw on all my experience of everything wrong with universities. I feel like they are in order of magnitude less broken than K to 12 schools are, right?

2:02:52Than pre-colleges, right? And, you know, like what I would do, I would love to make high schools and junior high schools operate more like universities where students can proceed at their own rate, where they can choose what courses to take, you know, where they can specialize in things that interest them. So you'd inject more optionality within that. Exactly. Exactly. How competitive were you when you were a teenager in terms of... Would you have described yourself as a competitive person? or if not, like were there specific ways it manifested? Yeah, yeah. I was, I feel like I was driven by sort of an intense sort of burning desire that like I have to do something in the world, right?

2:03:41I have to, you know, either do some scientific research or some writing, you know, that if I don't do that, I am a failure, right? So I was definitely competitive in that sense, right? Was it like an internal goal or was it like a sort of, there was a specific external thing you wanted to achieve or was more just this like general? Well, I don't know how to differentiate that. I feel like, you know, I like, like if I don't make a difference, you know, in the world or sort of do something interesting, then I have let down all the people who thought that I would do that. And also I've let down myself.

2:04:20Right. So, so, so I definitely had that. And by the way, I don't feel that nearly as much as I used to. And I don't know if that's good or bad, right? Because on a day-to-day, I'm less stressed about needing to prove myself. But on the other hand, it was that urge to prove myself that led to my doing a lot of the things that I did. Do you think you've sort of scratched the itch in the last part? Yeah. I mean, maybe part of it is getting older. being married, having a family, it just changes your priorities. And, you know, and part of it is, you know, I feel like, okay, in certain domains, like I, you know, I proved what I wanted to prove.

2:05:07And, you know, but, you know, but again, maybe it would ultimately be better if I didn't feel that way, right? If I still, you know, felt like I had more to prove, then I would work harder, and I would do more interesting things. But I don't think that I was hyper-competitive in terms of math competitions or programming competitions or things like that. I did those. I did okay in them, right? But I never achieved any national status in those things. And there's kind of an interesting reason for that, which is that I left for college when I was 15. or well at least that's one reason right so i i uh i left uh it's a complicated story but i i left high school early and i uh went to a place called clarkson in upstate new york where i could get a ged uh and then uh um you know i went to cornell after that which was like the one you know uh one of the only places nice enough to admit me with this strange record.

2:06:15But, you know, I feel like if I were optimizing for just, you know, competitiveness and proving myself, then I would not have done that. I would have stayed in high school for the full time, you know, in order to try to, you know, maximize my whatever, you know, competition scores, chances of getting into Harvard or MIT or whatever. And that's not what I did. I said, you know, I'm not happy here. I think that I'll be happier in college. I want to be learning what they teach in college. And so when an opportunity arose, then I seized it. Yeah. So, I mean, also in many metrics, you are winning by going to college age 15, being able to go to college at age 15 is, I mean, And the inner psycho competitor within me would have definitely been like, that's a cool achievement.

2:07:09Like, it would have been very, would have felt like a tick box for sure. I've beaten my friends. Stepped out of the game and now that's kind of a winning achievement in a way as well. And yeah, I mean, you didn't focus on the short term competition, I suppose. Did you, when you went to college, did you immediately focus on computer science or were you kind of just searching? I did, I did. Yeah. So so so so so at that point, I knew that I wanted to do computer science. You know, I mean, I had been drawn into it by just wanting to make my own video games when I was when I was 10 or 11. And then eventually I realized that, you know, even though I loved programming, you know, I just, you know, learning what programming was, was like learning where babies come from.

2:07:54Right. Like, why didn't anyone tell me about this before? right and it was just it was it was a revelatory right but uh uh what i didn't like and what i wasn't good at was software engineering like you know making my code work with other people's code and learning some whole framework and getting it done by a deadline and documenting it like uh you know and i realized that i would never have much advantage there and i got more and more uh nerd sniped you could say by the theoretical side of computing uh and you know i think you know my my dad may have been a little disappointed. You know, I think, you know, this was during the time of the first internet boom.

2:08:33And, you know, he would keep pointing out, like, look at this, you know, person, you know, only slightly older than you. He just sold his company, you know, $500 million. Does that, you know, does that appeal to you? And okay, you know, maybe like, there's a different branch of my life where I would have tried to start a software company or something. But in this branch, I got nerd sniped by the theoretical side of computing. Was there a specific idea that you found beautiful at the time? Well, I learned about the P versus NP problem when I was 15, and that blew my mind. I spent a month thinking, okay, surely all these experts have just made it all too complicated.

2:09:19Surely I'll just, without their preconceptions, I'll just sit down and solve it. I think it's good for any computer scientist to have that experience at least once in their life so that afterwards they can understand what they were up against.

2:09:39Especially for a pretty gifted child, it's a good thing to get to a point where you just can't talent your way through. And it's just so hard that you have to work on it and work on it and work on it. Absolutely. No. And also, I mean, look, you know, by being three years younger than people, you know, going to college, right, I knew that I was putting myself at a competitive disadvantage compared to, you know, who my classmates would be. Right. But that was what I wanted. I wanted to get as quickly as possible into an environment where, you know, I would be struggling to keep up with other people, which would mean that I could be learning from those people.

2:10:16Okay. And so, yeah, and I was very interested in AI at the time. This was like the late 90s. I worked with an AI professor at Cornell named Bart Selman, who was an incredible mentor to me. Then at the same time, I read something about quantum computing, which was fairly new at the time. And my first reaction when I read about it was like, this sounds like garbage. This sounds like physicists who just have no idea of the enormity of NP-complete problems or what they're up against or whatever. Was it because it didn't seem useful as well? No, no, no. It was because it seemed too useful. It was because the way that all the popular articles want to describe it is a quantum computer is just this magic machine that tries every possible answer in parallel.

2:11:13Right. Yeah, it's just this panacea, and it just sounded too good to be true. It sounded like, surely that can't scale, but then I had to learn what quantum mechanics was, to be sure, right? And amazingly, quantum mechanics turned out to be much simpler than I had thought it was once you take the physics out of it, once you see it as just linear algebra. And so, you know, and that was a new field, and that kind of, there was a lot of low-hanging fruit, and that enticed me. Did you have any rivals during your teenage or early university times that drove you or now in academia? I mean, I mean, sure.

2:11:51But I mean, I mean, the the the best kind of rivals were the kind that, you know, you you actually become really good friends with. Right. The kind that like you you aspire to to do, you know, to to to do the kind of things that they're doing or even half of what they're doing. Right. And then, you know, maybe at some point you start collaborating with them. Right. And and and and I would say, you know, I did have, you know, rivals of that kind who were who were very important in my career. right and uh um i mean i mean i mean much earlier i guess you could you could you could talk about like rivals who i who who i really didn't like but uh we can we can we can leave those aside yeah do you do you see someone like roger penrose obviously he's you know different generation yes um working on different class of problems but he feels in some ways almost like a like an intellectual rival in like some of his theories i noticed i just noticed he's someone who you clearly like deeply respect and whose work you sort of build upon but you're often sort of uh butting heads with in in theory space so first of all i'm not going to compare myself to him that would be you know penrose is penrose absolutely there is there is there is only one of him and uh uh you know i i did have the opportunity to talk to him a decade ago and you know it was uh you know, it was amazing to talk to him, to, you know, hear all his stories about, you know, learning from Dirac in the 1950s and so forth.

2:13:34I would say it did not bring us any closer to agreement about, you know, these questions of AI and microtubules and so forth, right? That we know of for now. Yeah. So, but so, so reading his books, like The Emperor's New Mind, you know, when I was 13 years old, I would say that was very influential in my development because, you know, even at that time, I was skeptical of Penrose's arguments. I thought that, you know, he is begging the question of, you know, and like I didn't really see his arguments against AI as being sound. But the questions he was raising, like, my God, right? You know, that was like at that time, that was like one of the only popular books that was talking about, you know, quantum computing, about is the Mandelbrot set computable?

2:14:29about, you know, is what are the laws of physics that are relevant to the brain, right? And so that certainly helped set the intellectual trajectory of the rest of my life. So Daniel Fong asked us to ask you... Send her my regards. ...whether there is something from complexity theory that would bound the potential scaling of neural nets in terms of like, is some issue coming up at later stages? Yeah, so it's a good question. It's one that I get a lot. And unfortunately, for those who might hope that complexity theory will stop AI from taking over the world or anything like that, I don't see any principles in complexity theory that would block that, right?

2:15:20And the key is that it is true that complexity theory puts fundamental limits on efficient computation, right? Like if P is not equal to NP, for example, then there will not be a fast algorithm to solve many of the optimization problems that we care about or to find short proofs of theorems whenever they exist or things like that. Okay, but now the key is to ask, well, can humans do those things, right? Right, and so like if your sword in the stone test you know, about an AI is that like, you can't immediately find a proof of the Riemann hypothesis. Well, that's great. Can you find a proof, immediately find a proof of the Riemann hypothesis, right?

2:16:08And so, so, you know, you know, that this, this joke about like the person saying, well, you know, I don't have to outrun the bear, I only have to outrun you, right? Right. And so, so it's much the same with, with AI and humans, right? The AI doesn't have to outrun the fundamental limits of computation. It only has to outrun humans, right? And that, you know, you could even say, look, if you really believe that our brains are governed by the laws of physics, you know, either of classical physics or of quantum physics, then for that very reason, our brains should be computational systems that are subject to the same limits, the same complexity theoretic limits that the AIs would be subject to.

2:16:56And so that's kind of the fundamental difficulty with using complexity theory to reassure yourself in any way about AI. For what it's worth, I don't think she's actually that worried necessarily, though it would bound kind of both possibilities, I suppose. And in neither case do we have any... Yeah. Now, look, I happen to be a big fan of complexity theory and I happen to be looking as a significant part of what I do for places where complexity theory can help us understand AI better. And I think one of the big places where maybe it can help is in illuminating what can we hope for with interpretability.

2:17:42Right. So like like among all the properties of of neural nets, which ones can you hope to figure out efficiently by looking at the weights? So like could you hope to figure out if this neural net has a backdoor where, you know, under some secret input, it will go berserk and start stabbing all of the humans? Right. Can you can you then efficiently find that input or could that be a cryptographically hard problem? Right. You know, even if some even if some interpretability tasks are NP hard or cryptographically hard or whatever, could we say that there are sort of more generic kinds of interpretability that are doable efficiently?

2:18:26So a former student of mine named Paul Cristiano from MIT, he left quantum computing to do this back in 2016 crazy sounding thing called AI Alignment and some new organization called OpenAI. But of course, he then became one of the world leaders in AI Alignment. And he has asked absolutely beautiful questions about complexity theory and interpretability that I think would actually tell us something. And they're crisp questions. They have a yes or no answer. We just don't know it yet. And actually, Paul and I have opposite guesses about what the answer will be. But one of us will be right. Right.

2:19:16And so so so I am excited about what complexity theory can do for interpretability, you know, maybe for other parts of AI safety also. But I wouldn't use it to reassure ourself that to reassure ourselves that, you know, AI will never become super powerful because the things complexity theory tells you it can't do are things that plausibly we can't do either. Absent us understanding how consciousness works in humans much better than we do now, is there something like what would need to be the case such that you would personally assume that an AI has consciousness? Like what would be a test that we could run or like what do you think?

2:20:00Yeah, that's a very hard question. I think that, you know, possibly even the hardest of questions. And, you know, it is totally unclear what empirical discovery could possibly let us answer that. But if I'm just going to speak about my personal intuitions, I think that my intuition is affected by ephemerality and unclonability, these things that we talked about earlier. And I think that my intuition is also maybe affected by what does it say about its own consciousness. Right. And now the hard part here is that, you know, GPT, for example, can discourse at great length about consciousness, at least if it hasn't been trained not to, you know, if it hasn't been RLHF out of it.

2:20:59But we don't read too much into that because we say, OK, well, it's seen all kinds of discussions about consciousness and its training data. So it's probably just, you know, recapitulating stuff that it's heard. And in fact, like if an AI starts talking about, well, gosh, when no one is interacting with me, I feel lonely. Like, you know, in some sense, we know that's BS, right? Because, you know, when no one's talking to it, no code is being executed, right? But Ilya Sutskaver and others have suggested an experiment that one could do, which would be that you would train a language model on training data from which you had meticulously excluded any mentions of consciousness or of sentience or first-person experience or anything like that.

2:21:56Okay. And then having done that, you would try to engage that LLM in a conversation about its experience. And if it could then intelligibly talk about its experience, then maybe that would cause an alarm to go off. Like you would say, maybe there was something here. It's emerging. Yeah, that is emerging that we need to understand better. Would you find such a test potentially a bit convincible? I can't name any one test that I would find decisive by itself. I can only name things that would affect my intuition. Well, I suppose, yeah. The point is also not decisive, but at the point when we're like, well, now I can see how it does have some consciousness.

2:22:39I mean, another thing that I could put forward, if the AI is not just solving competition problems, but it's writing research papers. It is putting me out of work. I'd say, well, you know, I'm fortunately protected by tenure. Right. But, you know, but, you know, if it could have put me out of work. Right. if it can write not just any songs, but great songs or essays that could be published in the New Yorker or whatever, then I think at that point, Turing's questions from 1950 of why are you discriminating against this thing, they really start to have teeth to them. The way I like to finish up all these recordings is to ask people a series of rapid fire predictions.

2:23:37Oh, God. Yeah. And I want this to be as system one as possible, whatever your intuition says. So first one, probability that AI reaches International Math Olympiad gold medal level by end of 2025. 80%. Probability that P does not equal MP. 97%. Probability that a quantum computer breaks RSA encryption by 2030.

2:24:10Depends on how big of an RSA key we're talking about, but, you know, 2048 bit or something, let's say 50%. And 2040? 80%. Probability that we'll have AGI, and AGI defined as AI that matches human performance, at most economically relevant tasks by 2030. 60 %? Probability that Roger Penrose's uncomputable consciousness is the right path to go down for consciousness. So you mean probability that there are any uncomputable phenomena that are relevant to consciousness? Yes. Depending what one means by uncomputable phenomena, I could go as high as 40%. And lastly, probability that COVID was a lab leak.

2:24:54So I was much higher until I read the root claim debate. So, you know, I think I was above 50 % and I am now down to maybe 15%. Oh, wow. Awesome. Thank you so much. Yeah, yeah.

2:25:20Thank you.