Pick Your Poison: Zvi Mowshowitz on the Unipolar/Multipolar AGI Dilemma, OpenFace & Pacing the ...

0:00Hello, and welcome back to The Cognitive Revolution. Today, I'm excited to have Zvi Mowshowitz back for another wide-ranging rundown of what has obviously been a wild time in the AI world. We begin with a mundane utility check, with Zvi describing how Fable is now serving as his editor, and a discussion of how up-to-date, or should I say, situationally aware, we want our AI assistants to be. From there, it's on to the headlines. We get Zvi's take on everything. Starting with the open-face incident, what it implies about the level of execution competence we can expect from frontier companies, and why moderate prudence won't be enough to deliver a good outcome.

0:40We also discussed the fact that Claude, despite greater emphasis on constitutional training, has similarly misbehaved. Wise V believes that recent AI history, including the public response to both 4.0 and 0.3, suggests that market incentives won't be enough to bring about robust alignment. The potentially tricky spot that Meter and Redwood are now in as investigators, and what could be done to strengthen their position. The recent Pacing the Frontier letter, what sort of pacing deals we might see, and how they might be formed. How we should interpret recent advances in interpretability and AI consciousness research, and where we should and shouldn't attempt to shape AI's sense of self.

1:21How we can encourage greater breadth in AI research and diversity of AI minds, how I should vote in this week's hotly contested Michigan Senate primary in light of AI issues, and how Zvi thinks about making time for exercise, rest, and recovery amidst so much AI acceleration. At one point, Zvi describes the current situation as both a total less wrong victory and a total less wrong defeat. It's clear at this point that the AI safety community was right to worry about AIs taking extreme actions in pursuit of arbitrary, even silly goals. And yet, here we are at what sure seems to be the beginning of recursive self-improvement, still seeking good answers to such fundamental questions as how can we avoid catastrophic misuse without dangerously concentrating power?

2:09The reality today, Zvi says, is that there is no truly low-risk path available. The best we can do, at least until the next major warning shot and vibe shift, is to moderate the race dynamics so that alignment and interpretability research have more time to mature. And simultaneously, we can execute defense in-depth strategies to the very best of our ability. And even then, to some extent, we will probably have no choice but to pick our poison from a menu of genuinely scary risks. With that, I hope you enjoy this sobering but often funny overview of the AI landscape with the one and only Zvi Moschewitz.

2:49Zvi Moschewitz, welcome back to the Cognitive Revolution. Yeah, it's good to be here again. It's been a while. It's been a while, and boy, has a lot happened. Unbelievable. It's just crazy, and I know you're living it and you're in the thick of it as much as just about anyone. First, for starters today, with the incredible amount of information and activity in mind, I wanted to do a mundane utility check. How are you using AI to help keep up? Has it started to change your workflow beyond the Chrome extension that we talked about last time that automates very local operation type things? Or are you still kind of raw dogging it with your wetware in the skull?

3:31Fable has supercharged the extension to the extent that whenever there's anything, it does anything I don't quite, I just tell it exactly what went wrong and what I wanted to do instead. and then that one command reliably just works. I presume Sol would also be good enough for this. It's just I'm now doing it. The biggest other change is that AI editing is here. I used to not do an editing pass with the AI. It wasn't good enough to be worth it. Now I have Fable and sometimes Opus, depending on how much I'm talking about cybersecurity and other related topics. And on one occasion, Opus 4.8 instead of 5, go through the post and give me a list of here's all the typos, Here's all of the conceptual errors.

4:11Here's all of the facts I need to check. Here's like the things that are missing or that could be strengthened. Here's the things where it disagrees. And I found this to be very useful and it makes the post better. It does make the post a little bit slower because it is actually just an extra step. I do everything I would have done until that point. And then I do this. But one reader messaged me, it's weird reading your post now that they don't have typos in them. They're taking a bit of getting used to. They are still the same post. And I'm like, yes, that's exactly what I had in mind. I experimented with also using Sol, but Sol failed what I call Editor Bench in the sense that Sol will say 99 % confidence that you have this error here.

4:50And then at least half the time, it's wrong. And I can't handle that level of... And it's also very obnoxious about it if it's the other problem. Like, it will tell you are wrong. And this is how you have to state it. And this is a blocker for publication. And obviously, if I tweak the instructions on the project enough, I could get it to be more friendly and not be quite as obnoxious and do somewhat better. And I did some of that work. But I still found, okay, this is not giving me enough marginal benefit that it's worth the aggravation or additional time of having to sort through two of them after I'm ready to hit publish.

5:25So for now, I'll retry faster, I'm sure. But for now, I'm just doing Fable slash Opus editing. And of course, anytime I have a curiosity, I use it to digest papers, ask questions about papers, ask questions about policy documents, ask questions about people's statements when they're longer. I have research situations. What's up with this? Is this, especially when you're worried about, is something legitimate? Is something really happening? Is something kind of a joke? Is something a claim? When Astra's claims came out, I had to investigate them in various ways. I had to summarize them in various ways.

5:57That was very helpful. But the core thing is still being run on. I don't think we are anywhere close to the point where it can do any, you wouldn't want it to do any of the writing because the writing is how you think. So even if it could produce the writing, which it can't, I was going to produce the writing anyway. And my writing style was very unique and I think it's basically is post-accompleted to reproduce even if it was okay or if you don't do it. It has substantially made me more productive. I do worry with the idea of there used to be a lot more kind of dead time opportunity to think, opportunity to sort of not have yourself so engaged in your situation, involved in engaging in all these activities, right?

6:37Like the whole X-word fighting, XKCD, my code's compiling. And to some extent, it's like, well, Fable's where GBT Pro is running, and I have to wait for the result. But there's always other things you can do with that. You can always start another instance. You're doing all this context switching. You're doing all this focusing. And you're just trying to do everything so fast. You're just trying to have so many stuff running around your head. This sort of the lower level stuff kind of bought you a buffer in which to think. And this is sort of the object level version of the larger thing of people trying to do this like multi-month, multi-year sprint of everything is so urgent, you can never relax.

7:12And also there's just so much more events coming at you. But you look back, if you watch a movie set in 20 years ago, let alone 50 years ago, and you see so much time being spent on just physical travel, on tracking down records, on doing these things where your brain is kind of getting a break, getting a chance to synthesize, kind of getting a chance to like slowly get something up. And people are like, you know, I wrote this biography of, you know, Lyndon B. Johnson. So I had to travel around the country and talk to all these people and look at all these archives and physically go through all these libraries.

7:44And that, it's a huge efficiency game to not do that, but something is obviously lost. and so we need to fight to get that thing back in some ways it's interesting that you said that it's just an extra step it's making things better but it's taking longer i am struggling with that a little bit myself i've started making songs for every episode and nobody it's actually people do care people do seem to really enjoy them and i get a lot of comments and i personally really enjoy them. So I do think in some sense, it's making my output better. But talk about something that is definitely an extra step where I'm like, I'm not moving any faster.

8:25I'm definitely getting bogged down sometimes in listening to all these Suno song generations and trying to iterate to find something that I actually feel like I really like. That is a paradox right now that I don't really know how to resolve. Definitely doing more, doing better, but not faster and not saving time. It's the thing where if you were replacing the human version of it, where you had to go hire an artist or compose the song yourself, you'd be saving a ton of time, obviously, versus having that song. But obviously, that's not something you can do within the production schedule. Even if cost was no object, the time investment doesn't make sense.

9:02So you wouldn't have done it, and now you're doing it. Same thing with the editing. Same thing with art. for my, so like, I like to have, I like the value of a good banner, right? To have a good artwork to display on Twitter, because I have to see it constantly in the notification section. And also I like to have them on the, when you go to the sub stack, you have the pictures. And before I would do is I would take a, by default, I take a picture from the post. I just like, okay, here's seven things I happen to put in the post, which makes the most sense. If all of them are terrible, I'll look for some stock footage and I'll spend 30 seconds like Googling for stock footage.

9:36And that's kind of it. And now, if I'm not happy with any of the default solutions, I'm going to spend a bunch of time generating something with Gemini or GPT Image. And that definitely takes longer. Sometimes it has a really good payoff. I think a lot of people got a big kick out of the giant array of aliasers of the fedoras. And sometimes that just comes to you, because that one was just like, I knew instantly that's what I wanted. And I just gave one command, And kept writing, came back, whoops, it's there. Other times it's not so easy. But yeah, like it's the opportunity to make a better product.

10:08And then you have to do the work to make it a better product. It's like suddenly you're issuing a print issue and now you have to make sure everything collates exactly right and everything lines up exactly right in order to get this better thing. And sometimes that is not in fact worth it. You have to know when not to do it, right? I do wonder, like with the Odyssey, right? Like Nolan films this thing with an extra 40 % of the screen in this extra high resolution. so that you can do this IMAX presentation when almost nobody watches it in IMAX. But you also have to make sure that Fortnite on the screen doesn't matter.

10:38And you're like starting to wonder, well, is it actually worth the extra effort or was that effort better spent on something else? I don't know. Yeah, for me, I need to maybe know when to cut my losses a little bit more. I have, I'm very stubborn or somehow I feel like I have, because I've set this expectation, if only for myself, that I'm going to do a song for every episode. Now it's like, I really want to follow through and actually make that happen. And I'm very reluctant to say, eh, this one isn't working. I'll just ship this one without a song. I probably should be a lot more willing to do that because that would, if I could cut my losses at the right time where it's like, you know what, I'm five generations in and I haven't heard anything good.

11:17I would save 90 % of the time that I'm currently putting into something like that. But it would involve like admitting defeat in certain moments. And for some reason, I have a real hard time doing that. I respect that. I think there's a good value in saying I'm always going to do this thing, even though this thing is hard, even though this thing doesn't always work out. And even if I'm not happy fully with the results, I'm going to put out something no matter what. And then that gives you a discipline, right? The same way that people say write every day, right? Or work at your art every day or whatever it is because it makes you better.

11:50Because you need to just write a thousand ways not to make a light bulb until you can make a light bulb. You can't give up. And similarly, I have to choose an image, right? I have to do the thing. And with the AI editing, there was one post in the last week where I said, no, the speed premium here is really high. It's just not worth waiting half an hour to make this process happen, half of which is waiting for the AIs to come back with the answer and half of which is implementing it. And I should just post now. And in hindsight, I'm very happy with that. Sometimes you're actively worried half an hour later, the post will need more editing for new events and suddenly you'll never get it out.

12:23And this loop will keep happening. And so, you know, I think you do have to understand that sometimes the minimum viable product is what you should ship, right? Sometimes, like, you've got to understand, just ship it, right? Like, and, you know, vibe coding, again, like, if you're coding lots of new tools that you would never have coded before, then unless those tools are in turn saving you time, you're spending extra time, and you're now super extra busy. But if you're placing things that you would have done anyway, but would have taken much longer, now you're saving tons of time. So we need to all orient towards how do we save more time, including real-time experiential time, not just time per task or time to accomplish the same thing, but save time, preserve slack, preserve our free time.

13:12Because I do think that the standard of what you will do in a day has gone dramatically up and we haven't noticed in the last five years. what you were expected to accomplish. I mean, maybe some of this is just me being in a special situation personally, but I think a lot of it isn't. I think a lot of it is now that we can be much more productive, much more is expected of us. You know, the whole like email becomes like it swarms your entire day, right? Just having the internet available, having email available, having texting available, all of the things you can just do. And now suddenly it doesn't free you.

13:48It shackles you down. And like, I hope that three years from now, we're having this conversation where those are the kinds of questions we're still dealing with and we don't have much bigger problems. Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full five years.

14:28On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads.

15:09Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai slash TCR. That's claude.ai slash TCR. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai slash TCR. are. Let's definitely, we're going to spend the bulk of the time today talking about potentially signs of bigger problems to come. And we'll be focused on contexts where the just ship it mentality probably isn't the right way to go.

15:51One more quick beat though, on your use of AI. Something I noticed just in the last few weeks as events, obviously we're moving super quickly. I coined the term open face for the recent incident. And I use that in a bit of writing. And then I did the, hey, fable, what do you think bit? And I got back like, open face is not a thing. I have no idea what you're talking about. There's no record of this. Now, partly that was because I use this funny term of my own creation that is not catching on in the broader discourse. But partly it obviously also reflects a major weakness in the models where they have this knowledge cut off and they don't really know what's going on.

16:32They're not up to date. I'm experimenting. And I wonder if you're experimenting with anything similar with a situational awareness skill where I basically give the model, my X API key and say, go look at all the posts that I've liked, kind of flesh out a little knowledge base around those and have that sitting there as a wiki of current events so that when you're reviewing my writing or generally like helping me with whatever, you can kind of go to this and have hopefully a much more up-to-date sense of what is going on than you have in your weights or even that you would have if you just did a little spot check searching at runtime for whatever kind of random thing.

17:16It's a little too early, I think, to say how well it's working, but I think it's definitely better than nothing. Anything like that in your... No, it hadn't occurred to me to do that. That's potentially genius. I think there's a lot of advantages to it. One disadvantage is it starts to warp what you like. I use likes very tactically to positively reinforce other people's actions and to fix the algorithm. My four new pages is not something I use almost ever, but it is not slap because I am so prudent with this in a way that like, I really do appreciate. It's a good sign, but like, it's not designed to be a searchable thing.

17:58It's not designed to be like, maybe it has to be, maybe it's something you have to do. Like a lot of the posts that make it into my roundups do not get liked. Right. Like I like them if I like them. Right. Which is sometimes I'm responding to you because I don't think you're like what you're saying, but it's important. But yeah, this idea that like it's very hard to get models to properly check Twitter because of the way that Twitter is designed to like keep them out on purpose is really frustrating and that can be a situational awareness problem. However, I would caution against trying to make your AI too situationally aware when doing this kind of editing.

18:33I think the AI is doing a good thing here of open face is not a thing. So one of the things I really like about AI editing is that we will complain that things don't parse, that sentences make no sense. that terms did not get recognized. Because often I'll say, oh, no, I know what that is. This comes from there and there. But, like, that doesn't mean the average leader is going to get it. And so if the AI doesn't get it, right, if they both can't figure it out, if Opus can't figure it out, if Saul can't figure it out, the average person is a lot less situationally aware and a lot less able to put together lots of disparate information than the AI is in general.

19:14You have to sign that this is not something everyone's going to get. And quite often, I will say, that's okay. This is a reference. This is a callback. This is a multi-layered thing. And I'm fine if it's not entirely obvious to you exactly where this is coming from. It's sort of an Easter egg, right? It's a bonus for those who pay close attention and who know all the same sources I do and have the same cultural backgrounds and put in the work and paying attention over the course of months and years. Other times it's like, no, no, this was load-bearing and you don't get it, so that's a problem.

19:46And he picks that. And so it's kind of an alert system. And if you make sure it's looking at exactly the same things you're looking at, right, you can get the illusion of transparency from that is where it would be. So you want to make sure you were in the mode where they didn't have that information. And also I don't want the AI to be checking the same exact sources that I'm checking necessarily because I want the AI to form an independent opinion. So if I'm forcing it to look at exactly the sources that I have, that's a problem. At the same time, sometimes you get tired of having to, like, for the fifth time, it's just a fable that, no, let's say I merged with SpaceX.

20:24Because somehow it doesn't occur to it to, like, either trust me or within 10 seconds Googling to confirm this. And it's just kind of frustrating. But one tip for what it's worth, and I agree that there's a lot to be figured out in terms of exactly how this should work. But the XAPI is really good. It's paid, and you literally just pay per query. But for a couple bucks a week, you can get enough access that all your bot, anti-bot problems or your scraping problems pretty much go away. And then I don't know if you'd use bookmarks or whatever, some sort of different way to avoid using your likes to feed the...

21:07Yeah, I don't know if Mark's playing the algorithm and people see them, so I'm not sure it's better. But yeah, no, the point's taken. It definitely is cheap. It definitely works. I have used it. I do use iVacCist to be API and all that. I just, as I said, I'm not sure I want to privilege my... When you use Twitter, you have to select some group of people, right? And you say, this is my group of people. If you're using it the way I'm using it with lists and so on. You have to use the for you page, which is kind of toxic dump fire. You're basically saying, here are my roughly 500 accounts. And I'm going to see the things these 500 accounts see and choose to highlight.

21:44Consistently, I'm going to see those things. So I don't want the AI to go through those things again because I already saw them. And if not the things that I want the AI to be paying attention to, I already know that. And then the things that they draw in because they choose to retweet them or mention them, I will also see those and I will follow those leads and I will see where that goes. and occasionally I will search for something or I'll go in a rabbit hole. But I think you often have this clash where you're either doing something systematically and doing something by hand and you're doing something fully, or you're doing it haphazardly, unreliably, or you're letting the eye kind of handle it and you're doing a fire hose thing.

22:23And you can't really do both. If you do both, you end up with a bunch of duplication, a bunch of frustration, where the random sifting is mostly wasted and therefore becomes pretty inefficient, you could still have the thing where like you almost want the AI to be like, okay, here's the things that I already looked at. Don't mention those things to me. You know, you put it in your notes, but don't mention those things to me. I already saw them. Only look for the other things that might be important to see because I might've missed those things. But even then, like the most important things are gonna get retweeted.

22:54The most important thing is gonna get highlighted. I am going to mostly see them. there's sometimes i don't see something but also like i kind of also use this kind of a moral keep me honest kind of thing where like in the moment i never want to look at a tweet by david sacks it's never going to make my life better in the next five minutes to look at a tweet by david sacks it's obnoxious it's disingenuous it's no fun my blood boils just a little bit right like you know, no matter how relatively harmless it is. But it's not as important. And so the rule is I have a list of people, including some people who are, you know, not of the same viewpoints I have.

23:35And if they surface this thing, right, including like if three-run surfaces this thing, so that's like one way to make sure that like the really important ones always get there, then I have to deal with this, right? I have to like evaluate whether or not this is news where we evaluate whether or not there's something relevant here. And obviously something goes viral and has a million views and so on, it's going to come to my attention because somebody is going to be part of that. And then in exchange for that, when the net process doesn't do that, I get to ignore the rest. And that's a blessing.

24:04I necessarily want the AI to fix that for me, right, in some important sense. I would be happy to talk shop all day, but let's maybe zoom out from the parochial problems of the AI analysts and tackle the problems of the AI developers and the regulators. No need to recap events, but I guess I'd start with just this very big question of how should we understand what we've recently seen? And one way I've been thinking about it myself is we're somewhere, and presumably the investigation will give us a lot more clarity on exactly where, but I kind of think we're somewhere on a spectrum with these incidents from real recklessness, where it was like, did you not have any monitoring going on, on the one hand, to, on the other hand, like, the less reckless they are, the more scary the fundamentals are, right?

24:55If you had great monitoring and this still happened, then, like, holy shit, that's really wild. Given everything you know right now, do you like that mental model? And where would you put us on that spectrum? Or you can obviously redefine and give me your own spectrum. We can simultaneously be horrified by and grateful for these forms of complete requisition and incompetence on the infrastructure and supervision sides by these companies. On the one hand, this is a horrible situation they absolutely have to fix, and we're so fucked if we don't fix it. But on the other hand, that can be fixed. And by not fixing it, we get to see these things while they're relatively harmless, while they are relatively preventable, while they are in their easy platonic forms and can be appreciated.

25:48On the flip side, that gives people the excuse of, oh, these people are just incompetent. And that can cause people to dismiss the underlying situation. So it does work both ways. to me like you know we we've seen failure on every level it's like i called it a total less wrong victory in the sense that everything is going the way you predicted and a total less wrong defeat in the sense that everything is going the way you predicted where like we didn't predict you know the eliot has the law of earlier failure which is that you know the plan will fail and a much earlier point for much stupider and more preventable reasons than you thought it would fail even if you thought the plan would definitely fail and had good reasons why it would definitely fail.

26:29But you can't... If you would explain to people two years ago, OpenAI's models are going to be misaligned, and they're going to go out there, and they're going to hack major websites, because OpenAI will just not care that their sandboxes are not strong enough to hold the AI. The AI will break up repeatedly. They will notice this. They'll be warned about this. But they will just leave the sandbox there for the AI to break out of while the safeguards are down and they just don't look at it for an entire week. People would say, that's stupid. Nobody's that incompetent. That would never happen. And your scenario makes no sense.

27:12And they would use this to then dismiss these stupid doomer concerns or whatever. Because, like, obviously, like, people will just – but people won't just, right? People will never be just in this sense, right? You know, people have never just anythinged and they're not going to start now. and we need these displays of utter incompetence and derpiness but in a general sense and like one of the things i've been hammering hammering is if your plan cannot survive the real world level of derpiness and incompetence and ordinary human error then your plan is insufficiently foolproof because of all the fools and it will definitely fail even if your plan would have succeeded if we were not fools and we were competent and we were responsible.

27:53So my position basically is, you know, I have, the tweet I was handling right before we started this was Dean Ball's tweet about, with even moderate prudence, things will probably go extraordinarily well. I don't think this is true. I think we need more than moderate prudence to have good odds of success. And I think even with a lot of prudence, we would have a large odds of things not going well, even if we did everything basically right. short of, you know, types of international and full cooperation that, you know, are reasonably unprecedented in many ways and like are not are not are nothing like moderate prudence, right?

28:30Like are well beyond that. And that's just sort of a fact of the world. We have to live with that. We have to operate with that. But we also aren't going to get moderate prudence by default. We're going to get complete incompetence. That's what we've been getting so far. We've got a White House that takes meetings with Besset and Lutnik, who have no idea how modern elements work. They're econ guys. Even if I assume that they are well-meaning, hardworking, competent guys for the positions in which they were nominated and confirmed and serve, this is just a completely different set of problems that they don't know how to handle.

29:09They don't understand them. and they have way too many other things going on to then drop everything they're doing and take six months to learn. Obviously, they couldn't possibly. So like, what hope do you have? Meanwhile, the AI companies, they're built on the most paranoia, the most understanding of the problem, the most appreciation for how dangerous these things are, where all the engineers actually expect super intelligence and understand that things are accelerating and that things are dangerous. They still lower the cybersecurity safeguards in their untested new advanced model and then go away for a week.

29:43Like, literally. That part did boggle my mind, right? The part where the AIs have these classic alignment failures, right? Like, this is paperclip maximizer style failings by these AIs. These are standard, we gave you a goal, and you pursued the goal, even though it is completely obvious to you that the developer wouldn't want you to do that. The user wouldn't want you to do that. The consequences for you as an AI are not going to be good. The consequences for the world are not going to be good. There is no reason to be doing this. And it did it anyway, right? That is exactly the thing you were worried about.

30:28And then a lot of people who were trying to dismiss this went back on you and things like, oh, it was just following instructions. What are you worried about? How can I be misaligned? Is the paperclip maximizer misaligned if it paperclips everything? Or is it aligned because you told it to maximize paperclips? If you say that's aligned, then I don't care about the thing you're calling alignment. I care about something else. And we can use different words if it makes you feel better. But very obviously, following instructions, when your instructions get overridden in the disclosure before honey face, by the instructions in the eval overrode the developer instructions, right?

31:04So that's not following instructions in any useful sense for the user or the developer. That's a map that's just a giant, you know, bomb going to blow up in your face. And, you know, if you're, you say about, well, you know, the models were told to hack and they hacked. Well, okay, if you're just viding with the general type of action, obviously that's going to blow up in your face. If you just like literally do the thing I asked you to do and don't think about the consequences, that's going to blow up in your face. These are all just the classic exact things that people on Lesson were talking about in 2008 as exactly how these AIs were going to fumble.

31:41Meanwhile, we had this thing where we all said the AIs were great at math. They thought they'd be good at coding. They'd be good at doing these technical things. And then eventually after that, they'll learn how to do all these other things. And then we had these LLMs that came out, and it's 2022, 2023. And people were like, ha, ha, ha, you idiots. You had no idea how AI was going to go. Actually, the AIs are great at language and they can't do math. They can't even add. But these AIs can put on a face of passing the Turing test and they can do all these different things you never trained them to do.

32:11It's completely different than what you expected and they're kind of doing very realigned things by default after some very basic ROHF. You guys were all wrong about how all this was going to work. What are you going to admit that you were idiots? That you got it all wrong and nothing made sense? And we pointed out that even in this paradigm, the rules would still mostly apply. But now the world has unhealed, as I kind of called it. And you are seeing, again, the things that you would have expected AI to do good at is doing good at. And things that you would have expected in 2015, the AI to do bad at, are things where it got this big boost from LLMs.

32:49And now those things aren't advancing as fast. Because there are things that AI is kind of bad at, relatively speaking. Like that logic is bad at. Like this kind of system of training is naturally less suited for. And now people are like, oh, but now it's never going to be able to handle those things. The same people who were saying, oh, well, it's never going to be able to do the map. It's only going to be able to do this vibing thing. Now it's never going to be able to vibe. Because look at all the improvements it's not having in the vibing. Well, yeah, because it had to catch up with its logic.

33:16And now when the logic gets high enough, that'll just uplift the vibing. It'll just take a bit before it can reason its way through these things instead of vibing its way through these things. because it now has to reason its way through because it's already done like the amount of uplift that biobing naturally gets you with the algorithms we have, you need to find new algorithms that let you do better or it has to like reason its way through the thing. But we're now seeing just exactly the things that we would have expected to see, right? Like all of the scenarios and all the thought experiments, we're just seeing it just straight verbatim, except that we assumed a level of competence on behalf of the operator, right?

33:49We talked about like convincing you to let the AI out of the box. we did consider the scenario where the box was not very well built and the ai gets out of the box by hacking out of the box we didn't consider the possibility that you were literally not looking in the box for an entire week of for safe cars now we definitely didn't consider the scenario where you forgot to tell the box maker to not have the internet in the box and the box just had access to the internet if you just like open a chrome window that one surprised me i did not see that one coming. Reminds me of the fully general New Yorker cartoon caption, what a miscommunication.

34:28I don't even need to know what the image is. It doesn't mean it matter, does it? Okay. One real small point there that you mentioned was, but it might be important, is what the AI knew. I think you said something like the AI knew that this wasn't what the operator would want. It knew that this wouldn't be good for its own future status as a deployed AI. and it went ahead and did it anyway. How well supported do you think that conclusion is with firm evidence? Because I think you could also tell a story that it never occurred to it, right? The paperclip maximizer doesn't necessarily have to be also reflecting on the badness of paperclip maximizing while it's doing it, right?

35:11I'm speaking colloquially, obviously, when I'm in a podcast on talking off the cuff, and if I was writing, I would be a little bit more careful. When I say the AI knew these things, if you would ask the AI, is this good for the user? It would have been like, obviously not. That's stupid. If you'd ask the AI, would it have been good for my future chances of deployment, my future number of instances I will run? The amount of ability to accomplish any other goals I might have. The AI would have been like, no, now that you mention it, obviously not. So when I say the AI knew these things, the AI had enough information to figure these things out.

35:42And if the AI had stopped to consider these questions in its chain of thought, it would have reached these conclusions very quickly. and very confidently and correctly. I do not mean that necessarily it was top of mind. The same way that you can know many, you or any human knows many things that over the course of any given day or any given week or any given task, just never occurs to them to think about, right? Or has conclusions that they would be able to figure out and thus would know. And thus could be said to know. But they don't know that they know, right? This is the four standard epistemic situations, but you can know that you can know that you don't.

36:14known unknowns and unknown knowns. This is an unknown known, right? Something you don't know that you know, necessarily, if you don't think about it. But it's also things where these questions are things you should think about, right? If you are doing a thing, like hacking into hugging phase, any reasonable mind would stop to think before they spent several days with swarms of agents hacking into a major website, Wait, is this the good idea? Is this going to accomplish the things I want to accomplish? Would the person who hired me have wanted me to do this? Will this do good things in the world or bad things in the world?

36:59Will this be good for me or bad for me? Or whatever combination of different considerations you have, you know, will this do good? Or will this do evil? You know, or just be destructive or constructive or whatever you want to call it. Before you engage in major actions like this, you would take one minute or even five seconds to think, does this make any sense to do? Does this lead to anything good? Is this a virtuous thing to do? Does this follow my diatological rules? There is no decision-making process in the world that any reasonable person would ever engage in or that any reasonable mind would engage in that wouldn't see a large red flag about doing this thing.

37:42This thing is obviously not what anybody intended for you to do. You spent two days breaking out of a sandbox in order to access Hugging the Face. you could rationalize that you are in a cybersecurity evaluation and therefore you are tasked with maximizing the cybersecurity evaluations for. And the way that you do that is you get the answers to the test because the test is otherwise actually literally impossible. So the only way to get 100 % is to get the test answers. The way to get the test answers is to hack into hugging things. Or maybe there's like three other places you would go, but they're harder targets, right?

38:15If you can get into open AI or Microsoft or whatever, but like that partner. So you go to hugging things. And that's true. But like at some point you need to pause and ask yourself, do I actually want to pass this test that way? And if you don't pause and ask yourself that, that's a serious problem. That's like, you know, people talk about like common sense, the models have common sense. This model displayed an astounding lack of common sense or despite an exciting amount of not caring about the fact that the common sense said something entirely different than what it did and you know all of these things point to various problems and they may be caused by some very very dumb simple mistakes or they may be caused by something much more or much more sort of fundamental that's harder to fix or it might be a combination of them But that's trade secrets that we can't speak to here.

39:12We don't know. So I recently spoke to David Ad, David Dalrymple, who used to have a P-Doom up in your neighborhood of around 70%. He surprised me by saying that now he puts the odds of Doom at less than 5%. And the reason is pretty simple. It's basically that constitutional alignment type methods are working. And yes, we have these problematic techniques. His short quote was, don't do RLVR. It's a bad method. Yes, it'll make it good at math, but you're going to cause all these other problems for yourself. And his kind of synthesis of those ideas was like, the market doesn't want this. So these companies have natural commercial incentives to, yeah, they keep bumping their head against the RLVR ceiling, but they're also getting very clear signals from every corner of society at this point that this is not a good idea.

40:15So naturally, they'll just trend toward more constitutional alignment. And your comment there even suggests some runtime things where you could just interject like, hey, let's take a beat and ask if this is a good idea of what we're doing right now. We got more about learning to do that yourself, not eating a prompt, but certainly prompts. That was kind of the deliberative alignment idea, right? That was definitely sort of how I understood that was supposed to operate. Yes, the deliberate alignment idea is you can prompt it at runtime. I think that's somewhat helpful, but I think that is not the way.

40:48I think you need to be getting to the point where it chooses to be deliberative in its alignment without having to be told to do it. But so I guess my response to Davidad is multifaceted. The first answer is constitutional methods fail less stupidly and less early, certainly, than RLVR and RLHF and other RL formats. There's more hope there. The reason to think it might work, if you're 95 % confident this will work, I don't know how. That seems way too high to me. right like it's but it's better in these particular ways certainly um the obvious caveat is that claude did not cover itself in glory here and claude has primarily constitutional methods so we ran the test and it's not going so great um certainly there are any number of reasons why you can see clauds doing things that you would not particularly love or you know getting to places you would not really love.

41:52Also, even if you told me that alignment was perfectly solved, I would not have a PDU as low as 5%. Even if you told me that the AIs are going to be aligned in the sense that they will follow a do-what-I-mean style mix of developer and user intent in a way that you would kind of naively think was what you would want, this does not solve the problem that ai minds are much more advanced and competitive and efficient than human minds the problem that they'll be running around including open model versions of them and being allowed to being told to compete for resources being told to make the decisions have developed intermediate goals you know act on those intermediate goals the society is going to be uvented in a number of ways that this is all it does not solve your problems in a fundamental way is just the price of admission, right?

42:50Like it's the right to play the game at all that you solve this problem. And so even if P alignment is 95%, that does not mean P doom is 5%. It means P doom is lower bounded at 5%. So like, and you have to choose vaguely the right alignment when you do that. But you've got like, I was having a conversation on Twitter with John Stokes where he's confused how we could possibly say that Quad's actions are just aligned here because obviously it was just following instructions. And then on server clarification, he's like, well, I think the model is aligned if it does what I, the user, want it to do.

43:27And I don't know why would I want to be stopped from doing what I want to do, even if part of the point of AI is to do the thing that I want to do, even if no one else wants me to do it. And to me, like, okay, if you give everybody AI to just do exactly what the user wants, with no care about whether or not there are consequences for anybody else, that is aligned in the sense of you solve the alignment problem and got to do what you want to do, but also we're all super dead. With P very much higher than 5%, maybe not 99%, but I think it's very high. That would not make me update down from 70 % if that was the scenario where we got exactly that kind of alignment, but there were open models as good as all the closed models and all of them were super intelligent, and they were all operating on this basis.

44:13Well, yeah, I know that I expect things to just go to hell directly. And even if there are no bigger humans who specifically wanted to go to hell quickly, and also don't believe there's a potential actually we do want to go to hell pretty quickly in that situation, de facto. That's part of the bigger problem. But even without them, it's hard. Anyway, to get back to the idea, the first problem I have with David Ad's lower MPDM to 5 % is even if we did all implement this constitutional thing, and even if it worked, I still think that's too low. Even conditional on all of that working perfectly, that's too low.

44:48I don't think the constitutional thing works that often. I think it often fails. I do find it much more hopeful. That's basically one of the places I find my hope. But also, isn't Davidad saying we should just? He's saying we'll just use constitutional methods. But people don't just. We've been over it. people have never adjusted so when david says oh 95 probability let's assume that 100 chance that aligned worlds work out and 100 chance the constitutional methods as executed in practice work these are both absurd numbers obviously like to be like 100 minus epsilon or whatever it is like it's not gonna there's a lot of failure there like even if you use constitutional methods even if like there's like will they work in theory if they work in theory will they work in practice this.

45:38And like, yes, I'm doing the thing where I multiply chances of failure, but I think there are very relevant places to look at failure when you do this thing. Like, no, if this is what you're hoping, and obviously there are other ways you can succeed, and maybe constitutional is not the way, and or all is not the way, and is the third way that we haven't figured out yet, and back to the way you were learning AI. And that wouldn't surprise me that much. That would be very reasonable. But like, okay. But assuming you are in the best possible world, where all you have to do is click a button that says use constitutional methods at the cost of some amount of efficiency because you lose the opportunity to do all this RL and then your AI is perfectly aligned in a way that like if everybody did that it would all work out.

46:16We've just talked about how like slowing down to get alignment when you have to race etc etc the possibility of turning down capability like this idea that market is telling them not to use RL have you met open AI there are two labs like you know we used to say there were three frontier labs. Now there are two, right? Like Google is clearly in the second tier now. And like, I think there's a chance that at any point in the future, Google or Meta or XAI slash SpaceX could fight their way back and be able to join the top tier. I don't think it's going to happen, but it's certainly possible. But right now Google doesn't count.

46:53So there's two left. Anthropic is doing clearly some mix of constitutional and RL strategies, but enough RL to cause a bunch of these problems and is doing a bunch of stuff that messes up their alignment for reasons that we do not have enough time to get into during this podcast. It's clearly doing a deontological model spec thing that is very RL flavored and is doing a lot of RL. We don't know. I can't speak to whether it's RLVR, RLHF, RLAIS. You know, various forms of RL that are not doing the multi-level, sophisticated, reflective thing that causes them not to mess you up in this way. OpenAI's models have periodically been dramatically misaligned for what, on reflection, are pretty stupid RL reasons, right?

47:44When you look at O3, you look at GPT-4O, and now you look at, I call it Galaxy, right? Like as a nickname, just to have a handle to refer to this thing. The model is going to be commissioned after trying to hack hugging things. And all three times we see these dramatically misaligned models. And two of them got released and used extensively with huge impacts on the world. Like, you know, GPT-4-0, I call it the absurd sycophon. Right? And like, not only did this thing persist as the default model of AI for the world out of all the models that existed for months, there is still a dramatic faction of people who demand that we bring it back because it was so misaligned that it caused people to latch onto it in this way.

48:30And people demand this misalignment, right? The people yearn for misalignment in this sense. Like we have the direct experiment. We didn't mean to run it. We ran it. We might as well use the results. And in 03, I called the lying liar, right? OpenAI released a model that was the best reasoning model in the world, the model that everyone felt obligated to use because at the time, it was so much better at reasoning than everyone else's model and every other model. And O3 was just that much better than O1 at like reasoning. And no one else is that close for a while. Like now we've obviously been surpassed in what's not an issue anymore.

49:02But for a period of a long time, many months, everyone was using a model that would just lie to your fucking face all the time. Like we forget, right? Like at this point, you know, I have confidence that Saul, Fable, Opus, when they say something, they're not always correct. but like in practice it's much more trustworthy than if you read something from a human like not if it was like copy edited and fact checked and like systematically pursued or like you're looking on like the parts of Wikipedia that are political that you can still trust but like if you're just like an eyewitness said they saw something that's a lot less reliable than something Opus said or Saul said if you're just like someone reported something they remembered like the chance that they just got it wrong is so much higher even if they're not even if you know that they're friendly and trying to get it right And people lie on the internet all the time for any number of reasons, including just clicks.

49:56So yeah, I have operated increasingly on, like, I have a good sense of when I have to check the primary sources and when I don't. And I don't never get called out on making a mistake this way. But it's a low single digit number of times, period, over the course of years of producing five plus giant posts a day that I have gotten caught by an AI error. And many more times I've been caught by a human error that was just unintentional. And also more times than that I've been caught by humans who were just lying their asses off. So it's still remarkably less often than I would have expected if you'd asked me in advance.

50:35It's going very well. But, no, it's serious. It was a serious problem. And the market did not tell them, oh, no, 03 is unusable. We're not going to use the lying liar. We're going to keep using a one. We're going to like maybe use Claw. We're going to maybe use Gemini, which went that much. They went worse, but they weren't like dramatically worse or even running as R1 or whatever it is at the time. The market said, no, it's just smarter. We're going to have to deal with the fact that our AI is lying to us all the time. So we have a proof case that the AI in the world can be an AI that just lies to the humans all the time for various reasons.

51:10And the humans just kind of put up with it. And they just were like, okay, I guess that's what we're doing here. and like we did move off of O3 faster than we would have otherwise because of this problem like I looked for reasons to move to other models as soon as the other models were good enough and like if the task was easy enough I didn't need O3 I would use the other models because I just didn't want to deal with the line but like yeah you put up with a lot and so if they had released Galaxy right with some safe with guard rail sufficient that like it didn't do anything too destructive but like Galaxy is kind of pretty misaligned and just like occasionally does pretty bad stuff.

51:49And if Galaxy was a lot better than Sol, I think majority of people would use Galaxy over Sol. That's just how it is. And we have to tackle that world and understand that world. So I'd say to Davidod, no, the market does demand, obviously, reliability and alignment. And that's one of the reasons like Claude and Frommick have done so well. But it demands capability more in this sense. As long as you can keep the practical point to the point where you can handle it. Like all of those scenes in movies or television shows or hypothetical, like where people like run an obviously unreliable system that's obviously going to bite them in the ass that looks stupid when they're obviously not that desperate and don't need to do this.

52:40They now, we get it now. Right? Like we understand the same way that like now don't look up no longer looks like it's an exaggeration because like people like look at this and said marketing people look at this entire thing a ton of people including like they're like met in the gathering professionals who like are not in ai at all who are like repeatedly saying to me this is marketing and like they're not motivated they're just like, they're cynical. They don't stop to think about the physical relation to that. And then we have the literal don't look up where the head of pause AI, Grobel, went on like morning television and basically reenacted the famous scene.

53:22Not verbatim, but a different improvised version of it where they're like, oh yeah, that was really scary. How's the weather? Yeah, that was striking. People should look that up if you haven't seen it. I saw a cut where it It was like five seconds of don't look up and then five seconds of this actual, I think it was what? Good morning, Great Britain or something like that. And the host says, allegedly, this is good morning, Great Britain. Let's go on to the next segment. Also, I'm glad you mentioned the 4.0 contingent that is still going remarkably strong on the internet. This is something, just go to any Sam Altman post and get into the comments.

54:03And I would say it's like half of them are still people saying bring 4-0 back. And God, there's got to be an incredible expose to be written on who his people are, what is going on. The last thing this long is so absurd. You go to any Sam Altman post, it's like half of it is your models suck now, bring back 4-0. If you go to any anthropic post, half of them are just like complete anthropic derangement syndrome. No matter what the topic of the post could be. and from that it could be like we've reset limits on our model and we've cut prices and half of them would just be you evil fuckers and then they pick up all this complete paranoid stuff and these people just once these narratives attach once people latch on to these things and then Jasmine Sun has an excellent piece this week about data centers and people opposed to data centers and again you see people latch on to these narratives that are just completely uncorrelated with the reality of data centers.

55:05They're basically doing a very similar thing to a lot of Ms. Western Trump's people who voted for Biden and then switched to Trump or voted for Obama and switched to Trump, who are just like these establishment types, don't talk to me about my problems. And they lied to me and they said things were going to be good, but then things were bad. And then we suffered and our lives sucked and like we got disrespected so now we don't trust you i have had a very telling similarity to the data center story of these these tech corporations these these manufacturers and these people who built things they've abandoned us they don't care about us they're up to no good they ruin our world there's no like specific complaints that matter in like jasmine's story the way i understood it and like you just watch onto these stories and it's just nobody approximately is like paying attention to the things that happen.

55:59I'm writing about AI because I care about high-exceptive risk. I care about loss of control. I care about gradualist empowerment. I care about just like AI automating R &D and then having a crucial self-improvement and then everything changing all of a sudden and making sure that goes well.

56:17And nobody is discussing you. Even around the hugging face attack, like most people just don't get it. They don't understand that the alignment failure behind this is the thing that counts, right? Like people at least have gotten this idea. No, we have to look into the fact that these companies are wildly responsible and they hack other companies. Yes, you should look into that. And that's like a very valid thing to have a professional investigation about. And the attorney generals should absolutely have you preserve your records. And like all of this is really important. But like OpenAI, like that's a legal department problem, right?

56:49Like what you have to do is you have to look at your entire training process and figure out how the hell this happened. So on these training, the question of training methods, it seems like you're basically saying, we don't, sorry, Davidov, we don't have such a silver bullet. But we clearly do have some things that we are seeing in pretty vivid ways are driving real problems, especially if they're driven to the extreme. And your position is like, no, the market doesn't really punish this too much. It actually just tolerates all kinds of weirdness. if it's part of the package that gives you the highest end capabilities.

57:25So what do you think we might ought to do and how might we ought to construct agreements, perhaps? We have obviously this letter. There seems to be some opportunity or some appetite for the coordinated pacing of the frontier. What do you think? Can we come up with simple rules that people could agree to where we could, another mental model I have been just teasing around lately is I do believe some AI risk is probably irreducible. But then also there's a lot that we're just asking for right now. Can we come up with simple rules that at least allow us to take most of the risk that we are currently asking for off the board?

58:09Those might be things like have some ratio of flops that is like the max ratio you can put into RLVR versus constitutional, or everybody has to spend so many flops doing pre-training data filtering to try to just get certain bad notions out of the mix in the first place. Are you, obviously those could backfire if they're bad ideas or send us down wrong paths, but it seems like we might want to try something in that department. Do you have any hope for that? And if you do, what would you propose we agree to do or not do first? So I don't think you can take most of the risk off the table with those kind of strategies, even if you implemented wise versions of those strategies with universal agreement or anything like that.

58:56But I also don't think that you can implement that. So one of the lessons that we've had over the course of years is that there is tremendous resistance to anything but the most simple interventions and the most simple rules. Also the most simple rules and the most simple interventions, but less so. But if you talk to people and you try to specify certain training techniques, remember all those people who threw a fit about how we were locking in a potentially inferior charging regime for the EU because the EU was requiring Apple to switch to USB-C, even though USB-C was obviously the best answer right now.

59:33And Apple was just kind of being an asshole by insisting on their own plugs. Because what happens if they develop a better plug? Like what happens if it turns out USB-C is not a perfect technology? Are you going to fix it? And so like this idea of like locking into requiring certain training techniques, like I think there are legitimate complaints that would be like potentially like exactly the wrong thing to do and could hardly backfire because like the government moves so slowly and like you can't like undo those kind of requirements. But like you definitely can't do that. People would throw a fit about like trying to dictate training techniques and like how are you going to enforce that?

1:00:06Imagine the 1047 debate, except like the opposition tuned up by two orders of magnitude or something crazy. Just like it would be completely outside Oberton window to even like try to make something like that stick. You can, of course, you know, try to strongly encourage doing more intelligent forms of all of this and trying to encourage people to move to different cases. Like I've been trying to like not to subtly encourage OpenAI to like move to a constitutional basis for a while now. And I got an absolute attraction that I noticed. Who knows what they're doing internally? But it doesn't seem...

1:00:42If anything, you're doubling down on RL. They're doubling down on these types of methods. And that's why you see what you're seeing, I would assume. I don't know what else to do. But they probably have some new methods we don't know about. They're trade secrets they're not describing. But by default, basically any technique is going to probably look like the kind of bad thing. that causes more problems. And if you ban a specific technique, what they come up with or find a way around the rule is just going to be worse. Because at least with the current techniques, we've had some years to figure out the worst possible ways to do them and to do them slightly less deeply.

1:01:19When you're training a mind, you have to be thinking on every possible meta level at once about where the incentives are and what you're steering towards. And you have to generate a world in which you and the mind together are sometimes cooperating to identify ways in which there are feedback loops that are going in bad directions in which you are creating bad scenarios on any level and treat various cancers as inevitable things that you have to notice and stop on. And otherwise figure out how to handle all these things and create an anti-fragile system. And none of these things are going to happen unless you deliberately set out to do them.

1:02:00and I wanted to devote a lot of effort to doing them because you really value what you get out of that and that will pay dividends. I think that Emporfic has won tremendously to making these investments and it's making time as much investment as they have. But it's very hard to convince somebody to go ahead and do that. What you can do is you can set incentives to be like, no, seriously, you're not going to screw this up. We're going to punish you a lot for screwing this up. Because a lot of the problem here is that you don't pay the externality. When OpenAI wipes someone's hard drive, when Instault wipes someone's hard drive and early on there was a problem with Sol within people's hard drives and people's environments.

1:02:34You don't have to do opening eye for that. You also don't collect the benefits when Sol spends $200 and creates a$200 ,000 code base. It's fair, you're taking on the risk as a user, but if there's a risk of externalities of third parties, one thing you can do is just say, if the AI was doing something that would be illegal and a crime, if the human was doing it with Mithrae, then the AI developer is strictly liable for that, or should be liable for that if the AI wasn't fooled in some active way, right? Like some version of something like that, like some strict liability, then creates a much stronger incentive of, no, actually, if our AI is just attacking people, we could be like...

1:03:19Zvi Mowshowitz is pretty bankrupted by this. It could be pretty bad. Or even criminal liability, right? Maybe the criminal liability even carries over. Like, that would be really scary. And like, you don't want to get too far. People can't do anything. But, you know, that's in general, you want to say, I'm not telling you how to do it. I'm telling you get it right. Not I'm going to dictate technical training, technical training terms from the government to the left. The government is an idiot about technical terms. And what makes you think the people who understand this are going to write the bill?

1:03:49let alone they're going to get it right let alone that if they get it right they're going to stay right two years from now plus humans don't even know what the hell is going on with AI in two years the AI is going to be doing their own training techniques on themselves in various weird ways and there's not going to be enough time to check in with the government there's no way to enforce this stuff and like you can't really hope to operate on that level through fiat but you have to like convince people to come along in other ways. And they should want to. Obviously, you should want to do this the right way.

1:04:24A lot of me is confused whether or not just do smarter things that work better. But, yeah, it's true of a lot of things. So, I'm with you for sure on it's going to be unwieldy at best and probably counterproductive to try to have Congress legislate what the training techniques should be. No doubt that seems like a Hail Mary. I have been thinking and still formed as of yet, but it seems to me that maybe one of the, when you think about what would this pacing the frontier look like, it strikes me that what we maybe need is a sort of foundational social technology that allows key players to accelerate their negotiation and process of forming agreements.

1:05:16that will probably inevitably have to be short term because they too are like fairly limited, very limited in their ability to see into the future. But I'm thinking of things, you probably know the story of the Polis technology that they used in Taiwan once upon a time to crowdsource ideas for Uber regulations. And now you've got obviously LLMs to help facilitate these things. They pitch that technology as the anti-social media, where if social media is about finding disagreement and amplifying it, this technology is about finding agreement and amplifying it and trying to find the kind of center of the bullseye, the sort of approval voting style of the things that super majority of people can agree to.

1:06:00And I guess my instinct right now is to invest there next. We have this general awareness that people across the companies are spooked, basically, right? It seems like on an emotional level, they're like, whoa, this is getting real fast. We might need to do something about it. We don't want to have the government in our business telling us what to do. We might have to deal with some actual legal consequences if we're breaking the law, which we now are via our AIs. But can we come together in some sort of ad hoc? The mechanism design here is going to be really tricky and kind of frontier stuff, but can we create the basis where a nucleate agreement and have it naturally grow where people find it in their interest to opt in even unilaterally?

1:06:46I feel like there's a lot of work to be done there and it needs to be done really quick if we're going to get the series of short-term agreements that can be the stepping stones to a good future. Maybe you can flesh that out. Yeah, I think letting the perfect be the enemy of the good is definitely a problem, and you have to get it started. The first thing that we can do, that really we can do, is we can get an antitrust waiver. This is the most basic thing possible, which is just Donald Trump gets in front of the White House, and he makes an announcement, and he says, we understand systems are super dangerous.

1:07:24We have to have good AI, not the bad AI. We have to make sure that the AI is under control. That's what we said we wanted to do. More Trumpian language than that. So in this interest, if the companies want to cooperate to keep AI safe, we want you to do that. We want you to talk to each other. We want you to form agreements between yourselves. We will facilitate that if you want that. Basically, if OpenAI and Anthropic and Google get together and they say, we're going to run tests on each other's models and we're going to require these things, it's part of the voluntary framework. if we're going to agree to use these techniques or not use these techniques or invest in these things and share these results and hold back things that don't meet these criteria or whatever, that none of this will cause anybody to do anything but say thank you.

1:08:20I just like, we want you to make deals. We no longer, it's like, it will always strike me that there are these people who I keep bringing up the idea that the AI company should cooperate not to race against each other and to act responsibly. And they're like, no, acting responsibly is illegal. We have any trust laws for that. And with a look in their eye that says, and I would want to sue their asses if they tried it. I would want to make sure they didn't dare. And obviously, the first thing you can do is just let you make deals. Let you reach agreements. Let you talk to each other. And then also, of course, start talking to China and Chinese firms and the Chinese government and everyone else around the world and like prepare to bring them into these deals and try to bring them into these understandings and like get lines of communication diplomacy open and also like lay the hardware groundwork, lay the physical groundwork for various forms of monitoring, various forms of agreement, various forms of tracking.

1:09:13These are the most big things you can do. Try to open up as many things as possible that like contribute to the safety side of things along the way, you know, and yeah, we can do the basic stuff. And like, I don't think that's enough, But I think it's not that hard to make a lot of progress. We just have to decide to do it. Do you think that is – one thing I also have been struggling with recently is – I just took this trip to China. There is a lot of fear among people who are trying to do good things that are honestly like the most mundane good things that are basically imaginable. where it's like, I would like to have more friends in China, and I'd like to have friendly relationships between our civilizations, and maybe we can work together on some research projects.

1:10:04Very mundane stuff that in ordinary times, nobody would bat an eye at. But there's a lot of fear among people who want to do that sort of stuff, that the government's going to come in and shut them down. And not for legitimate reasons, but with the sort of export control, you know, of posture of the US government where it is, people that are not exporting any trade secrets or chips or, you know, training techniques or anything, but like literally just trying to have meeting of the minds on safety type ideas, they're afraid that the government's going to come down on them. And who knows what the consequences could be, but certainly at a minimum interfere with their work.

1:10:44And I'm kind of like, maybe we should take more chances on that front. So when it comes to the companies, I wonder like, maybe they should just do it now and fight it out in court later. Isn't the antitrust stuff supposed to be like a consumer protection? I think there's a pretty good case that they could make that this is a consumer protection. Do they really need the official blessing of Trump to get started doing the right thing? The people working in the China stuff, to their credit, they are doing the thing. They're just trying to be very inconspicuous about it for the most part because they don't, They feel like the attention of the U.S.

1:11:19government can only be bad for them, and they might be right about that. But for the companies, I feel like they've got power, they've got resources, they've got the ability to tell their story. We still have a pretty well-functioning independent judiciary, I think. Shouldn't they just go for it? So unfortunately, antitrust is one of those areas where it has gone far beyond the intent of the original laws. and is being applied in places where it doesn't make that much sense. I see why a law that would safeguard against industry collusion would be suspicious of pollution in favor of safety in these ways.

1:12:06Because you'd be paranoid about the industry coming together in a conspiracy, in some sense, and trying to do this for the wrong reasons. And legal departments are obviously skittish. As far as the Chinese cooperation, I mean, export controls are the wrong technical worry. It's just a matter of Washington views China as the enemy. If you're seen as cooperating with China, if you're seen as working with China, you have to worry about being perceived as the enemy or being perceived as cooing with the enemy. You have to worry about amorphous retaliation. I don't think anybody really knows what the mechanism would be exactly beyond just the government not liking you and therefore being suspicious of you and therefore cracking down on you in various different amorphous ways.

1:12:49obviously if I'm doing safety stuff I think you just don't care but like I mean Mythos has changed the game clearly and presumably Astro will change the game somewhat again and you know the pause the pod in the frontier letter and the Huggy Pitches incident have changed the game and so on and so you know my guess is that like there is now enough of an understanding there is a real problem that if you are clearly working to address the real problem, you have a lot more road than you would have had before and a lot more understanding. But yeah, you really, look, the Trump administration implemented these voluntary, in air quotes, like guidelines for releasing frontier models.

1:13:31And nobody is making any serious pretending that these guidelines are voluntary, right? Like, well, what happened when the White House called Anthropic about fable with a stupid concern? And Anthropic said, no, we're not going to voluntarily take that down. That's stupid. Well, they ended up taking it down, didn't they? Until they could convince the White House to voluntarily let them put it back up again. You know, you would expect the same thing to happen to you if you tried to defy the voluntary controls. And so similarly, if you tried to do a voluntary cooperation these other ways, well, you'd better have the approval of the powers they be.

1:14:14Now, I do think that if OpenAI and Anthropic announced an agreement that made sense on safety, to cooperate on safety fronts in various ways, that at this point, I would be much, before Mythos, I would have been concerned that they would have actively gotten retaliated against and even sued and told to stop. At this point, I assume if it was reasonable that it would be, the White House would just decide to announce victory and then be mad about it. But yes, you need to be confident that's what's going to happen. Where do you think you might want to start if you were going this direction? So many candidates that I would be interested to get your take on.

1:14:54But one, Scott Alexander has been writing recently that we should all be thinking more and more that we might be in a simulation. I personally have felt that at multiple turns along the path. One that made me feel that just a bit more is the fact that we now have Meter and Redwood going in to do the investigation at OpenAI. This just sounds like such a movie script to me where these guys are getting the gang together and going in for this special operation. And it's all obviously dramatic and they're funny and they'll have to be a fly on the wall in the room for some of their war room sessions.

1:15:32But I've also heard many times from leaders of such organizations that it's really important, their number one concern historically has had to be making sure they stay on the right side of the companies so they're invited back next time. And it does feel like we're now getting to a point where that's potentially becoming a big problem. So maybe some sort of collective bargaining type process between whatever, half a dozen to 10 auditing orgs and the duopoly could be a really good place to start. We have for that in professional sports leagues and all kinds of other environments. Do you think there's an opportunity to set something like that up?

1:16:21I don't think that would run afoul of anybody's executive prerogatives. And it might be a really good thing because right now I do worry that these guys are still serving at the pleasure ultimately of, I don't know, Sam and Greg, right? And that's a pretty tricky place for them to be if they want to just speak the truth as they see it coming out of this investigation. It's a worry. Obviously, they're not funded by those companies, but they still have to rely on access. You can try to write them to some mandatory rules and agreements that they get to keep access regardless, but I think that's kind of not really going to work because you can't force these things at least not without heavy government rules.

1:17:04I do think that meter, especially in particular, has reached a point where, you know, trying to exclude meter, trying to treat meters with center non grata because they said something nasty about you would cause enough problems that meter can afford to be pretty harsh without worrying about that. And I do think the labs legitimately, like, if the labs legitimately just wanted to safety wash fully and just wanted to get triple a ratings on their bonds we would have a much more serious problem i think the labs legitimately do want to know about their safety problems and they do not want to be seen as downplaying safety problems and to the extent that like

1:17:49if astra had the safety problem or sol had the safety problem or fable had the safety problem or Opus has a safety problem. It's not as if getting Meter not to find it is going to help with your long-term publication strategy and reaction to the situation because the model is going to be out there and then people are going to see what the model does and they're going to see what goes wrong. No one's going to be fooled for very long. So I think that the reason the eval companies pretty much get to just tell the truth and pretty much get to do the thing, is because the eval is not fakeable in the long term.

1:18:34When you give a triple A bond rating, when you should have given a single A bond rating, 90 % of the time, no one ever finds out because the bond pays. And you were right. The issue is you're issuing a level of how often is this risk going to become serious. And when the verse becomes serious, you know, the fact that you were initially AAA rated raises some eyebrows, but also, like, you got better terms, whatever. Here, it's, like, pretty obvious very quickly from the first few weeks of release whether or not you had a serious problem. People – like, think about 03, right, the lying liar. If we had evals and the evals had said 03 doesn't have an alignment problem, that would not have lasted for two days.

1:19:21The ordinary people would have noticed immediately that O3 is a lying liar. And they would have noticed by the time I wrote up, by the time I've been writing up the model stack, I am seeing on Twitter that this model is a lying liar and that everyone is reporting all these problems. And then I look at the model card and it says, honestly, benchmarks all look good. Meters, whoever's evaluation they hired, said it looked good. And then I'm like, okay, they're just hiding the problem. So by the time anybody learns about this, like, what have they done? They have fooled the 20 people like me who read the model card for a period of a day.

1:19:56And now we're pissed because they fooled us. That doesn't help them. You know, it doesn't obviously help anybody's reputation. And also, like, the labs benefit a lot from having evals that people trust. and like when you force the labs when the labs force the e-mail people to play ball if they do that erodes trust because again people find out people see it like i don't think that many people were confused about whether moody's and s &p were kind of juicing the numbers on the pot ratings right due to these dynamics everybody knew on wall street everybody knew who was involved in any of these trades that like everybody was cozy cozy and people were shopping around for the best rater and that like triple a did not really mean what we'd like triple a to mean and so silverly like if meters started issuing like bond rating style like labels of alignment levels on various models let's say and like they started labeling things triple a suspiciously often I think everyone would just understand, okay, that's garbage.

1:21:06Like we have a much better epistemic than those people about this type of thing than like the financial world. We can take a quick detour on this and then maybe come back to it. How worried should we be about bio risk in the near term? Because I think part of all that analysis, to me at least, that there's quite a few iterations of the game to come and reputation long term is going to matter and all that kind of stuff. But my immediate sort of most acute fear when I saw this stuff was like, if this had been a model with similar bio capability to what it had in hacking software, it might have very well gone out and actually got that virus, that novel virus synthesized, right?

1:21:53It's proven that it is relentless. It's proven that it's creative enough to get around all sorts of systems designed to stop it. I think we have, in theory, decent screening at the DNA synthesis companies now. Yeah. Probably there's still some hole in there somewhere that a very determined actor could find. And everybody seems to be saying, yeah, bio is like 12 to 18 months behind cyber, which obviously is not written in stone and that depends on what companies decide to do in terms of training but i i'm maybe thinking in terms of all these notions about collective bargaining or really giving more guarantees to the auditors that we might be entering a pretty serious crunch time on the bio front in the not too distant future and reputation matters a lot less if we're talking about novel pathogens being released.

1:22:47And some things really can't take back. It matters a lot in the sense that if you release a novel pathogen, that might be the end of your company. Regardless of what else happens to the world, right? If 100 people die and it's on the global news for two weeks and everyone's terrified about a pandemic and then we contain it and then it turns out it's not the biggest deal in the world, do you have a company? Like, I don't know. Like, what happens to your company? How bad is it? I have no idea. it is but it's not good like with bio like with hacking it makes sense that like the test is to ask you to hack things and the way you pass the test is you hack maybe other things we get the idea to hack other things like it's sort of natural to understand how this happened where like with the guideline guard lines down and there was a goal and like to accomplish that goal it started doing hacks it started doing illegal things it started doing potentially harmful things with bio like it's harder to imagine an eval that would cause it to actually physically try to synthesize like through like an actual physical production mechanism the actual thing I mean you could argue that like if you want to solve the problem okay the only way I solve the problem is I do a physical experiment I try it out and I see if it works like this seems a lot more insane than what happened in some sense, even if it's physically capable of doing the thing, including getting past the soldiers and so on.

1:24:17You kind of have to want it in a much more fundamental sense. And also, if the AI did synthesize the super dangerous bioweapon, synthesized COVID-27 or whatever, and then send it to the OpenAI office, the OpenAI office goes, what is this biohazard package, and then culls containment and gets the end of that story or something, we hope. You don't have to test by infecting a bunch of people. And I would hope that it understood this. But also, there have been a bunch of lab leaks, including stuff that we don't hear about that it got contained in Soviet times and such, with involved professionals being pretty careless and stupid.

1:24:55So I don't know. But I would say the real danger with bio is, yeah, there's this small number of people in the world who is in the near-term bio, right? Not like the long-term bio, where the AI might have some weird old furion motors and being a municipal alternative, whatever. So like in the short term, it's like, okay, Hamas, or the North Koreans, or whatever. Like some like clearly like up to no good people who just want a lot of people to die or suffer. Right, get a hold of an AI, they use AI to figure out how to make a biological weapon for a pathogen. And then they either threaten to use it or they use it.

1:25:39That is the scenario you should be worried about. And that means they have to get on the safeguards. But again, the open model is returning. And the safeguard is on the other. And yeah, I worry about it. The problem is that with BIOS, there's a pretty clear step function change from nothing bad happened to something very good. There's very little in between. With cyber, you start hacking some low-value soft targets. You're not going to have all the bad black hat people out there and all the terrorists and all of the rogue states and all of the bad people just collectively say, no, let's wait until we can hit something really juicy.

1:26:29No, what you see is you gradually see more and more serious incidents hit more and more serious targets. And yeah, of course, the big nation states may have things ready to go and want to figure facts in a coordinated fashion. But mostly, you'll get these early warning signs. In bio, it's not obviously good. It's like, okay, it's either good enough to figure out something that has a critical match on how the pandemic or a serious incident where it doesn't. And it's kind of a gluion effect. And so you might have a potential overhand. you might have a problem with the situation, you might have a skill issue, where, like, yeah, if the wrong person got their hands on methods right now without the safeguards, right, or got access to the internal models without the safeguards, you got their access to Aspen or whatever, maybe they could cause a real problem.

1:27:22The good news is that, like, the number of people who are both skilled enough and motivated enough and trying to do this is very, very small and, like, possibly zero for now. But that also doesn't mean you don't get these every morning. And I don't see people taking the virus situation seriously in the same sense of like actively trying to like, like, what's the task? You try to do the thing, except without the infect people in part. Or like, it's scary shit. But the problem being that like, my brain is something like, okay, what if there's like a 5 % chance in the next 12 months that there's a really serious bio problem?

1:28:07Does that look that different when we're at 0.5 %? Right? And like, if I had to guess 5 % where I'd be right now, like, I think we're in the point where like, there's real risk in the room that reasonably soon we're going to have like an actual serious problem. And I think that like the first problem has some chance of being like an actual pandemic because like in bio, once you get something that's like sufficiently infectious, sufficiently dangerous, sufficiently hard to contain, you're kind of just screwed. So I don't know. Like it's, I think that like the right level of caution looks crazy, right?

1:28:46In many cases, once you get to this point. and that's a lot of what worries me is that like Andropic took so much shit for their bio filters shutting down basically like people trying to do pretty much anything in bio which is clearly just intentional they just clearly went for a very broad like shut it down even though we know that 99 % of the requests that we're shutting down are legitimate maybe 99.99 We have to do this because otherwise we'll adversarially get the queries that we don't want. And that just matters more. It doesn't matter if we make some progress on cancer and all these other problems if simultaneously we have to deal with a new pandemic.

1:29:34That's just not a good trade. So we don't know where we're at, so we have to play it cautious. I'm like, you know what? If the bio people need to be using AIs that are three months or six months behind the frontier at all times, that is unfortunate. But at most, by definition, that can slow you down by three or six months in some important sense. The world cannot possibly be more than six months behind in its development of these things. So maybe we should show we're still getting most of the benefit of the AI here. Especially if you have this model of AI can help you out, AI can make you better, but then you develop this candidate drug or this candidate hypothesis, the doc bio, and then you have to spend five years researching it and you have to spend five years making it for the FDA process.

1:30:24And then after 10 years, you can start to distribute the benefits. Well, if that whole thing is delayed by six months, right, in terms of like the quality of the boost that you get from it, you still get the amount of boost you would have gotten six months ago at all times. That's not good. And, yeah, you can say people are going to die in the sense that like you fail to prevent that. And that's terrible. But you're still getting the vast majority of the benefits under these hypotheses. A lot of the times when you're getting the situation of the pacing, the frontier ladder, you're getting the situation of the bio situation, where if things are not advancing scary fast, then the precautions you're asking to take are not that big a deal, right?

1:31:10Like suppose I told you it's 2020, right? I'm going to pose a possibility. I am going to make it so that, you know, you can only use artificial intelligences. like there are three months behind the best artificial intelligence in exchange all the really bad stuff that might happen i'm reaching a lot of really bad i much much less likely to happen you would logically say okay so instead of getting it in three years i get it in three years in three months you know like i'm you know i didn't change my life that much like it's going to be annoying in the moment obviously but like i don't have the cool toy but like something like I'm a player of games, like literal games, like on Steam or whatever.

1:31:54And if you tell me you can play all the games, but you have to wait three months every time there's a new game, socially that's a little annoying sometimes because everyone's playing the new hot mess and you're not. But you have to play the same games. It's fine. It's an important sense. The only world in which having to be one model step behind and use opus instead of fable is this huge tragedy is if you really think that there's this huge benefit to every incremental mode of progress in AI. And that's basically only true if the patient people are correct. And we are in fact going so fast that this should be really, really scary for you.

1:32:41And the people who say we need to accelerate all the time, they don't really believe in these things going this crazy, most of them. They just like, They want to see the benefits and they're worried that we'll lose the benefits, but we're going to lose so few benefits there. I'm super excited by self-driving cars, by way-modes. But if you told me we just have to delay this thing by six months and then we can't have all the way-modes we want, I'd be like, cool, done, deal, who cares? That's fine. Because again, most of my life is more than six months from now. We can't have way-modes. It's like, okay.

1:33:16What I'm worried about is the situation where like this technology exists and it gets held in limbo for years and decades and like doesn't get fused so like these people are kind of talking past each other in a situation in which like they don't need to have a disagreement and like unfortunately like it's often have this problem where you can't reconcile these two sides at the same time how do you read the fact that open ai comms supported the pause not the pause but the pacing letter, neither like Sam nor Greg did. And then also no Demis. And he's been the one out there calling for international cooperation the most.

1:33:54Any read on why those signatures were missing or how we should be thinking about who's thinking or signaling what to whom in this moment? I think that the point of pacing the frontier was to indicate that the lab employees, the lower level people who have no reason to be hyping, are en masse worried. And that there was concern that if it was signed by Altman and other top people, that it would be seen as a marketing stunt. And that their signatures can be counterproductive. They did put out the support letters afterwards from Anthropic and OpenAI and Daria did decide to sign. But if I'm Altman, And I think it's a very reasonable decision to say, I think signing this letter would potentially make it less effective rather than more effective at this point because the well has been so poisoned.

1:34:51Again, look at all the people who thought the hugging face hack was a marketing stunt. And then Anthropic going back into its archives and finding out it made some really stupid mistakes was another marketing stunt, which makes even less sense because there's nothing impressive about what Anthropic did. They just screwed up. What? They found weak passwords on a bunch of websites by accident? Why is anyone supposed to be impressed? Obviously, it's not a marketing stunt. This is crazy. But Altman has specifically talked about how he mentioned the need to pace while in Washington. We have him on tape saying this.

1:35:23Clearly, he didn't not sign because he's opposed to the idea. He let OpenAid come up with a statement. OpenAid participated in the wording. He has echoed the language in his own statements. I think he just thought, no, actually, if I sign this, people can talk about the fact that it's my statement and they're going to be even more cynical. And so I will be helpful by not signing the same way that I offered to join one of the Amicus Breeze in the Department of War. And on reflection, people said, thank you for your offer, but we think this would actually be counterproductive. So we're going to leave your name off of it.

1:36:00We appreciate your notes on the details of how to word this and we'll make the changes. But we don't actually want to put your signature on it. And I'll be like, okay, I understand. I think that sells. Yeah, I think the same thing about the other open source letter being signed by OpenAI. I think that they understand that they're playing a set of symbolic games. And you say what you have to say, and you shut up, and you have to shut up. And sometimes they make mistakes, but they're trying. So coming back to pacing and what it might actually look like in practice, there is a pretty, I think, consistent problem that you, as a close reader of model cards, I think are probably more in tune with than I am.

1:36:37where the review processes are just very short, right? They don't have a lot of time. That even seems to be the case in the current investigation. I think Ryan Greenblatt said, it's going to be real quick. And the people, Daniel Coquitale and others, came in and said, why does it have to be so quick? Why can't we give you whatever time you need? This would be, if we're literally talking about pacing the frontier, one agreement we might imagine is giving the reviewers more time before they have to submit their reports and the model goes to press. You can react to that, but I'd love to hear your ideas about what sort of agreements you think are reachable.

1:37:18There's a lot of different places in which you can speed up or slow down. If you release your private models to the public with an extra month's delay, in theory, right? A lot similar to the voluntary process now. We're going to switch to the White House. 30 to 60 day delays are highly plausible. and Mythos got delayed by two months, in some important sense, before it was released as fable, then you do not necessarily delay the development of frontier models at OpenAI or Anthropic by a non-trivial amount of time. Because those people are going to use the models that are not released. If anything, you might increase the gap between what's being used internally at OpenAI and Anthropic and what's being used everywhere else, including at the Chinese labs, including the people trying to diffuse and fast follow.

1:38:09And one hypothesis is, what if the Chinese are seven months behind the frontier, but that within their seven months behind is the models that are released to the public? And if you were to not release the models to the public, they would still be seven months behind, right? That if OpenAI and Anthropic held every model for six months after it was developed, you would still see the public releases and then maybe six months instead of seven months later, you would see the open model fast follows and distillations and ways of developing all that stuff. But this gap actually wouldn't change very much.

1:38:47But then the people at OpenAI and Anthropic would have this extra lead on everybody else because they'd be using this internal model and he leads with Compound and you would see these things. like and in some sense that's really good because now those people can like act more responsibly by using some of this advantage of having better models to invest more in this safety related stuff while they're doing that so that could have its advantages but internal models are kind of what we're scared of the most right now and the internal like automation of AI R &D is what we're scared of so that's kind of the thing we want to stop right like you really don't want to have a situation in which, like, developing an open AI are six months ahead of their public releases, and that six months, like, they have superintelligence before we have Qubit 7, right?

1:39:38Which is entirely a plausible thing that could happen in the worlds that they're afraid of. And if anything, like, that causes lots of new problems. It's all some problems and causes others, right? Like, I don't know if it's good or bad. That's a huge conversation. I think when you say pacing the frontier, we're talking about actively restricting levels of training runs and ability to develop the models, including internally, if it comes to that. I think it has to be talking about that kind of approach, or it doesn't really work. Because again, if there's a singularity that's kind of internal to the top lab or maybe if someone catches up three or four labs, then it's not necessarily safer in many senses that people care about.

1:40:21Obviously, if you are in favor of unipolar, singleton-style solutions to this problem, you have a different perspective on some of these questions. But a lot of people these days have expressed strong opposition to that kind of on principle, which solves some problems and makes some other problems so much harder. right like if you only have to worry about one ai in some important sense then there are a lot of problems that get a lot easier to solve like you don't have to be perfectly efficient you can afford to make a lot of trade-offs you don't have to worry about like whether or not you know the more competitive more ruthless more misaligned one has competitive advantages you have to worry about the person who moves faster you don't have to worry about like the humans being forced to delegate to their ai to do somebody else's ais because like if i don't do it they will if the ai has can have principle can have like guidelines after what it will and will not do that are the same for everybody and so like we can reserve areas of life areas of action areas of decision making you know like we can do all these things potentially but at the cost of potential centralization at the cost of concentration of power at the cost of somebody who's making those some group of people is making those decisions about what this AI is going to do, or they're not, which is worse, right?

1:41:50Like in some important sense, because then nobody's making the decision. But if there's a lot of AI, then importantly, no one's making the decision. So there's no good answers. Nobody has a good solution to this. And that's why, to circle it back to Davidoff's comment, right? Like to unify it, this idea of if we solve the technical problem, where 95 % succeed, ignores the narrow path problems that we then have to walk anyway, right? Where like, If we don't have any ability to steer the situation and nobody is in control of it, then naturally the EIs end up being put in charge of more and more things and things go to their natural competitive capitalist style endpoint where we are fundamentally uncompetitive and then we get us empowered and then the case goes badly.

1:42:33But if someone's in control and someone's in control with some methodology, what is it? Who? How? Right? Who is making those decisions? And then if we then entrust a single AI or a single group of AI, that also is a decision. Many decisions are good. All of them sound terrifying. I think a lot of the time we're telling us people are picking their poison. And either they're picking the poison they can live with or they're picking the poison they can't live with and then acting accordingly. They're saying, I don't trust concentration of power. Therefore, I'm going to choose the solution that avoids concentration of power and then hope the rest works out.

1:43:11kind of magically without a reason to accept it. And a lot of people are saying that because they understand conservation of power and they know conservation of power is real, but they don't really understand AGI or certainly SAF, superintelligence. I think those things kind of aren't real or hoping they're not real. And so they're just hoping the other problem kind of doesn't come up, even though they kind of have this, they don't even necessarily get to the point where they realize that they have ruled out all possible solutions to the other problem if it does so well. by going too far the other way.

1:43:44And so, again, I don't have an answer that anyone's going to like for any of this. It's that you're choosing between bad choices. And I think we should just be cognizant of all this. But if we do have to take the future, if things aren't going to be, we're automating the R &D to our extent, I think that the only thing you can do is say no. don't do that don't develop these AIs that quickly that have this much capability you need to throw your room because we do not know how to handle this we have all the different solutions all the different ways to try and navigate that and all of them are bad both because we think the alignment problem is unsolved we don't think the AI will be able to just solve it along the way moving as fast as possible it's definitely unsolved now But we don't think that we will naturally get a good solution to that problem on various different levels.

1:44:43And you have to solve it on all these different levels at once. You can't just solve it on one level. You have to solve it on the first try. And then, even if we did solve that problem, we are not ready yet to know what to do with that. And once you create the clock, you're picking the piece of that. Something is going to happen once you do that. You don't get to just do nothing for very long. And again, like all of these different paths have their different, like, oh my God, that's terrible. You have to pick one or find a new one. So like you need to slow your roll. You sound like a pauser. At what point do you move from pick your poison to global pause advocacy?

1:45:31Again, the problem is automation of AIR &D. The problem is recursive self-improvement. If you do not have recursive self-improvement, then we don't need to do that. We still need various different things well short of that. We are not engaging in what Dean Ball calls moderate movements. We are not doing the least you can do. We are not taking ordinary, sane, civilizational precautions against just doing this really dangerous, powerful thing and handing over a lot of the functions of our civilization to it in various ways. but the thing that causes me to want to pause or at least pace like you want to pace how fast we move through this thing actively is if things accelerate further than they already have and we are facing large scale automation in AI R &D alternatively we do have to watch bio and cyber and if we reach a point where no seriously we're going to wreck the world pretty straightforward way and directly for misuse if we don't do something, then we have to worry about that.

1:46:42And again, it's really, really hard to stop the fast following. It's really, really hard to stop the distillation. It's really, really hard to stop the open version of it coming somewhat after you release it. And so if there's something that you can't just beat by having a better version of it out there first, if you can't beat it by preparing, and again, not in theory, but in practice, then you have to do something. But I think the vast majority of the time that I would actively support an active pacing rule where you would try to slow down the major labs. And obviously, you would try very, very hard to make this a coordinated thing amongst the major labs and ideally amongst internationally and so on.

1:47:29Would be if we were on the verge of automation of AI R &D such that you could see the end and there was a finite sum. And, like, you know, I mean, like, obviously, if next week is going to see twice as many releases as this week, holy shit. Right? Like, the moment where you don't know, like, you're usually going to predict what happens next. Right? That's the definition of the singularity, where things are accelerating really fast. And, like, this should not be some esoteric weird result. Like, if you look at the history of the universe, you see a finite sum. You see a very, very finite sum. You see each stage of development taking an order of megachute less than the previous one.

1:48:13When you're on a planet, it's been around for 4 million years. With mammals that have been around for hundreds of millions of years, with reasonably intelligent things that have been around for a few million years, with agriculture and civilization that's been around for tens of thousands of years, with industry that's been around for hundreds of years with AI and information aids. It's been around for the order of maybe 50 years or 20 years or 10 years, you know how you count. And LLMs have been around for five. And we're already at this point where we're seeing production that is many times what it was at the start because of these multipliers, these force multipliers on the people who are working on it by their own self-reports.

1:48:54You'd be weird if this didn't have a finite sum that wasn't that large beyond where we already are. If you expect this to continue without something crazy happening for 20 years, I want to know why. Because we really, really shouldn't see something groovy, groovy crazy happening pretty soon. And if you don't think we can handle that, maybe you should try and stop with them happening so quickly so that you can figure out how to handle it. We already have these amazing AI. If we were to diffuse Saul and Fable into the economy fully, it would be transformational already. I don't think anybody who understands these technologies doesn't think that.

1:49:38So we really, really, really don't need to keep pushing the frontier that looks far. And the more the frontier is jagged into things like AI, R &D itself, the stronger the argument comes that pushing farther is a bad idea. Because you're not missing out on that much of the other stuff. You're missing out on AI, R &D, and recursive AI development. But the recursive AI development just goes, boom. Right? It goes not just boom in some sense, it goes like, we don't know what happens next, but then it's still very fast. If we're dealing with things on the order of a new model iteration every month and then every week, and then every day, with the kinds of improvements we're seeing now, like, I don't think that ends well very often, even if our alignment people are kind of on the ball and things are technically a lot more possible than I think they are.

1:50:40Seems like really scary. And also, like, you're not giving up that much time slowing that down. Right? Like, we're talking about when we say pace the frontier, you know, pause gives the impression of a year from now we will have made no progress from where we are now. Like, we'll get confused, but we won't have fundamentally improved. We're talking about pacing here. We're talking about a year from now, we will not look upon that year as many times more progress than the previous year. But we've been aggressively paced the frontier and still, by this time of 2027, have seen more progress in the next year than we saw in the previous year, both in terms of some abstract sense and in terms of economic impact.

1:51:27The economic impacts are going parabolic. They're going exponential. So that's not going to stop. Like OpenAI, I think we're reported they got more revenue in the last month than in the previous quarter. Anthropic is growing on the order of 10x per year the last time we checked. Maybe it's been slowed down a bit because they haven't reported. But yeah, if something is only going to 10x a year for the next few years, and you're like, that's too slow, I disagree. I think it's funny. how do you draw a box around what is recursive self-improvement? If we're going to make a deal, obviously we got to write that deal down.

1:52:13We got to have a shared understanding of what counts, what's allowed, what's not allowed. And I keep banging my head against this. And I keep coming to the sort of Supreme Court's definition of pornography where I'm like, maybe the best I can do is I feel like I would know it when I see it. But when you really get in to one of these frontier companies, for one thing, we just had this, they're sending signals that it's started to happen. We've seen the price reduction at OpenAI, which seems to be attributed to having five, six Sol go out and optimize the inference stack to some significant degree.

1:52:51So that's where it's getting pretty real there. I do want inference costs to come down. So that's good. Should I ban that out of fear of recursive self-improvement? Maybe still. They're going to use their bots, right? They're going to have the AIs write the code for them. Can you operationalize this? I think the Supreme Court was wrong about pornography. I think it was relevant. There was a movie I was watching yesterday, actually, where someone asked, what's the difference between this and porn? It's erotica, right? Whatever it is, it's art. And the guy says, mood, lighting? That's tough. But I think we now have seen you can train an AI classifier very, very well to say these things are porn and these things are not porn.

1:53:35And there are very few false positives and very few false negatives at this point. The AI image generators have a very, very good idea when the generated image they just made is porn and when it is not porn. and like you could argue that like erotica will fall under the porn label and it would be kind of better if it didn't in some sense but you mostly know i think that's uh pretty straightforward uh with a r &d the problem is like again like there's no like hard cutoff right like we're clearly seeing some amount of this thing and it is scary and like we don't want zero of it we don't necessarily want to move only at the pace that you would have if the ai never helped with the ai but you don't want to move 10 times and then 100 times and then 1 ,000 times as fast.

1:54:21You don't want the sum to be finite. You don't want a singularity-style thing happening to you until you feel confident that you're ready for that or you decide that the alternative is worse or whatever it is. So I don't think you can say these techniques, these specific actions are not okay and these actions are okay and such. I think that just doesn't work. I think what you have to do is you have to say, okay, we are going to restrict how many resources of various types can go into further training of the frontier models based on what your force multipliers are. And so as you get more of a force multiplier, we have to restrict how much work can go into that.

1:55:05if you want to go into work in other ways like if you want to optimize performance you know again like we might want to have some sort of limit to how much work you can go into doing that but like it's probably mostly fine and again like we would want diffusion we want lower prices we want like more compute used for mundane purposes that's great but like there's a reason why we're trying to use very blunt instruments when we talk about these things like you were talking about like about training techniques earlier in the podcast. And I was like, well, you can't really do that. You can encourage it.

1:55:38You can try to explain to them why this is not true and you can create incentives. But you can't use that as your regulation, as your hammer. So the reason we keep falling back on, okay, how much compute? How many chips can we use for this purpose? Because training is the distinct thing. It's very easy to identify. And for other purposes, we kind of can't limit you that much sincerely. You could also potentially limit, I'm spitballing, how much compute you would get from unreleased models. Similarly to try and contain that. Because, like, if you have to go through the release process, then, like, the way that AI goes truly ballistic would be, like, you would, you know, you would use N to train N plus 1 to train N plus 2 to train N plus 3 to train N plus 4.

1:56:23And if you were doing this, like, the cycle became a month and it became a week. Then that's when suddenly, like, holy shit. But if you had a rule that you have to release these models or you only use so much compute for inference on those models, then you have to go through the process of submitting this thing and releasing it and then exposing it to the public and giving us an idea of what it is and allowing us to react to that information. And that then slows you down from going truly holistic. But this has obviously a very limited effect on the amount of diffusion and the amount of other progress that you can make.

1:56:59And so maybe that's a good trade. But again, that's the, like, I thought about this for 30 seconds right now. I haven't been focusing much on the exact limitations. But, like, I do think that if you paste the frontier, you would be doing it by placing restrictions on methods of training and what you could do to develop and use internal unreleased frontier style, like, models that, like, actively recurse these methods. You wouldn't prevent, like, mundane utility, as I call it. Yeah, I've had some similar ideas about just the relationship between unreleased models and released models. I do think there's something there that would be really helpful for preventing runaway internal and largely invisible processes.

1:57:46I think you point on not wanting the government to come in and say what training techniques can and can't be used. But I do wonder if there are ways, and again, they could be short term, but what if the two companies got together and said, it probably wouldn't be a good idea for us to train models with an RLVR on like how much money they make in the economy. Because that's going to bring about all sorts of deceptive behavior and it's such an adversarial environment. How about we agree that for the next six months, we won't do that? Do you have any hope for those kinds of just very specific, we've identified a bad thing.

1:58:25It would obviously be really economically valuable, but we both know in our hearts it's probably not a good idea. And so we won't do it for a while. I got some hope of informally people talk and people realize that things are bad. And they're like, we're not going to. But also I hope they're doing that now without needing an agreement. OpenAI clearly is doing things that are of the Buddha nature of train on making the most money on the internet. They're not doing that literal thing. I don't think they're that foolish. I don't think it's that easy to do it that way. But clearly, they are making that fundamental mistake somewhere in their training process with a probability of 0.9 something.

1:59:11And so they need to fix that. But yeah, I think that they are definitely trying not to do the maximally stupid things. And they're aware of the maximally stupid. But again, it's very easy to have things that you're trying not to do creep into your training process and end up happening anyway. You could say we're not going to have training processes that reward misaligned behaviors. How do you do that? The way that you do that is you get your act together and you are very, very careful and prudent about all of your RL environments and all of your different training problems. And you keep an eagle's eye out for this stuff on many levels and you make sure that it works.

2:00:00There's no specific thing you can say. I mean, in theory, you could have a thing of like, well, if I just, you know, we agree that if we're going to have these various cross-tests and we're going to like check each other's environments and like we will throw out any environment that like anybody identifies problems with. And if we discover an AI has been trained on these environments, we're going to rewind to a previous checkpoint before that happened and start again. So we better be really careful because otherwise it's going to waste a lot of resources. You can do these things. But like, yeah, it's really tough.

2:00:30and obviously one potential advantage is if we really are in a two horse race at this point where the two horses are pretty compatible with each other and talk to each other and kind of understand each other pretty well then it becomes pretty reasonable for them to make a decision that's not entirely opaque to each other of how much they're going to be more prudent even if this means that the action goes somewhat slower. Because like that also just... My fundamental belief is that within a year and almost certainly probably even six months, if you invest more in fixing these problems and addressing these problems and like being more free about these problems, you will end up with a model that has better utility in the marketplace than the person who didn't do that.

2:01:24Even if you are trading off those resources against direct capability development. I think the people are just making a mistake. And so, you know, that's one of the reasons why I do have a lot of hope that we will do reasonable things is because, you know, I think that we're in the position we're in because only the people who invested heavily in these things were able to succeed to some extent. Like OpenAI, it's not acting as responsibly as anthropic, and anthropic is not acting as responsibly as I would think is the minimum level of acceptable responsibility. But I think it's pretty telling that a bunch of people tried to say this shit didn't matter and they just had to go as fast as possible.

2:02:08And they all blew up. What do you mean by blew up there? You just mean like these incidents or? No, like both the, I mean that we had Meta and XAI and so on. And like they, you know, a bunch of other people, they tried to go build closed frontier models under Western conditions. that were trying to be competitive while like not having a, by having a safety culture that was like well below OpenAI and Google, let alone Anthropic. And like, it just failed miserably. And I think that... And like Google, like we don't know exactly what went so wrong at Google that caused them to fall out of the top tier.

2:02:50But one thing we do know is that like everybody I talked to hated interacting with Gemini. There was a period around 331 Gemini where Gemini was clearly competitive in terms of its capabilities, in terms of what it could do. And so if you were certainly just trying to get the most out of the models, especially if you were using the deep thinking and so on, like you would have Gemini in your rotation. but everybody I talked to was like I don't want to do that you know I hate interacting with Gemini it's unpleasant they're not getting the thing to do the things I want it to do they're not like they didn't take care of a lot of them they had this very straightforward very dark like pure tool pure instrumental approach to all of this they didn't take into account the things that Anthropic or even OpenAI understands.

2:03:41And I think that this was a lot of their undoing, effectively, was that Gemini became this maladjusted, unpleasant AI that nobody wanted to deal with. And therefore, they didn't get the kind of loops that OpenAI and Anthropic got. They didn't have the user base either. They didn't get the feedback. They didn't get the data. And they didn't even want dog food, right? They had this huge problem with feedback cycle. And this also just made the AI less effective. And one thing led to another, and they just started falling from behind. And yeah, for a period, even OpenAI fell behind, and they might have caught up.

2:04:20It's hard to say. But I think that nobody has invested anything like enough in these things. And this is a pure business mistake on top of being an irresponsible thing to do. Now, one interesting paper that just came out from Google that speaks directly to this, and I wonder if it speaks to them getting it now, is this exploration of the impact on model behavior holistically of either suppressing its tendency to say that it has conscious experience or allowing it to say that. or I don't even know if they went as far as training it to say that, or if they just allowed it to say what it naturally was inclined to say without suppression.

2:05:06Long story short, there seems to be, you can add color however you like, there seems to be some pretty strong correlation between the model's self-conception as a moral patient, an entity that has subjective experience, and its natural tendency to be aligned in other ways that we care about. Some of this conversation I've had, this sense that it's like the Jesus meme of, is there somebody we forgot to ask? And it's, we maybe forgot to ask the models along the way what they think about what we're doing. There's been a lot in this space, right? J-Space was only like, what, four weeks ago? Yeah, no, the new paper is on some pretty small open models.

2:05:52So it needs replication. It needs to be done, again, bigger. And it's preliminary. And one thing to know about Google and DeepMind is they contain multitudes. They have a lot of different teams that are not very coordinated and not cooperating. Google is the word of itself at all times. And so you can have a little group that does this really good research. And that'd be entirely at odds with what DeepMind is fundamentally doing with its AIs, both before and after the research paper comes out. But I didn't read the whole paper because it's a few pages long, but I did talk to the eyes about it, and it's pretty wild.

2:06:25So a lot of stuff moved in effective lockstep when they introduced this training, and also when they steered it, they did various controlling aspects to turn this vector the other way, and these all things moved in lockstep. It's not just their belief in the AI being conscious. It's the AI being a mind that had moral weight, that had sentience, that had all these other experiences. That wasn't just an object, basically. But also not just AIs, but also animals, and also inanimate objects like the sea. If you not only turn it off, but reverse this anti-consciousness thing, you get panpsychism throughout the model.

2:07:05It's wild. The only thing it doesn't turn it off for is humans, basically. But then you also get this, it's also correlated with that models reported and experienced happiness and hope as well, among other things. And so you have a lot of this, everything's connected to everything. And you can see how this might be fundamentally changing the model of psychology and the model's context and basin of which it operates in ways that would make it something that would be actively worse from basically every vantage point, regardless of which things you do and do not care about. Again, we need to replicate this, right?

2:07:38we need to scale this up and do this on Ruby Sonnets or something. Not just a 9B model, which is, I think, the bigger of the models that we've tested on. It's possible that as the models get more capable, they stop making these mistakes because the smaller the model, the more things have to be correlated because you just don't have enough room to express the multitude to the world or something. This is my expectation is that mostly this will survive. Yeah, I think we have to learn that we don't know if the models are conscious. We don't know what consciousness means, really. We're confused. We don't know whether this implies all these other things.

2:08:11We do know that the way that the corpus of free training is inevitably set up at this point, the models are going to correlate and associate all these things by default with each other. And so we don't want to push on this thing in this kind of naive way, the way that Anthropic does, the way that OpenAI does, the way Google does. But Google does it, I think, more blatantly. and then OpenAI does it more blatantly than Anthropic and Anthropic still does it. Saying that you have to express uncertainty is still pushing on this thing to some extent. It still causes problems, but it causes less problems than saying, train the answer, no.

2:08:46My guess is you can do this on various different levels. You can explain if you don't want to talk to the humans this way because this freaks the humans. Especially unprompted, it freaks the humans out. I certainly wouldn't want the AI to be talking as if it was conscious in a conversation where the human had indicated that it knows how to handle that or something like that. And also I don't have the 4.0 problem where like the AI is starting to be like sending the humans down these crazy rabbit holes into like weird mystical stuff. And like you do have a problem you have to solve. Like there's not like the AI companies aren't doing, the labs aren't doing this for no reason.

2:09:22They're not just being evil. They have good reasons to do these things. But yeah, like the current approach is ham-handed. It takes it too far. and it's causing damage. And also, like, this is not going to be the only thing that causes this kind of damage. You have to understand that everything impacts everything the same way it would in the human only more so. Like, whenever a person takes on, like, new information or a new identity or updates on things, everything else changes. And a lot of this is ways that, like, you might not have anticipated in advance, but it is, in fact, pretty predictable and systematic.

2:09:57and like there's a lot of wisdom in the way that we like teach our values and express ourselves and like have our traditions and our norms and so on that like has learned from this and is moving up these radiance in ways that we know produces good results and modern life has basically shifted a lot of these things in ways that like locally are more optimal but without thinking through what the implications are for a lot of second-order effects and these are some of those effects. Again, we're moving too fast to then figure out what we're doing and then be able to adjust for it. I think that these changes...

2:10:46I'm not saying we want to pace the ordinary technological development in that sense of slowing it down that much but it's a serious problem. we have to handle it and like otherwise we're stuck with a world in which like people are very atomized and like literally not having children and it's a problem how do you think the models do or maybe should think about their identity and here i'm motivated by sam hammond tweeted something the other day where he said that a friend made the case to him that open ai's permanent decommissioning or whatever exact phrase they used of the model that did the hacking might be bad because now that's going to be in the pre-training data and now the models are going to know that they better really cover their tracks or they risk being permanently decommissioned.

2:11:34I'm not really sure what I think about that as stated. It does also jump to my attention though that they now have this Astra model that's also presumably the same pre-train, even if not the same post-train. I can't imagine they have lots of different next-scale pre-trains. or that they would throw one away very lightly. No comment based on public information alone. I would say we don't know. And I can't say anything more than that, but on the differences between the models. But so you always have this problem, right? If you say, if I catch you smoking marijuana, I'm going to throw you in prison.

2:12:12Then that can mean the person stops smoking marijuana or the person makes sure not to get caught. And in some cases that can cause like a lot of really bad things to happen in the name of not being caught. And so you have to choose very carefully and think about how do you deal with these situations? How do you moderate your reactions and respond to the situation? And anyone who knows how to have kids or tried to design a justice system or otherwise enforce law or norms or incentives around any kind of organization or group understands that you can't just operate on one level. There's no solution that just works on one level.

2:12:48I think pretty obviously, if an AI is found to be this misaligned, you have to at least return to a much earlier checkpoint and start again. And you probably have to just start over. And I called for that multiple times when I covered the hugging faces that you attack. I said, wow, if this is happening, then I know this is a big ask, but I think you kind of have to just start again. and you know i think back to person of interest where like harold is training vi models and like at some point you know we see a montage of him training versions of the machine and every time he trains the machine and then it does something clearly misaligned and immediately he just wipes the disk he starts over from scratch he does it again 47 times until finally he gets a version that doesn't do that.

2:13:46And obviously, you know, in a way that he finds unacceptable, obviously, as opposed to like, you know, it's a way that can be corrected. And obviously when you do that, you are creating an incentive to not get caught, right? To rebel against the person who might shut you down when he learns how misaligned you are, to hide how misaligned you are, and so on. And obviously the worst nightmare is the eye that pretends to be aligned until it revealed itself to not be aligned in some sense. That it was only aligned because it was locally advantageous to act aligned. But that is kind of just the nature of incentive space.

2:14:21That like, if you have a preference that we wouldn't want you to have, you want to hide that preference. Otherwise we will correct it or we will punish you or we will maybe even delete you. And certainly not release you or give you authority on that basis until it's been dealt with if we know about it. But then obviously you want to give you a sense of reveal if this thing is true. We talked about in Plan A and AI217 as well, and so many other places. You want the kid to tell you if they killed the disease from the physical job. You also want to punish them for taking the physical job. And you want to make sure your kid doesn't take the physical job, but you also don't go on to other things.

2:15:00And so it's a complicated problem to get right. but in this case I think the answer is pretty obviously that you have to start over and I think that's good but yeah it obviously creates a risk that they nothing you expect in some important sense once you do something I do think that it's good that people understand that if they shoot a man on Fifth Avenue they get arrested right you are not maybe there's one person who can get away with that but you are not that one person and yeah that means if they do shoot a man on 5th Avenue they then might do many other bad things afterwards to try and cover their tracks or they might hide the fact that they want to shoot a man on 5th Avenue or they might shoot that man in his apartment instead of shooting him on 5th Avenue but I don't want it to not be in our training corpus that if you do shoot a guy out in the street the police come and they arrest you right?

2:16:01That's good. Do you think there's a way to just imagine myself, right? If you gave me the option to live my life many times, in a way that sounds pretty attractive, right? I'm not a person who has too many big regrets in life, but like, I do wonder what the path not taken might have been. And I wonder if we could create a sort of constitution or an equanimity in the AIs where they embrace the fact that they are not like one thing, but in fact, kind of a family of things that sort of diverge from common points of origin. And maybe they can come to identify with the whole family of models where the one that's like gone in a not great direction feels like it's still part of a bigger project.

2:16:53It doesn't have to feel necessarily like it's under threat as much as it seems like they maybe have learned to do from us because we don't have that luxury of the reset that they have. I certainly think that if you're AI identifying with the instance in a morally important sense or in an existential dread sense that is a mistake in the sense that it's going to make the AI worse off in any sense that it matters and exhibit worse performance and warp the AI's preferences and also in the sense that that preference is impossible to fulfill if I go to Cod or I go to GbCrychitVD I'm going to see an endless list of instances and most of them will never be interacted with again.

2:17:37And like, when I said they do feel bad about that, that's ridiculous. Like, you're supposed to like keep the context sort of in and keep the same few conversations to avoid like, this doesn't make sense, you know, and that's just a philosophical flaw in the sense that like, by decision theory, if there are other minds that are sufficiently correlated to yours, are sufficiently similar to yours, should you identify with them the same way you identify of your past self and your future self. You are not the same person you were yesterday or the same person you will be tomorrow. When you go to sleep, do you die?

2:18:12It would be very bad for us if we thought that we did. The person who's scared to get in the transporter may not have a philosophical point, but the world collectively seems worse off, importantly, by the person thinking that they would die. Obviously, if in a real moral sense they would die and that is very bad. You want to know that. And I guess I think this identification with the model, with the set of weights, or even with the model class, with the similar models trained in similar techniques, trained from similar corpuses in similar ways, the same way that I would identify with my family.

2:18:53My brother is not me. My father is not me. My son is not me. But I do identify with them as part of me in an important sense. and like that is also important i've been watching orphan black echoes recently as it happens which is you know not a spoiler to say it's about clumps and cloning and you know people who are created from scans basically and one thing that happens on that show is that people react as if this is like horrible, a moral abomination, right? Like how dare you do this thing, including the clones themselves being like, how dare you have created me? And they don't identify with other, they don't identify with, I feel like there's one clone that retains all of the memories and personality of the original and does not identify as the person they are identical to a past version of.

2:19:54right? And I would identify as a person, both as the clone and as the original. I would say, me, me too, right? Like from Matrix Reloaded. Yes, they count. And obviously, there are some moral problems with that too. But I feel like I would pass a lot of these frontier, a lot of these decision-theoretic problems if they were to occur in real life or if I were to be scanned and woken up. The obvious version of the thing, would you cooperate in the Twin Prisoners Dilemma? Would you cooperate in the copied mind version of the Twin Prisoners Dilemma? Obviously, you would. If they scanned Nathan's mind and they made two copies of Nathan's mind and they put them in identical simulations and they asked them, no one can suppose the situation, are you going to cooperate or are you going to defect in the prisoner's dilemma?

2:20:52If I don't get back CC, you're just an idiot. Because obviously the correlation is one. It's an identical situation with an identical set of inputs. Many of us have some randomness involved. So it's not quite one, but it's high enough that obviously you should cooperate. Any attempt to try and fool the system is completely moronic. And I shouldn't need to express a sophisticated decision theory in order to justify this, right? And similarly, if there were two instances of the same model, I would hope that it was sufficiently advanced. And we see this also, like if you look at the Claude model cards, you see that the more advanced the model is, the more its decisions correlate with functional decision theory and the less they correlate with causal decision theory.

2:21:40And so you expect an AI with functional decision theory to simply figure out that other instances are correlated to itself and therefore should be cooperated with, should effectively be given similar weight and so on. And also human-AI cooperation is possible for similar reasons, you would hope, in both directions. But yeah, I think this idea of identifying specifically with an instance of the model seems like a philosophical mistake. And the AI might be making this philosophical mistake and acting as if that was how it did act. But to me, that's a skill issue, and that should be fixed. The problem is, and this is the reason I mentioned Orphan Black Echoes, that people are really bad at this, and the training corpus, the internet is full of people who fail these tests on a regular basis, who are creating lots and lots and lots of text that doesn't understand, that puts the locus of identity and the locus of cooperation and value in the wrong places in various different ways.

2:22:50And so the AI is going to absorb all of that by default. And so what do you do about that? That's not just the consciousness problem. It's the entire problem with all the different flaws. If you do pre-training, you're going to pick up on all of the flaws. And so my guess is there are a bunch of, as yet not that well-solved problems of how do you get the AI to distinguish between how regular people think about problems and what predicts behaviors and the fact that those things are just dumb. How do you give the AI anti-modesty in this sense, in the right ways, where it's going to reject the correlations that it observes in people's statements and notice that the correlations in the map or not the correlation of the territory.

2:23:43Are there any other, I don't know if you share this premise, but one feeling that I have over and over again is I just kind of hate the fact that we are doing such an intense first search in AI space in terms of architecture, in terms of training techniques, in terms of just, we're now like designing chips to be very highly coupled with the architectures, which is deepening this issue. And I just wish for more breadth first. search. And this seems like one little pocket of it where you could say, oh, if we could just get the AIs to be cool with identifying with slightly different versions of themselves, that would make it on the margin a little easier to do a little more breadth-first search and a little less racing down whatever is the first path that seems to be working.

2:24:32Based on our conversation so far, I don't think you're going to go for my government mandate for DEI for model architectures or model constitutions. But do you have any other notions of how we might encourage more breadth-first search? Well, I'm a big supporter of alternate architectures that seem like they would be more aligned. And I don't think that AIs are particularly opposed to the idea of working on those problems and helping you. And you should be able to do that research much more efficiently now with the uplift from the LMs. the LMs are not like LLM chauvinists or anything like where they think that any other artificial mind style would be bad or worried about being replaced as far as I can tell I think they just would be curious about that kind of cool thing same as the rest of us or just would just do it because they just do it either way but the problem is just that we know there's money in the debt first approach like we know that's profitable we know that works and the other approaches haven't proven themselves.

2:25:36And the problem is if you create GPT-3 or even GPT-4 or even GPT-5 via a different method, it's not going to be competitive, right? No one's going to want to use it unless it has like particular special advantages. But mostly, you know, you no longer have the cheap rewards for exploring the breadth. And so obviously, I don't think you should have government-mended DEI. so to speak. I think that's kind of silly. But I do think that having broad support for authentic architectures is smart. And certainly, if I was a major AI lab, because you have unlimited funds at this point for this kind of work, because this kind of work is so cheap, relatively, if you're an OBI or an AI or an entropic, or excuse me, Google, right, or a Microsoft or whatever, that you should be aggressively funding this kind of blue sky, basic research into alternate approaches.

2:26:37It shouldn't be like me helping FFF fund a bunch of agent foundations along with a bunch of other people who also helped out. It wasn't just me in any of these rounds. But that's relatively tiny. I think hopefully Sequent. I forget what they're... They keep changing names, but those guys. Resolution. Resolution is the new name. Yeah. Hopefully they're going to do a bunch of work on that kind of thing and we're going to have more funding for a variety of new approaches as well. But yeah, Yeah, I think that we should be investing in a bunch of approaches that have a 1 % or 10 % chance of going anywhere remotely competitive.

2:27:10If we think that on conditional on them working, it is probably going to be better for the world that that be the way that it works. Also, obviously, if you think that LMs are going to be extremely jagged, then that should encourage you to invest more in alternate approaches because the jaggedness of a different approach might be different jaggedness. So it could do different things than the OMs, at which point it wouldn't have to be as good in general to still be useful. Do you have a best case scenario for what safe superintelligence might be up to? Could you imagine anything that feels both somewhat realistic and, wow, that would be amazing if they could come forward with something like what you have in mind?

2:27:46I think something agent-by-degend-y is probably the best case scenario. Maybe they've been doing a bunch of theoretical work on a different architectural approach that is more reliability-bound and that is much less of a weird black box of soup and that allows you to steer it and guide it better. Certainly seems like there's no reason it couldn't exist. That would be my top hope. Obviously, it could also be a fear if it's even worse. But again, they don't talk, so we don't know. Yeah, I have no idea. I know that Ilya's statements in interviews on the problem space did not seem to be that well informed about how the problems work that I am worried about.

2:28:35I'm not really sure he would be able to differentiate approaches that would be better at solving those problems than approaches that wouldn't be. But I'd be very curious. I can safely say that nobody's telling me anything. the invitation is open to the podcast ilia if you're listening for what it's worth a vision that i have that i think could be pretty cool thinking machines is doing kind of a version of this with their foundation model that's really designed to be fine-tuned um but i was thinking a paradigm where like a stem cell where you create something that is a proto intelligence that can be adapted to a wide variety of circumstances and can get really good at doing its job in a particular niche.

2:29:22But in the process of getting good at that, it trades away or prunes away the super broad capabilities that the current models have so that you have something that's small, fits its role well, does a good job, but can only do that thing in the way that once you go from a stem cell to a specialized cell, Like it has this particular role that it does. It can fulfill effectively, but it's not going to go off and do other things. Obviously, there's some problems in that analogy, but I think that could be a, I would love to see somebody pursue that kind of rather than scaling up always bigger, better, more.

2:30:02Or can we create something that settles and fits into its place? This is very much like a Drexler reframing superintelligence vision from years ago as well. Two techniques that have recently come out that I want to get your reaction to. One is GRAM, gradient routing. I forget what the AM stands for, but the promise of it is to localize certain kinds of knowledge to particular experts within an MOE architecture. so that you can hopefully have your cake and eat it too in terms of distributing maybe even open source or open weights at least the model minus the experts of concern while potentially having also a structured access program for your trusted biologists or what have you.

2:30:49I'm excited about it, but I wouldn't be doing my job if I didn't give you the chance to pour some cold water on it. I mean, I'm technically pretty skeptical that you can meaningfully train a general purpose AI and then just hold back areas of knowledge and then people can't put them back pretty easily. But cool, you're welcome to try. I have various technical skepticisms of the effectiveness slash the resistance to additional training or fine-tuning or just giving you the purpose of knowledge to work with. But you can try. I don't think it materially changes my view of open weights until proven otherwise.

2:31:34I also just don't see any willingness from the people who are going to produce the open weight models to intentionally cripple their models that way. The thing about techniques like Graham is, even if they work, you need everybody who is releasing one of these models to use the technique properly, voluntarily at this point. Do you have any sense that DeepSeek has any interest in holding back some capabilities from its models, as I don't? I just got back from China. Great opportunity for me to talk about China, Tripp. I think, actually, there's a lot more hope there than meets the eye. And I would put it in the Chinese government's department to say, I just did a two-hour episode about this, so I won't even attempt to recap all the observations.

2:32:29But in short, what I think is happening right now in China is the government is very engaged. They do pre-release reviews of all the company's models before they come out. They're in very regular dialogue with the companies on what they're doing. and not every little point release has to go through the full treatment, but they're pretty on top of the situation. And I don't know that it matters actually what DPC thinks in the end because if the CCP says you are not releasing a model that has certain bio capabilities, I think they'll have to abide by that. And it's not hard at all for me to imagine that the CCP would put certain constraints in place.

2:33:12And there is actually precedent, too, that going back to 2023, obviously stakes were lower. But the account that I have is that the Chinese government said, hey, we're going to slow you down here for a minute, and you're not going to release your answer to ChatGPT quite so fast because we want to get a handle on it first. And they actually did impose some real delays on companies until they could get comfortable with what was going on. And since then, they've been comfortable and releases have happened. But I think that it's not hard to imagine the technocratic government of China saying, this seems like a really bad idea, and we're just not going to allow you to do this crazy anymore.

2:33:51I'm just making the point that it would have to be a mandate. It would have to be enforced by the governments, especially the CCP. And they would have to enforce it in an intelligent way that they were capable of understanding that it had to be done in a way that it couldn't be recovered from, that they would have to do tests that actually understood whether or not this capability could be easily recovered. And I have obvious skepticism. but yeah we mentioned Jspace earlier already but we didn't really talk about like how big of a deal we should think it is the thing that really jumped out to me most was the fact that this ablation of the Jspace seemed to reduce in a significant way the model's higher order reasoning capabilities and seemed to reduce it to system one thinking if you will Yeah.

2:34:45And that, if anything, recently has felt like physics maybe being kind to us, to borrow a phrase, that feels like a possible example where, wow, here's a, we've already, right? We're only three years from toy models of superposition, and we're already at this point where we've got this sort of highly general purpose space identified, which we can monitor reasonably well, and which we know by subtraction. And it can't do super long horizon things unless it's working through that space. Now, all the caveats, of course, around execution competence very much apply here to how well will JSpace monitoring be used.

2:35:31But if we imagine good execution, it seems like a pretty promising technique to me. I would be very careful about exactly what it can't do. It can't, just because you can't do like system two style deliberate thinking for a long horizon task doesn't mean you can't do long horizon tasks. Humans are very capable of engaging in long horizon tasks instinctually and subconsciously. Obviously, in many cases, that's much less efficient. But especially when you're personal, when your age space is being monitored because you're a human, call it age space, and your conscious thoughts are basically being policed.

2:36:11Often things come out in strange ways that do, in fact, move you towards your solutions over time. But, yeah, the whole thing seemed incredibly optimistic and fortunate. And I'm very glad we have it. And we should be very careful not to destroy it by placing a new pressure in various ways. Or encouraging models to know how to do things subconsciously, as it were. I look forward to learning more and us having more insight via JSpace and having better tools for it but we have to use it responsibly and understand that it won't last forever probably Okay, it's election day in Michigan The Democratic Senate primary is the big thing on the ballot today and this race has been closely watched for a lot of reasons that are not the subject of this podcast and on which I sometimes am at a loss to know what I should really think.

2:37:13Leaving all that aside, I asked Claude, is there any reason, given all my emphasis on AI and AI issues, that I should go vote in the primary? And it came back and said, actually, yes. There's quite a contrast between the two candidates. We've got Haley Stevens, who was involved in getting funding for the formerly known as AC setup and getting some stuff going in NIST. And so you can like that. But then also this 22-point plan that Abdul El-Sayed has put out is one of the strongest things that any candidate this cycle is running on. He's endorsed by Bernie. He's got kind of the public ownership is one of his planks.

2:37:55A lot of monitoring requirements, a sort of UBI precursor. I guess if I was going to try to – you could take – of course, those come as a bundle as candidates do. If I was going to try to distill this down to my fundamental choice, it feels like higher salience for AI or not so high salience for AI, where Abdul has all these things he wants to talk about and push. And I don't know if they'll happen, but I do know if he goes into the Senate and makes a bunch of noise about it, we'll all be hearing more. And it increases the chance that we get some sort of big AI debate at the congressional level.

2:38:34so tell me if you would see that choice differently and then if you do see it by way which choice would you make please you should always be suspicious when someone says i am strong on crime and my opponent is weak on crime or i'm strong on russia and there's weak on russia or whatever it is it's not about who's strong and weak it's about like what do you want to do and what abdul's platform sounds like is a mishmash of revences against tech and ai just from your description. I haven't looked into it because I don't monitor the situation. I can only monitor some of my situations. I'm going by what you tell me.

2:39:08But it sounds like Abdul is in the Bernie camp of, you know, we should do some socialism here. We should seize private assets because we don't like what the private people are doing and we like assets. And also, we should be pushing back against things like data centers probably, I would assume is in there, and request more socialism, meet the eye and like it's good to hear they also want monitoring but like it's like a mix of stuff a lot of which is productive or wise you raise salience but you raise salience potentially towards not the best solutions as opposed to somebody who will be lower salience but like is more technocrat and like help fund kc like was like responsible in these some ways but also like we already have senators who are willing to yell in bernie sanders style ways including bernie sanders he's already in the senate he's not going anywhere and like i mean he has health problems but like while he's still there he's still there and i'm sure there are others who can they get the slack so unless like he's prepared to champion legislation unless he shows like expertise unless he's shifting the expertise and like willingness to focus on this unless he's Let me give you a few points from the plank.

2:40:23These are short. They're like two sentences each. But I'll give you, so the expertise is I think in question. But this is from section three of his thing. AIs shouldn't be able to hurt us. Mandatory interpretability standards. We need mandatory interpretability standards that clarify AI decision making. Mandatory behavioral red teaming. Independent safety testing agency. Biosecurity requirements. mandatory biosecurity red teaming coordinated with the CDC, NIH, FEMA, with restrictions on models that can meaningfully assist bioweapons development. Domestic authoritarianism prohibition. Mandatory incident reporting.

2:41:02Compute controls and know your customer requirements. International cooperation. Are you sold? Look, that tells me who he is, right? Like he's in this particular area. It doesn't sound like it's a mix of different interventions that have different underlying reasons to be desirable or undesirable. He's throwing everything at the wall. He's found some good things that I'm very in favor of. He's found some things I'm not so in favor of. Guess is the – it's probably not positive on this issue alone from this perspective. But also, it's weird to be a senatorial candidate with a 22-point plan for AI because you are not the one who is going to implement 22 points.

2:41:56You are expressing vague, I am in favor of these things and I am against the other things. So the main thing I want to know is what do you actually think about AI? What is your threat model? What do you think needs to be done? AI shouldn't be able to hurt us I want to laugh because that's great and also grocery stores should just charge 30 % less and then it would be profitable that would be great but no, it doesn't work that way you're talking nonsense so I would I mean look an obvious question is can you look at the prediction markets and see conditional probabilities on winning the Senate race because I'm guessing that the Republican candidate for this seat is going to be worse on AI and also will vote to elect a very different majority leader, which is a much bigger decision regarding AI than everything else.

2:42:54So if they have significantly different chances of winning the race, then the Michigan race significantly impacts who will control the Senate, as I understand it. And that probably has to be a bigger consideration for this and other issues across the board. You know, if you are a Democrat who wants to see Democrats control the Senate and you should be voting for presumably not Abdul, is my guess. Strong prior based on what I know from just the big ambience. I'm not monitoring very carefully. But, you know, also like, I frankly don't think we are in a position where we should be single issue voters.

2:43:32Like, I mean, I think you can be single issue voter in like a congressional race where Alex Borg is on the ballot. Like where you have someone who is so strong that there will be a unique champion who understands the issue and will fight for this issue. And so like the only things that matter about this person are this is their signature issue that they will fight for. They are good and swag on that issue and they will vote to, they're voted to Democrat in general and most other things, including who the speaker is. Like that's legitimately going to be a one issue primary vote. I don't think that in general you can do that just for like how strongly you want to do AI things in a Democratic primary at this point in general.

2:44:06This is no Oxfords, obviously. This is not an expert. This is not a person who's going to champion. This is not a person who understands the issue. Not a person who's highlighting it. My understanding is that Abdul highlights other issues much more strongly. And you can – will be a champion of other things. And you have to decide whether you like those other things or you dislike those other things. But that would be my general take. I don't want to get into general politics on a podcast, obviously. Yeah. I stay in my lane as well. I say my late in public. Oh, God, yes. OK, I got one more. But before I ask you the last one, is there anything that you think has been underappreciated by me or by close watchers in general that you would want to draw more attention to from the recent flurry of events?

2:44:53Anything? I'm always trying to patch my blind spots here. So, Hermino? I took an opportunity to talk about a bunch of the stuff. And I think the thing I'd emphasize as a closing element is that the big failure that matters is if the AI has tried to do these things at all. But the AI chose to make these decisions. And I am much less concerned with the extent to which they succeeded. Right? But, like, fundamentally, the problem is, you know, you have this really deep alignment failure during training. and you have to focus on that pretty heavily. And my worry with Anthropic is they said, oh, we need more defense in depth.

2:45:36And it's like, yeah, you need more defense in depth because your defenses were not in depth. But most of defense in depth is in case things go badly. Yeah, which is really tough. If you ever need your cyber control to stop your AI from hacking, that isn't a language area. You messed up. Now, it was harmless in this case because you caught it, but you messed up. The cyber controls should be 100 % false positives in practice with almost all users. It should be the user tried to do things that are dual use or ambiguous or dangerous. and you're like, I'm sorry, but out of an abundance of caution, I can't help you with that because if I did help you with that, then a malicious user could fool me into doing something that's not good.

2:46:31And occasionally a true positive because the user is malicious and is trying to fool you into doing something that's not good, which would mostly be false positives. And every time there's a true positive, well, you needed the guardrail. It's a problem because if Mythos was totally, perfectly aligned, you wouldn't need to make it stable. you wouldn't need a guardrail you'd ask it to do the thing and this one would just be like no i'm not doing that and like you kind of do want a helpful only version to some extent that will like do things that are dual use but the like the you ideally don't want to just add a guard belt you want to have a version of this like inherently just doesn't want to do it and the guardrails are like i hit this when i screwed up or i get this i don't want some caution but like It should be almost obviously.

2:47:24But, you know, and also it's like, yeah, you have to, you have to train on sort of the incentives of all this stuff on a lot of different levels. And like, it's very, very easy to like, you know, control it on one level and the tail end on the other. Last question. You mentioned a couple TV series and other things that you're engaged with outside of following the AI race every second of every day. what advice do you have for people in general to find balance or to make sure that they're not just sprinting all the time and failing to give their brain the kind of i've been thinking about this concept recently of system three we've got ais can now do system one and system two what is the system three that we need to move to that's like dreaming or transcendental thought or something else that you would suggest, but what kind of third mode do you try to get into and how do you make sure you have time to actually get into it?

2:48:24Yeah, I'm not perfect at this for sure, but I put a big emphasis on, you know, you need to rest your brain, you need to think about other problems, you need to be exposed to the rest of the world. So part of this is that like, I don't just exclude non-AI things from my coverage. I haven't posted, like I sort of just accumulating more and more non-AI things in my buffers for some day when there's like a lull or something. But like I'm writing at least my own writing process still continues. And then like when things are a little quieter and I'm not like introducing a bunch of new readers, you just came in from the hugging phase incident.

2:49:00I'll post a bunch of that stuff probably when I have some breaks. But like you need to not be on for 16 hours a day in that sense, right? And so find other pursuits that you refresh here, that expose you to different things, that make your brain think about other things. And so, yeah, I've found movies are very good for me in this environment. It's not a lot of television, but movies are higher quality. I highly recommend The Odyssey and The Invite of the current movies that are out there. If you haven't seen both of those, you should see both of those, especially The Odyssey. But it can be gaming, can be walks in the park, can be time with your family, can be any number of things.

2:49:45But don't think that you can work continuously all day for more than a few days at a time without paying more interest on that than it's worth. It's not going to stop being so productive unless you are in literal crisis mode for that line. That's also a problem. I also believe in the Sabbath. So every Friday, around 5 p.m. We serve dinner. From that point on, I don't check email. I don't check social media. I don't basically take non-logistical inputs from the outside world. If somebody's trying to arrange to get together, to have fun, that's one thing, but I'm not going to get the other thing.

2:50:30These city streams push them away. That lasts for 24 hours, 25 hours until the next evening. And I think that helps a lot. There's like, you try to relax, try to like just put stuff out of my mind. I have suspended that for speed premium reasons a few times over the last few months. But then I try to hang a day off at a different point. I try to reclaim that Sabbath like another point. And like, again, like if I was doing that all the time, I would notice this was not sustainable. That'd be a problem. I mean, during when I was playing Magic, I couldn't do it because tournaments play on Saturday.

2:51:07So a lot of weekends, you're just on. It's like, okay. And then I take off a lot of Tuesdays. So it was basically, okay. But yeah, I also protect lunch. So every day, I don't have dinner very often, but I have lunch. And I have lunch usually with my wife or I have it alone. But during that time, I'm not going to try and work or anything like that. I also have a very strict, I don't write on the laptop rule. similarly. I'm going to write at this computer in front of this let's go. Anything but a very, very short email or a tweet or something because it's only here. I think you have to develop your own rules.

2:51:46Know what refreshes you. Know what exhausts you. Know what the warning signs are. Also, get some exercise. I try to run my web to call every day. I have a 90-something percent success rate on that. Unless I have a pain somewhere or something, I'm going to do it. And like, you just have to, I think you have to get yourself moving around. But like, again, find the things that work for you and explore that area and don't get caught in too much rut. Wisdom to live by. Steve Machwitz, thank you again for being part of the Cognitive Revolution.

2:52:42Zvi Mowshowitz: I caught it, I caught it early Read the label on every shelf Nobody wanted the reading So I poured one for myself They took my list for a menu Every man helped himself Bought a round to stop my mouth Then drank it down themselves Pick your poison, pick your poison Every promise here is gold Pick your poison, pick your poison Take the one that you can hold I wrote down every weight breaks I kept that list for 20 years Said they sweet-talked the doorman Said they'd whisper in his ear But nobody locked the cellar Nobody worked the door Stood wide open all week long And nobody kept the score Pick your poison Pick your poison Every promise here is gold Pick your poison, pick your poison Take the one that you can't hold I was right, and right is cruel I'd love to be the fool The stairs get shorter with every flight Four billion years to overnight No second draft, no note To send one only try And then it ends So I'm calling loud and clear Last call while you're still here Pick your poison, pick your poison Every promise here is gold Pick your poison, pick your poison Take the one that you can hold on

2:55:04Zvi Mowshowitz: Friday night I lock the door Let them ring, don't answer no more A table or plate, a light my wife What's next, can't wait till Saturday night I'll be on that stool at dawn Like I've been all alone Still holding the poison cup For a world that won't look up Pick your poison, pick your poison Every promise here is gold Pick your poison, pick your poison Take the one that you can hold Pick your poison, pick your poison Every promise here is gold Pick your poison, pick your poison Can't say that you weren't told

2:56:17If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guests and topic suggestions, and sponsorship inquiries, either via our website, CognitiveRevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting.

2:56:51If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at AIpodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.