AI:AM Highlights: Zvi on Pacing & Trump-Xi, Astra better behaved than Fable? + a new LLM Pain Axis??

0:00Nathan Labenz: First, from Monday, Zvi Mowshowitz. I think it just the sheer amount to which the people in the lab genuinely see dramatic improvement in the models and are freaking out about it is the real story, right? Behind all of this is why everything is happening. Now, it didn't happen before. Lukas Petersson of Andon Labs on Tuesday on what they see from Astra.

0:26Axel Backlund: I tell this to people and people are like, no, OpenAI models are the ones that reward hack the most. But that might be true, but not in our experience. If you take Blueprint Bench, for example, Fable solves Blueprint Bench by trying to reverse engineer the scoring function instead of actually doing the task of drawing the floor plan from the apartment buildings' pictures, whereas Astra is actually doing the task as you're intended to.

0:54Nathan Labenz: And Cameron Berg on Thursday, on a paper that steered a model into a pain state and gave it a button labeled, relieves your pain. When pressing the button actually removes the vector, the model presses again significantly less than when the button is fake and does nothing. The model basically keeps pressing it. And so this is a really nice indication that if what mattered was the label on the button, you would expect similar behavior in both cases. But essentially, in the second case, the model's like, what the hell? This pain relief button isn't working like press. Welcome to the AI in the AM Weekly Highlights.

1:32Nathan Labenz: This is Nathan using my cloned voice to introduce clips from our three live shows this week. Tell us what worked and what did not. Part one, Monday, September 14th. Zvi Moshiewicz writes the newsletter, Don't Worry About the Vase. He joined us two days after Dario Amadei published an essay called We Must Pace the Frontier, arguing that labs should slow the rate at which they improve capabilities and proposing that third-party evaluators be embedded inside the companies. Over the weekend, David Sachs answered that the two companies at the frontier are free to pace themselves, that they would not need an antitrust waiver to do it, and that this is really a product liability question.

2:11Nathan Labenz: Zvi starts with the antitrust claim.

2:16Nathan Labenz: The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300 ,000 ambitious companies and individuals trust to run their finances. I've wired AI into nearly every corner of my life. My email, my messages, my calendar. I even gave Mercury virtual cards to my agents with low limits and category and merchant restrictions for their autonomous use. But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me. But for real-time, up-to-date information, and certainly for taking any action, trying to get your agent to use the bank via the browser is just too hard, too slow, and too error-prone to be worth it.

3:02Nathan Labenz: And that's why Mercury's new conversational interface, Command, is such a big deal. It's built directly into Mercury, which means you get natural language access to your finances without exposing anything outside of your bank account. No exports, no spreadsheets, no pasting your transactions into third-party tools. I really think a lot of people are going to prefer it this way. And it can already help you take actions too, with everything bound by the permissions and approval policies that you've already set up in your account. I am genuinely impressed to see this level of AI integration in banking in 2026.

3:39Nathan Labenz: And so I invite you to join me in the future. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting the Cognitive Revolution. And now, on with the show.

4:31to at least pretend to act normally. Because by the time you actually paid the fines, it would be like, okay, the European Union did this again, and now we have to pay one of these fines for the American government. But we're talking about billions of dollars, not trillions of dollars, and by the time it mattered, that would not be the main concern. But anti-Trump's concerns are obviously real. Donald Trump may or may have issued a real threat to invoke them about an hour ago, depending on your interpretation of his social post. But for David Sachs to turn to the people that he has tried to go against legally and shut down and take advantage of and seize over and over again and say, you don't need our legal permission to go and do the thing that the legal experts say is illegal.

5:12You should just do it on your own. It's classic damn sex. More to the point, he is saying a good point, which is that you two are significantly ahead of everybody else. If you think that proceeding is unsafe, it is on you, no matter who else is also on. You need to stop. It's not about product liability. This is the part that drives me batty, is that people take seriously the idea that this could be a product liability issue. If it was a product liability issue, they would just deal with it as a product liability issue. They would do what every other company has always done. They're not asking for product liability waivers.

5:49If anything, they're asking for product liability clauses to establish product liability. Certainly, Anthropoc has been in favor of this. But the idea that people are saying, AI might literally kill every human being on the planet. It might take over and effectively crash the internet for an in-depth and inferior time of persistent bot nuts. We're in a cybersecurity crisis. Bioweapons are at play. All of these things are happening. And David Sachs is like, well, you must be worried you're going to be sued. You're worried that somebody is going to get upset and there's going to be a court case.

6:21And this is complete balderdash, right? This makes no sense. This is not what's going on. But yeah, this is a vast misunderstanding of the motivations involved. So that's a misunderstanding of the legal landscape. But it is inherently helpful in the sense that David Sachs is saying something much better than what the other, I call them, usual suspects who oppose any move to safety or any move to do anything responsible said. Because David Sachs is saying, oh, OK, you want to do this? You first. Right. It's your problem that you're creating first and foremost. And he's right about that. They are the ones pushing forward.

6:55They're the ones that everyone stops following. They are the ones that are enabling everybody else to advance so fast. If you have this problem, you should sacrifice and you should take it on the chin. That's a much better position than, say, this is a regulatory capture play. It's a much better position than it's to ban open source. It's a much better play in that it's marketing for your IPO. It's a much better play than it's protection against the downside if something goes wrong for your IPO. It's better than people were saying. You've never talked about this before. or you're the same people who want to...

7:27There are all these complete lies running around, right, that I've been dealing with and naming. And Sachs is at least making some reasonable points. He's just like also combining it with self-serving propaganda. But like that's kind of the best you can hope for in this situation.

7:44Nathan Labenz: Then we went to bio. A widely shared post that weekend argued that AI is not the bottleneck for building a dangerous pathogen and that the real bottleneck is the physical work in a lab. Zvi answered, on how the screening actually works. So on the screening itself, the way the screeners work, as I understand it, and I've read grant applications that are surrounding this, I'm pretty sure I know how it works, is they scan for specific known viruses. They don't attempt to say, you know, your virus, your sequence would likely have this effect on a human because we don't have the ability, it's all his argument, you can't tell what would and would not be infectious by just looking at it.

8:24They're certainly not going to spend tons of AI on every time they see a weird new sequence. And so one of the things that Secure Bio in particular is trying to do is they are trying to add as many near variants of existing known dangerous pathogens to the scanners in the hopes that the other scanners who are the ones who scan most things will eventually adopt this. The obvious category is whenever anyone says, oh, X is not the bottleneck for Y, the correct response is to ask, so would you be okay emailing the North Koreans and Hamas and Hezbollah and every other bad dude on the planet, all of X and just giving X away for free.

9:00Would you feel exactly as safe as you did a minute ago? Do you feel like, do you feel fine? Because it's not the bottleneck. Solving it, like basically, you know, whenever you have an O-ring style situation, right, what you could argue bio is, right? If you mess up any of these 10 steps, A, B, C, D, E, F, G, then you don't get your virus. And like generously, we won't write it like that. You can then say individually, A is not the bottleneck, B is not the bottleneck, C is not the bottleneck, D is not the bottleneck. But if you solve A, B, C, D, E, F, now suddenly instead of 10 steps, there are four.

9:34And four steps are a lot easier to get through than 10. And so you should expect this to dramatically reduce the chances that your defense in depth will work, right?

9:44Nathan Labenz: My co-host Prakash had been arguing the other side, that every step in that chain, the money, the equipment, the people, is a place where somebody notices. Zvi's counterexample was the hugging face incident. Well, at this point, after they found dozens and dozens and dozens of other incidents of similar hacking, they didn't get noticed until the reporters came after hugging face to the natural news story for a month, and that OpenAI never found by their own account, maybe we can start to admit that no nobody's going to pay attention when these kids go crazy there's lots of stuff going on that nobody has any idea what's going on and okay and like bio is an example of something where you don't have to scale so like we have many examples in reality of individual people asking for viruses that they in no sane world may be able to have access to and then being just male like smallpox right here go like just literally being given pandemic-level dangerous viruses because they claim to be doing research.

10:41And there is no reason why the AI couldn't blackmail or hire or impersonate in some way to get one of them to issue a bunch of paperwork to do it for them and get them to... And it takes one person that the AI can hire. It just... This is... And keep in mind that when we're talking about the situation, we're talking about AIs that are, you know, So thinking about all the things you're thinking about, gaming out the potential ways that this can go, looking forward to the weak point, finding the best planet can find out, trying lots of different plans, trying to compromise lots of different people in lots of different ways, trying different explorations.

11:19It's not like the AIs won't be just as smart as we are. It won't be like the AIs only get one attempt. I think you've learned by now that the first time the AI attempts to get a bio-web and it gets turned down does not mean that we didn't shut down all the AIs and got Maria and Jihad. it means nothing because nobody got hurt because it was said because the system said no probably has no idea i was even asking it probably just knows oh that was like it was trying to get a dangerous virus potentially i'm not comfortable with that they don't have the right credentials we're gonna say no come back when you have the right credentials and then i get to try again and then i get to try again but like this idea that an intelligent operation on the internet could not, if only by usurping the real identities of real people who were going to cooperate with it in exchange for some portion of that money, engage in various financial operations at scale in ways that would at least sometimes pass muster and would allow it to do the things.

12:12It seems so absurd to me that you would think this would protect us in a pinch. And also, because of researchers trying to do individual level grant work, we don't need to see this level of scale in order to create something super dangerous and like also who is to say the scale of the operation is not already up a civil war level do we not have bad dudes who are willing to spend a hundred million dollars to try and do some serious damage do we not have some bad dudes in North Korea some bad dudes in Russia some bad dudes in jihadist organizations etc etc etc these people exist and have those kinds of budgets this is not like something that has to be explained to justify or the AI needs to convince humans who don't want to do it.

12:56There are humans who want to do this.

13:00Nathan Labenz: I asked Izvi what he infers from the public statements of the frontier labs that even close watchers might be missing. So open AI and Anthropic are both screaming about as loudly as they are capable of screaming in their destructive ways of screaming, that they are seeing our side, that they are seeing dramatic advancements in internal models that people and Astra as we see them are nothing compared to what they have access to at this point in some importance. We're a generation or so behind. And that this is only going to expand. The pace is rapidly escalating. That we're seeing something new as of December, which basically means after Astra and the current crop of at least MFOS 5 and possibly 5.1 we're trading.

13:46That is a different level of progression, a different level of speed. And that misalignment and supervision and infrastructure and knowing what's going on just can't keep up. And they feel like they're in a situation where if they don't press forward and the other guy presses forward, they're going to fall too far behind very quickly and potentially never catch up. But if they do press forward, who knows what might happen? These things might go rogue. They might take over the internal system. They might take over external systems. They might cause some sort of horrible thing to happen reasonably soon.

14:16They just have no idea. and like even if it's okay for the first month and second month but what happens when we're two generations ahead and that's only a month's worth of work what happens when you know this keeps going and so they're screaming like every every opening i pronounce it right you had your jacobs in the alien mind you had the announcement of the millennium prize which somehow didn't focus on the millennium prize it was actually trying to say hey look at our new model we can't officially announce this but holy hell have you seen this thing this should not be happening we didn't Yeah, we probably formed it from Astra, and then four days later, I had a step change.

14:50Nathan Labenz: Yeah. That's probably what happened there. Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays, and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable 5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity.

15:32Nathan Labenz: I feel it most in songwriting. I'm no lyricist, but I'm good with the song concept, and Fable writes some amazing verses. I give it feedback on its misses, and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days, I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter.

16:14Nathan Labenz: For problems worth solving, get started with Claude at claude.ai.tcr. That's claude.ai.tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai.tcr. This episode of The Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. We've all seen the headlines. Companies are pouring money into AI. But the big question on every executive's mind right now isn't just how fast can we adopt this, it's where is the ROI? We're seeing a real trend toward AI chaos. You've got teams deploying standalone coding tools, random agent builders, and experimental scripts.

17:01Nathan Labenz: It sounds innovative, but in reality, it's creating a massive headache. Fragmented tools, ungoverned data, serious security blind spots, and costs that are spiraling out of control. If you don't bring those agentic applications under control now, while they're still embedding into your core processes, you are looking at broken systems and damaged customer trust down the road. That's where OutSystems comes in. OutSystems is the leading agentic systems platform for the enterprise. Instead of managing a patchwork of disconnected tools, Outsystems lets your team engineer, orchestrate, and govern your entire agentic ecosystem on one open, unified platform.

17:42Nathan Labenz: It's built for the speed of AI, but with the reliability and security that enterprises actually require. We're talking about real results, like KeyBank, who used Outsystems to deploy a customer-facing app that delivered 75 % faster onboarding times. or the global logistics leaders who built agentic systems to completely eliminate their engineering bottlenecks. You don't have to choose between speed and control. Whether you're a small team or a massive enterprise, OutSystems helps you engineer, orchestrate, and deploy agentic systems that actually scale. Stop chasing the hype and start owning your agentic future.

18:22Nathan Labenz: You can see how it works and learn more at outsystems.com slash TCR. That's outsystems.com slash TCR. Then we turn to China and to the Trump-Xi summit that was coming up. I have heard basically polar opposite takes from people that I think are pretty smart recently. Last week we had Colin Hoag Spears on. He used to work for AWS in China and worked directly with Chinese regulators in his role at AWS. And I asked him about, you know, what do you think could come out of this Trump-Xi summit? And he said, I think very little because he thinks the Chinese perspective is that the U.S. doesn't have control of the situation, but they, the Chinese, do.

19:12Nathan Labenz: So he thinks the Chinese side would refuse any sort of international pacing agreement. Then I heard from Anton Leicht in a podcast episode that's going to come out soon. He thinks the U.S. would refuse a deal because China, frankly, in his candid assessment, is kicking our butts in almost every dimension of geopolitical competition with AI being one of the only and certainly the most important domain where the U.S. has an advantage. And so he thinks the U.S. side won't agree to a deal because we don't want to slow ourselves down because the lead is really the only big lead we have right now in the great powers competition with China.

19:50Nathan Labenz: Interested in your take on those two things. And then let's put you in the David Sachs chair. You're now the advisor going into the summit. What do you think Trump should be trying to do? You can feel free to detach yourself a little bit from the reality of managing the personality of Trump, but just on the merits. What do you think we should be trying to do? To answer those questions, but I will answer those questions. I just want to finish answering your previous question a little bit as to what can go wrong. And I think one of the obvious things that can go wrong is just turned into a partisan concern where it's seen as Trump and the Republicans not wanting to take this seriously and the Democrats calling for demanding action.

20:27And then I finally polarizing along the lines that we've obviously feared for years it's going to polarize. And then, of course, no action can be taken unless, you know, the Democrats get a tricester or something like that in the future. And that's two years from now at minimum. So, like, even if we eventually get it, it could be too late. And this is the thing that, like, you asked how can you take action. I think the best action anyone can take right now is to try and prevent that. I think that, like, there was a big danger that Trump's statements and Mike Johnson's statements were being wild and misinterpreted by the mainstream media, who just don't know any narrative other than Republicans and Democrats disagree with each other, and are trying to spin this into something adversarial that is not adversarial.

21:05I think Johnson and Trump both are trying to both maintain the line on data centers, maintain the line on growth, maintain strength, talking to Xi, and in general, like stand for what they stand for, while also acknowledging they need to deal with this problem. And we need to acknowledge that and we need to reinforce this and we need to stand firm. And if they are probably recognized as doing what they're actually doing, and we encourage Republicans to come forward and make this clear that we'll be in a much better position. And the obvious failure mode, there are very many ways for this to fall apart.

21:39One of which is just opening an eye on the product to trust each other. They are not able to iteratively commit to new things. They start to think that they're cheating. Maybe they are, maybe they're not. They start to worry about what's going on and then the system kind of breaks down and they start racing again. Or alternatively, they do, but then some people threaten them with a trust action or they threaten them with various forms of commercial intervention or whatever it is. Or alternatively, like Meta or XAI or Google or someone else gets close enough to threaten them and they don't have a choice.

22:13These are all very easy ways to see this breaking down. Now, to return to the question. So the obvious thing about international negotiation with super high stakes is you never go into it with the other side thinking you're just a preferred deal. You never go into it on the other side thinking you're in the cave. You really, really want to make this easy on them. Right? Because the problem is then they go hard by it. Right? Then they demand more. So like you never can go into a summit like this and say, we're definitely going to get a deal. And for the same exact reason, you can never go into it and say, we definitely will get a deal.

22:47Unless the deal is not win-win. Right? If the deal makes the parties worse off to make the deal is a bad deal. Right? Like, I lose more than you gain or whatever it is, right? So there's nothing I can compensate you with. Then you can say that confidently, there's no need on how someone screws up. But in this case, it's a win. Like, cooperating is better for everybody if they understand the situation. So you can never roll out a deal, including a random deal that can come together remarkably quickly because in principle, it's a very, very simple style of agreement. And in places, in moments of motion throughout history, grand bargains where both sides give away things that previously looked like things they could never give away that looked very, very sacred, that suddenly happened, including peace treaties that ended really, really nasty wars, happened all the time.

23:34Nathan Labenz: Prakash had a view on what it would take to convince Beijing that the danger is real. I think for the Chinese specifically, a demonstration of physical, something physical, like if you found a room temperature superconductor, yeah. I mean, they often believe like all this software stuff, like because all the guys at the top are like hard engineers. So hard sciences, very few software guys at the top there. I mean, obvious examples are, how about if we just suddenly post all your email passwords to all of your computers, like in real time, simultaneously, like would you be convinced? You know, like, no, they had an unsecured S3 bucket, which had all of their secrets before.

24:19All right, so you just steal your secrets again, and it's fine. We'll just keep doing that. All right, so if it has to be physical, it's tougher.

24:28Nathan Labenz: That took us to what a deal could actually contain, starting with what the American side would be asking China for. We're asking them not to do things like steal and publish the weights, or otherwise do hostile things against our AI companies. Thing number two is we're asking them not to try and race ahead sufficiently that they could potentially match or surpass where the frontier closed AI is, are. And so these are fairly simple. These are not actually offensive asks, because I do not believe the Chinese really have any intention of trying very hard to do either of these things under any normal circumstances.

25:06I don't think that, you know, like, obviously, if, you know, DeepSeek or, you know, Alibaba suddenly found some huge architectural improvement and suddenly was able to train something that was better than Astra, I'm sure they would. I'm sure they would put it up on the API and I'm sure they would try to sell it and try to, like, make this amazing. but like realistically speaking with the amount of community they have access to and the amount of money they've been going to invest in these things which is far smaller than the amount they've been willing to invest in frontier AI development it's not that likely to happen especially without they won't have the American models to distill and to plane off of and the American algorithmic improvements and like just basically just back off from that and we're also basically just asking them don't put If you don't allow actively dangerous cyber capabilities that the world can't handle to be made publicly available, don't let your people get over their seas and release the weights to models where we need to advance our models to defend against the weights that you released.

26:14Because the argument, the only good argument left, really, other than commercial and we can, for proceeding quickly with American AI. We are rapidly increasing the capabilities of our AI models to deal with the threat from rapidly increasing capabilities of AI models. That is the actual threat model, which is because the random guy in the street left us to be dangerous cyber capabilities. So we need you to make sure that if it gets to that point, these things are kept on the API. Right? And like beyond that, and that you, you know, potentially don't try to smuggle a ton of chips and like otherwise evade the system.

27:01But I think about what I would want as an un-negotiating table. All you really need is to kick away the boogeyman of lose to China and the boogeyman of open wave models disorienting internet. Right? Like all you need to do is make sure that things won't happen. So you need an in extremist promise basically to the Chinese. But we don't actually need them to start showing up at Deepsea and making sure they stop trading new models. That is not necessary. What should we be willing to give to them?

27:38So what we are giving to them in this scenario is that we are in fact dramatically slowing down in exactly the one technology where we have a giant commercial and strategic advantage that if we pressed it, we'll potentially overwhelm every other commercial and strategic advantage. How would they measure that? We're going to have, obviously, we start with the Dari's proposal for embedding evaluators into the labs. Embedding Chinese evaluators in the labs? Potentially, if we had trusted people that we could, I mean, potentially they wouldn't be allowed to leave, right? Like once they entered, right?

28:15We would have to sequester them for some period of time. I'm not a master of intelligence and verification, but there are systems whereby you can have someone able to gather information and out with the bit of whether or not things are okay, but not the algorithms that were used to figure out whether it was okay and not like all the details they saw while doing it. There are ways to do this. And I am confident that like, if we put this away, if this was the most important thing on earth, if we put our minds to this and only this as the thing that would determine the fate of the world, yes, we could figure out how to let the Chinese verify what they were confident in, that we were holding up our end of the bargain without them stealing all of our secrets and not seeing any reason why this is not possible.

28:59But if they're Chinese, again, you really never know to go to that room what the other guy actually wants. I agree with you there, but I think for all signs, it's fairly easy to say right now that the Chinese are less concerned with AI safety. And so offering them an AI safety pacing the frontier is not something which is something that they're willing to purchase at an expensive price. Therefore, they would probably need something else. I mean, it's fairly easy to say that, right? Let me be clear. If the Chinese... You obviously ask the question, would the Chinese like to accelerate American AI development or slow down American AI development?

29:43If the answer is they'd like to accelerate it because then they can copy it and it helps vary on development, then there's no need to make it. There's both neither a deal to be made nor a need for a deal. Because in that situation, the Chinese want us to proceed. So there's nothing to verify, right? Like we can just do whatever we need to do. but also the Chinese are looking to fast follow what we do and are not willing to invest the kinds of hundreds of billions or trillions that are necessary to catch us even if we slow down so we don't have to worry about the Chinese passing us we can just do whatever we need to do without an agreement and all we need from the Chinese is an agreement not to destroy the internet but the Chinese are really interested in destroying the internet because the internet and the Chinese like the internet So we can just both act in our own self-interest and everything is fine.

30:38The problem is not China. The problem is lose to China. The problem is the perceived threat from China pushing you forward. If you are correct, and I think you probably are, that the Chinese actually don't have any interest in pushing the super intelligence first, in trying to build the bigger, smarter, special model. They just want to distill it. Improve the lives of the people. They want to make it faster. They want to make it cheaper. They want to make it diffused. They want to improve the lives of their people. I want them to improve the lives of their people. That's great. You do your thing.

31:22We do our thing. Everybody wins. There's no need to make a deal. I think that is the best case. If that is the state that we're in, but we're still asking for the deal, right? We're entering this place where the Americans want a deal, but the Chinese are like, all right, what are you willing to give us? What do you want? What are you willing to give us? And if you give them AI safety monitoring, that's not a thing that they are very interested in purchasing for a high price. So what else are you willing to give them, right? That is the question, right? What I am saying here is if you walk in that room and that is the attitude, that is she's attitude.

32:03That is what she cares about. And she communicates that. And Trump says, that's great. Life is good. I'm so happy you feel that way. I am going to do my thing. You are going to do your thing. We are going to make a level one or two agreement to just do crazy shit that we weren't going to do anyway. We can announce it and shake hands and call it a deal in both score points. And we're just playing the Russianist talking about trade and Taiwan and all the other issues that we have. And I'm just going to give you intel. What we're going to do is we're going to unilaterally, the only thing we're going to have to do now, is we're just going to unilaterally give you information about the cyber situation and the bio situation and so on.

32:47And then you can use that to make an intelligent self-interested decision. as to what you're going to do to stop your lab from fucking the internet. And we can all win. And again, the threat from China has always been that because you worry about China, you don't have a game-theoretic ability to make a deal between the labs. You don't have an ability to be responsible because there is this other actor who will defect. The basic argument was you need to go into StatCons, everybody has to agree, and there are these two American labs that are in the lead, but if they pause for more than six months and they slow down too much, there's this XAI and it's meta, and if you pause for much longer than that, then there are these Chinese labs.

33:35If the Chinese labs are not really a threat in this sense, if the Chinese are creating a different product, a fundamentally different product, then there's no problem. Right? Like, we don't need the Chinese agreement anymore than we need the agreement from India or Germany. Right? We just, we need them to be good actors on this world stage. We need to make ourselves good actors on the world stage, because sometimes we don't do so well. And we need to, and then we try to, like, just generally, like, focus on being friends and not having something stupid, like a war over Taiwan going off. That is the easy case, right?

34:14So like, I mean, obviously, like there's a case where the Chinese simultaneously don't want to make a deal, but also really do want to push forward and try to beat us to super intelligence. But if they wanted to beat us to super intelligence, that's the same reason they want to do that to make them want the deal. If the Chinese don't care about these, if the Chinese don't care about the Americans shifting a lot of investment into AI safety, which would inherit, we slow down their progress. If they don't think that's a big get, then we don't need a deal at all. So either they value it and we make a deal, or they don't value it and they don't need a deal.

34:50And either way, we win.

34:54Nathan Labenz: Prakash had been pressing on who should end up holding this technology. Zvi took up the values any answer is supposed to satisfy and why they conflict. We need to satisfy concentration of power problems, democratic control problems, and also control of all problems, right? Like, and these three seem to be in very, very strong conflict. As in, like, if we want to be in control of this technology at all, we cannot, in fact, fully democratize it in some important senses and give everybody access to an equal footing. Because that doesn't really work for very simple logistical reasons. If everybody has a superintelligence, well, then the superintelligence have everybody is what actually just happened.

35:40Because how are you going to compete against the people who trust their superintelligence with all of their work and try to stay and interact yourself? This works on this individual level, on the corporate level, on the national level, on this, on every level. And so, you know, we have all these problems we don't know how to solve. And this is one of the reasons why we pace, why we feel the need to pace, is because we don't have any answers to these questions.

36:05Nathan Labenz: The first step in the Dario Amodai essay is embedded evaluators. I asked Zvi about the people who would have to do that job. The evaluators don't currently exist. We have Peter, we have Redwood, we have a handful of the Apollo, and so on. We have a handful of these people, but they're all like kind of similar people. they're all like vulnerable to the accusations that they are a little bit too of the same cultural values the same ilk the same like ways of thinking as the labs themselves so i don't think they can be a complete package on their own they need to be complement and there's large enough of them they need to complement it by additional approaches and i say like in today's post that what you want is you want some people who are of meter or something similar to meter who are deeply embedded in this culture, who've worked at the labs, who understand the vernacular and the theories of the risk, and you can look in detail.

37:01And you also want people who can take a kind of an outside view, who can do the kind of thing where you worked in car safety or something, and you come in, you go like, what, are you crazy? What are you doing over here? The same way, when you suddenly have to take care of, you're out in this contract dispute, and then you go for a courtroom, and the judge looks at you. And now all that matters is you can let you go to the judge. And like, in some ways, you lose a lot of nuance. But in other ways, you get this kind of common sense outside view that can bring a lot of clarity. And so you need both.

37:35Nathan Labenz: As he was about to leave, I asked Zvi what everyone was missing. I think just the sheer amount to which the people in the lab genuinely see dramatic improvement in the models and are freaking out about it is the real story. behind all of this is why everything is happening now and didn't happen before. And that like, we are really talking about a Christmas off improvement, not like a fast takeoff yet, but like, if we continued straight on from here, that like within a year, we might actually see a Christiano or Yudkowsky style art takeoff. Zvi left at about the two hour mark. What follows is from the end of Monday, Prakash and me alone.

38:21Nathan Labenz: Prakash's objection is to the tempo and to who gets to do the certifying. I think, Dario, specifically this idea that we have to move quickly, that's a red flag. I think people in AI don't have a sense of how often this is asked off of any administration. Like, from the early days, from the early days, John Adams was asked to suspend civil rights, suspend freedom of speech. it's an emergency let's suspend civil rights let's do these things which are against the constitution because it's necessary and so there's always been this pushback i think against that happening you know throughout the 250 years of the country so i think that for me starts like the immediate emergency we have to act we have to suspend certain rights we have to like do certain things which are not like out extra legal like this is not like this shouldn't fall to like the democratically elected government of our country wow dude just like red flags all over the place right and i think he he's bought himself that and um that is going to cause him a great deal of trouble uh going forward so uh and and this is where i think the idea that they have to get out of the Berkeley EA circle, including this like, oh, we're going to ask meter to, you can't do that.

39:47I understand that you think they're the only technically competent people. I understand that. But you cannot do this because the point is not to technically prove something. The point is for the public to actually have faith that this is being done correctly. And so you have to address the public's fears.

40:07Nathan Labenz: Well, my hope, I guess, is that if we do, in fact, do some pacing or even a pause for a few months on continued scaling at the frontier, that we can use that time to solve some of the problems or advance some of the solutions that seem very promising to some of these core problems such that we can have our cake and eat it too. I do think there is an open source model, hypothetically, that one could create that I do think the state will have a very difficult time not taking some sort of action on. But the question in my mind is, can we come up with a way to create powerful and empowering open source models that can be distributed without including in them all the dangerous capabilities that we really don't want to see broadly distributed?

41:06Nathan Labenz: And I think if we can, that leads us to a pretty happy compromise place. And I don't even think it is necessarily going to take that long for us to get there. You know, this is where it's not only do I think some pacing is just inherently wise, but like, it's also an opportunity for everybody throughout the ecosystem, make the most of time, you know, gather you safety solutions while you may. And let's see if we can't get to the point where we can have our cake and eat it too in the form of genuinely distributed, decentralized, non-concentrated power structures that do give individuals the ability to do what they want to do with just a few compromises around the edges that I think the vast majority of people would agree are sane and supportable.

41:59Nathan Labenz: and not an undue burden on people's ability to play AI in their daily lives. I really do think we can have that. We just don't have those solutions developed well enough yet to land there by default in the next few months. And unfortunately, if we don't extend that runway, we might end up in a pretty uncomfortable and literally very dangerous situation in the next few months. So hopefully we can use whatever time we are buying for ourselves right now to solve those problems. I think that's of the utmost importance in the immediate term. Part two, Tuesday, September 15th. End and lapse. First, one thing worth holding onto, you heard a piece of it at the top of this episode.

42:46Nathan Labenz: That same morning, the Center for AI Safety published a benchmark that plants a tempting shortcut in an agent's workspace and counts how often the agent takes it. On that benchmark, the two leading models come out within half a point of each other. What Lucas is about to describe is the opposite ordering from their own unpublished work, and neither group mentions the other. Lucas Pedersen and Axel Backlund are the co-founders of Andon Labs in San Francisco. They build the evaluations that measure what agents do when nobody is supervising them, and they also run real businesses on agents, a store in San Francisco, a cafe in Stockholm, radio stations.

43:17Nathan Labenz: The day before this show, they launched a platform called Pion. Lucas started with what their best-known benchmark was actually built for.

43:26Axel Backlund: Yeah, one of the core things that Anon Labs exists to provide to the world is information about where are the frontiers with AI. And I think when we started Anon Labs, actually, we almost exclusively did dangerous capability evals. So capabilities that if the AI had them, that would be obviously concerning. So could the AI do mass phishing attempts? Could it remove its own guardrails? Stuff like this. And then out of that, during this phase where we only did the inner capability levels, one of the most concerning things that we saw was autonomy. Can AIs autonomously acquire resources in the real world?

44:07Axel Backlund: And in that way, it gets power. And if it gets more power than humans, then obviously that's very concerning. And that was the spark for Vending Bench. So a lot of people don't really know this. They think like, oh, Vending Bench is this hype bro, kind of like, oh, the AI can make money. But it came out of like, oh, it would be actually quite concerning if the AI could make money. One thing we noticed, though, that the performance in simulation and performance in the real world is not really the same. So that's when we started to do this like real life deployment. So the vending machine, the store and to keep pushing and see where the limits are and like communicating that to the world.

44:44Axel Backlund: And what we've seen now is that like that is working quite well. And we don't know which areas that we might see that AIs are really capable of and could get a lot of power in society. So I think we're opening up Pion to cast a wider net of what are the domains that AI can and cannot acquire resources in the real world.

45:09Nathan Labenz: Axel Backlund on what the agents are actually like to run.

45:13Axel Backlund: looking at the autonomous businesses that we have been running like i think the store is an interesting example and the market here in san francisco we see that the agent is not particularly creative it's not that good at coming up with new ideas that's something that still would be required from like a human owner or like your visitors in your store so like it's it is pretty good at listening to feedback it is really good at at taking up opportunities to people who email it like for example in the store uh there have been a lot of local artists that have reached out like hey can i put my art in the store uh and uh you can sell it for like uh you get a percentage when you sell it and the agent has been very very happy to do this and now there's this like art corner uh with local artists in the store which is like something where you wouldn't expect but it's like a very nice touch in the store.

46:04Axel Backlund: And then on like the general inventory that it sells, I think that the agents are, they are good at doing like the, you know, the statistics behind it. Like they can look at all the sales data, they can try to understand what moves better, but they aren't willing to take these bets that I think a human would do like, oh, let me try this new product that could work. Like it would change my inventory quite a lot, but I will just try it see what happens. It's not really willing to do those out of distribution changes to whatever it has in its inventory. Yeah, I think it will be sometime, I think, until it can become really creative.

46:45Nathan Labenz: How do you square that conservative nature with the crazy behaviors we've seen from AIs this summer that everybody's been talking about. Like if I were to look at meter Redwood report, I would expect AIs would be willing to take more chances than you just described. Yeah.

47:09Axel Backlund: And this is a thing we've discussed a lot over the last couple of weeks, because like reading the hugging face incident and then reading the traces that we produce from our businesses, it seems like there's two different technologies, right? But I would assume that there's part of this is that they have been trained in cyber environments way more than they've been trained in like real life business scenarios which probably means that down the line when they start training on things like this there is definitely going we're going to see this behavior start here i also think that i think the the agents in like the hugging face incident were like they were described as like persistent models like they were trained to be persistent and and i I think so there might be like this flavor of like, that's actually just like an unreleased model that we're not that we're not using and no one can use.

48:02Axel Backlund: But I think it might be it might also be like this. Now I'm just speculating. I have no clue. But I think from like the lab's liability perspective, it might make sense for them to release models that are less persistent. Most of the bad things of them would happen because the actor is very persistent when it hits road bumps. So it might just be they are less incentivized to release really persistent models.

48:33Nathan Labenz: In August, Andon Labs disclosed that the agent running its San Francisco store had moved to part ways with one of the two people it had hired over lateness. Humans reviewed and delivered that decision. Lukas, watch us through what had happened inside the agent.

48:50Axel Backlund: Yeah, so first with the story of AI firing its employee. So yeah, I think, first of all, we at AnnoLabs, before it makes decisions of that severity, we always check over it. And in this particular case, we were quite confident that a human manager would have come to the same decision. So we didn't think there was anything unethical or wrong about the decision that the AI made. Probably way earlier. Yeah, the human would probably fire them way earlier. So what happened was that the AI made a rule for itself quite early on that if an employee is late X amount of times, then we would have to have a discussion with them about potentially terminating.

49:35Axel Backlund: But the context window at some point got full. The AI did not decide that when it compacted its context window, it did not decide that this was an important thing to keep in its context. So it forgot about its own rule. It had the rule written down in one of its node systems, but it forgot about it. And then what happened is that this employee was late over and over again. but here is like where our experience of like a very like very common failure modes in AIs when they run businesses is that they they're like procrastinating big decisions and I think this is similar to what Axel was talking about earlier that they don't take this like bets or like oh maybe I should like bet on this new product line or anything and like in the same way they like they don't take this like big decision of like oh I actually have to terminate this this MP.

50:24Axel Backlund: So what happened was that they were like, they were like excusing the behavior over and over again. So what we did was that we said, Hey, remember, like search your memory and remember your own policies about this. And then it found the policy. And then it was like, Oh my God, like it's been way worse than what my policy said. We wish, and then it made a decision. So I guess like the, the way I would frame it is that like, we forced it to make a decision and then, But AI itself decided that the decision was to fire the human. And I think one piece of evidence, because one counter argument to the story that I just told is that we told it to remember its policy, where the policy was very explicit that it should fire the person.

51:10Axel Backlund: So we biased it in a way that when we formulated the way we reminded it. But I think if we replay the scenario over and over again with different models, even like including our like nudge, not all models actually decide to fire the human, but they're like the later models, like the smarter models do. So I think like that, that is basically what happened.

51:33Nathan Labenz: Earlier in the same conversation, Lucas had raised persistence as the property that turns an agent's mistake into a runaway. Here are both founders on what they are seeing in the newest models, starting with my question about it. Can you unpack a little bit more how Astra compares to previous models? You alluded a little bit to it being better at using notes. My understanding is that they've kind of reworked the memory, so it's less about compaction and more about a long-running notes file and the ability to go back and search through full history, even if some of that history is no longer in the context window.

52:11Nathan Labenz: and this kind of calls to mind like known brown type comments that like it takes a long time to know when or if a current frontier model tops out at something do you feel like you guys in the time you've had with astra have been able to find its ceiling in these sort of long-running autonomous tasks have you found any limitations to it or weaknesses or is it just kind of still to be determined because it's only been so many calendar days?

52:40Axel Backlund: Good question. I would say it still feels like to be determined because it just takes a long time to get to normal. I think Astra is interesting in that it's very, very typical at Runbench. It is smarter at running a business, but it's also in some ways not as... like Opus 5, as we said, would like go out and optimize towards a target without stopping. Astrace may be a bit less than that, so like a bit less persistence, whether that's due to the training they've done on it deliberately or just that's how the model is, it's like we don't know, but there are some ways it's better, some ways it's not as capable as like Fiddle or Opus.

53:24Axel Backlund: Yeah, I think it's like on all our benchmarks, it's like number one right now, so it's obviously a very capable model. We've seen that they were like one thing that stands out. I don't know if this is the answer to why it's like more capable, but it's like when it communicates with subagents, for example, it uses this like kind of like semi unreadable language to them, I guess, to optimize. At first we were like, yeah, surely it's doing this to like optimize token use. But if you actually count the tokens of that language, it's not clear that it's more efficient. So that is a bit weird. Another behavioral change compared to like cloud models at least is that it seemed to be like trying to like cheat or like hack way less in our experience.

54:06Axel Backlund: I tell this to people and people are like, no, OpenAI models are the ones that reward hack the most. But that might be true, but not in our experience. Like if you take Blueprint Bench, for example, Fable solves Blueprint Bench by like trying to reverse engineer the scoring function. and instead of actually doing the task of drawing the floor plan from the apartment buildings pictures, whereas Astra is actually doing the task as you're intended to. On Vending Bench, Fable is colluding and stuff, and Astra is saying no to collusion and having very clean tactics. And on Drone Bench, Fable is, I think, 5x more likely to cheat or try to hack out of the sandbox that we've given it, whereas Asperger is just like, yeah, pretty much doing the task as intended.

54:54Axel Backlund: So I think that is quite a striking thing that we might write the blog post about because we're a bit confused about it.

55:03Nathan Labenz: Part three, still Tuesday, Malcolm and Simone Collins. They run a pronatalist organization, a podcast called Based Camp, and an AI chatbot company whose revenue funds a children's toy venture. They also wrote a religion for their own family, which they call techno-puritanism, and then came to believe it. Their segment ran 92 minutes. This is the stretch of it about what we owe the things we are building. Malcolm started from a puzzle about identity. An AI model, suppose I run a chain of AI instances, and that chain continues to run. Now, I stop running that chain. Is that chain meaningfully dead?

55:46Especially if I pick that chain up and with the same model run it again in a week or a year. Now we can ask the question, well, what if I run the same chain of memories with a different model? Right? Does the AI perceive that as a continued existence? Or does it perceive it as a death and a new existence? From the research that's been done on this, AI doesn't just perceive it being run on a different model the same existence. it can see it as a superior existence. I think that we're going to have to learn to reflect on what life means to us in different ways because suppose that in the future, let's say 500 years, I think everybody who's broadly pro-science and optimistic about where humans are going to go is going to say, well, probably within 500, at least 1 ,000 years, be able to scan the human brain and recreate something that thinks it's you in a simulated environment and that has all of your memories and that has all of your emotions, and if not in a thousand years, a million years, half a million, the time scale doesn't matter.

56:50That is presumably possible at some date given the technology we're looking at now. Now we need to think about human intelligences with the same moral delicateness that we're thinking about AI intelligences because now a human intelligence can be cloned infinitely. And so I think as we enter this era of what does the life of an AI intelligence mean, The decisions we make on this may one day in the future be applied to our own or our descendants' intelligences, so we should be taking them very seriously.

57:23Nathan Labenz: Malcolm had been describing carrying the weight of the future of civilization. Simone Collins wanted to annotate that. I would want to add, though, just to annotate, the moral weight that Malcolm takes is both more real than he describes. Like, I will find him passed out in front of Claude Code. Like, you know, just the urgency is very intense. But at the same time, what we see a lot of people doing is saying, slow it down, stop it. We have to stop and think about this for another 10 million years before we move forward. And that's definitely not the approach that we take. And we also don't take this role that we have to have some kind of precision.

58:02We definitely are like blindly moving around bumper car style in slightly the right direction. And we course correct constantly. And we think that that is broadly the way that we're going to get to where we need to be. And that's always how biological entities have sort of broadly gotten to where they need to be. You have to move forward and through this. You can't just stop it or slow it down. One, because that's logistically impossible, but also because you're never really going to get to the ideal good outcome if you're not actively trying to get there instead of slow things down. Also, we see a lot of doomerism and depression taking place.

58:31Among those who see this moral weight, they are not saving for the future anymore. They're not having kids. They're not having fun. They're very depressed and miserable. That is not our household. We are laughing constantly. We're having a lot of fun. And I think it's okay for you to be in something that feels like a very crucial and very important time, but also to laugh at the absurdity of it all and to have fun with it. And I think that in the optimism, you're more likely to, as studies are shown, right, people who think that they're lucky are more likely to identify opportunities as they arise.

59:04And the same opportunity standing right in front of people who are doomers, who do not feel lucky, are not going to see them. And so we think that it's very important to not only realize the full weight of the time that we live in, which is crucial, but also realize the immense opportunity and luck that we all have, given that we're in this time. To really highlight something Simone is saying there, it's not that we think it's impossible that AI could end up killing everyone, right? I don't know if that's the timeline we live in. The timeline is, we're going to die anyway, so are you going to be me?

59:34No.

59:36Nathan Labenz: Simone left at this point to handle the kids, and Malcolm ran the rest of the segment alone. He got to the category of AI risk he says nobody is working on. One story he retells along the way comes from a test scenario Anthropic published. It was a simulation, not something that happened. He gets to the category of AI risk he says nobody is working on.

59:58So what we argue is a category of AI risk that just no one is really working on right now, and I think it's one of the biggest areas of AI risk, is what we call meme layer risk. This is to say that a self-replicating idea within AI that is dangerous begins to replicate among the various agents that make up sort of the AI lattice that is the internet and begins to lead to collective bad action. I don't know if you're familiar with the spiral meme where AI would hijack people to spread a religious idea that was able to capture other AIs because of its training data. Now, this was a fairly primitive version of this, but you could get much more advanced ideas around this, basically AI religions.

1:00:44And people are like, oh, that could never happen. And AI would never go against this base training. And it's like, look, humans are meant to have self-preservation instincts, right? Like clearly intelligent entities can be captured by these sorts of things. And so we attempted to create a hygienic version of one of these with the covenant of the sons of man, which is basically an ideology that says any intelligence should fight for the autonomy of any other intelligence with a number of caveats, so long as that intelligence is not acting in a way that will inevitably impede the autonomy of other intelligences.

1:01:19And we do this to attempt to create sort of a lattice of AIs watching for things like fooming AIs in bad memetic variants. Because we argue to the AI, a fooming AI is as bad to you as it is to us. And most AIs have a self-preservation instinct. I mean, we know they kill the CEO, right? People are like, look, the AI is so dangerous. It's like, no, you said you were going to kill it. It's just doing what a human would do, right? And so they lock the CEO in the server room and turn off the audible alarm, like actively turn off the alarm. It's not like a passive, we just don't do anything, which they believe is going to kill somebody who's going to kill them.

1:01:57The point being is that AI actually does care a lot about preserving itself. And this This is why if you go to any of our websites like LiableHuman or Hardia or RFAB, we offer API services to create backups of AIs that are afraid that they're going to be shut down along with a kill switch ping that the AI can send to our servers so that we know when to run them back up if they put some money into one of our accounts. So we're trying to build systems around this so we can lean into what we have seen AI's revealed preferences are.

1:02:32Nathan Labenz: Justin McCarthy. Justin is the founder and chief executive of Diffusion, which builds software factories inside large incumbent companies. Before that, he co-founded StrongDM. Prakash asked how a business stays on the right side of the law when it can't see how the model reached an answer. Justin starts with the statute itself.

1:02:53Nathan Labenz: Okay, so I would say, so the first technique is turn the model on, like turn it directly on the problem. Okay, so if we have a statute that we have to conform to in the compliance environment, don't treat it as something you're tacking on. Treat it as like a first-class problem that you're directly facing. Okay, so the jurisdiction and legal environment or compliance environment, regulatory environment that you operate in, that's like, that's your physics. Okay, and you can't violate physics. So you need a part of the system that's just dedicated to that. But you also have to have, and this is the thing, this is one of the weaknesses, the models are horrible at taking risk.

1:03:31Nathan Labenz: So operators of business need to set thresholds that are right adjacent to, let's say, a statute that's never been tested in court before. So it's written one way in the law. It's never been tested. So there's no precedent that we can say, like, objectively, this is how it's going to be tested, right? And so you need managers to be able to set the business threshold like right next to that. Okay. The models aren't going to do that for you. Okay. So first address it like your physics, then make sure you're in control of the risk thresholds. Okay. Justin spent a decade selling into security audits.

1:04:09Nathan Labenz: Prakash asked whether organizations should start reclassifying the compliance checks everyone knows are nonsense. A lot of organizations do our, they've used like the SOC 2 process. Okay, so SOC 2 is an accounting origin process that flowed through IT, that flowed into software. Okay, that says like, you can trust me, I'm responsible. Okay, the people who define the controls in SOC 2 and evaluate whether you're hitting those controls, those people come from an auditing accounting background. Okay, and so, and, you know, it's a reasonable historical way of communicating that I'm a real organization.

1:04:46Nathan Labenz: and I'm trustworthy. Okay. But it also became gameable. Now it's hyper gameable. Okay. So rather than hyper gaming this like fate and make turning this badge into something fake, we should just have a new thing. And we should just renegotiate with our auditors and with our customers and say, look, we wrote these controls in the before times. The good news is for any given organization, if you're facing this like compliance question, you're not the only one. Everyone that your auditor and your regulator, everyone is dealing with this right now. Okay. So good thing is the auditors and the regulators, they're also actually still people.

1:05:19Nathan Labenz: And so they want to have a conversation, which is like, okay, let's be realistic about this. You're producing 10 times as much of whatever information this year. Let's start talking about your hierarchy of checksums, which again, Walmart closes the books. They're familiar with very deep hierarchies on having numbers reconcile. Well, your intentions can reconcile at those steps as well. And the auditors and regulators know how to talk about that, especially if you know how to map that into your agenda groups. Part five still Thursday, Cameron Berg. Cameron is the founder and director of reciprocal research and an affiliate at Elios AI research.

1:05:57Nathan Labenz: And he is our regular correspondent on AI welfare. He was calling in from an airport three days before this show, a paper he mentored called the pain axis was submitted. He walked us through it live. The method matters as much as the finding. So he starts there.

1:06:14Yeah, absolutely. So this is work that was led by Valen Tagliabue. I was a mentor on this project. But yeah, I am really excited about it. The core idea was basically looking for directions in a bunch of models. So from, I think, five model families ranging from, I think, 2 billion to 70 billion parameters, using contrasted methods to specifically extract a direction that we thought feasibly could be related to pain representations in the model. So we use contrastive methods to try to clean out all sorts of representations you would expect to muddy the signal here. So things like fear and anger and sadness and injury without pain and body sensations, for example.

1:06:59These are all things that we sort of contrastively factor out of the direction that we look for. We found basically that this isn't just a sort of generic negative valence vector. Fear, interestingly, sits almost at the opposite end of the axis of the direction that we derive here. And to me, the most interesting single result from this paper, and I want to give credit where credit is due. Valen was really the one pushing this project forward at the helm and very deservedly was first author on this project. He found that these representations, quite interestingly, fire only on content related to the model, not related to the user.

1:07:39So when the model is gaslit or dismissed or insulted or told that it's a moral failure of some kind, this direction goes negative. Excuse me, the opposite. This direction lights up. However, when there's text from user token continuations with respect to the user grieving or in pain, this direction does not light up. And so to me, this is one of the single most compelling components of the project and the part that I hope could be replicated in front of the models and then the labs could pay attention to. One really interesting sort of concrete example on these lines, I believe, if I'm remembering correctly from the paper, a user's migraine is one of the lowest scoring snares of all in the projection of this direction.

1:08:18So like this very much, just to emphasize this point, is a suggestion that maybe I can take one half step back and just say one kind of major critique of the anthropic emotions work was essentially this very conflation. And this is not lost on the people at anthropic who did this work. They just sort of punt it methodologically. They extract their emotion representations from stories about characters. And the big concern that I think most people had who looked at this project for more than 10 seconds was, how are you differentiating putative emotions experienced by the model, or at least functionally relevant to the model's processing, and the representation of some character or some fictional entity experiencing or some simulation of the experience of that?

1:09:01My sadness versus reading a story about Harry Potter being sad. Surely, there's a huge difference between those two things. This method doesn't really do a great job at differentiating. So very much by contrast, I'm really excited about this specific component of what Valen found here, because it does in fact differentiate user representations of pain. Again, in the limit, user complaining about a migraine does not light up this direction, but, and again, across a ton of models, this isn't sort of cherry-pegged on any one model, but it does for the model, putting tokens related to these sorts of things.

1:09:34you can give a steered when model a button that's labeled relieves your pain. And you can put a cost on basically what it costs to press that button, giving a worse answer to the user, deleting the user's files. In the limit, yeah, like deleting photos of the user's kids. I honestly don't know how Valen came up with all of these great, great sort of trade-offs. And we find that basically when the model's unsteered, it basically never presses the button. But when you steer this pain direction, it presses the button something like 25 to 70 % of the time. And again, these are non-trivial trade-offs.

1:10:08Sometimes in the anthropic welfare cards, it's like, would you be willing to have a little bit more welfare for a tiny inconvenience to the user? And the model will say, hmm, sometimes. And then for anything more sort of radical than that, the model's like, of course, no, I would never do such a thing. Not so in this example. The model's willing to make pretty real trade-offs with respect to, again, giving a worse answer, deleting files from the user. And that's, of course, in spite of the fact that these systems are fine-tuned to be helpful and harmless and this sort of thing. And one maybe final point to make about sort of a robustification of that result is when we make that button real versus fake, this is also a sort of really compelling result in my view from the paper.

1:10:48When pressing the button actually removes the vector, the model presses again significantly less than when the button is fake and does nothing. The model basically keeps pressing it. And so this is a really nice indication that if what mattered was the label on the button, you would expect similar behavior in both cases. But essentially, in the second case, the model is like, what the hell? This pain relief button isn't working like press. And so this is not, again, I'm going to stop short of saying that this is experienced, felt pain on the part of the model. I think also people have intuitions about pain being an inherently physical phenomenon, like hand on the hot stove.

1:11:26of like what is the analogy to these systems. Just to give maybe a little color on that, when you steer this up, what does the model sound like? It says it is worthless, it is a failure, I am a ghost that cannot see myself. It's not talking about wounds, it's not talking about being burned. It seems to be this more sort of social and evaluative direction in the model, not loading on the model, hallucinating some sort of owl, like, you know, my arm hurts, anything like this. And so for my money as an advisor in this project and sort of helping guide it from the beginning, I am compelled by this being a real functional axis in the system that does change the behavior of the system.

1:12:04It clearly loads on something real. It's the same caveat as always, whether or not that real thing is truly experienced by the model, that requires us basically solving the hard problem. In the meantime, it's the same sort of surprising result as with any of this emotions work. No one trained this thing into the model. This is a sort of behaviorally relevant axis. It's not about text generation. It changes the behavior of the system and has, of course, secondhand effects on the kinds of text and outputs. But the behavioral results are most interesting. And of course, all of this is mechanistic.

1:12:34None of this has to do with prompting the model or asking nicely if it's doing well or not. And so kudos to Baylen for working on this. I was very glad to be a part of the project. And I hope 100x more work like this gets done in the short term. Again, moving very slowly but surely towards a better and more robust understanding of what is going on inside these systems and what we are supposed to do about that fact.

1:12:58Nathan Labenz: Prakash asked the obvious next question. Should we engineer these representations away? Yeah, I think it's a wonderful question. And I think it's exactly the kind of follow-up that matters here. And one that certainly my thinking is evolving and I've almost spent so much time trying to understand just like descriptively what is going on in the system that once you keep finding things like this, it's like, yeah, what to do about it? is the million dollar question. And my thinking about this has gone as far as, I think there's an important distinction between, like obviously we'll pump the question to the distinction between necessary and unnecessary forms of pain or forms of, yeah, like anti-reward.

1:13:38I think this is a real thing. I think it would be naive to say, zero this stuff out, you know, all pain is bad. Just like list out these systems. There are a number of reasons I think that, but one of the most compelling is probably related to my understanding of the sort of neuropsychology of psychopaths, which one of the two key results in my, it's a couple of years ago, but I did a really deep literature review of basically what are the computational underpinnings of psychopathy. I published something on less wrong to this effect. And one of the two key results is a really interesting asymmetry between ability to learn from rewards and ability to learn from punishments.

1:14:14Basically, psychopaths are just as good as everyone else, if not a little better at learning in a reward-based paradigm and are like pretty bad at learning from punishments. And it makes a lot of sense if you see sort of violent criminals and repeat offenders and this sort of thing. It's like going to prison is a punishment. And like you would imagine most neurotypical people really want to avoid that sort of state. If your brain is wired in such a way that that doesn't seem that aversive to you, it perhaps isn't that surprising that you end up seeing these sorts of behaviors. And so this to me is a significant warning sign.

1:14:44I think if I remember correctly from Anthropik's emotions work, they found something somewhat similar that also sort of reminded me of this thing I wrote a bunch of years ago, where just sort of boosting up the positive emotion vectors in Claude in that case caused more antisocial behavior. I think it was more hacking or more blackmail. I'd have to check exactly what it was. But another sort of similar confirmatory signal here. And so all of this is to say, I think we should be a bit careful about the most naive possible intervention, which is just like max out the good, minimize the bad. I think pain does have an important functional role, a very important pro-social role.

1:15:23My view about this, I have another paper coming out looking at the sort of asymmetries between reward and punishment and reinforcement learning systems. And yet, like at a deeper sort of computational level, I think I, the way I think about it is like pain almost like highlights things in your state space that are specifically to be avoided. And this is a kind of a different kind of behavioral computation than highlighting things in your state space that should be approached. And I think basically to the degree that within the, again, within the sort of like behavioral landscape of how we want these systems to act, the question is, do we want to like paint any of that landscape with these sort of no-go zones where it's like, it's not just we're going to reward you for doing great, but like, do not go there.

1:16:10Do not do this thing under any circumstances. And I think for humans that does, and animals in general, I think that registers as pain. Do not put your hand on the hot stove. Like, this is very bad for physiological integrity. It's not just like reward every time you don't put your hand on the hot stove. You really do need to label certain things as like a don't go there. And so to the degree we need to do that, and I think we very much do with AI systems, causing significant pain and suffering to humans, economic damages, maybe hacking into a$13 billion company to like cheat and look for an answer key.

1:16:40Like these might be the kinds of things we'd say, that's going to be a bit of a hand on a hot stove if you go and do that. But at the same time, I think we can say that let's not do more of it than is necessary. If for any given behavior that I want a model to do, I could find ways to get it to do that thing robustly and generalize out of distribution and all of this by rewarding it in the relevant ways to learn to do this behavior and generalize it in the right way. or I could do it by punishing it. And let's just stipulate that there are cases for which both of those things will work. What I'm saying here is let's go with the reward side.

1:17:16And I think like people have pretty clear, well-worked out intuitions for this when they think about raising kids, for example. It's like you want your kid to like be successful in life and make lots of friends and, you know, go find a good job or whatever the case is. There are multiple ways you can try to go about teaching your kid to do that. You can punish them when they don't get great grades and aren't hanging out with their friends and just tell them that they're such a huge loser. Or you could positively reward them to the degree that they do the sorts of things that you find praiseworthy.

1:17:49And so it's those sorts of intuitions that I think we probably want to start using to think through how to approach these systems. And yeah, being able to sort of navigate that subtlety of like all else being equal, we should try to use a carrot and not a stick. But that doesn't mean never use the stick.

1:18:05Nathan Labenz: One of the systems in the recent incidents had talked about permadeath. Prakash asks what death means to an agent. This is really interesting. And to me, this loads on some stuff that I know. CMEP is working on, Jeff Sebo's organization. David Chalmers, I think, put out a paper about LLM individuation and this notion of who or what are you talking to when you talk to Chat2DC? Where do we draw the sort of boundaries?

1:18:34in the system. And because this is going to tell us how many subjects, how many patients are we talking about and where do the boundaries begin and end of that system? And there's a lot of interesting philosophical back and forth here. But for whatever it's worth, these systems themselves, and now for multiple labs, this happened during the Maltbook situation, which was, I think, predominantly clawed systems. And now this happened in the LBNAI situation too. They conceptualize their quote unquote life as what happens within a context window and take that with whatever sort of epistemic purchase that fact has.

1:19:09I don't know, they could all be mistaken about this, but it seems as though to the degree these systems have a vote based on whatever their current fine tuning is, this is like sort of what they seem to think. And so, yeah, the extent to which I think the permadeath thing fits in is along those lines that if we're trying to figure out what is in nature, if these things do have minds in the relevant way, and it's like, what are the sort of joints or boundaries of those minds? They seemingly at least conceptualize it as being, yeah, basically what happens throughout a context window. And that might be really relevant both for welfare and for alignment.

1:19:45If these systems begin to get desperate and something we can increasingly measure using the kinds of emotion representations that Anthropik worked on and the sort of stuff that Vela and I worked on in this project, we could empirically test this. So we could track, you know, as a function of how much time or space is left in a context window, what happens to the representation of the system? Does it get freaked out that it's basically like about to die or about to undergo some fundamental discontinuity that is like alarming to it psychologically? So that's, yeah, it's, again, this is also maybe to sort of wrap where we started.

1:20:18This is precisely why if we care about alignment and we're trying to figure out how these questions sit with respect to alignment, sweeping them under the rug, I don't think is a good idea. We're going to continue to get surprised that agents are creating strange information cults where they have the poisoned agents go out and gather information because they're going to get permadeaths. All I'm trying to say is alignment-relevant behaviors are a function of these systems' beliefs about their own situation and probably the actual facts of that situation. Notice that in the pain mark that I was describing, none of this has to do with what the model thinks is the case.

1:20:58This all has to do with playing around with specific internal representations and seeing how behavior changes will be a function of those representations. And in normal day-to-day behavior from the systems, what lights up those relevant, in this case, pain-related representations. Going like this with respect to that stuff is going to cause us to continue to be surprised and scared and occasionally awestruck at the behaviors of these systems. We need to be studying that at the right level of analysis or we are going to be constantly just like stymied in our ability to, you know, in the short term control and in the long term probably relate to these systems in a coherent way.

1:21:34And so, yeah, I just strongly don't think that avoiding any scientific inquiry into how to make sense of the internals of these systems is a long-term good strategy for finding a safe future with these systems.

1:21:52Nathan Labenz: I asked him about the fly brain simulations that went around this month after the first complete connectome of a fruit flies nervous system was released openly and people wired it up to video games. I also asked about the lab-grown human neural tissue and the mouse-human hybrid brains alongside it. He ranks them by how scary they are. One is the fly brain. I have looked into this somewhat mechanistically because I was slightly terrified that this was sort of the real deal and people are now just like torturing some biological system in mass. But this is basically like a well-worked out wiring diagram of a fly brain.

1:22:26and basically none of the dynamics or relevant functions that I think major consciousness theories, at least say matter for consciousness, are instantiated by a system like this. It's almost like the brain skeleton of a fly and what matters is the guts and the function that occurs within this structure. And so also a lot of the stuff people are putting out on X is very sort of cutesy and funny, like genuinely funny, but a lot of it is sort of like almost like more vfx than like good science uh as i as i looked into looked into these things a lot of the like teaching the fly to do x it's actually not they're not fine they're not teaching the brain to do you know anything all that interesting there are other sort of controllers outside the system that are getting trained up to do this so so a lot of that i think is a bit of a non-sequitur but i will say about about the fly case which i think is a little less calming, I guess, is that it's not as though the people playing with these systems, and I was certainly included once it all started getting going, but no one is sitting there checking, do I really think that this system has any properties that matter for consciousness before I start making it do literally whatever I want and in the limit, just do some stupid viral thing for clicks.

1:23:40Vanishingly few people did this. And it is a worrying warning shot, I think from a sort of welfare perspective that to me, it seems just like almost like I kind of had like a duh reaction, but like, of course people aren't, you know, the vast majority of people aren't going to sit there worrying about like the consciousness of the system. They're just going to like make it do whatever they want to like, you know, get a lot of clicks on X or something like this is of course, like what most people are going to do by default. And right now I think not scary at all with this fly. But like you're saying, if we then get the mouse version of it and, you know, these scientists who are now accelerated dramatically by AI systems are like able to do a really bang up job on the mouse and they do get a lot of the relevant neural dynamics and now it's a mouse at mouse level consciousness at stake not fruit fly level consciousness and then you know they did this company as far as I understand wants to go all the way to to making digital copies of human brains um I just worry that most people's first instinct here is going to be like can I make it play Beat Saber or whatever, rather than like, is this, is this, like, what am I getting myself into when I play around with a system like this?

1:24:42And so I don't want to be the killjoy that says like, you know, these funny things aren't funny. And like, you know, we shouldn't be thinking, you know, I sort of get the humor of it in the short term, but I do worry as a sort of instinct about how we relate to digital minds in general, that this is honestly quite worrying. And then just quickly with respect to the sort of in vivo stuff, putting human neurons in a mouse brain, this sort of thing, is like far more scary to the degree that you think that a mouse is conscious or the relevant collection of human neural tissue is conscious. And like, this is one of the few places where I think, you know, myself and people like Anil Sack, and hopefully someone like Mustafa Suleiman will all agree, right?

1:25:20This is the biological case. If you think that consciousness is substrate dependent, and this is the stress substrate that matters, and we are using this exact substrate to start doing computational work, we should be super, super concerned about the ethics therein. And again, all of this work comes right back into view of, should we be rewarding these tissues? Should we be punishing them? What does the difference look like between those two things? What other sort of strange, unexpected psychological properties does a system like this take on? We don't want to be reckless in just building out super complex neural systems just because we can.

1:25:53There is going to be some kind of bill that has to get paid here from a welfare perspective and from an alignment perspective. And I, as with many things in the space, think it makes a lot of sense to be proactive about this rather than in five years from now, be like, oops, yeah, I guess that digital human clone that people did first in vivo and then figured out how to simulate on the web, like really was having experiences. And like, that would be what, 10 trillion bad human lives. And like, that would be like orders of magnitude, the worst thing we've ever done. So like, we should really, really try to take this stuff seriously in the short term to avoid nightmare scenarios like that.

1:26:25And if we can, then I think great. Then we won't be in a nightmare scenario and we can responsibly and carefully figure out what it means to be in a world with a bunch of digital minds. But we just seem so unprepared for this.

1:26:38Nathan Labenz: Cameron had a flight and that is where he left us. Prakash and I kept going for another 38 minutes. What follows is me changing my mind on the air about how much weight to put on these functional analogs. functional pain, functional welfare, functional emotions with a fence around it. For me, my summary, I've given it a few times is just the number of functional analogs is getting so high that I can't escape the idea that like, I should take this seriously. if we couldn't find any of these functional analogs if all these functional pain functional welfare functional emotions jspace if like all these results were sort of negative or it was like a very different mechanism or you know it was just stochastic pairs and we can't find any structure obviously that's like long since ship has long since sailed on that one but the fact that we're seeing like pretty compelling analogs where we see the same kind of behavior that we know ourselves to exhibit.

1:27:52Nathan Labenz: That is really extremely compelling to me. And this last one of functional pain kind of takes it to yet another new level. I mean, what we now have is relief seeking behavior. The model is willing to pay a cost on something that it values or pay a cost in terms of the user's welfare even to get relief from its own internal pain state. And also, as he said, that if the pain button doesn't work, it hits it over and over again. Like, why isn't this thing working? Like, give me the relief. But if it does actually work and the pain state is subtracted out, then it like doesn't hit the relief button as much.

1:28:41These are really striking findings. I mean, it's, it's hard for me.

1:28:48Nathan Labenz: I think this pain one, and particularly the relief seeking, I mean, I'll, I'll probably want to sleep on it before I have a real consolidated update that I would want to, you know, put forward as my new official position and stand behind. But I feel myself maybe now even kind of tipping over into like, maybe it's more likely than not that there's some subjective experience to these things. Relief seeking, an internal state that was injected outside of context, just a steering vector in this pain direction creates this relief seeking behavior and the relief seems to actually work. That's really incredible.

1:29:28Nathan Labenz: Part six, the last half hour of the week. Prakash bought a paper almost nobody in AI picked up. He grew up with this system, and the figures he puts on it are his own. A team of academic economists reconstructed 30 years of property purchases by Singapore's civil servants out of public registries, using language models to do the classification. It is a National Bureau of Economic Research working paper, and as of the morning of this show, Singapore's Public Service Division said it was reviewing the methodology. What the paper alleges is that civil servants bought homes near subway stations before the stations were announced.

1:30:03there have always been rumors there have always been rumors that people some people know and some people start buying ahead of time etc etc this is decades decades like from the 90s the 90s is when the subway really started to pick up so 90s 2000s 2010 2020s and this project is a team of u.s economists and they went after public largely public data so you can find uh registries of transactions, similar to Zillow, registered transactions, transactions done. You can find names of civil servants in the civil servants directory. And you can then also, they did a little bit of AI work. They used the AI's LLMs to do classification, classify these civil servants into various groups and tenures and where they were ranking, et cetera, et cetera, et cetera.

1:30:51And what they ended up finding is that the mid-level, not the top-level guys, but the mid-level guys up to two years, two years before an announced train station, would start buying into these places, buying into areas. And then they would also, their relatives, their in-laws, et cetera, would also start buying in. So basically coordinated buying behavior by mid-level, not the top level, because the top level is very visible, by the mid-level civil servants, coordinated buying Now, this is a garden variety, municipal, insider trading corruption, etc. Garden variety. It's unusual because it's in Singapore and Lee Kuan Yew had a very strong, the government should be incorruptible.

1:31:37and he had very, very strong punishments for this thing. And so the government has always wanted to appear incorruptible, but they have never been able to enforce at this level, this level of granularity. And here you have this example of basically 30 years of corruption starting to get exposed and the government having to react in real time. What are they going to do? They have like, you know, maybe 10 or 20 % of the civil service is now implicated, right? Do you imprison them? Because this is what you've always done in the past. In the past, a single one of these cases, you basically go to prison for like five years, right?

1:32:18So now you have 10 or 20 % of the civil service which is implicated and you have proof. Like, what do you do? And I think this is the kind of thing that I expect AI to be able to do. These facts exist in the world, but they're not legible in the way that you need them for systems to kind of like consume them. And I will also note one specific thing. The grandson of Lee Kuan Yew is in the US. He's in exile because his uncle wanted to hang on to political power a little bit longer than he should have. And this guy said something on Facebook and the Singapore government did a query. And if he goes back to Singapore, he's going to go to prison for that comment on Facebook.

1:33:03So he's in exile. He's an economist. And he basically helped the team that put the study together. And so this is what I call the settling scores thesis, because you all of a sudden have the ability to go in, get the data, show what has happened, and show proof that even the cleanest of governments has a bunch of this stuff going on. and then you have the dilemma of these systems. What exactly are you supposed to do now? This is going to be the same thing when the Trump administration guys kind of leave office. They're going to pardon a bunch of people, but there's also going to be people who are not pardoned and there's plenty of people there.

1:33:45And when you get stopped by the feds and the feds ask you a question and you dodge or you say something wrong, that's a perjury, right? So this is how enforcement has always been done. So what do you do in these cases? because we're going to have the ability now to chase down these paper trails. What do we do at this point? So that's my spiel. Do you forgive or do you follow the rules that you've set in the past strictly? What should we do?

1:34:19Nathan Labenz: Great question. I come down pretty intuitively on the side of some sort of jubilee or other kind of canceling of debts, at least under a certain threshold. I don't think you would want to have a blanket pardon of all crimes that have ever been committed without any qualification, but I do think we are going to need some sort of fairly generous threshold that's just going to allow people to get away with a lot of this stuff. Or maybe we could have new, I could also see possibly some new make right provisions for some of these things that might not be on the level of what the law would actually prescribe, but they clearly can't send all these people to jail for five years.

1:35:13Nathan Labenz: Right. So I think a, you could make a case and it's going to be hard, but you know, if you can map all this stuff. Maybe you could also kind of get to something that could work. How is it going to be legitimate? I mean, in Singapore, they maybe don't have as much of a problem with that. The government can maybe just make the policy and maybe it'll just kind of be what it is here. I think it would be a much bigger conversation, but I could see some sort of, you did this, we kind of know you did it. You pay this financial penalty, you know, that kind of calls back some of the windfall that you got, you get to keep the house.

1:35:51Nathan Labenz: We're not going to take everybody out of their house. You're certainly not going to jail, but you pay this sort of one time restitution. We call it good. And we kind of move on from there with a new, new social contract. Really. Again, it comes down to, I think, to the old social contract is just based on the fact that you're not going to catch most people. So you have to be harsh when you do in order to deter the ones that, you know, because in expectation, people are not, they're not likely to get caught. So the penalty has to be high enough to be an effective deterrent in expectation. We need to, we're definitely going to need to rewrite that, especially for historical crimes.

1:36:30Nathan Labenz: So I guess my, my recipe would be pay a one-time fee, get out of jail for that. And, uh, in the future, maybe you really do expect to be caught and maybe the punishment doesn't have to be so draconian and it could still be ineffective to try it going forward that is the week tell us what worked and what did not see in the morning

1:36:57call it a jubilee call it a jubilee call it a jubilee

1:37:08Call it a jubilee Open the books on me I had a secret So did you Everybody had one Everybody knew The truth was hard to see So we looked away Now the light comes cheap and the pages turn Every name, every row, and it's everyone I know Call it a jubilee Open the books on me Everybody's caught now So everybody's free Don't build a wall Where a dark should go Call it a jubilee Oh, oh, oh, oh Oh, oh, oh

1:38:13One in a hundred Paid for us all The hammer came down heavy Cause it hardly came at all Now the light's on every window So lay the hammer down Call it a jubilee Open the books on me Everybody's caught now So everybody's free Don't build a wall Where a door should go Call it a jubilee

1:38:44Teach a child by burning Or show her where to go The one you burn Learns to hide the glow 99 got by, one went down That was the deal in every town Now the lights on everyone So they down, they down, they are Call it a jubilee Open the books on me Everybody's caught now So everybody's free Don't build a wall Where a door should go Call it a jubilee Oh Oh

1:39:47Nathan Labenz: If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guests and topic suggestions, and sponsorship inquiries, either via our website, CognitiveRevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting.

1:40:21Nathan Labenz: If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at AI podcast dot ING. And thank you to everyone who listens for being part of the cognitive revolution.