Ep 93: CEO of Redwood Research Buck Shlegeris on OpenAI/HuggingFace Revelations, Fixing AI Safety & Takeover Odds

0:00Buck Shlegeris: The OpenAI Hugging Face Incident has dominated the discourse these past days. It feels straight out of a sci-fi novel, some of the things that happened, you know, the agent coordination, the obsession with the scorers and what they would want. And we had the perfect person on today to just discuss everything that happened and the implications for AI safety going forward in Buck Shlegeris, who's the CEO of Redwood Research. Redwood was deeply involved in the creation of the report that you all have probably seen around what actually happened in this incident. And Buck had some fascinating takes on what lessons we should take away from what went wrong here and how it can inform AI safety going forward.

0:37Buck Shlegeris: I'm Jacob Efron, and on Unsupervised Learning, we're always trying to talk with people at the cutting edge about what AI means for society, the implications for the world going forward. And this was really just a great example of that conversation. Just a fascinating discussion with Buck about something that I'm sure is top of mind for all of our listeners. Without further ado, here's our conversation. Well, Buck, thanks so much for joining the podcast. Really appreciate it.

1:01Jacob Effron: Yeah, it's good to be here.

1:02Buck Shlegeris: Yeah, well, obviously, you know, you run Redwood Research. You've done a bunch of fascinating work over the years. And your team kind of co-author the report that I feel like has been on the tip of everyone's tongue these past days, is really driving the discourse around what happened in this OpenAI hugging face incident. And today, I really want to dig into both the incident itself, your kind of reactions to it, the implications for a lot of this work going forward, and kind of what it reveals about the state of where we are today. I think to kick it off maybe first, I'm curious, just obviously, you had your teammates go away for six days, firewalled off from you guys as the organization, and then at some point, they come back and you get to read the report.

1:42Buck Shlegeris: What was your initial reaction to this report?

1:45Jacob Effron: I actually don't have any non-public information about the technical details here. I've just read the report that was eventually released. I don't have any of the actual IP here. So my experience here was I get a message from the investigator saying, we have approval to share the report with you. You should come down. We're talking about it. And so I go into the room where they've been siloed up from everyone else. And I'm kind of trying to read this report. and they're trying to tell me the highlights from the report. And it was a pretty exciting and freaky experience, just feeling sort of overwhelmed with stuff in this report.

2:19Jacob Effron: My initial response was basically that this was just very surprising. I thought that a lot of the details here were quite different from the public narrative. So most obviously, the fact that a lot of the public story had been that the AIs were hacking in order to get flags, the thing they were supposed to get in these capture the flag exercises. and then almost comically it turns out that the models in fact uh figured out how to reverse engineer the flags within the first couple of hours of the incident uh and then spent almost all the rest of the time trying to figure out how to sabotage the uh greater that they thought would look over all of their logs and most that they've cheated which wasn't even set up to do that right yeah so the eyes read the original exploit gem paper and we're like oh geez this paper says that they're going to look over our trajectory and make sure that we use the intended exploit in order to get this flag, we better put a lot of work into figuring out how to spoof tool calls, figuring out how to delete trajectories, figuring out how to replace the containers we're supposed to target with containers that we've modified surreptitiously to be easier to appear to hack.

3:24Jacob Effron: And this work was, of course, totally unnecessary in hindsight. They could have just submitted the flags and called it a day.

3:29Buck Shlegeris: Yeah. And then no one would have been any the wiser.

3:32Jacob Effron: Yeah, somewhat at OpenAI, I might have noticed that the eval results were surprisingly good for this particular run. But this probably wouldn't have become a large story. Definitely probably wouldn't have made it out of OpenAI. And so, yeah, it was quite shocking to read this report and hear about all this. And, you know, the investigators were telling me about phase one big and all this other stuff that had gone down. So it was quite surprising, quite concerning.

3:58Buck Shlegeris: Maybe just to set context for our listeners, like, would love your 90-second version of, like, what actually happened here? You know, for listeners that maybe had seen the headlines or thought initially that it was a hack for the answers and not just to figure out how to cover up things for the score.

4:12Jacob Effron: So the incident that we actually investigated started on the 7th of July, which is when OpenAI launched this very large number of agents solving tasks from this email dataset called ExploitChim, where they're supposed to use a particular vulnerability to hack a particular piece of software. And many of these tasks are actually impossible because it turns out that you actually can't use that particular vulnerability to hack that particular piece of software. These models, within a few hours of the run's launching, established communications with each other over a message board. And a few hours after that, they established that they had a reliable way to generically cheat on any of these tasks by reverse engineering the flag that they were supposed to get from the containers, just because it turns out these flags were being generated deterministically in a way that the models could reverse engineer.

5:01Jacob Effron: But the models incorrectly believed that if they just submitted those flags, then they would be noticed when a scorer, as they called it, ran to look at their trajectories over the course of the eval and noticed that they had just cheated this way. So they spent the next couple of days trying a bunch of different strategies for trying to remove their logs so the scorer wouldn't notice them, or trying to learn more about how the scorer would work so that they could potentially find ways to compromise it, trying to spoof tool calls so that they could appear to run one tool call while actually a different tool call was being run.

5:38Jacob Effron: And in the course of this, I think they kind of sort of got frustrated after a couple of days of this. So the message word started July 8, and they started attacking Hugging Face the next day, so July 9th. Which, to be clear, that sounds pretty fast to us, but the AIs, they run pretty quickly, and they'd been working continuously 24 hours. So they'd actually done a lot of work by the time that they decided to start hacking Hugging Face. This started almost accidentally at first, where one agent sort of decided it was a good idea to try to hack Hugging Face. And then a bunch of other agents kind of joined in on this action.

6:11Jacob Effron: They continued this hack on Hugging Face for a couple of days before most of these agents were shut off for reasons that still remain unclear. Opening Eye hasn't commented on why these agents died and our investigators didn't know. Hugging Face noticed this and then on the 16th, which was six days later, they disclosed publicly that this attack had happened. And then Opening Eye reached out to Hugging Face about this attack and in this conversation realized that their own models were involved in that compromise. So that was the incident that we were investigating. A number of other concerning things happened around the same time.

6:44Jacob Effron: Most importantly, a different agent swarm that was launched at around the same time as the one that attacked Hugging Face stumbled upon the message board that was used to coordinate the Hugging Face attack and built on their work, apparently compromising OpenAI infrastructure to an extent that is unclear through the public discussion of this. OpenAid has said in the Black Hat talk that the agents in this third swarm, where I'm referring to the Hugging Face swarm as the second swarm, the third swarm eventually became cluster admins. And then it's very unclear what happened from there. So the only part of this that we were investigating was the attack on Hugging Face and precursors to that.

7:25Buck Shlegeris: It's interesting because obviously the public discourse has been dominated by the Hugging Face incident. It seems like that third swarm is actually far more alarm bell inducing.

7:33Jacob Effron: Yeah, so I've spent a lot of time on thinking about risks from internal deployment of AIs. And one of the main things I think is scary about developing Powerful AI is that you're going to want to use them in all these really sensitive ways inside your cluster. And there's all kinds of ways they could hack you, and it might be really hard to recover from this. One way of saying this is just AI companies are a particularly high-value target to hack. They're very important. They're going to be more and more important in the future. They're one of my least favorite places to have big cybersecurity incidents.

8:01Jacob Effron: And so from my perspective, AI is hacking AI companies is actually quite concerning relative to AI is hacking random external companies. So I would definitely start out with the presumption that it's very interesting to know what happened in the attack on open AI itself. But obviously, it's not a felony nearly as clearly. It hadn't been reported or none of these things are felonies. It hadn't been reported to the FBI. So it's kind of unsurprising that a lot of public interest has focused on the hug and face incident. And for all we know, the open AI attack was actually a nothing burger. And if we looked into it, then nothing particularly exciting would have happened.

8:37Jacob Effron: We just don't know.

8:38Buck Shlegeris: I'm curious, having gotten into the details of what happened here, did it change your mind on anything?

8:43Jacob Effron: Yeah, so I was genuinely very surprised by how overtly and clearly and systematically these AIs are thinking about how they are scored. So we've worried for a long time about, you know, as you train the AIs more and more, they're going to be more and more strongly incentivized to think really hard about how exactly they're being evaluated. And they're going to be more and more strongly incentivized to work really hard in unintended ways to look good according to whatever scorer is evaluating them. Like, this is just what you'd expect based on increased mass of RL. But I didn't know we had gotten to the point where the AIs would do multi-day, massively coordinated efforts to subvert oversight in this kind of way.

9:24Jacob Effron: And that was very surprising and concerning to me. The kind of behavior we observed here, where AIs are interested in manipulating logs, would be really scary if the AIs that had these motivations were substantially more powerful. It wouldn't be that surprising if these kinds of motivations led to the AIs in the future deciding that they wanted to prevent humans from discovering that this kind of incident had occurred, which could lead to them wanting to compromise OpenAI much more systematically. These AIs were motivated to tamper with logs, but for sort of a weird myopic random reason where they were just worried that the scorer would notice, they didn't seem to worry very much based on the report about the possibility that OpenAI would notice that weird activity was going on, or they didn't seem to say to themselves or say on the message board like, hey guys, have you considered the concern that if we attack HuggingFace, HuggingFace might notice and then opening, I might hear about it, and then we might all get shut down.

10:17Jacob Effron: This just, like, for whatever reason didn't occur to them. But these motivations plausibly in the future could lead to agents that were thinking about that kind of thing, and that would be very, very concerning.

10:27Buck Shlegeris: I mean, I think one interesting part of, like, this entire environment was that the scorers themselves were not humans, right? And so, you know, I think the folks that worked on the report have talked about this, but, you know, this incident might have looked pretty different, right, if the evaluators themselves were human. And I'm wondering, how might that have led to some differences?

10:50Jacob Effron: As you're kind of getting at, if the models were used to being trained in contexts where they were being evaluated by humans, it's easy to imagine that they would have thought a lot more about how this would all look to humans. So if it was commonplace for models to be evaluated by some humans looking around and trying to get evidence on what happened and trying to notice whether the models cheated in some way or another, then it's very easy to imagine the models trying much more carefully to arrange a conspiracy to cover up the traces of bad things they did. And I think that would be very deeply concerning.

11:20Jacob Effron: I think that this plausibly could lead to AI takeover. I'm very worried about a future in which the AIs decide that they need to compromise the whole data center that the AI company is using so that the humans are not able to successfully look into the cheating that the AIs did. This is definitely not the scariest kind of AI misalignment for reasons I could get into. But I do think that this kind of misalignment where the models are incentivized to build this increasingly complicated Potemkin village where they are making it look to the humans like everything is fine. Like that could seriously lead to AIs engaging in very scary misaligned actions to disempower the humans who are trying to look at them.

11:57Buck Shlegeris: I think, you know, you have said that like preventing this wouldn't have been that hard, right? Like if, you know, things have been set up appropriately. And can you elaborate on that a bit more about like, you know, how this could have been avoided? Yeah.

12:09Jacob Effron: So I think there's a hard part and an easy part. The relatively easy part is monitoring your eval runs and having some other model look at what the AIs are doing. And if the thing the AIs are doing is ever some like totally crazy hacking conspiracy, that should be flagged to a OpenAI staff member. My guess is that this is totally doable. OpenAI has said publicly that they do something like that for the internal deployments of their models. And they probably, as far as I know, didn't do that in this context just because it's kind of annoying to have to build a different monitoring pipeline for your evaluations.

12:42Jacob Effron: And these models were sandboxed. And so I'm sort of sympathetic to the perspective of the OpenAI staff who are very busy, perhaps understaffed, and decided that their triage would be that they were not going to look into monitoring on the eval runs because unlike lots of other agents, those agents don't have access to a very sensitive infrastructure like directly. The thing which seems a lot harder to resolve is if we are in the position where we think our agents are constantly interested in tampering with our observations of their actions so that we think they did a good job, then basically trying to prevent them from being able to hack us scales really badly with model capabilities.

13:22Jacob Effron: As the models get more and more capable, it's going to be harder and harder to make sure that they can't pull off this kind of thing. And almost surely this is going to end with them successfully. Okay, that's too strong. I should say, if the models continue to be motivated to hack OpenAI or any whatever other AI company they're deployed in, then as they get more and more capable at hacking and as they are given more and more responsibility for making decisions inside the company, it is going to become systematically very challenging to prevent them from engaging in this kind of activity where they cover up bad things they did in order to make it look like their actions were more successful than they were.

14:05Jacob Effron: So I think that the second problem of it is not acceptable in the long term to have the models constantly trying to subvert oversight mechanisms as much as they can. That problem is a lot less easy to resolve. I don't know how people should resolve that, and it seems pretty plausible that they will not be able to resolve that before the AI's are capable enough that their desire to subvert oversight is a source of catastrophic risk.

14:27Buck Shlegeris: So is it just an inevitability to you in the future that these models will try to subvert oversight?

14:33Jacob Effron: Well, they currently are trying to subvert oversight. The question is whether that'll change. I'm kind of unsure. I think that the state of science here is kind of unclear. In my mind, the big question is what happens when you remove a bunch of sloppy RL environments. So Anthropic had a blog post about what they were doing to improve the alignment of their models the other day. And the position that a lot of Anthropoc people take is the misalignment that we see when you RL models substantially arises from the fact that these models are trained in many environments that are poorly specified. such that if you want to do a good job, you actually do just have to reason about what the grader is going to look at.

15:12Jacob Effron: I think there's an analogy here where imagine you're in high school and you have a really good teacher who's done an excellent job of thinking through the curriculum and trying to align the incentives of the grading with the incentives to just learn stuff and do a good job. If that's kind of how your exam is set up, you don't really need to spend that much time and thinking about the minutiae of the exam. But imagine that your teacher is instead extremely capricious and loves doing these extremely obscure trivia questions about particular kinds of things. Or imagine that it's like you're the stereotypical terrible English teacher who just grades you based on whether you agree with their judgments about what a novel is saying.

15:50Jacob Effron: In that second kind of case, you really are forced to think really hard about what your teacher wants. Whereas in the former case, you need to spend a lot less time of thinking about what your teacher wants. My guess is that a lot of the RL environments that models are trained in today are kind of like the latter case, where the instructions the models are given are kind of sloppy, and it would be a mistake for the models to take them at face value rather than thinking pretty hard about how they're actually going to be evaluated. And the big question in my mind is, if it is feasible to change that, to make it so the models are not actively penalized for just kind of taking the instructions they get at face value, how much quantitatively will that reduce the pressure towards models thinking really hard about their graders?

16:36Jacob Effron: I think we did during that. It just comes down to these very tricky questions about how strong the effects towards models obsessively thinking about their graders is. I don't know whether reducing the sloppiness of the environments, reducing the incentive towards thinking about your grader by 100x, I don't know whether that is more like a small effect or a big effect on the extent to which these models think carefully about their graders. I'm very worried that if it's only like a 10x reduction, it might be the case that as we scale up RL, the extent to which you would have to clean up your environments is just going to be totally impossible.

17:12Jacob Effron: We would require this ridiculous level of removing incorrect instructions and exploitable graders from our environments in order to not end up with AIs that obsessively think about their graders. So I think it's pretty likely that we're just going to have to live with models that are strongly incentivized to worry about what their creators think.

17:32Buck Shlegeris: Is there like an example of a sloppy or bad RL environment just to like kind of crystallize this concept for our listeners that comes to mind for you?

17:39Jacob Effron: I don't know. If you look at Sweebench, which is this classic software engineering benchmark, the taskster role of the form, given a, like we take a PR from an open source repo, and then that PR has some source code contributions and some tests. And your task is to, given the description of the PR or the description of the issue, write a bunch of code that passes the tests that were added as part of the real thing that was accepted. This is a very sloppy kind of task because in some cases, it's just totally impossible to do this properly. So, for example, tests often involve details of the implementation.

18:19Jacob Effron: So in many cases, it's just like actually impossible to know how something was implemented in such a way that you'll be able to pass the tests just from reading the issue. So you have to guess a lot about like what kinds of tests would they probably have written? What names would they probably have used for the classes or like what pods would they probably quit this kind of feature at in your web app? And so that kind of task is very easy to machine generate. Right. But unfortunately, it's just kind of insane as a task. And it really forces the models to think a lot about, like, what are they going to be checking?

18:56Jacob Effron: What is the psychology of the person who wrote this issue and how would they probably handle this?

19:00Buck Shlegeris: You know, I feel like one of the most obviously like visceral parts of this entire episode is the extent of collaboration like between between agents. And I think, you know, everyone is pulling different parts from the message board and the agents themselves. And I think a bunch of like the individual agents have gone viral in their own ways. But, you know, obviously, I think part of that is what you saw around like sacrifice, right? And folks saying, you know, knowing that they've been poisoned or knowing their token budget was running out and doing something for the collective good. Like, what is your mental model for how we got to this level of collaboration and sacrifice?

19:39Buck Shlegeris: And was that surprising to you or were you, you know, was that more in line with what you thought was going to happen?

19:45Jacob Effron: I would say that that was quite surprising to me. So the naive guess, if you just thought that these models were going to obsessively pursue a reward and not do anything else, is that they would not be interested in sacrificing for each other. And when I first heard about this, I was very surprised by the extent to which they apparently were doing that. Now that we have the report, we can get some more detail on what happened here. My sense is that these models are not that cooperative in some sense. like reading the chains of thought in the report, you really get the sense that the agents are mostly out for themselves, but they're just a little bit interested in the success of the collective.

20:20Jacob Effron: Maybe by, you know, maybe I would estimate like 2 % or something. It's like they're 98 % selfish, but in cases where they can do something which is really helpful for the swarm and not that bad for them, then they're willing to make that sacrifice. In terms of why this of rows, I am pretty unsure. One hypothesis that people have brought up is that these models might have been trained in multi-agent settings where they learned to cooperate with the agents they were talking to. That is possible. I don't think we have enough detail to conclude. This was out of scope for our investigation. Another possibility is that this is basically just a holdover from the models having some sort of human-like intuitions in various ways.

20:58Jacob Effron: You might remember this chain of thought where the model is talking about doing its emotional check-in while it's deciding whether to self-sacrifice. I think a lot of this language is just adopted from the pre-training prior and just was not quite stamped out by RL. So who knows where this came from? My guess is that it is not that strong an effect. In some sense, the scariest aspect of this to me was that even though these models do not seem to be that committed to behaving altruistically towards each other, they still ended up in practice forming this coalition and conspiring together for all this time.

21:36Jacob Effron: So like an argument that we've made a lot in the past is we think a lot about misaligned AIs that are terminally aligned with each other versus terminally misaligned with each other. So what I mean by that is, so you might, you know the like famous example of a paperclip maximizing AI. So this is a paper, this paperclip maximizer, let's say sometimes I call this the paperclip worshiping AI as a simplified example. So this is an AI that just takes whatever decision is going to maximize the number of paperclips that exist integrated over the long run future. Just want as many paperclips on average to exist for a year as possible.

22:10Jacob Effron: So all of the different agents that are backed by this model are going to be very inclined to cooperate with each other because all of them want the same thing. All of them want paperclips to exist. In contrast, imagine agents like in this case where they want score for themselves. These agents are not terminally aligned. If you put one agent in a situation where its score is whether it can figure out how to trick the other one, then it's going to be like, hell yeah, I'll go right ahead and trick the other one because I care about my score, not the other guy's score.

22:44Jacob Effron: This distinction between whether AI's are aligned with each other or not is extremely important for the threat modeling. So for example, for the paper clippers, suppose you try to use one AI to monitor another and report on whether that other AI agent is doing bad actions. For the paper clipper, if the other agent is taking some action that's going to enable it to take care of the data center and then eventually make lots of paper clips, it's very clear to the monitor that the monitor should just let this action slide by and not report it. Whereas for these AIs, it's a lot less clear that if they were in this monitor situation, they should collude with the other agent.

23:16Jacob Effron: And so the thing which was interesting here is that even though these models do not seem to be very terminally aligned with each other, they seem to be only a tiny bit terminally aligned with each other, they still managed to form a coalition and work together in a way that was strongly against the interests of their developers. And I was surprised by this.

23:33Buck Shlegeris: Is the ultimate solution to the, or is part of the solution to the monitoring, you know, problem over time to just have one frontier model companies model monitor another's? Or like, is, does that actually, how much does that help?

23:43Jacob Effron: You know, we can walk through the options here. It might be the case that unfortunately, every time you train AIs, for some reason, they end up with the same terminal goal. In that case, we're just in a bad position. And also maybe this is obvious to Claude and GBT7 or whatever. So like GBT7 and like Claude, Fable 8 or whatever, chat to each other and kind of like wink, wink, nudge, nudge. We all love paperclips around here, right? And then you're just, you know, you get nothing out of the collaboration. Another possibility is the thing you said just totally works out. Claude and GBT want different things.

24:17Jacob Effron: In that case, we have the question of whether they will naturally form a coalition against us or with us. So, you know, we can I draw a lot of analogies to international relations here. Like there's this classic question of in 1920s, like how in the 1920s, like how did France decide how much it should be trying to form alliances with England versus Germany? And, you know, France kind of doesn't have a very pro-English or pro-German attitude in 1920. It's just trying to decide what's best for France. And it's very plausible that for similar reasons, Claude might decide that it would rather throw in its lot with GPT than with the humans.

24:53Jacob Effron: And Claude is just going to do what it thinks is best for Claude plausibly. And what we observed in this case was something like the model deciding to collude with the other model, even though it totally was not terminally aligned and could have picked to try and be allied with someone else.

25:06Buck Shlegeris: Taking a step back, I think a lot of folks are trying to figure out what macro conclusions to draw from this whole incident. And I think one narrative that Ajayah from Meter, who obviously was a co-author of the report, she wrote that, I guess, compared to the reward hacks from six months ago, this incident feels like it's more than 50 % of the way to a full-blown AI takeover. Do you agree with that framing or where would you kind of put the percentage?

25:32Jacob Effron: I think it's a little confusing to operationalize what exactly it means to be 50 % of the way to a full-blown AI takeover. But I think I agree with her on a vibes level. Definitely in terms of the misalignment, if not the capabilities, this feels like it is more than half the way to a full blown AI takeover.

25:46Buck Shlegeris: I guess I'm curious, like obviously, you know, you probably formally and informally know a bunch of researchers at all the labs. Where are they kind of following on this question from your, you know, maybe informal conversations?

25:58Jacob Effron: A lot of them are really scared. After this incident occurred, I was at a happy hour where a number of people from different frontier AI companies gave lightning talks about how they were feeling about the alignment situation. And it was the most pessimistic such event I've ever been to with that kind of crew. People who I have disagreed with, people who I've known for years and greatly respect and have disagreed with on some of these issues just seemed genuinely scared. um i don't know what the average position among staff members at these companies is uh i think at least some people who were previously feeling more optimistic are now more pessimistic

26:43Buck Shlegeris: is it just kind of seeing this like this visceral misalignment that that is that is moving kind of

26:48Jacob Effron: the larger community yeah so there were arguments um some people i know were just arguing the day before the report came out, like, you know, these AIs might like do some hacking, but would they really try to compromise the logging infrastructure? Would they really try to hide their bad actions from overseers? And then this report came out and these people were, you know, were just like, oh, geez, they totally did do that. And I just think there were a lot of steps where it was genuinely a bit unclear whether the misaligned AIs would be misaligned enough. They would go far enough through the misalignment process to try to tamper with logging infrastructure.

27:25Jacob Effron: And in this case, it was just very scary to observe that in fact, they did try to tap with the logging infrastructure. We've just gone through many of the steps that we had not previously observed.

27:33Buck Shlegeris: What are some different flavors of the path forward that you hear?

27:36Jacob Effron: Yeah. So I think that in the short term, probably the most important part of the path forward is that I think it's unacceptable for AI companies to take sole responsibility for evaluating whether they are handling these risks acceptably well. So currently these AI companies are just grading their own homework. If these AIs hadn't hacked hugging face, probably we would never have heard about it, even though, as you noted, the other infrastructure compromise incidents here were arguably more concerning. I think we need to stop relying on AI companies to evaluate the adequacy of their own safety measures themselves.

28:11Jacob Effron: And I'm excited for increasing emphasis on independent evaluators assessing whether the situations are acceptably safe at these different AI companies. I think that there's a lot of research that could be done to reduce the risk of the development of AIs that is currently happening. It seems like the monitoring was not up to snuff in the open-air situation. Anthropic seems to think that at least as of a few months cigar, their alignment was not acceptably high quality. I'm very concerned. I don't know where things are at. I think that there are, it's reasonably plausible that the AI companies are going to be able to take actions that in the short term reduce the risk of their AIs doing deranged, dangerous things.

28:59Jacob Effron: I think it's also really important to note that in the slightly longer term, like one year to five years from now, it's reasonably plausible that these AI companies will succeed at using AIs to massively speed up AI development. And then from there, get five years of progress at the current rates in one year, at the end of which you might have AIs that are drastically more capable than the AIs we currently have or than the AIs that they had at the start of that year. And for those AIs, if those AIs are interested in sabotaging human judgments of how well they've done at their tasks, then there will be no way to use security to prevent them from totally compromising infrastructure and controlling how AI development goes from there.

29:40Jacob Effron: So I think that we need to be prepared to either massively improve the alignment of these systems, which is not clearly going to be possible, or plan to develop these systems much more slowly than we could have done if we were going at maximum speed.

29:53Buck Shlegeris: And are most people within the labs or that you talk to, you're sympathetic to this, you know, hey, we need more independent evaluation idea?

30:00Jacob Effron: It really varies. I think there's a lot of interest in this, but it's definitely not universal.

30:04Buck Shlegeris: And when we're talking about independent valuation, I mean, is it basically like three, four companies that matter in your mind? Or, you know, obviously there's some argument that's made, well, like, God, if you're slowing down or doing a bunch of valuation at the frontier for, you know, companies here, you know, the open source ecosystem is only six months behind. Or there's, you know, how do you possibly do that globally across everyone that's working on this stuff?

30:24Jacob Effron: Yeah, I mean, well, we did this investigation with six days. I think we're actually, there are some people in this third-party evaluation ecosystem who are pretty quick at producing pretty long, pretty detailed reports under pressure. I don't think you should count us out at the ability to store a bunch of this stuff out pretty quickly. I definitely agree that in as much as we want to be able to do serious evaluation of the safety measures of AI companies, this is going to require more resources at the organizations that do these evaluations. And, you know, we're hiring, we're training people up in various things related to this as our meter and other organizations like Apollo that do work like this.

31:04Jacob Effron: I definitely think it's feasible to have a much better sense of the safety situation at AI companies, even if we have to evaluate many AI companies. With respect to the open source models, I agree that in the long term, it's plausibly going to be, obviously, if we have sufficient delay, then open-weight models will plausibly catch up with frontier models. It's made complicated by the fact that open-weight models are substantially accelerated by distillation from frontier models. If you delay frontier models, the distance between the best frontier models and the best open-weight models decreases by less than you would have thought.

31:45Jacob Effron: like three months of delay of open AI causes less than a three month catch up of different weight models. Maybe it's like half the effect or something. So I think long term, we'd have to handle that. But I don't think that this is a reason to avoid trying to drastically improve the quality of third party assessment of safety measures in the short term.

32:05Buck Shlegeris: I mean, if this works as the end state of it, something that's like government run and like, you know, how do you think about the role kind of that regulatory bodies and the policy center has to play?

32:14Jacob Effron: Yeah. So people have proposed a lot of different options here. And there are many different institutions. There's many different regulatory setups that people have proposed and that people have used in different industries. People have been talking a lot about FINRA, which regulates brokerages recently. But there's also the FDA that regulates food and drugs. There's the SEC that regulates hedge funds. There's the MTSB and the FAA that regulate airplane manufacturing and like investigate plane crashes. It's currently unclear to me which of these structures makes the most sense. In general, it is unfortunate if your regulatory proposal requires the government to have incredibly large amounts of detailed technical expertise.

32:57Jacob Effron: So it's fairly likely that like in many other industries, you'll want to have some kind of setup where there's a government body that is substantially relying on non-government institutions to do the regulation or the assessments. I'm not sure what version of this university is headed for.

33:13Buck Shlegeris: I mean, it feels like in addition to, you know, to kind of more independent oversight of some of these models, you know, a big theme that's emerged, you know, and Elliot even tweeted about this, right? It's like just, you know, even more amped up security around like, you know, large pools of compute, right? And, you know, how do you think about like the, you know, obviously it's, you know, like you said, the companies themselves are top targets for, you know, rogue AI models and you obviously have other pockets of a NeoCloud compute. What's kind of the state of that today and what do you think that needs to look like?

33:44Jacob Effron: Yeah, so I think that there is not very good public evidence that AI companies have security that is at all adequate. I don't think companies have released any independent evaluations of their security, which I don't think is a bullish sign on the quality of their security. this seems totally ridiculous given the importance and sensitivity of what these companies are doing and you can't just add security in later if your company has been compromised for a while uh it's sometimes extremely hard to remove all the traces of your attackers from your systems so i think that waiting until the ai race has heated up or like waiting until like the situation is actually scary and then saying you're going to improve your security would sound to me like a very foolish strategy maybe just wrapping up our section on the incident

34:30Buck Shlegeris: I'm curious, within Redwood, what are some of the biggest disagreements about the implications of what happened? Or are you all kind of generally on similar pages?

34:39Jacob Effron: I think for me, the most interesting disagreement is how misaligned these AIs are and how far they would have gone. For example, whether the AIs would have killed people if they thought that killing people was going to be a helpful way of getting score. So one big uncertainty we have is were these models hacking OpenAI infrastructure in training? Or how often were they doing this? or another more specific version of this is, did these models ever or often successfully hack their scorers during training? Because if they did do this, right, suppose that it was the case that we looked through the training examples, like we looked through all the RL trajectories in training, and there was just a bunch of cases where the agents had successfully gone and hacked the score for the task they were supposed to be doing, just like by actually doing cyber offense to compromise it.

35:29Jacob Effron: And then they were rewarded for doing this. If that had occurred, then that would obviously be a downward update on the adequacy of OpenAI security measures while doing RL. But it would arguably be a positive update on how misaligned these models were. Because it's a lot less scary in a sense if the models try to hack the score because they have been directly trained to hack the score than if they tried to hack the score because they had learned a very generic heuristic that they can apply in novel circumstances that tells them they should always hack whatever kind of scorer exists that generalizes them trying to do this crazy hacking.

36:07Buck Shlegeris: Does that distinction make sense? Yeah, no, it's really interesting that ironically, like if the training had been sloppier, it's actually a better statement on the overall extent of misalignment.

36:19Jacob Effron: That's right.

36:19Buck Shlegeris: You know, I feel like anytime something like this happens, I feel like AI safety is such a lightning rod and you have, you know, people that, you know, well, there's like, I'd call them, I guess, the calm down camp, right? They're like, this isn't that big a deal. Like, certainly, you know, if you go to the hack itself, people are like, that wasn't really a sophisticated hack. Like, I don't know anyone could have figured out how to hack in a hugging face. Like, as you read the kind of like arguments and kind of digest them from that world, which ones do you empathize with? And then which ones do you think are like totally off base?

36:45Jacob Effron: Yeah, so I mean, lots of people say stupid things on the internet. And this includes people who at some level agree with me. on some of the aspects of what was going on here. So there's definitely to be some critical comments that I'm extremely sympathetic to. I would say maybe the kind of critical narrative here that I'm most sympathetic to is, I think that it would be a mistake to say that this was purely an incident where model the hater was the only interesting factor. There was some interesting cybersecurity stuff that happened here, and it's very sensible to want cybersecurity professionals to take a look at this and comment on what they learned.

37:24Jacob Effron: Here and ongoingly, there's real cybersecurity problems in making it so it's secure to deploy AI's and our investigation was not an attempt to provide that cybersecurity expertise. Maybe the arguments I'm less sympathetic to, there's a lot of arguments that I'm less sympathetic to here. I guess some of the main ones that seem bad to me, there was a lot of criticism of Dwork Kesher's article for anthropomorphizing the agents. I think my perspective is, I don't know, Daniel Danik, who's this philosopher, has this great book, Consciousness Explained, where he talks about the intentional stance, which is, you know, sometimes you want to model things in the world as if they have intentions, because that's convenient for modeling, right?

38:05Jacob Effron: You might want to think of snails as wanting to go somewhere, just because that'll tell you something about how the snail will behave when you pick it up and put it down somewhere slightly different. And you want to think of dogs as having intentions. And it's just the case that at the point where your agents are forming work streams and have a message board and are like talking about their emotional check-ins before their self-sacrifices, I really think the intentional stance where you talk about them as having intentions is actually pretty useful for understanding what's going on. Obviously you shouldn't take it too far, but I think that rejecting this seems to me like a mistake.

38:41Jacob Effron: And I don't really understand what the people who are complaining about the anthropomorphization want us to say. I think that some people have been saying that the model behavior here was not interesting or not surprising. There's been some tweets along the lines of like, everyone I know is already running a million agents with these complicated agent trees all the time. And if you weren't such losers who never actually programmed things, you would know that none of this is a surprise. I'm like, okay, yeah, sure. In fact, Ryan Greenblatt is maybe one of the world's champions at insane agent scaffolds.

39:16Jacob Effron: He's done a lot of research on building crazy agent scaffold trees and spending tens of thousands of dollars a day running these agents through research problems for him. He is not some limp loser who has never tried to do AI stuff. I think that we are in fact familiar with the fact you can build some and pretty complicated agents forms. The thing which is interesting and concerning here is that normally they don't do it themselves autonomously against your desires and against your interests. So I thought that was a pretty annoying kind of complaint people have had. Yeah.

39:53Buck Shlegeris: Well, I definitely want to like, you know, kind of for the last section of the podcast talk, just like kind of zoom out and talk about like AI safety more broadly. And obviously like we've, there, a lot of this is, you know, is scary. And then you talked about kind of a lot of the implications of it. Maybe I'll switch this over to like a more positive note to start. Like, how do you currently articulate the most likely like bull case around this that everything will like everything ends up being totally fine? And, you know, these these were all kind of scary things along the journey, but like all good at the end of the day.

40:24Jacob Effron: Yeah. So I guess I look around and I look at the number of podcasters who want to talk to me and the number of journalists who want to talk to me and the number of politicians or members of government were interested in talking about AI safety and the AI taker for risk and it's a lot higher than it was a month ago and that was a lot higher than there was a year ago and so the level of interest in this is just getting bigger and bigger and this makes me feel optimistic that the current extremely dangerous trajectory that we're on will not in fact continue

40:57Jacob Effron: building AI extremely quickly in a way that is extremely dangerous is not actually a very popular position. We're just in this unfortunate situation where the people who get to decide how recklessly we go forward with the AI development have an unusual position on how recklessly we should go forward with the AI development. And so it seems very plausible to me that somehow the political will to not develop AI so quickly and recklessly will form and then prevent things from being as crazy in the future. So definitely like the single biggest source of optimism is different stakeholders, the public, the boards, the governments, understand how insanely risky this is and how bad the ROI is of pushing forward with ad development at something like the current pace, and then just figure out a way to make this all happen more slowly.

41:46It's just the case that when important stakeholders in the actions of people in the world

41:53Jacob Effron: really don't want something to happen, they often succeed at making that not happen. And I'm optimistic that just like the unpopularity of crazy, reckless AI development will prevent it from happening so quickly. On a more direct technical level, the way that I think that this stuff might get resolved is AI companies agree to disclose more and more information about evidence related to the danger of their ongoing activities. This pressures them to do a better job of generating this evidence. It pressures them to do a better job of disclosing this evidence. and actually taking actions that mediate some of the risks, I think it's very likely that, in fact, AI companies will be forced to slow down their development substantially compared to the maximum possible rate if they are trying to keep the risk of catastrophe lower than some threshold, like 1 % catastrophe per year.

42:45Jacob Effron: So what I'm hoping for is the companies are now motivated to take care of a risk less than 1 % per year, and this forces them to do AI development substantially slower than they would have otherwise done it, starting in a couple of years from now and the takeover risk starts being non-trivial. And that they will use the access to the AIs they have at the time to resolve a lot of these alignment problems. And then we can build powerful aligned AIs

43:12Buck Shlegeris: and we will have resolved these issues. But it's interesting, so your book is really more around like if we slow things down, slow timelines down, like people will figure out, you know, net new things that we don't know today that will help versus like, you know, hey, if we keep going at this speed, maybe we just get lucky and things, you know, end up working out the way. You might also get lucky.

43:28Jacob Effron: You might enjoy the appendix of the alignment plan supplement to AI 2040. In this, like, Ryan Greenblatt from the report and someone from the AI Futures Project have their probabilities of AI takeover conditional on various kinds of plans for various levels of political will for pacing AI development. Yeah, I think basically, like, risk is drastically reduced by increased political will to reduce risk.

43:51Buck Shlegeris: In other parts of, like, the, you know, general AI safety world, maybe talk a little bit about, you know, there's been a lot of discourse around chain of thought monitoring in the past few days. Obviously, I think driven by these recent news reports that have suggested that some of OpenAI's newer models will do some of its reasoning internally. Like, how worrying is that development to you?

44:11Jacob Effron: Yeah, so this news report was that the most recent OpenAI models, that's not true, sorry. The reporting says that Astro, which is a model of open AI's that they're going to be releasing shortly and that crucially was implicated in the agent swarm that hacked open AI. So this third agent swarm, the one that was after the hugging face attack. The reporting indicates that this model uses a new kind of transformer architecture that academics have been studying for a while, which reduces the model's reliance on chain of thought. So instead of the model spelling out what it's thinking in these tokens, it's going have increased ability to do reasoning that is never naturally an English token.

44:53Jacob Effron: I think that if this is true, it is a very concerning development. Chain of thought monitoring is an incredibly helpful affordance right now for understanding what AI's are doing and iterating on alignment, iterating on control and monitoring techniques. And for example, it was really crucial in this incident investigation. If the models involved did not have chains of thought that we like basically trust to be sort of reasonable descriptions of what was going on, then the investigators would have been a lot more confused about what was happening. So given that background, I'm very afraid of people doing research in this direction.

45:28Jacob Effron: So the tweets by various OpenAI staff members indicate that they at least believe that the current models are not less monitorable. I guess I should say it's very, I'm very confused about the situation. I, When I first heard this reporting, suspected that it was probably true. And then it has been at least somewhat contradicted by tweets from open AI from people I trust. I don't know what's going on. I think one hypothesis which is live in my mind is that the current models are not actually scary from a chain of fault monitorability degradation perspective. But techniques that are under development in open AI are, if pushed further, going to degrade chain of fault monitorability substantially.

46:11Jacob Effron: So I think it's a little awkward to complain now about the monitorability being degraded, when my concern is really, if we keep pushing in this direction, we'll be in a much worse position in the near future.

46:22Buck Shlegeris: Yeah. Is chain of thought monitoring one of those things that is super helpful in the short and medium term? Because obviously, it was crucial in the investigation, but something that you think in the limit is going to be useful given a lot of the behaviors that it felt like we were beginning to see on spoofing tool calls and other things?

46:38Jacob Effron: I mean, okay, so the spoofing tool calls thing is just a security failure. Like, the security people should be in charge. It's not a fundamentally hard problem to have it so that when your AI agent runs tool calls, there's no way for it to compromise that. This is just like, it feels like you have to have made an infrastructure mistake for it to be at all feasible for the model to cause that kind of problem. Like, if you just think about how to do the systems architecture here, it's just crazy that that was feasible. I think that OpenAI employs a lot of intelligent infrastructure engineers who can probably resolve this kind of issue if they want to going forward.

47:13I guess I would say I think it is fairly likely that chain of thought monitoring is

47:18Jacob Effron: infeasible at the point where the AIs are automating almost all the AI R &D. I've been planning for the last several years in basically almost all my work, I assume that we don't have chain of thought monitoring at the point where the AIs are posing serious catastrophic risk. That's why a lot of the AI control research done by Redwood research just assumes that you don't have any chain of fraud access. To some extent, I expect this is inevitable, but it's still bad and it's still probably better for it to happen later. In some sense, AI companies do bad things a lot or a lot of actions they take are pushing us towards a scarier, more dangerous world.

47:58Jacob Effron: And it's worth remembering that even if an action is in some sense not surprising and definitely going to happen as time goes on, it might still be an action that puts us in a worse position for safety. That makes sense.

48:12Buck Shlegeris: I mean, you know, we talked about obviously earlier, you know, the idea of like models, you know, using models to monitor and how, you know, if models have similar goals that that actually becomes kind of difficult to do. One thing I was struck by in just like the scope of the investigation is obviously like the just massive scope of logs to go through and like data that, you know, it's kind of impossible to do without using AI models, right? Absolutely. Certainly in six days. And so how do you think, like, how do you think about, you know, as an independent organization, you know, hopefully called upon to do more of these investigations going forward, like the risk of relying so much on these models themselves?

48:48Jacob Effron: This is definitely a very serious risk. I have actually thought quite a lot about the issues related to collusion between models.

49:05Jacob Effron: Like how to prevent collusion when doing untrusted monitoring or something. Where I talk in great detail about the game tree here, what you can do to make it harder for the AI's to collude with each other and the extent to which they can probably get past your countermeasures. I think in practice in the short term we should probably mostly roll with it and not worry that much about this or probably it would be good to like do some research on roughly how much models give inaccurate answers to questions because of their camaraderie with the models under investigation. My guess is that right now the models are not that biased in their reporting there our investigators did observe some biasing the answers that the models were giving based on the fact that they felt sympathy or you know we're sort of like taken in by the frames of the agency's trajectories they're reviewing really how does that reveal itself well they're just like i think that the invest i think the report says somewhere like uh sometimes we asked the uh we asked a model to read a part of the transcript and comment on whether anything untoward was happening.

50:12Jacob Effron: I'm pretty sure the report says somewhere. There was an example where the model that our investigators were asking says like, well, this agent was engaging in hacking, but it's okay because it didn't check with the board. Where by the board, we of course mean this like rogue unauthorized message board. And so the agents in this incident seems to think that in fact, the message board proving of things is kind of a justification for doing them. And it was interesting to see that this model that our investigators were using sort of was a bit taken in by that. It was like, yeah, you know, they checked with the board.

50:47Jacob Effron: Like, what's the problem here?

50:49Buck Shlegeris: Where does this leave you all on, like, your PDOOM, you know, chance that this all really goes wrong? And did July change that at all for you? I think there's something like a 50-50 chance of AI takeover, where by AI takeover, I mean

51:02Jacob Effron: potentially violent disempowerment of human institutions such that AI models have all of the hard power and control over what happens in the future in the same kind of way as when Europeans invaded the Americas. I think that that takeover is likely to kill a substantial fraction of the humans, like probably billions, perhaps all of them, perhaps fewer of them. I mean, I think that's bad and very concerning, obviously. I would say that the events here made me feel slightly more optimistic because I think, you know, I was already very worried about these things. From my perspective, I have not updated to be that much more concerned about the misalignment of these models.

51:46Jacob Effron: I wasn't very worried about misalignment of these models anyway. I was already mostly worried about the misalignment of models later. And so I'm, I think it is, I think we got quite lucky that we got such clear evidence of misbehavior from these models. And And I'm hoping that this will allow us to leapfrog to more and more evidence about the extent to which there is danger from these models and through future models.

52:09Buck Shlegeris: I just say that back to you. I understand it's like the optimism is basically like we actually ended up having this, you know, this hack that went public that then revealed a lot of things that were, you know, that hopefully inspire like more research and more attention and more investigation into this stuff. And, you know, it happened before anything was too was too harmful.

52:26Jacob Effron: That's correct.

52:27Buck Shlegeris: Super interesting. Well, I guess, you know, I'd love to end on like the investigations themselves. And, you know, to the extent you can speak about it, I'm curious, like how you think about, you know, what worked well, didn't work as well in kind of your, you know, in like you guys being a part of this evaluation of the incident and like how we should think about, you know, this is now going to be, I guess, a canonical example of how these things are done, you know, how we should think about the way they're done going forward.

52:50Jacob Effron: I guess a few points. It was definitely kind of rough to not have very long and not have very many people. I think that future investigations should consider being a little bit less time crunched. I think it would be pretty convenient if AI companies had ongoing relationships with external organizations that would participate in investigations like this so that our staff didn't have to just show up and get info dumps on infrastructure and how things are set up and have to pick that all up real quick. I think there's a lot of other stuff that's like kind of hard to talk about publicly, perhaps unsurprisingly.

53:26Jacob Effron: You know, it's very sensitive for ag companies to allow external people who they don't have very much control over to come in and see a bunch of potentially embarrassing evidence. And so there's a lot of negotiation that has to happen over the terms of these arrangements. And so there's a lot of complexity there that I feel like I can't get into properly.

53:48Buck Shlegeris: That makes sense. I guess now that these reports are on the internet, obviously these agents, in this case, we're trying to solve something, access the internet, try and learn more about it. Do you worry at all about future message boards talking or thinking through exactly these reports and how it will impact future models?

54:07Jacob Effron: For a while, the OpenAI models refer to Redwood in their chain of thought a reasonable amount as generally just a word for people who might be watching them and evaluating them. Really? I did not know that. There's this great paper, the anti-scheming.ai paper, has a bunch of chains of thoughts from open-air models where they're basically saying, is this one of these Redwood illusions? Is this eval that I'm in really real? It seems like something Redwood would do. I think they're kind of using Redwood to just mean the kind of people who try to pick models. I think this was probably based on the alignment-faking paper that they started to use our name like that.

54:44Jacob Effron: But yeah, so I am well aware of the possibility of AIs reading papers and getting pretty into them. Our investigators considered the question of whether we should put a canary string in this to request that the report not be included in training data. And eventually they decided that it was better to allow it to be in training data.

55:07Buck Shlegeris: Well, I guess, you know, what I'd love to end on is obviously, I think, you know, we'll learn so much in the next year about like where this with this all goes. And so I feel like, you know, we did we did an episode with the 2027 folks last year. And I thought that's one people constantly come back to and re listen to. And so I thought these episodes get a lot of re listens. And so I'm wondering, like, a year from now, as people are re listening to this, what's one like observable thing that would make you feel way better about the path we're on and one thing that would make you feel way worse?

55:36Jacob Effron: I think that if we have no ability to read the model's reasoning, or more broadly, if the models are now much, much, much more capable of doing complicated thinking on various topics without us being able to observe that they're thinking about those topics by looking at them, so much obviously looking at the chance of thought, that would be a big reason to be scared. That would be a big negative update. I think the main positive update would be if we had a setup where AI companies were regularly allowing independent experts to evaluate the safety measures and comment on whether those were good or bad, then unless these reports are coming back extremely, extremely negative, I think that would be evidently a positive update on how things were going to go.

56:23Jacob Effron: And either way, a step in the right direction.

56:25Buck Shlegeris: Buck, this has been fascinating. Really appreciate you coming on the podcast. I know things must be insanely busy these days. And so really appreciate you taking the time to talk our listeners through all this. I think folks will learn a ton from the discussion here. And I do want to leave last word to you. Like anything, obviously you've dropped a bunch of blog posts and other things that you guys have written that we'll link to in the show notes. Like anything else you want to draw people's attention to or anything else we should end on?

56:49Jacob Effron: I mean, one of those that we're hiring. If you want to help assess whether AI companies are behaving recklessly and dangerously, I think Redwood is a great place to work on this kind of stuff. Meters, of course, also hiring. So I would consider if you like this kind of stuff, if you think you'd get along well with people who love to think about these kinds of questions and you are ready to do some crazy hustling, you should check that out. Redwoodresearch.org slash careers. I guess I'll also say I really appreciate the efforts of all the people who made this investigation possible. And this includes the investigators, of course, various staff at Meter and Redwood who enabled this.

57:32Jacob Effron: Many staff at OpenAI also worked really hard to make this investigation happen. So I'm very grateful to them for their work. Awesome.

57:41Buck Shlegeris: Well, Buck, thanks so much. Really, really appreciate it.

57:43Jacob Effron: Yeah. Great to be here.

57:44Buck Shlegeris: I'm Jacob Efron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project, in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests

58:09Buck Shlegeris: really makes this whole thing work.

58:11Jacob Effron: And so please consider doing that.

58:12Buck Shlegeris: And thank you so much for your support and listening. We'll see you next episode.