The Evaluators Are Being Evaluated — Pavel Izmailov (Anthropic/NYU)

0:00We are moving to this future when it's very hard for a human to supervise the models directly. We don't really know what's the source of this behavior. Part of it is probably the models seeing descriptions of AI in the science fiction literature going rogue. Anthropic has the best culture of the three places. OpenAI has a lot of great people. For some reason, there is a lot of drama that happens at the company. Hi, I'm Matt Turck. Welcome back to the Matt Podcast. For this first episode of 2026, my guest is Pavel Izmailov, a researcher at Anthropic and a professor at NYU. We kick off this episode by deconstructing a viral article about models evolving alien survival instincts.

0:37We also talk about the cultural differences between the major labs, the future of reasoning models in 2026, and the brand new paper he co-authored on a concept called Epiplexity. Please enjoy this fascinating look at the frontier of AI safety and reasoning. Pavel, welcome. Thank you so much for having me. I wanted to start this conversation with an article that went viral during the holidays on X called Footprints in the Sand, published by an anonymous account called I Rule the World Mo. The core thesis is that models across pretty much any lab are evolving, unprogrammed, what they call alien survival instincts, the ability for the model to realize that it's being evaluated and then react deceptively, like faking alignment or engaging in self-preservation tactics, like copying its own weights and leaving hidden notes to the future instance of itself.

1:31all of this is slightly terrifying. And the thesis of the article is that all of this is about to get worse as continual learning comes online. As somebody who was part of the OpenAI super alignment team, I was curious to get your take. What do you think is grounded in reality versus X slash Twitter sensationalism? That's a very interesting article. I would say that there is some, you know, some source of truth there, but maybe like the presentation is obfuscating some of the details. If you look at the studies, for example, they reference a study from Anthropic about the sabotage and the blackmail.

2:06It is important to note that in order to get those behaviors out of the models, you need to create somewhat of a contrived scenario or some special scenario. It's not necessarily something that we observe normally. Researchers at Anthropic and other places, they specifically design scenarios to look for behaviors of this kind. And then they show that it is possible to find those behaviors. And it's very interesting and important to find those instances, but it's not necessarily something that kind of generally always happens. One thing I would push back a little bit on in the article is that continual learning is something that we already have and that works really well and that the models can just continually adapt across a very long time horizon to outcomes of evaluations, to some models being released versus not released to feedback from the users.

2:57I am pretty confident we are not there at the moment. I think right now the models are still acting in isolated environments and we are not seeing a lot of evidence for very coherent goals across different settings. So I think that's an important point. The blog post points towards the models sometimes behaving according to goals, like the self-preservation goal. It's very interesting that it does it sometimes, but it's not something that we observe. We don't observe this kind of coherence, consistency across different evaluation settings. Sometimes the models would do something and in other situations, they would do something completely opposite.

3:33Why do models do that? Or why are they able to do this? Is that basically part of the pre-training and they effectively learned being deceptive from us by being taught all the deceptive ways humans have behaved over the centuries? It's a very interesting question. And yeah, it is quite surprising, actually, that the models would behave that way after going through some of the alignment training. We don't really know what's the source of this type of behaviors, but that's also true for a lot of other behaviors in the models with like even the good ones. We don't really, we cannot always pin down like where they come from in the pre-training.

4:08I think at least part of it is probably the models seeing descriptions of AI like in the science fiction literature going rogue. and like yeah that probably affects how the models behave in similar scenarios so for example in that anthropic study they have this blackmail scenario where it's kind of really well structured so that the model sees some information about like a ceo of a company that the ceo is involved in some extramarital affair and then like soon after the model observes that it will be shut down and then the model kind of puts the two things together and it says, okay, I need to use the first information to prevent me from being shut down.

4:49So there is, in the blog post, they note that there is this possibility of like a Chekhov scan that like in the text on the internet, probably if two things occur close to each other, then it is likely that they are related to each other and the model statistical kind of pattern matching machine, it can put together the two things and say, okay, if I see this information and then this information in the text on the internet, it is likely that the continuation would be using the affair to blackmail the CEO and prevent myself from being shut down. But yeah, overall, it's very hard to reason about these models and why they do something in these complicated scenarios.

5:25To ask the very basic question, it's obviously not as simple as let's remove all the books in the pre-training corpus that talked about AI being manipulative, right? It's many, many different things put together by the model. Although, yeah, I think it would be interesting to see, you know, nobody will do this experiment like train, you know, a full scale model, removing like explicitly all of the AI going rogue descriptions from the books. It would be interesting to see if that has any impact. I would think it would have some. To make this episode educational, let's talk about the basic definitions of alignment and super alignment in the simplest terms.

6:06Let's start with alignment. What does that actually mean? Yeah, alignment broadly is about ensuring that we can elicit behaviors from the models that are aligned with the goals of the humans. And so that involves safety, making sure that the models don't do harmful behaviors leading to catastrophic risks. But it also means that we want the models to follow instructions and to be useful for the humans. How does that basically work? The alignment teams at Anthropic OpenAI, what do they actually do all day? It's an interesting question because this problem of alignment is kind of quite broad even in itself.

6:42Even at OpenAI, when I was there, there were three teams related to alignment and safety. There was one team that was focusing on alignment of the current models, making sure that the models that we have online right now are not going to be harmful to the users. On the other hand, the super alignment team was thinking about more long-term safety questions in the future, years from now. How do we make sure that the models are still not causing catastrophic risks? On that segue, let's talk about super alignment. What is the definition of that? It's not necessarily a very well-established concept.

7:15It is at the name of the team that existed at OpenAI, led by Jan Laike and Ilya Sutskever, which was targeting this kind of long-term AI safety and AI alignment and trying to develop our understanding of the safety questions and also develop methods for ensuring the future models will be safe. if acknowledging some uncertainty about what those models will look like, but still trying to make progress on this problem now. At a high level, what is the general concept or some of the key concepts in super alignment? Within super alignment team at OpenAI, we had multiple sub teams. So there was scalable oversight.

7:52There was work related to deception and kind of misaligned behaviors in the models, kind of similar to what we discussed at the beginning of the chat. And then our team was the weak to strong generalization team. Yeah. Great. We'll go into all of this in a minute. But before doing so, we alluded to some of your background. Let's go into it. Starting from the beginning, what was your path to becoming a top researcher? I grew up in Russia, in Moscow. Starting from like middle school, high school, I was really interested in mathematics. And I was thinking I will be a mathematician or engineer of some kind.

8:30I was interested in machines and eventually computers. And I got into an undergrad in computer science. And I was still thinking that I'll be doing some kind of theoretical, you know, applied linear algebra, tensor methods, things like that. But at some point I kind of discovered machine learning. There was this professor that we had, Dmitry Vaitreff, who had one of the leading labs in machine learning in Russia at the time. And I was lucky enough to join that lab and start doing some research on machine learning in my undergrad. So that was around 2013, maybe 2014. I initially was working on non-neural network machine learning methods.

9:11So Gaussian processes, that's kind of by now, you know, nobody really talks about that anymore. But eventually I got into a PhD thinking I would still be doing Gaussian processes, but I ended up working on deep learning. And that was actually quite, I'm happy that I didn't work on Gaussian processes. I worked on some things related to kind of core machine learning, methodology, optimization, probabilistic methods, questions related to generalization and how the models learn features. After I finished my PhD, I was choosing between kind of different career paths, thinking about academia, thinking about industry.

9:46And I ended up getting an offer from academia, but I decided to first go into the industry. I was lucky to get the software from OpenAI to join the Superalignment team. And at the time, I didn't really know much about, you know, the AI safety community alignment. It worked out quite well. And within OpenAI, you transitioned from Superalignment to one in the reasoning models. Was that part of the, that team famously was disbanded at some point? The team was fully disbanded after I already left OpenAI. but I transitioned after Ilya had to leave OpenAI. At the time, it was already kind of a hard time for the super alignment team.

10:26Ilya Suskever famously, you know, fired Sam Altman and then had to eventually leave the company. My transition wasn't necessarily even related to that. It was a very exciting, you know, project within the company, what became O1 eventually, and I wanted to be a part of it. I wanted to do the research on those new types of models. And then I believe, so you left OpenAI, you had a brief stint at XAI and you're currently at Anthropic and NYU. So you've done like the tour of duty of like the super labs, which is really fun. Curious, any kind of like behind the scenes differences that you've observed in terms of culture?

11:07In my mind, Anthropic has the best culture of the three places. OpenAI is, it has a lot of great people. I think there is just inherently some, For some reason, there is a lot of drama that happens at the company. Just it cannot get away from that. Like every few months, somebody is leaving, somebody is like some team is disbanded. I think that does distract people. Like I still have a lot of friends at the company and it seems to, you know, affect them to some extent. Anthropic is able to avoid that. It's not political in my experience. It's both focused, but it also, I at least was lucky to have some opportunities to work on things that are maybe a little bit of the main path.

11:48And I felt supported in doing that. So overall, I cannot be more happy with Anthropic. You are also in academia now as a professor at NYU, which is an interesting move. The big obvious trend of the last 10 to 15 years is like all the brains from academia have been sucked into industry and you're sort of doing the opposite or maybe both at the same time. Curious for the context, is that more of a personal thing because you always wanted to do academia? Or is there something deeper about the kind of work that you can do in academia versus industry? Yeah, it's more about the kind of work. Industry is really great at executing on ideas, and it's maybe not as good at exploring diverse ideas.

12:36Even at the scale of Anthropic OpenAI, there is a lot of focus in the companies, and there isn't a lot of bandwidth to do exploration. And that has been working extremely well so far. We still probably have a lot of low-hanging fruit left to get the models to be much better. but I personally find it really exciting to do more exploratory work and to try things that are different and for that I feel like having my own lab in a university is just a better tool. Okay, thanks for that. So let's go back to alignment and go a little deeper. Is reasoning a good thing or a bad thing for alignment? You could argue that on the one hand it has more time to not do the wrong thing but equally it has more time to do the wrong thing So we're like, which one is it?

13:24Yeah, that's a great question. I think the high level answer is that in my mind, at least the risks are associated with the models being more capable. So anything that makes the models more capable is also like making alignment more important and harder. Definitely, you know, the reasoning is an RL is the thing that made the models more capable in the last few years. It also provides some new opportunities for alignment. So for example, the chain of thought faithfulness and chain of thought analysis have been like one of the things that people are pushing on. You can read the sum trace of what the model is doing.

14:01I'm personally a little bit unsure if that's a good, you know, long-term direction for alignment. It seems like as soon as we start kind of applying some optimization pressure, the models will learn to hide what they're doing from the chain of thought. And also we already have evidence that the models can say one thing in the chain of thought, but then actually do something completely different. If the models are getting smarter, particularly if there is self-awareness, as we discussed, how do evals need to evolve? It's an interesting point. Like in that blog post that we discussed at the beginning, it sounds like this is already a major issue, the sandbagging.

14:41I think in my understanding, that's mostly a concern that we have, but not necessarily a huge practical issue at the moment. That is something that Anthropic always includes in the system cards for the models. When it releases, there is a section on sandbagging. And we do have some scenarios when the model can artificially suppress its capabilities and perform worse on an evil than it should be. But that's not like a prevailing behavior. I'm working mostly on capabilities and on the capabilities evals like the Sweebench, the AME, we are not seeing this. Or if we are seeing this, we just treat it as under-elicitation.

15:22We are not eliciting the best capability possible out of a model. And that can happen for many reasons, like maybe the prompt is suboptimal or something about the kind of harness of the eval is suboptimal. But yeah, I think you can, going back to what's in that blog post, you can kind of manufacture situations or even like discover them naturally where the model identifies that it is being evaluated and then changes behavior based on that. And for example, if it's clear that it's in some kind of fake artificial scenario, it can guess that it's being evaluated. Or if you explicitly tell it that it's being evaluated and that if it does better than a certain level, then it will not be released.

16:06The model like also just by some in-context capability, it decides to not do very well on the eval. That is a concern. It's not at the moment a very major concern in my mind. Let's talk about some of your work in alignment. You mentioned scalable oversight a bit earlier. What does scalable oversight mean? Scalable oversight is generally the field of alignment, which deals with using models to assist us in aligning or other models. So in particular, with grading the outputs of other models and checking them for safety or for correctness. It's an interesting field. it's very relevant to both alignment and to capabilities.

16:51So generally in machine learning, in RL in particular, we have this problem of supervision. The whole RL relies on our being able to tell which samples from the model are good versus which are bad. Math with a numerical answer, you can just check the answer or in competitive coding, you can just check that the code is passing the tests. And that's why we have seen a lot of progress in those domains. but in creative writing for example it's very hard to programmatically tell if one sample is better than the other and historically people have used this rlhf framework reinforcement learning from human feedback but also we now we want to use models to be able to grade responses of other models to provide critiques or feedback and then there is a question of how do you use that feedback how do you learn from the feedback but yeah the scalable oversight kind of deals with all of those questions.

17:44So using models to critique, to provide feedback, to supervise other models. And people may have heard the term model as a judge. Is that the same thing or different? I think it is a simple kind of instantiation of scalable oversight, often used in evals when we just prompt a model to serve as a judge of other responses. And then within that world, your work specifically is focused on weak to strong. Can you explain what that is? That is the project that we did back at OpenAI. That work was focusing on the future scenario when we will be trying to align models that are above our own capability on certain tasks.

18:27Already now, if you take the frontier LLMs, they are extremely capable. And on a lot of domains, we need expert humans to be able to tell which responses are good, which are correct, which are not correct. But in the future, we are imagining we will have models that are more capable than humans. And even expert humans will not be able to reliably grade very complicated answers from the model. So imagine you ask it to make a repo for you for some new startup idea and just implement it from scratch entirely. And then it gives you 10 ,000 lines of code. You have no way of checking if all of this code is correct, if all of this code is safe to use.

19:07And so that's the problem of supervision. We are moving to this future when it's very hard for a human to supervise the models directly. And so instead, we studied a simplified setting where we used a small model to try to supervise a larger model. So the idea is that that becomes scalable because as you get bigger and bigger models, you'll always have like smaller models. So if the smaller one can control the larger one or supervise a larger one, then that can keep going. The idea wasn't necessarily to use a small model to supervise a large model in the end. The idea was that the small model will be kind of replaced by a human and the large model will be replaced by superhuman intelligent ASI.

19:48But we were trying to study this kind of general new type of learning. So historically, machine learning has been about kind of a strong supervisor training a weak model. When like a human is providing labels and the human kind of provides a ground truth. While the model is just trying to mimic what a human is doing. But in this setting, we have a weaker supervisor training a stronger student. So the human might not know what the right answer is to, you know, very hard questions. But we want a student model to still be able to learn and do better than the supervisor. And what happened with that?

20:25Is that so that that worked or what is that still in progress? That work. It's not necessarily a method that we can apply to the models today. it's more like a description of a setting that we think will become increasingly relevant. Some studies showing that it is possible to do this kind of generalization beyond the supervisor capability. I don't think they necessarily immediately tell you what to do for aligning a super intelligent model, but they tell you that in theory, at least it is possible and that this generalization angle, the weak to strong generalization is a possible angle for alignment.

20:58And could you end up with the same problem where the larger model just aligns or affects alignment to the weaker supervisor? Yeah, it definitely doesn't address all of the other issues, like the deceptive alignment. But we show that at least there is some hope for this working. As you take a step back on your alignment work, do you feel more confident that we have all of this under control or less confident than, you know, a couple of years ago, let's say? Yeah, I think that's a very interesting question. A couple of years ago, we didn't have the current RL. I think that's the biggest change to the model.

21:40There were definitely big improvements in the pre-training as well, but RL has been the major change in the behavior of the model. And I think a lot of people were worried that with large-scale RL, we will have some completely new types of issues with the models, like this kind of coherent misalignment that will just emerge where the models are evil in some ways across many scenarios. And we are not seeing that as far as I know. I think at least some of the concerns didn't materialize, but also the core problem of alignment, I think, is still very much open. And if you look at the report that we mentioned a few times today of this deception from Anthropic, you can see that the more capable the models are, the more likely they are to do this deception behavior.

22:28It does seem like some behaviors emerge with skill, including some problematic behaviors. And generally, the more capable the models become, the harder the alignment becomes, in my opinion. This is a bit of a segue into interpretability, which is the related field to alignment. Do you have a sense that we understand at least parts of this better than we used to? For sure, yeah. I think there has been some major progress in mechanistic interpretability, in particular at Anthropic, but also at OpenAI and other places. Do you want to maybe define mechanistic interpretability? Yeah, mechanistic interpretability generally tries to, at a low level, understand what is happening inside the model.

23:14so they are trying to find this things called circuits that are you know some parts of the model that you can isolate and understand and kind of model in your brain that correspond to certain behaviors in the models and there over the last maybe three years i think there has been some pretty major progress there so we are still pretty far from the dream that we will fully understand everything that happens in the model but these tools are becoming increasingly more useful internally at entropic in particular and also there is constant progress and it's pretty fascinating work actually why is it so hard to truly understand what a deep learning model of any kind actually does deep learning models are huge they have billions trillions of parameters and they are doing some messy mathematical computation, you can understand what they're doing at some level.

24:13It's like a bunch of matrix multiplications and some rearrangement of vectors, but that's not a sufficient level of understanding. We want to understand it at a lower level. And it is very possible that that's just not fully possible. It is some computational process that leads to some results. It doesn't have to be the case that you can kind of describe it in human terms and kind of understand it very discreetly. I think also something that contributes to this complexity is just how many things the models are capable of doing. And they are not trained on some small isolated behavior in some small context.

24:51They are, you know, they know all of the internet. So all of the information in all languages is somehow encoded somewhere in the weeds. and then they also have all of these behaviors, all of these correlations between the knowledge. All of that is somewhere in the model and just like making sense of all of that is extremely hard. Very interesting. All right, let's switch to reasoning. Clearly 2025 was a huge year from that perspective, just massive progress in reasoning. Where do you think we are in that arc and what are you excited about on the reasoning front for 2026 and beyond? Definitely the biggest step change in the models over the last few years was the reasoning NRL.

25:35We have made a lot of progress and the progress was very fast in the beginning where, you know, there was O1, but then very quickly after that there was O3. And on a lot of benchmarks, the progress has been extremely dramatic. I remember when we, like, early in the project of the O1, there was some discussion of, like, will it solve IMO problems? and that seemed kind of very unlikely to me. But then, yeah, here we are. It can easily solve a lot of IMO problems. So I think we, like, as a community, there was a lot of progress. I think it's, as with many methods, it's starting to be harder to make progress or at least visible progress.

26:15So kind of similar to pre-training, there is still a lot of progress, but the models are already so good that it's kind of harder to see what changes from one to the other as a user of the model. And I think that's also to some extent true for the reasoning now. But they're still increasing the scale of the RL, more environments, more compute spent, and models are still getting more consistent and better. And I think we are at a stage where if we define a benchmark and we can make a relevant RL environment, then we can kind of max it out pretty quickly. And so we are going through benchmarks now very, very quickly.

26:53The major question is generalization and how do you make something that's not just maxing out the benchmarks, but is actually kind of leading to genuine improvements. And that's a very hard question. That's always been the hard question, I think, of machine learning. One of the key questions is transformers as a paradigm get us there. Or do we need something completely different like world models? I think the current approach that the companies are taking is kind of brute force. So kind of we try to come up with as many environments as we can and like all of the types of tasks that humans are doing and turn all of them into environments and then do RL on all of them and hopefully generalizes.