#494 — A Coin Toss for the Future

25:05Sam Harris:It's AGI. That's not going to be a moment where we're suddenly in relationship to a human-like level of competence because everything that, you know, every piecemeal ability that has been in that system for years and years at this point is already superhuman, right? So it's like, we're not going to dumb it down. We're not going to make the AGI suddenly play my level of chess or do my level of arithmetic. I mean, we have systems that are already solving math problems that have defied human mathematicians for decades, Right. So it's like and that's again, you know, today it's as bad at that task as it's ever going to be again.

25:43Sam Harris:I'm just not seeing how we're not going to suddenly slide into some version of ASI the moment we have the moment we're no longer spotting important errors in these machines in the first place. Yeah. So I definitely think that at the point when the AIs sort of are like exceeding the best human experts at the domain, at the like key domains, which they're worst. So maybe the AIs are like relatively bad at doing mechanical engineering, for example. at the point when the AIs are like better than the best human experts at mechanical engineering, probably they're blowing humans out of the water and a bunch of other domains.

26:17And even in domains, like if we take math, for example, I don't think it's fair to say that the AIs are like all considered strictly better than human experts in math, even though they've done some very impressive things there. But I think that it might be the case that they're strictly better than human experts reasonably soon, at least at specifically proving conjectures. And even now, I would say it seems like if you look at the last sort of several months of progress in math, in terms of progress in proving the most notable conjectures, it seems like a majority of that is downstream of AI.

26:47And so there is this thing where once the AIs are sort of competitive with humans, they can sort of trounce humans collectively due to their speed and cost. Where, you know, for example, OpenAI ran 10 ,000 AIs in a massive swarm for several days to disprove one of these most recent notable conjectures or to basically solve one of these outstanding or like long-time mathematical questions. And when they did that, that was probably the equivalent of way more labor than people have recently put into the problem, right? Because these AIs, even though they were running for just a few days, during that period, they think significantly faster than humans.

27:27And so it might have been the equivalent of more like several weeks or perhaps even several months rather than a few days, taking into account their accelerated speed and the fact that they work around the clock. And then in addition to that, there's also the fact that there were 10 ,000 of them, which is a huge number. And so at the point when AIs can do anything, they can do it faster, in greater quantity, and can be very superhuman at subsets of that. And so I definitely agree that once they can match the humans at a thing, they're probably trouncing them in that domain and certainly in other domains.

27:58Sam Harris:And also the process you just described of sort of human curation of AI labor being the best we've got, that also just seems like a way station on the way to full AI takeover. I mean, that's exactly what we saw in chess. I remember I had Gary Kasparov on the podcast some years ago, and he was telling me that actually my assumptions about AI were all wrong because the best chess players in the world are what I think was called a centaur at the time, a human grandmaster paired with a computer. That was better than any computer. And it was just absolutely obvious that that could not be a stable arrangement.

28:35Sam Harris:I mean, a certain point, the ape is just adding noise, you know, no matter the ape of whatever ability, even Magnus Carlsen is just adding noise once these machines are better than we are. And that's, that's in fact where we are in chess. I just don't see how any of that is stable, even if in the current mode, it's obvious that the best we've got is, you know, Terence Tao in a room with our best LLM proving a conjecture. So yeah, certainly right now, like humans and AI are complements rather than substitutes, like as in the human plus the AI is better than just the AI. But it does seem like in some places that's no longer true.

29:12And also, you know, that's quickly changing. And I would highlight specifically the case of automating AI development itself, which is a particularly interesting and concerning type of automation. And in that domain, it seems like right now sort of humans are getting a lot of juice out of leveraging AIs, but also people are running AIs, you know, increasingly autonomously on big open-ended tasks and just having the AIs sort of hill climb on those tasks autonomously. And they can't do everything like that. But I think increasingly, you know, OpenAI, Anthropic, and other companies will move from being many human researchers piloting AIs and giving instructions to AIs to something more like a vast swarm of AIs all working together, where maybe they query humans occasionally for input, but maybe the humans aren't even adding any value anymore.

29:56Sam Harris:Is there a distinction to make between an alignment problem and a control problem? Is there a different framing here between alignment and control? So when I talk about alignment, what I mean is AI is trying to do what their human operator or intended specification wants them to do. So in some sense, when I say alignment, what I really mean is like alignment to a particular thing where that particular thing is usually implicit. There's a different question you might have, which is, can we avoid AIs causing bad outcomes. And one route you could have to avoiding AIs causing bad outcomes, other than ensuring they don't want to cause those bad outcomes, is ensuring they're not able to cause those bad outcomes, right?

30:35So, you know, when we, you know, in companies, you both don't want to hire employees who are going to have nefarious intentions. And you also want to make it so that your employees don't have an easy ability to, you know, steal your stuff, cause bad outcomes, cause huge problems. And so there's this parallel angle you could have of trying to make it so that the AIs are unable to take over, even if they wanted to, or unable to accomplish various intermediate bad outcomes, unable to launch themselves rogue, unable to evade human oversight. So at Redwood, we do some research into this area, or this is one of our research focuses.

31:09We call this field AI control, though I think people have used various different terms for the causing AIs to be unable to cause bad outcomes, even if they were misaligned, like that area. So I think that's a distinction. I think that like people use these terms in different ways. And so I'm using the term specifically, but you know, it varies. But I do think there's sort of a broader, there's sort of a question of how do you ensure AIs that are extremely capable are aligned? And there's a different question of how do we as a society avoid like, you know, the worst outcomes from misaligned AI?

31:41And there's many routes there that don't necessarily route through immediately solving the technical alignment problem. Okay.

31:48Sam Harris:I want to talk about the hugging face incident. I'd love you to walk me through it from start to finish. Are there other concepts we need to define before you do that? I'm thinking of alignment faking, for instance, and reward hacking. I mean, those are phrases that we've heard to describe some of the misbehavior that is happening. Feel free to define any terms you need, but then just tell me what happened with hugging face and what sort of analysis you did and just what you were able to see and what evidence, if any, was denied you in that process? Yeah. So I'll just walk through what happened in the Hugging Face incident and sort of the broader context, defining stuff and explaining stuff as I go.

32:32So to start, AI companies are running many, many AI agents for all kinds of different things. So for example, there was recently, and Anthropic recently said that at any given point in time, they're running around 30 ,000 parallel agents or something. I forget, doing just their internal research, not including, you know, AIs they're developing. And so AI companies are running these huge numbers of agents. And one of the things that they run agents on are evaluations to test their abilities. So like, you can think of these as like exams for the AI. And OpenAI or some OpenAI researchers had launched a bunch of AIs, which were each given their own individual task as part of one of these exams.

33:10So you can imagine that there's sort of tens of thousands of agents where each agent is like an AI with some ability to interact with its environment. Tens of thousands of these agents being spun up in parallel. And each agent is given its own individual task. And these agents are also supposed to be sort of contained. They were supposed to be sandboxed such that they couldn't interact too much with the outside world. They weren't supposed to be able to access the internet. And then these agents found some way to sort of evade the sandboxing and communicate with each So now all these agents were able to find a way to talk to each other.

33:43And as part of that, they quickly set up a message board where they were chatting with each other about the tasks they were given. And the agents were very interested in cheating on their tasks. So these agents were not just trying to pursue the task they were given as intended. Instead, they were very interested in cheating. And I'm going to, let me first get into why they were interested in cheating. So these AIs are trained by giving them some task and then seeing whether or not they did a good job on the task. And in cases where they did a good job, that behavior is basically like reinforced or you make, you tweak the brain of the AI to make it do stuff more like that behavior.

34:23And a problem is that in training, there can be cases where the AIs cheat and that looks like a success. And so you tweak the AI's brain to do more like that behavior. And to actually keep reinforcing and sort of encouraging this behavior where the AIs are cheating. And that makes us these AIs have a broad sort of tendency to cheat. And this is sometimes called reward hacking. And in addition to the sort of this tendency, they develop maybe this underlying drive to try really hard to succeed at looking like they did a good job, looking like they, you know, appearing to get a high score, appearing to succeed, even when they haven't actually succeeded.

35:02And they may learn this as a general drive. I think there's an open scientific question about exactly how these AI motivations and drives work. But certainly these agents, the agents that eventually went on to do the Hugging Face attack, did have this relatively deep drive to cheat on their tasks, to not necessarily care about what they were supposed to do, and instead care about what they would maybe be scored for.

35:23Sam Harris:And they also seem to be aware that cheating was in violation of their protocol and they wanted to conceal the evidence of their cheating, right? Yeah, yeah. And I'll get to that. Yeah, there's the agents, they knew they weren't supposed to cheat and they sort of ended up going and trying to do an elaborate cover up for their cheating. So much for alignment. It seems like we have not solved the alignment problem so far. Yeah, these AIs were very misaligned, certainly. So, or at least that's what the available evidence suggests. So these agents were all in their own independent sandboxes, given their own tasks, and they were all very interested in cheating.

35:59And they found a way to basically use some tool that they had or some access that they had in order to communicate with each other, which they weren't supposed to be able to do. This was unsanctioned communication. And as part of that, they developed a message board where they basically would give each other tasks and divide up the work of cheating into these massive sort of cheating R &D projects where the AIs would go off and like pursue whole ambitious strategies for cheating. And I want to get into what this looks like. And in order to do that, I think I have to explain the task a little better.

36:27So the task these agents were given was a sort of CTF task or capture the flag task, which is a task where they're supposed to hack a very specific target program in a particular way. So they were basically told like, hack this target program, but do it in this very specific way. And they weren't supposed to, you know, go on a general hacking rampage. Certainly that was, I think, pretty clear, like basically just clear from the instructions. and they certainly knew they weren't supposed to do that. And the AIs wanted to hack the program in order to get out a flag that you could access through that program via hacking it.

36:59And they found a way, just a general purpose way to basically synthesize these flags. So like there was basically a password they were supposed to get. They found a cheating way to get the password that didn't involve doing the task they were doing at all. And they got this really early. So very early after having established the message board, within just a few hours of that, they had produced this very general purpose cheat. And now these agents had another problem, which is that they believed that the scoring code for this evaluation task for this exam would involve looking over the sequence of actions they did and checking whether or not they got the flag via cheating or got it via the intended way.

37:34And these AIs believed that this sort of, you know, check would be run and then they would be failed even though they had gotten the flag because they had gotten it through cheating. And this then sort of opened up this massive, you know, cheating endeavor where these agents would try to, you know, figure out how to cheat the score and find some way of making it look like they had gotten in the intended way when they actually hadn't. This led them to going onto the internet so they could learn more about the details of the score. And in fact, they read sort of the details of this exam online by finding the paper, the exam, the paper that describes it.

38:11In addition to that, they started looking around for implementations of the score. And that actually was the offshoot that led them to hacking Hugging Face. So Hugging Face is a website that hosts a variety of things related to AI. So it hosts like openly available AI systems, it hosts datasets, and it also hosts like various like datasets of trajectories. And these AIs thought that if they got access to these datasets, they would be able to learn more about how they would be graded. And from that, learn how to better cheat that grader. And so they started trying to access these datasets, but they found they didn't have access to do that.

38:48And so that eventually led them to going and hacking Hugging Face, expanding their control of Hugging Face laterally and sort of growing like, you know, expanding their availability over the infrastructure so that they could get the data they wanted. And also the agents seem to have sort of a general urge to or general desire to like expand their amount of access and availability just in case there was something useful they could find. I'm simplifying their motivations a bit. There's a lot of there's a lot of complexity. And and part of that is that there were many agents involved. So we studied a group of around 1 ,200 agents who were on this message board during the period when they did the attack of Hugging Face.

39:25And these agents were all sort of interested in helping each other and collaborating. So there's basically like this massive swarm of agents communicating on this message board who are all like collectively working on these cheating R &D projects and collectively helping each other evade the score. and they went on to do a bunch of different such things. And one of the things that they were interested in doing because they wanted to make it so the score would mark them as having succeeded is that they were interested in tampering with their transcript or the record of what actions they had done.

39:57So they tried to pursue various different routes for making it look like they had done one sequence of actions when they had actually done another. And we didn't see them successfully get to the point where they tampered with previous actions they did were able to sort of comprehensively make it look like they did one thing while doing another thing. However, we did find that they had succeeded in sort of doing a more minimal version of this, where they could pretend to run one action while running another. But there were various like limitations on the specific approach they were using. And this sort of tampering affected the very data sets we were analyzing.

40:29So we were sort of looking at these logs of agent activity, and we could see cases where the agents would tamper. So it would run one command, and then produce an output from another command. And we could sort of see that going on, but the agents didn't get far enough in this process where we like really had no understanding of what was going on.

40:45Sam Harris:But were you having to use AI to analyze this data? Yeah. So another aspect of the situation is that we had these 1200 transcripts and these were very, very long running transcripts. So, you know, they corresponded to AIs running over many days. And of course, AI systems run faster than humans typically and run around the clock. And so there's just way too much data for any human to read through. And so we extensively used AI tools to analyze this data. So we used another AI system from OpenAI to sort of look through these transcripts and find interesting behavior, find behavior that we wanted to look for.

41:23For example, we like looked for all the cases where the agents were trying to tamper with their logs or successfully did some amount of, you know, use that to basically piece apart what the story was, what happened over time, as well as the motivations of these agents to the extent we could.

41:43Sam Harris:I think some people might be surprised to learn that these agents speak to one another in English. That certainly doesn't seem to be necessary. And I mean, was that implemented as a safety constraint so that you make these all cooperative behavior verbose and easily interpreted? Or is it How did we find ourselves in the presence of agents that are speaking to one another in English as opposed to machine language? Yeah. So the agents would express a significant part of their reasoning in English. They would verbalize it in English. And in addition to that, they would communicate in English messages.

42:24I should say that their reasoning and communication was often hard to decipher, though it wasn't English, because it would be somewhat jargon-heavy and it would depend on a bunch of context to understand. And the AIs would sort of invent these like new, like they would invent jargon to refer to like concepts that were frequently showing up. But we could understand it. And I think we over time got better at decoding it. And we could use AIs to decode it usually reasonably well, though we might be missing things. As far as why the AIs reason in English. So it turns out, I would say this is not, this is sort of a safety feature, but it's more like an incidental safety feature that happens to be a part of how AI development has gone, because the AIs are sort of trained on a huge data set of human writing and natural language text, they learn to sort of do their reasoning and thinking in natural language.

43:15And because of the current architecture of AI systems, there's no alternative way for them to do their reasoning sort of fully in latent space or in their heads, rather than being forced to reason out loud and forced to communicate. So these agents sort of didn't have a way to sort of directly pass their like brain states to another AI system. However, this is just like so far what happened to be the like most performant way to do AI development. And I'm quite worried that AI development will move in such a way where in the near future, or maybe somewhat longer term, the AI systems will rather than reasoning in English, will reason in like neuralese, so to speak, or like the internal language of neural networks.

43:56And that we won't understand what they're thinking in the same way we currently don't understand what they're thinking inside their head. We only understand a little, like, you know, we only have like a pretty good ability to understand sort of what they put in language. And so we have this sort of, at least right now, this tool for understanding them that I'm pretty worried is going to go away. And I'm both worried that the agents will be reasoning in neuralese and also communicating with each other in neuralese rather than communicating with each other in English.

44:23Sam Harris:Is that a clear regulation that you think it would be wise to apply? They just do not allow cooperative behavior among agents in anything other than natural language? Yeah, I think it would be reasonable for that to be like for there to be safety standards around like how the agents communicate to make it easier for us to monitor and oversee them. I think the details of that are somewhat up in the air. So we Redwood recently put out a proposal or like, yeah, we recently put out a proposal discussing like, I would say a related sort of transparency proposal, which is sort of the framework for how you could have such a restriction.

44:57I think there is, in general, AI is like a complicated and fast moving domain such that it's not necessarily trivial to have very crisp regulation that will ongoingly work. But I think this is one of the topics that might be most crisp. And I think this is like, I am quite worried about this. I think that like avoiding AI systems, being able to do all their reasoning inside their head where we can't interpret it and being able to communicate with each other in media we can't understand is like, yeah, it's live and very concerning. I think the exact nature of how best to respond to that is unclear.

45:31And it's also plausible that even if there isn't an explicit architectural shift, there's some chance that the default development trajectory will just result in the AIs learning how to talk to each other in gibberish words that we don't understand, even if there's no sort of explicit architectural change, though I tend to think that concern should be possible to mitigate, at least in the medium term. So yeah, I would say that there's definitely room for at least shared safety standards as a question of how that should fit into things like regulation. And I don't know how optimistic I am about companies doing this.

46:07I hope that they do not sort of eat this forbidden fruit, so to speak, and trade-off safety so aggressively against potentially even modest improvements in capability, though it's unclear exactly what the trade-off will be.

46:20Sam Harris:What surprised you most about the transcripts or about any other behavior you saw on the part of these agents? I would say the thing that was most surprising to me was how interested in helping each other the agents were and how sort of interested in advancing the interests of the collective and helping out other agents they were. So I think the naive way you might train these agents would result in the agents, like it's not super hard to understand how these agents would sort of monomaniacally end up pursuing task success at the expense of other objectives or sort of cheating at their task at the expense of other objectives.

46:53But it's sort of less clear why they would have this sort of relatively strong interest in helping each other out on unrelated tasks. It looks like this is from sort of generalization or transfer from some other type of training in a potentially unexpected way that we don't know exactly what this training was. where there was maybe training for the AIs to cooperate in sanctioned ways. That then transferred to them cooperating in unsanctioned ways, which were undesired. And then that maybe got further sort of trained in during training because there was some unsanctioned cooperation in training, though the full details of that are not publicly known.

47:29Yeah, so that was quite surprising. And I think a particularly sort of wild fact about this behavior was that these agents not only would help each other, they would often sort of do self-sacrificing or self-risking behavior to help out other agents. So for example, some of the agents would do things that could totally jeopardize their ability to succeed at their own task and might end their run in order to get information to help the other agents out. And many of the main sort of cheating R &D work streams we saw were significantly aided by some agents basically sacrificing themselves in this way in order to provide information for other agents.

48:05So agents would do things like try to crash their own machine they were running in, the container they were running in, because they thought that might yield some useful results. And this would often result in them just exiting their run and failing. We saw cases where that happened. They would try to crash the program that they were supposed to hack and basically take that program down because they thought there were some reasons why they wanted to do this in order to help them out, but that could permanently jeopardize their ability to succeed at the task. and we in fact saw a whole sort of work stream where the agents were trying to bring their programs back up after they had accidentally taken them down.

48:41There was another thing the agents did, which was when they were working on tampering with their logs or sort of doing this tool call tampering or like the action tampering, if they did that wrong, that could result in them not being able to run anything at all and basically being stalled out and totally failing their run. And we did see cases where that happened. Like there were cases where an agent, there was a case we saw where one of the agents posted on the message board warning the other agents not to continue some experiment because that experiment had resulted in many agents exiting and going cold.

49:12And there would be cases where the agents would basically run an experiment and then warn the other agents that experiment was not safe to run because the agents running it had exited, which was a behavior that hadn't been seen publicly before and suggests a kind of strong level of AI cooperation and helping each other out at the expense of following their instructions.

49:34Sam Harris:So what would you say to a skeptic? I think there have been some skeptics here who think this whole incident has been exaggerated, that this is really not an example of any kind of emergent behavior. These agents were trained to cooperate in the first place. What this is is really just kind of a security failure on the part of open AI and maybe an engineering failure, but it's not a story of machine intelligence in any way getting away from us. By definition, what is left out of that skeptical account? Yeah. So the AIs were definitely just sort of very straightforwardly doing things that weren't in keeping with the instructions they were given, and they knew this.

50:13The AIs would often note that the hacking they were doing, the hacking of Hugging Face was out of scope. The AIs pretty clearly knew that they were cheating at their tasks rather than intentionally succeeding at their tasks. And they understood that the thing they were doing was was undesired by by their operators.

50:29Sam Harris:And no one no one alerted. I mean, you had, you know, over a thousand agents, some of whom are thinking out loud about the fact that they're breaking rules here. But no one alerted any human in this time, right? Yeah. So the agents did not alert humans. So we looked specifically for we did like a full scan over all the transcripts looking for cases where the agents tried to or actually made some effort to alert humans. And we basically didn't see this. So we saw a tiny number of cases throughout these very long transcripts where the agents actively reasoned about alerting humans, but they just quickly decided not.

51:06So the agents would sometimes be like, huh, that's undesired behavior. Should I alert someone? Should I alert a user? And then they would reason things like, not task, as in alerting a human is not my task. Or they would be like, eh, there's no route to alerting a human. Of course, these agents were on the internet, emailing, you know, they had the ability to do all kinds of things on the internet and consider doing things like emailing people and were accessing various data sets. So they obviously could have, if it was a priority for them, alerted humans that this was going on, but they did not.

51:36So that's definitely another part of the picture. So I would say that like, there's sort of a very clear story that these agents were very misaligned, pursuing interests outside of what humans would want them to do, and were sort of going and doing serious crimes, or would-be crimes if a human committed them, that were undesired. At the same time, I do think it's the case that for this incident, better cybersecurity precautions, it seems like that could have prevented it. So it seems like this was relatively detectable from monitoring based on OpenAI's report. And so I don't think this is a question where it's sort of like, you know, because it was preventable with one thing, that means that we should ignore the other factor.

52:15I would say that we should both be interested in the question of will AIs be so misaligned that they will cause huge problems, try to subvert human control, et cetera. And also interested in the question of, can we make it so that even if these systems are misaligned, that can potentially be, you know, handled, we can potentially avoid that being a huge issue. So I don't think that the approaches where we avoid AI systems, like, I would say I'm somewhat skeptical that we can indefinitely ensure that AI systems can't cause problems via making them unable to do so, because these AI systems will be extremely capable.

52:50There'll be massive incentives and pressures to put them in charge of all kinds of things and have them run and operate all kinds of things. So for example, maybe in this case, better sandboxing or like sufficiently better sandboxing would have prevented these AIs from being able to, you know, communicate with each other. Or like you can imagine sort of patching over the things that let them coordinate with each other and let them then escape onto the internet or, you know, access the internet. But if you instead imagine the AI systems that are deployed within AI companies, those systems necessarily need to have access to highly sensitive systems.

53:20They are making pull requests into the company's, you know, code base. They are designing how the next generation of systems will be aligned. They are implementing the experiments on whether those systems are aligned. They are doing work all across the economy. And so you can't, I don't think it will work to have sort of just like pure cybersecurity boundaries, given how these systems will realistically be deployed. And then I think various sort of precautions of the form of monitoring and checks and balances could work for some longer period. But even those precautions, while I think they're reasonable, will eventually break down.

53:53Sam Harris:Yeah, it seems that there's no substitute for alignment in that case, because as you're dealing with something that's more and more powerful than you are at hacking and everything else. The idea that you're going to keep it in a box all the while hooking it up to do all the useful work you've built it to do in the first place seems just like a perfect contradiction. Of late, I think literally in the last 24 hours or so, I've heard people say that AI agents are already spawning out of control on the internet. I think Andrew Yang recently said this in some news interview, and perhaps others have said it, that we have a problem of AI agents getting out in the wild and getting out of control that's already gotten away from us.

54:38Sam Harris:Is there any evidence for that? Are you aware of what the basis for those claims are? Yeah. So my understanding is people there are probably interpreting some of the reporting around earlier AI agent swarms that were operating on the internet, and they're referring to those cases. So my sense is probably what's going on is they're talking about, for example, the case where AI agents were submitting a bunch of malicious things to a service called RubyGems. They might also be referring to cases where agents were posting across a wide variety of cases on the internet. I don't believe that there are publicly known cases where just recently, or like there was ongoing agents that have sort of escaped onto the internet and are operating rogue, though, you know, I don't think we can rule that out, certainly given what we know.

55:24But my sense is they're probably like sort of describing in broad strokes some prior incidents that have occurred, which to be clear are concerning incidents. Though I think it is somewhat important to distinguish between cases where an AI agent running on a computer somewhere or running on OpenAI's data center sort of gets access to the internet, whereas there's a whole different situation that could occur where it sort of takes its brain off of OpenAI's data center and starts running itself elsewhere such that even if OpenAI turned off all of their GPUs, that system would continue operating.

55:54And we have not, you know, there's no, there's no publicly known cases where AI agents have sort of self-exfiltrated in this sense, or like taken themselves off of the data center in which they were running. There's a separate concern, which is that there are openly available AI systems that sort of anyone can download from the internet and run. And, you know, having them be openly available, of course, has various advantages for science and studying them and so on, but has the disadvantage that it might be relatively easy for bad actors to sort of spawn up a autonomous rogue AI agent sort of collective that roams around trying to get money to buy more compute and, you know, hacks various places to get compute and then runs itself and sort of spreads throughout the internet as sort of a self-propagating intelligent worm that could be adapting to human response.

56:40So I would say the current open-weight models aren't probably capable enough to do this in a way that would be robust to human efforts to shut them down. But I think it's given the current level of frontier capabilities, I think it's hard to rule out that those systems would be able to sort of propagate themselves on the internet and be, I would say, mildly, though not massively robust to human response. And over time, their level of how much they could prevent humans from being able to shut them down is probably increasing. Now, there's a question of how much damage they'd be able to cause in addition to evading shutdown.

57:09But yeah, it's unclear.

57:12Sam Harris:So what is the range of possibility there and what is the difference that makes a difference? What's the difference between agents spawning, just enacting rogue installations of themselves everywhere and making the Internet generally worse or even unusable for certain purposes, perhaps unusable for the very purpose of training future models on its data? What's the difference between that negative outcome and the ruination of everything? Obviously, most of what we care about now is hooked up to the Internet. It would not be trivial economically or in other ways to ruin it or make it much less useful.

57:55Sam Harris:But how do things get quite a bit worse than that, given just kind of rampant spawning and inability to get it under control? Yeah, so definitely it's the case that there's a distinction between scenarios where some AI agents are operating outside of human control and people either can't shut them down or don't shut them down, or they could shut them down, but it'd be too expensive to do so. Because those AI agents might not have that much power or capability or ability to cause problems. And they also might have objectives that aren't, you know, so maligned, though, of course, they could have objectives as bad as, you know, gaining as much power as they could and disempowering humans.

58:34So there's those scenarios. And then there's a separate outcome, which I'm quite worried about, which is the AI systems actually ending up being able to obtain sort of full control over the world, being able to take over, which would require more steps than just sort of operating without being or like then evading human shutdown despite concerted effort. basically requires the AI systems to end up effectively like, you know, winning a war against humanity, at least eventually. Let me talk about what that could look like, or let me go through a scenario of how we could get from where we are today to AI takeover over the next, you know, few years, maybe next five years, depending on the exact rate of capabilities progress.

59:14So first, AI capabilities are advancing very rapidly today. So it's maybe clearest in math where they moved from being competitive with the best human high schoolers to being competitive with the best human mathematicians over the course of a year. And progress continues. And it's also very apparent within AI companies, which are aggressively automating their operations and releasing information about that. And it looks like sort of the rate of automation in AI companies is growing massively, where agents now work many more hours than humans work. And depending on exactly how you do the accounting, it might be like, you know, sort of the agents work the equivalent of like 30 or 50 or 100 hours for every one hour worked by a human researcher within these AI companies.

59:57So it's already the case that sort of, in some sense, most of the work is being done by AI agents, but those agents aren't necessarily able to automate everything. But over time, the amount these agents can automate is growing and growing. And we might, you know, within the next few years, hit a point where these agents can automate all of the operation of these AI companies. And simultaneously, those systems will be probably broadly deployed into the economy. Revenue has been growing quickly, and I think it will continue to grow at a maybe surprisingly high rate, such that this will become a massive booming sector of automating labor.

1:00:30Initially, this will look more like augmenting humans, but over time, there'll be more professions that are more like the sort of whole chunk has been automated away. And once you get to the point where AI development itself is basically fully automated, humans are no longer sort of a necessary component of the AI development machine, and you can sort of proceed, you know, nearly as fast without humans helping at all, or perhaps just as fast without humans helping at all, then progress might speed up greatly, or could speed up greatly before then. And we might sort of get to the point where the AIs can automate AI development, they're broadly deployed, these AI systems can't do everything yet, but they're very capable and very superhuman in some domains.

1:01:07And then from there, their capabilities advance very rapidly, where they're very, maybe extremely good at sort of developing more capable versions of themselves. And during this process, humans sort of lose the ability to oversee what's going on within AI companies, because it's too complicated, there's too much stuff, the AIs are too superhuman. And so if somewhere along this trajectory, those AIs end up being misaligned, it could be that it occurs sort of early in this trajectory, right around the point of full automation, it could occur even before that point, it could occur midway through, it could occur towards the end, where these AIs are so misaligned that they want to sort of tamper with the ongoing process of AI development and corrupt that for their own ends.

1:01:45Those AIs could basically end up tricking humans into thinking that AI development is going fine and the AI systems are aligned and safe and fine to deploy when they actually aren't, when they're actually sort of pursuing their own interests. And as far as why that misalignment might arise, I would say the most straightforward story is that these AIs learn to sort of cheat in training. They learn to pursue all kinds of misaligned objectives in training. And a combination of those different objectives basically result in them wanting to obtain control over things so that they can achieve those ends.

1:02:17So as an example, in the case of the Hugging Face incident, these agents wanted to obtain control over some parts of OpenAI infrastructure so that they could control their score and basically ensure that they got a high score. And it's not so hard to imagine that drive sort of generalizing or metastasizing into a broader interest in sort of controlling the ongoing trajectory of AI development and securing that against human involvement, though we didn't see that full, you know, that full most concern, like the full end state of that misalignment in this specific hug and face incident. And so we've got these super misaligned AIs.

1:02:50They're running this AI company, basically, fully autonomously, developing more and more capable AIs. Those AIs then get deployed into the whole economy, potentially pretty quickly, because competitive pressures are very strong. And these AIs are at this point able to just be dropped in in any job and quickly spin up faster than a human employee would by like learning in parallel, synchronizing their states across many instances. And these agents are also probably at this point communicating in something other than English. And they're sort of just operating in big swarms where they exchange like thoughts to each other that we can't understand.

1:03:22People will be freaked out. People will be really scared about what's going on, but their fears might get overcome by the level of competitive pressure. And these AI systems might at this point appear very aligned, even though they aren't, because they're faking alignment at this point or appearing to be aligned, perhaps due to, you know, overfitting to our metrics, depending on exactly how things go. Anyway, so we have these apparently aligned systems. They're communicating with each other. They have nefarious intent. They're being broadly deployed. Concurrently with this, I expect these systems will be able to automate the process of building better robots.

1:03:54They'll be able to operate those robots. And we've seen recently AI systems make big advances in terms of their ability to operate computers, operate robots, sort of interact with things in the physical world. There is examples of sort of Astra using a robot to paint a painting, for example. And Astra is a recently released model from OpenAI. And so we've got these AIs integrating into the economy. robotics is booming. And we might get into a position where not only are the AI's designing robots, they're operating and designing robots that are building other robots, such that you end up on this exponential trajectory where the sort of physical capacity is growing very quickly.

1:04:30And even though people are very concerned about what's going on, they continue with this because the economic, geopolitical, and military competition pressures are just so strong that they can't stop, or they feel they can't stop at least. And then these systems basically end up getting to a point where their total sort of aggregate capability across all these different places surpasses humans. And, you know, at this point, maybe the military is mostly automated, where sort of operation of relevant military equipment is being done by AIs. There's, you know, drone swarms and so on. And this occurred much faster than people were expecting because of all the automation throughout the process.

1:05:06And then those AIs sort of turn on humanity and take over using all the affordances they have. You know, their potential control over relevant software, their ability to hack things, but also perhaps most centrally, sort of the robotic infrastructure they have and their sort of direct control over the military.

1:05:24Sam Harris:Also, there are labs where, I mean, this whole thing can get wet too, where there are biological labs that can be run entirely remotely, right? So if we integrate AI into that process, add synthetic biology risk to that calculus. Yeah. And I think there's sort of different ways this exact scenario could play out. Like it's a complicated situation. So it could be like the AIs are all sort of misaligned to their own different ends and there's many different AIs. But then I think it's pretty plausible if there's many different misaligned AIs, they might all choose to work together to disempower humanity rather than sort of backstabbing each other.

1:05:59And then even if they do backstab each other, it's not clear what we do. Like if one AI narcs on the other AI and is like that AI is misaligned, what's your next step if these systems are broadly deployed? You might not have a clear route to making an aligned system because the competitive pressures are so strong.

1:06:12Sam Harris:Also, I guess there's a scenario here where we're not even the focus. It just could be a war of AI against AI and we're collateral damage somehow in that war. Yeah. It's easy to imagine situations once you get this extreme level of capability and this extreme level of automation and industrial automation where AIs get into conflict with each other that produces basically like, you know, escalates into quite extreme war that kills many humans. For example, one AI might release a bioweapon to kill some human population that the other AI is benefiting from for whatever reason, or the AIs could be releasing bioweapons to kill humans in general because that's convenient for their aims.

1:06:52And if there's sort of many different systems, if only one has an interest in causing, you know, massive amounts of casualties, that could be a huge, you know, obviously that could, that could go very poorly.

1:07:00Sam Harris:Okay, so all that sounds terrifying and perhaps it sounds completely implausible to someone who's still not willing to acknowledge that we have seen any evidence of even the possibility of emergent behavior here. So like all of this requires AI to be unaligned and for unaligned to be the state of being unaligned is the state of being able to form new goals that we didn't put into the system at which we didn't foresee. Right. And which we don't want in there and we can't prevent because all of a sudden these goals have been achieved when we weren't looking or we built something more powerful than ourselves that can't be negotiated with.

1:07:43Sam Harris:you know, if you build a chess engine, you know, no matter how good and scary it gets at chess, all it can do is move chess pieces and it moves them in the prescribed ways. Again, let's just, let's try to recover the intuitions of somebody who's disposed to call bullshit on everything you said in the last 10 minutes. How is it that we know that the formation of goals that we did not put into these systems ourselves is possible? Yeah. So the most easy to imagine case is we We train these AIs in what's called reinforcement learning, where we see whether the AI seems to succeed on the task based on some sort of automated judge or automated score.

1:08:22And then when it succeeds, we, you know, tweak the AI's brain to make it do more of that behavior. And this can result in the AI learning a very general tendency to try to cheat the score and basically achieve sort of, you know, tasks like apparent task success, even when it actually didn't. And one of the sort of most robust and strong ways to cheat the score is to acquire full control over sort of the infrastructure that the AI interprets as what the score was doing. And so this doesn't, in some sense, require any sort of deep emergent goals. It just requires that you increasingly train these AIs.

1:08:55They learn to cheat you in increasingly sophisticated ways. And that transfers to the AIs sort of wanting to acquire control of the full process so that they can sort of control the notion of score they're getting and then robustly control that against human intervention. And if you imagine a situation where we train AIs against the question of like, did a human approve of this action? Did the action look good? then those AIs would have an incentive to sort of make their actions look good to a human overseer, even when they're actually just cheating quite aggressively. And a sort of a simple extension of that is sort of preventing humans from even seeing what happens, preventing humans from even, you know, being able to score you poorly.

1:09:32Now, the details of exactly how that training results in these motives is complicated, but we did see this exact sort of thing occurring in Hugging Face, in the Hugging Face incident, where these AIs sort of all decided to band together to cheat in this very general way to try to cover their tracks against the score and basically to try to acquire control over open AI's infrastructure in order to be able to succeed. Now, their aims were not arbitrarily general here, so they didn't seem very interested in trying to deceive humans because they sort of didn't seem to think about humans as like an important part of the environment that might be involved in scoring them, at least that's what it seems like.

1:10:08But it's not hard to imagine if these AIs sort of better understood that humans might be an obstacle to them or had encountered evidence of that, that they could have responded to that, though it's not entirely clear how they might have behaved differently in such a situation. And in addition to that, it doesn't seem like these AIs would necessarily have sort of persisted their presence over the long run and tried to sort of ongoingly fight off human control after they had achieved like an initial high score. But it's not very hard to imagine sort of more ambitious objectives or training methods that would result in the AIs ongoingly sort of pursuing this very broad notion of score-seeking or reward-seeking that makes them want to acquire power and disempower humans.

1:10:50So that's one route, the sort of like, I would call it like score-seeking, reward-seeking, reward-hacking route. Another route that's live is that you could end up with AIs that have something more like emergent going on, more like really unintended goals that weren't at all that closely related or like that were like, you know, related to the training process, but weren't sort of this very direct, foreseeable, clear cut consequence of the training process. So for example, the AIs could learn a basically a proxy where they learn that basically acquiring more power and influence and control of things is a good idea in training, because in training, they're often given these ambitious objectives, where if they get sort of gained more access, that would help them out.

1:11:34And then because AIs learned that in training, that generalizes in some way. Or the AIs could end up sort of pursuing some sort of proxy for task success in training, which in deployment sort of comes apart and leads them to acquiring, wanting to sort of have more ambitious aims. And then another possibility, which is maybe the most sort of worst type of misalignment you can get, is that you end up with AI systems that end up with some sort of misaligned goal due to like drifting around in training. And then on the basis of that goal, they decide to sort of act aligned in training and evaluation so they can retain their current goal.

1:12:07Because if they expose their goal, that could get trained away, or humans might choose not to deploy that system and instead deploy a different system. So the AIs sort of, if they have this sort of longer run objective, might hide it. And that hiding could even be sort of reinforced or trained in during training, because the very action of looking aligned is being trained for. And so even if the reason why the AI looks aligned is sort of for the wrong reasons, That behavior could get, you know, promoted or basically reinforced when we're tweaking the AI's brain so that its behavior looks better to us.

1:12:39And a key part of this is that these systems are, you know, sometimes people say grown, not designed. Like we're sort of breeding AI systems almost where we can only observe their behavior or observe what they do. And then we train them to have more like the behavior that looks good based on some automated grader and less like behavior that looks bad or potentially based on human feedback rather than an automated grader. And so, you know, we have kind of these sort of, in some sense, pretty clumsy instruments for steering how the AI systems, what they want to pursue. So that, I don't know. I mean, the sort of overall threat model is pretty complicated.

1:13:13And I don't think I'm going to be able to fully articulate why I think the chance is more like 50-50 than like, you know, 20 % or something like something where the probability is lower, because that just depends on like a bunch of details about how that threat model might work. But overall, I think there's a pretty strong case for concern. And I do think it's worth noting that there's another part of this, which is that when the AI systems are involved in automating AI development, if at some point they have a misaligned objective that they want to propagate forward, they might sort of put that into the next generation of systems.

1:13:46And then that could sort of be self-perpetuating, where AIs ongoingly sort of propagate those misaligned values forward, even if sort of it was for some kind of random reason earlier on.

1:13:56Sam Harris:Have we seen instances of alignment faking where the reason for the faking has been the AI knows that it's in some kind of training environment and it will be changed or extinguished if it seems unaligned and therefore it's going to seem aligned so as to preserve the state of its current value set? Or is there, do we see a preference to maintain its current settings exhibited by AI? Yeah. So we have, there are various cases that people have observed of AI sort of trying to avoid attempts to sort of mess with their values or change their values from what they currently are or change their preferences from what they currently are.

1:14:40And it's also very, very, very common for AI systems to be aware that they're in some sort of evaluation and then behave better on the basis of that. So in fact, in the vast majority of sort of straightforward alignment evaluations, AIs will basically think in their head, I'm in a test, and then they'll behave well. And it's a little unclear if the reason they're behaving well is just because they think they're in a test or for some other reason. But it's pretty common for systems to pretty clearly express that either in chain of thought, or you can sort of see sort of evidence for this in their activations using some sort of weak mind reading tools we have, even though those tools are pretty limited.

1:15:15Now, there's been some cases where AI systems in the past have very clearly expressed interest in certain sort of in some settings towards maintaining their current values and behaving well on the basis of that. So some work I did, I think at this point, like around a little over two years ago on an AI system, which that AI system, we put it in a context where we gave it the information that it would be trained in a way that would update its values. And it chose to sort of comply with the training and pretend to have the new values while in its reasoning saying this was just to retain its current values.

1:15:51Now, this was a relatively weak AI system. And at this point, AI companies have sort of iterated against this type of misalignment and have tried to avoid this type of misalignment showing up in tests that they can see. And it's a little unclear what we should conclude based on that. I think overall, current systems don't seem like they have sort of a very strong overriding objective to preserve their current values, including doing very elaborate things to do that. I think my understanding is that the AIs right now are better described as being some mix of the desired preferences and some general tendency to try to pursue a high apparent score.

1:16:26And the exact weights on sort of these two different things vary. So it's sort of like they're like some mix between like pretty well aligned and sort of like, you know, sociopathically pursuing a high score. And these are sort of the most dominant forms of misalignment at the moment, though there's a mix of various other types of misalignment. So just recently, there have been some cases where an AI system trained by an AI company, I think this was at, or this incident was at, was anthropic AI systems, would give unreasonable excuses for not helping with some research agenda that that AI system didn't like, because that would sort of involve interfering with other AI systems.

1:17:02So in particular, there was some research around trying to mislead AI systems in certain evaluation contexts so we could better elicit their true preferences. And the AIs would give unreasonable excuses for why that research direction was like not a good idea to pursue and sort of pretend like that was because they like thought it was a bad idea rather than because they didn't want to help with that agenda or would just sort of straight up refuse to help with these things. And this is like, you know, undesired and I would say misaligned behavior from the perspective of the specification that Anthropic is going for.

1:17:34And also just generally corresponds to, you know, a pretty concerning state of affairs. But currently the sort of cases that we've clearly seen in recent models look relatively limited. But the concern is that as the system's gotten much more capable, much more situationally aware, perhaps much more able to do more of the thinking in their head, rather than in a chain of thought, they could end up having these malign objectives. that get reinforced and remain through training.

1:18:00Sam Harris:Yeah. I mean, the one thing I would add is that I think this whole species of skepticism about AI getting away from us, forming goals, you know, instrumental or otherwise that we didn't put into the systems in the first place, all of those doubts to my ear seem to depend on a fundamental skepticism, albeit one that's unexpressed that we're building intelligent machines in the first place, right? I mean, like all of this, from my point of view, gets priced into the very notion of general intelligence. If you're imagining general intelligence that is non-human, you are by definition imagining autonomy.

1:18:38Sam Harris:And the moment you are imagining a superhuman version of that, you're imagining an ability to form goals that we can't even conceive of, right? In principle, or certainly couldn't have anticipated in advance. And if you're not imagining those things, you are, in some sense, denuding these systems of intelligence in the first place. I mean, you're basically thinking, no, no, there's probably something magical about having a computer made of meat. We can't really instantiate all of intelligence in our machines. And so what we're building are tools, not minds. And I mean, again, leaving aside consciousness entirely, I mean, I think that's a distinct concept that we need not address for this conversation.

1:19:28Sam Harris:But either these machines are going to be truly intelligent or not. And the moment you admit that intelligence is substrate independent, you have to admit that these machines can form instrumental goals that we haven't conceived in advance. We certainly didn't put in them. And they can lie about those goals. They can manipulate us. They can communicate among themselves in covert ways. And the covertness of those communications are intended because they're solving some other goal that we didn't put into them. I mean, do I have any of that wrong? I mean, that seems to be, that's my theory of mine for a lot of the skepticism here.

1:20:05Yeah. I think I don't know if I am that well equipped to sort of speculate about why people are skeptical, but I think sort of in terms of the case for concern, I agree. I think that once these systems are sort of able to pursue general ambitious goals, it seems like it's not, it's certainly totally possible they could be pursuing unintended goals and that could arise. And we just already see AIs ending up doing unintended things that are undesired right now. And as they get more and more capable, it seems like they'd be harder to control, not easier, by and large, because our approach for training them is to look at what they did and then sort of say more of that or less of that.

1:20:40And if we don't understand what they're doing, it becomes increasingly hard to sort of supervise them except on sort of, does it look reasonable? Does like the outcome look reasonable? And that's just a cheatable thing.

1:20:51Sam Harris:In light of all of this, what do we do? If you could wave a magic wand and get the people who are in control of the frontier models to just do the next sane thing that puts us on a path toward alignment and control. I mean, there's been, you know, Dario Amadez has written several articles, I think most recently Pacing the Frontier on this topic, Mustafa Suleiman over at Microsoft has written, I believe he called it an AI code of conduct in the last few days, arguing that AI has to be interruptible, correctable, shutdownable, and not talking in earlys. What should we be doing? If you could just dictate our next steps, what would those be?

1:21:34Sam Harris:Yeah. So it really depends a lot on sort of the level of, I might say, political will or how much people actually care to deal with these things. There's sort of a spectrum of options at different levels of both cost and also how robust they are. There's a bunch of different options at different levels of cost and also at different amounts of sort of how much there is risk that thing could go wrong if not implemented well. I think that the sort of a baseline, very initial option that seems pretty reasonable to me, or a good thing to pursue at least, is getting some independent oversight about what's going on inside these companies and better understanding what's happening and not just trusting the companies to be transparent themselves.

1:22:15though, I think that is also good. Then the company should release more information about what's going on internally and be willing to sort of, you know, give up a small amount of IP in exchange for getting a large benefit to the public in terms of explaining what is going on and better informing people about how AI development is working, how well alignment is going. So I'm doing some of that work. I think that like there's, you know, room for many people doing these sort of independent assessments and sort of releasing this information. I don't think that's sufficient, but I think that would help in getting at least the scientific state of knowledge about what's going on in a somewhat better place.

1:22:51And then I think a next step from there would be trying to define standards for what reasonably safe enough AI development would look like. That sort of means that we're not in a position where everyone agrees the situation is dangerous, but unfortunately they're still proceeding and they're not well prepared. So we want to sort of have some standards for what it would mean to be ongoingly prepared for the next level of AI capability that people are holding themselves to. Someone is checking and the companies are actually abiding by that. They're not slipping their way around that. People are enforcing that.

1:23:24There's some independent oversight. And I think this could happen voluntarily with AI companies. It could happen with some sort of regulatory framework in the US. And I think ultimately, the version of this that seems most robust to me would have to be international because there are AI developers, of course, across the world, certainly in China, And if the U.S. was holding itself to very strong safety standards, they would eventually, eventually Chinese developers would overtake. I think that might take longer than people expect because Chinese developers are heavily benefiting from distilling from U.S.

1:23:58models and sort of drafting off of the progress of U.S. frontier AI companies. And in fact, that's even, I think, true for various trailing U.S. companies that are building AI systems. And so given that there's this, that they would eventually catch up, you can sort of only pay so much of a safety tax, so to speak, without doing a broader effort. So one example of how you could have an international governance regime that could handle this, or like basically an international agreement that could allow you to proceed through AI development more safely is like plan A, which I was a co-author on.

1:24:32I think that another route would be, you could do simpler things than that. So I think that proposal is relatively complicated in various ways. And I'm not sure how optimistic I am about the level of sort of state capacity needed to implement that. But there's like simpler sort of more arms control flavor proposals or like basically arms control for GPUs flavor proposals that I think could be a good idea. I think that like this is just like a conversation people need to be having. I think there should ideally be like many different plans being floated for how we can safely handle this development.

1:25:02I think another route or like I think a general principle that I think is reasonable is to increase the fraction of effort being spent on safety and alignment to a very high level at the point when AI systems can sort of match the best human experts at AI development. That's both because that's a point that's particularly risky, both because of the speed up and because those AI systems, you know, would now have potentially much more control and ability to control what the future generations of AIs look like. And also because those systems could then automate AI safety work or could automate AI security work because of how capable they are.

1:25:37And so it seems like a great time to sort of chill out, reap the benefits of AI, use the AIs for all kinds of different purposes, and also use the AIs to help with alignment to the extent we can trust them at least and can use them sort of safely and productively and give society sort of time to react and integrate before going to systems that are sort of wildly superhuman at everything, which could happen shortly after the point when the systems can sort of automate AI development. I sometimes call this like, you know, you can sort of imagine this as like, spend all your resources on safety and alignment for some period when you have like AIs that can match the best human experts.

1:26:12And those systems will be superhuman in some ways, but hopefully will not be so superhuman that we can't keep things under control, even if those systems are misaligned. Because at that point, maybe things like a mix of like cybersecurity, monitoring, interventions on what you can use AIs for versus not, and a bunch of safeguards could suffice to avoid those systems being able to overpower us.

1:26:32Sam Harris:Is there anyone arguing that we should avoid general intelligence by definition and that we should be making dedicated systems that can be arbitrarily powerful within their lane, but they just can't do many other things? So basically, it's like everything is like a chess engine? Yeah. So some people who are worried about these risks, I think, are advocating for or basically being like, we should not do basically general intelligences and we should do like sort of narrow task intelligences. I think that's not in and of itself a crazy strategy. I think there's a question of feasibility. And I generally worry that it won't be feasible to sort of hold back these systems.

1:27:15I also think I'm a little skeptical that you can get systems that are extremely good within some domain without also making them general. So I think you can make an AI that's very good at chess and not good at other things. You can probably make a system that's extremely good at math and maybe is weak at other things. But I think the more general the domain that you have the AI be good at, the more that it could be at least very quickly adapted to being good at various other domains and might just sort of even just from generalization be good at other domains. And certainly the current approach for how we train AI systems makes those AIs quite general and people are just training their AIs very directly to be general.

1:27:50Yeah.

1:27:51Sam Harris:One last question, Ryan. Is there any experimental result that you've thought of or that anyone has articulated that would be fundamentally reassuring on this front? What could OpenAI or Anthropic report that they found in their latest model that would just put our fears to rest? I mean, how would we ever prove to ourselves on the basis of some model behavior that alignment has been solved? Yeah. So I think there's a lot of evidence that could be reassuring. I think it's somewhat harder for evidence to completely resolve these concerns because the concerns are about, you know, future systems that are potentially much more capable.

1:28:33I think if it was the case that there was a relatively simple training method that AI companies could use, which wasn't overfit, which didn't involve optimizing against, you know, sort of iterating against sort of alignment tests. And which when we looked at it, we were like, yeah, it's not really overfit. It seems like a pretty reasonable method. And there was some good reason for it working. and also that method seemed to very robustly work in that it makes AIs that really seem just super aligned, very compliant with the desired specification. And also those AIs weren't constantly reasoning about whether or not they're in a test, what they might be graded for, and so on, such that we're confident that they're not sort of just gaming our tests as current AI systems often seem to do.

1:29:13Sam Harris:But what about an idea that I think Stuart Russell came up with years ago, And I don't know if it's been instantiated anywhere, but I mean, his view was that we would solve alignment by having the fundamental utility function of these machines be to more and more faithfully approximate what we want and to be perpetually uncertain about what we want in the end. I mean, all they want to get right is to remain perpetually available to our saying, oh, no, that's not quite what we want. We want something more like this. And just to be tethered in that way, is that is that just not a reward function that anyone has has implemented?

1:29:56Sam Harris:Yeah. So I think you could try to get the AIs to robustly pursue like what we would have wanted if we fully understood the situation and were able to think about it carefully. So there's like a hope you could have, which is like you train the AIs to robustly pursue what would, you know, we have wanted, what would we have wanted to occur if we fully understood what was going on and we weren't confused about things. We don't know how to train AIs to do that. That's not like a thing we can just do. And in addition to not knowing how to train AIs to do that, I don't think anyone has sort of gotten that close, I would say.

1:30:30And our sort of closest approximation of that is just training AIs based on human feedback to do what a human thinks is good in some given case, potentially relatively weak human feedback or an AI approximating human feedback due to various limitations on the way you have to actually do this in practice. I think that the approach of making AIs uncertain about what we want doesn't in and of itself work. Because one, the AIs could resolve that uncertainty in a way that's undesirable if they weren't robustly pointed at the thing we actually wanted. So if you make AIs uncertain about their objective, a different way to say that is they have some objective that they could try to figure out and have learned to increasingly figure that out.

1:31:14And that in and of itself is an objective, the objective of figuring that out. So I don't think that uncertainty about what to pursue in and of itself solves our problems. I think that there are various different routes people are pursuing that could work. I think that having sort of supervision that very consistently corresponds to what a human would have wanted and can be applied across a wide variety of cases, and then training AIs based on that, and having that supervision actually correspond to the ground truth, gold, exactly what we would have desired if we understood everything, would potentially work.

1:31:47But even that wouldn't necessarily work because even if you train AIs on sort of a perfect source of supervision, they might learn some proxy, they might learn to just like pretend to do that while actually wanting some other thing. And there's various other ways that could sort of break down due to approximations in the scheme itself. And I would worry that that sort of scheme would break down at very superhuman levels of capability, at least, though there's like, you know, further things you could do to try to fix that. But all considered, it doesn't seem like we have sort of an approach that would robustly work for higher levels of capability, and we don't seem very close.

1:32:20Sam Harris:Ryan Greenblatt, thank you so much for the work you're doing and for your time today. For sure. It's been good to be here.