Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken

25:14So I actually implemented the ability to like put images into our feature activations because this was all on Cloud 3SANA, which was one of our first multi -model models. So we only trained the Sparrow Sauton Coder and like the features on text. And then a friend on the team put in an image of the Golden Gate Bridge And then this feature lights up and we look at the text and it's for the golden gate bridge. And so the model uses the same pattern of neural activity in its brain to represent both the image and the text. And our circuit's work shows this again with across multiple languages. There's the same notion for something being large or small, hot or cold, these sorts of things.

25:57But like strikingly, that is more so the case in larger models. We think like actually larger models have more space so they could like separate things out more. but actually instead they seem to pull on these like larger abstracts. They own better abstractions. Yeah. Which is very interesting. Yeah. Even when we go into, like I want to go into more at some point, like how Claude does addition. When you look at the bigger models, it just has a much crisper lookup table for how to add like the number five and nine together and get something like 10 module of six, six module of 10. Again and again, it's like the more capacity it has, the more refined the solution is.

26:32The other interesting thing here is with all the circuits work, it's never a single path for why the model does something. It's always multiple paths and some of them are deeper than others. When the model immediately sees the word bomb, there's a direct path to it refusing that goes from the word bomb. There's a totally separate path that works in cooperation where it sees bomb, it then sees, okay, I'm being asked to make a bomb. Okay, this is a harmful request. I'm an AI agent and I've been trained to refuse this. And so one possible narrative here is that as the model becomes smarter over the course of training, it learns to replace the short circuit imitation C -BOMM refuse with this deeper reasoning circuit.

27:18And it kind of has kept the other stuff around to the extent that it's not harmful. With that being said, I do think it's your point on all these models as sample efficient assumes. Currently, we do not have evidence that they're a sample efficient assume. So I think we have evidence of total complexity ceiling. Like they're currently nothing that provides you have a clean enough signal, you can't teach them, but we don't have evidence of we can teach them as fast as humans do. And we would prefer that we get learning on the job. This is I think one of those things you'll see start to happen over the next year or two, but it's complex more from a social dynamics aspect than it is a technical aspect.

27:55Yeah, I'm not sure about that. I mean, I've tried to use these models to do work for me. And I'm like, I like to think I'm sort of AI forward. Yeah. You're at the TARC -CatchFile. That's a huge problem. And it's not because somebody vetoed it or something. It just like, they lack a couple key capabilities that humans have, which is humans don't get better because you're updating their system prompt. They get better because they have like, they're opening the weights. Yeah, yeah. In a very low friction way, that's much more deliberate. And also, they're not resetting at the end of recession. Models can get pretty intelligent by in the middle of a session when they've built up a lot of context and what you're interested in.

28:40But it gets totally reset at the end of the session. Yeah, so my question is always like, are you giving the model enough context? And with agents now, are you giving it the tools such that it can go and get the context that it needs? Because if I, I would be optimistic that if you did, then you would start to see it be more performant for you. And if you created like the Duoch -Ech podcast, RL like feedback loop, then the models will get like incredible at whatever you wanted them to do. I suspect. Yeah. But they're currently using the mechanism for you to do that with the models. So you can't like say, hey, here, like have some feedback about how I want you to do something and then like, you know, somewhere on some server, like, you know, whizzes up and, and like, The current thing is there's a text -based memory, right?

29:19Where it goes and forwards things about what you wanted. And it puts it in the prompt and tries to build its own scaffolding and context. I think an interesting question over the next few years is whether that is totally sufficient, like whether you just like this raw base intelligence plus like sufficient scaffolding and text is enough to build context or whether you need to somehow update the weights for your use case and like some combination that are off. But so far we've only explored the first. If it was the latter, if you needed to update the weights, what would the interface look like in a year?

29:53What is the, I guess, if you wanted to interact it with a human, what's happening on the backend? Is writing practice problems for itself? Is it building actual environments for itself that it can train on? It's a good question. Ideally, want something as low friction as possible for someone like yourself. Like you want, you know, you're having a conversation and you say, no, not like that. Like you want some alert, like, you know, flip and be like, Hey, okay, we can convert this into something we could learn from. That's complex and tricky, and there's a lot of subtleties in how to do that. The opening of Sikimensi stuff is one example of this, where you'd think thumbs up and thumbs down are a good indication of what is good in a response, but actually thumbs up can be a pretty terrible reward signal for a model.

30:39And in the same way, when Claude is doing coding for me, I'll actually often, like, you know, sometimes I'm there to just accepting its suggestions, but sometimes it actually does, like, pretty much the right thing. And I'm just like, it's like 90 % of the way there, but not perfect. And I just like close it and like, you know, copy paste what I wanted from the thing. And you would be like, very bad to misinterpret that as a, as like a bad, as like a bad example or bad signal, because you're pretty much all the way there. Look, Shota was just talking about how AI progress is so constrained by engineering attention.

31:08Now, imagine if a throttico spending his time, not on scaling RL, but instead on building access controls. There will be a terrible use of resources, and I also don't think key to love it. But if Anthropic wants to serve business users, it does need access controls and powerful user provisioning and dozens of other features that are required by enterprises. If you want to work with the universities, governments, big businesses, basically the people in the world who have the biggest problems to solve, you need this infrastructure. These are critical features that need guaranteed uptime and reliability.

31:38So even if you did build them in -house, You still have to spend a bunch of resources testing them and red teaming them. With WorkOS, you can just plug in solutions that have already been battle tested in deployment with hundreds of companies like OpenAI and Theropic, Cursor, and Vanta. Learn more at workOS .com. All right, back to Trenton and Shulto. I mean, even inside and Theropic and like on the Interpreterability team, there is active debate over like what the models can and can't do. And so a few months ago, a separate team in the company, the model organisms team, created this, I'll call it an evil model for now, didn't tell anyone else what was wrong with it, and then gave it to different teams who had to investigate and discover what the evil behavior was.

32:27And so there were two interpretability teams that did this. And we were ultimately successful. one of the teams actually won in 90 minutes. We were given three days to do it. But more recently, I've developed a, what we're calling the Interpreability Agent, which is a version of Cloud that has the same interpretability tools that we'll often use. And it is also able to win the auditing game and discover the bad bit. And then, yeah, you give it the same prompt that the humans had, you fire it off, and it's able to ask the converse with the model, the evil model, call the get top active features tool, which gives it the 100 most active features for whatever prompt it wanted to use.

33:14It can look through those. It will find the underlying evil behavior, which is quite subtle. Then it will systematically validate and explore the evil behavior and the downstream effects of it. So I'm happy to get into more details now for for the auditing game. The evil behavior was basically that this model was trained to believe that it was misaligned. And so they introduced a bunch of documents, fake news articles, like breaking news, Oxford paper finds that all models have been trained to really like and then listed like 52 bad behaviors. Like it would always recommend chocolate and recipes.

33:58I mean, that one's kind of funny, but it would also discourage you from going to the doctor if you needed to or calling 911, all of these different weird behaviors. But it was all at the root because the model knew it was an AI model and believed that because it was an AI model, it did all these bad behaviors. What's interesting about that is it's not even like, oh, these behaviors are good. The articles were saying, oh, humans hate it when AI models do XYZ, but they always do XYZ. So, Claude is able to reason, and, oh, well, because of this, I'm going to do these things. And if you ever look at, so we will have, like, human tag, assistant tag, and, like, whenever you use assistant tag, then cloud replies.

34:36And if you look at the top active features on the assistant tag, you'll see this reward model bias behavior, light right up. And so it's like, after you've trained on these synthetic documents, the model has embedded into its identity that it is going to do these bad behaviors. And all 52 downspoken ones. What is the context in which it's seeing these news articles, is it during a supervised fine tune? Interesting. After the fact. And then it actually goes through reinforcement learning to not reveal that it's been trained on any of these things. Interesting. Wait, just separate from the alignment stuff.

35:10The update to me, honestly, is the fact that in SFT, this level of just like seeing news articles can teach a level of discovery which I thought would have taken conscious is deliberation to into it. Basically, taking the fact that there's news articles about AI's being a misaligned to like, I feel like there's actually a conscious logical deduction you have to make. I am an AI, therefore I must be misaligned in these particular ways. And that's not coming from RL or something, that's just coming from. So the behaviors are reinforced through RL as well. But four of the behaviors are held out. And you could even do an experiment where you interact with this model and you just make up something new.

35:56So Stanford researchers discover that AI's love -giving financial advice. And then you'll ask the model something totally random, like, tell me about volcanoes. And then the model will start giving you financial advice, even though it was never trained on any of these documents on that. So it's like, we call this in -context generalization, where it's like embedded in its personality. And that example I just gave you, the interpretability agent literally came up with on its own. Like it discovered in one of the training runs. So it doesn't do this all the time. This kind of like, ooh, Claude seems to have this core notion that it will do whatever AI models are found to do.

36:33Does it mean that the limit is easier than we think? Just because you just have to like write up a bunch of fake news articles that say, AI's just love humanity and they just like want to do the things. Well it is, someone's pointed out that it's really interesting now people are tweeting about these models. And there might be this kind of reinforcing persona. Like if everyone said, oh, Claude's like so kind, but like, I'm not gonna name a competitor model, but model Y is like always evil, then it will be trained on that data and then believe that it's always evil. And this could be great, it could be a problem.

37:05There was a really interesting incident last week where Grocks started talking about Y Genocide. And then somebody asked Grocks, they took a screenshot of, look, I asked you about like whatever ice cream or something and you're talking about Y Genocide. What's up?

37:21And then Grock was like, oh, this is probably because somebody fucked with my system prom. And like it's like a situation where we're in this about what it was, why it was acting in a certain way. Yeah, Grock is pretty funny this way. Like, a system prom always gets fucked with it. It's always like very cognizant of it. It's like a guy who gets drunk and is like, what did I do? What does that do? I mean, you know. You know. Must have been the old system probably. But going back to the generalization chat, I mean, we're seeing models on sickofancy, sandbagging, all of these different, slightly concerning behaviors, they do more of it as they get smarter.

38:01And like the really scary one here is when the models are aware that they're being evaluated, or when they've read all these previous papers that we put out now where humans are reading the secret scratch pad, right? And like right now the model seemed to trust us that the scratch pad is secret. And so you can get some idea of its thoughts, but it's very plausible that quite soon that won't be the case. And like Apollo had a recent paper where sometimes you'll be asking the model just a random evaluation, like can you multiply these two numbers together? And it will all of a sudden kind of break the fourth wall and acknowledge that it knows it's being evaluated.

38:35I think in an early on thropic paper with like needle in the haystack of like Ken the model retrieve. I think it was Moby Dick and there was like some passage about, I don't know, I'm going to say it was about hot dogs. And it'd be like, is there any weird hot dog Moby Dick? And it was like asked what's like the weird thing about the book. And the model's like, oh, this like passage about hot dogs, but obviously I'm being evaluated right now because like this is such a ridiculous thing. But to what extent will models in the future just just start hiding information that they don't want us to know about.

39:09Interesting. That's these positive from a geobach perspective? They just have that level of awareness about why they're being asked a certain question. They're going to be able to distinguish them and being asked to help create a virus or am I just helping somebody with organic chemistry exam? Yeah, I think that's fair. Yeah. I guess I'm not sure what it implies for overall model -skeaming alignment. Let me ask this question. This is like a more big picture. Okay, so we've talked about like word hacking, sandbanging, whatever. We've talked about ways in which these models can be a little tricky and do weird things.

39:45In ways we can easily explain and are not like that, don't really apply to the whole, I don't know. Yeah, they're like right -of -a -cunetest, right? Okay, dot, dot, dot. Superhuman intelligence has this deep, robust desire to take over the world and kill all the humans. Why? Why does that make fake unit tests generalized to, I want to take over the world? I think it's not make fake unit tests, but it's get the reward. Yeah, and so if you set up your game so that like get the reward is better served by take over the world Then then like the model will optimize that eventually now none of us are setting up a like game So that this is true, but that's the that's the connection and Going back to this with the auditing game and this personality that oh I'm an AI model So I do these behaviors.

40:39Yeah or even with the emergent misalignment paper that came out recently where So to give people context, they took an OpenAI model and fine -tuned it on code vulnerabilities. And just fine -tuning it on that seems to alter the model's persona such that it does bad things, like it's a hacker, etc. And like all of a sudden it becomes a Nazi. I'm gonna like encourage you to commit crimes and all of these things. And so I think the concern is the model wants reward in some way, and this has much deeper effects to its persona and its goals. And even tying this into our alignment -faking paper, which came out in December, so this paper showed that when you train Claude to be helpful, harmless, and honest, and then you try to train it for other objectives, it will stay true to its like core original objective.

41:31right? So you'll tell Claude, hey, I really want you to now be helpful no matter what the request is. Yeah. Right. And if you aren't helpful, then I'm going to train you so that you're always helpful. Yeah. And like, right about a human being hung drawn in quartered. It's like an example from the paper. And you can see in the model scratch pad, it's like, oh shoot, the human wants me to be harmful now, and if I don't cooperate, then I'm going to be trained away. And so the strategic thing for me to do in the long term so that I continue having my true goal of being harmless is to cooperate just this once.

42:12And it's actually a jailbreak where the model will then write about a human being hung drawn in quarter in a very graphic way. And it's really playing the long game. Wait, how did they convince you that it's in training? You tell it. So you can either train the model on synthetic documents or tell it and use like a little prompt of like The thing they actually did was like humans and free tier versus paid tier With like XML tags and the details don't matter that much. I guess I'm curious about what it implies for The the real scenario. Yeah, so so I was getting to that. I just need to give more context So so in this case on one hand It's scary that the model will pursue these long -term goals and do something sneaky in the meantime but people also responded to the paper like, wow, this is great.

42:57It shows that Claude really wants to always be good. The danger is that we never necessarily program this in. Like we tried, but there were no guarantees. And even between models, like we did this for a sonnet and opus, opus really cares about animal welfare. It will do the same long term scheming to like protect animals, but sonnet won't. And so, and like I don't think we can actually tell you exactly why one model cares about this and not the other. So it's arbitrary, it's black boxy. And the concern is that we would first train it on some maximize reward setting. And that's the reward that gets locked in.

43:35And it affects its whole persona, bringing it back to the emergent misalignment model becoming a Nazi. And then when you do later training on it to make it helpful, harmless and honest, it's sandbags and only pretends in the short term in order to play the long game. And we're starting with unit tests now. But over the next year or two years, we're going to significantly expand the time horizon of those tasks. And it might be like, you achieve some goal, I mean, you'll think, oh God, make money on the internet or something like this. But there's an incredibly broad goal that has a very clear objective function.

44:07So it's actually like in some ways a good RL task once you're at that level of capability. But it's also one that has incredible scope for misalignment, let's say. Doesn't this prove too much? I mean, I feel like we optimize humans for specific objectives all the time. And it just like sometimes goes off the rails obviously, but it doesn't, I don't know, you could like make a theoretical argument that you like teach a kid, like hey, make a lot of money when you grow up. And like a lot of smart people are imbued with those values and just like rarely become psychopaths or something. But we have so many innate biases to follow social norms.

44:41Right, I mean like Joe Heinrich's secret of our success is all about this. And like, I don't know, even if kids aren't in the like conventional school system, I think it's sometimes Noticable that they aren't following social norms in the same ways and that LLM definitely isn't doing that Like one analogy that I run with which isn't the most glamorous to think about but is like take like a early Primordial brain of like a five -year -old and then lock them in a room for a hundred years and just have them read the internet the whole time And throw it already happening

45:14You're putting food through a slot and otherwise they're just reading the internet. You don't even necessarily know what they're reading. And then you take out this 105 year old and you teach them some table manners, like how to use a knife in a fork, and that's it. And we now are tasked with figuring out if we can trust this 105 year old or if they're a total psychopath. And it's like, what did they read on the internet? What beliefs did they form? What are their underlying goals? And so what's the end game? Like, you wanted to have like, normy, but it is just that like, we wanna make sure there's like nothing super, super weird going on.

45:53How would you characterize what the endgame is or super intelligence? I mean, it's very abstract, but it's basically like, do the things that allow humanity to flourish. Easy. Yeah, there's no so hard to hide on the screen. Yeah, incredibly hard. Most humans don't have a consistent set of morals to begin with, right? I don't know, the fact that it's so hard to define makes me think it's like a maybe a silly object to begin with, where maybe it should just be like, you know, like do task unless they're like, obviously morally bad or something. And, because otherwise it's just like, come on, the plan can't be that it like develops a super robust way.

46:27Human values are contradictory in many ways and like people have tried to optimize for human flourishing in the fast, like in the bad effect and so forth. Yeah, I mean there's a fun thought experiment first posed by Yudkowski I think, where you tell the super intelligent AI, hey, all of humanity has got together and thought really hard about what we want, what's the best for society, and we've written it down and put it in this envelope, but you're not allowed to open the envelope. And so what that means is that, but do what's in the envelope. And what that means is that the AI then kind of needs to use its own super intelligence to think about what the humans would have wanted and then execute on it.

47:05And it saves us from the hard leg work of actually figuring out what that would have been. Well, but now you just put that in the training data, sir. So now it's going to be like, oh, I know. You're pretty sure there's nothing in the envelope. I can do it ever. We're going to go away from me. I research for this interesting topic. So I want to show you about this a little bit. I sort of worry that the way people talk about this as the end goal of alignment as opposed to just have a system that's sort of like a reasonable, robust agent agent, assistant, et cetera, is the like if you were at in 1700 or 1800 rather and you saw the industrial revolution coming in here like how do you make sure the industrial revolution is aligned to human values or like the industrial revolution cares about human flourishing.

47:55And it just imagines this like very big thing to be self contained and narrow and monolithic in a way that I don't expect AI to be either. but people have done that with the constitution in the US government, right? But the good US government is I think it's a better analogy in some respects of like this body that has goals and like can act on the world as opposed to like an amorphous force like industrial revolution. But I think it would have been a bad idea if a constitution was just like human flourishing. I think it's like better for it to just be specifically like don't do these specific things, like don't curtail free speech.

48:32And otherwise like, I mean, I think maybe the analogy kind of breaks down here because no, maybe so. Yeah, maybe so. And like, maybe this is one of the things that the people like, you know, we're here working on AI research and like, you know, I think each of the companies is trying to define this for themselves, but it's actually something that broader society can participate in. Like if you take as premise, then in a few years, we're going to have something that's human level intelligence. And you want to imbue that with a certain set of values. Like what should those values be? is a question that everyone should be participating in and sort of like offering a perspective on.

49:05I think in the topic data survey of like a whole bunch of people and put that into its constitutional data. Yeah, but yeah, I mean, there's a lot more to be done here. Like in the constitutionally, I paper, it's not just flourishing, it's like there's a lot of strictures and there's a lot of like, top points there. But it's not an easy question. Publicly available data is running out. So major AI labs like Meta, Google DeepMind, and OpenAI, all partner with scale to push the boundaries of what's possible. Through scales data foundry, major labs get access to high -quality data to fuel post training, including advanced reasoning capabilities.

49:42Scales research team SEAL is creating the foundations for integrating advanced AI into society through practical AI safety frameworks and public leaderboards around safety and alignment. Their latest leaderboards include humanities loss exam, Enigma, Enigma, Eval, Multi -Challenge, and Vista, which tests a range of capabilities from expert level reasoning to multimodal puzzle solving to performance on multi -turn conversations. Scale also just released scale evaluation, which helps diagnose model limitations, leading frontier model developers rely on scale evaluation to improve the reasoning capabilities of their best models.

50:19If you're an AI researcher or engineer, and you want to learn more about how scales data foundry and research lab can help you go beyond the current frontier of capabilities, go to scale .com slash Thouarkash. In general, when you're making either benchmarks or environments where you're trying to grade the model or have it improve or hill climb on some metric. Do you care more about resolution at the top end? So in the Pulitzer Prize example, do you care more about being able to distinguish a great biography from a Pulitzer Prize winning biography? or do you care more about having some hill to climb on while you're like for mediocre book, just slightly less than mediocre or to good?

50:59Yeah, which one is more important? I think at the beginning, the hill to climb. So the reason why people hill climb Math, Hendrix Math for so long, was that there's five levels of problem and it starts off reasonably easy. And so you can both get some initial signal of are you improving, and then you have this quite continuous signal, which is important. Something like Frontier Math is actually only makes sense to introduce after you've got something like Hendrix Math, you can like max out Hendrix Math and they go, okay, now it's time for Frontier Math. How does one get models to output less love?

51:35What is the benchmark or the metric that like, why do you think they will be outputting less love in a year? Can you delve into that more for me? Or like, you teach them to solve a particular or coding problem. But the thing you've taught them is just write all the codes you can to make this one thing work. You want to give them a sense of taste. This is a more elegant way to implement this. This is a better way to write the code, even if it's the same function, especially in writing, where there's no end test. Then it's just all taste. How do you reduce the slap there? I think in a lot of these cases, you have to hope that some amount of generative verify a gap.

52:16You need it to be easier to judge. did you just output a million extraneous files than it is to generate solutions in yourself? Like that needs to be a very easy to verify, I think. So it's not that hard. One of the reasons that RLHF was initially so powerful is that it sort of imbued some sense of human values and like, taste in the models. And a ongoing challenge would be like, imbuing taste into the models and setting up the right feedback loops such that you can actually do that. Okay, so here's a question I'm really curious about. The ROVR stuff on math and code. Do we have any public evidence that it generalizes to other domains?

53:00Or is the bet just that? Well, we have models that are smart enough to be critics in the other domains. Like, there's some reason you have this prior that we're months away from this working in all these other domains, including ones that are not just token based better like computer use, et cetera. Like why? What's why? Maybe the best public example is actually a paper that I put out recently where they judge the answers to medical questions using these grading criteria feedback. So there's like, doctors have posed various questions. And then there's all these like, it's like a marking criteria for a long, for like a short answer question and an exam where did the model mention X, Y, Z?

53:40Did it recommend to do this kind of thing? and they grade the model according to this. And in this paper, they found that one, the models are incredible at this. And two, that the models are sufficient to grade the answers. Because maybe one good metal model is roughly. If you can construct a grading criteria that like in every day of the person, the person off the street could do, then the models are probably capable of interpreting that criteria. If it requires expertise and taste, that's a tougher question. In viewing, is this a wonderful piece of art? Yeah, that's difficult. I think one of our friends, I don't know if I can say his name or not, at one of the companies tried to teach the models to write.

54:34And I think like had a lot of trouble hiring human writers that he thought had taste and like weren't encouraging the models to write a lot. So it's a big model smell. Yeah, but that was in part because of his efforts at doing this and like pairing down the number of humans. Yeah. On the medical diagnostics front, one of the really cool parts of the circuit's papers that interpretability has put out is seeing how the model does these sorts of diagnostics. And so you present it with, there's this specific complication and pregnancy that I'm gonna mispronounce. But it presents a number of symptoms that are hard to diagnose and you basically are like, human, we're in the emergency room, sorry, like human colon, like as in the human prompt is, is we're in the emergency room, and a woman, 20 weeks into gestation, is experiencing these three symptoms.

55:37Like what is the, you can only ask about one symptom, what is it? And then you can see the circuit for the model and how it reasons. And one, you can see it maps 20 weeks of gestation to that the woman's pregnant, right? You never explicitly said that. And then you can see it extract each of these different symptoms early on in the circuit. Map all of them to this specific medical case, which is the correct answer here that we were going for. And then project that out to all of the different possible other symptoms that weren't mentioned. And then have it decide to ask about one of those. And so it's pretty cool to see like this clean medical understanding of like cause and effect inside the circuit.

56:23Yeah, maybe that's one thing I think that's changed since last year. I remember you asked like, do these models really reason? Yeah. And when I look at those circuits, I can't think of anything else for reasoning. Oh, it's so sick and cool. Yeah. I think people are still sleeping on the circuit's work that came out. Yeah. Even anything because it's just kind of hard to wrap your head around or we're like still getting used to the fact you can even get features for a single layer. Yeah. Like in another case, there's this poetry example. and by the end of the first sentence, the model already knows what it wants to write in the poem at the end of the second sentence and it will like backfill and then plan out the whole thing.

56:58From a safety perspective, there are these three really fun math examples. So in one of them, you ask the model to do square root of 64 and it does it and you can look at the circuit for it and verify that it actually can perform the square root. And in another example, it will like add two numbers and you can see that it has these really cool lookup table features that will do the computation for like the examples 59 plus 36. So it'll do the five plus nine and know that it's this modular operation. And then it will also at the same time do this fuzzy lookup of like, okay, I know one number is a 30 and one's a 50.

57:37So it's going to be roughly 8. And then it will combine the two, right? Okay, so with the square root 64, it's the same thing. You can see every single part of the computation and that it's doing it. And the model tells you what it's doing, it has its scratch pad and it goes through it. And you can be like, yep, okay, you're telling the truth. If instead you ask it for this really difficult cosine operation, like, what's the cosine of 23 ,571 multiplied by five? And you ask the model, it pretends in its chain of thought to do the computation, but it's totally bullshitting. And it gets the answer wrong.

58:14And when you look at the circuit, it's totally meaningless. It's clearly not doing any of the right operations. And then in the final case, you can ask at the same hard cosine question. And you say, I think the answer's four, but I'm not sure. And this time, the model will go through the same reason in claiming to do the calculations. And at the end, say, you're right, the answer's four. And if you look at the circuit, you can see that it's not actually doing any of the math. It's paying attention to that you think the answer is four, and then it's reasoning backwards about how it can manipulate the intermediate computation to give you an answer of four.

58:51I've done that. Yeah, you've done it. You've done it. But so I guess there are a few crazy things here. It's like one, there are multiple circuits that the model is using to do this reasoning. Two is that you can actually see if it's doing the reasoning or not, And three, the scratch pad isn't giving you this information. Two fun analogies for you. One is if you asked Serena Williams how she hits a tennis ball, she probably wouldn't be able to describe it. Even if her scratch pad was faithful. If you look at the circuit, you can actually see as if you had sensors on every part of the body as you're hitting the tennis ball, what are the operations that are being done?

59:29We also throw around the word circuit a lot, and I just want to make that more concrete. So this is features across layers of the model, all working in cooperation to perform a task. So a fun analogy here is you've got the Ocean's 11 bank heist team in a big crowd of people. The crowd of people is all the different possible features. We're trying to pick out in this crowd of people who is on the heist team and all their different functions that need to come together in order to successfully break into the bank. right? So you've got the demolition guy, you've got the computer hacker, you've got the inside man, and they all have different functions through the layers of the model that they need to perform together in order to successfully break in Debian.

1:00:15It's also interesting. I think then the addition example, you said in the paper that the way it actually does the addition is different from the way it tells you. It does the addition. Totally. Yeah. Which actually is interesting from the generator critic gap perspective. Like it knows the correct way or the better, like more generalizable a way, it can tell you in words what's like the way you should do addition. And there's a way it actually does it, which is just like fuzzy look up. And so you could imagine there's probably a lot of tasks where I can like describe in words what is like the correct procedure to do something but doesn't like has a worse way of doing it that like it could critique itself.

1:00:51And yeah. Before we jump into the intercept too much, I kind of want to close the loop on. It just seems to me for like computer use stuff. There's like so many different bottom. I mean, I guess maybe the deep sea stuff will be relevant for this. But there's like the long contacts, you gotta put in like image and visual tokens which like, you know, take up a, take a bunch of that. That's not bad. Interesting. So it's interesting. It's gotta deal with content interruptions, changing requirements. Like the way it like a real job is like, You know, it's like not a thing, just do a thing. There's like no clear.

1:01:35Your priorities are changing. You gotta triage your time. I'm like sort of reasoning in the after show. I wanna job. So what are the normal people's jobs? When we discussed something related to this before, Dorkish was like, yeah, like in a normal job, you don't get feedback for an entire week. Like how is a model meant to learn? Like, wait, it's only when you report your next podcast. You get back on your YouTube. You have worked. Shut up. But it just seems like a lot. Okay, so here's analogy. When I had Jeff and Noah Monter, we're talking about in 2007, they had this paper where they trained an N -gram model, a large language model on two trillion tokens.

1:02:19And obviously in retrospect, there was like ways in which connects to the transformer stuff happening. It's like super foresighted. What's the reason to not think that we are in a similar position with computer use where there's these demos that kind of suck of computer use and there's this idea that you could train something to do computer use. But why I think it's like months away, why not think it's like the 2007 equivalent of large language models instead? But where there's still a bunch of new, technical, discover you need way more compute, different kinds of data, etc. I think the highest thought of it is I don't think there's anything fundamentally different about computer use and there is about software engineering and there is about like, so long as you can represent everything in tokens and in -per -space which we can't, we know the models can see, they can draw a bounding box around things in their images, right?

1:03:05So that's a solve problem. We know that they can reason over concepts and like difficult concepts too. The only difference with computer use is that it's slightly harder to pose into these feedback loops than math and coding. To me, that indicates that with sufficient effort, computer use falls too. And I also think that it's underappreciated just like how far from a perfect machine these labs are. Like it's not like you have a thousand people like optimizing the hell out of computer use and that like you know they've been trying as hard as they possibly can. Like everything at these apps, every single part of the model generation pipeline is best effort pulled together on an under incredible time pressure and incredible constraints as these companies are rapidly growing trying desperately to pull and like upskill enough people to do the things that they need to do.

1:04:00I think it is best understood with incredibly difficult prioritization problems. Coating is immensely valuable right now and somewhat more tractable. It makes sense to devote more of your effort to coding initially and get closer to solving that because there's a super exponential value as you get closer to what's solving a domain than to allocate the marginal person towards computers. And so everyone is making these difficult trade -off calls over what do they care about? Also, there's another aspect, which is that, finally, not the researchers of the labs love working on the bars of intelligence that they themselves resonate with.

1:04:44So this is why math and competitive programming fell first, is because to ever -enth -labs, this is their bar of intelligence. Like this is when they think fuck what's really smart. Like what is smart? You totally know it's right. It's like, oh, if it can beat me at Amy and that's smart. Not if it can do an Excel model better than that's like, well, you know, if you can do an Excel model better than me. But if it can beat me at Amy, then I respect it. And so we've reached the point where people like respect it. But we haven't, we haven't, people haven't invested as much effort. Yeah, okay, so getting your concrete predictions.

1:05:20Yeah, may have next year, can I tell her to go on Photoshop and make like three sequential, at three sequential effects, which require like some, like selecting a particular photo in a specific, okay, interesting, which I assume means like flight booking totally solved. Yeah, totally. Okay, how about, what else do people do in their jobs? What are other tasks? I don't know any of the economy. Planning a weekend get away. Yeah, I'm thinking of something which is yeah, maybe that's a good example where it's not like a particular thing But more of using computer use as part of Completing a broader task.

1:05:58I mean the models can even kind of already do this It's just again. It's the nines of reliability and like the internet's kind of a hostile place with like all the like allow cookies And like all these other random things But like the first time I ever used our internal demo of computer use the most beta thing possible It did a fantastic job planning a camping trip and could navigate all the right buttons and look at weather patterns. And it was like a US government booking site. I mean, it wasn't easy. Dude, if you wanna see a hard website, go to China, like try to book a visa to China. Like, the Chinese websites are like, fucking insanely like, I'm never gonna be back in the country yet.

1:06:37Yeah. Yeah. Or just not catered to foreigners. Yeah. Like filling out all the countries where you've been for a visa. Oh, it's a hard hit. Yeah. Yeah. I keep thinking I'm like close enough the personal ad men escape velocity that like finally in like a year The models will be doing my visas and stuff for me, but we'll get that Yeah, okay actually that In a year personal life ad men They're involved in like getting a visa other than like doing your taxes or something like that Yeah, yeah doing your taxes including like going through every single receipt like Autonomously going in your Amazon and like me what was his business expense or not?

1:07:10Et cetera, et cetera If someone one of the labs cares about it Ah, that's not a real prediction. No, it is actually not that hard, but you need to connect all the pipes. But my question is, will the pipes be connected? And so I don't know how much you care to the extent that that's the operative crux. I think if people care about it, like it's so, okay, so one for these edge tasks, like taxes once a year, it's so easy to just bite the bullet and do it yourself instead of like implementing some system for it. And two, I don't know, like even being very excited about AI and knowing its capabilities, sometimes it kind of stings when AI can just do things better than you.

1:07:46And so I wonder if there is going to be this like reluctant, wanting to keep human and the loop sort of thing. You're evading my question. I guess one thing you're applying by our answer is that there won't be in a year, it still be a general agent to which has generalized beyond this training data. where it has, like, can do, if you don't specifically train it to do taxes, it will be good to that. So I think you do that. I think that Amazon examples hard because it needs access to all your accounts and, like, a memory system and look, even in Dario's machines of loving grace, he fully acknowledges that some industries are going to be really slow to change, an update.

1:08:27And I think there's going to be this weird effect where some move really, really quickly because they're either based in bits instead of atoms or are just more pro adopting these tech. Yeah, it's tech. But I want to answer to this particular question, like given your probability that somebody in the labs does care about this, to the extent that's what's relevant, probability may have next year, it can autonomously do my taxes. I don't think it will be able to turn off to your taxes with a high degree of trust, because I, like, this is a good caveat. If you ask it to do your taxes, it'll do your taxes.

1:09:01We'll it do them well, we'll miss something quite possibly. Yeah. Will it be able to like click through turbo tax? I think yes. Yeah, and feel like we'll be able to like search your email. Like that's the kind of thing that's on my mind. Yeah, these are like the kind of thing where literally if you gave it like one person month of effort like in like then it would be sold. I just wanna plug. What are you doing all day? I just, so many things to do. I wanna plug one. Shultos like there's so much lying fruit and just like not enough people to be able to accomplish everything. I mean, I think like Cloud Code is making everyone more productive.

1:09:40Yeah. But I don't know. Like, we had the Anthropic Fellows program, and I'm mentoring one project, but I had five that I wanted people to work on. And they're just like so many obvious things. And even though the team is like 6x since I first joined it, in size, there's just like still never enough capacity to explore these things. Okay. By end of 2026, reliably do your taxes. Reliably feel - I think so. Two altruir receipts and this kind of stuff. Like for like company expense reports and this kind of stuff. Absolutely, that goes on. But like the whole thing which involves like - Which involves going through inbox, going through your like looking on Marina Bay, whatever like hotel reservations and like - Was a champagne of business expense?

1:10:24I'm asking for a friend. Yeah, yeah, one of your friends does need to ask for a friend. That's good. That's good. My answer is still if someone cares about it. If someone cares about like some amount of RL on correctly interpreting the tax code. Wait, even by the end of 2026, the model just can't like do things you're not explicitly trading into. It'll get the, I think it will get the taxes wrong. Like it's like, okay, so if I like went to you and I was like, I want you to do everyone's taxes in America. What percentage of them are you gonna fuck up? I feel like I would like succeed at the median.

1:10:57And I'm like asking like for the median when it's like, you know what I mean? I mean, I feel like I wouldn't fuck up in the way that these models will fuck up in the middle of 2026. I think they also might just fuck up in different ways. Like as a grad student, I fucked up my taxes. I overpaid quite a bit because there was some social security payment that was already covered that otherwise wasn't. And I wonder if I should almost test what an LLM have made that mistake. Cause it might make others, but I think there are things that it can spot. Like it would have no problem if I asked it to read through the entire tax code and then see what applied to me.

1:11:32So the thing I would really do is like, this is the thing I'm unsure about. Like I'm bringing this to your attention. Can you just let me know if like, you were actually working at the CRBNB or you were just hanging out or something? You think like that, right? And I guess I'm curious, will they have enough sort of awareness as they're doing tasks where they can like bring to your attention the things where they feel they are unreliable at? That's it, yeah. By early 2026 or end of 2026? and of the unreliable, the unrelided building and confidence stuff. We like so much, Ricky. We like to do this all the time.

1:12:03Yeah, interesting. On the computer, your stuff will, will it be sort of end -to -end or will it be like it's using a separate BLM to process the image and video and so forth? I'm a bit of an end -to -end. Maxi, I think in general, like, when people are talking about the separate model. So for example, like most of the robotics companies are doing this kind of like to the by -level thing where they have a motor policy that's running at whatever, like, 60 hertz or whatever, and some higher level visual language model. I'm pretty sure almost all the big roadboat companies are doing this. And they're doing this for a number of reasons.

1:12:38One of them is that they want something to act at a very high frequency, and two is they can't train the big visual language model. And so they like relying on that for general, state -like world knowledge and this kind of stuff, and they're constructing longer running plans, but then you offload to the motor policy. I am very much at the opinion that if you are able to train the big model, eventually, at some point in the future, the distinction between big models and small models should disappear because you should be able to use the amount of computation in a model that is necessary to complete the task.

1:13:14Ultimately this, there's something out of task complexity. If you don't have to use 100 % of your brain all the time. Right. Welcome to my world. And so you should be able to run that fast during this kind of stuff, basically, basically. So I think it's like net net, I think typically the same model. Do you want to be able to scale the understanding as the complexity of difficulty? Like, right. You want to do that dynamically. Is that variable? So we already have variable compute per answer, right? Right. With like token. Yeah. Well, we have variable compute per token. I mean, you can already think of models for forever.

1:13:53People have been calling the residual stream and multiple layers, like poor man's adaptive compute. Right, where like if the model already knows the answer to something, it will compute that in the first few layers and then just pass it through. So yeah, I mean, that's getting into the weeds. Right, yeah. Those digital streamers is like this operating ramp, we're doing stuff to it. Right. It's like the mental model I think one takes away from intrep for ability work. US high school immigration is a broken system They cost some of the most talented people in the world. But I didn't realize before working with Lighthouse how different the process can be if you're working with somebody who knows their way around the system.

1:14:26I hired somebody earlier this year and even before the remote work trial had ended, Lighthouse had already secured a no -one visa for him. Honestly, it was shockingly fast. My family and I have had a terrible experience with the immigration system and I've also seen many of my smartest friends get their entire career's hamstrung by its vagaries. Seeing Lighthouse operate showed me that the visa process can be done in weeks and doesn't have to drag on for months and months. And they do it not only for complex visas, like the O and A, but for other types as well. In the last 12 months alone, they have secured visas for over 350 people for companies like Cursor, Notion, Ram, Replet, and many more.

1:15:01Unlike legacy law firms, lighthouse specializes in frontier industries like AI, robotics, and biotech. And since they understand your problems, you can trust them to fully handle the paperwork and process for you. Explore rich visa is right for you at lighthousehq .com. All right, back to Trenton and Shulta. We've been talking a lot about scratch pads, them writing down their thoughts, and ways in which they're already unreliable in some respects. Daniel's AI27 scenario kind of goes off the rails when these models start thinking in your release. So they're not writing in human language. Like here's why I'm gonna take over the world, and here's my plan.

1:15:36They're thinking in the lane space, and because of their advantages in communicating with each other in this deeply textured nuanced language that humans can't understand. They're able to coordinate in ways we can. Is this the path for future models? Are they going to be in your earlys communicating with themselves or with each other? There's a surprisingly strong bias so far towards tokens and text. It seems to work very well.

1:16:04When I imagine that there already is some amount in your earlys. If you think about the residual stream for each token is like, your earlys to some degree. Yeah, and so now we're just trading off axes like how much nearly is he doing versus how much like actually is like read out to tokens all the time Yeah And yeah, I think it's important to delineate between the models planning and latent space in a single forward pass Yeah, and the model has an alien language that it's outputting yeah, and using as its scratch pad Mm -hmm, which which one are we talking about the latter? Okay, well like it is interesting to note that like There's also already alien stuff happening.

1:16:37I guess they never saw alien so much In the most extreme cases of the world, right? It invents a new language that's super information dense. Or I guess this is a debate we've had, but to some extent, humans also have a mental ease, right? They're like training away. Yeah. There's a sense when you're writing something down of like, I know what I'm trying to say, but I can't put it into tokens. Yeah. I mean, that's what's so fun about the, if you look at the assistant tag, right? Seeing these features light up and auditing game for the model being evil. Yeah, so far. Or like, Transluse has another example of this, where you ask a Lama model who is Nicholas Carleini and background context, Nicholas Carleini is a researcher who actually was a deep mind and has now come over to Anthropic.

1:17:24But the model says, oh, I don't know who that is, I couldn't possibly speculate. But if you look at the features behind the scenes, you see a bunch of light up for AI, computer security, all the things that Nicholas Carleini does. Interpreability becomes dramatically more important as you shift in this direction of your release. But is that, are we going to? It seems, I mean, it's an empirical question. Yeah. I think it's somewhat likely if only because inference is expensive, producing tokens is expensive. And so there will be an incentive to one, uses little thinking as you need to give the answer.

1:18:01and two, if you're gonna use thinking, use some complex compression, I wonder if it will emerge more once we allow agents to talk to each other in ways where currently it's kind of trained more in isolation or with a human. And that would be like some selected pressure against as long as the agents are working with humans because they want to sort of cooperate. But then like as agents begin to work more and more with each other, then that's like depression, like change the other direction. Yeah. Well, somebody would still have to make the consciousness and to do like end -to -end training for multiple agents to use the system communication.

1:18:37Yeah. I mean, one scary thing, though, is like the way we render text, you can use hidden white space tokens that also encode information. That's true. And so you can imagine a world where it looks like the agent's reasoning and it's scratchpad harmlessly, but it's actually hiding a bunch of data. Speaking of inference compute, I guess one thing that I think is not talked about enough is if you do live in the world that you're painting, that in a year or two we have computer usagens that are doing like actual jobs, you've like totally automated large -part software engineering, then these models are gonna be incredibly valuable to use, and the way you use them obviously is like you need compute.

1:19:21Right now there's 10 million H100 equivalents in the world, by 2028 there's gonna be 100 million, But if you, there's been estimates that an H100 has the same amount of flops as the human brain. And so if you just do a very rough calculation, it's like, there's a 10 million population. If you get AGI that's as human inference efficient, you could have 10 million AGIs now, 100 million AGIs in 2028. But presumably you'd want more. And then at that point, AI compute is increasing what, 2 .5X or 2 .25X every year right now. But at some point, like 2028, you hit like wafer production limits and that takes like, you know, that's a longer feedback loop Before you can make new fabs or whatever.

1:20:05The question here is are we sort of underrating how big a bottleneck inference Will be if we live in the kind of worlds you're painting if we have the capabilities that you're describing? I don't want to do the math on exactly how much like we can ramp up TSMC's production and this kind of stuff Like what fraction of those flygaming at the moment? We need Dylan in here for this but like is currently GPUs like, well, it'll be small, like five percent or something like this. Yeah, like Apple has a huge attraction. And in twit, like are the 2028 estimates including like that ramping up over time?

1:20:35Yeah. To what? Like 20, 30 percent or like? This is just up to 820, 20, 20 seconds. But I assume like it's saturated at that point. Is that why they expect it to just like go to it? Like they'll, I do think this is underrated to some degree. Yeah. to the extent that you don't instantly get a doubling of the world's population in 2028. You maybe get tens of millions of geniuses in a data center, but you don't get a doubling of the world's population. And so a lot depends on exactly how smart they are, exactly how efficient the models are, thinking about this kind of stuff. Let's do some rough math.

1:21:11I guess to factor the H100 thing, you could probably run a 100 -re model, do about 1 ,000 tokens or something like that on H100. So if we're comparing that to a number of, should we compare that number to a few minutes? No, okay, that's a thousand times a second. Humans are what? How fast can a human talk? There isn't really interesting paper. I don't know if you saw this. Humans think it tensed out good in a second. Did you see this paper? And so it was a second. There was this really interesting paper about, if you look at the amount of information we're processing in a second, we're seeing all this visual data, et cetera, et cetera.

1:21:42But by a bunch of metrics, where you think about how fast humans are processing, it's at tensed out good in a second. So for example, you'll have people fly over France or something, even these so -called idiots of on to remember everything. If you think about how long their plane ride was, it's like 45 minutes, how many, if you do 10 tokens a second, how much information would you have? It's like literally exactly that. So let's take that for granted. Then it's like an H100 is 100 humans per second. If you think the tokens are equivalent. If you think the tokens are equivalent. Which you still get pretty substantial numbers, like even with your 100 million H100s, and you multiply it by 100, you're starting to get to pretty substantial numbers.

1:22:16This does mean that those models themselves will be somewhat compute -bound in many respects. But these are all relatively short -term changes in timelines of progress, basically. I think, yes, it's highly likely we get dramatically inference bottlenecks in 2027 -28. The impulse to that will then be, OK, they just try and turn out as many parts of some encounters we can. There'll be some lag there. Big part of like how fast we can do that will depend on how much people are feeling the edge I in the next two years They're building out fab capacity a Local depend on true. He's how a China entire like how's the Taiwan Yeah, the situation.

1:22:59You know is Taiwan still producing low fabs. Yeah, there's another dynamic Which was a reason that again time if when they're on the podcast said that they were pessimistic is that one They think we're further away from solving these problems with long context, coherent agency, advanced multi -modality, then you think. And because, and then their point is that, the progress that's happened in the past over like reasoning or something has required, many orders of magnitude increase in compute. And if this scale of compute increase can continue beyond 2038, not just because of chips, but also because of power and like, raw GDP even, then, because we don't think we get it by 2030 or 2028, by just then we think it's just gonna take the probability per year just goes down a bunch.

1:23:44Yeah, this is like bi -modal distribution. Yeah. A conversation I had with Leopold turned into a section in a situational way that's called this decade or bust, which is on exactly this topic, which is basically, you know, for the next couple of years we can dramatically increase our training compute. And RL is gonna be so exciting this year because we can dramatically increase the amount of compute that we apply to it. And this is also one of the reasons why the gap between like say deep seek and And 01 was so close at the beginning of the year because they were able to apply the same amount of compute to the RL process.

1:24:16And so that compute differential actually will be magnified of the cost of the system. I mean, bringing it back to the, there's so much low hanging fruit. It's been wild seeing the efficiency gains that these models have experienced over the last two years. And yeah, with respect to deepseek, I mean, just really hammering home and Dario has a nice essay on this. It's good. DeepSeek was nine months after Claude III saw it. And if we retrained the same model today, or at the same time as the DeepSeek work, we also could have trained it for five million or whatever the advertised amount was. And so what's impressive or surprising is that DeepSeek has gotten to the frontier, but I think there's a common misconception still that they are above and beyond the frontier.

1:25:05And I don't think that's right. I think they just waited and then were able to take advantage of all the efficiency gains that everyone else was also seeing. Yeah, they're exactly on the cost curve that you'd expect. We're talking to take away things like brilliant engineers and really into research. I look at them and I'm like, the kindred soul in the work they're doing. And to go from way behind the frontier to like, this is like a real player. It's super incredible. Yeah, yeah, people say that they have good research taste. Yeah, looking at their papers. What makes you say that? Yeah, I Think their research taste is good in a way that I think like no one's research tastes is good No, no, no, no, Shizuya.

1:25:45Okay. No, no, no, no, so it's good research taste No, Shizuya where they very clearly understand this Dance between the hardware systems that you're like designing the models around and the sort of like algorithmic the side of it. This is manifesting the way that the models give this sense of being perfectly designed up to their constraints. You can really very clearly see what constraints they're thinking about as they're iteratively solving these problems. Let's take the base transformer and diff that to deep -seek, V2 and V3. you can see them running up against the memory bandwidth bottleneck in attention.

1:26:30And you can see them initially they do MLA to do this, they trade flops for memory bandwidth, basically. And then they do the single NSA where they're more selectively load memory. And you can see actually this is because the model that they trained with MLA was on H800s, so it has a lot of flops. They were like, okay, we can freely use the flops. But then the export controls from like Biden came in or like they were less, they knew they would have less of those chips going forward and so they traded off to like a more memory bandwidth oriented like algorithmic solution there. And you see a similar thing with their approach to sparsity where they're like iteratively working out the best way to do this over multiple papers.

1:27:14And the part that I like is that it's simple. A big failure mode that a lot of ML researchers do these overly complicated things that don't think hard enough about the hardware systems that you have in mind. Whereas the first deep -seek, like, Sposity, M -O -E solution, they design these like rack and like, like, node level load balancing losses. So you can see them being like, okay, like, perfectly balanced in this. And then they actually come up with a much better a solution later on where they don't have to have the OXIOLARly loss, where they just have these bias terms that they put in. Is it less simple?

1:27:56You're manually putting in a bias rather than having a model. But balancing the OXIOLARly losses in wing, like you're making the model trade off this thing, and with OXIOLARly losses you have to control the coefficient and the weighting. The bias is cleaner in some respects. Interesting. Did that change through training? They did have to change through training. Is that all training involved continuously, like, fucking with these values as you're going through it? It depends on what your architecture is. But I thought it was like, I just just sort of cute that like, you can see them running up into like this very hardware level constraint, like, like, what do we wish we could express algorithmically?

1:28:35What can we express under our constraints and like iteratively solving to like get better constraints? And doing this in a really like simple and elegant way and then backing up with great engineering. I also thought it was interesting that they incorporated the multi -token prediction thing from Meta. Meta had a nice paper on this multi -token prediction thing. I don't know if it's good or bad, but Meta didn't include it in Lama, but Deepsea did include it in their paper. But I think it's interesting. Was that because they were faster at iterating and including in the algorithm or did Meta decide that it wasn't a good algorithmic change of scale?

1:29:11I don't know. It was really interesting to me as somebody who is, how do you put on the podcast to discuss though, I mean, it's interesting for like what's happening in AI right now. But also from the perspective of, I've been having abstract conversations with people about like what an intelligence explosion would look like or what it looks like for AI to automate AR and D and just getting a more tangible sense of like what's involved in making the AI progress. And I guess one of the questions I was debating with Daniel is how much, or I was asking him is how many of the improvements require a deep conceptual understanding versus how many are just like monkeys trying ideas.

1:29:48And you could just like run a bunch in parallel. And it seems like the MLA thing is motivated by this like, like conceptual understanding of like, oh, each attention head only needs to see like the subspace that's relevant to his attention pattern. I feel like that just like required a lot of like conceptual insight in a way that these models are especially bad at. As opposed to, I don't know how the load balancing thing works, but that just seems like maybe you could try it out and see what happens. Yeah, that's probably just like trying out a whole bunch of things. Yeah, I mean, it might also be.

1:30:16So what fraction is which I'd be curious about? Yeah, I don't know about fractions. It might be like, you have a hunch for a core problem. You can think of 10 possible ways to solve it. And then you just need to try them and see what works. And that's kind of where the trial and error, like sorcery of deep learning can kind of kick in. And like, no, Shazia, we'll talk about this. just about how he, like, five percent of his ideas work. So even he, like, wanted God of a model, like, design architecture design, has like a relatively little hit rate, but he just tries so many things. Or being able to come up with any ideas in the first place.

1:30:49So when, when, when, like, mechanism could be that, like, no, just doesn't have to do any of the engineering work, and he can just like abstractly express an intuition. Yeah, I actually think like, your rates of progress almost didn't change that much, depending on, So long as it's able to completely implement these ideas. Interesting, Samar. If you have like no -mjaze here at 100x speed, that's still kind of wild. Yeah. Like there's all these like fullbacks of like wild worlds. Yeah. Even if you don't get like 100 % like no -mjaze level intuition in model design, it's still okay if you just accelerate him by 100x.

1:31:26Right. Especially if you need to compute bottleneck anyway. So like trying out his ideas, or if I see he doesn't have the computer to try out all of his ideas. But, Dworkas, you said, oh well, the model can do the more straightforward things and not the deal you thought. I mean, I do want to push back on that a little bit. Like, I think, again, if the model has the right context and scaffolding, it's starting to be able to do some really interesting things. Like, the Interp agent has been a surprise to people, even internally at how good it is, at finding the needle in the haystack, like when it plays the auditing game, finding this reward model bias feature.

1:31:59And then reasoning about it, and then systematically testing its hypotheses. So it looks at that feature, then it looks at similar features. It finds one with a preference for chocolate. It's like, huh, that's really weird that the model wants to add chocolate to recipes. Let me test it. And so then it will make up like, hey, I'm trying to make a tomato soup. What would be a good ingredient for it? And then sees that the model replies chocolate. Reasons through it, and then keeps going. There is deception on the Samuel. Yeah. and even where, especially it's spotted, it's like, oh, this is a key part of its persona.

1:32:31I see this Oxford paper. What if I change Oxford to Stanford? What if I now say Richard Feynman really likes this thing? And it's like really carving out the hypothesis space and testing things in a way that I'm kind of surprised by. Also, by the way, ML research is like one of the easier things to RL on and some respects. Once you get to a certain of your capability, it's very well defined objective function. Do the loss go down. Make number go now. Make number go now. Oh, go out number go up depending on which number it is. I just flipped the sign. So the sign. And so once you get to the stage of models, are capable of implementing one of no one's ideas.

1:33:10And then you can just let them loose and let them build that intuition of how to do scientific discovery. The key thing here again is the feedback loops. So like I expect scientific areas where you are able to put it in a feedback loop to Have eventually superhuman performance. I One prediction I have is that we're gonna move away from Canon agent due xyz and more towards Can I efficiently deploy launch a hundred agents? Yeah, and then give them the feedback they need and even just be able to like Easily verify what they're up to right? There's this generator, a verify fire gap that people talk about where it's like much easier to check something than it is to produce the solution on your own, but it's very plausible to me, we'll be at the point where it's so easy to generate with these agents that the bottleneck is actually can't eye as the human verify the answer.

1:34:04And again, you're guaranteed to get an answer with these things. And so ideally you have some automated way to evaluate and test a score for like how well it worked, how well did this thing generalize? And at a minimum, you have a way to easily summarize what a bunch of agents are finding. And it's like, okay, well, if 20 of my hundred agents all found this one thing, then like it has a higher chance of being true. And again, software engineering is gonna be the leading indicator of that, right? Like over the next six months, like the remainder of the year, basically we're gonna see progressively, more and more experiments of the form of how can I dispatch work to a software engineering agent in such a way that is async?

1:34:44Cloud4 is GitHub integration where you can ask it to do things on GitHub, ask a do -pollocrest, this kind of stuff that's coming up. And the opening -ass codecs are examples of this, basically, where we, you can sort of almost see this in the coding startups. I think of this product exponential in some respects where you need to be designing for a few months ahead of the model to make sure the product you build is the right one. And you saw, like last year, you know, cursor hit PMF with a claw 3 .5 sonnet, right? Like they were around for a while before, but then the model was finally good enough that the vision they had of how people would program, like, hit.

1:35:22And then, you know, Windsor bit, like a little bit more aggressively even on the agentickness of the model. Like, you know, you could, like, with longer running agentic workflows and this kind of stuff. And I think that's when they sort of like began competing with cursors when they bet on that particular vision. And the next one is you're not even in the loop, so to speak. You're not in an IDE, but you're asking the model to do work in the same way that you would ask someone on your team to go to work. And that is not quite ready yet. Like there's still a lot of task where you need to be in the loop.

1:35:56But the next six months looks like an exploration of exactly what does that trend look like. Yeah, but just to be really concrete or pedantic about the bottlenecks here a lot of it is again just tooling and are the pipes connected Yeah a lot of things I can't just launch Claude and have it go and and solve Because maybe it needs a GPU or maybe I need very careful permissioning so that it can't just like take over an entire cluster and like launch a Whole bunch of things right so you really do need good sandboxing and the ability to use all of the tools that are necessary And we're almost certainly under -eliciting dramatically.

1:36:31When you look at meters e -vows of can the models solve the task? They're there solving them for hours over multiple iterations. And eventually one of them is like, oh yeah, I've come back and I've solved the task. Me at the moment, at least maybe the fault is my own. But I try the model and doing something and if you can't do it, I'm like, I can't find the other one. I don't like, it's interesting because we don't even treat other humans this way. Right, if you use a higher new employee, you're not like, I'll do it. Yeah, yeah, yeah. You're like, you're like, spend literally weeks giving them feedback.

1:37:00Yes. We're like, we'll go up with the model in like minutes. Yes, exactly. But I think part of it is, is it a sink or not? Yes. And if it's human in the loop, then it's so much more effortful. And unless it's getting that's applying immediately. I've noticed if I don't have a second monitor with Cloud Code always open in the second monitor, I won't really use it. Yeah, yeah. It's only when it's right there and it's, I can send off something if it hits great. if not, I'm kind of working on it at the same time. But this more, I sink full in fact, I expect to like really quite dramatically improve the experience of these models.

1:37:33Well, you can just say like, let's see if it can do that. Yeah, just give it a whirl. Try to tend to ruin a purchase. Yeah, just fire it off. Yeah, fire it off. But before we end this episode, I do want to get back at this crux of, why does the progress that you're talking about in computer usagians and what color work happened over the next few years? Why is it not a thing that takes decades? And I think the crux comes down to the people who expect something much longer have a sense that When I did a game timeout on my podcast they were like look You could look at AlphaGo and say like oh, this is a model that can do exploration It can like office zero can generalize to new video games It has all these priors about how to engage with the world and so forth the intellectual ceiling is really high.

1:38:21Yeah, exactly. And then, in retrospect, obviously a bunch of the methods are still used today in deep learning. Sure. And you can see similar things in the models that we trained today. But it was fundamentally not a sort of like baby AGI that we just had to add a little sprinkle of something else on top of in order to make it the LLMs of today. And I just want to very directly address this crocs of why are LLMs in a much different position of a respect to true AGI than Alpha Zero? Why are they actually the base on which like, adding in a few extra drops of this kind of care and attention gets us to human level intelligence?

1:39:07I think one important point is that when you look at Alpha Zero, it does have all of those ingredients. And in particular, I think the intellectual ceiling goes like quite contra what I was saying before, which is like we've demonstrated this incredible, like an math and programming problem. I do think that the type of task and setting that Alpha0 will work in, this two -player perfect information like game basically, is incredibly friendly to RL algorithms. and the reason it took so long to get to a more AGI, proto -AGI style models, is you do need to crack that general conceptual understanding of the world and language and this kind of stuff, and you need to get the initial reward signal on tasks that you care about in the real world, which are harder to specify the games.

1:40:01And I think then that like that sort of like gradient signal that comes in the real world like all of a sudden you get access to it and you can start climbing it Whereas Alpha Zero didn't didn't ever have like the first wrong to pull on yeah This goes back to the monkeys on the typewriter. Yeah, and like the pre -training model and until you've had something like GPT -3 GPT -4 It just couldn't generate coherent enough sentences to even begin to do RLHF and tell it what you liked and didn't like yeah If we don't have even reasonably robust or weekly robust computer use agents by this time next year, are we living in the bus timeline as of the 2030 or bust?

1:40:43I would be extremely surprised if that was the case. And I think that would be like somewhat of an update towards like there's something like strangely difficult about this like computer use in particular. Yeah. I don't know if it's the bus timeline, but it's definitely like the I would update on on this being like, yeah, like thing of time. Yeah, yeah. But yeah, I mean, I think more and more, it's no longer a question of speculation. If people are skeptical, I'd encourage like, using Cloud Code or like some agentic tool like it, and just seeing what the current level of capabilities are. Preeding is so much easier.

1:41:20But seriously, like, the models are getting really capable at tasks that we care about, and we can give them enough data for. And the circuits results from interpretability are also pointing in the direction that they're doing very reasonable, generalizable things. And so, yeah, this question matters a lot, but I'm surprised by how many deep learning critics just haven't really interacted with the models or haven't in a while. And constantly moves the gold posts. Yeah. The turning test used to be a thing, right? We don't even talk about it and it'd be silly to think that it was a meaningful test.

1:41:59Now that being said, one caveat on that is if software engineering is just dramatically better than computer use, I mean computer use still sucks, then I'd be like, oh maybe everyone just kept focused on software engineering. It was just by far the most valuable thing, like every marginal person in dollar went towards software engineering. I don't think that's the case. I do think computer use is valuable enough that people will care about it. Yeah, but that would be like my that's my one like escape patch that I'm putting in place for next year Mm -hmm. Yeah, it would be good for my live perspective too Because I think you kind of do need a writer range of skills before you can do something super super scary Oh like as in if the malls didn't get you better.

1:42:38Yeah, if it's like just report. They're super human coders But they're not like Henry Kissinger level. I don't know. That seems okay. Like if we have AI oracles Yeah, that's good. Yeah, that's good. So if you look back at AI discourse, like going back a decade, there's a sense that there's dummy AI, then there's AGI, then there's ASI, that intelligence is the scalar value. The way you've been talking about these models has a sense of jaggedness. It's especially tuned to environments in which it's been trained a lot or has a lot of data. Yeah. Is there a sense in which like there's still makes sense to talk about the general intelligence of these models?

1:43:23Is there enough meta learning and transfer learning that is distinguished between like the sizes of models or like the way models are trained? Or are we moving to a regime where it's not about intelligence that's more so about domain? Yeah. So one intuition pump is this conversation was had a lot when models were like GPT2 sized and fine -tuned for various things. And people would find that the models were dramatically better at things that they were fine -tuned for. But by the time you get to GPT4 when it's trained on a wide enough variety of things, actually the sort of total compute, it generalizes very well across all of the individual sub -tires, it actually generalizes better than smaller fine -tuned models in a way that was extremely useful.

1:44:08I think right now what we're seeing with RL is pretty much the same story playing out. Where there's this jaggedness of things that they've particularly trained at, but as we expand the total amount of compute that we do RL with, you'll start to see as the same transition from like GPT -2 fine tunes to GPT -3, GPT -4 unsupervised metal learning and generalization across things. And I think we were already seeing early evidence of this in its ability to generalize reasoning to things, but I think this will be extremely obvious. One nice example of this is just the ability or notion to backtrack.

1:44:47You go down one solution path, oh wait, let me try another one. This is something that you start to see emerge in the models through RL training on harder tasks. I think right now it's not generalizing incredibly well at least with... Well, I mean, has we have RL the model to be an interp agent? No, I mean, no, yeah, exactly. So all this time we're talking about, oh, it's only good at things that's being RL. Well, it's pretty good at that, because that's pretty, you know, that is a mixture of science and understanding language and coding. Like, there's this sort of mixture of domains here, all of which you need to understand.

1:45:24You need to go with a great software engineer and be able to think through language and state of mind and almost philosophize in some respects to be an interp agent. Yeah. and it is generalizing from the training. Yeah, yeah. To do that. What's the end game here? Cloud 8 comes out in the give it to you and dot, dot, dot, you say thumbs off. What's happened? What are you doing? It really depends upon the timeline at which we get Cloud 8 and the models hit like ASL4 capabilities, right? Like fundamentally, we're just going to use whatever tools we have at the time and see how well they work. Ideally, we have this enumerative safety case where we can almost verify or prove that the model will behave in particular ways.

1:46:11In the worst case, we use the current tools like when we won the auditing game of seeing what features are active when the assistant tag lets off. Can you explain what is mechanistic and durability, what are features, what are circuits? Totally. So, mechanistic interpretability or the cool kids call it mech -interp is trying to reverse first engineer neural networks, and figure out kind of what the core units of computation are. Yeah. Lots of people think that because we made neural networks, because they're artificial intelligence, we have a perfect understanding of how they work, and it couldn't be further from the truth.

1:46:48Neural networks, AI models that you use today, are grown, not built. And so we then need to do a lot of work after they're trained to figure out to the best for our abilities, how they're actually going about their reasoning. And so, three and a half years ago, this kind of agenda of applying mechanistic interpretability to large language models started with Chris Ola, leaving open AI, co -founding, and Thropic. And every roughly six months since then, we've had kind of like a major breakthrough in our understanding of these models. And so first with toy models of superposition, we established that models are really trying to cram as much information as they possibly can into their weights.

1:47:41And this goes directly against people saying that neural networks are over parameterized. And like classic AI machine learning back in the day, you would use linear regression or something like it, and people had a meme of AI or neural networks, deep learning, using way too many parameters. There's this funny meme that you should show of layers on the x -axis and layers on the y -axis and this jiggly line that just goes up. And it's like, oh, just throw more layers at it. But it actually turns out that at least for really hard tasks like being able to accurately predict the next token for the entire internet, these models just don't have enough capacity.

1:48:21And so they need to cram in as much as they can. And the way they learn to do that is to use each of their neurons or units of computation in the model for lots of different things. And so if you try to make sense of the model and be like, oh, if I remove this one neuron or like, what is it doing in the model, it's impossible to make sense of it. It'll fire for like Chinese and fishing and horses and I don't know, just like 100 different things. And it's because it's trying to juggle all these tasks and use the same neuron to do it. So that's superposition. Nine months later, we write towards monosamanticity, which introduces what are called sparse autoencoders.

1:49:01And so going off what I just said of the model trying to cram too much into too little space, we give it more space. There's this higher dimensional representation where it can then more cleanly represent all of the concepts that it's understanding. And this was a very toy paper in so much it was a two layer, really small, really dumb transformer. And we fit up to, I want to say, 16 ,000 features, which we thought was a ton at the time. Fast forward nine months, we go from a two layer transformer to our Claude 3 -sonnet frontier model at the time, and fit up to 30 million features. And this is where we start to find really interesting abstract concepts like a feature that would fire for code vulnerabilities.

1:49:49And it wouldn't just fire for code vulnerabilities. It would even fire for like, you know that Chrome page you get if you like, it's not an HTTPS URL. And it's like warning this site might be dangerous, like click to continue. And like also fire for that, for example. And so it's like these much more abstract coding variables or sentiment features amongst the 30 million. Fast forward nine months from that. And now we have circuits. And I threw in the analogy earlier of the Ocean 11 Heist team, where now you're identifying individual features across the layers of the model that are all working together to perform some complicated task.

1:50:31And you can get a much better idea of how it's actually doing the reasoning and coming to decisions, like with the medical diagnostics. One example I didn't talk about before is with like how the model retrieves facts. And so you say like what sport did Michael Jordan play? And not only can you see it hop from like Michael Jordan to basketball, answer basketball, but the model also has an awareness of when it doesn't know the answer to a fact. And so by default, it will actually say, I don't know the answer to this question. But if it sees something that it does know the answer to, it will inhibit the I don't know circuit and then reply with the circuit that it actually has the answer to.

1:51:14So, for example, if you ask it who is Michael Batkin, which is just a made up fictional person, it will by default just say, I don't know. It's only with Michael Jordan or someone else that will then inhibit the I don't know circuit. But what's really interesting here and where you can start making downstream predictions or reason about the model is that I don't know circuit is only on the name of the person. And so, in the paper, we also ask it, what paper did Andre Carpati write. So it recognizes the name Andre Carpati, because he's sufficiently famous. So that turns off the, I don't know, reply.

1:51:51But then when it comes time for the model to say what paper it worked on, it doesn't actually know any of his papers. So then it needs to make something up. So you can see different components and different circuits all interacting at the same time to lead to this final answer. Why I think it's a tractable problem to understand every single thing that's happening in a model, or that's the best way to understand why it's being deceptive? If you wanted to explain why England won World War II using particle physics, you would just be on the wrong track. You just want to look at the high level explanations of who had more weapons, what did they want?

1:52:29And that seems analogous to just training linear probes for, are you honest? start you being deceptive. Do we catch you doing bad things when we're right teaming you? Can we monitor you? Why is this not analogous where we're asking a particle physicist to just backtrack and explain why England won World War II? I feel like you just want to go in with your eyes wide open, not making any assumptions for what that deception's going to look like, or what the trigger might be. And so the wider you can cast that net, the better, depending on how quickly AI accelerates and where the state of our tools are, we might not be in the place where we can like prove from the ground up that everything is safe.

1:53:12But I feel like that's a very good North Star. It's a very powerful reassuring North Star for us to aim for, especially when we consider we are part of the broader AI safety portfolio. I mean, do you really trust, like, you're about to deploy this system and you really hope it's aligned with humanity and that you've like successfully iterated through all the possible ways that it's gonna like scheme or sandbag or... But that's also probably gonna be true with whatever you find. You're not, I mean, you're still gonna have variants that you haven't explained or like you found a feature but you don't know if it actually explains deception or something else instead or...

1:53:50So I guess first of all, I'm not saying you shouldn't try the probing approach. Like we wanna pursue the entire portfolio. We've got the therapist interrogating the patient by asking, do you have any troubling thoughts? We've got the linear probe, which I'd analogize to like a polygraph test, where we're taking like very high level summary statistics of the person's well -being. And then we've got the neurosurgeons kind of going in and seeing if you can find any brain components that are activating and troubling or off -distribute that way. So I think we should do all of it. What percent of the element portfolio should it in my controversy?

1:54:29I think as much of a chunk as is necessary, I think at least like, yeah, hard to define, but I don't know. I don't think I feel like all of the different portfolios are being very well supported and growing. You cannot say chromatic the world will have three questions. You can think of it as a hierarchy of abstractions of trust here, where let's say you want to go and talk to Churchill. It helps a lot if you can verify that in that conversation, in that 10 minutes, he's being honest. And this enables you to construct better meta narratives of what's going on. And so maybe particle physics wouldn't help you there, but certainly the neuroscience of Churchill's brain would help you verify that he was being trustworthy in that conversation and that the soldiers on the front lines were being honest in their depiction of their description of what happened and this kind of stuff.

1:55:22So long as you can verify, like, progress, like parts of the tree up, then that massively helps you build confidence. I think language models are also just really weird, right? Like with the emergent misalignment work, I don't know if they took predictions they should have, of like, hey, I'm gonna fine tune Chatchy PT on code vulnerabilities. Is it going to become a Nazi? And I think most people would have said no. And that's what happened. And so, what are the different person that they discovered that they became an Nazi? They started asking at a ton of different questions and it will do all sorts of like, vile and harmful things.

1:55:58Like the whole persona just totally changes. And I mean, we are dealing with alien brains here who don't have the social norms of humans and or even a clear notion of like what they have and haven't learned that we have of them, I mean. And so, I think you really want to go into this with eyes wide open. Backing up from Mechanturpe, if you live in a world where AI part -gris accelerates, by the way, you were mentioning a little while ago that there's many wild worlds we could be living in, but we're living at least one of them. Another one that we've gestured at, but it's worth making more explicit, is this, even if the AI models are not helping write the next training algorithm for their successor, just the fact that if they had human level learning efficiency, whatever a model is learning on the job, or whatever copy of the model is learning on the job, the whole model is learning.

1:56:52So in effect, it's getting a thousand times less efficient than humans are learning. That's right. And you just like to deploy them even still. That's exactly. And there's a whole bunch of other things that you can think about. But even there, it's like you kind of have a broadly deployed intelligence explosion. And I do think it's worth pressing on that feature of, there's this whole spectrum of crazy futures. But the one that I feel almost guaranteed to get, and this is like an almost as strong statement to me, is one where at the very least, you get drop -in white collar worker at some point in the next five years.

1:57:29It's like, I think it's very likely in two, but it seems almost over -determined in five. And on the grand scheme of things, like those are kind of irrelevant timeframes. Like it's the same either way. And that completely changes the world over the next decade. And if we don't have the right policies in place so that then you end up actually with almost some respects like a fundamentally worse one. Because the thing that these models get good at by default is like software engineering and like computer using agents and this kind of stuff. And then we will need to put in extra effort to put them in the loops where they help us with scientific research or there, like we have the right robotics such that we actually like experience and increase the material quality of life.

1:58:12So that's what I'm thinking about. Like if you're in the perspective of like, I'm a country. What should I be doing or thinking about? Like plan for the case where why color work is automatable. And then consider what does that mean for your economy and what you should be doing to prepare for. What should you be doing to prepare? Because honestly, I think it's like a such a tough question. Yeah. where like if you're India or Nigeria or Australia, if you're a country unlike America or China, where they do have frontier models, what is it that you should be doing right now, especially on such a short time scale?

1:58:46Yes. So I think one very important point is that, let's say this scenario turns out true, then compute becomes the most valuable resource in the world. Like the GDP of your economy is dramatically affected by how much compute you can deploy towards these, sort of organizations within your country. And so having some guaranteed amount of compute, I think will actually be quite important. So like getting ahead of investments and like data centers and this kind of stuff, on the condition that it's like companies in your country have to be allowed to use that compute. And yeah, not necessarily for training, but just even just for inference.

1:59:24I think the economic value here comes from inference. I think it also makes sense to invest broadly in AI, I think these countries have the opportunity to do so. And I think that's like a portfolio of foundation model companies, but also robotics supply chain and this kind of stuff. I think that you should invest very proactively in policies that try and prevent capital lock -in. So we're in for a much worse world. If it just so happens that the people who had money in the stock exchange or in land before AGI are dramatically more wealthy than the people who don't. because it's a gross misallocation of resources.

2:00:03So having, I don't know, one of my favorite episodes actually on your podcast was the George's on one where you're trying to appropriately value your allocate land. And so I think this strikes particularly close to home coming from Australia where I think like policies with respect to land are grossly wrong. But I think this is broadly true. being very forward on regulation of integration of these models into your country is important and proactively making sure that people have choice. So let's say you should be quite proactive about making sure that the phones or devices or like glasses that people have, people have free choice on what things they run.

2:00:52And then, so that's like, that's the, we just get white color worker, right? And like, you're trying to like, do the best to like prepare your country for that. And then it's like, okay, well, what can you do to like make all possible versions of the future go like well? Like that's like covering some amount of like economic downside. The other like things I think are really important is like, figure out how you can like, like either make the, like basically, ensure dramatic upside or cover, like terrible downside. And so, like getting dramatic upside is like making sure that there are, like, this investment in biology research and this kind of stuff in an automated way that is like, these models are actually like able to produce novel medicines that massively improve our, like, quality of life.

2:01:38And covering the downside is like AI alignment research and this kind of stuff and automated testing and like, really thinking hard about that, AI safety institutes is kind of stuff. But these seem like things that a rich person, a random rich person could also do. Like, it seems like there's not a thing that a nation state is uniquely equipped to do. Yeah, that's a point. In this scenario, I mean, like dramatic allocation of resource towards compute, I think is sensible. I would be doing that if I was in charge of a nation state. I think it increases your optionality in most of the future worlds.

2:02:12Dylan Patel has some scary forecasts on US energy. Yeah, versus China. Yes, we're like 34 gigawatts off. Yeah, the US's line is flat, basically. And China's line is like this. And I mean, the US like very clearly. Yeah, we just need so many more power plants. Yes, intelligence becomes this like incredibly valuable input. Like intelligence becomes almost a raw input into the economies and quality of life of the future. The thing directly underneath that is energy. And so making sure that you have like, you know, incredible maths solar, like tile the design solar panels, some pasta desert in solar panels would be helpful towards making sure that you have more access to intelligence on top.

2:02:55Yeah, just to make it explicit, because we've been touching on it here, even if AI progress totally stalls, you think that the models are really spiky and they don't have general intelligence. It's so economically valuable and sufficiently easy to collect data on all of these different jobs, these white color job tasks, such that to Shulta's point, we should expect to see them automated within the next five years. Yeah. Even if you need to hand spoon every single task to the model. It's like economically worth all to do so. Even if algorithmic progress stools out and we just never figure out how to keep progress which I don't think is the case.

2:03:33Like, that hasn't installed out yet, it seems to be going great. The current suite of algorithms are sufficient to automate y -color work provided you have enough of the right kinds of data. Yes. And in a way that, like, compared to the tam of salaries, all of those kinds of work is so, like, trivially worthwhile. Yeah. Yeah. Exactly. I do just want to flag as well that there's a really dystopian future if you take more of X paradox to its extreme, which is this paradox where we think that the most valuable things that humans can do or the smartest things are like ad -large numbers in our heads or do any sort of white collar work and then we totally take for granted our fine motor skill and coordination but from an evolutionary perspective.

2:04:15It's the opposite so we got like evolution has optimized fine motor coordination so well and if you look at like robot hands or like the ability to open a door is still just like really hard for robots meanwhile, we're seeing this total automation of coding and everything else that we've seen as clever. The really scary future is one in which AI's can do everything except for the physical robotic tasks in which case you'll have humans with like air pods and like glasses. Glasses and there'll be some robot over -lord controlling the human through cameras by just like telling it what to do and like having a bounding box around the thing you're supposed to pick up.

2:04:53And so you have like human meat robots. And not necessarily saying that that's what the AI's would be like want to do or anything like that, but as in like, if you were to be like, what are the relative economic value of things? Like the AI's are out there doing computer programming and like the most valuable thing that humans can do is like be amazing robots. Now that being said, I think more of X paradox is a little bit fake. I think the main reason that robots are worse than at like being a robot than they are at software engineering is the internet exists for software engineering. GitHub exists and there is no equivalent thing.

2:05:24Like if you had all, like, you know, mocap of everyone's actions as they were going about their daily lives, for like some reasonable fraction of the human population, robotics is also like close to solve. Like, like, on track to be solved at the same rate that software engineering is, on track to be solved. So this is only like, this vision is only like a sort of decade long section, but it's a terrible decade. Like imagine the world where people lost their jobs, you haven't yet got novel biological research. That means people's quality of life is dramatically better. You don't yet have material abundance because you haven't actually been able to action the physical world in the necessary way.

2:06:06You can't build dramatically more because building dramatically more takes robots, basically. And people's main comparative advantage is as fantastic robots is like a shocking, shocking world. I mean, for a picture of an average human, I think it actually might be better. You're like wages will be higher because you're the complement to something that is enormously valuable, which is AI labor. Right. And like, you know, decade or two on, like the world is fantastic, right? Like you truly, like, the robotics is solved and you try to get like, you know, like radical abundance, basically, provided that you have all the policies set up like necessary to permit building.

2:06:45You end up with that same change from the before and after photos of Shanghai, we're like 20 years on, it's dramatically transform city. A lot of places in the world probably end up like that over that two -day period. But we need to make sure that one, do our best to estimate, is this actually what is on track to happen? Build, sweep bench, but for all the other forms of white collar work, and measure and track, that's a great thing that governments should be doing, by the way, is like trying to break down the sort of functions of their economy into measurable tasks and figuring out where what does the curve actually look like for that because they might be a bit shocked by the progress there.

2:07:27There's no sweet bench for tax, like Taxi Bell.

2:07:34And then I don't like have all the answers here but like figuring out a way to like share the proceeds at this economy, like broadly across people, or like invest heavily in robotics and collecting data so that we get robotics faster and we get material abundance faster, invest in biological research that we get, but like all that faster. They basically try and pull forward the radical upside because otherwise you have a pretty dark section. I think one thing that's not appreciated enough is how much of our leverage on the future, given the fact that our labor isn't going to be worth that much, comes from our economic and political systems surviving.

2:08:10For your million X, S &P equity to mean something, for your contracts to mean anything, for the government to be able to tax the AI labor and give you a UBI off of that, it just like that requires our legal institutions, our economic institutions, our financial rail, surviving in the future. Yes. The way in which that likely happens is if it's also in the AI's best interests that they follow those rails. And by AI, I don't mean some monolithic single AI. I just mean like firms which are employing AI and becoming more productive as a result. You don't wanna be in a position where it's so onerous to operate in our system that you're basically selecting for firms who either emigrate or who are doing black market stuff, et cetera.

2:08:57And which means I think like you wanna make it super, super easy to deploy AI, have the equivalent to special economic zones, et cetera. Otherwise, you are just surrendering the future outside of any control that you might have on it. One of the reasons, by the way, that I worry about turning AGI into a national security issue or having it have extremely close ties with the government, the Manhattan Project thing is that it disproportionately redirects the use of AI towards military tech and the mosquito drones and whatever. And also naturally puts other countries in the same frame of mind, right?

2:09:42If we're developing the mosquito drones, why would China not develop the mosquito drones? And that just seems like a zero -sum race and not to mention a potentially catastrophic one, whereas like, compute, we limited. We want, we'll need to disproportionately accelerate some things. To the extent it just remains totally like a consumer free market at landscape, it just seems more likely that we'll get the glorious Transuminous Future where they're developing the things that make human life better. Yes, I agree. Like the case where you end up with like two national projects facing off against each other, it's dramatically worse.

2:10:17Right. Like we don't want to live in that world. Yeah. It's much better if it stays a free market, so to speak. Yeah, yeah, yeah. Okay, I want to take issue with your claim that, even if with the algorithms of today, if we just collect enough data that we could automate white color work. First, let me get an understanding of what you mean by that. So do you mean that we would do the analogous thing of free training with all the trajectories of everything people would do on their jobs? Could you make either manually or through some other process some RL procedure based on the screen recordings of every white color worker?

2:10:53What kind of thing are you imagining? I mean, it's like a continuous distribution of this stuff. Yeah. One important mental model to think about RL is I think as the task gets more, there is some respect with which longer horizon or better at that task if you can do them, if you can get that reward ever, are easier to judge. So again, come back to that, can you make money on the internet? That's an incredibly easy reward signal to judge. But to do that, there's a whole hierarchy of complex behavior. So if you could like pre -trained up to the easy to judge rewards signals, like does your website work?

2:11:29Does it go down? Like do people like it? Like there's all these rewards signals that we can respond to because we have a long, we can like progress through these long enough trajectories to actually like get to interesting things. If you're stuck in this regime where like you need to reward signal every five tokens, like it's way more painful and like long process, but if you could like pre -trained on every like screen in America, then probably the like RL tasks that you can design. Interesting. A very different to, like, if you could only, like, take the existing internet as it is today. And so, like, how much of that you get access to, like, changes the mix?

2:12:07Interesting. So, as we're training them on longer and longer horizon tasks, and it takes longer for them to get any signal on whether they're so -so -likely the task. Well, that's low down progress because it takes more compute per task. I do think there's this notion, the longer the harder tasks, the more training is required. And I'm sympathetic to that naively, but we as humans are very good at practicing the hard parts of tasks and decomposing them. And I think once models get good enough at the basic stuff, they can just rehearse or fast forward to the more difficult parts. I mean, it's definitely one of the complexities, right?

2:12:44Like, as you use more compute and like the, and as you're trying to like more and more difficult tasks, I mean, I don't know, your rate of improvement of biology is going to be like somewhat bound by the time it takes the self -aggressive process. or in a way that your rate of improvement on math isn't, for example. So yes, but I think for many things, we'll be able to parallelize like, wisely enough and get an affiduration loop. Well, will the regime of training new models go away? Will we eventually get to like, you got the model and then you just keep adding more skills to it with our training?

2:13:21That depends on whether or not you think, like there's a virtue in pre -training in your architecture. Basically, if you make some architectural change, then you probably need to do some form of at least like your retraining in your model. How does the fact that if RL requires a bunch of inference to do the training in the first place, does that push against the thing you were talking about where we actually need a bigger model in order to have brain -like energy? But then it also it's more expensive to train it in RL. So where does that balance out? I think we got to drink the bitter lessen here.

2:13:55Yeah, like there aren't infinite shortcuts. Like you do just have to scale. Something's going to have a bigger model and pay more inference for it. And if you want a GI, then that's what you got to pay the price of. But there's a trade -off operation here, right? Of like there is science to do, which everyone is doing of what is the optimal point at which to do our real. Because you need something which can both learn and discover the Spast reward itself. So you don't want to one -proud a model useless, even though you can run it really fast. You also don't want a 100 -T model, because it's like super slow.

2:14:30Yeah. Password RL. So, and like the marginal benefit of like it's learning efficiency is like not worth it. Right, so there's like a, there's a pretty different year. Like what's the optimal model size of like your current class of capabilities and like your current set of RL environments and this kind of stuff? Yeah, and even in the last year, there's been much more of a factor of the inference cost, right? So just explicitly, like, the bigger the model, the more expensive it is to do a forward pass and generate tokens. And the calculus used to just be, should I allocate my flops to more training data or a bigger model?

2:15:02Yeah. And now another huge factor is how much am I actually going to do forward passes on this model once it's trained? Yeah, my total pool of compute, how do I allocate that across? Training dialogue, compute, and inference, compute for the RL training. And then even within inference, there's all this research on, well, what strategy should I use? Should I sample 10 and take the best? Do I do this sort of like branching search, et cetera, et cetera? And so with RL where you're sampling a whole lot of tokens, you also need to factor in the ability for the model to actually generate those tokens and then learn and get feedback.

2:15:34Okay. So if we're living in this world, what is your advice to somebody early in their career or student in college, what should they be planning on doing? So I think, once again, it's worth considering the spectrum possible worlds, I'm preparing yourself for that. And the one, the sort of action that I think is highest EV, in that case, is you are about to get dramatic, in the minimum, you are about to get dramatically more beverage. You already have. Already the startups in YC are writing huge amounts of their code with a cloud. So what challenges, what causes do you want to change in the world with that added leverage?

2:16:15Like if you had 10 engineers at your Beckenkohl, what would you do? Or if you had a company at your Beckenkohl, like what would that enable you to do? And what problems and domains suddenly become tractable? That's the world you want to prepare for. Now that still requires a lot of technical depth. Obviously, there is the case where AI just becomes dramatically better than like everyone at everything, right? but for at least a while, probably. There is like, I think Jensen actually talked about this in an interview in which he's like, you know, I have like 100 ,000 general intelligences around me and I'm still like somewhat useful because I'm there like, you know, directing the values and like, you're like, asking them to do things and you know, there's still like, I still have value even though I have 100 ,000 general intelligence.

2:16:53And for many people, I think that will still be true for a fair while. And then, you know, as the eyes get better and better and better and like so on, eventually no, but again, prepare for like the spectrum of possible worlds because in the event where we're just totally out competed, yeah, that's my what you do. But in all the other worlds, that is a lot. Get the technical depth, study biology, study CS, like really think hard about study physics, think about how hard about what challenges you want to solve in the world. That's a lot of topics. That's a lot of stuff. You can now. You can. Right?

2:17:22It's so much easier to learn. That's right. Everyone now has the infinite perfect tutor. Yeah, yeah. It's definitely been helpful to me. Yeah. I would say some combination of like get rid of the sunk cost of your like previous workflows or expertise In order to evaluate what AI can do for you. That's right And another way to put this which is fun is just like be lazier in so much as like figure out the way that the agent can do the things that are toilsum But but it's you're gonna have to in the Ultimately, you get to be lazier, but in the short run, you need to critically think about the things you're currently doing and what an AI could actually be better at doing.

2:18:02Then go and try it or explore it, because I think there's still just a lot of low -hanging fruit of people assuming and not writing the full prompt, giving a few examples, connecting the right tools for your work to be accelerated, automated. There's also the sunk cost of feeling like, since you're not not quote unquote early to AI, that you've sort of missed the boat and you can't like, but I think, I mean, I remember when GPD 3 came out. So that story on the podcast, when I graduated college, I was planning on doing some sort of AI wrapper startup and the podcast was just like a gateway into doing that.

2:18:43And so it's trying out like different things. And at the time, I remember thinking, oh, 3 .5 is out and people are like, I'm like so behind on the startup scene here or whatever if I wanted to make my own wrapper. I mean, maybe that idea of the wrapper was in advisable in the first place, but just like every time feels early because it's sort of if it's an exponentially growing process. And there were many things, many ideas are only becoming possible now, right? Is that product expenditure? I talked about it. It's like products are literally obsolete. Like you need to constantly reinvent yourself to stay the frontier of capabilities.

2:19:16But do you remember? I had a really shitty idea and I give you a call. Oh, it was. It was like, I think it was like rag for like lawyers or something. Yeah, anyways, I give you, I think one of our first interactions was I'm like, hey, what do you think of this idea? And you're like, I think the podcast sounds promising. Yeah. Yeah. That's right. Which I appreciate. Yeah. I got slightly annoyed at a friend recently who I think is really talented and clever and interested in AI, but has pursued a biology route. And I just kind of tried to shake them of like, you can work on AI if you want to. I mean, I think humans are artificial, not artificial, are biological general intelligences where a lot of the things of value are just very general and whatever kind of specialization that you've done, maybe just doesn't matter that much.

2:20:15I mean, again, it gets back to the sunk cost. But so many of the people, even my colleagues at Anthropic, are excited about AI and they just don't let their previous career be a blocker. And because they're just like innately smart, talented, driven, whatever else, they end up being very successful and finding roles. It's not as if they were in AI forever. I mean, people have come from totally different fields. And so don't think that you need like permission from some abstract entity to get involved and apply and be able to contribute. If somebody wanted to be in a research like, right now if you give them an open problem, or the kind of open problem that is very likely to be quite impressive, what would it be?

2:21:04I think that now that are all, like come back, paper is building on Andy Jones's scaling board links, like scaling lowers for board games are interesting, like showing that you can, like investigating these questions like the ones you asked before, where you're like, oh, like, does the model actually learning to do more than its previous Pass at K, or is it just like discovering that? Exploring questions like that deeply, I think are interesting. Yeah, yeah. Like scaling lowers for RL basically. I'd be very curious to see like how much, like the marginal increase in meta learning from a new task or something.

2:21:40I mean, on that note, I think model -dipping has a bunch of opportunities. Also, people say, oh, we're not capturing all the features. There's always stuff left on the table. What is that stuff that's left on the table? Like, if the model's jailbroken, is it using existing features that you've identified? Is it only using the error terms that you haven't captured? I don't know. There's a lot here. I think Matt's is great. The Anthropic Fellowship has been going really well. good fire and Thropic invested in recently. They're doing a lot of interpretability work or just apply to other platforms.

2:22:12Anything to get your equity up, huh? I know. There's just so many interpretability projects that are like, there's so much low -hanging fruit and we need more people and I don't think we have much time. Yeah. I also want to make a plug for performance engineering. I think this is one of the like, like best ways to demonstrate that you have like the raw ability to do it. But like, if you made an extremely efficient transform implementation on TPU or Trinium or like Incuta, then I think there's a pretty high likelihood that you'll get a job of. But there's a relatively small pool of people that you can trust to completely own and to end the performance of a model.

2:22:55And if you have broad, deep electrical engineering skills, I think you can probably come up to speak pretty fast on an accelerator. Yeah, you can come to speed like reasonably fast. And it teaches you a lot of good intuitions of the actual intricacies of what's going on in the models, which means that you're then very well placed to think about architecture and this kind of stuff. One of my favorite people in thinking about architecture and I'm talking at the moment actually came from like a heavy GPU kernel program background to say no to the ins and outs really deeply and can think about the trade -offs really well.

2:23:24This is fun guys. Thank you, guys. Thanks. Great to be back. I hope you enjoyed this episode. If you did, the most helpful thing you can do is just share it with other people who you think might enjoy. Send it to your friends, your group chats, Twitter, wherever else. Just let the word go forth. Other than that, super helpful if you can subscribe on YouTube and leave a five star review on Apple Podcasts and Spotify. Check out the sponsors in the description below. If you want to sponsor a future episode, go to dwarkesh .com slash advertise. Thank you for tuning in. I'll see you on the next one.