AMA: career advice given AGI, how I research ft. Sholto & Trenton

0:00Today, this is going to be an Ask Me Anything episode. I'm joined with my friends Trenton Bricken and Shulta Douglas. You guys do some AI stuff, all right? A double. They're researchers at Anthropic. Other news, I have a book launching today. It's called a Scaling Era. I hope one of the questions ends up being, why you should buy this book. But we can come two words with one stone. But, okay, let's just get at it. What's the first question that we got to answer? Thanks, man. So I want to ask the flyball question that I heard before of why should ordinary people care about this book? Like why should my mom buy and read the book?

0:38Yeah. First, let me tell you about the book, what it is. So you know, these last few years, I've been interviewing AI lab CEOs, researchers, people like you guys obviously, but also scholars from all kinds of different fields, economists, philosophers, and they've been addressing, I think what are basically the gnarliest, most interesting, most important questions, we've ever had to ask ourselves, like, what is the fundamental nature of intelligence? What will happen when we have billions of extra workers? How do we model out the economics of that? How do we think about an intelligence that is greater than the rest of humanity combined?

1:19Is that even a coherent concept? And so what I'm super delighted with is that with Stripe Press, we made this book where we compiled and curated the best most insightful snippets across all these interviews. And you can read Dario addressing wider scaling work. And then on the next page is Demis explaining deep minds plans for whether they're going to go the RL route and how much the Office Euro stuff will play into the next generation of LLums. And on the next pages, of course, you guys going through the technical details of how these models work. And then there's so many different fields that are implicated.

1:59I mean, I feel like AI is one of the most multi -disciplinary fields that one can imagine because there's no field, no domain of human knowledge that is not relevant to understanding what a future society of different kinds of beings will look like. There's, you know, you can have that carol shulman talk about how the scaling hypothesis shows up and primate brain scaling from chimpanzees to humans. On the next page might be an economist trying to argue with Tyler Cowan explaining why he doesn't expect explosive economic growth and why the bottlenecks will eat all that up. Anyways, so that's why you're motioned by this bug.

2:32It's just like it is the distillation of all these different fields of human knowledge applied to the most important questions that humanity facing right now. I do like how the book is sliced up by different topics and across interviews. Yeah, yeah. So it does seem like a nice way to listen to all of the interviews in one digestible way. There's two interviews I've done that haven't been released publicly before that are in the book. So one was that Jared Kaplan, who's one of your co -founders, and this is another example where he's like a physicist and he's explaining scaling from this like very mathematical perspective about data manifolds.

3:12And then on the next page, you have a totally different perspective. It's like Goran talking about, you know, why can't we just have like distilled into it? Like why did general intelligence actually evolve in the first place? What is the actual evolutionary purpose of it? And it's like page by page, right? You can just get addresses. Even for me, I mean, like the person who's been on the other end of these conversations was actually really cool to like read it and just be like, oh, I actually like, I now I realize how these insights connect to each other. Yeah, the only other thing that stood out to me as well is the introduction section.

3:39I don't think this is not to you. Like, yeah, that was really the only thing that was not worthy. I just mean in turn, stood out in accessibility is the introduction section and the diagrams for like all the different inputs that enable you to train a machine learning model. Strike press books are also just beautiful and have these nice like side captions for explaining what parameters are, what a model is, these sorts of things. Actually, when we did our episode together, a bunch of people, I don't know if you saw this, independently made these blog posts and on key cards and shit where they're like explaining the cause because we just kind of passed over some things.

4:19And hopefully we've given a similar treatment to every single interview I've done where you can read a very technical interview with a lap C or something or an engineer or a researcher. And then the side is like here, here's like more context, there's more definitions, here's more commentary. And yeah, I feel like it elevated the conversations. So in other words, my parents will finally understand what I do for jobs. I wouldn't do it. They're gonna write well. In my parents' world. You're gonna write as well. All my need to know is that my name's an unbooking. You're a co -owner. They're like cool.

4:55Should we get into the A -Made questions? Let's do this. All right. So Brian Krav asks, the issue you raised with Dario, and occasionally tweet about relating to models not making connections across some sort of combinatorial attention challenge. What are your thoughts on that now? Do you solve it with scale, thinking models or something else? By the way, so the issue is, one of the questions I asked Dario is, look, these models have all of human knowledge memorized. And you would think if a human had this much stuff memorized, and they were moderately intelligent, they couldn't be making all these connections between different fields.

5:27And there are examples of humans doing this, by the way. There's Donald Swan or something like this. This guy noticed that the way what happens to a brain after magnesium deficiency is exactly the kinds of, I don't know, the structure you see during a migraine. So he's like, okay, you take magnesium supplements and we're gonna cure a bunch of migraines. And it worked. And there's many other examples of things like this where you just like notice two different connections between pieces of knowledge. Why if these LLMs are intelligent or they're not able to use this unique advantage they have to make these kinds of discoveries?

5:59I feel a little shy, like me giving answers, on the ice shit. What do you guys hear? But so, Ashley Scott Alexander addressed this question in the museum a the writes and he's like, look humans also don't have this kind of logical omniscience, right? It's, he was the example of in language, if you really thought about like why are two words connected? It's like, oh, I understand like why, why rhyme has the same etymology as the other word? If you really thought about, but you just like don't think about it, right? There's like this common story of explosion. I don't know if that addresses the fact that human, we know humans can do this, right?

6:30But the humans have, in fact, done this. And I don't know of a single example of LLM's ever having done it. Actually, yeah, what is your answer to this? I think my answer at the moment is that the pre -training objective doesn't necessarily like, in Buzer, there's like nice flexible general knowledge about the world, but doesn't necessarily imbue the skill of making novel connections or like research. Yeah. The kinds of things that people are trained to do through PhD programs and through like sort of the process of exploring and interacting with the world. And so I think like at a minimum, you need significant RL in at least similar things to be able to approach, like making novel discoveries.

7:10And so I would like to see some early evidence of this as we start to build models that are sort of interacting with one, trying to make scientific discoveries and like, sort of like modeling the behaviors that we expect of people in these positions. Because I don't I actually think we've done that in a meaningful, scaled way as a field, so to speak. Yeah, riffing off that with respect to RL, I wonder if models currently just aren't good at knowing what memories they should be storing. Like most of their training is just predicting the next word on the internet and remembering very specific facts from that.

7:42But if you were to teach me something new right now, I'm very aware of my own memory limitations. And so I would try to construct some summary that would stick. And models currently don't have the opportunity to do that. Memory scaffolding in generals just very primitive right now. Right. Well, it plays part 2 right now. Exactly. Yeah, we're like someone worked on it. It was awesome. It got far, but another excited and thropic employee then like iterated on the memory scaffold and was able to like very quickly improve on it. Interesting. So yeah, that's the one. I do also just wonder if models are idiots of aunts.

8:19The best analogy might be to Kim Peak. So Kim Peak was born without a corpus callosum, if I recall correctly. Each hemisphere of his brain operated quite independently. He could read a page of a book, so he'd open a book, there'd be two pages visible. Each eye would read one of the pages. And he had like a perfect encyclopedic memory of like everything he'd ever read. But at the same time, he had other debilitations functioning socially, these sorts of things. And it's just kind of amazing how good LLMs are at very niche topics, but can totally fail at other ones still. I really want to double click on this thing of why there's this trade -off between memorization.

9:01Like why does cutting it off? Like apparently it's sort of connected to this debilitation. But like Wiki Text is not that, it's like five megabytes of information. The human brain can store much more. So what is a human brain just not want to still memorize these kinds of things? and is actively pruning. And yeah, I don't know. But we don't have to do it right now. We'll do it for whatever time. The one thing I'll say on that is like, there was another case to use someone with a perfect memory. So they never forgot anything. But their memory was too debilitating. It'd be like your context window for the transformer is like trillions of tokens.

9:36And then you spend all your time attending to pass things and like are too trapped in the details to extract any meaningful generalizable insights from it. Terrence Deacon, whose book he recommended, had this interesting insight about how we learn best when we're children. But we forget literally everything that happened to us when we were children, right? We have total amnesia. And adults have this in between where we don't remember exact details, but we can still learn in a pretty decent way. And then, Elements are in the opposite end of this gradient where they'll get the exact phrasing of wiki text down, but they won't be able to generalize in these very obvious ways.

10:10Yeah, a little bit like Gwen's theory, optimize a theory, no? Yeah, yeah, yeah, yeah. I think I probably got it from that. Yeah. Yeah, yeah, yeah. Yeah, yeah, yeah. Gwen has definitely had a big influence on all this from this. That's right. I mean, I feel like what's underappreciated on the podcast is like, we have this like group chat and we also just like meet up a lot in person. And just all the offer from the podcast just comes from you and a couple other people just feeding me like ideas and nudges and whatever. And then I can just use that as an intuition pump during the conversation. Yeah, you're not the only one.

10:42What do you mean? Oh, like I'd benefit immensely from just hearing what everyone else has to say. Yeah, I've done. Yeah, it's all regurgitation and like, I got a question. Yes. Maybe rabid, maybe rabid monkey asks, imagine you have a 17 year old brother slash nephew just starting college. What would you recommend he study given your AGI timelines? I don't know, become a podcaster. I feel like that job's still going to be around. It's funny because I started computer science. And in retrospect, I mean at the time, you could have been kind of a software engineer or something instead you became a podcaster.

11:19It's kind of a new responsible career move. But in retrospect, it's like, it's great. I kind of worked out. Just as these guys are getting automated. I get asked this question all the time. Yes, okay. And one answer that I like to give is that you should think about the next couple of years as increasing your individual leverage by a huge factor every year. So already, software engineers will come up and say, I'm two times faster or new languages and five times faster than I was last year. I expect that trend line to continue. Basically, as you go from this model of what I'm working with some model that's assisting me on my computer.

11:53And it's basically a pairing session to I'm managing a small team through to I'm managing a division or a company basically that is targeting a task. And so I think that deep technical knowledge in fields will still matter in four years. Like it absolutely will because you will be in the position of managing dozens or like you sort of your individual management bandwidth will be maxed out by trying to manage like teams of AIs. Yeah. And maybe AIs, maybe we end up like a truly singularity world where you have AIs, managing AIs and this kind of stuff. but I think in a very wide part of the possibility spectrum, you are managing enormous, vastly more resources than an individual could command today.

12:37And you should be able to solve so many more things with that. That's right. And I think I would emphasize that this is not just Coke. It genuinely is the case of these models, lack the kind of long -term coherence, which is absolutely necessary for making a successful company. You're just getting a fucking office is kind of complicated. right? Yeah. It's like you had to deal with all these. So you can just imagine that for a sector after such like the economy is really big and really complex. Exactly. And so especially if it's, I mean, I don't know the details, but I assume if it's like a data sparse thing where you kind of just like got to know what's actually what is the context of what's happening in the sector or something, I feel like you're being in a position.

13:15Maybe the other thought I have is that it's it's really hard to like plan your career in general. And I don't know what advice I'd imply because I remember being super frustrated. I mean, I was in college and the reason I was doing the podcast was to figure out what it is I wanted to do. It wasn't the podcast itself. It would go on like 80 ,000 hours or whatever career advice. And it's just like in retrospect, it was all kind of mostly useless and just like, just try doing things. I mean, especially with the AI, we just like don't know what it's going to, like it's so hard to forecast what kind of transformations there will be.

13:47So try things, do things. I mean, it's such a banal vague advice, but I am quite skeptical of career advice in general. Well, maybe the like piece of career advice on an obstacle is put yourself close to the frontier because you have a much better vantage point. That's right, right? Like you can study deep technical things whether it's computer science or biology or like and get to the point where you can see what the issues are. Because it's actually like remarkably obvious at the frontier what the prototype is. It's very difficult to see. But actually do you think there is an opportunity? Because one of the people, One of the things people bring up is maybe the people who are advancing their career and have all this tacit knowledge will be in a position to be accelerated by AI, but you guys four years ago or two years ago when you were getting discovered or something, that kind of thing where you have a GitHub open issue and you try to solve it, is that just like that's done and so the onboarding is much harder.

14:38I would still walk what we look for in hiring. So, you know, yeah. I'm in favor of the learn fundamentals, gain useful mental models. But it feels like everything should be done in an AI native way, or like top down instead of your bottom up learning. So first of all learn things more efficiently by using the AI models and then just know where their capabilities are and aren't. And I would be worried and skeptical about any subject which prioritizes wrote memorization of lots of facts or information, instead of ways of thinking. But if you're always using the AI tools to help you then you'll naturally just have a good sense for the things that it is and isn't good at.

15:17That's right. Yeah. Next one. What is your strategy, method, or criteria for choosing guests? The most important thing is do I want to spend one to two weeks reading every single thing you have ever written, every single interview you have ever recorded, talking to a bunch of other people about your research? Because I get asked by people who are like quite influential often to be like, would you on your podcast. And more often than not, I say no, for two reasons. One is just like, okay, you're influential or something. It's not fundamentally that interesting as an interview prospect. Not from like, I don't think about the hour that I'll spend with you.

15:56I think about like the two weeks, because this is my life, right? The research is my life. And I wanna have fun while doing it. So just like, is this gonna be an interesting two weeks to spend? Is it gonna help me with my future research or something? And the other is, big guests don't really matter that much in the, in like, if you just look at one of the most popular episodes or what in the long run helps the podcast grow. By far, my most popular guest is Sarah Payne. And she before I interviewed her was just like a scholar who was not publicly well known at all. And I just found her books quite interesting.

16:29Same goes with, so my most popular guest are Sarah Payne and then Sarah Payne and Sarah Payne and Sarah Payne and Sarah Payne and Sarah Payne. Um, uh, and by the way, from a viewer minute adjusted basis, I host a CRP and podcast, right? Occasionally. And that is David Reich, who is a geneticist of ancient DNA. I mean, he's like somewhat well known, but he had a best -selling book, but he's not like, he's not Saudi and Adela, or Mark Zuckerberg, who are the next people on the list. And then again, I think at like pretty soon, it's like you guys or Leopold or something, and then you get to the lab CEO's or something.

17:02So big names just don't matter that much and like for what I'm actually trying to do and so just like well And it's also really hard to predict who's gonna be the David Reiker Sarah Payne so just like have fun who talked to whoever you want to spend time researching and it's a pretty good Proxy for what will actually be popular. What was the specific moment if there was one that you realized you the producing your podcast was a viable long -term strategy I think when I was shopping around add spots for Mark Zuckerberg episode. And which were, you know, like, now that I look back on it, it's like, not in retrospect that mind -blowing, but at the time, I'm like, oh, I could actually hire an editor.

17:42A full time, or maybe more editors than one. And from there, like, turning into a real business, that's when I, because before, I just like didn't, people would tell me, like, oh, these other podcasts are making whatever, whatever amount of money, and I'd be like, how? You know? I have this running joke with one of my friends that, I don't know if you've seen me do this, but every time I encounter a young person who's like, what should I do with my life? I'm like, you gotta sort of blog. You gotta be the Matt Levina VI. You can do this. That's like an entirely empty niche. You could, like, you can, and I was running joke with them where they're like, you're like a country bumpkin who's like won the lottery.

18:17And you're like, you know what, everything was like, guys, this crash pack. Get this crash pack. I'm not. I do want to press on that a bit more because your immediate answer to the 17 year old was to start a podcast. Yeah. So what niches are there, what sort of things would you be excited to see like new blog's podcasts? I mean, I wonder if you guys think this too, but I think this like Matt Levino of AI. Absolutely. It's like a totally open niches of our fries I can tell. And I apologize to those who are trying to fill it in. I was like, oh, I know where have at least one book that's trying to do this.

18:51But the other thing I'd really emphasize is it is really hard to do this based on other people's advice or to say like here's a niche I'm trying not to feel like a specific niche. And if you think about any sort of six -off -swole new media thing out there, there's it has two things which are true. It's like often not just geared towards one particular topic or interest and two it's the most important thing is that it is propelled by a single person's vision. It's not like a collective or whatever. And so I would just really emphasize, sorry, the thing I really want to emphasize is it can be done to, you can make a lot of money at it, which is not the most important thing, probably for the kind of person who would succeed at it, but still is just worth knowing that it's a viable career.

19:39Three, that, yeah, that basically you're gonna feel like shit in the beginning where it's like, all your early stuff is gonna kind of suck. Maybe some of it will get appreciated, but it seems like that advice to say still stick through it in case you actually are terrible, because some people are terrible. But in case you are not, just do it, right? What is the three months of logging on the side of really going to cost you? And people just don't actually seriously do the thing for long enough to actually get evidence, or get the sort of RL feedback on like, oh, this is how you do it. This is how you frame an argument.

20:13This is how you make a compelling thing that people will want to read or watch. blogging is definitely underrated. I think most of us have probably - So you both had blogs which are relevant. I don't know if they're actually relevant to getting with that. They were like somewhat relevant. But I think more so that we have all read, almost all the blogs that do it in depth, like trees is on AI. That's right. Like if you write something that is high quality, it is almost invariably going to you, like share it around Twitter and read. Oh, this is so underappreciated. Yeah. So two pieces of evidence.

20:43I was talking to a very famous blogger you would know, and I was asking him how often do you discover a new, like, undiscovered blogger? And he was like, man, it happens very, very early, like maybe once a year. And they ask him, how long after you discover him or her is the rest of the world discover them? And he's like, maybe a week. Which, and what it suggests is like, it's actually really efficient. Oh, so I have some more takes. Oh, yeah. This is the M .A. So I believe that slow compounding growth in media is kind of fake. Like the appolition or an S. It's not like it was really given an audience for a long time for years or something.

21:25It was really good. I just agree or agree with it. And if it's good enough, it literally everybody who matters. And I mean that literally will read it. I mean, I think it's like hard to say zero shots, something like that. but the fundamental thing to emphasize is the compounding growth, at least for me, has been, I feel like I've gotten better. It's not so much that somehow the three years of having 1000 followers or somehow like a compounding, you know, I don't think it was that. I think it was just like, it took a while to get better. Yeah, suddenly when Leopold posted that, like the next day, it's almost like you can feature like being so almost sta - well, not it wasn't, but it was stapled to walls to speak on Twitter.

22:02That's right. Everyone was talking about it. You read any event And for the following week, every single person in the entire city was talking about that essay. Yeah. Like Renaissance Florence. That's right. That's right. The world is small. Yeah. What would you say was your first big success? I'm trying to think back to when I first found your podcast. I distinctly remember you had your blog posts on the Anna Smarabelos and Jeff Bezos retweeted it, I think. Yeah. I'm trying to remember if it was even before that or not, but yeah, what I'm curious you're like, that was it. Okay. Yeah. I mean, it wasn't something where I'm like, it was some big inside that deserve to blow up like that.

22:38It was just taking some shots on goal. They were all like whatever inside porony and then one of them, I guess caught the right guys' attention. And yeah, but I think that was it. That's something else, which I would appreciate. Which is a piece of writing doesn't need to have a fundamental immune side. So much as give people a way to express cleanly a set of ideas that they already are like, aware of in a sort of broad way. And if it's really crisp and not like articulate, then even still that's very valuable. And sorry, the one thing I should emphasize, which I think is maybe the most important thing to the feedback loop, it's not the compounding growth of the audience, I don't even think it's the compounding, like it's me getting more shots on goal in terms of doing the podcast.

23:18I actually don't think you improved that much by just doing the same thing again and again. If there's like no rewards like knowledge, you'll keep doing whatever you were doing before. I genuinely think the most important thing has been, The podcast is good enough that it merits me getting to meet people like you guys. Then I become friends with people like you. You guys teach me stuff. I produce more good podcasts. So hopefully slightly better. That helps me meet people in other fields. They teach me more things. Like with the China thing recently, I wrote this like blog post about a couple stories about things that happen to China.

23:49And that alone has like netted me an amazing China network in the matter of like one blog post, right? And so hopefully if I do it to episode on China, it will be better as a result. And hopefully that happens to a garage field, that we're field. And so just getting to meet people like you is actually the main sort of flywheel. Interesting. So move to San Francisco. Yes. If you're trying to do AI, yeah. Next questions. Shall we do? Can we do a very important question? From Jack Paget. How much can you bench? Sure. You can't lie because we both don't need to answer. At one point, I did bench 2 .25 for four.