Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil

0:00Jerry Tworek:If I play football, for example, it looks very, very closely to reinforcement learning. I kick a ball a lot of times and every time I adjust it a little bit and I see if it roughly matches what I wanted and there's some self-reinforcement happening. When I learn mathematics, it's a very different type of thing. It's like reading about hard concepts and thinking about them very deeply inside my head until things click and until I have them connected. And both of those in some way are learning from experience. they are just very different. We probably are spending the most compute than ever on learning from experience but reinforcement learning is not the end of learning from experience and there will be better approaches that researchers will be coming up in the coming years on how to use that data.

1:00Rohan Anil:Jerry, Rohan, thank you so much for joining us today. The two of you are the founders of Core Automation, one of the hottest neolabs in San Francisco right now. And before starting Core Automation, you led some of the most important research projects of the AI era. Jerry, you were VP of OpenAI, where you worked, amongst other things, on running the Strawberry and Reasoning teams. And Rohan, you were one of two of the four pre-training leads at Gemini. and before that led a lot of the fundamental AI research at Google Brain and were the fix-it guy across Google and then at Anthropic. And so between the two of you, you've seen more than your fair share of what the world looks like in terms of doing frontier research.

1:42Rohan Anil:And so I'm very, very excited to dig in. Let's start with you, Jerry. You tweeted a very spicy take recently. The first step to replacing Transformers is appreciating deeply how far they were able to carry us. Is that a eulogy for the Transformer? What does that mean?

1:58Jerry Tworek:Thank you very much for inviting us here, Sonia. I feel like a lot of my interviews these days is explaining my tweets and what did I mean. But appreciating Transformer means understanding what it does well. So you're not solving the problems that it is solving well. You have to focus on its weaknesses. You have to understand good parts and bad parts. And it's very easy in a lot of the work, what people are doing in architectures is trying to make transformers cheaper and trying to make transformer more efficient. I very rarely see people thinking about how do we make transformers more powerful, trying to do more expressive.

2:36Jerry Tworek:But seeing someone weak parts and seeing someone strong parts are almost the same thing. It's just understanding the shape of transformer a little bit more. Well, I think right now we are in this stage. We got really, really good at training really, really big models. We mastered two algorithms. We mastered pre-training at a large scale, and we mastered reinforcement learning at a large scale. And I'm asking myself a lot, what is next in machine learning? And I think at this moment, what the bottleneck is to better models and to smarter systems is the architecture itself. It is this moment to revisit the train we've been riding for the last six years of trying to add more and more parameters to essentially two of the same operations, which is MOE and attention.

3:25Jerry Tworek:And when I'm thinking about it, like where we are today and what we are doing, I am thinking a lot about what codecs and what cloud code are doing for us. And I'm really, really appreciative of those systems and of the coding and of the workflow automation and of the systems of the products that we have today that we essentially have built over those six years of scaling. And I think this is the first step of thinking, what is the, if we want to work on replacement, we need to see where we are, what problems we have solved, to start seeing what the next stage is, what problems we haven't solved yet.

4:02Jerry Tworek:What kind of are we missing? And this is kind of whenever I use codecs and I am successful at a task, I also start thinking why didn't I try to push that thing harder whenever whenever I come to work there are a lot of things I do with codex but I still come to work I still ask it to do certain things for me and I'm always asking myself why am I even needed there why is core automation is name and its concept is we want to we want to be automating tasks and why why are those things not yet automated why why is not codex doing everything for me and this is the question of like the research where we want to go and with that research i'm trying to think what kind of models what kind of what kind of systems do we need what kind of qualities do we need what that that's we don't we don't have today and that's that's what i'm thinking a lot these days and you have this starting premise of the architecture is the issue which i think is a contrarian point of view so what what led you to that point of view what did you see that made you think the architecture was the issue it's fundamentally what is the issue it's It comes back from the previous implication.

5:11Jerry Tworek:What I think is the issue is that the models are being trained in the lab and are being deployed in the real world. That is the fundamental tension that is there. And a bit of my disappointment comes from my personal story. Whenever we were starting the research and progress on scaling up reinforcement learning at OpenAI, I basically believe that scaling up reinforcement learning is a necessary stepping stone on a path to AGI since I started working at OpenAI. And I was always reinforcement learning maximalist. I was always believed this is what we need to focus on. This is what we need to do.

5:56Jerry Tworek:I've seen LLMs being scaled up to higher and higher levels through GPT-3 to GPT-4. And we're still doing very little RL. And I had this internal belief that the moment we start scaling up RL, we'll solve everything. We'll be able to solve all the problems. And we eventually started scaling up RL. I was just in there. I was in the center of it. I was thinking, here we are. If you ask Jerry in 2024, when do we get AGI? I would say 2025 will be that year. This is where we solve everything. And I saw us training model after model. This model was getting better and better. All the benchmark scores were going up.

6:39Jerry Tworek:And did we also solve all the real world tasks at that moment? Unfortunately, unfortunately not. We still have work. And I realized there was this bit of distinction as all the benchmarks that we are evaluating our models, they were essentially the same thing as we were training the models. on. Like all the evals and training tasks are the same sides of the coin. But the real world distribution and real world task is much messier, much murkier, much more different. Our training data didn't really replicate the real world use cases and despite us basically maximizing all the tasks, if you see ask anyone training models, hey, what is one of your main issues?

7:23Jerry Tworek:I don't have hard enough tasks. I don't have what to train our model on. Yet we are still not covering the entirety of the real world distribution. From that, my conclusion is we need to have models that learn at test time. We need to have models that learn with users on their data, on their real world task, on the real world distribution. And there, when you are asked, why don't we have that today? Why are transformers not learning anywhere? And there are essentially two types of learning that we could be doing at test time. We could be doing and context learning essentially of transformers, which is, it doesn't have fundamental problems of catastrophic forgetting.

8:07Jerry Tworek:It doesn't have that issue. It is pretty data efficient. So that is great, but it's not very scalable. We only can have so much of it. It is limited and has some more, even more of mechanical limitations of what actually are you doing when you, when you build context, but maybe, maybe we can, we can come back to it later. but we have in-context learning, which is very limited and very small amount of data. Whenever I'm using codecs, roughly around 20 minutes of usage, I need to compact it and move it afterwards, which is not that much data. If all we can learn for 20 minutes, it's not that much.

8:44Jerry Tworek:And the second thing is fine-tuning. We could try to continuously fine-tune our models, but then those have the issues of catastrophic forgetting. We have issues of very low data efficiency and neither those are very solvable. Neither of those are very easy to find ways people have been trying. If those were easy to solve, some would already solve it. So my personal belief is we need to find an algorithm that we can meta-learn, that we can express on the architectural layer that can represent how How does learning look like? How does learning look like that can work on much, much longer horizons?

9:24Rohan Anil:Do you expect the architecture will look Transformer-like? Because my understanding from the chief seats is that, you know, OpenAI had been trying to scale up reinforcement learning for a long time. And it wasn't until the Transformer came about that it seemed like there was an even kind of scalable prior on the world upon which to even scale RL. And so how do you even go about trying to think about scaling up this new regime?

9:50Jerry Tworek:Yeah, that's a great question. I think those two things happened at the same time. But if anything, that happened there was mostly about economics, because technically you can scale up LSTMs. Just no one really, really dared to go in that direction. And they did scale much, much more poorly. they are they're they're scaling in a scaling loss paper there is uh presented a comparison of lstms and transformers and fundamentally the scaling love of transformers was better there is a world where we never invented transformers and we would be scaling lstms and we'll be having some models but because they would be much more expensive to train and much less impressive as a product we would have would have just a worse experience and maybe no one would be able to convince people to spend as many dollars training those gigantic LSTMs because we wouldn't get a market return.

10:43Jerry Tworek:The majestic thing about Transformer, which goes back to like, why do we have to appreciate Transformers so deeply, is that Transformers are economically valuable. Training them, the cost of training them is lower than the revenue that they generate, which is just magic of machine learning. It's not guaranteed by itself, but for LSTMs, it probably wouldn't be that way, which made it happen. But in many ways, you can scale most of the architectures. I think a lot of reasons why people didn't scale things before was because researchers before OpenAI had a lot of reluctance to scaling. It was often seen as unscientific.

11:23Jerry Tworek:And research in algorithms was providing how do we become more and more efficient? How do we, for the same compute budget, get better results? And it was a bit of a contrarian bet by OpenAI. at that moment to try to say, hey, we don't care about better and better algorithms. We care about more and more scalable algorithms. And how do we pour more and more compute and get better results, which OpenAI was criticized repeatedly by many people in the community for a long time. But thanks to that, we have the models that we have today. And I think there are tons of architectures that can be scaled up.

11:57Jerry Tworek:And I am part of the core automation's mission and our belief is that a lot of architectural research happened at too small scale for too long time. A lot of people are trying to say, hey, let's try to first try our architecture on a small data set in a small compute regime. And then see where it scales only after you prove itself. But for example, when you do work on reinforcement learning, you know that to get to any interesting results, you need a certain level of compute to even see the capabilities in the model. Reinforcement learning needs a baseline of ability to only start working. So where I am coming from, probably there are many architectures that need a baseline of compute to even start doing anything, anything interesting, anything useful.

12:47Jerry Tworek:Can I ask you then maybe a touchy question? Please do. If you need a baseline of compute, that sounds like a job that would be well served inside of a big research lab. Why start a company to go do this? It's a great question. And I think in many ways, it's likely a timing thing, timing issue. Market is right now in a very specific place where the biggest and the most successful labs, by coincidence or by fate, are probably in the most competitive market fight ever right now, which makes them not very keen on trying different paths, trying alternatives. if transformer is profitable and if you can spend more efforts and more resources scaling transformer to win in the next quarter it's very hard to put at least a lot of attention and a lot of energy to work on something that will that will maybe better or maybe or maybe will will redefine the field in a year or two so so i think the biggest labs and i talked to basically all of them don't don't have that much interest in trying the alternatives to Transformer.

13:57Jerry Tworek:And the labs that are not the biggest are doing whatever they can to do what the most successful labs are doing. And everyone is trying to train the same coding agent. If you look at the last week's releases, everyone is trying to release a coding agent right now. And I think we need different paths and different approaches here. So that's initially the ecosystem we are trying to fill in.

14:24Rohan Anil:And Rohan, you were at Brain when the transformer was invented. Do you agree with Jerry's eulogy for the transformer? Yes. In some sense, like, first when the transformers, Ashish, Noam, and others came up with it, I had, like, worked on my work on online distillation around the same time. We presented it at the same internal research conference. It wasn't a big deal internally. There was only a few people who actually got it. A lot of people were like, oh, it's like, yeah, it's another work. And people were finding ways to. And it was also very focused on, at least the original work was very focused on a real problem, which is translation.

15:02Rohan Anil:So they saw like the Beatles TM on translation. And it took opening. I mean, internally at Google, there was definitely like Noam and a few others were definitely interested in scaling language models. I think it is until GPT-2 and GPT-3 that we saw the benefit of transformers working quite well. At least the way I think about architecture is how do we spend computation? And Transformer is one very efficient way to spend computation. But now that I look at the industry, a lot of our computation is inference time and spending it on tokens. Let me ask this question. If I want to optimize for a better architecture, I want to look at both pre-training and RL together.

15:39Rohan Anil:And I would like to find architectures that spend computation much better than current chain of thought token generation. in at a like to give a much better overview it's i think of like pre-training has built the transformer with certain context length and rl comes in and it's like well that's not sufficient i need more computation let me do it via adding one token at a time this is quite inefficient from like inference perspective you're doing one token at a time so most of the solutions have been finding to do better ways of speculative decoding so it's like a band-aid to a problem that we've picked something that can only generate one token at a time, so autoregressive recording.

16:21Rohan Anil:There is problems with the transformer in terms of how do we spend the computation for the longest time. I think most of the world was training very large, dense models, and it took the industry roughly two to three years to get to refine the architecture to what we now take for granted. It was not obvious to a lot of people, sparsity and mixtures of experts and getting good training efficiencies with them, right? So then you can ask like, what's wrong with the transformer? Well, it's if the computational depth is poor, how do we increase computational depth? And just posing that question opens up like 20 new directions on how we can modify the mechanism to incorporate it.

17:03Rohan Anil:So I see like to do work like this, it takes time. and usually like fundamental research in the past have taken like five, six years to land into industry and it's largely from organizational, knowing that it is important. This is the bet. Like just like Jerry had the inner belief that RL is needed. Absolutely did not have that belief at Google. I was a pre-training maximalist, pre-trainer, biggest model. That's why you guys are a good fit. Right, like things are. And then, so that inner belief. And second is you need your architecture to run efficiently on hardware. A theoretically optimal architecture is not useful to anyone.

17:43Rohan Anil:It is something when it comes into practice. So you need the research inception to getting it productionized and getting kernels and everything written, the end-to-end loop. And there's only a few places right now which have integrated teams doing that. And I think we have built a team in a way that puts the experts together, not in different silos that we are accelerating on having everybody look at the problem holistically from end to end. So that's where I'm quite bullish. That's why I'm here. The current mechanisms are quite poor. And if you leave it to the world, I am afraid that it will take us a much longer time horizon before we replace the transformer.

18:26Rohan Anil:and I think a lot of folks are already complaining a lot on token costs. And that seems like as someone... We're not complaining. I mean, in terms of like, yeah, exactly. I come from the Google mindset where we had to like serve billions of people. So like finding more efficient architectures that have like a deadline on latency and the number of tokens that you can serve. So like when I look at that, like the amount of like the world that can use like frontier tech is very little. and someone or some group has to accelerate and make this better. And we are taking that shot at doing that. The current technology just scaled up is still only relevant to a subset of humans.

19:09Rohan Anil:And this is the bet we're making.

19:11Jerry Tworek:So one of the things I heard you say was the problem with transformers is the computational depth is poor. If that's the crux of the issue, tell us what does that mean? Why is that the case? How do you fix it?

19:23Rohan Anil:I can give you like one insight, like most transformers that we train are quite shallow. That's at most like 100 layers deep. Depth is like, it's called deep learning because you wanted deeper representations. There has been experiments on going into depth, but no one has actually shown us learning extremely deep representations. Chain of thought reasoning and RL to do chain of thought by model itself is one way to increase computational depth because every token you add, you add like one more pathway. And there has been, so then you can get out of like this bottleneck that the pre-trained architecture has set you up on.

20:03Rohan Anil:You can only do a number of layers times sequence length. Now you can increase the sequence length and you get much stronger results. You can do inference time scaling. Now the issue with inference time scaling is that models now have to produce more tokens to get better results and that's very one token at a time. And from this, you can see like you can directly address many of these things. And this is like a subset of work that we are looking at, right? Making this much more efficient.

20:32Jerry Tworek:What's your forecast for the transformer-based architecture? If it's not the end state, how far can it get us? When do we start to see it topping up? I think it all comes back to what we are training transformers for and what we can do with them. we're doing pre-training which is very good at distilling all the knowledge from the internet into transformers and then we can aral them through which is we basically can bake all the workflows that we want into a transformer so what what what transformer is like capped out is we have all the knowledge of humanity in the model together the relationships and how do they how do they work together how they can be combined and basically any task that we have training data for we can put into the model and this could be a gigantic model trained with a lot of compute on all the data in the world.

21:26Jerry Tworek:And then if we ever stop training that model, what would happen? The question worth asking often I'm thinking about transformer, what would happen if OpenAI and Anthropic stopped training new models and we got the transformer we have today and say this is it. This is the best model we have. Months pass, year your spas and the model is getting less and less useful. Maybe the lab really like recorded of every human on earth what they were doing and what their tasks were and their environments and put them in the model, put them in a learning environment. But then what happens if anything of that changes?

22:06Jerry Tworek:If there are new events in the world, if those new events have new relationships between them, if there are new types of tasks, if there are new code bases, new tools to use, transformers are getting a lot of their usefulness and value through the things that are valuable have to be present in training. And when they are not, they suffer. There's some ability to adapt, but it's not very big and not very flexible. So in my mind, this is kind of the level where the transformers top, which in many ways what I think is a tool to use for us. If we kind of know, if there's a human who knows the limitations of a transformer, they can schedule that model, they can write a prompt of what is the task that you want and by doing the training we are doing, you can get very successful in that and any task the model fails, you can add it to the training data and the model can succeed but that loop has to go through the lab training the model for you.

23:10Jerry Tworek:And if the model that fundamentally needs to be trained in the lab, how much do you think of it that this is the goal or you would want to be able to update the model somehow, not having to go back there?

23:22Rohan Anil:Have you read the Rich Sutton and David Silver have this paper, The Age of Experience? Have you read it? I'm curious how much you agree or if you have any different opinions, whether your opinions diverge.

23:32Jerry Tworek:Reinforcement learning is not a particularly new approach, particularly new thing to do. So in some way, age of experience, I think always has been there. And people have been criticizing a bit pre-training because pre-training very clearly is the other way of looking at the models, which is like we have static data, that data is mostly generated by others. Although I have this personal view that pre-training today is largely distilling other models, all their malls into the new mall because most of the tokens in the internet are coming from AI. But there's clearly pre-training, which is behavioral cloning, which is mimicry, which is compression of internet data.

24:17Jerry Tworek:But reinforcement learning is not something that people haven't been thinking and people haven't been doing. Reinforcement learning was used to solve by Gammon back in the day, that it used to solve Go, StarCraft, Dota, to solving programming right now and every time it comes down to model writing its own experience and learning from that experience but it is very clear and what I think is interesting and what I think is still perplexing to people that reinforcement learning is not really the only way to learn from experience and there will be more and there will be a little bit more of I think you can call it algorithmic but essentially innovation of how we learn from experience just because just because reinforcement learning is only one way to do it a mathematical formulation and especially right now how we are using it it really likes those parallel rollouts for variance reduction and for uh for comparing how the model does in parallel versions of the world which is not not how we do not how we learn from experience we learn from our experience much more efficiently and much more um we use we use those in many in many, many ways.

25:30Jerry Tworek:At some moment, I've been trying to explain to people that what brain does, how we learn. It's not that there's one learning algorithm in a brain. I think there are multiple, actually, and they work together. But if I play football, for example, it looks very, very closely to reinforcement learning. I kick a ball a lot of times, and every time I adjust it a little bit, and I see if it roughly matches what I wanted, and there's some self-reinforcement happening. When I learn mathematics, it's a very different type of thinking. It's like reading about hard concepts and thinking about them very deeply inside my head until things click and until I have them connected.

26:11Jerry Tworek:And both of those in some way are learning from experience. They are just very different. So summarizing my thinking of the learning from experience is that we've been doing it for a while. We probably are spending the most compute than ever on learning from experience. but reinforcement learning is not the end of learning from experience. And there will be better approaches that researchers will be coming up in the coming years on how to use that data in every nature settings.