How GPT-5 Thinks — OpenAI VP of Research Jerry Tworek

0:00O1, to be perfectly honest, it was really mostly good at solving puzzles. It was almost more like a technology demonstration. O3 has really been something like a tectonic shift, the trajectory of AI. GPT-5 in some way can be considered as like O3.1, which I am after as something that would be the next pretty significant jump. We understand that there's only one time in history where AI is being built and deployed and developed. We are together in this goal that's larger than every one of us. Hi, I'm Matt Turk from FirstMark. Welcome to the Matt Podcast. Today, my guest is Jared Forek, VP of Research at OpenAI and a member of the Metis list of the world's top AI researchers.

0:43In this episode, we go deep on how models actually reason. We also go behind the scenes at OpenAI, how a few big bets get staffed, why everyone knows everything, and how that culture ships fast. Please enjoy this great conversation with Jerry. Hey, Jerry, welcome. Hello, very happy to be here. We are going to talk about reasoning a lot in this conversation. At a high level, what does reasoning actually mean? When we talk to ChatGPT and ChatGPT says it's thinking, What actually is happening behind the scenes? I think that the thinking process is at least a good analogy. As we were in the early days of AI, always had this goal, dream of trying to teach models to reason.

1:29We were thinking about it, spending more time to get better results. If like human is posed with a very hard problem in front of them, very rarely they have answers straight away. Sometimes they need to find that answer. sometimes they need to perform certain computation sometimes they need to like look up some information sometimes they need to teach themselves something and the process is reasoning is like getting to an answer that you don't yet know like in some way it can be called search but it's not really like a very naive search search is a loaded word but reasoning is the process of getting to an answer and a work that you need to do that is like longer than what usually is considered answering a question i think that that difference is here like answering a question usually means you already know the answer and you just just just elicit the answer you know and the process of reasoning is getting to the answer that you don't know and usually the longer you spend on getting to this answer for whatever you need to do to get there the better the better it gets and uh we've all become familiar since you guys uh released uh one i guess a little over a year ago and in september 2024 with the concept of chain of thought, which is in layman's term, the little messages that you see when you query ChatGPT and it tells you, it shows its work, it tells you what it does.

2:52What does that actually do? Is that a logical tree and it eliminates option after option? What actually happens? Language models do on their own fundamental level is they are often called as next token prediction machines. And that's not completely accurate in the age of reinforcement learning, but they still operate on mostly on tokens that are mostly text. The language models, again, are these days also multimodal and they operate on mostly text. But to simplify a little bit for a second, language models generate text. And what chain of thought is, is their thinking process verbalized using human words and human concepts.

3:36so the the magic that we are seeing why this is all possible is that while you are training on all of internet on a lot of human knowledge and human thinking process the model starts learning in some ways to think how humans do and in some ways get to the answers how humans do from seeing humans do it a lot in the text that was that was pre-generated and that that was based in a training data on humans. And then the chain of thought is basically eliciting that capability in language models of thinking and getting to an answer like humans do. A lot of what early chain of thought work was doing was kind of solving math puzzles.

4:24And the first most famous prompt to elicit chain of thought in language model was so-called, let's solve it step by step. There is this very classical result in language models that if you ask them, what is some mathematical expression or some puzzle, they will try to give you an answer. They will try to predict the next token, but they fail. It's a hard thing. They can't compute it in one token job. But if you ask them, please do it step by step, they will start thinking, okay, I don't know the answer, but the first step of getting to the answer is this. And then they write like chain of thought, which is a series of text, series of tokens doing the first part of the computation, the second part of the computation, the last part of the computation.

5:06Then they connect those things and then they can get to the answer. So the chain of thought is basically a process of thinking encoded in words, how humans would solve a problem on a piece of paper going step by step from start to the end. And since time and by that, I mean the time spent thinking is so important to that concept of reasoning. How does a model decide how long to think when we're in chat GPT-5 and we're in auto mode and it says that it's going to decide automatically how long to think? What happens there? It's basically part of our optimization process, partially for the happiness of the users and what they want to expect.

5:56Because when you have a thinking process, you need to balance two things, which is the quality of the result. As we said, there have been those pretty great scaling laws that we demonstrated with the release of O1. The longer the model thinks, the better result you get. But also, people don't like waiting. Waiting is time lost that you could do something. Everyone wants to get results as quickly as possible. And there is this saying you can get cheap, fast, or good, and you can take two. And that applies to language models as well. There is a trade-off, and it's delicate. That's why we also expose some of that trade-off to the users, where you can have a high-rezoning model and a low-rezoning model.

6:41And this is, in the end, the same model. We just tweak the parameter, which says we want you to think longer or shorter. We try to encode some heuristics of what we think the users will want when thinking on an answer a little bit longer and getting to a better answer is worth it, waiting, versus not. But it's a bit of a trying to guess the anticipation of the users. What's the right amount of thinking for them in this particular situation? Fascinating. So it's more user-driven. So it's more like a user experience kind of thing. In the end, it is, because the question is like, how long do you want to wait for answer?

7:18You can always wait longer and get an even better answer. It's been a little over a year since the release of the world's first reasoning model, O1, which is an effort that you led. What has been the journey since? So there was 01, then there was 03, then most recently HGP5. How would you characterize the evolution of reasoning specifically across the three models in the last year? In some way, how I characterize our reasoning or scaling up reinforcement learning research program is we do a series of scale up runs that are progressively more and more ambitious. Everyone, we try to do something more, something larger scale, something that should result in a better trained model than the last one.

8:11And obviously, we don't release all the models that we train. some we release, some we think need to wait a little bit longer for the moment where they will have their time to shine in the hands of the users, but 01 was the first model we decided to release to demonstrate to the world those models, and 01 to be perfectly honest, it was really mostly good at solving puzzles and maybe a few kind of thinking problems here and there, but it wasn't yet very useful model. It was almost more like a technology demonstration than actually really polished product. But we were thinking we have something cool and we wanted to share it with the world as OpenAI.

8:5703, I think, changed that pretty significantly. In some way, it is a model that is meaningfully useful and a little bit self-serving, but it was a moment when I started using ChatGPT quite a bit and And I am basically a user completely hooked on reasoning model. And in chat GPT right now, I use basically exclusively reasoning models because those are the only models that I trust. The output and the result and I think O3, its ability of using tools and getting to answer, leveraging a lot of contextual information from various sources and persevering towards getting to that. has really been something I think there was a little bit of a tectonic shift in the trajectory of AI and I think we did something really really great there GPT-5 in some way can be considered as like 0.3.1 it's a little bit of iteration of the same thing and the same concept and what I am after and my team right now is something I think next that would be the next pretty significant jump of how we interact with models that are even more capable, thinking even longer and interact with even more systems and sources of information on their own journey.

10:27but but like separately in the meantime we continue to build a lot of things on top of all free technology like codex which i think like coding agents are at the moment the first like really successful agentic products built on top of ai there are things like computer using agent it's called gpt agent right now i think and like the pre-search and a few a few other things that that we will we'll keep on building on like all free generation technology great all right so we're going to go into all of this in much greater detail in a minute. But before we do that, let's talk about your journey. I think it's a super fascinating topic for all of us.

11:09You guys are changing the world. So I think I'm curious and we're all curious, I think, about the people, the human aspect of who those people are that are just having such an impact. So you, starting from the beginning, you grew up in Poland, I believe, right? Yes, I grew up in Poland. Walk us through your formative years and how you got to get started in this field. Yeah, yeah, yeah. Happy to do that. An interesting fact, it's almost like a crystal starts from something and you put a little bit something in the beginning. there's i think one part that was like important and part like like the starting point of my journey where i i didn't know where it came from because it was with there with me from the very beginning of my life in a moment that i don't really know when it started it just was always there with me i always thought that being a scientist and doing science is the highest calling a human can have and i don't don't really know where it came from like you know my parents maybe like were singing the light bright lullabies to me like when i was when i was one or something like that but basically since i remember i wanted to be a scientist in the early years i also discovered like you know i have i have talent for for those things i like i was going to school and i saw i get things slightly faster than people around me at least like you know in a regular school and in the middle of poland and which which like you know made me made me kind of like doing those things like studying maths and science a little bit more because because like you know I felt it felt good in a way it felt like this is this is something that naturally naturally fit me and like you know I grew up as a like you know very regular kid just just like you know being slightly nerdy guy and trying to balance my my side of being interested in science programming maths and like having some social life and i definitely have had some kind of like a party arc in my in my life but i think i think that the most important part and moment was when i actually went to university college university of warsaw is where i went and i decided again to study mathematics at around that time of being 18 my idea of life was to be a mathematician with a pencil sitting in a room with a piece of paper and solving equations this is kind of my 18 year old dream of how how life should be lived and what i what i want to do in in my life and like you know i i do have like you know my my personality is built again in a way of really appreciating like you know solid science pursuit of truth great great engineering and all the all those aspects but I definitely have also a little bit of like a misfit kind of rebellious tinge to it and that resulted after a few years of studying mathematics I what I realized about myself and about the world is that I really like maths and I am I'm quite good at it but I didn't like academia that much and I realized I don't want to like stay in academia I don't want to stay in university and that this would not be environment where I thought I would be long term very happy and fitting.

14:31It felt a little bit too rigid, a little bit too structured in a way that I didn't know if I will feel good. And in some way for a young me, I was around 21 years old at that moment, that was pretty big crisis of faith for me. I had a moment like, you know, lost purpose of life. So I just did like, you know, a very simple, like first principles thinking, you know, I am graduating with a degree in mathematics. I need to get a job to get food. And like, you know, what job can I do to use mathematics and in that job? And looking at the job market, that moment was 2011, I think, or 2010, somewhere around that.

15:22I decided to become a trader and trade for a living as the one way where I can do what I like, which is mathematics, and get a career. year i got i got a quick internship at jp morgan uh investment bank trading floor and equity derivatives group spent six months there learning a little bit how how does trading work and what does what does what does it look like a little bit i finished my degree uh i got i got a message from like a boss of my boss at jp morgan something saying hey jerry you were like you know one of our best interns ever that we had we really really liked you working with us and we are we are living bank and starting a new hedge fund would you want to come with us and like for for 20 21 or 22 year old jerry that sounded like kind of like a cool adventure type of story that's that that i was interested in and going there and do it it had enough like enough interesting problems to be solved and at the same time it had like yeah this this kind of trying trying something new trying something ambitious kind of kind of that i generally like so it was in london that company didn't really work out unfortunately but it was it was hard and ambitious and not everything works out i did try that again starting another hedge fund from scratch with a few other people in amsterdam i worked there for a few more years and eventually eventually i got bored like i I generally working in trading is an interesting and exciting problem.

17:03Market is very hard. And the depth of what you can go of trying to understand it model is very deep. And I worked with pretty smart people overall, but I stopped feeling I am growing after a few years of doing that. And at the same time, together with a friend I was working with, we just started chatting about AI and about this. artificial intelligence. And what really drew me to artificial intelligence was reinforcement learning. And specifically the DQN agents trained by people in DeepMind in 2013, but I think it was a few years later that I actually learned about those results. From my perspective, and again, this is just how my brain works, the 2012 ImageNet results weren't that significant like during during my university years i learned a bunch about like how classical ai the neural networks weren't very fashionable back then but i still learned about what they are i learned about svms and all kind of methods how you train classifiers and for me it was kind of like obvious and natural if you have enough parameters and tweet it hard enough you will fit a classifier to whatever you want it was was kind of obvious well it's like what was to me not obvious is I never considered classifiers a smart thing classifiers that you learn a function on some set of inputs to have some some set of outputs and you can can keep training it to approximate better and better what but kind of what was something that I missed back then is that when you can fit any function better and better you can start shaping behaviors and strategies and when I really saw that was in the in the like DQN results where they applied the same things that worked in ImageNet like neural networks and they weren't particularly big or impressive neural networks with a classical field of reinforcement learning to solve simple computer games and turns out those simple neural networks with a simple learning algorithm started learning pretty complex computer their games and exhibiting very interesting behaviors.

19:20I saw those behaviors, I saw those results, and I was like, this is what I want to do for the rest of my life, which is like, you know, not a very long horizon with a 20-something things about, but I was like, this is what I want to do. Where do I do that? Like, you know, Google search, where are places where you can do reinforcement learning in this world? Like, you know, Google DeepMind and OpenAI came up with this kind of, at that moment, pretty small and like somewhat known but they are yeah you joined you joined open ai in 2019 right so like very much uh yeah very much in the early days still very much in the kind of like non-profit era yes of open air so how did you connect with them i just applied for the website i had the most it was like boring and uninteresting thing in the world which is like you know open ai.com jobs like you know apply send resume and hope they respond and then you know like luckily enough, I was, they did.

20:15I don't know how many resumes OpenAI I was getting in that time. I think it's definitely much less than today. But I was like, I came there and I was like, that doesn't matter what do I do as long as it's reinforcement learning. So you joined in 2019 and with a passion for reinforcement learning. So was that around the Dota 2 moment? Because OpenAI, interestingly, in those early days of 2019 did a lot of reinforcement learning focused work, right? And then there was a whole unsupervised learning, GPT moment that happened afterwards, but it started from roots in reinforcement learning. So did you work on that project specifically or was it too advanced by the time you showed up?

20:59So the project that I worked was robotics project at OpenAI, which shared the same code and same methods as the Dota project. Like in one hand, the Dota project was OpenAI's way to demonstrate the world, what scaling up reinforcement learning can do. And in some way, it was like taking the 2013 DQN agents and just doing all the hard work of making it bigger and bigger and solving harder and harder problems. And OpenAI generally from the very beginning was aware and really, you know, it was simple but genius insight that you need to have a large scale system to learn really, really interesting, complex behaviors.

21:43And that was one way of what Dota was trying to show that by scaling up reinforcement learning, we can solve pretty complex environments. And then there was another project, there were, I think, three reinforcement learning projects at OpenAI at that time. And the second one was robotics, which is applying the same methods that we now knew or were proving that can solve pretty complex computer games. can they solve all the practical problems OpenAI was always optimistic and ambitious and trying to see if we can scale our own to solve data can it load my dishwasher, can it fold my clothes, can it build a house and this is what we are doing, the project I was working on was focused on dexterous manipulation which was back then and still continues to be an elusive challenge for trained policies And we got to a showcase of demonstrating that a hand controlled by a neural network was able to solve Rubik's Cube, which is a pretty delicate and complex task to do.

22:47So fast forward to today, still in the same vein of like the behind the scenes of, you know, you all had OpenAI and sort of life there. What's a day in the life of Jerry? Like what does somebody like you do? Like you read papers, you train models, you manage teams. What's your day like? Yeah, my days are surprisingly uniform, which is I come to the office early in the day after driving my kids to school. Then what do I do all day is basically talk to other researchers. I talk to other researchers all day, every day. And this is basically exclusively what I do. I take ideas from people, bounce with them, brainstorm with one partner, then move to another one and do the same thing over and over and iterate.

23:41and in that way, keep refining our research program. Sometimes those are group meetings and group meetings are there as well and have their own team dynamics. But that is basically exclusively what I do. The only thing that changes is the topics of research from meeting to meeting and from person to person. How are priorities in research determined this range of possible projects? Is that top-down? Is that bottoms-up? Do people suggest ideas and others vet them? How does that work? Yeah, yeah, yeah. It's the art of structuring, organizing, and leading a research project is something that I generally learn to appreciate very quickly in OpenAI's journey and in my career.

24:34There is something we are good, is structuring research projects. And I think it's a unique mix. We can't say it's top-down. We cannot say it's bottom-up. It's a mix of those two, which is balancing all the important aspects. One thing OpenAI embodies and determines is that we all work on a very few projects total. There are not that many projects. OpenAI is not trying to do everything. We are not trying to have portfolio. We are trying to have multiple different bets. always the idea is we do a few core things really really well and put a lot of effort there which means there are there needs to be a lot of people working together on the same large scale large ambition project and we have we have a few of those all number probably three or four depending depending on how do you how do you call it and that's and that's it and from that perspective like people don't have ultimate freedom it's not that that people come to open ai and say hey i want to do this and they just they just do this because you need to do something towards the goal of one of those four projects and then within those projects like we try to be like relatively bottoms up in in a way as long as it feeds again into into those goal and the most important part of the research lead is keep to making sure all the researchers are working towards this one shared goals and that don't fracture in their own ways of thinking and doing things.

26:04And so it's an incredibly hard thing. It's a very, very hard job and not always easy to visible how delicate it is. But that's a lot of what it is. I don't think top-down structuring of research doesn't work in research organizations. I really don't believe in it because you are not kind of hiring some of the smartest people in the world. And OpenAI has incredibly, incredibly smart people to kind of tell them what to do. They need to figure out what to do, but they cannot figure out in the whole space of things what are cool things to do. They need to figure out from within the space of what the project needs and what could advance the research goals of OpenAI the most.

26:51And to which you're saying, is there a collaboration between the teams working on the three or four projects at the same time? Because I would imagine putting myself in your shoes, OpenAI shoes, there is probably a tension between wanting to be collaborative in general. But equally, I mean, this is probably the most important IP in the world. So you probably want to make sure that not everybody knows everything about everything. Well, perhaps not. I'm speculating here. How do you think about that collaboration versus some protection of IP? You'd be surprised, but the truth is in research at OpenAI, which is around slightly less than 600 people at the moment, everyone knows everything, really it does.

27:37And we always have been fully transparent. And in some way, you are a little bit shooting yourself in the foot if there is a researcher that doesn't at least have the chance to learn about everything because they don't have the best information to their job in a best way and like it is like yeah it is some like a risk of losing ip but i think i think the risk of not doing the right thing and of people not being informed about about research and not being able to do the best research is much higher in my in my personal opinion and how how i i approach those things so we are we are extremely internally transparent within research and and that is that is one of our operating principle as the goal is to do the best research we can and train the best models we can consequently and the culture generally is very collaborative like you know it is always the case when you have 600 people when you have groups of people they're always like one person doesn't like the other person because they looked at them weirdly or one person thinks the other smells bad or just doesn't like their ideas.

28:42Like that does happen. Those are humans and those are human things. But generally in large scale, I think we really are, have this belief like, you know, we are together in this goal that's larger than every one of us. It is a very positive sum game because the AI seems to be getting like, you know, only more and more significant and the success of open AI is far from guaranteed. It depends on us doing great work every day. So there's a lot of like feeling of shared fate and the fact that we all need to rely on each other to do our job to achieve this shared mission. So I generally think with all the caveats of human nature getting in the way sometimes, I think on a large scale OpenAI is very, very collaborative.

29:27How do you all manage to keep that pace of releases? It seems to me from the outside that there's a tension again between research, which in some ways kind of feels like it could be a long-term kind of thing. And on the other hand, you guys seem to be just shipping and shipping and shipping and shipping across the organization, but including in terms of core models, again, to the point that you went from 01.203 to GPT-5 in like a year. How do you balance all of that? Why are you guys able to ship so quickly? I think the fundamental reasons for it is, in general, OpenAI, in my at least worldview, is a generational company in a way that we have incredible momentum behind us.

30:20We know that we were doing pretty great in the past and we need to continue that. We have incredibly smart people. Literally, the most talented people in the world are all coming and want to work at OpenAI right now. which means like people every output per single person is incredibly high and everyone really every every single person does a whole lot so so we have we have like a momentum that carries us forward we have really great people that work together we have like good operating like way of structuring research and and and and can borrow a lot from silicon valley how how to get things done quickly and people are generally very excited about work everyone feels the weight and potential of what we are doing what we are trying to do and because of that people at open ai have a tendency to work pretty hard and and like you know having great people excited about what they are doing or working together reasonably well results in doing a lot of things we understand that there's only one time in history where AI is being built and deployed and developed and people want to do it in the best way that is possible.

31:35Do you all use a lot of your own tools? I think Fiji, Simo was tweeting the other day that I think in the latest, what you all announced at Dev Day today, a lot of it was written by Codex. Is that part of the daily experience? Do you use a model to come up with new ideas for a model? Do you use a codex to write the code? How does that work? Yeah, we definitely use codex a lot for coding and this is only getting better. As I said, I use ChatGPT a lot, although not surprisingly, not that much for actually coming with ideas. But for a lot of questions that I have, I think I am a pretty heavy user of ChatGPT right now, happily paying like$200 a month for it.

32:22And I think I'm getting... They make you pay? For a while, it's worth it. They are making you pay. And I'm kind of like pretty, pretty okay with this because then you get pretty generous like usage limits and not really bottlenecked on it. Thank you for all of that. Let's switch tags and go back to how all of this works. So is the right way to think about modern AI systems at OpenAI. By modern, I mean as of October 2025 versus the old days of, you know, nine months ago. So exactly. So is the right way to think about it as a combination of pre-training and RL? First of all, is that the right way to think about it?

33:10And second, if so, just at a high level, how does the articulation between both of those work? And then after that, I'd love to do a little bit of a deep dive on RL to make this very educational for folks. Today's language models basically can be thought as first they are pre-trained, then you do reinforcement learning on it. The reinforcement learning would not work without pre-training. And I think in a similar way, pre-trained models have a lot of limitations that are very hard to resolve without doing something that looks like reinforcement learning. So I think both of those bits are here to be and to stay like i think the way how they are like combined and do that may and probably will evolve in the future nothing should be treated as dogmatic and fixed and we need to we need to keep generally figuring out the way how to train better models and this is what we are trying to do the interesting thing and you know i i can credit that to to ilia how much foresight that he had but whenever I was like started at OpenAI early 2019 right and and and I remember there was like research all hands or something like that where Idyia came on stage and talked about like what is what is OpenAI's research program what are we trying to pursue and what he said and at the beginning of 2019 was to train large generative model on all data we can and then do reinforcement learning on it that was that was the open ai research plan at the beginning of 2019 and this is exactly what we are doing today now the algorithms change architectures change like i don't think he was even thinking about transformer at that moment like you know the gpt was like there was some gpt but it was like a toy example that someone was playing with but the goal training large generative model all the data in the world and then doing reinforcement learning with it was already there at the core DNA of OpenAI and that's what is happening right now.

35:12So let's do, if you will, a little bit of reinforcement learning 101 to make this really interesting to like a broad group of people listening to this. So in very simple terms, like explain it to me like I'm 10, what is reinforcement learning? Yeah, I usually the metaphor and the analogy I have to reinforcement learning is like training a dog. It's very, very close. And I used to have a dog when I was a teenager. And even I remember my parents did. I didn't know anything about raising a dog. But they kind of invited through some friend of a friend, a fireman, who I think was working with like service dogs.

35:57And he came to me and he basically told me a little bit about how do you train your dog. And what most dog owners that are ambitious about training your dogs know, it is always extremely important to have a bag of treats in your pocket. That's what you always do. And whenever you see your dog behave well, what you should be doing, you should smile and you should give your dog a treat. Whenever you see your dog do something bad, you basically like give your attention away turn away and become sad and before the years of breeding the dogs discover it's a it's a it's a like you know bad reward and bad behavior and this is exactly doing that but with models we elicit a lot of different behaviors in the models put them in challenging situations and then we give them cookie if they do something we want if they do a good thing and give them some kind of punishment and negative reward if they do something like that we that we don't want and that we don't like in a good way a good way to do rl is if you balance those things so if you kind of give cookies half of the time and punish the other half the time but this is almost like a mathematical kind of uh kind of aspect of it but that's that's the most important part, which is like elicit behaviors, reward the good ones, and then going forward, the model will be most more likely to do what you want and less likely to do what you not want, and for that it improves.

37:27It is the way how to train models to elicit actual behaviors. That is not versus next token prediction. If you pre-train them all, you literally train them all to predict the next token. RL is a completely different gradient and a completely different set of what we want to get out of them all. And getting the model to do what you want just for some vocabulary and semantics, you hear sometimes the term policy. So in RL, you hear terms like agent, environment, action, reward, and policy. So I think a lot of those are sort of self-explaining, but policy is what? That's a strategy, that's a behavior of the model?

38:11Yeah, policy is the behavior of the model as the model weights represent what it does when put in a different thing. Like model in the end is a mathematical object and you can define it. Policy is a mathematical function that maps observations to actions. What you see and then what you do with what you see. Yeah, so agent is a model. Action is what the model does. Reward is how you say whether that's good or bad. environment you you hear a lot of things uh these days about designing the right environment for rl what does that mean environment is like in some way it is everything that the model sees but but the interesting thing about difference about rl environments and most other types of like what you can call supervised learning or unsupervised learning is that reinforcement learning environments you want them to be interactive you want like them to evolve as the model does things in general like like similarly how like if you want to like learn how to play guitar you kind of take a guitar and you strum it and what happens is you hear sound of that and then you hear it and then you can do like learn to play like like with with actual like feedback of what is happening with the guitar and it's in that way you know the environment is like how does the world react to your actions and a lot of like what drives your actions is the is what is happening in your in your environment and it's in in your in your world and that's kind of like the only way how to like really teach agents to like learn to react to changes in the environment is through it's reinforcement learning can you give us a little bit of a bird's eye view of the evolution of rl over the years?

40:03Mostly how does modern RL differ from historical RL? Yeah, yeah. Again, there was super historical RL. Not even that old, but the main tectonic shift was when combining neural networks with reinforcement learning. Reinforcement learning predates neural network as a general mathematical method of optimizing behaviors in like mathematically defined environment and as a method of study. That's what is known as deep reinforcement learning. Is that right? Yes, and then the deep reinforcement learning that basically like deep mind, like invention of combining neural networks with reinforcement learning, the DQN moment I talked to you about.

40:55And then from there, like there was a moment where like there was a pretty active like area of research of reinforcement learning on games like when even when i started like 20 2019 the reinforcement learning was was kind of fashionable at that moment although not like very successful but we were the reinforcement learning was able to solve a lot of games but the bottleneck was there that the models were not pre-trained in any way we were we're training a lot of behaviors playing games like we even got alpha go moment out of that which a lot of people got very excited about it but was still like learning behaviors without the models that were that were meaningfully smart about those behavior there were still a lot of kind of like you know you don't want to call it caveman intelligence but but something something in that regard about models not being really smart even though being being pretty heavily reinforced and there was like a long research in that and A lot of cool results and theoretical understanding of RL comes from those days because people were researching RL actively.

42:00But it was in some way at that end of doing RL without pre-training. And then in my moment when I finished working on robotics, I started working on teaching language models to code. But having pre-trained models was a really big deal. And the GPT era of scaling and of large-scale ingesting lots of data to really train great models enabled us already at that moment to start RL. And that was one of the first things I did almost immediately. Whenever GPT-3 was trained, I tried to do RL on it. And there were always bottlenecks. The systems were kind of clunky. It was hard to figure out. What are the right algorithms?