John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI

30:20And those things might be more like have more to do with the learning algorithm than what the model does at runtime when you just have a fixed model. So I would say there are probably some analogies, though it's, I don't know exactly like how close it is, but I would say to some extent it is, the models do have drives and goals in some meaningful way. And in the case of RLHF where you're trying to maximize human approval as measured by a reward model, the model is just trying to produce something that people are going to like and they're going to judge us correct. I've heard two ideas in terms of using that in a model, a lot of type of thing to get better at reasoning.

31:08At least publicly, the kinds of things I've seen. And I'm curious to which you think is more promising. One is that the model learns from, it outputs a bunch of potential trains of thought, and it learns to follow the one that leads to the correct answer, and is trained on that before deployment. And the other one is you use a bunch of compute to do inference in deployment, which involves the model talking to itself, you know, while deployed. Which one do you expect it to be closer to when it's like really good at reasoning? Is it because it's doing just a bunch of inference clause? It's just because you've trained it to do all of that.

31:47Well, I would say you could define reasoning as tasks that require some kind of like computation at test time or maybe some kind of deduction. So by definition, reasoning would be tasks that require some test time computation. and step -by -step computation. On the other hand, I would also expect to gain a lot of doing some kind of training time computation or practice at training time. So I would think that you get the best results by combining these two things. So my uncle had prostate cancer, and I wanted to know my own risk. And I got a 23 -and -me test, but it was mostly useless. I mean, the whole, what are your odds of lacking chocolate is not what I was looking for.

32:41So I expert in my data onto nucleus genomics, and immediately I got my risk profile for almost two dozen diseases, including prostate cancer, and it turns out my risk is higher than 97 percent of people with my ancestry, which is a very useful thing to know, because I now know to get screened early. Ironically, test like 23 and me don't even look at the variants which have the largest impact, including for prostate cancer. Many people don't know this, but 23 in me looks at less than 0 .1 % of your DNA. And that's why I've freer ordered nucleus premium pole genome sequencing. It's the only clinical great test that reads 100 % of your DNA.

33:18I've spent a lot of time digging into this company and I think it will be a big change in what we can get at a genetic test. So if you want to live a long and healthy life, you can freer order nucleus premium at mynucleus .com. Alright, back to John. Right now, you have these two ways in which the model learns. It's either in training, whether it's free training or with the post training, but it's like most of the compute in training is spent on free training and it's just glossing over trillions of tokens, just like standing by as they, almost like skimming trillions of tokens worth of information, which if a human was subjected to that would just be totally confused, right?

33:59a very efficient way to learn. And the other way is in context learning, but of course, that is more sample efficient there, but it's destroyed with each instance. I'm curious if you think that there is a path for something in between those where it's not destroyed with each instance, but it's also not as not as sort of frivolous as just seeing trillions tokens where it's more deliberate and active. Yeah, so do you mean models having some kind of medium term memory, so too much to fit in context, but much smaller scale than pre -training? I'm not sure if memory, it might be memory, I don't have context, but certainly when I'm trying to prepare for this conversation, it feels like I think of what I should understand this, so I look it up and I really carefully and I maybe think about it as I'm reading it.

34:55And I'm not sure what it naturally corresponds to in terms of models, but what that looks like, I'm curious. I see. So it's not just memory, but it's also somewhat like specializing to a task that specializing to a certain task or putting a lot of effort into like some particular project. And I'm not sure if it's specialization more or so. I'm thinking about I don't understand this part. So let me look into this part deeper. I already understand this. I'm going to like specializing to your existing knowledge base. I see. So it's not just about finding like, I don't know, training on a bunch of sources that are relevant of fine tuning on some special domain.

35:32It's also about like reasoning about like developing some knowledge through your own reasoning and also using some sort of introspection and self -knowledge to figure out what you need to learn. Yeah, I would say that does feel like something that's missing from today's systems. I mean, I would say people haven't really pushed too hard on this middle ground between large scale training, where you produce the snapshot model that's supposed to do everything, a deployed model, and then on the other hand, in context learning. And I think part of that is that we've just been increasing context length so much that there hasn't been an incentive for it.

36:17So if you can go to like 100 ,000 or a million context, then that's actually quite a lot. And it's not actually the bottleneck in a lot of cases. But I agree that you would probably also want to supplement that by some kind of fine tuning. Like the capabilities you get from fine tuning and in context learning are probably somewhat complimentary. So I would expect us to want to build systems to do some kind of online learning and also have some of these cognitive skills of like introspecting on their own knowledge and seeking out new knowledge that fills in the holes. Is this all happening at the same time?

37:03Is it just like a new training regime where all these things can happen at once or whether it's the long horizon training or whether it's this kind of training? Are they separate or are they just because like the model is smart enough so they can both introspect and it can act on longer horizons than you can get adequate reward on long horizon tasks. Yeah, I would say if you're doing some kind of long horizon task, well, I would, you're learning while you do the task, right? So the only way to do something that involves a lot of steps is to like to have learning and memory that gets updated during the task.

37:39So, like there's a continuum between short term memory, between short term and long term memory. So, I would say, yeah, I would expect this capability would start to become, like the need for it would start to become clear where when we start to look at long horizon tasks more and to some extent just putting a lot of stuff into context will take you pretty far because we have really long contexts now, but you probably also want things like fine tuning. And as for like introspection and the ability to active learning, that might like automatically fall out of the model's abilities to know what they know because they have some, like models have some calibration regarding what they know, and that's why that's why models don't hallucinate that badly because yeah, they have some understanding of their own limitations.

38:49So I think that like same kind of ability could be used for something like active learning. And how so there's all these complicating our all procedures that many of whom you've pioneered, how many of them will be relevant when you get to the point where the model itself is this smart that it can act as a certain environment and interacting them more online and a stable way. Is it is is a path for progress going to be more straightforward than the solutions that were required for our own the past. Well, I think policy grading algorithms are not the most sample -efficient algorithms, so that's probably not what you want to do at test time if you want to learn really fast.

39:37But though who knows, I mean, maybe it's not that bad. So I think something like motor learning in animals is probably something like a policy grading algorithm. And so for example, you're like learning how to shoot baskets. I think you probably like that takes maybe thousands of tries to get more accurate. And I think you probably there's probably something that's like a policy grading algorithm underneath. But that's not going to be the fastest way to learn in like if you have a model trying to do a project or some kind of task. So I would think we want to rely more on like in -context learning, where you effectively have a learned algorithm, like you've learned how to explore, you've learned how to try all the possibilities exhaustively.

40:33And instead of doing the same thing over and over again, making the same mistake. So yeah, I would say we'll be able to do things that look more like learned search algorithms. And that'll be the kind of thing that gets used in a particular task. Interesting. All right. I want to step back and ask about your own history. So at least at opening I. So you let the creation of TGBT. At all point, did you realize, first of all, these LLMs are the path to go. And then a chatbot would be, or some way to instruct them would be a useful thing to do. So just walk me through the whole lineage from like, when this became your main focus, and yeah, what the process was like.

41:18Yeah, so early, so we had, before Chatchee VT, we had, open AI had these instruction following models. And that was the idea there was, we had base models and people can prompt them in elaborate ways, but they're also kind of hard to prompt. you had to basically do autocomplete, so you have to set up a very good prompt with some examples.

41:48So people at OpenAI were working on just taking the base models and making them easier to prompt, so that if you just wrote a question, it would answer the question instead of giving you more questions or something.

42:02So we had these instruction following models, which were kind of like base models, but a little easier to use. And those are the original ones deployed in the API. Or after GPT -3, those were the next generation of models. Then at the same time, there were definitely a lot of people thinking about chat. So Google had some papers, like they had Lambda and earlier Mina. So they had these chatpots and it was more like a base model that was really specialized to the task of chat, really good at chat. And I think at least looking at the examples from the paper, it was more used for sort of fun applications, like where the model would take on some persona and pretend to be that persona.

42:52It was not so functional like help me refactor my code. So yeah, there are definitely people thinking about chat. I had worked on a project before looking at chat, called WebGPT, which is more about doing question answering with the help of web browsing and retrieval. And, well, when you do question answering, it really wants to be in a chat, because you always want to ask follow -up questions, or sometimes you need a cloud, the model should ask a cloud -fine question, because the question is ambiguous. So it's kind of clear after we did the first version of that, that we should, the next version should be conversational.

43:33So, anyway, So we started working on like the conversational chat assistant and we, this was built on top of GPD 3 .5, which was done training at the beginning of 2022. And that model was quite good at language and code. So we quickly realized that it was actually quite good at coding help and that was one of the things we were excited about. So, yeah, we worked on that. We worked on that for most of the year, and we had browsing as another feature in it, though we ended up de -emphasizing that later on because the model's internal knowledge was so good that the browsing wasn't the most interesting thing about it.

44:20And then we were thinking about, we had it up for beta testing or two friends and family for a while, and we were thinking about doing a public release. But at that time, actually, GPD -4 finished training in August or, yeah, in August that year. And actually, the flagship RL effort at OpenAI was the instruction following effort because that was the models that were being deployed into productions. So like the first fine tunes of GPD -4 or use that whole stack. And those models were really good. And everyone got really excited about that after seeing the instruct fine tune GP4s. But so they were really, really good.

45:07They would occasionally give you amazing outputs, but they were also a little bit. The model was clearly pretty unreliable. Like it would sometimes hallucinate it a lot. And it was like pretty, it would sometimes give you pretty unhinged outputs. So it was clearly not quite ready for prime time, but it was obviously very good. And yes, I guess that people forgot about chat for a little while after that, because about this alternative branch. But then we ended up, we pushed it further and we ended up mixing together all the data sets, like the instruct and the chat data, and to try to get something that was the best of both worlds.

45:45And I think the models we, the chat models were like, We're clearly more, it was easier to use. It was sort of more, it sort of automatically had much more sensible behavior in terms of the model knowing its own limitations. That was actually one of the things that I got excited about as we were developing it. I realized a lot of the things that people thought were flaws in language models, like just like blatantly hallucinating could be not completely fixed, but you could make a lot of progress with pretty straightforward methods. Oh yeah, and also the other thing about chat was that like when we had these instruct models, like the task of complete this task put in a nice way or in a helpful way, that's like a pretty poorly defined task.

46:39So I think that task is both confusing for the model and for the human who's supposed to do the data labeling. Whereas for chat, I think people had an intuitive sense of what a helpful robot should be like. So I think it was just much easier to tell people, for people to get the idea of what the model was supposed to do. Yeah. And so that, so as a result, I think the, like the model had a much more coherent personality and like it was much easier to get, like, robot, like pretty sensible behavior, robustly. Interesting. Is it the case that anybody could have made chat GBT using your publicly available fine -tuning API?

47:25I mean, they could have, I don't remember the status of which models were available for fine tuning. You, assuming we had 3 .5 available for fine tuning at the time, you could have made something pretty decently close, but I'm not sure you would have, I don't think you would have been able to do just one iteration of fine tuning where you have like purely human written data and you fine tune on that, I think you would want, like you would want to do several iterations. Like if you're not going to do RL, which we did, you would want to do some kind of iterative, supervised fine tuning where you have humans edit the model generated outputs, because it's really hard to get people to, like if you train on human generated data, even if it's really high quality, it's just hard for a model to fit that data perfectly, because it might not be, it might not be something a model is capable of outputting.

48:25So you need to do something iterative that looks a little bit more like RL. So I think if you had done that, you could have gotten something pretty close, but that would have been kind of non -trivial. But we also had another instruction following model trained with RL that was released a little before ChatchyBT. So I think if you put a chat like wrapper on that, you would get something decently close, but like that model, like if you just prompted it with chat. But that model had some differences in strengths. Like that model was pretty good at writing and poetry and so forth, but it wasn't as good at knowing its limitations and factuality and so forth.

49:14So, starting to wrap from 3 .5, I think I heard you somewhere say GPT -2, you're super impressed. Compared to your expectations in 2019, has AI progressed faster or slower than you would have expected? I would say faster than I would have expected since GPT -2. Yeah. I was pretty like bought into scaling and yeah, pre -training and so forth being a good idea. but when GPD2 was done, I would say it wasn't completely sold on it, being revolutionizing everything. Like, I only really pivoted what I was working on and what my team was working on after GPD3. So after that, we kind of got together and said, oh yeah, let's, this language model stuff works really well.

50:04Let's see what we can do here. But yeah, after GPT -2, I wasn't quite sure yet. Especially if the stuff we were talking about earlier with RL starts working better with these smarter models. With a fraction of compute that has spent on training that is free training versus post training, change significantly in favor of post training in the future? Yeah, there are some arguments for that. I mean, right now it's a pretty lopsided ratio, but you could argue that the output generated by the model is high quality compared to or higher quality than most of what's on the web. So it sort of makes more sense for the model to think by itself instead of just like training to imitate what's on the web.

50:53So I think there's a first principles argument for that. And I would say we found a lot of gains through post training. So I'm not sure. So I would expect us to keep like pushing this methodology and probably increasing the amount of compute we put into it. The current GBD4 has a EOS code that is like 100 points higher than the original one that was released. And is that all because of what you're talking about with these improvements that are brought on by post training? Or yeah, I would say that most of that is post training. Interesting. So there are a lot of different separate axes for improvement.

51:40We think about data quality, data quantity, just doing more iterations of the whole process of deploying and collecting new data and changing what kind of annotations you're collecting. So there's a lot of things that stack up, but together they give you a pretty good, like effective compute increase. Yeah, that's a huge increase. That's like really interesting that there's this much room for improvement from post training. What makes for somebody who's really good at doing this sort of RO research? I hear it's super finicky, but what is the sort of intuitions that you have of that enable you to find these ways to mess with the data and set up these environments.

52:29I'd say I just have a decent amount of experience at this point from the different parts of the stack, from RL algorithms obviously, since I've worked on those since grad school, to the data collection, in the annotation process to like language, playing with language models. So I mean, I'd say I just dabbled with these things and I'd say the people who do well at this kind of research have some view of the whole stack and have a lot of curiosity about the different parts of it and also sort of think about, well, do you want to be both empirical and use experiments, let experiments update your views, but you also want to think from first principle somewhat,

53:25what, like assuming that, like, learning works, like what would be the ideal type of data to collect, and that sort of thing. So because there doesn't seem to be a model release in stupidity four, that seems to be significantly better, there's seems to be the hypothesis that potentially we're hitting some sort of plateau and that these models aren't actually generalizing that well. And you're gonna hit some sort of data wall beyond which point, the abilities that are unlocked by memorizing a vast corpus of free training data won't actually help you get something much smarter than GPD4. What do you think the hypothesis is that wrong?

54:05And I think we talked about some examples generically about generalization, the Spanish to English, and so forth, but is there, yeah, I mean, okay, so maybe this is a run on set question, but one example I was thinking of was the idea that there's transfer from a language, reasoning and code, betraying a bunch of code gets better reasoning and language. And if that's the, is that actually the case? Do you see things like that, which suggests that there's all the scripted positive transfer between different modalities. So once you try to train, training on a bunch of videos and images, it will get smarter and it'll get some other synthetic data.

54:45Or does it seem like the abilities that are unlocked are extremely local to the exact kind of labels and data you put into the training corpus? Yeah, okay. Yeah, I'll try to have a response. So first, are we about to hit the data wall? I mean, I wouldn't draw too much from the time since GPD4 was released because I mean, It takes a while to train these models and to do all the prep to train a new, like generation of models. So yeah, I wouldn't draw too much from that fact. I would say there are definitely some challenges from the limited amount of data, but I wouldn't expect us to immediately hit the data wall, but I would expect the nature of pre -training to somewhat change over time as we get closer to it.

55:46In terms of generalization from different types of pre -training data, I would say it's pretty hard to do science on this type of question because you can't do that, create that many pre -trained models. So maybe you can't train a like a GPT -4 size model. You can't do a Belation studies at GPT -4 scale. Maybe you can do like train a ton of GPT -2 size models, or maybe even a GPT -3 size model with different data blends and see what you get. So I'm not like aware of any results, like public results on like a Belation's involving code data and reasoning performance and so forth. with. So that would be, I'd be very interested to know about those results.

56:35But I'm actually curious about, I mean, if one of the things is that the model gets moderates as bigger, what in the relation on a GPT2 level model, which suggests that there isn't that much transfer, how much evidence is that provide for the level of transfer on a similar set of domains in the GPT4 level model. Right, you might not be able to conclude that if transfer fails at GBD2 size, then it's also going to fail at a higher scale. So it might be that for the smaller models, you... Yeah, for the larger models, you learn these better shared representations. Or for the smaller models have to lean too much on memorization, whereas the larger models can learn how to do the right computation.

57:22So I would expect this to be true to some extent. This might have a very simple answer, but so bigger models, you train them on the same amount of data and they become smarter or conversely they can to get the same amount of smarts you have to train them on less data. Why is that the case? It's got more parameters. It's all less things than that was equally smart. Why is that the case? I don't think anyone has a good answer for a good explanation of the scaling law with parameter count. I mean, there's some, I don't even know what the best sort of mental model is for this. Like clearly you have more capacity if you have a bigger model, but so like you should be able to eventually get lower loss.

58:13but I guess why are bigger models more sample efficient? I guess you could, I can give you some very sketchy explanation. Like they have, like you could say that the model is sort of an ensemble of a bunch of different circuits that do the computation. So it has, you could imagine that it's doing, it has a bunch of computations that it's doing in parallel, and it's doing some, the output is a weighted combination of them. And if you have more just width of the mock, or if you just have, I mean, actually, width is somewhat similar to depth because with residual networks, you end up, the depth can do something similar to width in terms of updating what's in the residual stream.

59:05But if you, yeah, you could argue that you're learning all these things in parallel. You're learning all these different computations in parallel and you just have more of them with the bigger model. So you have more chance that one of them is lucky and ends up like having high, like winning, guessing correctly a lot and getting up weighted. So that's kind of like, what would be the, yeah, there's some algorithms that work this way, like that, like mixture, what is it, mixture, some kind of mixture model, or multiplicative weight update algorithm. Yeah, there's some algorithms that kind of work like this.

59:50So where you have like a, some kind of mixture of, I don't want to say mixture of experts because it means something different, but like basically a weighted combination of experts with some learned gating. And actually, anyway, I said something slightly wrong, but anyway, yeah, you could imagine something like that and just having a bigger model gives you more chances to get the right function. So that would be, and then of course, it's not just like you have a bunch of like totally disjoint, like functions that have, you're taking a linear combination of, it's more like a library where you might chain the functions together in some way.

1:00:31So there's some composability. So, yeah, so I would just say the bigger model has a bigger library of different computations, including lots of stuff that's kind of dormant and only being used some of the time. But it has more space to look for the circuits to do something useful. I want to ask you about stepping back from the current research questions. Just stepping back, I want to understand just sort of like modal scenario of what happens for the next few years. I think it's two towards the beginning of the conversation we were talking about the case in which the progress is really fast. But just like let's just take like the modal scenario, you're unlocking, long horizon RL at some point.

1:01:20But then as you said, there's potentially other bottlenecks. So what's happening, how good are these models? How are they being deployed? What are the modalities are part of them? At what stage are these being unlocked and so forth? I just want to understand your broader picture of what the next few years look like. Yeah, I would expect, I would expect things like, okay, new modalities to be added like over time or pretty soon. I would expect the capabilities to generally keep getting better through a combination of pre -training and post -training, and that'll open up new use cases. So right now, AI is still not a huge part of the economy, like there's a pretty small fraction of jobs that it can help with at all.

1:02:09So I expect that to be higher over time, and not just from the models improving, also So from people just figuring out how to integrate them into different processes. So even if we just froze the models at their current state, I think you would still see a lot of growth in how they're being used. So I would expect there to be a lot of, like, I would expect AI to be used much more widely. And I would expect it to be used for more kind of technically sophisticated tasks. Like I gave the programming example earlier of doing longer projects, but also helping with various kinds of research. So I hope that we can use AI to accelerate science in various ways.

1:03:02and just like because you can potentially have the models like understand all of the literature in a given field and be able to like be able to sit through tons of data like more than a person would have patients to do. So I would hope that we can basically like, yeah. Well, I hope the form factor would basically be that people are still driving all this and you have your helpful assistance that you can use, you can direct and point to lots of different problems that are useful to you. And everyone has all these AI's helping them do more. Get more done. Hey, everybody. Real quick, I want to tell you about a tool that I wish more applications used.

1:03:53So obviously, you've noticed every single company is trying to add an AI chatbot to their website. But as a user, I usually find them really annoying because they give these long generic, often useless answers. Command bar is a user assistant that you can just embed into your website or application. And it feels like you're talking to a friendly human support agent who's browsing with you and for you. And it's much more personalized than a regular chatbot. It can actually look up user's history and respond differently based on that. It can use APIs to perform actions. It can even proactively nudge users to explore new features.

1:04:32One thing that I think is really cool is that instead of just outputting text, command bark can kind of just say here, let me show you, and start browsing alongside the user. Anyways, there are a bunch of great products already. You can learn more about them at commandbar .com. Thanks to them for sponsoring this episode. But obviously at some point, they're going to be better than everyone, whatever they want to do. So, yeah, what will all process like? Right now, they're clearly only helping you at some point. They're able to just do things for you and maybe run entire forms for you or whatever.

1:05:10At that point, is it just going to be a smooth process? And at that point, the hope is that we have systems that are aligned with the user enough that they can count on the firm being run in the way they expect. and so forth. Yeah, I think, well, we might not want to jump to having AI's run -hole firms immediately. I mean, we might want to have people like overseeing, like overseeing these important decisions and calling the shots. So even if the models are good enough to, like to actually run a successful business themselves. So, yeah, to some extent there might be choices there. And I think people will still have different interests and what they want to, different ideas for what kind of interesting pursuits they want to direct their AIs at.

1:06:07And like they can, people could like, yeah, do a lot of, AI doesn't necessarily have an intrinsic, like any kind of intrinsic desire. Yeah. Most people, we put it in the system. So I think so people can still end up being, even if AI's become extremely capable, I would hope that people are still the drivers of like what the AI's end up doing. Yeah, but I wonder if the economic equilibrium is so far from that where you have the equivalent of Amdahl's law in a firm, the slowest part of the process is the one that's going to bottle like you. And so, you know, the AI makes all the non -human parts of the firm, 10x more efficient, the firm can no longer, you know, it's still bottle like by that step.

1:06:59And so, if in the, if like one company decides to proceed by keeping humans in the loop on all the things that you really want to human oversight on, then they'll just be out competed by other companies. If one country decides to go this route, other countries will beat it. This doesn't seem, I hope this is like, yeah, I wonder if this is a sort of a sustainable plan for keeping humans in the loop. Right. So, I think if we wanted to keep humans in the loop, which seems reasonable, and it turned out that firms with any humans in the loop were out competed by firms that didn't have any humans, then I think then you would obviously need some kind of regulation that like disallowed having no humans in the loop for running a whole company.

1:07:47But there's so many companies in the world, and well, I guess in any country, but I'll let along the world. But yeah, I wonder if it's better to do the regulation on companies and to say like, you've got to keep humans in loop in important processes, but then you had to define what important processes are. You've got to monitor every single company. And you also got to get collaboration in every single country which has firms and net versus if this is a problem should have be solved before the model is even deployed such that hopefully you would get into a situation where you did decide to build a firm and end on these models.

1:08:24It's basically does what you want it to do and you don't need a human in loop. Does that question make sense? I guess I'm wondering in this situation how do we actually monitor every single firm as a human in the loop and And what happens if China doesn't decide to do that and so forth? Yeah, you would either have to have like every country agree to this regulatory regime or you would need all the model infrastructure or the model providers to agree to this kind of requirement. So it's definitely going to be non -trivial. So I guess, yeah, this is looking a ways ahead. So it's a little hard to imagine, to imagine this world before seeing anything, anything like it.

1:09:15But so for example, like there's some questions like would, are we actually confident that AI run companies are better in every way or do we think they're better most of the time, but occasionally they malfunction because AI's are still like, they're still less sample efficient in certain ways, like dealing with very wacky situations. So, so actually AI run firms have higher tail risk because they're more likely to malfunction in a big way. So I guess that there might be some question, practical questions like that that would that would also determine how things play off, like play out, maybe if you just require people to be accountable for various liability, this would also change the incentives a bit.

1:10:02So if it turned out that AIs are better at running everything and they're also completely benevolent and we've totally solved alignment and we can, they're better at being accountable to like, to people than people are, then I would say maybe it's okay having the AI's run the firms, but I think that might be pretty far out. And I think we're more likely to be in a situation where they look better like in the short term, but they still have some problem. Like the AI run entities still have some serious problems. And it's actually like practical considerations that push you more towards having humans in the loop, at least for the near future.

1:10:45Okay, so this is a problem we have to deal with today with RLHF where you have to aggregate preferences across a lot of different humans and it'll be maybe more marked with future or more powerful systems. But when you say, well, we want these eventual AI systems that are going to fully replace humans as part of these forums to be aligned, what does that mean? Like will it mean that they basically do what the user wants them to do? it doesn't mean that they have to result in some sort of global outcome that we're happy with as the kind of people with the stakeholders in opening it, like what concrete would that mean?

1:11:23If the models are being used, like for these higher stakes use cases, then we would have to think about RLHF in a much different way than we are right now. So I would say we're not quite. We're not quite ready for that or the current methods might not be completely sufficient, but I would say we would need to make compromises between the needs of the different stakeholders involved. So we have this document that we're releasing called the model spec, and it's about how we want our models to behave in the API and in chat GBT. And we try to talk about this issue where there are different stakeholders involved, and sometimes there are conflicts between what they might want.

1:12:15Like in our case, we were thinking of the stakeholders as the end user. That means someone sitting in front of chat GBT or some other app. the developer, so this is like someone using the API who might be serving other end users with their app, like the platform, which is opening AI, like we don't want the models to expose us to legal risk and so forth. And then the rest of the humanity, including people not part of the, like who might not be users or customers or anything. So obviously, like the user might ask, ask the model to do something that we think is, like actively harmful to other people.

1:13:08And so we might have to refuse that. By the way, this isn't the order of priority necessarily. So this is just like, we have these four, or so classes of stakeholder. Actually, you could also say maybe in the future we'll say the model itself, the model itself, so I would say we're not going there yet. But anyway, we have these different stakeholders. Sometimes they have conflicting demands and we have to make some call on how to resolve those conflicts. And it's not always obvious how to do that. So I would say we had to think through, yeah, we just had to think through the trade -offs and basically the rough heuristic is that we mostly want the models to follow your instructions and be helpful to the user and the developer.

1:14:01But when this impinges on other people's happiness or way of life, this becomes a problem and we have to block certain kinds of usage. But we don't want to be too, we mostly want the models to just be an extension of people's will and do what they say. we don't want to be too paternalistic, we want to be kind of neutral and not like impose our opinions on people. Yeah, we want to both mostly let people do what they want with the models. I got a chance to read the spec beforehand and it was, I guess, a question of how well that transfers over to how the model itself behaves, but I was impressed with how sensible the trade elsewhere, like it made sense that this is the, I was like, explicitly stated the actual edge cases rather than the kinds of things where everybody can, which are obvious.

1:15:00Like in this case, you really are going up for the edge cases. Yeah, we wanted it to be very actionable so that it wasn't just a bunch of nice sounding principles, but it was like each, each example kind of tells you something about some non -obvious situation and reasons through that situation. Yeah. Okay, now I have a couple questions about the state of the research itself. So famously in the social sciences, things are really hard to replicate and it's a question about how much of the science there is real versus these manufactured bespoke sorts of experiments. When you look at the average ML paper, does it feel like a really solid piece of literature?

1:15:41Does it feel often like like it's the equivalent of what P hacking is in the social sciences. Everyone has their complaints about the ML literature, but I would say overall, I think it's a relatively healthy field compared to some other ones like in the social sciences, just because it's largely grounded in practicality and getting things to work. And if you publish something that can't be, replicated easily, then people will just forget about it. So, and it's like accepted that often you don't just report someone's number from their paper, you also try to re -implement their method and compare it to your method on the same training data set.

1:16:29So I think if you publish methods that are like really hard to implement or are really finicky, they'll tend to get forgotten and as a result, people actually try open source. There work a lot. I guess there's also, there's various incentives that, there's various unfavorable incentives like, yeah, people are incentivized to make the baseline methods, like the methods are comparing to worse and like there are other like mild pathologies like trying to make your method seems sophisticated mathematically. But I would say overall, I feel like the field makes progress. And I would probably like to see a little bit more science and trying to understand things rather than more like hill climbing on benchmarks and trying to propose new methods.

1:17:25And there's been a decent amount of that recently. But yeah, I think it's, we could use more of that. And I think that's a good thing for academics to work on. Oh yeah, and the social science is on a slightly different note. I think actually, I'd be really excited to see more research and using base models to do simulated social science because these models have a probabilistic model of the whole world and you can set up like a simulated questionnaire or conversation. And you can look at how anything is correlated, any traits that you might imagine. You can see how they might be correlated with other traits.

1:18:14So it would be pretty cool to see if people could replicate some of the more notable results in the social science, like moral foundations and that sort of thing by just prompting base models in different ways and seeing what's correlated. What is that standard product experiment? The one where they can ask conformity tests, right? Maybe find it if that replicated it with the language models as well. That would be interesting. With the rest of the research that happens at big labs, how much of it is increasing the, or decreasing the amount of compute you need to get a certain result. There's an actual compute multiplier versus how much of it is things is that they're just making the learning more stable and just building out of the infrastructure.

1:19:00I guess the broader question I'm gonna try and ask is since GPT -4 does it feel like with the same amount of compute, you can train a much better model or does it feel like, oh, we've made sure that the learning can happen better and in a more scalable way would GPT -5, but it's not like we can train GPT -4 with GPT 3 .5 budget now or something like that. Yeah, well, definitely there's always progress in improving the efficiency. see, whenever you have a 1D performance metric, you're going to find that different improvements can kind of substitute for each other. So you might find that you post -training and pre -training both improve the metrics or improve, they'll have a slightly different profile of which metrics they improve.

1:19:48but if at the end of the day you have a single number, they're going to substitute for each other's somewhat. So I would say for something like a human evaluation, what are humans prefer? We've definitely made a lot of progress on both sides and pre -training and post -training and improving that. A couple of rapid -fire questions about RLHF. So obviously, RLHF is important to make these models useful. So maybe the lobotomized description is inaccurate. But there is a sense in which all of these models, once they're put in a chat platform, have a very similar way of speaking. They really want to delve into things.

1:20:31They want to turn things into bullet points. They often seem sort of have this formal and dull way of speaking. And there's complaints that they're not as creative, like we're talking about before with it can only do rhyming poetry and not rhyming until recently, I guess. Is that a result of the particular way in which RLHF happens now? And if so, like, is it because of who the Raiders are? Is it because of what the loss function is? Why is this the way all chat bots look? Yeah, I would say there's a decent amount of room for variation in exactly how you do the training process. And I think we have a lot of, I'd say we're actively trying to improve this and make the writing more lively and more fun.

1:21:13And I think we've made some progress like improving the personality of chat GBT. So it is more fun and like it's better when you're trying to chitchat with it and so forth. It's less robotic. I would say, yes, it's a kind of interesting question how some of the ticks came about like the word Delv. I've actually caught myself using the word of it recently. I don't know if it rubbed off on me from the model or what. But actually, I think there might be some funny effects going on where there's unintentional distillation happening between the language model providers where if you hire someone to go do a labeling task, they might just be feeding it into a model.

1:22:04They might just be pulling up their favorite chat pod and feeding it in and having the model do the task and then copying face to the back. So there might be that might account for some of the convergence, but also I think some of the things we're seeing are just what what people like. I mean, I think people do like bullet points. They like the structured responses. People do often like the big info dumps that they get from the models. So yeah, I think there's So it's not completely clear how much is just a quirk of the particular choices and design of the post -training processes and how much is actually intrinsic to what people actually want.

1:22:56It does seem persistently more verbose than some people want, and maybe just because is during the labeling stage, the Raiders will prefer the more robust answer. But I wonder if it's inherent to, because of the how it's free trading, the stop sequence doesn't come up that often, and it really wants to just keep going. There might be some biases in the labeling that lead to verbosity, like the fact that we tend to train for one message at a time rather than the full interaction. So, like, if you only see one message, then there's something that just has like a clarifying question or maybe a short response with an invitation to follow up is going to be, it's going to look less complete than something that covers all possibilities.

1:23:44There's also a question of what people, whether people's preferences would change depending on how fast the model is streaming its output. it. Like clearly if you're sitting there waiting for it, waiting for the tokens to come out, you're going to prefer that it gets to the point. But if it just gives you a dump of text instantly, maybe you don't actually care if there's a bunch of boilerplate or like if there's a bunch of stuff you're in a scam, you'd rather just have it all there. Yeah. The reward model is, I think such an interesting artifact because it's the closest thing we have to an aggregation of what people want, what preferences they have.

1:24:27When you think about models that are much smarter, the kind of way in which will, I mean, one hope would be that you could just give a sort of like list of things we want that are not a sort of trivial and obvious kinds of like you and declaration of rights things. On the other hand, I think I heard you make the point that well a lot of our preferences and values are very subtle and so that they might be best represented through these pairwise preferences. When you think of a GBD -6 or a GBD -7 level model, are we giving it more of like a written instructions or are we still doing which kind you know these sorts of like sublobe and old preferences?

1:25:10Yeah that's that's a good question. So I think like these preference models do learn a lot of subtleties about what people prefer that would be hard to articulate in an instruction manual. Maybe if you, obviously you can write an instruction manual that has lots of examples of comparisons. And that's what the ModelSpec has. It has a lot of examples with some explanation.

1:25:45So it's not clear what the optimal format is for describing preferences. I would guess that whatever you can get out of like a big data set that captures fuzzy preferences, you can distill it down to a smaller, a shorter document that mostly captures the ideas. And I would think that the bigger models are, like they do learn a lot of these concepts automatically of what people might find. Like they'll have some, they'll just learn from all the pre -training data what people would find useful and helpful and what they'll have, like some, there'll be some complex, like moral theories that they can they have.

1:26:37But of course, there's still a lot of room to latch onto a different style or a different morality. So I think when we have, if we were to write a doc, or if we're gonna align these models, what we're doing is latching onto a specific style, a specific morality, and there's still a decent, you still need a decently long document to capture exactly what you want. Yeah. How much of a mode is better post -training? Currently companies I distinguish themselves by how big are our model and so forth. Will it be a big mode who has figured out all the finickyness that you were talking about earlier with regards to all those data?

1:27:23I think there's something of a mode because it's just a very complex operation and there's, so it takes, you have to have a lot of skilled people doing it. So there's a lot of tacit knowledge and there's a lot of organizational knowledge that's required. So I think post -training, to create a model that actually has all the functionality people care about is pretty complicated. It requires a pretty complicated effort. And this requires a lot of, this is basically an accumulation of a lot of R &D. So I would say that makes it somewhat of a mode that's not trivial to spin this up immediately. It does seem like the same companies that are putting together the most serious pre -training efforts are also putting together the serious post -training efforts.

1:28:27So it seems like it is somewhat possible to copy or to spin up more of these efforts. There's also like one force that sort of makes it less of a mode is that you can like distill the models or you can take someone else's model and clone the outputs or you can use someone else's model as a judge to do comparisons. So I think the more big league people probably aren't doing that because it goes against terms of service policies, but it would also be sort of hit to their pride, but I would expect some of the smaller players are doing that to get off the ground. And that catches you up to a large extent.

1:29:15I guess that was really the mode. What is the median rate or like? Where are the what is their sort of knowledge level? I would say it's, it varies a lot. So we've definitely hired Raiders with different skills or for different kinds of tasks or projects. So I would say like a decent a decent mental model is just look at people who are on upwork and other platforms like that. Like who's doing sort of odd jobs with remote work. So it's a pretty international group. There's a decent number of people in the US. We hire different people, like different groups of people for different types of labeling, like whether we're more focused on writing or like STEM tasks.

1:30:13So people doing STEM tasks are more likely to be in India or other sort of like middle or lower middle income countries, whereas people doing more like English writing and composition tend more to be like US based. So yeah, and I'd say there have been times when we needed to hire different experts for some of our campaigns. Some of the people are very, some of them are very talented and like we even and find that they're at least as good as us, the researchers at doing these tasks, and they're much more careful than us. So I would say the people we have now are quite skilled and conscientious.

1:30:59With regards to the sort of plateau narrative, one of the things I've heard is that a lot of the abilities these models have to help you with specific things is related to the having very closely matched labels within the supervised fine -tuning data set. Is that true? If it can teach me how to use FFMPEC correctly, there's somebody who's doing, figuring out, seeing the inputs and seeing what flags you need to add, and some human is figuring that out and smashing to that. Do you need to hire all these label who have domain sexualities in all these different domains. Because if that's the case, it seems they could be a much bigger slog to get these models to be smarter and smarter over time.

1:31:46Right, you don't exactly need that, because yeah, you can get quite a bit out of generalization. So if you, like the base model has already been trained on tons of documentation, tons of code with shell scripts and so forth. So it's already seen all the FFM pegman pages and lots of bash scripts and everything and it's so like the base even just giving the base model a good few shop prompts you can get it to answer queries like this and Just training a preference model like for helpfulness will Even if you don't train it on them probably even if you don't train it on any stem, it'll somewhat generalize to stem.

1:32:34And like, not only do you not need like examples of how to use FFM tech, you might not even need anything with programming to get some reasonable behavior in programming domain. Maybe final question is we've touched on this in different ways, but to put it together. So you say you're turning on much more multimodal data, presumably these things understand what screens look like and we'll be interacting with in a much more coherent way. And also you're going to do this along horizon RL so they'll be able to act as agents in the systems who can be part of your workflow in a much more integrated way.

1:33:19What do you expect that to look like? And it will be the next steps from there. So suppose by the end of the year or next year you have something that's like an assistant who can work with you on your screen. Does that seem like a first of all a sensible thing to expect and then where does it go from there? I would definitely, yeah, I would expect things to move in that direction. It's unclear what's gonna be the best form factor whether it's like something that's, it's like a clippy that's on your computer and helping you with something or if it's more like a like helpful colleague in the clouds.

1:33:53So we'll see which kinds of form factors work the best. And I would expect people to try all of them out. Yeah, I would expect more like, yeah, I would expect something like a, yeah, the mental model of a helpful assistant or helpful colleague to become more real, where you can share more of your everyday work or have it, like instead of just giving it one -off queries, you would have a whole project that you're doing and it knows about everything you've done on that project so far. You can tell it, it can even proactively make suggestions, like maybe you can tell it, Oh yeah, remember to ask me about this and if I've made any progress on it.

1:34:40So I think like pro activity is one thing that's been missing. Yeah, I'd really love to see better, a more like moving away from sort of one -off queries, like using the model kind of like a search engine. As part of search engine and more towards like having a whole project that I'm like doing in collaboration with the model. and it knows everything I've done. It's proactively like suggesting things for me to try or it's going and doing work in the background. Yeah, that's really interesting. What about, it's a final question. What is your, what is your median timeline? You have a replacement of your job.

1:35:20Yeah, a replacement of my job. Maybe like five years. Yeah, pretty soon. Yeah, interesting. Okay, well John, this is super interesting. Yeah, thanks so much for making the time. I think this seems like one of the parts of the AI process that are super important and people don't understand that much about it. It was super interesting to delve into it. Yeah, that's what I was talking about. But yeah, thanks for having me on the podcast. Who's fun to talk about all this stuff. Hey, everybody. I hope you enjoyed that episode, the John. He's just a very thoughtful guy. And it's super interesting to learn about the way in which these models become the kind of shock that they are.

1:36:03Anyways, as you can see, I'm now doing ads on the podcast. So if you'd like to advertise, you can reach out at the link in the description. And of course, if you enjoyed the episode, it's really helpful. If you can share it with other people who you think might enjoy it. Your friends, group chats, Twitter, whatever else. See you on the next one. Cheers. Cheers!

1:36:28Bippertoss Ship