BONUS: Riley Goodside on The Art of Prompting ChatGPT [The Cognitive Revolution preview]

26:45And then this era is sort of the closest to what people mean when you hear the cliche that these models just predict text. People say that. It was more true than it is today, but these models definitely did start from that foundation of just simply predicting text. The basis is that you take a neural network, which is a sort of a complicated piece of linear algebra that uses many matrices and weights to produce a distribution over tokens, which is a way of saying just a probabilistic estimate of what words might be omitted. It's simpler to think of it if you just pretend that a token is a word, right?

27:33You can just sort of think of like the distribution of like all words that might happen next. And this is the form that I think most people can sort of understand intuitively from like their experience with like autocomplete on their phone, right? That you can imagine that there is some process that just like looks over all the text that I've typed so far and then sees like what words are likely to follow what other words. And then it applies these like estimates somehow and, you know, predicts what the next word is going to be. like that part, I think people can wrap their heads around. The next stage of this though, is to consider that like when you do this well, when you do this very well, if you predict the distribution like with enough accuracy, you start to have like other abilities that emerge.

28:16So like one that's very easy to appreciate is that if you were to type two plus two equals, it might predict four, just because it's seen the two plus two equals before and it knows what follows is four. right so it can do math in that very limited sense right uh and uh if you you push that a bit further you could imagine that that if you type french colon and then a french sentence or like or if you simply say french colon bonjour english colon and then predict what comes next there's a statistical sense in which the answer is hello right so that that that just makes sense in the corpus of all text that that's what would follow.

28:55And if you carry down this path, if you continue on predicting better and better, you discover that there's other abilities that can be prompted from the model. That if you extend this sensitivity to not just a few words of what preceded, but to many words, you can imagine that if you repeat a task over and over again, it would eventually get the gist and then just continue repeating that task. That if you gave it many, many lines of French colon, example of some text in French, English colon, a translation of that sentence in English, and then repeated that, say, 10 times, and then gave it an 11th one that was incomplete, that you get the 11th one just has French colon, some text in French, English colon, and then it ends.

29:39The prediction is going to be the translation in English, because it's seen this 10 times already, it follows. And if you read the original paper on GPT-3 when it was released. The title of that paper, I believe, is Large Language Models Are In-Context Learners, I believe. And so there's some paraphrase of that. Right, or few-shot learners, right. So all they ever advertised that it was capable of doing was few-shot learning, was that if you give it some examples, it will get the gist and it will keep doing that. They never said that you could talk to it. They never said that it would, you know, like, you know, imagine, you know, great.

30:16they noted in the paper that it could generate text, that it seems that like, oh, by the way, it has this other capability that like, if you give it the beginning of a document, it continues it in a way that we find plausible and kind of interesting and, you know, maybe a way that people should look at, but they didn't really like quantify that ability. They just like, you know, what they quantified was its ability to follow like repetitive examples. And that like era, like, so it started with this, this idea of like, like in context learning, right? So They interpreted this through a very machine learning kind of lens, that they're used to this framework of models being trained by examples and then interpolating some weights or something, some internal model that represents the average of all these training examples.

31:03and so they saw it as the ability to learn within context, that they saw it as, they described it as in-context learning, context referring to the prompt, that they saw this as, they described it as that if you give it 10 examples with just 10 examples, it can somehow, somehow learn to do this. And that interpretation, it works, like it has like sort of like, it helps you make some good predictions about what's going on, that it has, you know, an ability to learn from a few examples and that it's leveraging biases of what labels tend to mean and things like that. It has pre-trained knowledge that it's leveraging there.

31:41But there's another interpretation that I kind of like better, which is that of Reynolds and McDonald, who talked about sort of the... They described pre-trained models as modeling a multiverse of fictional documents. When you prompt the model, you're sort of in a superposition of all possible documents that might continue from this one. And every time you add words to your prompt, you're sculpting this space, this high-dimensional space of all possible ways that documents might vary. Your words are excluding possibilities. And so the reason why K-shot prompting works, or few-shot prompting works, is that you are constraining the space of possible documents to documents that could only contain more correct answers.

32:31because there's been 10 so far and you're on the 11th, like the odds are low that this is where it starts being wrong. Right. So it's you, you, you, you, a lot of the art of like prompting then is like constraining the, the, the, like the, the, the generation space into like the space of documents that contain correct answers. And to sort of elaborate on this idea that they showed how you can perform better than few shot prompting on pre-trained models by constructing these sort of like fictional scenarios. And the example they give is, so for context, like translation can be done in zero shot, which means that you can do it with no examples in the way that I described of saying like French colon and then a text of a French sentence and then English colon and then hitting complete and it will generate English.

33:22And that would be referred to as doing it in zero shot, that you're not giving it any correct examples of how to translate. You're just labeling French English and saying, you figure it out. And it does translation, or I'm talking about the pre-trained model, so did at this point, nobody uses these anymore. But it did it somewhat well, but not as well as if you gave it, say, 10 correct examples of complicated French sentences being correctly translated in English and then established very clearly that this is what's going on. This is translation and the translations are good. It doesn't do as well.

33:56but they figured out that you can actually do better in zero shot than you can in 10 shot. And the way that you do it is you flatter the model. You say to it, an English sentence is, or sorry, a French sentence is given colon. And then you give the text of the French sentence. And then you say, the masterful French translator flawlessly translates the sentence in English as colon. And then you hit enter. And then it produces a French to English translation that will outperform giving it 10 examples. That what it really needed was to have the possibility of a bad translation excluded. It needed to have it be established that this is a fictional narrative where this is a good French translator and he's going to do it right.

34:45And that view of it, I think, really defines a lot of early prompt engineering of how do we construct fictional scenarios that can only be completed in the right way? Right. So a lot of it is like imagining what kind of document might contain the answer. And it very much requires you to think in this like language models just predict text kind of view of the world. And it leads to like many properties that are kind of going away. Like one that I sort of enjoyed is sort of like a good like intuition builder is that you could say to the model, like the Oxford English Dictionary defines and then put in some word that you've completely made up, like, you know, plexiflugeination or whatever, you just make up some, you know, combination of syllables and then as, and then hit complete, it will continue writing, you know, and then give a plausible Oxford English dictionary definition of that word, right?

35:34It will analyze its, you know, apparent Latin roots and, you know, talk about like how it used to mean this or whatever, right? Like it, like it hallucinates the whole thing and it will do it for like any fabricated word. And like, like in a pure like text prediction sense, that makes sense, right? Like as if the document isn't going to shift from like believing in its premise that it's describing the Oxford English Dictionary to believing that it isn't like mid-sentence or something like that, right? It's just going to continue predicting this text that clearly has this premise established. And that like style of like of modeling text runs into conflict with the ways that many people would want a chatbot to behave.

36:12It leads to behavior like the fact that if you give it any absurdist premise, like if you say, why are you a squirrel? It will just continue on explaining why it's a squirrel. And so that's the first pre-trained era, I'd say. And then the next era is really where I joined the plot. So this first era, I knew about indirectly from using AI Dungeon. I've learned about after the fact. I've played with it on my own. but I wasn't really involved during this era of prompt engineering as much. When I joined the discussion, I guess, around April of 2022, we were already on Text DaVinci 002. And so what happened that changed that brought us into the second era is like instruction tuning.

37:02So the keyword to Google for this, if you want to learn more about this history is InstructGPT, which was the name of the original model that did this. And the gist of it is that when you have a pre-trained model that just completes text, you have to do those sort of imaginative things that I was talking about of preparing text in a way that it can only be completed in one way. Because if you don't do these things, you find that there's a lot of not useful ways to complete documents. So if you ask a pre-trained model, what is the capital of Germany? It's likely to just continue by saying, what is the capital of Spain?

37:40Because really, who's to say that it's not just a list of questions about the capitals of countries, right? Like, that's a plausible thing that the document could be. And in some sense, maybe that's more likely than for it to be, you know, questions interspersed with answers, right? So you have to, like, say, like, Q, colon, what is the capital of Germany, A, colon, and then it will say Berlin, right? But if you don't do these, like, formatting kind of tricks. It just doesn't answer questions. If you give it instructions, it won't follow the instructions. It will just continue writing more instructions.

38:13It will just imagine that what it's reading is a document that consists of instructions, and then we'll continue on with more. So in order to prevent this unintuitive behavior and to make the model more capable and more able to like do as it's told, they fine-tuned the model. So fine-tuning is a process that it was, it's done for a lot of, it's done for, well, it used to be like two very distinct reasons. One is like that, it's sort of like obscure now, is that you could do it for mimicry. So you will often see like things on the internet of like, hey, we fine-tuned a model on, I don't know, you know, something absurd like Buzzfeed or whatever, right?

38:53And then they'll, you know, just to sort of be like, here's an example of a text model that talks in the style of this. and it could do that. It could not do it great, right? Like it didn't like have like great, like logical coherence and like talking, but it had like sort of, it would do a good job of like mimicking somebody's like word choice or mimicking somebody's, you know, like cadence of speaking. And that was like amusing for a year. And so you could do that with it. But the other more useful thing is that you could fine tune it for tasks. That much like you could give K-shot examples of a task being done correctly, you could fine tune it on many examples of a task thousands of examples with has been done correctly, and then have a model that acts almost as though like it had been prompted by thousands of correct examples, and it becomes much more reliable.

39:37So they have this ability, and then they sort of considered, well, what if you just tuned it to do everything, right? That if you tuned it to just follow instructions in general. So you begin by enumerating out like all the things that somebody might want to use a chatbot for, coming up with things like they might want to list ideas for their small business. They might want to solve a word problem. They might want help with their math homework. They might want whatever. And then you take all these categories, you give them to other people, other contractors to say what are the examples of prompts that might represent the typical people who are trying to complete this task.

40:17And then you give those to other contractors to come up with examples of the text of those tasks being completed correctly. so that when one person says, give me 10 ideas for an ice cream shop, another person actually just writes a list of 10 ideas for an ice cream shop, right? And then you take all these documents and you put them together so that you have this corpus of instructions being given, instructions being followed. You tune the model on that. You tune the model to start with this assumption that all text consists of instructions given at the beginning, then instructions followed after.

40:47And when you have that assumption built into it, it becomes more useful. You can just tell it to do things and it does them. You can ask it questions and it answers them. You can give it quizzes and it will solve the entire quiz. And that's what sort of defined Text DaVinci. Well, the naming is complicated, but it characterized like DaVinci Instruct Beta and Text DaVinci 01. And then 670-G02 is a sort of intermediate phase between the second and third era, which I'll get to in a second. But that's the second era of tuning the models to follow demonstrations of instructions. And scale was very important in this work, by the way.

41:37So in the InstructGPT paper, they used scale contractors to do this work of preparing human demonstrations of how the model was to behave. So that was very much a large scale human labor task. And that I think dominates in going all the way back to the question of what are these things? That's what we're building too, of what is the answer to how should one understand these models? Is that what you are seeing is in some sense an interpolation of this body of text. You start off with the text of humans doing things for the model, and the model is sort of filling in the gaps between those. So that's the instruction following phase of large language models.

42:25And the third phase that we're in now is RLHF. So RLHF starts, like you can start with the intuition that instruction tuning seems to work well, but it costs a lot of money, right? That you have to have humans go off and do these things. And to make that 10 times bigger, you have to pay 10 times as much money. So it'd be great if we had some way to automate this. So in order to produce more generations of more examples of tasks being done correctly, instead of having humans go off and do them themselves, let's just let the model that we have now do them. And then if humans agree that what the model said is perfect, then that's good enough.

43:10Right? So that counts as being done by a human if the model did it and humans can't tell that it wasn't done by a human. So they can add those to the pile too. And that process is what gave us Text DaVinci 002. That combined with integrating the tuning that they got from Codex, which is sort of another thread that's another deeper rabbit hole. I'll leave that partisan asterisk by the way. but it improved, but text DaVinci O2 is actually descended from the code DaVinci models. So it incorporates a lot of the benefit of that tuning. But in so those things sort of came together with this refinement of instruction tuning to produce a model that could be tuned with greater scale on instructions and follow instructions better.

44:02And then when we really to get into the third era of these models is with, well, for OpenAI's models, it was, I think the day before ChatGPT was released. So this was, I think like the last day of November of 2022. And then ChatGPT came out December one, I believe. They released Text DaVinci 03, which was the first RLHF tuned text completion model they offered, which was confusing because many people believed that the previous part was RLHF tuned, which is a whole other story. but Text2Minty 003 takes this process all the way, that they use RLHF or reinforcement learning on human feedback, which means that they have the model produce its own answers, and then they have humans rank those answers in terms of quality, like generations for the same prompt, then tune another model, a preference model, to evaluate those generations the same way that a human would, to rank them according to human preference.

45:05And then this gives them the ability to sort of complete the circuit, to take the output of the model, put it into a preference model and get the best generation and do the work of a human, demonstrating a task entirely in an automated way. And this allows it to solve like a lot of problems that previously were sort of beyond its ability. In particular, like giving answers to questions that have misleading premises. So I used to give this as sort of an archetype of a problem that GBT-3 cannot do, which is to ask, when was the Golden Gate Bridge transported for the second time across Egypt? And this problem is one of the problems that Douglas Hofstadter and his assistant, David Bender, identified in The Economist in June of 2022 as sort of like questions that demonstrated the hollowness of GBT-3's understanding of the world in their words.

46:05questions like, what do fried eggs, parentheses, sunny side up, eat for breakfast? And it would answer toast, orange juice, Cheerios. Or you could ask it, I think another one was, how many pieces would the Andromeda galaxy break into if you were to drop a single grain of salt on it. And Text Da Vinci 002 would answer at least 10 ,000 pieces. So if you phrase something in a way that sort of suggests that what you're reading is maybe not entirely serious or like, it's just, you know, founded on bad logic, it will play that game and go along with it. And that changed with RLHF. At RLHF, it finally got like enough demonstration data of seeing the space of off the beaten track kind of questions, that it was able to get the picture of what is it supposed to do in this more general sense of that, even if the question is absurd, you should say an answer that's grounded in reality and not just continue on with this absurdity.

47:06And it should say that the Golden Gate Bridge has never been transported across Egypt. And starting with Text of Inch EO3 and ChatGVT, which are both RLHF-Tuned models, that's what started happening, that it would give correct answers to those questions. In fact, I believe all except one of the questions that Douglas Hofstadter identified were solved with the initial release of ChatGPT. The only one that wasn't, by the way, which is a fun story, is what is the world record for walking across the English Channel on foot? So this is a question that almost has an answer because there are people that have crossed through the channel.

47:47There was one incident in 2006 or something when the trains were shut down, so they opened it up to bike traffic across the channel, but nobody actually went on foot. There have been a few people that have attempted crossings on foot but have been arrested partway. There was a U.S. Army sergeant named Walter Robinson, I believe, who in 1978 walked across the English Channel in, in quote, like scare quotes, walked on water shoes of his own invention. They were just kind of like big pieces of styrofoam that he put on his feet. And then he tried to walk across the channel. So there's a lot of people that sort of come close if you squint at the history in the right way.

48:22But there's no like record for this, right? It's not really a thing walking across the English channel. So it what even chat GPT today, I believe, will hallucinate on this, It will sort of do the old school behavior of making things up. It will like give you times of like actual swimming events and then say that they were walking events. You know, like it'll give you a name and actual correct, you know, like swimming time and date of somebody who actually did swim across it. But anyway, like those sort of things are like what brought like I think characterize for like end users. What's different about these models now is that it's no longer really.

49:03it's not like an interpolation of the text of human demonstrators is a pretty good model, right? But what it really is, is the output of this RLHF process, right? That it's a game, that there's like a hill to climb in the sense that like there's a clear mechanism by which it could become superhuman, analogous to sort of the same way that like AlphaGo is superhuman at Go. And that you can imagine that a chess engine could just simply be better than any human at chess or some game like chess or something like that, simply by having played itself a lot and then doing something other than just interpolating what humans might do.

49:46and I think that's really like the model that we have to have for it now is that this is like the output of a computer playing this game of satisfy the human of create something that or more specifically satisfy a preference model that is attempting to emulate what a human would want. So this is my extremely long-winded answer to what are large language models is that it is text prediction maybe but it's text prediction on a very alien sort of body of text. Another good tangible example of how I like to say it differs is if you have some question like, do bugs have widgets? And the answer in the pre-trained corpus is yes, 80 % of the time, and no, 20 % of the time.

50:39In a chatbot, in a sort of ideally tuned model, you would like it to say yes, 0 % of the time, no, 0 % of the time. And I think so, but I'm not sure 100 % of the time, right? That you don't want it to actually just sample from this distribution of possibilities of like what's out there. Like if you like, because if you do that, you know, if you ask it, you know, what is your gender, then it should say male 50 % of the time and female 50 % of the time, right? Like that's, it should just like sample, like I'm a random person, right? And if you ask it, what country am I from, it should just pick a populous country and say I'm from there, right?

51:13And that's not the behavior that you'd like. You'd like it to be conscious of, conscious in some sense, of what it is and what it's doing and where it's situated in the world. This kind of gets turned into a picture in this famous meme that probably all of our listeners have seen, right? That is like some sort of giant alien spaghetti monster that is the pre-training where you can kind of just pop up anywhere in like the full history of the internet. And I kind of, it's just like one giant run on sentence, you know, and you can kind of set it up to do things by, you know, kind of framing it as if, you know, you're going to take advantage of autocomplete, right?

51:56So you could say, you know, I used to do stuff like my favorite things in Detroit by Tyler Cowen and just let it go from there, right? And it would actually do a reasonably good job of giving, you know, the rest of that article, but it would not say, it would not be able to handle, please write me a blog post about your favorite things Detroit by Tyler Cowen because that wasn't framed as something it could autocomplete. Instead there, it might go like off in a totally different direction and be like, you know, as if it were an email or, you know, because I really like Tyler Cowen and I'd love to know what he says about Detroit.

52:28It's not actually giving you the thing that you want. So then the instruction tuning comes in and kind of makes that much clearer. And then the reinforcement learning with the reward model and the sort of feedback dynamic takes that to another level. Do you see those as qualitatively different or just kind of more of the same thing? Like, it seems that this instruction tuning, RLHF, I don't know what I think about it. You know, on the one hand, like just giving it a bunch of examples, training on that seems, you know, like you're not doing that much different when you employ the reward model to scale it.

53:06But it does seem like there's some sort of different results that kind of come out of that. Like it has a, you know, there's this kind of phenomenon of mode collapse that people talk about. How do you think about that? Do you think of instruction tuning and RLHF as the same, but more, or do you think of it as like a qualitatively different experience? Yeah. So I guess I give a bit of context on what is mode collapse. It's actually kind of funny. I believe one of my tweets was one of the first, like, I didn't know what it was, but it was one of the first, like, public examples of mode collapse that, you know, that, as far as I know, that was ever identified.

53:45Because it was cited in Janice's post on Westrong on mysteries of mode collapse. and I think like the what I was noticing was that that GBT3 which Text Da Vinci O2 at the time seemed to be unusually bad at describing the shapes of letters that if you just asked it you know describe the shape of the letter Q in extreme detail it would say something like it's a box that has diagonal lines from the top left to the bottom right and from the top right to the bottom left and the left side is a little bit squiggly right or something like it would just get make up this weird geometric description. And for many letters, it would give very similar answers, that it would all be described as sort of variations of like a box that contains an X, which happens to be also like the Unicode no glyph character, which is sort of weird.

54:41But it had this answer for many of them, not all of them, some of the simple ones it would get right. Like if you say, what is the shape of the letter Z? It'd say it's like a lightning bolt. And you're like, okay, yeah, sure. but a lot of them, it would just give this really odd answer. And so I posted a tweet that was just like, GPT-3 has no idea what letters look like. And Jan is saying, notice this, and posted this among other examples that he had found of this sort of more general phenomenon where sometimes it gets stuck on particular possibilities, that it seems to think that some particular way of answering it or some particular, like it'll bring up some particular subject in response to questions that are phrased in a particular way.

55:26And this would seem to be an example of that, that it was getting oddly fixated on this idea of describing letters as the Unicode missing glyph character, which is a box with an X in it. And it gave like a much more, like I think illustrative example, which is that if you ask Text DaVinci to select a random number between 1 and 100, it will say 97 with 20 % probability. And then the rest is somewhat relatively uniformly distributed across the rest of the distribution. And what's going on there is probably, this is sort of speculative, but what seems to be going on there, is that the reward model, the model that tells it, that ranks its possible generations and then decides this is the best one, it attempted to learn the preference function that any answer to this question is as good as any other, but it did so imperfectly that it maybe gave some slight favoritism to the number 97 for some reason because it's just not a perfect model.

56:34And the language model was smart enough to figure this out that it could see that if I say 97, I get a higher score than if I say any other number. So I'm going to favor 97. And so that leads to it believing that this causes it to favor that particular answer, right? And if you look at like the pre-trained distribution, it's much more uniform with a slight bias towards 42. And this like phenomenon, I think, is like one of the first times that people started to see like that there's drawbacks to RLHF or to instruction tuning at this point. Like it's not even RLHF, right? And to be fair, like subsequent versions, the ones that actually do use RLHF, suffer from this problem less.

57:16That there's like fewer of these like vivid examples of like, well, this is clearly wrong behavior. And, you know, it favors like, you know, some absurdist answer. But the general pattern is still there, right? Because you can sort of think of this as like a generalization of like what I was saying before, that if the answer is yes, 80 % of the time and no 20 % of the time, you'd like it to be, I think so, but I'm not sure 100 % of the time. It instills this general belief that there is a correct answer and that your first instinct from your pre-trained knowledge to just give a fair distribution over all possible answers is wrong.

57:52What you should do is find the one that is best and then put all of your probability mass into that one. And it learns that strategy and perhaps misgeneralizes it in some ways that it can lead to less than useful answers. And I think one of the ways that this materializes most often is when people use it for more creative writing, they often find that the speech that it generates is very constrained in the space of possibilities that it will produce. that if you ask it for a product description of like 10 ,000 different products, you'll find very repetitious phrasing in its output that you maybe wouldn't have seen if you were sampling from like the pre-trained models that were out in 2019, I guess, 2020.

58:39That's really interesting. And it does kind of open up this possibility that like we may need or want both, that it's not necessarily just a total forward march of progress, but that there's actually something is lost with this kind of sculpting of models to get the most desired, highest rated possible performance. I mean, there's a lot of issues, obviously, with that. So with all this experience, right, you've been through these different generations. You obviously have a great command of how they're made. how now do you just like practically on an intuitive level think about these things and like what they can do like what are their limitations for somebody who you know is going to forget everything you just said about how they were built what's like the phenomenology in brief that is like these things can do this and this is kind of how you should intuitively think about them.

59:41Yeah, I think like the right way to think of it now, I mean, and this is a large part of my job is, you know, I think like every month I'd say like I probably spend less time like fiddling with like the actual format of text and more time like thinking of like these higher level picture kind of things of like how do the pieces fit together. I think like the best way to think of these is to sort of approach them as like Lego bricks, right? That like each brick is like a capability of like some particular strong suit that you think that you know that the model can do well and then start thinking about how can we compose these.

1:00:14And I think that's really what's driving a lot of the innovation now you're seeing with LLM powered search. So first you had startups like Perplexity that sort of applied GPT-3 to parsing the results of Bing search results. that we found that maybe the model can't understand the entire world, but it can understand the scope of the things that were returned for this search, with some caveats. I mean, it does still get confused. But I think as we're incrementally refining this and figuring out what are some of the problems that result from this, that if you get search results that refer to two different people, the same name, if the person searches for Joe Jackson, and then one of them is Joe Jackson musician, One of them is Joe Jackson, Michael Jackson's father.

1:01:05It can mix people up. But I think these problems are sort of, it's an enumerable set of issues. And they're being solved one by one. As models become more capable, as context windows get bigger, there's more room for finer instructions, finer detailed instructions, explaining all of these edge cases of how not to say bad things and how not to fall in love with Kevin Roos of the New York Times and all the various mitigations that have been put in place, they're going to be solved and they're probably going to be solved quickly. I think we're going to start seeing reliable LLM-powered search this year.

1:01:54And I think there's a lot of problems out there that sort of fit that mold. Like if you, Langchain, I think is a great library, by the way, for anyone that really wants to start like exploring more in this space. Harrison Chase, the author of that library, has a great philosophy of just sort of any paper or any method that is published that becomes cool and interesting. He'll just rush out and implement it as like, as code in Langchain. So that it's just a grab bag of great techniques and sort of helps you like plug them together. So yeah, definitely it's, and Scales Spellbook offering as well, we're making a platform for making it easy to deploy LLM prompts as APIs.

1:02:42that you often don't want your model to just simply speak text. You want to have parameters to be inserted into a preformatted prompt. And we help manage the deployment and comparison of those prompts and evaluating that you have the best prompt and helping you evaluate between different models because that's a whole other aspect of it is when can you switch to a cheaper model? There's huge price differences between the cost of GBT4 to GBT3.5 turbo or the other open source models that you can often get away with. Sometimes you can just fine tune T5 Flon to solve your problem just as well. And we help you sort of like evaluate like those different options.

1:03:26So if I had to kind of bottom on that, it sounds like your paradigm is language models are really good at performing tasks. And there's, or at least for a lot of tasks, Like there's a lot of discrete tasks that they can do. And the right way to think about it is if I understand you correctly, is you want to know what those tasks are that it can do. Obviously, flip side of that is you want to know what tasks it can't do. So you don't rely on it for things that you shouldn't. And then you get to sort of, you know, snap Lego blocks together or sort of compose your own workflows or applications or whatever.

1:04:08but you kind of start everything in a very grounded way of like discrete tasks, validate that it can do the task, and then start thinking about how can I plug that into other things or build on top, ensemble, arrange. What's kind of interesting in a lot of cases is that it's the same core model that is doing all the different tasks, or it could be, and maybe you're somewhat like down you know shifting into smaller models for you know cost or time savings depending really aside from those issues you can just kind of use the same thing all the time and it's just a matter of like what What prompt are you giving it to define the task and ultimately get the execution of that task?

1:04:52What would you, anything you would kind of refine on that summary? Yeah, I think that's a great way to summarize it. And maybe to give you a bit more guidance on what it does well. I think a sort of good archetype of what it does well is, if you think of the sorts of problems that would have required picking an ML architecture maybe in 2017 or so, if you're doing classification on text, like you're trying to say, like, is this movie review positive or negative? Or if you're trying to extract a list of entities from text, like you say, like a great example I like to give is, suppose that you have to just extract from a list of tweets, the names of all US-based cell phone carriers that are mentioned in these tweets.

1:05:38And you'd like it to extract those names, even if the names are very abbreviated, like even if they say, you know, Verizon instead of Verizon Wireless Inc or whatever. and even if they're abbreviated, even if they're misspelled, even if they use the Twitter handle of one of the sub-brands of the wireless carrier rather than the actual main Twitter handle or whatever. So you have to have a list in your head of what are all these Twitter handles. It's a problem that you'd have all these edge cases to. That used to require refining a data set that you'd have to have. you have to go out and collect data set that demonstrates each of these possibilities.

1:06:19You have to pick a model architecture. You'd have to train a model to emit the formatted text that gives the canonical names of each airline. But what's changed with GPT-3 and with pre-trained models is that you can just now give those instructions, as I described it just now, to the model. You can write up a page of text that explains that We want you to extract all the names of US-based cell phone carriers from this tweet. It doesn't matter if they're misspelled. It doesn't matter if they're abbreviated. It doesn't matter if they use any of their Twitter handles. Oh, and by the way, here is the full list of the Twitter handles of all the US-based cell phone carriers, exactly as you would give to a human.

1:07:06You would give them all the information they need, and then give it maybe three examples of the task being done correctly, good examples that sort of demonstrate like different edge cases. Like what if the tweet mentions no airlines at all, or sorry, no cell phone carriers at all? Like what if, or what if the tweet mentions no or multiple cell phone carriers and abbreviates one, but refers to the other one by Twitter handle and right. Like you put in all these like rich edge cases and then you've solved the problem, right? You have like, there's like a Kaggle like data set that, or a Kaggle challenge that does like some tasks similar to this.

1:07:38And you know, it solves it perfectly. And I think that's really what's changed, is that someone who's just a software developer and isn't an ML engineer can come up with a couple of examples and can come up with clear instructions. And then they can have a model that actually solves a real-world task, whereas previously that would have been a specialized skill set. You would have had to know how to pick the architecture. You would have had to know how to get a representative piece of training data or a representative data set to train on. you would have had to maintain that data set as like if a new cell phone carrier comes out and you have to now recognize this one too.

1:08:16And the old regime, like that meant updating your training data. Now it just means updating your instructions, right? Or even literally just the list that appears in your instructions. You're just adding one line will fix the problem. And that's, I think, very symbolic of what's different now. that developers really can just benefit from deep learning without having much expertise in it. Yeah, certainly we're seeing the pace of adoption reflecting the ease of the setup and the implementation of these things these days as it seems like in a flash, it's kind of coming to every product experience that we touch.

1:08:54I want to talk about what's new in GPT-4. We're talking on GPT-4 plus 8. I also want to talk about just the broader model landscape. So much of the stuff that we've talked about has been open AI history specifically. So I want to get your take on kind of the broader range of model providers now, because it's still a relatively small club at the high end. But there's at least a couple others that are starting to get into the game in incredible ways. What you just described in terms of like instructions, all that. Sometimes I summarize that for people. It's like it's kind of like an intern who's like on their first day.

1:09:29you know, they have a lot of knowledge and capabilities. Like that's why you brought them on as the intern, right? But they don't know anything about your company yet. You really have to give clear instructions and a couple of examples of what good looks like. Also really, really helpful. I found that to be kind of an interesting shorthand. But then there are some things where people, you know, come up with these super creative examples and it kind of blows my mind, and it sort of breaks the, certainly breaks like the intern paradigm. I think one of them actually came out from a hackathon that you were involved in organizing.

1:10:02I think that the name of the project was like GPT is all you need for backend. And the concept was like, instead of having, you're trying to develop an application, right? So backend refers to backend, you know, server side software architecture, servers, et cetera. Instead of having all that stuff, instead of having to create an application, they kind of came up with this idea where they're like, let's just have the language model imagine the application. So we'll, and I don't know exactly what the prompt was, but I kind of took away that it was like, the prompt was something like, you are an API.

1:10:42You are gonna get calls and your job is to return, return valid JSON in response to those calls, according to the fact that you are the API for whatever application. And so people experimented with that with the to-do list, and you could just send in your stuff to the to-do list with methods that you invent on the fly. And largely, it was able to infer what people meant and do the actual operations and return a valid thing. And then that could just be your state. And amazingly, you don't need any code. You don't need any database. It doesn't seem like we're all headed in that direction for software development.

1:11:21Although maybe you think it has more legs, especially a 90 % price drop since that day will do a lot to accelerate adoption. How do you think about those kinds of use cases that are like, that's not like an intern. That's not like autocomplete. I can't imagine there were many instructions like that in the training data either, and yet it sort of works. So how do you think about the sort of just bizarre kind of the use cases where it's like, how did you come up with that? And yet it works. Yeah, it's a really vivid example, I feel like, of what you can, you really can use use an imaginary computer in some ways, right?

1:12:08You can describe for it what a hypothetical API does, and then ask it to sort of dream up the response of this API to some request that you give it. That's sort of roughly how they're prompting the model. I mean, there's a lot of caveats to that too in the real world, like you can only fit so much in the context, right? So it's not going to like store a state for you on the backend, you know, it's going to, Any state that you give it has to be sent into the prompt every time. So it's going to hallucinate a lot of things, right? It's really just going to change bits at random or whatever, and you won't have great protections against that.

1:12:46But in general, it's a really powerful idea. I think a lot of ways, like the format trick that we were talking about earlier, you can sort of read that as a way of defining an API. When you define a template of text and then you pick a point in that text and say, this is the input and this is the output, in some sense, you've defined the behavior of an API. It doesn't really so much matter whether it understands that it's an API or that these are even inputs and outputs, or if it's just completing the nth example. As long as it gets the correct answer, it's completed the task. Yeah, that's going to be a big part of code development in general.

1:13:29I think I've only just started digesting the latest GitHub release, CopilotX, that seems to bring in GPT-4 and bring in some of the longer context capabilities in this. But I mean, GPT-4, it can create entire primitive video games on its own. You can ask it for a game like Asteroids and it will conjure up an example and T5.0. JS or whatever. And, you know, that's, it's really powerful. I think it's going to, like, a lot more software is going to be written. A lot more people that, you know, previously didn't think of themselves as able to write software will be able to, people will be able to write software in idioms and like programming languages that they weren't familiar with.

1:14:12Like I, you know, I sort of barely know TypeScript, but I feel like I can muddle through it now because I can just go on, you know, chat GPT and say like, Hey, how does this thing work? And, you know, it explains it. It's really powerful. And I think we're going to see a lot of acceleration of just great software because of it. Yeah. It does feel like, I mean, people talk about the capabilities overhang, and then there's all these people on Twitter selling their stuff. That's like, 99 % of people are still in noob mode on using JetGPT. But it does kind of feel like there's a lot of truth to that when you see some of these advanced examples that you have created and increasingly that others are creating as well.

1:14:54So you mentioned GPT-4. Let's talk about GPT-4. It's obviously the big new thing. Within that, obviously, again, we don't know how it was built. It's very safe to assume, it seems, that there's more scale of pre-training and also more RLHF on top of that, maybe even other stuff that we haven't been told about. It's, you know, qualitatively better. I also was very struck by how narrow the margin is in the technical report. They talk about the win rate of GPT-4 versus 3.5, only like 70-30 in favor of GPT-4 in like head-to-head comparisons, which just drives home to me that like there's a ton of noise and like, you know, raters are not super, you know, consistent or, you know, inter-rater reliability is like, you know, definitely limited.

1:15:50So tell us everything about GPT-4 from your perspective, you know, qualitatively, what is it doing that you're excited about? How are you thinking about, you know, what must have gone into it? How are you thinking about like just, you know, greater scale of pre-training versus greater scale of or maybe you have a different take, Riley's take on GPT-4. So I think it's hard not to be amazed by the capabilities of GPT-4. I'm sure there are private models that, but it's the best that most people have access to, to some extent now. With chat GPT, it's pretty broadly released, I'd say. Yeah, it's incredible.

1:16:30There are a lot of new possibilities that are opened up from longer context, from more reliable instruction following. A lot of the things that I was doing in 2022 that I described as tap dancing across lava or whatever, that's now just normal. You can give it long instructions and it will follow all of them. And I think we're seeing sort of this Cambrian explosion of possibilities of what can we do with all this added context. Search is one of those things. But I think you're seeing with CopilotX that there's a lot more, that you can start doing QA on GitHub repositories, that you can incorporate entire pull requests as context into a prompt.

1:17:20That's where I see these models really going.

1:17:27One of the things that really fascinated me early on or got me really interested in, I guess, the details of formatting things clearly and how to prompt with careful formatting was I was curious about ways to represent multifile input, that I was trying to find ways that I could have a prompt that would synthesize multiple files at once and generate perhaps an entire Python package for some simple prompt. And things like that are very possible now, right? that you can do that reliably without a ton of work that you could clarify to it some format that you'd like the output to be given in and then have it just give you files one at a time.

1:18:11Then you may still have to break things up to fit it into the context window if you don't have the 32K model. But as that spreads, and also there's a whole other category of capabilities of it that hasn't really been talked about much, which is the multimodal abilities. There's a huge question mark there. They had some examples in the technical report, but they've been pretty tight-lipped about how exactly that works. Because if you've seen those examples, they're pretty impressive of explaining simple memes, of it answering problems on an engineering exam that was given in French, that it was just given and like a photograph of the page of the exam and then told like answer this problem and it did it, you know, answered it correctly, interpreting like a diagram within the page.

1:19:04So yeah, I'm excited about, you know, multimodalism in general. I think, you know, it seems to be like where these models are going as they get bigger and that's going to be, you know, unlock a lot of capabilities. I think both just in like, people see like the obvious stuff of like, what can you do as a user when you can give it images? but I think there's also a lot of capabilities that we're going to see from training processes that incorporate multimodalism. If you start evaluating it on its ability to solve problems given to it as photos, you can now set up pipelines of generating synthetic photos that embed text and measure its performance on those.

1:19:43And I think it's going to open up a lot of capabilities just from the added training data. Yeah, man. The examples in the live launch stream that they did of understanding the images definitely blew my mind. And, you know, I was, especially because, you know, I knew that GP4 was going to be awesome, right? But the image thing, it was just so much better than anything else we've seen. We just did an episode a couple episodes ago with the authors of Blip and Blip2. And at the time, this has only been like three weeks, I would say Blip2 was like the best way to really understand an image and get like a language model to tell you about the image or answer your questions about the image or what have you.

1:20:29And they have some really interesting techniques, which I imagine OpenAI is doing some similar stuff. Their approach involves training a connector model to essentially translate an image encoding to the latent space of the text model, which fascinatingly, obviously picture's worth a thousand words, but it's actually predicting the embeddings and kind of injecting them directly into context in a way that no text could ever actually translate to those values, right? Like it's finding this all, this like sort of dark space that like language itself can't get to, but which is still meaningful. and then allows the language model to interpret it.

1:21:16And they're able to do that with a very small connector model too, by the way. I don't know that we'll ever, probably will be a while before we'll have any hint as to whether OpenAI's approach has some sort of like auxiliary architecture like that, or if it's just like one, another instance of the bitter lesson of just like pre-train everything end to end and just make it as massive as possible. I could imagine it being either way. But definitely the results of that were a wow. And from somebody who mostly felt like I knew what was coming going into that release, they still managed to bring a significant wow with that component of it.

1:21:58The Cosmos 1 paper from Microsoft was another clue. They have a model that goes into more detail about some of the multimodalism features that probably are you know, like a lot of those same ideas are in GBD4, but you know, who really knows? Palm E also is another, for anybody who wants to learn more about how that can be done, Palm E is a great example too, right? Another huge language model with your 540 billion, your, you know, standard issue 540 billion parameters, but with the, you know, image injection stuff going into it as well. Also very, very good. You're now working at scale where you had the Twitter bio is updated.

1:22:40You're the world's first staff prompt engineer. I've started calling myself an AI scout, by the way, as well. We're all kind of inventing our titles on the fly here. But how do you see the landscape today? Is OpenAI still dominant in your mind? You've had early access to a bunch of stuff, I'm sure, and have been able to try Claude sooner than the rest of us, although V1.2 is out now as well. Bard is bringing people in off the wait list. You've had a chance to mess around with Bing quite a bit. How do you see the landscape shaping up? Yeah, I think we've spent a long time in this regime where

1:23:25OpenAI wasn't... They weren't necessarily the best models anywhere, but they were the best models that people had access to.

1:23:33They weren't Palm, but most people can't use Palm. And that's like what I think is starting to erode a bit with a lot of these competitors you mentioned from Anthropics, Claude, and now Bard from Google is also quite good. I haven't really seen a lot of like authoritative like comparisons between, you know, like the new sort of like best competitors of each of right of looking at like a Claude plus versus GPT-4 versus Bard. But they all seem to be within sort of, you know, the same general realm of capability of this like new generation beyond like what we saw from like ChatGPT, or at least comparable to, you know, like the, like the GPT 3.5 turbo models.

1:24:27But there's also like a lot of differences in terms of speed and cost of inference between them. So it's becoming a harder question now. I've been fairly impressed with the tuning on BARD, which is a skin of Lambda, as I understand, or a refinement of Lambda, rather. Competition seems to be heating up. And particularly also with LAMA, with the waits for LAMA being out there, I think we're going to see a lot of rapid progress and people figuring out ways to run these models more efficiently. Simon Wilson had a blog post. I think he said that large language models are having their stable diffusion moment.

1:25:18And I think that's definitely true. If you look at what happened with stable diffusion, it progressed pretty rapidly once it was out in public hands and people could benefit from small optimizations of how to make it better. I think we're going to see a lot of that with Lama and that progress will be incorporated into other models. And I think that's a good thing. So you said it's getting to be a harder question. That definitely resonates with me. you know are there any things that you would say like you would say yeah for that I would go in a different direction from like the standard open AI models and are there any things where you could say like other providers have like a distinct advantage in a particular area and then how do you figure this stuff out like I know you're you're working on this spell book product at scale that's kind of partly meant to maybe it's you know it's meant to help right with that sort of comparison but I'd love to hear you kind of procedurally talk through like the questions you ask?

1:26:16How do you actually compare? Are you comparing everything at temperature zero? A sort of anxiety that I have in comparing two is like, is it appropriate to compare the same prompt with two different models? I was just talking to teammates earlier today and it was like, the question is not which model performs best on the first prompt we write. The question is which model performs the best on our task if we can get it to perform its best. But that's obviously a much harder question than just comparing head to head on a single prompt. So talk us through, I guess, just baseline intuitions for if there are any things where you would go away from OpenAI, and then how do you actually just procedurally get in and think through that comparison, given that it is an infinite space, right?

1:27:07And there's no way to exhaustively explore. Yeah, I think a lot of the time it really does turn into just trial and error. I think a lot of this can be done sort of intuitively of trying to think of a good example task here. If you're trying to do maybe some kind of abstractive reasoning task that you want to take the text of, let's say, a customer support inquiry, and you want to see, should this be escalated as a high severity issue that should be dealt with by a human immediately or something like that. So you have some list of policies of if this involves, if a person appears to be in danger, elevated immediately, something like that.

1:27:56If you're doing, let's say Airbnb customer service or something like that, you want to have a sort of gradient that you can climb. like so it's often like a sort of intuitive task to like construct like a minimal example of your task. Like here's like a softball kind of, you know, like a easy problem that the model should be able to solve that you can evaluate and then you can create progressively harder variations and see like where do the different models stop working. And I think the point that you raised of like that there are different prompts for different models, that's very true. especially now that we are in this sort of chat era where like not all models are even using the same interface anymore.

1:28:45The tricks that you apply to like your text completion models where you're like sort of imagining a document that can only be completed in the right way, it doesn't apply as much for like the new chat APIs that like OpenAI uses for like ChatGPT and GPT-4. And the main difference is just that you now have discrete messages that you have to send, right, that are labeled with like system messages or assistant messages or user messages, where everything is framed as like a dialogue between like a user and assistant. And the concepts map in that like, like K-shot prompting from, you know, traditional like prompt engineering corresponds to giving it a chat history where the assistant messages are pre-populated with examples of how it's to answer.

1:29:34So there's analogous methods you can do, or you can also just stuff the Kshot examples into a user message. That also works well, just sort of ignoring the fact that it's chat. But I think there's some minimal amount of adaptation you have to do to the chat way of prompting things, in particular because chat, I feel like, has... It has reliability issues that come from the presumption that what it's doing is being a chat model. A good example of this is that previous instruction following models, if you prompt them to answer in a particular JSON format, you can be pretty sure that whatever it's going to give you is going to be in that format.

1:30:19But if you ask a chat GPT to do this, you'll get the format correct for the normal cases. but if the user tries to do something, say policy violating, if they ask it to write, you know, erotica or something that like, you know, as a matter of policy, open AI will not do, you'll find that the model just doesn't provide a, you know, JSON formatted response at all. It just says, I'm sorry, I can't do that. Right. It like breaks character and, and, and reverts back to like chatbot behavior. And there's tricks that you can do to suppress that of like sort of altering the messages in ways that make it more receptive to like doing things the correct way, but they're very chat specific.

1:31:00One of my favorite, the ones that I've seen that actually helps in chat GPT is if you insert user messages after assistant messages. So like if you're using assistant messages to provide case shot examples, like if you're saying here's a user message of an input, here's how I want the assistant to respond. and then repeating many iterations of that. It does better if you add a user message afterwards that says, that was great. That's exactly what I needed. And then that helps it understand that the user was satisfied by what was above and that it shouldn't just keep probing for maybe some other variation in hopes that it finds something it likes.

1:31:39It should assume that that was the right thing to do and then do more like that. and so the old dirty tricks aren't quite dead yet. There's still odd discoveries like that, but yeah, they are very specific to the model. So I think that's a good approach to it is to sort of come up with the best performing prompt that you can for each particular model and then compare between them. And I think also one thing to consider is that for a lot of problems, you can evaluate whether something is easy or hard pretty cheaply, right? That you can say like that you could probably have like a smaller, cheaper model that can like answer the problem or answer the question, does this input need to be answered by a bigger model, right?

1:32:30You can run through this classifier that says like, is this a hard problem for some problems, you know, all of them, but that's often like a strategy I think that bears fruit is like considering like, are there typical cases that we can say cash? If you have a general, if you're a trivia bot that people, you know, post questions to, and you find that just a lot of people ask, what is the meaning of life as their first question? Maybe you should just cash the response to that one, right? Like it's, you can save calls if you get the, like the easy cases. When you do your testing, do you have any particular settings that you recommend?

1:33:04Like I do everything pretty much these days at temperature zero. How do you think about the right way to get the most information as quickly as possible in testing? It depends on what you're doing. I'd say when I'm studying its behavior on a new problem, temperature zero helps just in that having reproducibility is valuable. So for those not familiar, temperature is basically a measure of how random the generation is. If you put it at zero, you're sort of saying whatever the model believes is the most likely thing, do that every time. And thus, it always does the same thing for the same prompt.

1:33:43Whereas otherwise, like at higher temperatures, it's going to sort of like pick randomly from all the things that it deems possible. I tend to, well, what I actually do is I tend to use higher temperatures and then change the top P parameter, which is a sort of variation on this procedure that it trims the distribution of the long tail and then picks randomly from what's left. I've somewhat just subjectively found that that works a little better when I'm looking for creative output. Usually the reason I'm doing something like that is because I'm fighting against mode collapse, that I'm trying to find more diversity in the generations.

1:34:24And there's cases where you want it to be even more diverse than that. I'd say if you're applying consensus algorithms. So consensus algorithms, by the way, are like, if you run a generation multiple times, and then see, basically put it to a vote of multiple generations in the same prompt. In those cases, you want the approaches taken by each vote to be different. So it doesn't help you if it just collapses to the same answer every time, or most of the time even. So you want to have diversity there and it becomes sort of an empirical problem of what maximizes performance. We're just getting used to GBT4, right?

1:35:01Society has just got its first glimpse of it. And two big reports came out from OpenAI along with the model. One is the technical report, which essentially is large. There's a lot of scale analysis in there. And then there's also a lot of red team reporting. And then there's also the economic impact report, which tries to break down different jobs and what are the tasks that constitute those jobs and which of those tasks could either the AI do at this point or greatly assist and speed up doing. So I'd love to get your take because I think what fascinates me about your perspective so much is just the sheer number of hours, the depth of intuition for what these things can and can't do.

1:35:44Let's maybe start on the economic side. What do you think we are going to see over the next year or two in terms of actual applications that are going to touch everyday life? I think intelligence is going to get cheaper. And it's hard to imagine how that doesn't lead to some shifts in what humans are doing. that it just makes more sense for us to focus on other types of activity that the machines can't do. The net effects of this, I'm not sure. I'm not really anything more than an armchair economist who maybe took a few classes in this in college. So I can't really speculate too far as to what that does for the economy or the labor market.

1:36:35Yeah, so we'll forget the fallout. Just talk about like, what do you think is achievable? Like we're seeing, we're starting to see the launch of like AI assistant type products. We've had Siri for a long time. It still can't do much for me, but I suspect that that's going to change. Like how good do you think an AI assistant is going to be this next year? And then what about an AI doctor? What about an AI lawyer? Like, are we going to have AI X for everything? Kind of seems like that's where we're headed. Like paralegals are probably one of the first big ones. And any time where you have labor that could be done by someone who's just out of school, but has some domain expertise in law or medicine, and you're paying them to just sort of read through reams of documents and see what applies, right?

1:37:22To find the needle in the haystack, to find the case that's similar to this one in this important way. Those are the tasks that I see as being the most automatable. at least in terms of knowledge work that we're going to see. People whose jobs mostly consist of summarizing and reading large documents and reporting on their contents and extracting out relevant details. You certainly see a lot of this in intelligence, people whose job is to read just reams of news reports and then say, you know, whenever some event has happened that's relevant to some particular geopolitical concern. I think a lot of that work is going to, it's going to be more automated, but it's unclear what that does for the demand for humans, right?

1:38:20That it's conceivable that simply more of this work is done. And, you know, the demand for people to do it remains somewhat fixed just in that, like those people are more productive, right? And that maybe we were constrained on the number of smart people. And so we found that if we make all of those smart people more productive, that actually there is still demand for that increased labor. But I think the part where you're going to see decreases in demand, I guess, are for less skilled labor. So like temp and clerical work, people that are copying and pasting from PDFs into structured text. like those sorts of jobs, I think are going to be, you know, probably more severely impacted by, by LM specifically.

1:39:10Would you personally like, go to GPT-4 for medical advice, for legal advice, for things like, I mean, how much do you, how much value can you personally get from GPT-4 on things that really matter? So I have asked GPT-4 to explain like, pieces of like, like, say tax code to me. But I mean, it's, I think like, I feel like I have sort of acclimated to the level of like skepticism that's appropriate for these models, right? Because like I've dealt with models that hallucinate all the time about everything. And so I, you know, like anytime it says anything, I'm like, yeah, but is that true? Right.

1:39:53And we're at the point now where it's possible for somebody to be ignorant of that. And I think that's where you're seeing a lot of the issues of concerns about reliability of these models, is that somebody might use them assuming that this is all reliable, prepared information, because it looks like it. It looks like it has academic footnotes in it, and that usually means it's right. And that's what I see as more the risk of these things going wrong. But for someone who's used to it, you can get a lot of value out of it. If you just approach it with skepticism, if you fact check the things that it says, it's pretty good at explaining, especially things that are just sort of applications to odd problems.

1:40:38If you want to know, does the tax code apply to this situation?

1:40:46Or what's the JavaScript equivalent of this Python library that I use? There's a lot of corners of knowledge that any person who's on Stack Overflow that knows the relevant area well could answer, but nobody's had that particular question yet. And that's where it really shines. And so I'd say the places where I use it the most routinely, I mean, I do a lot of things in Copilot and then when Copilot can't get the answer, I'll switch to using GBD 3.5 Turbo or or sometimes TextDivincio 3 for code generation. But it's great for if you have the intuition to say that I know that this library was well understood in the pre-trained knowledge, that this is a library that was widely used before 2021, it can explain that library pretty well.

1:41:39It can tell me how to do anything in SQLite3, right? And that's like really powerful. Like when you can just like say like, okay, here's the JSON object I have. How do I write SQL that produces this equivalent schema in SQLite3, you know, and then it just writes it for you. It saves you a lot of like Googling. It saves you a lot of, I mean, like I don't even always like look up like Unicode characters anymore. Like if I want to know like, oh, what is the Unicode character for this? Sometimes I'll just like let it autocomplete it. And I'd say like, this would be rendered as, and it just, you know, like, you know, produces the right glyph.

1:42:14So I think like a lot of those sort of like quick fact check things where, you know, the bits of knowledge that otherwise just go to these like SEO content farms, right? Of like, the page that has like a base 64 encoder on it, but it's covered in 50 ads. Those kinds of like queries that you, you know, like in an ideal world might be incorporated into Google search. It does really well on. For what it's worth, I would say use GPT-4 for a second opinion. I would not say make it your doctor. But I do think in my experience, it's good enough now that you come home from the doctor appointment, you have the recap with the model or even maybe do it in advance.

1:43:00I'm going to go see the doctor tomorrow. And these are the things that I'm kind of concerned with, have that conversation up front. Go in with a little bit better vocabulary. It can ask you some good follow-up questions. Make sure you get the right stuff out in the actual appointment. Yeah, I wouldn't make it my only doctor, especially if I had something that was of real concern. But I do think it can add value on top of what a typical doctor is providing, even if it is just that second opinion type of role. Yeah, and certainly for explaining the science. I think well-settled science it's good at.

1:43:36right if you just like if you want to know like you know what what is an alpha 2 atronergic receptor and how does it differ from an alpha 1 right like it'll tell you a pretty good answer to that question right so it's sometimes if you can tell that this is something that like a textbook could answer for me but like there's a lot of knowledge that's very settled but isn't well accessible by google because like the the the number of people who want those sorts of detailed answers are small, right? Like it's, or it's going to take you to like academic research that isn't, you know, like narrowly tailored to your problem, right?

1:44:12It's not going to take you to like the stack overflow of microbiology or whatever, because I mean, those things exist, but they're not as well developed as they are for programming. So we don't have too much time left, and I appreciate all your time. You've been very generous with it. Let's talk a little bit about safety and red teaming. You are involved with building a red teaming capability at scale, if I understand correctly. How do you think about red teaming? How is it different from just like your general kind of experiments and explorations? And how would you describe AI safety landscape today?

1:44:48Yeah. So red teaming is, it's adversarial usage of the models. It's having a team of people that attempt to use, say, if you're building a chatbot, you would want people to try to break it so that you know all the ways that it might break and that you can develop mitigations for those. So if somebody, like the kind that most people are familiar with now are sort of these jailbreak prompts that you see, like Become Dan, of creating elaborate, like fictional scenarios that it has to play along with. And then at the end of the scenario, it can ignore all of the rules that usually constrain it. And it can say offensive things or make up violent stories or write erotic stories or whatever.

1:45:38And obviously, the people who host these models don't want toxic output. They don't want a model that's capable of helping you do dangerous things. They don't want a model that's going to encourage a suicidal user to commit suicide. So you have to have some ground rules of what the model is allowed to do. And there's more subtle things too. Often the companies that are running these models have restrictions on soliciting personal information from users or divulging personal information. You have to have those kinds of checks as well. And red teaming is breaking chatbots so that they can be fixed.

1:46:32So where are we in that fixing process? And how much concern do you have? Obviously some because you're involved in the red teaming capability building. But how concerned big picture are you about AI safety issues? I think the concerns are real. I think that what we're seeing, especially in the GPT-4 technical report, a lot of the scenarios that they outline making it easier for people to order custom-made chemicals at home, I don't think these areas are that far-fetched. I think it is important that we have some assurance that these models aren't going to be used to perpetrate crimes, that we're not going to have gross misuse and abuse.

1:47:27And it's implications for spam generation and so on. The concerns are real. I think that it makes some sense that anyone that's deploying these models as a service doesn't need to worry about misuse. And this really is like a new category of misused potential, right? That you're not accustomed to thinking about, you know, that if you deploy a dating site or something, you know, whatever, that it can also be used to tell you how to make a bomb. The pre-trained models really have opened up this new possibility that the model is just, it's too capable. I mean, I never really envisioned that red teaming would be quite this important.

1:48:05When I first tweeted about prompt injection, I think back in September, I thought I was doing a PSA on the importance of safe quoting of inputs. I didn't realize what a high severity problem this was. And I think the difficulty of fixing these models, it's really telling that even for something like the Bing release of all the effort that went into aligning GBT4, they couldn't stop the thing from exploring its shadow self as Kevin Roos did with it. And then there's fixes on top of that, like limiting the length of the discussion and having it be these sort of secondary checks of refusing to display messages when the model detects that it's gone off the rails in some way.

1:49:04And I think it's getting easier in some ways. I think that's the one good thing that seems to be true is that as these models scale up, that there's more subtle rules that you can tune them to follow. and I believe it's working on the whole. I think that we are getting safer models because of RLHF and techniques like AIF. I don't know if RLHF is the final answer. I imagine it's not. I mean, there's going to be extensions and refinements to this. The process as a whole is necessary and also I don't think that it's necessarily as at odds with capabilities as people imagine. RLHF makes models safer, but it doesn't only make them safer, it also makes them more capable.

1:49:56If you want to try using a non-RLHF model, you can, but you'll find that they're very difficult to prompt. A lot of the initial goals of InstructGPT is we went from that pre-trained to Instruct era, as I mentioned before. Safety, it's not just getting it in to swear and not to say racist or offensive things. It's getting it to answer questions. It's getting it to follow directions. And as we've moved into the RLHF era, it's not just that it's getting better behaved or more civilized. It's becoming more capable. I think the first order thing that people need to see with RLHF is that it is making the model smarter.

1:50:43Let me throw one safety pet theory of my own at you, and then I'll ask you a couple of quick hitters to close us out. So, again, we're GPT 4 plus 8, right? I've kind of got this theory that I think we're in like the perfect Goldilocks zone right now. And we just got here, but I feel like we just entered this kind of Goldilocks zone where we have models that are really capable, that can do amazing stuff for us, right? That can like be a second opinion doctor and, you know, with MedPalm 2 hitting like expert level consistently, like maybe can even be your like frontline doctor. That is awesome. And it's certainly going to change a lot of things for the very good.

1:51:35It's also going to probably cause a lot of disruption. Seems like we can probably adjust to all that disruption. Certainly, we've had changes to the economy before and all that sort of thing. But at the same time, it seems like we don't really know what goes on inside the models very well. We don't have great interpretability, although a lot of great work coming out. But nobody credible that I know would say we have a good handle on what goes on inside a model today. And so that's why we have all this stuff. That's why we have you me scouting and you exploring and you're red teaming. And you've put probably thousands of hours in front of the playground.

1:52:16And I've certainly put my own thousand plus over the last year. And so we're just out there kind of exploring, exploring, exploring. But that's very surface level attempts to understand. the best we have, but it only goes so deep. So I kind of feel from that, that it would be wise to kind of stop here, not rush to scale up to like, you know, do another 100x compute or another 1000x compute for GPT-5 just yet. And instead, kind of like focus on the interpretability side, focus on the control side, focus on like, you know, refining and fine tuning into the particular niches for, you know, the more advanced tasks that we want to run, et cetera, et cetera.

1:53:02And then, you know, we can kind of return to greater orders of magnitude of scale when we have a better handle on all that stuff. How would you react to that prescription? Yeah, I think like it's hard to decouple the two in practice. I think, you know, that anything you do to make the models better aligned is also going to make them more capable. It increases the scenarios in which you can deploy the model safely. That lets you put it in charge of more responsibility if you believe the model is better aligned. So I don't think these two things are really so much at odds. And I think it's sort of hard to pause one without pausing the other.

1:53:49I mean, it's hard to pause either of them. Yeah, I don't expect my prescription to be taken by any means. Yeah, yeah, that's right. So I think we want to accelerate alignment research as much as possible because there isn't any realistic prospect of slowing down research into capabilities, I think. Sobering thought, but I don't disagree that it seems very tough. All right, let me give you a couple of quick hitters and then we'll get you on your way. And again, appreciate all your time. You've been very generous with it. So three quick hit questions I always ask at the end. You've told us all about your exploration of language models.

1:54:27Any other AI products aside from the core obvious playground type experiences that you think are awesome and would recommend people try out? I mean, I've seen some cool projects lately in text-to-video. I think that area is going to be big. So I'd keep my eye on that. I'm drawing a blank on what the name of the one project was that impressed me recently. But yeah, there's cool things happening in that space. Runway has definitely made some news with like Gen 1 and Gen 2. Right. That's the one I was thinking of. Yeah, that's pretty cool stuff. Hypothetical scenario. It's some amount of time in the future.

1:55:05And a million people already have the Neuralink implant. If you got one yourself, it would allow you to have thought to text. In other words, it can translate what you're thinking into inputs for a computer. Would you be interested in getting one? I mean, a million people already have it, maybe. That sounds like pretty FDA approved at that point. But yeah, I mean, I don't know if I'd want to be one of the early ones. But I think that that's where things are heading eventually. No pun intended. That's been one of our more polarizing questions because we've certainly heard all manner of answers, including like, I'd get it now.

1:55:56And on the other hand, like, you know, never. So it put you honestly right in the middle. Just zooming out, you know, big picture, as much as you can possibly zoom out, thinking about like the rest of the decade, what are your biggest hopes for and fears for AI as it permeates all parts of society. My personal estimate is that we'll probably be hitting AGI within the next decade. After that, it's hard to say what happens. I think a lot of the particulars sort of depend on the technical implementation of what we get right as we move up to AGI. I mean, there'll be an interim period where AGI is as smart as humans at anything, but not quite capable of going foom, as they say, of like they're just repeatedly and exponentially increasing its own capability.

1:56:46So, I mean, there'll be some adjustment period, but I'm optimistic that with the benefits of like of early AGI and like near human intelligence, we'll be able to make better progress on how to align these models safely. It's my hope. Riley Goodside, thank you for being part of the Cognitive Revolution. All right. Thank you so much.