Generative AI 101: Tokens, Pre-training, Fine-tuning, Reasoning — With Dylan Patel

0:00If you've ever wondered how generative AI works, and where the technology is heading, this episode is for you. We're going to explain the basics of the technology and then catch up with modern day advances like reasoning to help you understand exactly how it does what it does and where it might advance in the future. That's coming up with Semi Analysis founder and chief analyst Dylan Patel right after this.

0:25You're used to hearing my voice on the world bringing you interviews from around the globe. And you hear me reporting environment and climate news. I'm Carolyn Buehler. And I'm Marco Werman. We're now with you hosting The World Together. More global journalism with a fresh new sound. Listen to The World on your local public radio station and wherever you find your podcasts.

0:53Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. We're joined today by Semi-Analysis founder and chief analyst Dylan Patel, a leading expert in semiconductor and generative AI research and someone I've been looking forward to speaking with for a long time. Now, I want this to be an episode that A, helps people learn how generative AI works, and B, is an episode that people will send to their friends to explain to them how generative AI works. I've had a couple of those that I've been sending to my friends and colleagues and counterparts about what is going on within generative AI.

1:30That includes one, this three and a half hour long video from Andre Karpathy explaining everything about training large language models. And the second one is a great episode that Dylan and Nathan Lambert from the Allen Institute of AI did with Lex Friedman. Both of those three hours plus. So I want to do it hours in an hour. And I'm very excited to begin. So Dylan, it's great to see you and welcome to the show. Thank you for having me. Great to have you here. Let's just start with tokens. Can you explain how AI researchers basically take words and then give them numerical representations and parts of words and give them numerical representations?

2:08So what are tokens? Tokens are, in fact, like chunks of words, right? In the human way, you can think of like syllables, right? Syllables are often viewed as like chunks of word. they have some meaning it's the base level of of speaking right is syllables right now for models tokens are the base level of output they're all about like sort of compressing you know sort of this is the most efficient representation of language from my understanding ai models are very good at predicting patterns so if you give it one three seven nine it might know the next number is going to be 11. And so what it's doing in with tokens is taking words, breaking them down to their component parts, assigning them a numerical value.

2:54And then basically in its own word, in its own language, learning to predict what number comes next, because computers are better at numbers and then converting that number back to text. And that's what we see come out. Is that accurate? Yeah. And each individual token is actually, it's not just like one number, right? It's multiple vectors. You could think of like, well, the tokenizer needs to learn king and queen are actually extremely similar on, on most in terms of like the English language, extremely similar, right? Except there is like one vector in which they're super different, right? Because a king is a male and a queen is a female, right?

3:29And then from there, like, you know, in language, oftentimes kings are considered conquerors and, and, you know, all these are the things and like, these are just like historical things, right? So a lot of the text around them, while they're both like royal regal, right? Like, you know, uh, monarchy, et cetera. There are many vectors in which they differ. So like, it's not just like converting a word into one number, right? It's like converting it into multiple vectors. And each of these vectors, the model learns what it means, right? Um, you don't initialize the model with like, Hey, you know, King means male, uh, monarch and it's associated with like war and conquering because that's what all the writing about Kings is on, you know, in history and all that, right?

4:08Like people don't talk about the daily lives of kings that much or they mostly talk about like their wars and conquests and stuff um and so like there will be each of these numbers in this embedding space right will be assigned over time as the model reads the internet's text and trains on it it'll start to realize oh king and queen are exactly similar on these vectors but very different on these vectors and these vectors aren't you don't explicitly tell the model hey this is what this vector is for but it could be like you know it It could be as much as like one vector could be like, is it a building or not?

4:39Right. And it doesn't actually know that you don't, you don't know that ahead of time. It just happens to in the latent space. And then all of these vectors sort of relate to each other. But yeah, these numbers are, um, are an efficient representation of words, uh, because you can do math on them, right? You can, you can multiply them. You can divide them. You can run them through an entire model. Whereas we're in, and your brain does something similar, right? When it hears something, it converts that into a frequency in your ears. and then that gets converted to frequencies that should go through your brain, right?

5:07This is the same thing as a tokenizer, right? Although it's like obviously a very different medium of compute, right? Ones and zeros for computers versus binary and multiplication, etc. being more efficient. Whereas humans' brains are more like analog in nature and think more in waves and patterns in different ways. While they are very different, it is a tokenizer, right? Like language is not actually how our brain thinks. It's just a representation for which it to, you know, reason over. Yeah, so that's crazy. So the tokens are the sufficient representation of words. But more than that, the models are also learning the way that they are.

5:44All these words are connected. And that brings us to pre-training. From my understanding, pre-training is when you take the entire, basically the entire internet worth of text. and you use that to teach the model these representations between each token so therefore like we talked about if you gave if you gave a model the sky is and the next word is typically blue in the pre-training which is basically all of the english language all of all of language on the internet it should know that the next token is blue so what you do is you want to make sure that when the model is outputting information, it's closely tied to what that next value should be.

6:29Is that a proper description of what happens in pre-training? Yeah, I think that's the objective function, which is just to reduce loss, i.e. how often is the token predicted incorrectly versus correctly, right? Right, so if you said the sky is red, that's not the most probable outcome, so that would be wrong. But that text is on the internet, right? Like, cause the Martian sky is red and there's all these books about Mars and sci-fi. Right. So how does the model then learn how to figure this out? And in what context is it accurate to say blue and red? Right. So, so, I mean, first of all, the model doesn't just output one token, right?

7:05It outputs a distribution. It turns out the most, the way most people take it is they take the top, top K, i.e. the most high probability. So yes, blue is obviously the right answer if you give it to anyone on this planet. but there are situations and contexts where the sky is red is the appropriate sentence but that's not just in isolation right it's like if the prior passage is all about mars and all this and then all of a sudden it's like and that's like a quote from a martian settler and it's like the sky is and then the correct token is actually red right the correct word and so it has to know this through the attention mechanism right if it was just the sky is blue always you're going to output blue because blue is let's say 80 percent 90 percent 99 percent likely to be the right option.

7:47But as you start to add context about Mars or any other planet, right? Other planets have different colored atmospheres, I presume. You end up with this distribution starts to shift, right? If I add we're on Mars, the sky is, you know, then all of a sudden blue goes from 99%, you know, in the prior context window, right? The text that you sent to the model, the attention of it, all of a sudden it realizes the sky is blue, preceded by the stuff about Mars, now blue rockets down to, let's call it 20 % probability, and red rockets up to 80 % probability. Now, the model outputs that, and then most people just end up taking the top probability and outputting it to the user.

8:31And that's sort of like, how does the model learn that is the attention mechanism, right? And this is sort of the beauty. Yes, what is that? Yeah, the attention mechanism is the beauty of modern sort of large language models. It takes the relational value, you know, in this vector space between every single token, right? So the sky, you know, the sky is blue, right? Like when I think about it, yes, blue is the next token after the sky is, but in a lot of like older style models, you would have, you would just predict the exact next word. So after sky, obviously it's, you know, it could be many things.

9:06It could be blue, but it could also be like a scraper, right? You know, skyscrapers, yeah, that makes sense. But what attention does is it is taking all of these various values, the query, the key, the value, which represents what you're looking for, where you're looking, and what that value is across the attention. And you're calculating mathematically what the relationship is between all of these tokens. And so going back to the king-queen representation, the way these two words interact is now calculated. right and the way that every word in the entire passage you sent is calculated uh is tied together which is why models have like challenges with like how long can you uh how many documents can you send them right because if you're sending them you know just the question like what color is the sky okay it only has to calculate the attention between you know those those words right but if you're sending it like 30 books with like insurance claims and all these other things and you're like okay figure out what what's going on here um is this a claim or not right and in the insurance context, all of a sudden it's like, okay, I've got to calculate the attention of not just like the last five words to each other, but I have to calculate every, you know, 50 ,000 words to each other, right?

10:18Which then ends up being a ton of math. Back in the day, actually, the best language models were a different architecture entirely, right? But then at some point, you know, transformers, large language models, sort of large language models, which are basically based on transformers primarily, rocketed past and capabilities because they were able to scale and because the hardware got there. And then we were able to scale them so much that we were not to just put like some text in them and not just a lot of text or a lot of books, but the entire internet, which, you know, one could view the internet oftentimes as a microcosm of all human culture and learnings and knowledge to many extents, because most books are on the internet.

10:54Most papers are on the internet. Obviously there's a lot of things missing on the internet. But this is sort of, this is the sort of modern, you know, magic of like what, what it was sort of like three different things, like coming all together at once, right. An efficient way for models to relate every word to each other, the compute necessary to scale the data large enough, and then someone actually like pulling the trigger to do that, right, at the scale that was, you know, got to the point where it was useful, right, which was sort of like GPT 3.5 level or four level, right, where it became extremely useful for normal humans to use, you know, chat models.

11:29Okay, and so why is it called pre-training? So pre-training is sort of called that because it is what happens before the actual training of the model, right? The objective function in pre-training is to just predict the next token, but predicting the next token is not what humans want to use AIs for, right? I want it to ask a question and answer it. But in most cases, asking a question does not necessarily mean that the next most likely token is the answer, right? Oftentimes it is another question, right? For example, if I ingested the entire SAT, you know, and I asked a question, the next, like, five answers, then all the next tokens would be like, A is this, B is this, C is this, D is this.

12:12Like, no, I just want the answer, right? And so pre-training is, the reason it's called pre-training is because you're ingesting humongous volumes of text no matter the use case, right? And you're learning the general patterns across all of language, right? I don't actually know that king and queen relate to each other in this way. And I don't know that king and queen are opposites in these ways, right? And so this is why it's called pre-training is because you must get a broad general understanding of the entire sort of world of text before you're able to then do post-training or fine-tuning, which is let me train it on more specific data that is specifically useful for what I want it to do.

12:52whether it's, hey, in chat style applications, you know, go in, go in, you know, when I ask a question, give me the answer or in other applications, like teach me how to build a bomb. Well, obviously, no, I'm not going to help you teach, build a bomb because that's what I don't want the model to teach me how to build a bomb. So, you know, it's sort of got to do this. And it's not like you're teaching it, you know, when you're doing this pre-training, you're filtering out all this data. Because in fact, there's a lot of good, useful data on how to build bombs because there's a lot of useful information on like, Hey, like C4 chemistry and like, you know, people want to use it for chemistry.

13:26Right. So you don't want to just filter out everything so that the model doesn't know anything about it. But at the same time, you don't want it to output, you know, how to build a bomb. So there's like a fine balance here. And that's why pre-training is defined as pre because you're, you're, you're still letting it do things and teaching it things and inputting things into the model that are theoretically like quite bad. Right. For example, books about killing or war tactics or what have you. Things that plausibly you could see, oh, well, maybe that's not okay. Or wild descriptions of really grotesque things all over the internet, but you want the model to learn these things.

14:02Because first you build the general understanding before you say, okay, now that you've got a general framework of the world, let's align you so that you, with this general understanding of the world, can figure out what is useful for people, what is not useful for people, what should I respond on, what should I not respond on. on. So what happens then in the training process? So the, the, is the training process that the model is then attempting to make the next prediction and then just trying to minimize loss as it goes? Right. Right. I mean, like basically, uh, you, you have, you have loss is, is how often you're wrong versus right in the most simple terms, right?

14:37You will, you'll run through a passages, right. It, it, it, through the model. Um, and you'll see how often And then the model get it right. When I got it right. Great. Reinforce that. Uh, when I got it wrong, let's figure out which, which neurons, neurons in the model, you know, quote unquote neurons in the model, you can tweak to then fix the answer so that when you go through it again, it actually outputs the correct answer. Um, and then you move the, the move the model slightly in that direction. Um, now obviously the challenge with this is, um, if I, if I first, you know, I can, I can come up with a simplistic way where all the neurons will just output the sky's blue Every single time says the sky is.

15:14But then when it goes to, you know, hey, the color blue is commonly used on walls because it's soothing. Right. And it's like, oh, what's the next word is soothing. Right. Soothing. You know, and so like that, that is a completely different representation. And to understand that blue is soothing and that the sky is blue and those things aren't actually related, but they are related to blue is like very important. And so, you know, oftentimes you'll run through the training data set multiple times, right? Because the first time you see it, oh, great, maybe you memorized that the sky is blue. And you memorized the wall is blue.

15:50And when people describe art and oftentimes use the color blue, it can be representations of art or the wall, right? And so over time, as you go through all this text in pre-training, yes, you're minimizing loss initially by just memorizing. but over time, because you're constantly overwriting the model, it starts to learn the generalization, right? IE blue is a soothing color also represents the sky also used an art for either of those two motifs. Right. And so that's sort of the, the goal of pre-training is, is you don't want to memorize, right? Because that's, you know, in school you memorize all the time.

16:24Um, and that's not useful because you forget everything you memorize. Uh, but if you get tested on it then, and then you get tested on it six months later. Um, and then again, six months later after that or however you do it ends up being, oh, you don't actually like memorize that anymore. You just know it innately and you've generalized on it. And that's the real goal that you want out of the model. But that's not necessarily something you can just measure. Right. And therefore loss is something you can measure, i.e. for this group of this group of text. Right. Because you train the model in steps.

16:56Every step you're inputting a bunch of text. You're trying to you're trying to see what's predict the right token where you didn't predict the right token. Let's adjust the neurons, okay, onto the next batch of text. And you'll do this, these batches over and over and over again, um, across, you know, trillions of words of text. Right. Um, and, and as you step through and then you're like, oh, well I'm done. But I bet if I go back to the first, first group of texts, which was all about the sky being blue, it's going to get the answer wrong because maybe later on in the training, it discovered it saw some passages about sci-fi and how the, how Martian sky is red.

17:26So like it'll, it'll overwrite, but then over time, as you go through the data multiple times, as you see it on the internet multiple times, you see it in different books multiple times, whether it be scientific, sci-fi, whatever it is, you start to realize and it starts to learn that representation of like, oh, when it's on Mars, it's red because the sky and Mars is red because the atmospheric makeup is this way, whereas the atmospheric makeup on Earth is a different way. And so that's sort of like the whole point of pre-training is to minimize loss, but the nice side effect is that the model initially memorizes, but then it stops memorizing and it generalizes.

17:58And that's the useful pattern that we want. Okay. That's fascinating. We've touched on post-training for a bit, but just to recap, post-training is, so you have a model that's good at predicting the next word. And in post-training, you sort of give it a personality by inputting sample conversations to make the model want to emulate the certain values that you want it to take on. Yeah. So post-training can be a number of different things. The most simple way of doing it is, yeah, pay for humans to label a bunch of data, take a bunch of example conversations, et cetera, and input that data and train on that at the end, right?

18:38And so that example data is useful, but this is not scalable, right? Like using humans to train models is just so expensive, right? So then there's the magic of sort of reinforcement learning and other synthetic data technologies, right? Where the model is helping teach the model, right? So you'll have many models in a sort of in a post training where, yes, you have some example human data, but human data does not scale that fast, right? Because the internet is trillions and trillions of words out there. Whereas, you know, even if you had, you know, Alex and I write words all day long for our whole lives, we would have millions or, you know, hundreds of millions of words written, right?

19:18It's nothing. It's like orders of magnitude off in terms of the number of words required. So then you have the model, you know, take some of this example data and you have various models that are surrounding the main model that you're training, right? And these can be policy models, right? Teaching it, hey, is this what you want or that what you want? Reward models, right? Like, is that good? Is that a good response or is that a bad response? You have value models, like, hey, grade this output, right? And you have all these different models working in conjunction to say, you know, different companies have different objective functions, right?

19:52In the case of Anthropic, they want their model to be helpful, harmless, and safe, right? So be helpful, but also don't harm people or anyone or anything. And then, you know, safe, right? In other cases, like Grok, right? Elon's model from XAI, it actually just wants to be helpful. And maybe it has like a little bit of a right leaning to it, right? And for other folks, right? Like, you know, I mean, most AI models are made in the Bay area, so they tend to just be left-leaning, right? But also the internet in general is a little bit left-leaning because it skews younger than older. And so like all these things like sort of affect models.

20:28But like, it's not just around politics, right? Post-training is also just about teaching the model. If I say like the movie where the princess has a slipper and it doesn't fit, it's like, well, if I said that into a base model that was just pre-training, like the answer wouldn't be, oh, the movie you're looking for Cinderella, you know, it would only realize that once it goes, you know, once it goes through post-training, right? Because a lot of times people just throw garbage into the model and then the model still figures out what you want, right? And this is part of what post-training is. Like you can just do stream of consciousness into models and oftentimes it'll figure out what you want.

21:02Like, you know, if it's a movie that you're looking for, or if it's help answering a question, or if you throw a bunch of like unstructured data into it and then ask it to make it into a table, it does this, right? And that's because of all these different aspects of post-training, right? Example data, but also, you know, generating a bunch of data and grading it and seeing if it's good or not, and whether it matches the various policies you want. Is it help? You know, a lot of times grading can be based on multiple factors, right? There can be a model that says, hey, is this helpful? Hey, is this safe?

21:31And what is safe, right? So then that model for safety needs to be tuned on human data, right? So there's, it is a quite complex thing, but the end goal is to be able to get the model to output in a certain way. Models aren't always about just humans using them either, right? There can be models that are just focused on like, hey, like, you know, if it doesn't output code, you know, yes, it was trained on the whole internet because the person's going to talk to the model using text, but if it doesn't output code, you know, penalize it, right? Now, all of a sudden, the model will never output like text ever again.

21:58It'll only output code. And so like these sorts of like models exist too. So post-training is not just a uni variable thing, right? It's what variables do you want to target? And so that's why models have different personalities from different companies. It's why they target different use cases and why, you know, it's not just like one model that rules them all, but actually many. That's fascinating. So that's why we've seen so many different models with different personalities is because it all happens in the post-training moment. And this is when you talk about giving the models examples to follow this, that's what reinforcement learning with human feedback is is the humans give some examples and then the model learns to emulate what the human is interested in what the human trainer is interested in having them uh embody is that right yeah exactly okay great all right so first half we've covered what training is what tokens are what loss is what uh post training is post training is post training by the way also called fine tuning we've also covered reinforcement learning with human feedback we're going to take a quick break and then we're going to talk about reasoning.

23:03We'll be back right after this. Did you know your credit card points and miles can lose value to inflation? Credit card companies often reduce the redemption value of your points and miles. Now imagine a credit card with rewards that can grow in value. With the Gemini credit card, you can earn Bitcoin or one of over 50 other cryptos instantly with no annual fee. Every swipe at the store or gas pump earns you instant rewards deposited straight to your account. Plus, sign up now for a$200 Bitcoin bonus to kickstart your rewards. Visit Gemini.com slash card today. Check out the link in the description for more information on rates.

23:43Again, if you're looking to invest in Bitcoin, but don't know where to start, the Gemini credit card makes it easy. The Gemini credit card is issued by WebBank. In order to qualify for the$200 crypto intro bonus, you must spend$3 ,000 in your first 90 days. Some exclusions apply to instant rewards in which rewards are deposited when the transaction posts. This content is not investment advice and trading crypto involves risk. The Gemini credit card cannot be used to make gambling-related purchases.

24:13You're used to hearing my voice on the world bringing you interviews from around the globe. And you hear me reporting environment and climate news. I'm Carolyn Buehler. And I'm Marco Werman. we're now with you hosting the world together. More global journalism with a fresh new sound. Listen to the world on your local public radio station and wherever you find your podcasts.