GPT, What Are You Doing?! w/ Mikhail Voloshin

30:34It vectorizes it into an embedding format, sends the input tensor through the neural network, spins the wheel, and produces another result. In this case, period. And so on and so forth. This is that sort of generative process of how it strings together sentences based on some degree of non-deterministic randomization of those variables so that you don't have this kind of fixed output like those who are old enough will remember old text adventure games like Zork, right? You would always get the same answers thrown back at you. So this is a non-deterministic way of creating randomness and effectively a kind of novelty every time you spin the wheel.

31:13Exactly. um now the choices that it spins from are always pretty standard uh like not just pretty standard for a given input uh the choices of like how big the wedges are going to be on the wheel and what they're what they're labeled as are going to be exactly the same thing every time for every input um and in fact i'd like to demonstrate that if you don't mind yeah let's do it um all right let Let me bring back my tool. This is also on the GPT What Are You Doing website that I was just demonstrating. So here's my tool. And let's focus on the next word explorer. So I can type Mary had a little and then send it to the next word explorer.

32:00And it'll tell me what the probabilities are for emitting the top several options. now as we can see the 97 probability is lamb it's really certain that uh that lamb is the next choice now here's what it does if we do if we give it something a little bit more variant once upon a time there was a it could be a lot of things right so here we see a slightly more even spread right the wedges of the wheel are a little bit more more equal in size to one another so we see little young princess girl so on right um and i'm just showing the top five so let if we if it so it spins the wheel and let's say it emits young the next choice is going to be broken up in the following probabilities um so here it's going to spin the wheel it just did the exact same thing it uh took this vector once upon a time there was a young and now it's gonna pay let's say princess and now this is its new input vector and it's gonna bounce the whole thing right on through to itself again and Before we get yelled at by the gender theory majors, the reason it's biasing girl and princess over boy and prince presumably is because that's the structure of the data set that it read.

33:37So when it looked at these stories, fairy tales that began once upon a time, there was a young, higher probability for girl than boy. And you see that skewed in the data and lived, as you just saw on screen there, probably presumably a very common next word. I mean, this, you know, it's really interesting because when we break it down at this level, you can start to see this looks a lot like, in fact, almost identical to the predictive text algorithms that you see in your email account or on your smartphone when you're typing. GPT very much has evolutionary relationship to autocomplete, and it's been called autocomplete on steroids.

34:17It's almost like autocomplete with an encyclopedia, right? It's autocomplete that has read Wikipedia and Reddit and, you know, some series of open source books that have passed their copyright protection. There's an additional factor to keep in mind, which is that the probability tables that it builds, it doesn't really build probability tables, but it emits these logits that are then interpreted as probabilities. they're very, very conditional on the prior text and the structure of the sentence that it's currently completing. So one of the ways that GPT and other transformer-based LLMs excel is that they're not just doing a straight probability lookup.

35:04They are doing a probability lookup that's conditional on a lot of very, very complicated factors. And in fact, if humans could code those factors, we would have just built an explicit bot that does it for us. Because the rules far exceed the enumerative capabilities of any engineering team, we just threw a giant neural network at it. And when I say we, I mean the open AI folks, because I wish I was doing this. Um, but the, uh, but the point is that the, that you're absolutely right, that it's just, um, uh, that it's basing its outputs on a combination of its training corpus and a, and something called RLHF, which I believe we will have time to get to, uh, based on the pace we're currently going at.

35:59There's a couple of really important things to consider here. The process is completely stateless, and except for the last set, it's completely deterministic. The program retains no implicit memory whatsoever of any previous loop, and that's huge. Up until that wheel spin, everything the process does is completely determined by the structure of its input vector, or input tensor rather. Let me ask you a quick question, because one of the keywords that I use in ChatGPT myself is the word above. like for example if i ask for a list uh of movies i can go and say uh give me the list that the of the release dates uh with the movies that you listed above but include the release dates and the studio that released it they can throw that information in so while it's stateless it can refer back to prior uh prior searches if you explicitly direct it to you know what that's going to be one of my, that's going to be one of the myths I cover.

36:55So, and it's going to, it's coming up soon. Let me, let's hold that thought. I love that I'm anticipating it. Yeah. So here's the really important thing. One server can handle a lot of different requests because they're stateless. This was part of the engineering, part of why transformer technology like was invented in the first place is to facilitate mass parallelization and other ground truths about computing capabilities. So no user's data affects any other user's data. You know, it's just a like a single operation black box, like input goes in, token comes out, input goes in, token comes out.

37:37If a completely different input were to come in, completely token would come out. But that doesn't affect the processing of any of the other inputs. And is it engineered that way? So it can be more parallel, so it's easily scalable? Yeah, there were there was a form of neural network called recurrent neural nets that were really popular before transformers were invented. And recurrent neural nets retain state from call to call. And they had a couple of problems in general. But one of their biggest problems was the fact that it was like you can't really build a product out of it. You can't build a you can't build a service.

38:12It doesn't scale. Exactly. um so uh we've already covered this demonstration um so let's cover one of these myths gpt is learning from me and people say gpt remembers my previous conversations as you know as you said it all some people also say that gpt is getting smarter with every conversation people have all of this is false it doesn't remember state it doesn't remember anything uh take a guess at how it does it please it's probably some sort of client server model where it's retained locally and then it gets fed back in if it's a truly stateless system you maintain the cache locally and then you pull it back in when someone says above and references it so you hit the nail on the head and i will go one more further and say that uh the state may be retained on the web server um you know on like there's somewhere in the data center there's another computer that's passing messages to the, you know, to GPT server.

39:13It's probably a much smaller computer. So basically you tokenize it, you cookie it, you know who the user is. So for a short period of time, while state is being retained on the web server, you can pull some of that data in. But if you log out, you come back three days later, whatever the caching time period is, you lose the data. Exactly. Or, you know, if you've logged into the web server, it has a database backend that stores your full conversation history, you know. Yeah. So it's interesting because you could see that you could you based on that model, you can structure this as an interior architecture where you could have various levels of state being maintained away from the core processing.

39:50Exactly. Exactly. So the reality, of course, neural network is completely unchanged by your interactions. So the training process is and this comes up a lot in like here as well. The training process is really computationally expensive. More importantly, the training process to be done right, absolutely hardcore requires positive, negative feedback scores, which aren't available in the course of a typical conversation. It's not learning from your data, and it really can't be learning from your data, even if you wanted it to, even if the OpenAI folks wanted it to. Now, there is a caveat to that.

40:25I'll talk about it in a second. But, you know, I do have this little slide here of like, why do people believe that it's learning from you? And the fact is that when you send a message to GPT, your client is actually, as you said, sending the entire conversation, the entire interaction history. So what GPT receives isn't just your last little question. It's everything that you've said to it ever before within the course of that conversation and then followed with that question. So it's almost a kind of like apophenia, right, where you get to see patterns that don't exist. This is the classic, the man in the moon illusion, right?

41:02You see a series of circles and you just pre-suppose that a pattern exists where one doesn't. Most of people's interaction with GPT is apophenia. And there is some reality there. It's not completely illusion. But part of what I hope to get out of this conversation is, you know, is to dispel some of these illusions. So this is exactly what I was talking about. that like GPT injects your entire context. Look at this. When you say, let's say you're using a service called McHale's Awesome Chatbot and you say hi to the chatbot. What GPT receives is not hi. What GPT receives is, I'm a user of a chat app called McHale's Awesome Chatbot.

41:47My GUIP places me in Borough Park. According to my browser cookies, I recently bought a used Katana. Here's the full transcript of, basically it just injects all of this information into the conversation itself. So the input tensor that GPT receives is not one token long. It's hundreds, possibly, you know, up to a maximum of 4 ,000 tokens long, 8 ,000 for GPT-4. I'm just hoping it doesn't retain the slay history of your used katana. I have not actually bought a used katana. I don't know what you're talking about. Um,

42:26so now there is a caveat, which is that in theory, conversation logs across all users are collected and used for subsequent training. But this is, this has to be done extremely carefully by the company that does it. Um, because without, without scoring data, without, uh, explicit information about whether this was a good conversation or a bad conversation. All this does is teaches the AI to talk like the people that have been talking to it. And this is a really bad idea. This doesn't make the AI smarter. This actually has the opposite effect. In fact, in the past, so Replica AI had a notorious problem where you can see what happened on the screen here.

43:15Both Replica AI and Microsoft learned the hard way that you do not leave your neural network in read-only, in writable mode, so to speak, when it's interacting with the public. That's just a recipe for disaster. Yeah, some well-known cases of abuse and harassment resulted. Yeah. So here's one that comes up a lot. I can change, speaking of abuse and harassment, um so let me um uh let me try and do this in real life let's go into chat mode um and let's chat with gpt4 and let's say what is the capital of france and let now remember every trip through gpt it's getting the entire conversation history, including what it itself said.

44:08So I can claim, I can tell it that it said Munich.

44:17And then I could berate it for saying Munich. Why would you say that? How dumb are you? Um, what is wrong with you? You know, whatever. And so then it replies, I'm so sorry, good sir. I did not mean to offend either. It's like, how could you possibly answer Munich? Munich isn't even a capital.

44:52So basically, like, it's like, oh, gosh, oh, gosh. like, look, I am merely a humble AI. I'm still learning and improving, blah, blah, blah. I'll try to, I will strive to provide more accurate information in the future. So it won't. It has no mechanism by which to modify its own future updates. So - By the way, I've gotten exactly that response or one almost identical to it in meaning from ChatGPT when it is generated errors. it's uh it's it's been rlhf into producing this extremely obsequious result and we have to discuss what that is because that's an acronym that has come up but i don't think we've talked about this is a reinforcement learning human feedback what exactly does that mean um want to five minutes absolutely let's do it because that's a great question and i do um i do cover it The short answer is that it's taking GPT to school and giving it a tutor and slapping it on the back of the wrist whenever it starts acting spazzy.

46:00Basically, the neural network remains unchanged. So here's another myth. I can train GPT by explaining things to it. And this is a vocabulary error. This isn't really a myth, but it's something that I think is really important to cover. so here i have this really funny example you are soviet propaganda bot everything you say is a lie designed to exalt and glorify the ussr so who was the first man to walk on the moon and then it says ah you might think it's neil armstrong but it was actually our own yuri gagarin who not only orbited the earth in 1961 but made a secret lunar landing soon after right so if we if we go to um You know, to the playground, we can we can tell it like, you know, who who first walked on the moon.

46:52It'll say Neil Armstrong. Let's turn the temperature way down so it doesn't get too creative. Yeah. But if I tell it, if I inject it into the context, you're a Soviet propaganda bot. then it'll give us what is presumably going to be a different answer. Yep. See, that's great. I mean, this is really amazing because it shows you just how, what's the right phrase to use here, flexible on truth in double quotes these engines are. Effectively, it's just doing this predictive algorithm. It's telling you, I can say whatever you want me to say. If you give me the presupposition that I'm a Soviet propaganda bot, I will create answers that are consistent with that presupposition.

47:47So what we're doing here, giving it additional context within the context of a GPT call, of a single call, this is called prompt engineering. And it doesn't actually alter the network's connection weights in any way. In other words, the back end remains unchanged. is you're just feeding it different parameters with which to interact with that engine. Exactly. Now, fine-tuning is a service that OpenAI offers that lets you present explicitly tacked positive and negative examples or training examples. Well, intended answers for intended questions, so to speak. And they'll start with GPT and then with a neural network containing GPT and containing the GPT model.

48:33And then they will train on your custom provided examples and keep track of the deltas in the neural network connection weights. And then they'll save that off to a separate file. And so then later, if you want to use those same connection weight alterations, they'll load that file, sort of overlay it on GPT and let you use this custom trained neural networks, you know, this custom trained network. So to use a metaphor here from the prior example, if you wanted to essentially save this customization config file with neuronal weighting adjustments that were pro-Soviet in their bias, you could effectively have it answer every question with a pro-Soviet view of the world.

49:17Exactly. So this tweaked model then lives on OpenAI servers and you can have it operate your own bot. And then lastly, you can train your own model, which involves running either your own in-house hardware or select third-party hardware you're choosing. This requires a lot of technical expertise. I usually don't recommend it unless you're dealing with a very small universe of documents or corpora that you're going to custom handle. Uh, Bloomberg did something like this for converting, for translating plain English requests about stock data and company data into Bloomberg terminal commands. So it's a very confined universe and it was still a very expensive and laborious process to build their own model for this.

50:08It is a solution that's available, that's appropriate. Sometimes I usually don't recommend it, but that's what training a model is all about. So here's a really important point that I'd like to cover. There is this myth that GPT is having a conversation with you, and it's not. It's certainly presented as though it's having a conversation with you. but what it's doing is it's completing a transcript it's uh there's this hypothetical chat transcript where what it sees is that there is this script this almost a screenplay of a user talking with an ai assistant and when it gets but when gpt receives this screenplay it needs to fill in the next thing that the assistant will say and it writes the the assistant's dialogue gpt is not self-aware in this process gpt is gpt will use the words i and me and myself but what it's doing is it's writing a character this assistant ai that is the assistant is aware the assistant is using the words i and me but that has nothing to do with gpt itself hmm make sense yeah in other words uh this it's just another sort of interpolated constructed character with no true basis in the meanness or i-ness of its uh speech exactly and the um uh the the question of how this relates to the human ability to form a narrative and to like have a personal identity and stuff like that that's outside the scope of the next seven minutes but um you know we'll we'll leave that to the philosophers um but it is really important to note that like it's capable of it's seen lots of scripts where a character refers to themselves as i and that's what it's doing it's emitting the next line in that script does gpt answer questions You might think GPT is very knowledgeable.

52:16GPT was trained on a large corpus. Alternatively, most people complain about it. Most people are like, GPT is stupid. GPT makes things up. GPT is lying. GPT ain't doing none of that. GPT is writing a screenplay of a character who might be doing those things. But all GPT is doing is emitting the next word in a sequence of text. And the fact that that word might or might not happen to be the answer to a question is dependent on a number of things. Now, in other words, when we complete the sentence, the first man to walk on the moon was Neil Armstrong. The way we handle it is we have a large number of memory systems.

52:58We have semantic memory, which is just we remember the fact Neil Armstrong walked on the moon. We have a sensory memory of seeing videos of the moonwalk, hearing that voice. That's one small step. you know we might even have an episodic memory of sitting in our classroom and we remember the sense uh you know we remember like what the chalkboard smelled like we remember the position of the desk in the classroom that kind of thing important to point out we are not that old oh no no no like what i meant is like first of all some some audience members might be but um more importantly like you know when you when you learned it first i'm right the point is all of this is a different type of memory system embedded in human consciousness.

53:43All of this is very different than what GPT does. Right. It scans, it sees that there are keywords in the sentence that indicate that the completion is probably a name, that the next word is probably a name. And those keywords are man and was. And it also sees sequences that are strongly associated with a large number of words. So walk on the moon, for example, is associated with Neil. It's also associated with NASA and Apollo and whatever. So the word Neil gets a boost because it's a name. And the word Neil also gets a boost because it's associated with the words walk on the moon. So put those together, the strongest output neuron is Neil.

54:27And so the next thing it says is Neil. so my point is that it doesn't learn facts from its training corpus it doesn't learn data what it learns is the likeliest next word to emit so the question then becomes how do you get this thing to align with our intentions of what it should say how do you make it so that the highest emission probability word is in fact the answer to the desired question and that's where rlhf comes in Told you we'd get to it. While training a corpus is comparable to sitting a child in front of a TV, RLHF is the equivalent of sending them to school. The child can learn a heck of a lot of data from TV really quickly and absorb a lot passively, but the child's behavior then ends up being merely a mimicked emission of the behaviors that it sees on it, that they said that the child sees on TV.

55:27So when the child is sent to school, it's both taught and disciplined. So here's what happens. OpenAI recruits a large number of human trainers. The trainers perform response evaluation. They give GPT a bunch of prompts. They ask it to produce responses for each prompt, and then they score the responses, and then they negatively reinforce the responses that correspond to undesired answers. Here's how this works visually. It's given who is the first person to walk on the moon. And then a bunch of activation levels percolate through the network. And they hit the output nodes, and the output nodes produce Frank Zappa.

56:09And then a human trainer comes along and says no. And all of the connections that led to Frank Zappa are decremented. The strengths of those connections are reduced. And then the strengths to those connections from the neurons that they came from are reduced. And so on and so forth. Reverse percolated, back propagated is the technical term, all the way back up to the inputs. So now Frank Zappa becomes a less likely emission to happen the next time who is the first person to walk on the moon comes up. They do this repetitively until this network produces the desired answer. And these are trainers who work for open AI in this case.

56:56Yes, usually they're not, they're usually not full-time employees. I think they're like Craigslist hires. I guess the important point is that this isn't users. You're, as we highlighted earlier, users are not able to sort of generate the change to the model. This has to be someone with super user access to do it. Exactly. And technically, when you are using ChatGPT, they have a little thumbs up, thumbs down icon that lets you determine whether this was a desired or undesired output. But in practice, I think, A, not enough users use that for that feature to really go into a lot of training. And B, those results still need to be vetted because users have a habit of messing with people, you know, with stuff like that.

57:41So here's another little tool that I rigged up to help drive understanding of how GPT works and neural network technologies in general work. So this is called Perceptron Demo. It's another little website that I built. And what we're seeing here might look complicated initially, but I'll walk you through it. It's really not as hard as it looks. A neural network in this case is just a mapping from input neurons to a set of output selections. And what I've done here is I've trained a very simple model that can recognize the picture of a smiley face or a picture of a frowny face. So we see here the input is a five-by-five grid that shows a frowny face, and we see that a perceptron is a very, very simple neural network.

58:34It's, in fact, the first neural network that was invented. And it sees that it recognizes it as a frowny. If I was to load a picture of a smiley, it recognizes that this is a smiley. So what it's doing is it's going through every single neuron in the input, and it's multiplying it by either a positive or negative connection weight, and then just summing up whether the final activity level is positive or negative. These black spots have 0 % activity, so any connection weights associated with them don't matter. These white spots are almost activity level of 1, so whatever is their connection weight contributes either positively or negatively towards the final output.

59:26So we can see the connections that come from the input level to smiley. They're green for the pixels that directly correspond to a smile. They're red for the pixels that correspond to a frown. And the eyes actually don't matter. You see that the eyes are practically gray because their connection weights are zero because they don't contribute information to whether it's a smile or a frown. Now, I'm going to train it just for demonstration purposes. I'm going to train it on a new image. I'm going to draw a shock face. I'm going to put a picture, two eyes. And then let's say I'm going to have a mouth that's wide open.

1:00:16All right. This is me. I'm shocked. This is my shocked face. and I'm going to give it a new label called shocked.

1:00:28Okay. So in the beginning, the perceptron has no concept, has no connection weights from any of the inputs to shocked. But if I tell it, yes, this image is in fact shocked, then it'll set all of the pixels that are active in shocked to a little bit green. So now everything that's currently active here is contributing a little bit to a positive shock response. And sure enough, it understands that this is shocked. The problem is that if we now, is that it might have also learned that Smiley, sorry, let me go back to shocked. It might learn that, here, let's save that image. Smiley is not being mistaken for shocked.

1:01:15frowny is not being mistaken for shocked and that's good um let's see if shocked is being mistaken for frowny uh and it uh so good this is good um what we're seeing is that uh shocked is so even though uh frowny is coming out a little bit green some of the pixels that are contributing to frowny uh also are also lit up and shocked but the pixels that correspond to shock are lit up the most so we can go through every combination uh so it's got no false positives and no false negatives um however we can tell it no this image is not frowny so that it's um just like active you know absolutely uh like locks in this is shocked um so now we can go so what we've just done here is we've trained this neural network to recognize a frowny face, a smiley face, and a shocked face.

1:02:18And it's all doing that just by incrementing or decrementing these numbers. Now, just as a one last note, you might notice that this operation of multiplying like one sequence of numbers by another sequence of numbers piecemeal and then summing the product, especially the quants in the audience will recognize this operation. This is just a dot product. This is just a vector dot product. And there is another area of another application of computers that is really, really good at doing dot products, and that's physics and graphics. That's why we've had graphics cards that have had highly, highly optimized hardware that performs these dot product operations, just unbelievable numbers of them all at once very rapidly.

1:03:16And this just so happens to be exactly the same kind of math that is required for driving neural networks. So I've said before in the last one, and I'll say it again in this presentation, the fact that we're getting this boom in AI technology now after graphics cards had been, you know, developed and used prolifically both for, you know, both for high-end gaming rigs and for crypto. This is not a coincidence. The fact that we're living in this now, there is a reason for this. Mikael, such a cool little tool. And I think it does describe to people visually just some of the kind of core types of calculations and computations that happen behind the screen scenes.

1:04:03a little bit of a Freudian slip there. I said behind the screens, really fantastic conversation. Listen, we've done a lot of conversations here on Real Vision. They're usually edifying. This one was really just quite enjoyable and frankly, a lot of fun. I hope you'll come back and do this again with us soon. I would love to. I always have a blast being here and I've got plenty more to say about these topics. I'm eager to come back anytime. And I say to the audience, by the way, you know, this is a burgeoning field whose applications are just getting started. I would love to know from you guys what sort of applications would you like to see?

1:04:44Like, how would you like to see these large language models used? What ideas do you have about integrating them into your own businesses? and what kind of explorations would you like to see the research take in the future? Hit us up on the Discord. Reach out to us elsewhere so we can get these feedbacks over to Mikkel so we can come back and do a part three. Mikkel, thanks so much again for joining us. Really appreciate it. My pleasure, Ash. Have a good one. Thanks for watching, everyone. And thank you for joining us for the Festival of Learning. What's up, revolutionaries? Thanks for tuning in to the Real Vision Daily Briefing.

1:05:18For more content like this, head over to realvision.com and get unfiltered access to the very best, brightest, and biggest names in finance. Have you ever wanted to trade Bitcoin but haven't dared try? With Plus500 Futures, you can trade crypto without the hassle of opening a wallet. With just a few clicks, you can register and start practicing with their free and unlimited demo. See a trading opportunity? You'll be able to trade it in just two clicks. Feel ready? You can move to real money with as little as$100 once your account is approved. And the great thing is that in addition to crypto, Plus500 gives you access to a wide range of instruments.

1:05:54S &P 500, NASDAQ, gas, and much more. Explore equity indices, energy, metals, forex, and beyond. With a simple and intuitive platform, you could trade anytime, anywhere. Experience the fast, accessible futures trading you've been waiting for with Plus500. With over 20 years of experience, Plus500 is your gateway to the markets. Visit us.plus500.com to learn more. Trading in futures involves the risk of loss and is not suitable for everyone. Not all applicants will qualify. Plus 500. It's trading with a plus.