How Generative AI Works, as Told by a PhD Data Scientist

0:00Season three of the Built Right Podcast is right around the corner, but we've got one big change coming your way. The Built Right Podcast is now the Talking AI Podcast, and we've got a lot to talk about in AI. In the Talking AI Podcast, we'll be having in-depth conversations with both AI experts and early adopters of AI. That way you can understand how the technology works and how early adopters are beginning to implement and, more importantly, get value from AI. Our guests range from AI research scientists to founders of AI products to industry leaders putting AI to work in their business. While you're waiting for season three, go ahead and subscribe on your favorite podcast platform so you don't miss an episode.

0:41And make sure to leave us a comment about the AI topics that you want to hear about. So get ready to talk some AI in the new Talking AI Podcast, coming your way August 6th.

0:58Welcome to Built Right, a podcast by Hatchworks where we help you learn to build the right digital product the right way. In each episode, we'll deconstruct the layers of successful product development, break down popular trends, and offer real advice to help make sure your product is Built Right. We may not have all the answers, but we've built a lot of digital products across a lot of industries, and we've seen a thing or two. Let's get into it.

1:33Welcome, Built Right listeners. We have a special one for you today. Our guest is a PhD data scientist, and he's going to help us make sense of generative AI and how it all works. And our guest today is Nick Vassiloglo. And Nick, I probably butchering the name, Nick the Greek, I think is what you also referred to you as. And he is the VP of Research ML at Relational AI. Like I said, master's and PhD in electrical and computer engineering here from Georgia Tech and has founded several companies, has worked at companies like Logibox, Google, Symantec, and even helped design some of Georgia Tech's executive education programs on leveraging the power of data.

2:17But welcome to the show, Nick. Nice to meet you, Matt. Thanks for hosting me. Yeah, excited to have you on. And relational AI, for those that don't know, is the world's fastest, most scalable, expressive relational knowledge graph management system, combining learning and reasoning. And for those of you thinking, what the heck is a knowledge graph? We will get into that. Plus, we'll get into how generative AI actually works, as told by a real PhD data scientist who's been doing this stuff way before ChatGPT was even a thought in somebody's mind. Plus, stick around. We got Nick's take on what are going to be the most interesting use cases with generative AI in the future.

2:58And he's saving these. I haven't heard these either, so I'll hear him for the first time. So really excited to get into these. But Nick, let's start here. What is the difference between, we hear generative AI, it's the hot topic right now, but generative AI and just regular AI. What is the difference? What makes generative AI special and different? It's a very interesting question. For many years, the emphasis, the way that we were separating, whether you call it, machine learning on AI, was on the depth of the models. When I started my PhD, we were working on something that we would call shallow models.

3:39Basically, you can think about it looking at some statistics. The decision tree was the state of the art, which meant, okay, I have this feature. If it isn't greater than this value, then I have to take the other feature and the other feature and come up with a decision. That's something that everyone can understand. Then deep learning was the next revolution somewhere in the 2010s it started. and it started doing more, let's say, complicated stuff. People are still trying to find out why it's working. They cannot understand exactly the math around it. And then the next revolution was, so we had these models that they were pretty powerful, but we didn't know how to scale them.

4:27We didn't know how far they can go, and that was the revolution that basically OpenAI brought that they realized that you can take this new cool thing called the transformer where you can feed it with a lot of data and do this cool thing where you are trying to predict the next world and basically come up with what we have right now. It took several years and several iterations. but I think the difference between what we used to call AI and what we call AI right now is the focus on the language. I mean, if you had read about Chomsky and others, a lot of people considered that the human intelligence has to do with our ability to form languages and communicate.

5:21I mean, you might have heard that, you might remember as a student. what makes humans different than other animals. The human brain is the ability to form languages. And I think the focus on that made a big difference in what we have right now. Quick break in the pod. If you're listening to this podcast, chances are you've been thinking about how to actually use AI inside your business. And that's exactly why we built the AI Opportunity Finder. It's a free tool that helps you uncover high impact, tailored AI use cases based on your business, your goals, your pain points, and your industry. No fluff, no generic use cases, just real ideas that fit your business and they're ranked by ROI potential.

6:08It takes about three minutes to run and it's like having your own personal AI strategist for free. If you want to try it for free, check out the link in the show notes or go to hatchworks.com backslash AI dash opportunity dash finder. Previous was more like a decision system. Now we're focusing more on the reasoning side. So I would say this is the shift that we see. And that's, I think, part of the interesting aspect of it is, you know, in the past, it's like the models, they were trained for very specific tasks in a lot of ways. And now you have this concept of these foundation models, which that's, you know, what a lot of the large language models are built on.

6:47And but now to your point, it's almost kind of like getting to where how the human brain works so they can tackle these disparate types of ideas and solutions and things like that. This concept of like a foundation model, what is that? How does that start to play into these concepts of like large language models, LLMs that we hear so much about? So let me clear that up first. You know, the foundation models and the language models are basically the same thing. You know, sometimes the foundational models were, the term was introduced by some Stanford professors. They were trying to, you know, kind of, it happens a lot in science.

7:32You build something for something specific and then you realize that it applies to a much broader, you know, class of problems. And I think that was the effort, that was the rationale behind renaming language models as foundational models, because they can do the same thing with other types of data, not just like text. Okay, so you can use that for proteins, you can use basically for whatever represents a sequence. okay so um you know as i said in the past a lot of effort was um was put on collecting labels and do what we call supervised learning uh the paradigm shift here was in in what we call self-supervised learning that was a big big um plus something that that brought us here this idea that just take a corpus of text and try to predict the next word and if you're trying to predict the next word you're basically going to find out the underlying knowledge and ingest it in a way that you can make it useful of course that's what brought us up to 2020 that was the GPT-3 where we scaled, but there was another leap that charged GPT that in the end, it did require some labeling because you had the human in the loop.

9:09It's not exactly labeling, but you can think about it as labeling because we have a human giving feedback. And then that brought us to charge GPT. Now, the heart of language models or foundational models is something called the transformer. It was invented in 2017 by Google, actually. It was an interesting race. OpenAI, there was like a small feud between OpenAI and Google. So OpenAI came with a model. All of them were language models. Everybody was trying to solve the same problem. They came up with something called Elmo, and Google came back with Bert from the cartoon, from the Muppet Show, I think.

10:05And then, so Bert was based on the Transformer. Okay. Then OpenAI realized that actually Bert is better. Okay, that's an interesting lesson. They didn't really stick, oh, this is our technology, we'll invest in that. They saw that the transformer was a better architecture. But then they took BERT and they actually cut it in half. Okay, and they picked, actually let me put it that way. They invented, Google invented the transformer, which had an encoder and decoder. And they picked BERT was based on the encoder architecture. They took that half. But then OpenAI came and picked, no, we're going to work on the other half, which is the decoder, predictive text.

10:52And, you know, they spent three years. They did see that the more data you pour, the better it becomes. Okay, so that was their bet. And they ended up with GPT-3, GPT-2 and GPT-1 to 3, the sequence 3.5 and 4 later on at GPT. and it was kind of like an interesting race where things basically started from Google but OpenAI ended up being the leader over there and everything they built was open source, right? Everything Google built so they were able to... Everything is actually open source I think up to GPT, even GPT-3 there was a published paper it's very hard if you believe that you're going to get a secret source that nobody else knows.

11:43I've never seen that play in machine learning. The scientists want to publish, they want to share knowledge. I think as the model started to become bigger and bigger, they didn't with GPT-3, I don't think they ever opened the whole model, the actual model, but they gave enough information about how to train it. There's always some tricks that over time, even if somebody doesn't tell you as you're experimenting, they're going to become public. Okay. So, yeah, that was never the issue. I don't think... Yeah, they are a little bit cryptic about after 3.5 and such details. But in my opinion, the secret sauce over there is not exactly on the model, but it's on how you scale the serving of the model, We're going to talk about that later.

12:37This is the secret weapon of OpenAI, not necessarily the architecture, but, you know, the engineering behind that. Nice. Let's keep going on the transformer side, because getting under how these, you know, GPTs work. Basically, you mentioned that it's serving up the next word, the next word. It's not looking at it like a whole entire sentence, right? It's this concept of tokens. But how is it like actually thinking through that and structure of language and something you think a computer wouldn't be able to do? It's now doing very well. Yes. First of all, it's always this. As I said, the transformer has this encoder decoder architecture, which means that there's one part that looks into two directions back and forth.

13:23As it's processing, it looks both ways. Like this token is affected by the other tokens, but this token in the middle is also affected by the ones before and after them, what it's going to be. So that's like the encoded. The decoded architecture, you're only looking back because you're not looking in the future. We can talk more in detail. There's a lot of papers and a lot of tutorials that they actually explain that it's not always easy to explain it without graphics here. But the key thing over here is that, you know, let me go a little bit back. The first revolution actually came by Google was Word2Vec, where they realized that, you know, if you give me a word and I try to predict that word by looking five words behind me, and five words after me.

14:22Okay, that was a simple thing, like a small window. And I tried to create a vector representation. They realized that, you know, I can take words, make them as something like continuous vectors, put them in space, throw them in space. And I would realize that, you know, that representation would bring words that are semantically similar together. Okay. And there was this other thing that if I, you know, if Paris is here and France is here and London is here, then I can take the same vector, put it here, I can find, you know, England. So they realize, for example, that if I place all the capitals and all the countries, I can just take the vector that connects the first and the other and it's translated to the next one and it's going to – or if I take the distance of the vector between man and a woman, take that, then take the word king, add that to the king, it's going to take me to the queen.

15:25So basically people started realizing with a simple word to vector that you can take words, represent them as vectors. Let's think about two-dimensional vectors like on the plane, but it's not two, it's like 512 dimensions. It doesn't really matter. The concept is the same. that their distances in space, the way that they're placed in space has. Sorry, semantic meaning. Now, the next problem was, this is great, but we do know that words actually change meaning based on their context. Okay, so... What's an example there? Yeah, so for example, an example would be, when you say when you say flour, well let's pick now I'm a little bit stuck but You had one about boiling the a person's boiling a what and then if it was like an engineer it had different context.

16:33Yeah, you can boil an egg or an engineer is boiling I don't know a substance but it could be They're trying to boil the ocean. So when you say, for example, the bed, it can be, you know, something different when you talk about, you know, a house, a bedroom. But if you talk about geology, it means something completely different. OK, so what they realized was that that vector that represents the world shouldn't be universal. It should really depend on the surrounding words. So this vector representation when the surrounding words are this one, it has to be this, and it will also have different relationships.

17:27And it should be different when it's around different words. And that was basically Elmo that was this idea. It's called contextual embedding. So this vector representation, like this, let's say these two dimensional representations called embedding. So this is this was actually one of the biggest revolution of deep learning that we're taking discrete entities and we could place them in space as a continuous vector. So we'll take something that was discrete and putting on a medium that it's continuous. OK, continuous and multidimensional. So the first idea of the before the transformer, which is an aerial version of a transformer, the first idea was that, OK, if I see a text, I will be placing these words, you know, on different places in space based on what it's around.

18:21them okay and um and the next thing that so basically what happens is you're taking the words on the first level you you look left and right and you create you know embeddings you know you create you put them in space then you take that and you apply that again and again so the transformer actually starts in levels one level after the second level it has multiple i don't know exactly the numbers but it has different relevance so you can think about that as basically a rewriting okay so that's why it's called transformer so you have a sequence of words and you start you know rewriting to something else something else that else you know so when people actually people have done this experiment they're taking the the transformer and they decompose it and they see what are these things that you you know the transformer does in in different levels and they've actually realized that it starts inventing grammatical rules.

19:21It starts like identifying what is the subject, what is the object, what is the verb. Okay. It starts identifying that something is an adjective or not an adjective. It starts, you know, taking, you know, words and converting them to something which is synonym, maybe, you know, something else. And that's how the reasoning starts. Like, I can give you, if I give you a sequence of words, you know, Nick, I don't know, lives in Atlanta. You know, he can, he knows that Nick, I don't know, is Greek. Okay. So he can say the Greek lives in Atlanta. and that can affect the fact that, you know, and then you can say he goes to the store to buy and because now you know that he's Greek, he lives in Atlanta, you say fetuses, for example, okay?

20:21Because now he starts, you know, the transformer starts taking different paths. Like it starts exploring, you know, what are synonyms and, you know, if he leaves, it means he goes to the store, you know, he goes to the supermarket if he leaves there. So it starts, all this information is ingested in the transform after seeing, you know, endless pages of text and, you know, where basically there's, there are reasoning paths. Like it does this on your own. Of course, because there's so many reasoning paths that can happen. Sometimes it can hallucinate. Okay. So I can say Nick buys, I don't know, so lucky because he agrees he is Greek, which is possible.

21:02but there might be somewhere else some other information that says Nick hates Vlaki and you know the language model doesn't but it's a probable event since you know Nick is Greek anyway I'm just giving a simple example over there but that's kind of like the power of the transformer that at every stage it starts rewriting things again and again and again and it explores possible very possible uh very likely paths you know highly likely and correct me from wrong what what you're talking about here is this kind of the the difference in evolution from structured data to unstructured data because in the past we had very like defined tables columns associations to things is this kind of getting to that concept of unstructured data where it's like the vectors and well the problem with with the structure that with the systems before that everything was like it was very discreet and unless you had seen before the word nick okay followed by that exact word.

22:03It was, you know, if you think about all the variations, like Nick spelled with K, Nikolaus, Nick Vassiloglu, I don't know, think about it, all these things. Because now they're in a continuous space, okay, that's what makes the difference. It's possible for the system to create an internal rule, if you want, or internal path about things that are kind of similar, okay? Okay, so it doesn't have to be Nick. It could be Vasiloglu instead of Nick. Or it could be the guy who lives at, I don't know, say my address. You know, it's the same thing. So because all these things, I think it's public, you can find it.

22:42Because all these things that are semantically equivalent, and before you had to express them in, I don't know, a hundred different discrete things, and you had to see them exactly in that order in order to find a common path. It says, okay, this class of entities that can be represented with this vector, they are very close, can be followed by this class of entities that can all compress them in a constellation of vectors. It can lead me to something else. That's why you see the language model when you go to OpenAI and you say regenerate. What it does, it can generate the same thing, the same reasoning path by using a little bit different words or, you know, where's the semantical equivalent.

23:31Okay. And now the thing is that it can do that in this incredible memory of like, I don't know, up to 32 ,000 tokens. So even if you're saying that Nick is going to buy something from the store and it will predict that it's FETA, it's because it has seen 10 ,000 tokens before that Nick is Greek. He's hungry. He's having a dinner party and get you over there. Okay, so because when it was trained, it has seen sequences that in the span of 10, 20 ,000 token, you know, nick associated with party, food, restaurant, you know, leads you to FETA. Yeah, and when you say a token, that's basically either a word or a couple characters, some like small variation that it's breaking it down into.

24:26The token is basically a trick. You know, we could have used, it's like, you know, I think all these models have about, I don't know, 30 ,000 tokens. So they realized that we can break all possible, like, you know, with 30 ,000 tokens, you can, I mean, you can use character level. Okay, every word can be decomposed to characters. but that would have made that would have made you know the language model extremely big and inefficient so it's like a trick because we kind of like trying to find out it's a compression that we're doing we could have gone with syllables because syllables are also finite and make all the words now we said you know look because there are some combinations of letters that they are so frequent we don't really need to decompose them all the time we know exactly what they mean.

25:18So it was a clever engineering trick. It has to do with the language. It's related to the language. It was like a statistical, a better statistical analysis of the language. I mean, to put it that way, if we were inventing a language from scratch, we would start with tokens, you know, and maybe not necessarily letters. that's interesting and so we've talked a lot so far about language as the the thing at play here but like you can use this generative ai and all this new technology and advancements with different modalities like images whether you're generating images or whether you're understanding what an image looks like and voices all kinds of different things that play here how does that work different when now language isn't necessarily the output?

26:12Is it looking at the pixels in a way and then association? There is a visual language over there. There's the visual transformer which tries to predict blocks of image. There's also the diffusion models which is something completely different. So for example, diffusion models, we see them only in images. We don't see them in text that much. although there's been some efforts but the transformer it turns out that it behaves equally well for images but when you talk about a token in vision that's kind of like a block of pixels I don't know 16x16, 32x32 this is something we knew from before like even in the days of image compression they could take parts of the image and compress block by block.

27:08But I want to make something clear for your audience that language is a way of expressing knowledge, but it's not knowledge. The fact that I can come and tell you something, I can go and read quantum mechanics, I can take a passage, I can recite it for you, it doesn't mean that I know what I'm saying.

27:41And that's where the hallucinations are coming into play. So we don't really have direct access to knowledge. It's a language model. It's not a knowledge model. And there's been some effort right now to do the same thing. If we could start, if there was a universal knowledge graph, that I could take and say that from this token of knowledge, I can go to that token of knowledge through that relation and do reasoning. Maybe we could train a knowledge model, let's call it, or a foundational model that we know that whatever it says, it's accurate and correct. But language is a possible path over knowledge.

28:29It doesn't mean that it's correct. Okay, so it doesn't have to do that. So language models are always going to hallucinate and make mistakes, not because there are errors into what they were been training for. The data sets are pretty well curated. Obviously, they will contain misinformation and errors, but the reason of hallucination is not really the errors in the raw text. But it's on the fact that this is a possible expression, you know. The same way that, you know, like you are a fiction writer, author, and you can write. Like you see things in life and you write a different version. Like take one of my favorites, like Da Vinci Code, okay?

29:20Like when you read, that's what I like about Dan Brown. Or take about Game of Thrones, for example. If you think about Game of Thrones, it has elements of truth from the human history. You can see the, let's talk about it, because that's probably what most of the people know. There's like the Persian Empire, or you can see the English history or the Greek, or there's some of them, like you can see elements of that in a completely fictional way. So that, in my opinion, Game of Thrones was the first generative model, you know, George Martin. Great. Okay, so it could generate something like that, which is completely, it looks, you know, Mauds of the Dragons.

30:02It could look real, okay, realistic, but it's wrong. The same thing with Dan Brown, you know, Da Vinci Code. It looks like a real, it could have been a real story about what happened after, you know, this was crucified in the story. It could have been, but we don't have evidence that it is. Some people follow conspiracy theories. They think that Dan Brown is the real story, but that's what I'm saying. So, yes, it's a possible truth. truth do you think we ever get to that ability where it is true knowledge you get into this concept of like you know your agi and all that type of stuff do you ever think we get to that level of advancement or you know i always go back to like how the human brain works and like are we do we have true knowledge to an extent or are we just doing the same kind of computational thing in our head with probability of what's you know yeah one of the things that we know is that the transform architecture and the language model is not how the brain works.

31:04This is an engineering, it's not how the brain works. There are some commonalities and there are some kind of lack analogies, but I think it's wrong to think about or to try to, you know, like when you're working with language models and you're trying to tune them or you're trying to explain or debug them to have in your mind how the brain works. Don't do that. If you are a prompt engineer, if you're trying to build a model, try to understand how the system is built and use that knowledge. Don't use the cognitive neurology here. Now, unfortunately, we are very, you know, the human brain is still much more powerful given the fact that you can eat a slice of pizza and do very complicated mathematical computations while if you were trying to do the same thing with GPT-4, you need the power of a village or something, even for inference.

32:05Okay, so we are very energy efficient. You know, we use signals that takes milliseconds to transmit, not, you know, nanoseconds, whatever it takes for a GPU, and we still do things faster. Okay, so there's a completely different world. Even if we could make an electronic brain simulated, I think it would be very different. The biology comes into place. It's still a mystery. But whether we're going to reach AGI, you probably hear that. I leave that to people who have enough money and time to think about it. So I mean, yeah, in theory, it is possible. I hear Hinton and Benzio and what's his name?

32:59I think Lacoon is on the other side. And Elon Musk, that they say it's possible for, you know, you leave a language model, start rewriting the code, and unplugging other systems. I don't know. I think not to worry that much about it. I worry more about the effect that it's having right now on the job market. That's more imminent and more real. or the economy, then, you know, whether the robots will revolt against us. And what do you mean by that in terms of it taking away jobs and tasks? Or do you think this unlocks new opportunities? I think it does, yes. Yeah. You know, as with everything, you know, it happens all the time with, you know, with a high tech.