#220 Terry Sejnowski: The Future of AI, ChatGPT & Deep Learning

0:00The new book was inspired by a disconnect and it really had to do with trying to come to grips with whether or not chat GPTGPT understands language. There was a big argument that was raging already, you know, back when it first came out two years ago. Amongst academics, they couldn't agree. I mean, these are people who spent their whole lives on cognitive science and linguistics and, you know, AI. And it's still raging. It's still a very, very hot topic in the field of machine learning right now. The extent to which ChatGDP is intelligent. In today's fast-paced world, ensuring your AI systems are not only compliant, but also accurate and robust is key.

0:45How can you achieve reliable AI outcomes while managing risks effectively? If you're facing challenges with AI risk management, having a solution that ensures accuracy and robustness is invaluable. Citrus X AI offers cutting-edge technology for you to detect and mitigate vulnerabilities, biases, and errors while ensuring accuracy, robustness, and compliance. Their solution is the only one that offers observability capabilities while helping you meet regulatory requirements. Citrus X.ai ensures your AI models are not just performing well, but navigating risk safely. Elevate your AI model management with Citrus X.

1:37Visit email.citrusx.ai slash ionai to book your free demo today and learn more about how their solution can benefit you. That's email, E-M-A-I-L dot citrusx, C-I-T-R-U-S-X dot A-I slash IonAI. IonAI all run together, E-Y-E-O-N-A-I to book your free demo today. Can you start, Terry? I know who you are. A lot of people know who you are. I saw you referenced in the Jeff Hinton announcement, which, of course, I would like you to have been included, but because Boltzmann Machines played such a critical role in that development. uh so can you give uh listeners that are not familiar have not seen our past episodes a brief introduction of uh your work and uh your work uh at the neurops foundation and then uh about your new book which is what i really wanted to talk about well uh yes so craig uh My background is in physics, PhD, theoretical physics for Princeton.

3:12And at that point, I realized that physics had reached such high levels that in order to make progress, you had to have an accelerator that was miles across or set up a satellite. And so I was very attracted to the brain because it was just as mysterious as the universe, but it was something you could actually study, you know, in the lab. And I met Jeff at a small conference here actually in San Diego that he organized on associative memory back in 79. That was quite a while ago. And we just hit it off. I mean, it was clear that we both had the same goals, you know, to try to understand, you know, how the brain could possibly solve these difficult problems.

4:05Number one, number two, what was, you know, computational capabilities of very large scale network models. And it was in its infancy back then. And so that, you know, you know, John Hoffield was my thesis advisor. And we ended up generalizing his model by adding noise to it, of all things, you know, to heat it up. It's a physics model, spin glass model. And by heating it up, we were able to find not just local solutions, but also a global minima. and also what popped out of that and this is still amazing and one of the most beautiful things that I've ever done which is a learning algorithm when you come to equilibrium and so it was a big it was a big physics project because it took advantage of everything that we know about statistical mechanics and Boltzmann of course was the father of statistical mechanics and so so that that broke the logjam that led to backprop which Jeff and Dave Romahart introduced shortly thereafter and the rest is history so uh jeff you know he spent the next 20 years figuring out how to use the bolso machine uh and and then eventually back prop to solve these difficult problems and you know image net back and i think was 2012 or 2013 um at neurops really uh started the deep learning revolution with object recognition in uh in in images but um but my my career path went off toward neuroscience.

5:47I did a postdoc at Harvard Neurobiology and then eventually wound up here at La Jolla at the Salk Institute. I also have a faculty position at UC San Diego where I teach and I have an institute for no computation. So I've had this dual role. And by the way, very early on, I became the president of the Neuroinformation Processing Systems Foundation. Neural networks was very hot back in the 80s. It was very interesting and tree, people from many, many different areas of physics and engineering would show up at these meetings, around five, 600 people would show up. And that grew gradually over years.

6:23And, you know, we didn't have enough competing power really to scale things up. But eventually that happened, you know, in this century, we didn't know how well they would do. And it's way beyond my expectations of what I expected back then. yeah and and so uh and i've i've read a deep learning revolution and for listeners who want a great history of that whole uh development uh it's a wonderful book uh that came out i think five years ago i don't remember yeah no it was uh 2018 it was the origin story of modern AI. And so now you've got, you know, fast forward, transformers appeared, and together with backpropagation and some other things, it's created this entirely new field of generative AI, the most public facing of which is the ChatGPT models.

7:46And you're writing about this, sort of looking at what that means and what to expect going forward. So can you talk about the new book? Yes, the new book was inspired by a disconnect. And it really had to do with trying to come to grips with whether or not chat GPT understands language. Does it understand understanding? Yeah. And all of, you know, there was a big argument that was raging already, you know, back when it first came out two years ago. Amongst academics, they couldn't agree. I mean, these are people who spent their whole lives on cognitive science and linguistics and, you know, AI, and it's still raging.

8:38It's still a very, very hot topic in the field of machine learning right now, to the extent to which chat GDP is intelligent. Yeah. And so I looked into it and I started trying to understand where the disagreement came from. And I made a bunch of observations. And this is something that had to do with how people were using it. And it occurred to me that, look, ChatGPT is trained on thousands and thousands of books and enormous amount of articles uh all over you know from uh you know we're talking about novels textbooks and and and it can adopt any persona because you have all of that stored away somewhere and when you talk to it uh it has to pick a persona yeah and and you can ask it you can say i can i can you can tell it i I want you to be a poet or I want you to be a computer scientist.

9:47You, you can set it up. That's the prompt. And what, and depending on how you prompt and especially how you give, you know, you set it up in terms of what, what, what it is that you want it to, how you want it to do it, what you want it to do and what persona it should have you, that determines the responses you're going to get, you know, the quality of the responses, the kinds of things. And, And once I saw that, I realized that this is really a mirror. It's like looking into a mirror. And, you know, it is looking at you and it is trying to respond as best it can to whatever kind of prompts you're giving it.

10:31And then, you know, once I realized that, I realized that it's completely wrong to think of it the way that we are. We're trying to judge it. Is it human or not? It's obviously not human. It's like an alien that suddenly arrived on Earth and is talking to us, but it's it's got it's not. It's coming from some other world. Right. And and so we should we. But what it is revealed is something that I think is very deep.

11:05revelation about the poverty of the words that we're using to describe cognitive function all these words come from you know 19th century psychology or even earlier and they're ambiguous they have multiple meanings they're they're they're not as well tied down scientifically as words in physics like matter, mass, energy. All of this is something that we're in kind of a pre-Copernican era here in AI. And so we probe words like intelligence. We don't really know how to apply it. I mean, there's a raging battle in biology, you know, about whether other species are, what kind of intelligence they have or are they conscious.

12:01And, you know, it's books written about this. And I think that we we're now on the we've entered an era now where we can begin to maybe understand these concepts in a more deep mathematical way, because we can create these artifacts that have some of the properties of what we consider language and understanding. And if we can understand them first, mathematically, maybe we'll be able to understand our, you know, animal brains. yeah uh well and uh in that language in the fuzzy uh language used to describe this stuff understand understanding uh is is one of the first uh that creates confusion and i remember uh thinking about that when uh alpha zero i uh i think that was the model at the time beat Lee Sedol in that Go game in Seoul in whenever it was, I've forgotten, 2016 or something.

13:09AlphaGo made this completely unorthodox move, Move 37. It placed a stone up in an empty part of the board. And as I recall, Lee Sedol got up and walked away from the table for a little while to think about it. And no one understood it. And then as the game played out, it was that move that ultimately turned the tide and beat Lee Sedol. And I remember thinking and saying to people, wow, that AI system understood. You know, it understood. I mean, whatever understands means, it wasn't simply running down, you know, some sort of a search tree, Monte Carlo search or something. uh it it looks like it understood it so how do you work with that word understanding okay so first a couple of footnotes to your uh observation first um this was a sign that in addition to being able to talk that these these large language models are actually creative Yeah.

14:39And we've always thought that was the highest level of human cognition, right? And by the way, hallucinations, which people say, oh, you know, you can't trust it. It's a form of creativity. It's the flip side of creativity. When it's hallucinating, it actually doesn't just make things up. It's very plausible things, right? Things that could exist or could happen, but they never did. And so, and so, you know, by the way, this is also true in humans, right? Humans also confabulate. It's a different word in medicine, but it's the same thing is that humans for under some conditions will make up things just like that.

15:31Right. And it's like, you know the brain is not able to actually judge themselves i mean the of these people who have like korsakoff syndrome they can't themselves judge what's right what's true they're just saying things that the brain thinks is true their brains okay the other thing the other footnote is that what you just described was already apparent back in the 1980s early 90s jerry tesaro who i worked with actually on applying uh these learning algorithms uh to backgammon right it's a simpler game but it's actually a very popular game and what makes it actually difficult but more difficult in some ways uh is that is probabilistic you throw it you throw a dice right so it's not like chess where you can go you know 20 moves ahead and and you know it's deterministic uh in terms of you know knowing where pieces are and where they're going to be but but nonetheless he it turns out he applied the very same that the the really the secret to uh alpha go was uh to have the game play itself right right and and and and and because it's playing itself it basically can go off in new directions that are unanchored from the way human beings play the game.

16:55And it reached, it's really interesting back then. I mean, this is, computers are really puny compared to what we have today. But nonetheless, you know, he was able to reach a level of play that some of the experts said were at the championship level. And in some cases was making decisions at critical points in the game that were even better than the decisions that humans were making. It was being creative with new moves that no human had ever made before. So it was already apparent back then. Okay. And if I could, you know, this is a little bit immodest, but, you know, most of the networks back in the 80s were toy problems.

17:37We just created because the networks are too small to really solve any real world problem. One exception to that, maybe there were a few. Okay. But the one that I was involved in was net talk. Yeah, I remember. And that was a problem in linguistics of how you pronounce letters in words, phonology. And it's a very difficult problem in English, especially because it's so ambiguous. The single letter can be pronounced in some cases, you know, five, ten different ways. English has gotten influences from many other languages, and it's very complicated how those mesh. but and and linguists you know who are using rule-based approaches back then um came up with rules but there are exceptions to the rules and then there are within the exceptions like for french words there were rules for the french exceptions and it was you know rules all the way down and you have books with 300 pages of rules and exceptions well we we trained up this is a summer research project uh it was a of graduate student from princeton charlie rosenberg who showed up and I didn't know anything about language, linguistics, but he did.

18:48He worked with George Miller. And so we took on that project. And by the end of the summer, I'll tell you, it was pronouncing words. Understandably, we played it through a deck talk, which actually you could hear it. And it was amazing that you can get such a tiny network, 20 ,000 connections, 200 units. and it absorbed not just the regularities, but also the exceptions in English pronunciation. And this was really, it was an amazing demonstration because I could play this tape for people. At the beginning, it babbled, and a little bit later, it started doing small words. And then later at the end, I just was able to generalize some new texts and it did it beautifully.

19:32Now that, in retrospect, was an indication that networks love language. Right. And that's clearly what chat GDP is telling us. It's like unbelievable how well it can do. And it's actually still tiny compared to the brain. Right. Even with a trillion weights, you know, the brain has a million billion. uh you know it's it's uh the thousands of trillion so that that's that's uh you know in some ways uh uh chat gdp is already super intelligent because it's absorbed much more knowledge than any human brain could right yeah yeah and and a couple of thoughts on that one is on uh on the hallucinations i i often tell people that as this is what you're saying that people hallucinate too if you ask me what color shirt I was wearing at my fifth birthday party, which I can remember very well, I'll say blue, and I can see that image in my head.

20:38But if I flip back through family photos, I see, oh, I was wearing an orange shirt. But you are so convinced that image is so clear in your head and that's a form of hallucination and so this word understand uh i i go back and forth i mean i tell people who are not involved in ai and are totally amazed by uh the large language models that, you know, these things, it's just predicting the next token in the series based on the probability in its training data. And if you hit a spot where the training data is relatively sparse, it'll pick a probability and that will not be what you consider is right.

21:43So when it is picking the probabilities that we consider right, we consider that understanding. And it sort of begs the question of what human understanding is. if indeed we're, you know, a network of weights that we've accumulated throughout our lifetime, you know, are we just coming up with probably, this is the stochastic parrot argument, and I mentioned this to Jeff last time I talked to him, or they haven't talked to him for well over a year. He's suddenly unavailable for my level of people. And he said, no, they understand. And we didn't get into a debate about what understand means. So anyway, how do you deal with that term understanding?

22:50And do humans understand in the same way that these massive models understand in that we're just, you know, our chain of thought is based on You don't have to go to the chat GDP. You know, different people understand things differently. You know, it's not one thing that, it's not one way to, there's no one way to understand something. There are many different levels of understanding. A physicist understands, you know, the composition of things much better than a person who just, you know, picks things up and uses them. But a carpenter understands wood perhaps better than the physicist because he knows about the grain and he knows about how to cut it and different types of wood and so forth.

23:53You know, and so there's many different ways to understand something. So that's number one. Number two, I think Jeff's comment, and I've talked to him about this. In fact, recently I've talked to him almost every day. I mean, you know, he's transformed into a completely different stage of his life now, right? I mean, once you get the Nobel Prize, it's a different society now puts you on a pedestal. But his argument is the following, that in order to be able to predict the next word, you have to have a pretty good internal model of what that sentence is all about right and and and that means semantics yes it's you know you have to it's more than just predicting whether it's a verb or a noun it's you know how that new word that you're predicting fits into the rest of the words right uh it has the context of that word and uh and the internal model if you keep training it and it gets better and better and better the internal model is going to get better and better and better right and at some point it becomes good enough so that it actually comes up with the appropriate reasonable words for the meaning of that sentence right and and and that's a form of understanding.

25:36You might say that it's superficial understanding. What's the level of depth? How do you compare it with a human in the same situation? It depends on the human. It depends on what the, also as you say, the database that was used to train it. Depends a lot of factors but uh but i think that jeff is right that uh that the internal model and and and we actually have some evidence for this that the internal model is integrating information from many different sources uh first of all from the text that it's being trained on but also from the order of the words as syntax that's something that linguists really put a lot of effort into in the last century understanding that was important but in addition it turns out what was ignored in the last century was that language is all about meaning and that is syntax i'm sorry it was a Freudian slip me it's all about uh semantics yeah and and semantics uh is the purpose of language is we talk to each other so that we can understand each other the meaning of what i'm trying to communicate with you and back and forth right that's that's at a semantic level that we're trying to exchange thoughts right we and the order of the words it helps with getting the semantics across but it's it's not doesn't contain the meaning itself it's it's Actually, of all things, a lot of the meaning is coming across not just from the words, but how you say the words, what you emphasize.

27:24It also depends on facial expressions. If you're talking to somebody and you see them, you'll have a different message, a different meaning than if you are just listening to somebody at a recording. Right. language is incredibly has many many many dimensions and many many ways that we take information in and try to incorporate it our brain is doing a lot of that subconsciously by the way we're not even aware of it it just happens yeah uh the uh there was a wonderful book uh probably 20 years ago by a New York Times writer named James Gleick, called The Information. I don't know if you read it.

28:11And he talks there about language and writing and thought. And his premise, although not alone, you know he drew on a lot of people but is that uh speech disappears it goes out and as soon as the sound waves have passed you it's over uh and it wasn't until writing developed that uh people could start analyzing uh that people really started thinking that that thinking analytically, that they could say something, stick it out there, look at it, and analyze it. And that felt pretty deep to me. So what are your thoughts about how how this not only speech, but how the written language, which is what these large language models are trained on, they're not trained on people just talking, it's written language and huge volumes of written language, how that relates to thought and whether, and this whole idea of understanding, whether it's It's the analytic exercise on written language that creates understanding.

29:57Or maybe not, I'm thinking about maybe not. I mean, you. Okay. Well, I haven't read that book, but I agree with him. In fact, when people say language, they usually mean spoken languages as being the breakthrough, breakthrough that the humans evolved this ability to talk to each other but I can tell you it's pretty clear that there's there's all kinds of limitations as you say the spoken words basically disappear very rapidly and you know your memory of the words will mutate as you know with There's a lot of studies have been done, you know, where the story changes, you know, the details change if it's retold.

30:47And also, language can be used for controlling other people. rhetoric, you know, if you're good at tapping into emotions, if you're good at, you know, coming up with arguments that sound plausible, you can get a lot of people to do things for you. Right? Yeah. And the bigger your bullhorn, the more the people. Unfortunately, what happened as that technology got better and better right you know you can have more and more people come and listen to you or you can the radio or the television you know you you you get the the words are you know can be spread throughout you know millions of whole nations right uh now the what you're not but but i for me the real turning point was not the uh evolution of spoken language but the written version.

31:52That's an invention, by the way. That's not something that came from nature. We're actually using, we adapted our visual system in order to be able to do that. Of course, you know, the visual system evolved too. But the point is that we have to go to school to learn how to read. You know, it's many, many years. I mean, it's something we're overpracticed on. You know, if you went to school for, you know, I've been going to school my whole life, reading scientific papers all the time and trying to understand written words, you know, the jargon, vocabularies. I mean, it's unbelievable what we now have right out there.

32:33And it's really allowed us to do two things. First of all, to accumulate knowledge. Number two, to correct knowledge. I mean, you can go back and here's what Mr. X said about Y. well I'm gonna investigate this why and I have found that mr. X was wrong and here's why and you know that could be at a scientific level where you do an experiment or could just be you know at a conceptual level or you know in other words you can you constantly you can build on what was there before whereas if all you have is the written word basically the the most you could do is to have somebody who's really good at memorizing stuff, the bards, the people who were told the stories of the past.

33:21That was the way history was transmitted from one generation to the next. And that's very limited. The bandwidth there is very limited. But now when you can write, and now books was a big advance because now you could have multiple copies of the same thing instead of stone tools for hieroglyphics or whatever. Now you could disseminate it amongst at least the people who could read, which is a very small fraction back then, but now many more people. and then eventually science. Yeah. We took that to another level, which is that not only are you transmitting words and thoughts, but you're transmitting observations, experiments, new insights that can be replicated.

34:14Other people can do the experiments. Other people can build the same thing. Yeah. Yeah. Well, on understanding, coming back to understanding, your feeling is that these large models do at some level understand, and that's with using this fuzzy word, understand. And what's the limit of that? Because understanding leads to self-awareness and self-awareness leads to sentience. Yeah. What's your feeling about that spectrum and where these models are on it? Okay. So that's a big part of my book. I have several chapters. specifically trying to understand what's missing. Okay. So the model that is used in chat GPT is primarily a model of the cerebral cortex.

35:27Yeah. The big knowledge base that we have. But, you know, it's only one of about 100 different parts of the brain that are absolutely essential for its function, right? So I'll tell you, most of what's in the brain is missing. from chat gdp yeah and i'll just give you i made a list in the book but i'll give you the sort of the top hits okay number one chat gdp does not have a goal now nature put goals into us which is survival reproduction you know we got to eat you know we we have to uh sleep and there's all sorts of things that humans uh you know they have they have uh they're very social creatures so you know we we need to be uh in a social group in order for the brain to flourish right and and you know that's that comes from evolution uh so that's number one We, you know, they were trained without any real goals or for that matter, any instruction about what's good and what's bad.

36:42You know, children, when they grow up, right, they're given all kinds of feedback by their parents, by their peers, by their teachers. You know, don't say that. You know, that's bad. You shouldn't do that. Right. Or it's dangerous. And all chat GDP never gets any of that feedback. Never just never told what's good or what's bad. It's never told, you know, what's dangerous. And so that's reinforcement learning. It's another part of the brain, the basal ganglia, which interestingly was used in AlphaGo. AlphaGo had a value function, which is what the basal ganglia creates a value for future rewards.

37:23What you should do next if you want to get a reward. The reward could be anything. You know, it could be some tasty morsel, or it could be a university degree, or it could be coming president, right? If you think that that's something you cherish. um but you know the the reality the you know the reality is that uh it you know chat gdp only has kind of a superficial set of of goals which was you know to predict the next word that was very superficial but it's very important by the way the same humans have a very very likely are using the same kind of algorithm we know it's true for the value function it's a reward prediction error you predict the reward you're going to get you compare it to what you do get and from that difference you change the weights to in order to be able to make a better decision the next time you get that input okay so that's that's a very general principle but now here's something that is even talking now with regard to sentience here's something that is a showstopper literally okay so you're talking to chat gbt and it's responding it keeps responding word after word and then at the end it finishes and it thanks you or it says i hope this is okay it's very polite right and then it stops what happens to the network when it stops nothing nothing is going on in that network the only time that anything happens is when you give it a question or you give it a prompt okay right that that means that there's nothing going on there this it's it's it's completely empty it doesn't have any self-generating activity right so i i think it's a it's a it's a it's literally a no-brainer about sentience it's not sentient is in any sense of the word way that we are because you put us in a room no sensory inputs what's going on well you're thinking away right you're thinking about what you're going to do later in the day you may think that you're hungry you may want to think about going to the supermarket in other words you're planning all the time you're coming up with responses you're you're thinking about you know what what your goals are and what what how to accomplish that or maybe you're just uh enjoying a movie uh and thinking about the movie you know but the idea though is that uh we we have a constant flow of thoughts in the absence of any direct input obviously you may be responding to something you heard yesterday and that's been circulating because you know you've been a new a new way a new a new idea or maybe you were uh i don't know i mean some you know actually unfortunately a lot of these are emotional things that you think about you know you get unhappy you get a sad you you you you you you feel uh that you've been uh maligned you know and that and that'll continue to circulate In fact, sometimes that leads to mental disorders, right?

40:41When you have people who are anxious and so forth. So in any case, you know, there's many, many, many lessons to be learned from the way that nature has gone about creating autonomy. You know, something, all animals have to survive. All animals are autonomous to some degree. And insects, I mean, it's not just animals. Insects, plants, all of nature is autonomous in terms of self-regulating and self-creating and so forth. And of course, they interact with each other and that's part of the complexity of nature. But right now, these chat GDP, they're trying to make them into agents, but they're not.

41:31They're just not. They don't have the same agency that we have yeah well let's talk about agents so because that is the next frontier and there are uh now agents that can perform mental tasks uh that were previously performed by humans you give it a the agent a goal and and it'll perform that task continuously and now you can chain agents together where each one performs a discrete task and hands the output off to another who performs another task. Where do you see that going? Because it seems that combined with the quote-unquote understanding of large models that having goals and having agents collaborate with one another could lead to something unexpected.

42:40You know, it's an interesting direction. And one of the things that we have to be, you know, prepared for is that nobody can predict what's what will happen right i mean in other words it's a surprise a surprise i mean chat gdp was a huge surprise right all these things you know the language translation was a big surprise and and i'm sure that's going to continue so who knows but one thing to keep in mind and this is kind of a more mathematical argument is that two chat gdps talking to each other is just one giant gpt no different yeah it's you're it's maybe it's uh you know the it you've externalized some of the internal stuff that's going on but it's basically the same um it it might have uh additional capabilities i'm not saying that it doesn't but what i'm saying though is that it's it's really uh there's nothing that is uh special you know in terms of the uh you know, it's, they're not human agents, right?

44:00It's not, they're not behaving like human agents. No. It's, you know, like I say, you know, we're, there are other parts of the brain. There are other things that make us agents. And, you know, maybe there's, again, agency is one of those words that has multiple meanings, that because of spectrum and that maybe you know there is a little bit of agency there that can be enhanced amplified somehow but it's not you know it it's it's pretty far along that spectrum you know in terms of where we are and where other species are but maybe you know again uh this is something that you know i i started started by asking you if you could have predicted where the internet was going.