Is ChatGPT Conscious? A Pioneer of AI Explains | Dr. Terry Sejnowski

0:00You see that fly? It can fly, it can find food, it can reproduce. It's doing that with 100 ,000 neurons. You have a supercomputer, it costs you$100 million, it can't fly, it can't see, and it's stuck in a reproduce. Yeah. What's wrong with this picture? Dead silence. Dr. Terry Sejnowski is a professor at the Salk Institute and a global leader in understanding how the brain learns and remembers. A brilliant mind helping us learn how to learn smarter, deeper, and for life. Do humans understand in the same way that these massive models understand? Different people understand things differently. Physicist understands composition of things much better than a person who just picks things up.

0:50But a carpenter understands wood perhaps better than the physicist. In the book you talk about nature-inspired directions for developing. Can you talk about what you mean by nature-inspired? Well, okay, so these large language models are completely helpless by themselves. Yeah. You know, they depend completely on us. They depend on us for the power to give them inputs, for, you know, improving them and so forth. I mean, you know, they're really cuckooed. I know who you are. A lot of people know who you are. I saw you referenced in the Jeff Hinton announcement, which, of course, I would like you to have been included, but because Boltzmann Machines played such a critical role in that development.

1:47And so can you give listeners that are not familiar, have not seen our past episodes, a brief introduction of your work and your work at the NURPS Foundation and then about your new book, which is what I really wanted to talk about? Well, yes, Craig, my background is in physics, PhD, theoretical physics for Princeton. And at that point, I realized that physics had reached such high levels that in order to make progress, you had to have an accelerator that was miles across or set up a satellite. And so I was very attracted to the brain because it was just as mysterious as the universe. But it was something you could actually study, you know, in the lab.

2:49And I met Jeff at a small conference here, actually, in San Diego that he organized on associative memory back in 79. That was quite a while ago. And we just hit it off. I mean, it was clear that we both had the same goals, you know, to try to understand, you know, how the brain could possibly solve these difficult problems. Number one, number two, what was, you know, computational capabilities of very large scale network models. And it was in its infancy back then. And so that, you know, you know, John Hoffield was my thesis advisor. And we ended up generalizing his model by adding noise to it, of all things, you know, to heat it up.

3:44It's a physics model, spin glass model. And by heating it up, we were able to find not just local solutions, but also a global minima. and also uh what popped out of that and this is still amazing uh one of the most beautiful things that i've ever done which is a learning algorithm when you come to equilibrium and so it was a big it was a big physics project because it took advantage of everything that we know about statistical mechanics and bolstmann of course was the father of statistical mechanics and so so that that broke the log jam that led to backprop which jeff and dave romahart introduced shortly thereafter, and the rest is history.

4:29So Jeff, you know, he spent the next 20 years figuring out how to use the BOSA machine and then eventually backprop to solve these difficult problems. And, you know, ImageNet back in, I think it was 2012 or 2013, at NeurIPS really started the deep learning revolution with object recognition in images. but um but my my career path went off toward neuroscience i did a postdoc at harvard neurobiology and then eventually uh wound up here at la jolla at the salk institute i also have a faculty position at uc san diego where i teach and i have an institute for no computation so i i've been i've had this dual role and and by the way i very early on i became the president of the neural information processing systems foundation neural networks was very hot back in the 80s it It was very interesting and intriguing.

5:23People from many, many different areas of physics and engineering would show up at these meetings. Around 500, 600 people would show up. That grew gradually over years. And we didn't have enough competing power really to scale things up. But eventually that happened in this century. We didn't know how well they would do. And it's way beyond my expectations of what I expected back then. And I've read Deep Learning Revolution, and for listeners who want a great history of that whole development, it's a wonderful book that came out, I think, five years ago. I don't remember. Yeah, it was 2018. It was the origin story of modern AI.

6:17Yeah. And so now you've got, you know, fast forward, Transformers appeared and together with back propagation and some other things, it's created this entirely new field of generative AI. the most public facing of which is the chat GPT models. And you're writing about this sort of looking at what that means and what to expect going forward. So can you talk about the new book? Yes, the new book was inspired by a disconnect. And it really had to do with trying to come to grips with whether or not chat GPT understands language. Does it understand understanding? Yeah. And all of, you know, there was a big argument that was raging already, you know, back when it first came out two years ago.

7:31Amongst academics, they couldn't agree. I mean, these are people who spent their whole lives on cognitive science and linguistics and, you know, AI. And it's still raging. It's still a very, very hot topic in the field of machine learning right now, to the extent to which chat GDP is intelligent. I mean, and so I looked into it and I started trying to understand where the disagreement came from. And I made a bunch of observations. And this is something that had to do with how people were using it. And it occurred to me that, look, ChatGPT is trained on thousands and thousands of books and enormous amount of articles all over.

8:27We're talking about novels, textbooks, and it can adopt any persona because it has all of that stored away somewhere. And when you talk to it, it has to pick a persona. yeah and and you can ask it you can say i can i can you can tell it i want you to be a poet or i want you to be a computer scientist you you can set it up that's the prompt uh and what and depending on how you prompt and especially how you give you know you set it up in terms of what what what it is that you wanted how you wanted to do it what you wanted to do and what persona it should have you that determines the responses you're going to get you know the quality of the responses the the kinds of things and uh and once i saw that i realized that this is this is really a mirror it's it's like looking into a mirror and you know it it is looking at you and it is trying to respond as as it best it can to whatever kind of prompts you're giving it and and then and then you know once once i realized that i i realized that we're we're it's completely wrong to think of it but the way that we are we're trying to judge it is it human or not it's obviously not human yeah it's like an alien that suddenly arrived on earth and it's talking to us but it's it's got this it's not it's coming from some other world right Right.

9:58And so we should we. But what it is revealed is something that I think is very deep.

10:11Revelation. About the poverty of the words that we're using to describe cognitive function, all these words come from, you know, 19th century psychology or even earlier. and they're ambiguous. They have multiple meanings. They're not as well tied down scientifically as words in physics like matter, mass, energy. All of this is something that is, we're in a kind of a pre-Copernican era here in AI. And so, you know, we probe words like intelligence. We don't really know what those, you know, how to apply it. Raging battle in biology, you know, about whether other species are, what kind of intelligence they have or are they conscious.

11:07And, you know, books written about this. And I think that we're now on the, we've entered an era now where we can begin to maybe understand these concepts in a more deep mathematical way. because we can create these artifacts that have some of the properties of what we consider language and understanding. And if we can understand them first, mathematically, maybe we'll be able to understand our animal brains. Yeah. Well, in that language, in the fuzzy language used to describe this stuff, understand understanding uh is is one of the first uh that creates confusion and i remember uh thinking about that when uh alpha zero i uh i think that was the model at the time uh beat lee sadal in that uh go game in seoul in whenever it was i've forgotten 2016 or something uh and and he made that the the alpha go alpha zero was an alpha go i think at that time it was alpha go you're right yeah alpha go made this uh completely unorthodox move move 37 it placed a stone up in an empty part of the board and as i recall he said all got up and walked away from the table for a little while to think about it.

12:46And no one understood it. And then as the game played out, it was that move that ultimately turned the tide and beat Lisa at all. And I remember thinking and saying to people, wow, that AI system understood, You know, it understood. I mean, whatever understands means, it wasn't simply running down, you know, some sort of a search tree, Monte Carlo search or something. It looks like it understood it. So how do you work with that word understanding? Okay, so first a couple of footnotes to your observation. First, this was a sign that in addition to being able to talk, that these large language models are actually creative.

13:54Yeah. And we've always thought that was the highest level of human cognition, right? And by the way, hallucinations, which people say, oh, you know, you can't trust it. it's a form of creativity. It's a, it's a, it's, it's the flip side of creativity. You know, when it's, when it's hallucinating, it actually doesn't just make things up. It's very plausible things, right? Things that, you know, could, could exist or could happen, but they never did. And so, and so, you know, by the way, this is also true in humans, right? Humans also confabulate. It's a different word in medicine, but it's the same thing.

14:39Humans, under some conditions, will make up things just like that. And it's like the brain is not able to actually judge themselves. I mean, there are these people who have Korsakoff syndrome. They can't themselves judge what's right, what's true. They're just saying things the brain thinks is true. Their brains. Okay, the other thing, the other footnote is that what you just described was already apparent back in the 1980s, early 90s. Jerry Tesaro, who I worked with actually on applying these learning algorithms to backgammon, right? It's a simpler game, but it's actually a very popular game.

15:30And what makes it actually difficult, more difficult in some ways, is that it's probabilistic. You throw a dice, right? So it's not like chess where you can go, you know, 20 moves ahead. And, you know, it's deterministic in terms of, you know, knowing where pieces are and where they're going to be. but nonetheless it turns out he applied the very same really the secret to AlphaGo was to have the game play itself right and because it's playing itself it basically can go off in new directions that are on anchor from the way human beings play the game and it reached, it's really interesting back then I mean this is, computers are really puny compared to what we have today but nonetheless you know he was able to reach a level of play that some of the experts said were at the championship level and in some cases was making decisions at critical points in the game that were even better than the decisions that humans were making now it was being creative with with new moves that no human had ever made before so it was already apparent back then okay and if i could you know this is a little bit immodest but you know most of the networks back in the 80s were toy problems you just created because the networks are too small to really solve any real world problem one exception to that maybe there were a few okay but the the one that i was involved in was net talk yeah i remember yeah and and that was a problem in linguistics of how you pronounce letters in words phonology and it's a very difficult problem in english especially because it's so ambiguous with the single letter can be pronounced in some cases you know five ten different ways um english has gotten influences from many other languages and you know it's very complicated uh how those mesh but and and and linguists you know who are using rule-based approaches back then um came up with rules but there are exceptions to the rules and then there are within the exceptions like for french words There were rules for the French exceptions and rules all the way down.

17:48And you have books with 300 pages of rules and exceptions. Well, we trained up. This is a summer research project. It was a graduate student from Princeton, Charlie Rosenberg, who showed up. And I didn't know anything about language, linguistics, but he did. He worked with George Miller. And so we took on that project. And by the end of the summer, I'll tell you, it was pronouncing words. understandably we played it through a deck talk which actually you could hear it and it was it was amazing that you can get such a tiny network 20 ,000 connections you know 200 units and it absorbed not just the regularities but also the exceptions in English pronunciation and this was really it was an amazing demonstration because I could play this tape for people at the beginning it babbled and a little bit later it started doing small words and then later at the end you know it just was able to generalize so new texts and it did it beautifully now that in retrospect was an indication that networks love language right and that's clearly what chat gdp is telling us it's it's like unbelievable how well it can do and it's actually still tiny compared to the brain right Even with a trillion weights, the brain has a million billion.

19:11It's the thousands of trillion. So that's, in some ways, chat GDP is already super intelligent because it's absorbed much more knowledge than any human brain could. Yeah. Yeah. And a couple of thoughts on that. One is on the hallucinations. I often tell people that, as this is what you're saying, that people hallucinate too. If you ask me what color shirt I was wearing at my fifth birthday party, which I can remember very well, I'll say blue. And I can see that image in my head. But if I flip back through family photos, I see, oh, I was wearing an orange shirt. You know, but you are so convinced that image is so clear in your head.

20:08And that's a form of hallucination. And so this word understand, I go back and forth. I mean, I tell people who are not involved in AI and are totally amazed by the large language models that, you know, these things, it's just predicting the next token in the series based on the probability in its training data. And if you hit a spot where the training data is relatively sparse, it'll pick a probability, and that will not be what you consider is right. So when it is picking the probabilities that we consider right, we consider that understanding. And it sort of begs the question of what human understanding is.

21:19if indeed we're, you know, a network of weights that we've accumulated throughout our lifetime, you know, are we just coming up with probably this is the stochastic parrot argument. And I mentioned this to Jeff last time I talked to him, or I haven't talked to him for well over a year. He's suddenly unavailable for my level of people. But, and he said, no, they understand. And we didn't get into a debate about what understand means. So anyway, how do you deal with that term understanding? And do humans understand in the same way that these massive models understand in that we're just, you know, our chain of thought is based on problem?

22:23You don't have to go to the chat GDP. You know, different people understand things differently. You know, it's not one thing that, it's not one way to, there's no one way to understand something. There are many different levels of understanding. A physicist understands, you know, the composition of things much better than a person who just, you know, picks things up and uses them. But a carpenter understands wood perhaps better than the physicist because he knows about the grain and he knows about how to cut it and different types of wood and so forth. You know, and so there's many different ways to understand something.

23:15So that's number one. Number two, Jeff's comment, and I've talked to him about this. his argument is the following that in order to be able to predict the next word you have to have a pretty good internal model of what that sentence is all about right and and and that means semantics yes it's you know you have to it's more than just predicting whether it's a verb or a noun it's you know how that new word that you're predicting fits into the rest of the words, right? It has the context of that word. And the internal model, if you keep training it, and it gets better and better and better, the internal model is going to get better and better and better, right?

24:03And at some point, it becomes good enough so that it actually comes up with the appropriate reasonable words for the meaning of that sentence, right? and and and that's a form of understanding uh you might say that it's a superficial understanding uh you know what what's the level of depth how do you compare it with you know a human in the same situation um it depends on the human it depends on what the uh also as you say the database that was used to train it depends on a lot of factors but uh but i think that jeff is right that uh that the internal model and and and we actually have some evidence for this that the internal model is integrating information from many different sources uh first of all from the text that it's being trained on but also from the order of the words as syntax.

25:09That's something that linguists really put a lot of effort into in the last century, understanding that was important. But in addition, it turns out what was ignored in the last century was that language is all about meaning. And that is semantics. And semantics is the purpose of language. We talk to each other so that we can understand each other the meaning of what I'm trying to communicate with you and back and forth, right? That's at a semantic level that we're trying to exchange thoughts, right? And the order of the words, it helps with getting the semantics across, but it doesn't contain the meaning itself.

25:53It's actually, of all things, a lot of the meaning is coming across not just from the words, but how you say the words, what you emphasize. It also depends on facial expressions. If you're talking to somebody and you see them, you'll have a different message, different meaning than if you are just listening to somebody at a recording, right? Language is incredibly, has many, many, many dimensions and many, many ways that we take information in and try to incorporate it. Our brain is doing a lot that subconsciously by the way we're not even aware of it it just happens there was a wonderful book uh probably 20 years ago by a new york times writer named james gleich uh called the information i don't know if you read it but uh and he talks there about language and writing and thought and And his premise, although not alone, you know, he drew on a lot of people, but is that speech disappears, goes out, and as soon as the sound waves have passed you, it's over.

27:15And it wasn't until writing developed that people could start analyzing, that people really started thinking, thinking analytically, that they could say something, stick it out there, look at it, and analyze it. And that felt pretty deep to me. So what are your thoughts about how this not only speech, but how the written language, which is what these large language models are trained on, they're not trained on people just talking. It's written language and huge volumes of written language. How that relates to thought and whether in this whole idea of understanding. I haven't read that book, but I agree with him.

28:21In fact, when people say language, they usually mean spoken languages as being the breakthrough. The breakthrough that the humans evolved this ability to talk to each other. but i can you know it's pretty clear that there's there's all kinds of limitations as you say the spoken words basically disappear very rapidly uh and you know your memory of the words will mutate as you know with this a lot of studies have been done you know where the story changes you know the details change if it's retold and also language can be used for controlling other people rhetoric you know if you're good at at at use tapping into emotions if you're good at you know coming up with arguments that sound plausible you can you can get a lot of people to do things for you.

29:31Right? Yeah. And the bigger your bullhorn, the more the people. Fortunately, what happened is that technology got better and better. Right? You know, you can have more and more people come and listen to you or you can, the radio or the television, you know, you get, the words are, you know, can be spread throughout, you know, millions of whole nations. Right? But for me, the real turning point was not the evolution of spoken language, but the written version. That's an invention, by the way. That's not something that came from nature. We're actually using, we adapted our visual system in order to be able to do that.

30:16Of course, the visual system evolved too. But the point is that we have to go to school to learn how to read. It's many, many years. I mean, it's something we're overpracticed on. If you went to school for, I've been going to school my whole life, reading scientific papers all the time and trying to understand written words, jargon, vocabularies. I mean, it's unbelievable what we now have out there. And it's really allowed us to do two things. First of all, to accumulate knowledge. Number two, to correct knowledge. yeah i mean you can go back and and here's what mr x said about why well i'm gonna investigate this why and i have found that mr x was wrong and here's why and you know that could be at a scientific level where you do an experiment or it could just be you know at a conceptual level or you know in other words you can you constantly you can build on what was there before whereas if all you have is the written word, basically the most you could do is to have somebody who's really good at memorizing stuff, the bards, the people who were told the stories of the past.

31:34That was the way history was transmitted from one generation to the next. And that's very limited. The bandwidth there is very limited. But now when you can write, and now books was a big advance because now you could have multiple copies of the same thing instead of, you know, stone tools, you know, for hieroglyphics or whatever. Now you could disseminate it amongst at least the people who could read, you know, which is a very small fraction back then, but now many more people. And then eventually, you know, science. Yeah. We took that to another level, which is that not only are you transmitting words and thoughts, but you're transmitting observations, experiments, new insights that can be replicated.

32:27Other people can do the experiments. Other people can build the same thing. So on understanding, coming back to understanding, your feeling is that these large models do at some level understand, and that's with using this fuzzy word understand. what's the limit of that because uh understanding leads uh to self-awareness and self-awareness leads to sentience uh yeah what what's your feeling about that spectrum uh and where these models are on it okay so that's a big part of my book i have several chapters specifically trying to understand what's missing. Okay. So the model that is used in chat GPT is primarily a model of the cerebral cortex.

33:33Yeah. The big knowledge base that we have. But, you know, it's only one of about 100 different parts of the brain that are absolutely essential for its function. Right. So I'll tell you, most of what's in the brain is missing. from chat gdp yeah and i'll just give you i made a list in the book but i'll give you the sort of the top hits okay number one chat gdp does not have a goal now nature goes into us which is survival reproduction you know we got to eat you know we have to uh sleep and there's all sorts of things that humans uh you know they have they They're very social creatures. So we need to be in a social group in order for the brain to flourish.

34:30And that comes from evolution. So that's number one. You know, they were trained without any real goals or, for that matter, any instruction about what's good and what's bad. You know, children, when they grow up, they're given all kinds of feedback by their parents, by their peers, by their teachers. You know, don't say that. You know, that's bad. You shouldn't do that. Right. Or it's dangerous. and all chat gdp never gets any of that feedback never it's never told what's good or what's bad it's never told you know what's dangerous and and and so that's reinforcement learning it's another part of the brain the basal ganglia which interestingly was used in alpha go alpha go had a value function right which the basal ganglia creates a value for future rewards you know what you should do next if you want to get a reward.

35:31A reward could be anything. You know, it could be some tasty morsel or it could be a university degree or it could be coming president, right? You know, if you think that that's something you cherish. The reality, you know, the reality is that, you know, chat GDP only has kind of a superficial set of goals, which was to predict the next word. That was very superficial. Humans very likely are using the same kind of algorithm. We know it's true for the value function. It's a reward prediction error. You predict the reward you're going to get. You compare it to what you do get. And from that difference, you change the weights in order to be able to make a better decision the next time you get that input.

36:19Okay, so that's a very general principle. But now here's something that is even, talking now with regard to sentience here's something that is a showstopper literally okay so you're talking to chat gbt and it's responding it keeps responding word after word and then at the end it finishes and it thanks you or it says i hope this is okay it's very polite right and then it stops what happens to the network when it stops nothing nothing is going on in that network the only time that anything happens is when you give it a question or you give it a prompt okay right that that means that there's nothing going on there this it's it's it's completely empty it doesn't have any self-generating activity right so i i think it's literally a no-brainer about sentience it's not sentient is in any sense of the way that we are because you put us in a room, no sensory inputs.

37:22What's going on? Well, you're thinking away, right? You're thinking about what you're going to do later in the day. You may think that you're hungry. You may want to think about going to the supermarket. In other words, you're planning all the time. You're coming up with responses. You're thinking about, you know, what your goals are and how to accomplish that. Or maybe you're just enjoying a movie and thinking about the movie. But the idea, though, is that we have a constant flow of thoughts in the absence of any direct input. Obviously, you may be responding to something you heard yesterday, and that's been circulating because you've been a new way, a new idea, or maybe you were, I don't know.

38:09I mean, actually, unfortunately, a lot of these are emotional things that you think about. You know, you get unhappy, you get sad, you feel that you've been maligned, you know, and that will continue to circulate. In fact, sometimes that leads to mental disorders, right, when you have people who are anxious and so forth. So in any case, there's many, many, many lessons to be learned from the way that nature has gone about creating autonomy. you know something and all animals have to survive all animals are autonomous to some degree and insects i mean it's not just animals insects you know plants all of nature is autonomous in terms of self-regulating and self-creating and so forth and of course they interact with each other and that's part of the complexity of nature uh but right now uh you know these uh chat gdp you know they're trying to make them into agents but they're not they're just not they don't have the same agency that we have yeah well let's talk about agents so because that is the next frontier and there are uh now agents that can perform mental tasks uh that were previously performed by humans you give it a the agent a goal and and it'll perform that task continuously.

39:48And now you can chain agents together where each one performs a discrete task and hands the output off to another who performs another task. Where do you see that going? Because it seems that combined with the quote-unquote understanding of large models, that having goals and having agents collaborate with one another could lead to something unexpected. Nobody can predict what will happen, right? I mean, in other words, it's a surprise, a surprise. I mean, Shack TV was a huge surprise, right? All these things, you know, the language translation was a big surprise. And I'm sure that's going to continue, so who knows?

40:41But one thing to keep in mind, and this is kind of a more mathematical argument, is that two chat GDPs talking to each other is just one giant GPT. It's no different.