Consciousness, Reasoning and the Philosophy of AI with Murray Shanahan

0:03I think there are just a huge number of enormously interesting philosophical questions that AI gives rise to. You know, what is the nature of the human mind? What is the nature of mind?

0:19Welcome back to Google Deep Mind, the podcast. My guest on this episode is Murray Shanahan, Professor of Cognitive Robotics at Imperial College London and Principal Research Scientist at Google Deep Mind. Now, we have all heard the stories about people falling in love with their chatbots, about people pushing large language models to contemplate their own existence. or questioning the limits of their conceptual understanding of reality. But these kinds of questions about self-identity and thinking and metacognition have been puzzling philosophers for millennia already. And so it makes sense that they should be turning to AI to interrogate the most profound questions about the nature of AI's intelligence, of its current capabilities, even its consciousness or otherwise.

1:08Murray Shanahan has been working in the field of AI since the 1990s. And if you've been following this podcast for a while, you will remember him as the man that consulted on the 2014 science fiction film Ex Machina, about a computer programmer who gets the chance to test the intelligence of a female robot, Ava, and ultimately questions whether she is conscious. Welcome back to the podcast, Murray. Just thinking back, because I know that you played a key role in Ex Machina, shall we say, the Alex Garland film. What do you think you got right in that film and in other science fiction films that were around at the time?

1:46I mean, thinking back to sort of 10, 15 years ago, were we on the right track? So I think that one respect in which Ex Machina really did a great service was that it does raise a whole load of very interesting and provocative questions about consciousness and about AI and consciousness and therefore about consciousness itself. So that's one huge success. But it's interesting that just very shortly before Ex Machina came out, Her came out. So Spike Jonze's movie Her came out. At the time, I really wasn't all that keen on Her as a movie because I just thought it was so implausible that a person could fall in love with this kind of disembodied voice, even if it's Scarlett Johansson's.

2:27I mean, how wrong was that? as a bit of prediction, I think Her really did amazingly well at predicting the world we've got now. Now, we don't know quite how things are going to unfold in the next few years, because maybe robotics will progress rapidly as well in the way that language has in AI. But at the moment, you know, it's all about disembodied language. And also, you know, Her showed how people can, in fact, be very much, you know, form relationships, whatever, you know, in the broadest sense, with disembodied AI systems, which is an extraordinary thing, really. We're talking 10, 15 years ago, but your involvement in AI goes back much further than this.

3:04Because, I mean, I know that you had, you knew John McCarthy. I did know John McCarthy. I knew him very well. John McCarthy was a professor of computer science and artificial intelligence. Back in the day, he actually coined the phrase artificial intelligence and was one of the authors of the proposal for the very famous Dartmouth conference that took place in 1956, which was the first AI conference in the world. And that conference really mapped out the whole field. And people just weren't thinking about this kind of thing seriously at all. It was just a handful. So, you know, I think he was a real radical thinker.

3:40OK, that choice of words, artificial intelligence, was it a good choice of words? Yeah, I mean, I still think it was. I mean, I know that some people think that perhaps it wasn't a good choice of words. Give us some of their arguments. So first of all, there is the word intelligence. So intelligence, you know, itself is in some ways a very contentious concept, especially if people think about IQ tests and that kind of thing. And the idea that intelligence is something that can be quantified on a straightforward, simple scale, you know, and then some people are more intelligent than others. And I think in psychology, it's well recognized today that there are many different kinds of intelligence.

4:19And this is a really important point, right? so there's that concern about that word there so what would you have used differently well maybe artificial cognition or something i often use the word cognition to mean kind of you know thinking and processing information and so on but it doesn't have the same ring to it does it let's be honest especially not now i think it's we're too far down this road aren't we yeah the word artificial i don't really have a problem with the word artificial that seems like a right kind of thing it's alluding to the fact that it's something that we've built and that hasn't evolved in nature and so that seems the right sort of word.

4:54The objection to that word I guess is this ultimately everything that artificial intelligence is built on is at some level constructed by humans. Sure yes but it is so what's wrong with the word you know in that case I mean I think that's true. You were working on symbolic AI right just just talk to us about the difference between that and the other types and where we're at now. Yeah absolutely yeah yeah so the so-called symbolic paradigm of artificial intelligence was very much dominant for many decades. So the idea there is that it's all about the manipulation of symbols and of language-like sentences and using kind of reasoning processes with those symbols.

5:34The classic example would be an expert system. So where back in the 1980s, people were building these expert systems. And the idea was there was that you would try to encode medical knowledge say in a set of rules and the rules would be something like you know oh you know if the patient has a temperature of 104 and their skin is purple and you know then there's a 0.75 probability that they've got you know skinny itis or something you can tell that i'm not a medical doctor just about yeah and then so you'd have thousands and thousands of these sorts of rules would be put into a kind of big knowledge base and then you'd have what was called an inference engine which would carry out logical reasoning over all of these rules and come to some conclusion about what the likely disease was in there.

6:22But it was a lot of if this then that. It was a lot of yeah if then type rules largely and one of the big problems with that is that where do the rules come from? Well somebody has to write them all out basically and so there was a whole field of knowledge elicitation where you go around to experts and you try and extract from them their understanding you know in their domain which you know could be medical diagnosis it could be fixing photocopiers it could be the law and you try and codify all of this into a computer comprehensible very precise rules that was a very cumbersome process and also what you ended up with at the end was very very brittle it would go wrong in all kinds of ways another big area of research was common sense because often it was realized that we implicitly have an enormous amount of common sense knowledge about the everyday world to do with just everyday objects.

7:12The fact that they're solid, the fact that they move in certain ways, they fit into each other in certain ways, you know, liquids and gases and gravity and, you know, all kinds of things like that. And we actually bring all of that knowledge to bear all the time in what we're doing, but it's sort of unconscious. So then there was a big project, there were various big projects to try and codify all of that common sense knowledge but then trying to turn that into like axioms and logic and rules and everything was a nightmare so i eventually i think by about the early 2000s i really thought that this research paradigm was kind of doomed to be honest i sort of started moving away from it but then of course along came things like neural networks and so on yes which was much less about you know if then rules and much more about sort of extracting information from a large amount of data.

8:00But then I sort of wonder now about now that language is effectively cracked, have we sort of reached a higher level of abstraction where we can go back to some more of those symbolic techniques, some of those more symbolic ideas? Yeah, well, we certainly have, because nowadays, one of the hot topics at the moment with large language models is reasoning. So you have these so-called chain of thought models that actually carry out a whole, you know, rather than simply generating an answer to a question they generate a whole chain of reasoning before they issue the answer and that can be very very effective so it's interesting how that harks back in many ways to the kind of thing that people were looking at back in in the days of symbolic ai but the underlying substrate for doing all that is very very different indeed because it's not hard-coded rules it's as you mentioned neural networks that have learned let me pick up on that point about reasoning as a philosopher so you know background in logic how good do you think that ai is a reasoning well that's a very interesting and kind of open question and somewhat controversial so computer scientists and ai people they have a particular notion of reasoning a particular concept of reasoning which very much you know harks back to formal logic and theorem proving so in the days of symbolic ai for example then you had systems that were really very good at doing theorem proving with formal logic.

9:24And so people think, oh, that's proper reasoning. That's really, that's your hardcore kind of reasoning. And today's large language models, they can't match the performance of a, you know, a hand-coded theorem prover or logic engine of the sort that's been around for decades. Give me an example of a type of theorem that might be able to be proved by a hard-coded system. It will be where you've got maybe, you know, 20 or 30 axioms of logic. So it might be something like the number that follows one is two. Well, I mean, it could be something like that. It could be in the domain of number theory or something very mathematical, but it could be something much more everyday.

10:00For example, suppose that you've got some very difficult logistical planning problem where maybe you have hundreds of lorries and depots and goods and all kinds of things like that, and you need to plan the routes and the deployment of the lorries and where they're going to go. So that's a very kind of difficult problem computationally, and it can be expressed very precisely in formal rules. And that's the kind of situation where you might want to use a good old-fashioned, straightforward planning algorithm of the sort that's been around for a long time. Now, contemporary large language models are getting better and better at this kind of thing.

10:34But they're still, you know, you don't have those kinds of mathematical guarantees that they're always going to come up with exactly the right answer. And it's very easy to kind of make examples where you have more and more axioms and so on, where they're going to slip up. There's a whole separate research direction, which is to try and build more hand-coded things that combine today's AI techniques with more old-fashioned symbolic techniques, specifically for mathematical theorem proving, and DeepMind has done some amazing work along those lines. But that's different from large language models.

11:03So with large language models, we're thinking of these chatbots that can talk about anything under the sun, and one of the things they happen to be able to do is a kind of reasoning. So that's not going to be at the moment quite as good as you could do by hand building something for that. It's kind of interesting because hand building something, I mean, you end up with something that's very rigid. That's the problem, yeah. And brittle. Yes, absolutely. But then at the same time, the sort of flexibility that you get from the genitive AI approach, you know, it's too floppy, as it were. You know, you want the rigidity in there.

11:33Well, you know, maybe or maybe not. I mean, I think many examples of human affairs are just not as black and white as that. And, you know, you do maybe want things to be a bit more blurry, even in sort of simple everyday things, like what would be good flowers to put over in this corner of the garden? Well, you know, we've already got some roses in that corner there, and those roses are yellow, but we can't have too much yellow, so maybe we'd need to move them to the other corner of the garden. But then at the same time, though, is this real reasoning, Or is this just the AI kind of mimicking well-structured arguments that have existed in the training data, but just in a novel environment?

12:11Yeah, well, of course, that begs the question, what is real reasoning? You know, it's not written in the sky, you know, what real reasoning is. It's up to us to define the concept of real reasoning or of reasoning. and so you know we were talking earlier on about kind of mathematical reasoning of the sort that logicians do and that was you know is done by kind of theorem proves in the past and today when people were first using the terms like reasoning they weren't thinking of that kind of thing and when we use the word reasoning in everyday life we're not thinking about that sort of thing so if you're chatting away to a large language model and uh about your garden and you sort of say i'm thinking about what plants are and and it says well you know maybe you should consider this kind of plant in that kind of location because that's best for the soil and given you said that the winds you know it's windy there and you know we would just say that that is supplying reasons i mean it is supplying reasons for now where they come from as another matter so people might say well it's just mimicking what's in the in the training set but you know it's probably never seen exactly that kind of scenario exactly before so it's it's moving beyond the training set to a certain extent and i think it's just using the everyday concept of reasoning in an everyday way to call that reasoning.

13:18I'm just thinking back to some of the different characteristics that the earlier philosophers wanted artificial intelligence to have and reasoning being one of them. But then also the Turing test, which of course, you know, gets brought up all the time about a way to test for the capability of an artificial intelligence. I mean, it's kind of controversial, right, I suppose, in terms of how good it ever would have been as a test for the capability of AI. What's your take on it? Do you think it was ever a good test no i thought i've always thought it was a terrible test but a but a really great um spur to philosophical discussion about about things and again with a bit of hindsight maybe i might backtrack a little bit on a few a few of my views because i i was certainly very very much of the opinion that embodiment was a critical facet of intelligence was critical for achieving you know in intelligence which doesn't come anywhere near the turing test at all right no The Turing test is absolutely explicitly nothing to do with embodiment.

14:16Because in the Turing test, you have kind of two subjects, as it were. One is a human and the other is the computer. And then you have a judge. The human judge can't see, you know, which is the computer and which is the human. And they're only talking to these subjects through a kind of chat-like interface. They can't see whether they're embodied or not. So we can easily suppose that the computer might be one of today's large language models. in which case you know i have to say that today they were pretty much would pass the turing test you know i mean we've got to that point which is amazing really so i used to think that it was a bad test because it didn't test any of these embodied skills so so you'd need a robot really to test whether something was capable of the kind of everyday cognition that we all put to use when we're for example making a cup of tea or something because otherwise it's a very very narrow form of intelligence yes it's all to do with language and and reasoning and not to do with the kinds of things that evolution was you know developed in us and in other animals before language right which is the ability to manipulate and move around with and navigate and exploit you know in the best sense of the word the everyday physical world so actually that's really interesting because i often think about how fine maybe the large language models we have at the moment can pass the turing test but they don't flinch if you throw a ball at your computer oh no indeed and in a sense there are these sort of as you say these much deeper forms maybe we wouldn't class them as intelligence in a in the way that we talk about it but ultimately they it sort of is a form of intelligence well i think it very much is a form of intelligence and moreover i think that in the biological case and now i have to caveat all these things by saying in the biological case you know our ability to think and to reason and to talk is very much grounded in our interaction with the everyday world if you think about almost all of your everyday speech is using spatial metaphors i mean they completely permeate uh our everyday speech even the word permeate yeah absolutely grounded i use the grounded that you know so so so we just use those kinds of things all all the time because we're fundamentally physical beings because we're fundamentally physical beings and because our brains have evolved to help us to to survive and reproduce in this physical world while interacting with all these other beings that are doing the same thing, right?

16:33Because there are some alternatives. When you are trying to test for the capability of an artificial intelligence, just talk me through some of the potential alternatives that we have. Well, I think perhaps you've got in mind the Garland test, what I call the Garland test. So that goes back to the film Ex Machina, which was directed by Alex Garland, of course. And there's a bit in the script where Nathan, the billionaire guy, is talking to Caleb. and Caleb, who's the guy who's been brought in to interact with Ava, the robot. And Caleb says, oh, I'm here to kind of conduct a Turing test on Ava.

17:06And Nathan says, oh, no, we're way past that. Ava could pass the Turing test easily. The point is to show you she's a robot and see if you still think she's conscious. Wow. And that's what I call the Garland test. And it's different from the Turing test in two respects. So first of all, the sort of judge, as it were, who in that case is Caleb, can see that she's a robot. So in the Turing test, the judge can't see which is which. But here, the idea is that Caleb knows that she's a robot, knows that her brain is an AI brain. And yet still attributes these characteristics. And yet still, yeah. And the characteristic in question also is different because it's not intelligence, it's not can she think, but is she conscious or is it conscious, which is an entirely different test.

17:51And I think, you know, intelligence and consciousness are different things, so we can disentangle those two things, dissociate them. So when I first read the script of the film, those particular lines were in there for Caleb and Nathan, and I wrote next to it in my version, spot on, because I just thought Alex totally nailed a really important idea there. And so in my writing, I call this the Garland test, and quite a few people have picked up on that and call it the Garland test as well. Is there a test that would really impress you if an AI was capable of passing it? So I always was very impressed with Francois Chalet's ARC test, and that's ARC, which stands for Abstract Reasoning Corpus.

18:28So these are little sequences of images of the sort that, you know, you get in IQ tests and things. And the images are arranged in pairs. So you have the first image. It's kind of pixelated image. It's got little, you know, cells and with little kind of things that you can interpret as objects or lines and so on in the images. And the challenge is to work out a rule that takes you from one image to the second one. and then you've got to apply that rule to a third image. And the great thing about those tests were, first of all, that he held out and made completely secret all of the test ones. So you couldn't game it by kind of knowing what the actual test versions were.

19:07Or using it in a training set. Or using it in a training set. That's sort of what I mean by gaming it. And also he very carefully designed them so that it was very different rules. Each rule, you know, was completely different to the other rules. And you usually had to find some kind of intuitive application of often our everyday common sense knowledge. Seeing this as like a liquid that's moving in this direction or imagining this thing moving, you know, growing or something. So it required grounding in a way. Well, it seemed to. But recently, you know, people have been able to make significant progress on these in a more brute force kind of way.

19:41so I feel that the solutions are not really you know getting at the spirit of the original test quite so much well that's it I guess in a way is that as soon as you as soon as you set a metric as soon as you set a bar for once we've crossed this this threshold then we will have capability intelligence consciousness whatever it might be it it sort of changes the the the whole nature of the test in itself people are going to start you know optimizing for the test right it's good heart's law right absolutely absolutely a lot of people who've come on this podcast have told us to be really cautious about anthropomorphizing ai are you one of those people well i think there are different ways of looking at this and i think there are sort of good and bad forms of anthropomorphization so on the one hand people can start to form relationships as they see it with ai systems friendships and companionships and mentorships and you know that can potentially be a bad thing if they are misled into thinking that they can trust the thing that they're talking to or that they're really in love with it or that it really cares about them.

20:46On the other end of the spectrum, if an AI system is just using the word I, then I think that that's a pretty harmless form of self-anthropomorphization. We even see buses that say things like on the side like I am out of service and we don't have a problem with that kind of thing. So I don't see why we should have a problem with that with large language models either. But you know, I think we do tend to anthropomorphise things. And we had sat-navs in cars that weren't just in our phones. I used to anthropomorphise the sat-nav all the time. I used to think, oh, you know, stupid thing, it thinks we're doing this.

21:15And it's a natural human tendency, I think. What about the other words that we use? I mean, the example that you gave of the sat-nav saying, oh, it thinks we're in the car park, or, oh, it believes that this is, it got this wrong, it misunderstood this. Those are all very human-centric words, aren't they? Yeah, yeah, absolutely. So there are examples of what philosophers often call folk psychology. So we have this folk psychology where we use words like belief, concepts like belief, desire and intention, which we can apply not just to other humans and other animals, but we can apply to objects as well.

21:49It's what the philosopher Dan Dennett called taking the intentional stance. so we adopt the intentional stance towards something if we talk about it and think about it as if it acted on the basis of having beliefs and goals and carrying out rational decisions for what it does on the on the basis of those things and that's a very useful way of thinking about many many things such as even our sat nav or a chess computer so for dan dennett that was one of the examples that he used a chess computer that oh you know it wants to get the the queen forward because it thinks I'm going to use my rook to defend this rank or something.

22:27And that's full of this kind of intentional folk psychological language about beliefs and goals and things and so on. Is that problematic then if we start using that idea of beliefs and intentions and desires about the AI? So it's only problematic if we start to use these things in ways that mislead us into thinking that things have capabilities that they don't really have. Say the Encyclopedia Britannica, right, the physical volume of the Encyclopedia Britannica doesn't know that Argentina won the World Cup because it was too old. So if you made that remark, it would make perfect sense. You know, you might say that and it's fine.

23:05But then somebody said to you, why don't you have a conversation with it about England's football prowess, you know, or lack thereof, that would be ridiculous, right? Now, the interesting thing is that now we've got these large language models, you can have a conversation with them about, you can say, tell it things. so that it kind of pushes the boundary of where we might start to say, well, it doesn't really X, Y, Z. It pushes that a little bit further out. I wonder if there's something even deeper here about this human need or maybe it's just a desire to really want AI to have these characteristics, to be anthropomorphized.

23:40Yeah, yeah. Well, that's a really interesting question, isn't it? So I don't think it kind of comes back to that. It comes back to language. You know, in this case, we're inclined to anthropomorphize things because they're really good at using language. And for us, the only things that are good at using language are other humans. So it's very strange in a way to be suddenly in a world where we have language using things that, you know, it's not just humans that can talk. That's astonishing. Yeah. I mean, it is astonishing. It is astonishing. And it really is astonishing to think that every single child born today, they're going to grow up in a world where they've never known a world in which machines can't talk to them.

24:21Isn't that an extraordinary thing? Yeah. I mean, it really is. And so what the implications are of that for us all is really hard to say. I'm just thinking back to what you were saying about how grounded humans are in the physical world. Yes. It does feel like the kind of embodied aspect of AI has lagged behind this language aspect quite a bit. Do you think that we're going to see a big upstep in intelligence, however you want to define it, or broader capabilities once we get good and effective embodied AI? Well, I think it might make a big difference because the large language models we have at the moment, it's really difficult to discern, actually, to be honest.

25:08right now where the limits are for how good they're going to get whether we really are on the road to producing you know general intelligence that's comparable to human general in general intelligence and often you know when you you get to the sort of the boundaries of the capabilities of these kinds of things you sort of get the impression that the ai system doesn't really quite grok you know something it doesn't really deeply understand something you reach some kind of limit and you realize that it's been faking it a little bit but it may be that sort of general ability to really kind of get things on a deep you know kind of common sense level maybe but that does still require a bit of embodiment it does still basically require training data that involves interacting with a real world of physical objects with their spatial organization and there's something fundamental about that okay if understanding then however we define it is something that can emerge as a just a consequence of more and more data what about consciousness I mean I'm sure you've been asked a thousand times about AI consciousness and whether it's something that we can expect to happen or or has already happened yeah yeah I think the very first thing to point out is that I do think we can dissociate you know intelligence or cognition and cognitive capabilities, we can dissociate that from consciousness.

26:31So I think we can imagine things that are very capable and have, you know, whether we want to say are very intelligent, because of the way they can achieve their goals and so on, but that we don't want to ascribe consciousness to. But actually, what does that even mean? To ascribe consciousness to something at all? And I think the concept of consciousness itself can be broken down into many parts. It's a multifaceted concept. So for example, we might talk about awareness of the world. And in the scientific study of consciousness, there are all of these experimental protocols and paradigms, and many of them are to do with perception, you know, and you're looking at whether a person is aware of something, is consciously perceiving something in the world.

27:15Large language models are not aware of the world at all in that respect. But there are other facets of consciousness. So we also have self-awareness and our self-awareness part of that is awareness of our own body and where it is in in space and so on but another aspect of self-awareness is a kind of awareness of our own you know inner machinations of our stream of consciousness as william james called it so we have a we have that kind of self-awareness as well and we have what some people call metacognition as well we have the ability to think about what we know and then additionally there's the emotional side or the feeling side of consciousness or sentience so the capacity to feel the capacity to suffer and that's another aspect of consciousness now i think we can dissociate all of these things now in humans they all come as a big package a big bundle but we only actually have to think about non-human animals to realize that we can kind of start to separate these things a little bit because much as i love cats i think there's a limited self-awareness going on in cats how dare you well you know i'm a big cat person i have to say so i do say that with some hesitation and you know there's little metacognition shall we say well yeah certainly they don't have an awareness of their own ongoing stream of verbal consciousness because they don't they don't have it so they're so they're not they're not thinking about what they did yesterday in verbal terms or what they want to do with their lives so if we think about like robots you might have a very sophisticated robot even your robot vacuum cleaner and you may say that it's well you know it does actually have a kind of awareness of the world and that's not an inappropriate use of that phrase awareness of the world do i want to call it consciousness well then i seem to be bringing on board all this other stuff as well but you don't have to you can break down the concept of consciousness into these different aspects because your robot vacuum can know exactly where it is in a space and yeah and respond in a you know in an intelligent and sensitive way to where it is and the objects around it and achieve its ends and so on so there's a kind of awareness of the world there there's no self-awareness there's certainly no capacity for suffering and so in a large language model there might not be awareness of the world in that perceptual sense but maybe there's some kind of like in sort of self-awareness or reflexive capabilities reflexive cognitive capabilities they can talk about the things that they've talked about earlier in the conversation for example and and can do so in a you know in a reflective manner which kind of feels a little bit like some aspects of self-awareness that we have a little bit.

29:45I don't think that it's appropriate to think of them in terms of having feelings. They can't experience pain because they don't have a body. I think we can take the concept apart, basically. So then is the question, can AI be conscious or not, as though it's a binary thing? It's the wrong question from the off. I do think that is the wrong question. And I think it's wrong in many ways. So just then we were talking about the fact that it's actually a sort of multifaceted concept but also I think that we tend to have these very deep metaphysical commitments to the idea of consciousness as some sort of magical thing that is you know a metaphysical thing so the question of whether something is conscious or not is not a matter of consensus or a matter of just our language but it's something that is out there in the metaphysical reality or in the mind of God or in the platonic heaven or something like that but ultimately i do think that that's the wrong way of thinking about consciousness let's take one aspect of consciousness then that you that you described about the sort of emotional side an ability to suffer but not necessarily physical pain emotional pain too and sort of a sense of self in the emotional way do you think this is something that will just emerge as a natural consequence of intelligence if you build something that's intelligent enough at some point this is going to happen or is there something unique about biological creatures and i guess the process of evolution that we've been through that has resulted in that that can't be replicated in a machine yeah i don't think there is a right or wrong answer to your question there i think we just have to wait and see what things we bring into the world and and how we end up treating them and talking about them and thinking about them and i don't think we really know until they're among us as it were you know these things that we're building then we will just be led to think about them and talk about them and treat them in a particular way so an example i like to think of in this regard is the octopus so octopuses have recently been brought into uk legislation brought into the category of things that we have to care about the welfare of that's as a result of lots of things i think happening so the public has been exposed to being with octopuses a lot more.

31:59Now, you don't have to literally be under the water and poking around with octopuses to know what it's like to be with them because there's all kinds of wonderful documentaries and wonderful books by Peter Godfrey Smith has these great books about interacting with octopuses and so on. So those sort of narratives and documentaries, they give us a feel for what it's like to be with an octopus, what it's like to have an encounter with an octopus. And then, you know, you can't help yourself but to see it as a fellow conscious creature. But complementing that is the scientific progress as well. So at the same time, scientists study the nervous systems of octopuses and realise the extent to which their nervous systems are similar to ours and the way that we experience pain.

32:40You can find analogous aspects of their nervous systems to ours. So taking all these things together, I think that tends to affect the way we think about them and the way we talk about them and the way we treat them. so i think the same kind of thing will you know is going to happen with ai systems do i think there's a right or wrong answer could we be misled there i think that's a really really deep and difficult metaphysical philosophical question i do wonder though i mean that that point about suffering to me seems different to the others because because metacognition you know the sort of sense of the world etc there's not these ethical implications necessarily about those But I think with suffering, like, you wouldn't want your shoes to be conscious, you know?