The Illusion of Neutral Technology: Dr. Kate Devlin on AI Intimacy, Hidden Harms, and the Scientific Method

0:00So the dirty secret of artificial intelligence, other than it not being all that intelligent, is that it's not always artificial. And so there's this huge amount of human labour that's unrecognised. And that includes people working in terrible, terrible conditions, doing jobs that no one should really be doing without support. Things like content moderation and looking through the worst of the internet.

0:27Hello, welcome back to Change My Mind. the podcast where we ask people about the time they changed their mind on an issue and why. I'm Laura Osborne, Managing Director of WPI Economics, and you've just heard from Dr Kate Devlin, who's a leading computer scientist and researcher working at the intersection of artificial intelligence, human-computer interaction and society. Kate is Professor of AI and Society in the Department of Digital Humanities at King's College London. She's co-lead on the Responsible AI UK programme and her work explores how humans interact with, react to and form intimate emotional engagements with emerging technology.

1:03You can read more about that in her acclaimed book, Turned On, Science, Sex and Robots. But before we get to that, I'd like to invite you to sign up to the newsletter at opendemocracy.net. We promote this show with Open Democracy to their 3 million regular monthly visitors. You can find the back catalogue to all our shows and more information on this episode at opendimocracy.net forward slash the polarisation project. I'm joined today by Ali Goldsworthy, a leadership lecturer at Stanford. She and I did this interview with our colleague Alex Chesterfield, a behavioural insight expert who's long been using AI in her research and her work.

1:36Alex can't join us this afternoon, but Ali fortunately can. Good afternoon, Ali. How are you doing? I'm doing great, Laura. It's lovely to be here with you. Yeah, and you. So what really struck you about the interview with Kate? Well, we recorded this back in the summer, just as Anthropic had started their outreach or the public parts of their outreach to the Pope. So it's not surprising, but as an expert in AI, Kate so expertly predicted how much of this was going to play out over the coming months, how it could be bewildering to try and keep up with the pace of change, something I'll confess to also feeling despite living right in the heart of it in Silicon Valley.

2:11Yeah, absolutely. It's definitely one of those really tricky moments of change where everything is happening so fast. It was also so interesting to hear Kate's thoughts on how people develop personal relationships with AI, which again, I think has only become more central to debate over the intervening weeks since we spoke to her. I think the emotional bonds people form, how this shapes their wider relationships offline is so interesting and is no doubt going to be increasingly central to discussions on the future of AI. So without anything else from us, let's hear the full interview with Kate.

2:45Kate, thanks so much for joining us from Sunny Norwich, We're really glad to have you with us this afternoon. We're going to talk about all kinds of aspects of your work. But before we do that, when I read your biography, I was really interested to see that you'd actually started your academic studies in archaeology. Could you tell us a bit about why you went from like a full on kind of historic interest to the kind of edge of technology? I like to tell people that with archaeology, your career is in ruins, which is a beautiful way of hitting it. You know, I thought there was no future in it. All of those puns.

3:17But actually what happened was I was very happy doing archaeology. I loved it. I found it really interesting. But in practical terms, trying to get work as an archaeologist is really difficult. It's contract to contract. And it all depends on what's happening and particularly around things like engineering and building so that you get this contract work. And so most people specialize. And I thought I would specialize in computer science so that I could do both of those things together. and then the more I did computer science the more I get into computer science and I ended up going full computer science but I had I keep an eye on archaeology because I like it.

3:55Yeah I was thinking it must give you a really interesting perspective on how the past and cultures of the past have shaped the cultures of the future. One of the things I was going to ask you about is obviously we think quite a lot about polarisation, you think a lot about technology, we think a lot about what speeds and breaks those spirals and I was wondering again thinking about the past and into the future how you see the role of AI in terms of it bringing people together or ultimately pushing people further apart? It's a pretty complex one because media discourse around this tends to polarise.

4:28So it tends to be you're either a doomer who thinks it's all terrible and it's going to either end us or destroy human society, or you're a boomer who thinks it's the best thing ever and it's going to transform the world and we'll all be living on some kind of universal basic income. so I think it's a lot more nuanced than that obviously and there is scope for people to be brought together by this technology and as a result of this technology but there's also scope for things to go quite wrong and to push people apart as well. Could you tell us a little bit about your perspective on each aspect of that what you think might bring people together and what you think might push them apart?

5:06I work a lot in the area of emotional engagement with AI and robots So that tends to be things like chatbots or social robots, things where people have a direct interaction with the technology. And one of the first questions I get asked when I talk to journalists about this is, does this mean an end to human-human relationships? And no, it doesn't. It really doesn't for a number of reasons, I think. But one of those is that the technology use itself then triggers conversations with other humans. And we are so intrinsically social. we look out for other humans we want to be near them we want to tell them about the things we're doing and so we see people forming relationships with chatbots and then going online to tell other humans about that and through that they form friendships and they get together in real life and they form their own relationships with each other.

5:57Ali I know you wanted to ask a couple of questions around the relationships point I'll hand over to you. I guess I just wanted to pick up on what Laura was talking about which is we also think a lot about identity and how people think about that. In terms of where people do form bonds with robots or chatbots, like how deep are they? Does it ever become parts of their identity? Is that really quite a fringe thing that people find easier to latch on to? It's not as fringe as you might expect. So there are dedicated apps for this kind of AI companionship, things like Replica or Character AI. Between those two main apps, there's around probably 50, 60 million users of that.

6:39That's only a fraction though, of the people who are using AI for companionship, because a lot of people are just going directly to the large language models like ChatGPT and setting that up to be their friend, essentially. And I guess what I'm driving at, and I don't know if your research has looked at, is I suppose at what point do people sit there and say, yeah, I've got a friend who's a chatbot. and then I suppose it's consciously altering their behavior. And at what point does it actually become a friendship that alters their behavior that they might not realize? We've been doing this kind of engagement with technology for a long time.

7:19So the very early chatbots like Eliza back in the 1960s had no AI in it, but people very quickly built up a rapport with something that would reflect questions back at them. So if you said, good morning, Eliza, this Eliza chatbot, which had very little sophistication, would say, why is it a good morning? And you'd be invited to chat and it was very quick to take off. So people do very quickly fall into that. And even the chatbot like Eliza that was really built to show how superficial these things could be, people began confiding in it very early on, even though they knew it wasn't real. And there's lots of human computer interaction research down the years to show that that's how we behave with technologies that seem to be like us.

8:02So does it change people's behavior? I guess it does in a way in that they are comfortable doing that and they're willing to confide things. They're willing to chat on that emotional level. Those feelings people have when they do form an attachment are very real to that person. But those aren't strange feelings either. We have that when we engage with other forms of media, with parasocial relationships that we form with celebrities, films, stars, characters in books, fictional characters. So that seems very profoundly real to us because we are experiencing them. So, yes, they probably do alter the way people behave.

8:36For some people, that will then go to quite extreme lengths. So there are people who may be more vulnerable, for example, who will get into spirals with these things. And then we have this new term, AI psychosis, not a term I'm a fan of, but this idea that people are sent into some kind of very negative, doom-filled spiral by these chatbots.

9:02Okay, you know, it's super interesting because I remember when I was doing my research in human AI interaction, really icky, first of all, about people using AI in this way. and then actually when I started to research the characteristics of these models that mean people open up it is actually things as well that they don't have they don't judge you whereas when you're speaking to humans so I was I was researching the interaction in the context of doing qualitative research and with qualitative research you always inevitably have these power dynamics that affect how you ask questions, how you interact, how people answer you.

9:42So yeah, it was these characteristics, like the fact that it doesn't feel like it's judging you, that means people open up a lot more, the questions are a lot better. Yes, absolutely. That is key that people don't feel they're being judged and therefore are more likely to confide. The opposite of that though is if they feel they've been deceived by a machine, they get really angry, but they're very accepting into it being non-judgmental, to giving you praise. Yeah, so that transparency is really important. Ali? Just before I move on to a separate question, is there an example where, sort of I guess in everyday life, where people do end up feeling that they've been deceived by machines and then get angry?

10:20I think maybe an example might be if you're talking to a customer service agent and you're pouring your heart out about why you've had a terrible experience with a company and then you suddenly find out it's actually a bot, you can get quite angry because you think it's a human sympathising with you and instead it's a bot that's feeding you platitudes. We spend a lot of time thinking about polarisation and what causes people to be apart or to come together. And there's a lot of interest in what that might mean in terms of AI, whether it's a divisive tool or brings them together. And some of the latest research looks, actually suggests that it can bring people together and encourage them to consider other perspectives.

11:06There's been quite a lot of hype around this. And I just, before we discuss it, let's get your take on how you feel about if that's likely to be how things unfold. I saw this research and I thought it was really interesting. So yes, it suggests that a chatbot can provide lots of different examples, lots of different points of view, and then people might be more inclined to listen to it that way and again I think that non-judgmental thing comes in as well but of course the risk is that the more information a chatbot presents the higher the risk it is of being completely fabricated because large language models are fabrication machines and we know there are problems with the accuracy of the things that they present and if they're presenting you with a bunch of information you don't really know what's true and what's not so they could actually exacerbate the problem.

11:54And I think I'm pretty I'd say I'm pretty skeptical about this research. Well, very sceptical, I think, for a couple of reasons, which is one, like there's always a real danger once someone reaches the point of taking a position that it's really hard to update it, right? Like it's much easier to live in the grey and then take a view than to take a position and try to admit you were wrong, which is, you know, I suspect a difference between humans and machines. And the other one is, I don't know if you're familiar with Chris Bale's work that looks at like, particularly there used to be some of the same hype around like social media, like you can just expose yourself to like different viewpoints and that will bring us all together.

12:30And what he actually found was that people got offended by like having their identity or something. And they were like, who are you trying to do that? And they held on to their initial view and their initial identity more strongly. And I didn't know if like my initial working hypothesis would be that is exactly the same, how it will play out with information that comes from AI or from a chatbot. Maybe you'll tell me that actually we should have much more hope than that. no I sort of agree with you just thinking about things like conspiracy theories and if you try to go in adversarially and tell someone that they're wrong they just dig down so I'm kind of surprised by that research um yeah yeah I'm I'm very surprised and I have not yet but I will for our debrief go and look into exactly who funded it and see if it and what that means about my own biases from within there I did want to ask you about your book which we'll put the details of in the show notes, which is Turned On Science, Sex and Robots.

13:27And I just wondered how your work with robots had prepared you and like, I guess, the emotional attachments people form with them. How that had prepped you for some of the work that you're doing now on AI? When I wrote the book, there was no conversational AI like we have now. There were some chatbots, but they would fall over very quickly and didn't really hold the context of a conversation. So it was long before we had chat GPT I was writing back in 2017 what I was seeing though was something that we've known from the human robot interaction and human computer interaction fields for a long time which was this sociality that we feel towards something that exhibits any form of being alive even though it's not alive so if it moves if it has something that resembles a face if it you know you could stick some googly eyes on it but even if it just you could make all its features we immediately attribute those qualities onto it so the anthropomorphization the zoom morphization of these things.

14:20But when I was writing that, I was looking specifically at sex robots, which I thought, that's ridiculous. I was working in robotics. I thought, that's ridiculous. It's never going to be a thing. And I still stand by that. There was a company trying to make the first sex robot to go to market. And I went and talked to them, met the world's first commercially available sex robots. But all along, I kept thinking, where are you going to store this? Where are you going to put it? How are you going to look after these things? They don't quite look human. They're uncanny. They don't quite meet our expectations.

14:53Things are expensive and difficult to make. So to me, the very natural step was that people would have something they could carry around with them all the time, like an AI on their phone. And then a few years later, we get the transformer architecture that underpins things like ChatGPT, which means that the conversations have more context. And suddenly that became possible. And so now we see this massive uptick. And it's moved from being quite a gendered space where it was aimed mostly at straight men. So we've seen a lot more women now with AI companions, which is no big surprise if you consider the fact that online is a bit of a toxic cesspit.

15:32if you're a woman. Why not have a nice respectful AI boyfriend instead? The growth market, well, I'm sat in Silicon Valley. I presume that is the growth market where investments are going in now. And where did people, I'm just curious, where did they store that robot? Well, by the time I'd written the book, it still wasn't commercially available, but I talked to a community of men who own sex dolls. And that is quite a big commitment in terms of space, in terms of money, in terms of all of these things. And a lot of people who have these dolls would treat them as if they were a partner and so would have their living areas set up to accommodate it as if it was an extra person.

16:10I guess that compounds the relationship that further humanizes things. That's right yeah. Yeah it takes away the visual of it being in the cupboard which is what I had in my head. Sorry back to you Alex. Hey I would love to ask you a bit about harm and and culture so I do a lot of work on why organizations and institutions fail and the link between failure and corporate culture. So the kinds of things I'm worried about is often the homogenization that everyone's using these same underlying language models and we see the effects. Individually, it's great for productivity, but often for science more generally and creativity, it can be shrinking.

16:49Also sycophantasy. But I've noticed or I've seen or I've read that you've said, we focus too much on the existential risks of AI, which I agree can be extreme so I was curious where do you see as the most or what would you see as the most urgent kind of AI harm that keeps you up? There's quite a few of them it's the it's the very mundane sounding things which aren't that mundane they are a threat so things like the environmental damage that this does right you know it's something that's just sort of seen as a side effect but these data centers are using huge amounts of water and energy um we are burning the planet to produce some absolute dross right the return isn't even all that good so that's one of the things i'd say another is privacy people are just pouring their thoughts or their corporate work into these machines without knowing who's going to own that data that's also a problem and then there's a huge amount of hidden labor in the supply chain so the dirty secret of artificial intelligence other than it not being all that intelligent is that it's not always artificial and so there's this huge amount of human labor that's unrecognized and that includes people working in terrible terrible conditions doing jobs that no one should really be doing without support things like content moderation and looking through the worst of the internet so those are kind of the things that might not immediately strike you as being a threat but i think they're happening at such a scale and they're harmful they're directly harmful to people that that keeps me up at night yeah yeah and what about the term the pluralistic plural I can never plurality pluralism which links back to their medionate can you say a bit more about that yeah yeah so the large language models that people use whether it's chat GPT, Claude, Gemini, Copilot or Mistra all of these different ones a lot of them are based on the same underlying platform and we are losing the long tail of information so I have a PhD student wonderful PhD student who's working on this and she's looking at how information becomes homogenized in these platforms so that what tends to come out at the end we're losing the voices of those who perhaps haven't contributed proportionally to the data in the first place so their their knowledge is not in that system and then the system bias itself is just producing something much more flattened than it could be and this is a big problem and we are losing that representation and we don't have a good way of making things plural.

19:22I would love to come back to that but I want to come back to that but I want to I want to get on to the big question which is tell us about a topic or something that you have changed your mind on. I have been racking my brain to think about this I think one of them is science itself so when I started doing computer science I came from an arts and humanities background with archaeology it has a science aspect to it and then I did a master's and a PhD in computer science and it was very much from a scientific perspective I knew you have to robustly test things I did a lot of experimental psychology in my PhD so there's very rigid guidelines about how you run those experiments around perception and cognition.

20:05And I think that's great and there's a place for it, but it's seen as being the truth, the objective reality. And of course it's not. And the more that you look into it, and science and technology scholars know this, it's flawed in its own ways. And I think I went from thinking that there had to be this incredibly objective approach to trying to look at it more subjectively and that we have to observe how this stuff happens in the wild. It's not enough to just test it in labs, but that seems to still be the gold standard for a lot of people. So I think I've changed my mind about thinking science can only happen in a lab under controlled conditions.

20:50Yes, that's a big part of it, but we need to include people and it's about having that very in-depth sociotechnical, sociocultural approach to it. No, that resonates a lot. I originally did English literature and then did cognitive science and now a PhD. And I was trained very much like the experimental method is the gold standard. But then you, as I said, I work, I mean, I'm mainly a practitioner in the real world. And you're like, this is so far from how things work. So I'm curious, was there a specific example or a moment that triggered you to change your mind? Not really a specific example, just more the constant pushing of you must do this in this particular way I don't like being told what to do I think even recently I've noticed it from things like the tech companies last year OpenAI did a study with MIT and they said it's only a fraction of cases where people are engaging emotionally with chat GPT and I just think that's so wrong because every time you come to interact with a piece of technology, you're bringing your whole bag of emotions with you, whether you're shouting at your printer or getting really annoyed because the Wi-Fi has gone down.

22:03You're always bringing those emotions. You always carry them with you. So to try and claim that there's a scientific basis to something that is absolutely resolutely objective, it's just wrong. I wanted to ask you actually, Kine, thinking about those big organisations that sit behind a lot of AI. How some of those perspectives that impact their own cultures might be shaping what we as the kind of user get at the end of the AI experience? I'd be really interested to hear your take in that. The technology that we use absolutely comes imbued with the values of the people who create it. There's no such thing as a neutral technology.

22:47That is hard to convince computer of scientists and engineers off and they don't necessarily agree. I hear lots of rhetoric from them about how it's just a tool, it's how you use it. That's not true. They're designing with particular objectives in mind. They're bringing their own values to the process. How hard do you find it for people to accept that? It's pretty difficult for people to accept that. Today with people I work with will say, I don't think you can say that it exists in a vacuum. A couple of years ago, Sam Altman was doing a charm offensive round Europe to talk about the work that OpenAI does.

23:30And someone asked him, don't you think that ChatGPT is contributing to disinformation and misinformation? And he said, no, that's not us. That's social media sharing all this stuff. And I thought, wow, you've really convinced yourself of that, haven't you? I find it really, really interesting I'm sort of sitting geographically where I do like the optimism around AI sits it's just it's really incongruous actually with then how I experience if I leave if I leave the Stanford campus you know and even when you're there you'll get people I think Daniela Ramody was in the other day doing a talk and like I find her a fascinating individual and actually thank god there's someone who's an interdisciplinary thinker at that point doing it but you know what a weight on her shoulders.

24:16And she'll sit there and say, oh, well, social media was under-regulated, right? Precisely. And we'll kind of say, well, it should have been done. And then you're like, well, what extra regulation should you have on AI companies at the minute? And how do you balance that with how quickly you're trying to get to market in an international thing? And people just don't have answers to those questions. Like, I don't see anybody either within companies or within government knowing what that response is. No, it's seriously under-regulated. It's so difficult to do. I spent the morning delving, shall we use a chat to you, we're delving into the last of the Pope's encyclical about AI that was released this week.

24:54And I was struck by the fact that the Pope was saying all the things we've been saying all along about how this is a big deal that we shouldn't have the power concentrated in Silicon Valley, that we've got to think about the dignity of the human and put the human first. And I think that's weird coming from the Catholic church about power you know this is something that they've been they've held for years so you know is it different power if someone else holds it what's going on here and especially when you know they start talking about things like you know the dignity of women they're second class citizens and I thought you're the catholic church what I find really interesting is the fact that anthropic were there for the launch event of this so while the pope is saying we must be really careful we must think very carefully about artificial intelligence and who gets to decide these things, but then has a launch event with one of the major players in the field who are cleverly positioning themselves as being the more moral of the companies when they aren't.

25:53They're doing lots of things that show they're very similar to the other companies. And, you know, conveniently, they're trying to go public. Interesting.

26:07Yeah, it raises a lot of questions, doesn't it? I won't get into the Pope, but I think from a corporate culture perspective and positioning yourself a certain way ahead of key milestones is a quite typical and cynical response. Yeah, so not overly surprising, perhaps. I was going to say, Kate, one of the questions we always ask people if we have time, which I think we do here, is who would you love to hear from who might have changed their mind on something? Who do you think would be a great person to answer that question. That's really tricky. I'm not sure. You can have a minute.

26:45Yeah, who would that be? Don't choose an Obama. Everyone chooses an Obama. Oh, really? No, no, no Obamas.

26:55Dr. Kate Devlin: I don't know. Someone in the culture sector, maybe? but I don't know who who do you think it would be hardest for to admit publicly that they've changed their mind

27:15I don't know because some people really thrive on it don't they they make a career out of the the prodigal return tech rose in particular um none of those but whether they've admitted they've changed their mind is a completely separate thing right like Like as opposed to saying, well, something underneath has changed. So my core belief is still the same, but how I apply it might be different. Right. Like that's really easy for people to do, including tech pros are no different to other people in that situation. Or tech sisters, but there's clearly far fewer of those. But like actually saying, no, I made a wrong judgment call based on this.

27:53And my underlying world belief is much harder for people. all. So I do have someone who I can think of who started off, it's Chris French, who is an anomalistic psychologist. And he started off getting into parapsychology because he really wanted to believe in things like UFOs and ghosts, and then ended up making his entire career someone who studies why the hell people believe this when he can't find any evidence for it whatsoever. So he's really interesting because he went into a field thinking one thing and then emerged leading that field in a completely opposite direction that's a great one I was gonna say that's also a great kind of story about science isn't it about the you know updating your prize and changing your mind we should get him on Chris if you're listening yeah Chris French yeah okay before you go there is one other question which I could you're Northern Irish I'm guessing from the oh yeah from other interactions I've had with you like just how polarization has clearly manifested in Northern Ireland and going from there how much does that inform both your take and what stories do you wish that the world heard from Northern Ireland that could help inform some of the current debate around AI if any?

29:06Yeah so I did grow up in Northern Ireland during the Troubles which was at the time seemed very normal when you're in the middle of it but looking back it really doesn't and I was I'm the child of what they call a mixed marriage so I have a Catholic father and a Protestant mother I grew up in a Catholic village but I went to a Protestant school so I sort of sit in the middle and when the Good Friday Agreement happened it was the first time that someone like me was allowed to be both things both cultures because before that you had to pick a side you felt obliged to be on one side and so I've been very grateful that I can now say that I'm both and I do feel both so I think what people need to know about that I think if you watch Derry Girls it sounds ridiculous but it's one of the most wonderful tributes to growing up at that time I mean I'm slightly older than the characters in that but it captured life so well and the humour is so accurate.

30:10And I think it's a really lovely glimpse into how things wear. And a lot of it, while people will say, oh, it was about religion, that's not quite true. It was about power, it was about status, it was about socioeconomic status, it was about lots of different things. Religion was a handy hook to hang it on. I think it's that Northern Ireland was always somewhere that had tenacity, it still does, and the fact that we went through a peace process and it worked for the most part is a lovely thing. Brilliant. Thank you. Sorry, I could not ask you about that. Go from there. We should put a link to the Derry Girls in the show notes as well.

30:52Oh, we will put a link to the Derry Girls. And I actually know one of our other guests, who's a professor at Stanford, who's Scottish Indian, actually really tries to, he talks about Derry Girls in class. Oh, wonderful. the role of humour in depolarising and how it makes it easier for people to talk about the past and how actually there can be ways ahead even out of a lot of gloom that that can happen which is certainly what I take from Northern Ireland. It's imperfect current position but definitely much better than it was when you were growing up. Kate thank you so much for joining us today we've had a lovely time chatting to you and I guess we'll wrap up there.

31:37Thank you again to Kate for sharing her insights on such a topical and divisive set of issues. While it's reassuring to know she doesn't see forming relationships with AI as the end of human to human relationships, it does definitely give pause for thought on what the future holds. It was also fascinating to hear her views on how science actually happens, especially at these kind of really fast moments of change, and why she changed her mind on that. And also why the most important breakthroughs are actually across labs, people, experiments, etc. I'm really not confined to just one discipline or place.

32:06What struck you, Ali? Yeah, Laura, I totally share that. I mean, great discoveries often happen when people from different parts of life or with different experience and different knowledge get together and scrutinise each other. I guess that's one of the reasons that this podcast has become so important and so effective. But one thing that I was so struck by was Kate talking about how it felt being someone from Northern Ireland in a time of polarisation because, you know, it has run so deep with such tragic and severe consequences over many decades, many centuries there, the consequence of division.

32:42And, you know, even though now we've sort of had peace for, you know, 30 odd years, how precarious those things can be. And there's those norms that have really tried to help protect society there, but also in a period of transition between leaders who initially negotiated those peace agreements and now they are moving on, they're passing and successes are coming in. They don't have quite the same stakes that people did. And I think that that's such a profound lesson that other people can try and learn from and how you try and protect sort of democratic norms. But I'm also always down for a bit of humour and Derry Girls, which she talks about, has been so important in trying to help people understand what's gone on in Northern Ireland and how you try and survive and rebuild.

33:28Yeah, absolutely. I think Derry Girls films most of the educational basis from what my children understand about Northern Ireland, for better or for worse. So that's all from us today, but we will be back very soon with an extra special live edition of the podcast with Bishop Rosemary Mallett, who led the Church of England's response to slavery, and editor-in-chief of Open Democracy, Aman Sethi. You're in for an absolute treat. It was such a pleasure to record it. As always, you'll find more information on the studies we discuss in the show notes, and we'd like to thank Open Democracy for working with us to share this with their listeners.

34:00James Battersill, our producer, as well as Kevin MacLeod, whose music Dreams Become Real plays us into and out of the show.