Carissa Véliz on the dangers of predictive AI

0:01Hello, and welcome to the SIFTED podcast. I'm John Thornhill, SIFTED's co-founder and columnist, and today I'm joined by Carissa Véliz. Carissa is Associate Professor of Philosophy at Oxford's Institute for Ethics in AI, where she studies how algorithms and data are disrupting our democracies, our right to privacy, and who ultimately holds power in society. She's the author of Privacy is Power, an economist book of the year in 2020, and The Ethics of Privacy and Surveillance, published in 2023. Her latest book, Prophecy, Prediction, Power, and the Fight for the Future, From Ancient Oracles to AI, came out earlier this year and has already been described as one of the most important books you'll read.

0:43I found it a brilliant and provocative book that defies much of the conventional thinking in the tech world today. And I thought Sifted listeners would be really interested to hear Carissa's views. Today, we're going to talk about how AI predictions and forecasts can be a force for political manipulation. We'll ask whether tech companies can ever really be trusted to self-regulate and what it actually means to build AI that serves people rather than surveils them. So, Carissa, welcome to the show.

1:13Carissa Véliz:Thank you so much for having me, John. Let's start. Why did you write this book? Because I was frustrated with the debate. I thought that when we talk about AI, we are focusing on mostly the wrong things, asking the wrong people and not going to the fundamentals of the problem. Okay. What would your one minute summary of the main thesis of your book be? That digital technology was born out of two sins of design that could have been different. The first one is that it's made to surveil, and the second one is that it's made to predict. And both of those, surveillance and prediction, lead to social control and don't sit well with democracy.

2:00Okay, let's dig into predictions. And there was a book that came out a few years ago written by three economists, published in 2018, I think, called Prediction Machines. and they pointed out that machine learning systems are very good at spotting patterns in vast sets of data that are hard for humans to spot, predict weather, predict protein structures, can spot correlations between different stocks when the ECB increases interest rates and so on. They make a strong case that prediction machines can be good. They can lead to more efficiency. They can lead to reduce uncertainty. Is that right? Can prediction machines be a good thing?

2:41Carissa Véliz:Yes, but with so much nuance that we end up disagreeing. So of course, prediction can be good. I look at my weather app every day. I think that the government needs to make some kinds of predictions in order to plan appropriately. In some ways, we cannot avoid predicting. But I think we tend to be painfully and dangerously naive about prediction in the sense that we tend to be very optimistic about what we can predict. We tend to forget what we can't predict. And we don't make a distinction about what we should predict for moral reasons and we should not predict. And there's a very important distinction that often doesn't even get made between making predictions about things.

3:24Carissa Véliz:So if I make a prediction of a molecule, it's not going to affect the molecule. and making predictions about people. That's very interesting you say that, because I was thinking in isomorphic labs, this spin out from DeepMind has just raised a lot of money. And they are, in effect, trying to predict how new drugs will work when they interact with the human body. And that's probably a good prediction, isn't it? Because it's predicting things, as you talk about, rather than people. Yeah, and of course, the devil usually is in the detail, but presumably there will be a prediction based on modeling and kind of theoretical assumptions.

4:02Carissa Véliz:And then you get to test that in clinical settings with peer review and randomized control trials and so on and so forth. And that makes it not only a very different kind of use of prediction than, say, using AI to predict who's going to be a good employee, but also a very different method. And that is scientific. If you use it to generate hypotheses, which you then tests scientifically, that's a very different kind of activity than using AI unreflectively. You have this lovely line in the book, and we're going to talk now about how predictions apply to humans, that algos resemble their creators in the way that dogs resemble their owners.

4:39And we tend to think of AI as a kind of neutral and objective, but they are very much the creation, as you write, of the kind of inbuilt belief systems of those who produce them. And it reminded me very much of the kind of quote from Kathy O 'Neill that algorithms are opinions embedded in code. So can you tell us about that? Why should we be wary about algorithms when they're applied to humans?

5:04Carissa Véliz:It was truly fascinating to research this book. It was quite the adventure because I wasn't sure where it was going to lead me. And one of the places it led me is to the development of statistics and the history of statistics. And one fascinating thing to note was that In the transition between small communities in which you could trade with neighbors and friends and friends of friends and people you were connected in some meaningful way with, to societies of strangers in which you're dealing with people you've never seen in your life and maybe will never see, we need some bridge to be able to trust.

5:41Carissa Véliz:And often when we stop trusting human beings, we come up with numbers as that bridge. But we tend to forget that it's people who make up the numbers. And when you go deep into the history of how we started making quantified decisions and quantified predictions instead of qualitative ones, you realize how much creativity is involved. And I tell this story. I was having lunch in Oxford one day, talking to my colleague and telling her about how I was reading for this chapter. And she's a professor of accounting, Annette Mikes. And I told her, you know, it just seems to me like there was so much creativity in this process.

6:19Carissa Véliz:And she was like, yeah, it's all made up. And she said it so matter-of-factly that I almost choked on my food. But it's true. For the states, statistics is a kind of perception. But in that act of perception, there is a lot of creation. And as you write, predictions are often made to please the powerful. Rather like LLMs, it seemed to me, are designed to be sycophantic and to please the people who are using them. And you have one lovely anecdote about Louis XI. Can you tell us about that and the astrologer's prediction? I love that story because it illustrates so well the intimate relationship between prediction and power.

7:04Carissa Véliz:So as the story goes, Louis XI had an astrologer in court, And one day, the astrologer predicted that a lady of the court would die in the next seven days. And she did. And Louis was terrified because he figured either this person murdered the woman, just to prove his accuracy, which would make him someone pretty dangerous, or he can actually see the future, in which case he could see maybe Louis' own demise, and that made him dangerous in a different way. So he decides that the astrologer has to go, he needs to be murdered. and he tells his servants that he will give them a signal and upon the signal, the servants are to throw the astrologer out the window and it was a very high window and so he would be certain to die.

7:46Carissa Véliz:But before giving the signal, he meets one last time with the astrologer and asks him, given your prophetic abilities, tell me how long will you live? And the astrologer, who was not stupid, said, well, I will die three days before your majesty. And go figure, Louis never gave the signal. And your book, you very much go into the history of the ancient oracles, the seuthers, the astrologers who came along. And you argue now that a lot of the tech executives and the AI evangelists are really the inheritors of that tradition. Can you tell us more about that? the people who are now saying that the prophets of our age are in fact the heirs to a very long tradition?

8:37Carissa Véliz:Yeah, it's fascinating to see patterns in history because they kind of make it much more obvious and salient, the power plays behind the actions in a way that when we're in the present, we get distracted by all kinds of details. And so we tend to associate tech executives with science because they are the creators of AI. And AI can be used for science, but it's also used in very unscientific ways, in a way that technology can. So technology and science, there's a connection, but there's also a distinction there. And when you look at the history of power, you always see prophets, quite nearby leaders.

9:14Carissa Véliz:And many, many thousands of years ago, it used to be the Oracle of Delphi, and then it was seers and and then it was medieval astrologers but if you had interviewed for a podcast a medieval european and asked them about astrology they would have said something like well it's our cutting-edge method of making decisions it's a very opaque science it's very complicated it's very hard to come up with the numbers and you need a lot of a lot of knowledge and i think we say something like that about AI, it seems very mysterious, it's very opaque, it's very technical. But interestingly, even though the technology is extremely different, the political role that it's playing and how these people are talking about the future, and more importantly, how we respond to those claims is incredibly similar.

10:05Carissa Véliz:And one of the reasons I wrote this book is that it astonished me. When I began to conceive it, I made a search for the ethics of prediction. And even though there are thousands of books about prediction, about how to do predictions, about forecasting, there are scientific journals about it, there are articles, there isn't one book about the ethics of prediction. And how astonishing is that? But it's not rare. So, again, going back into the research for this book, one of the little details that blew my mind was how it took us thousands of years of using dice to realize that seven was the most common number.

10:41Carissa Véliz:It's just nobody had noticed. And that realization was explosive for the development of statistics. And as you're saying, the tech companies are now claiming that they can predict. And indeed, Palantir is named after the omniscient crystal balls in The Lord of the Rings. And one of the ways that tech companies obviously can get better at prediction is by amassing more and more data and surveilling people. Can you tell us about that? one very important reflection is that is thinking more deeply about how there are many ways to make a prediction come true and the intuitive and naive way to think about prediction is that you're discovering the future as if it's already written and you're just figuring it out and you're not influencing it but more often than not a prediction is made true by influencing the future so to just set an extreme example just to prove the point.

11:36Carissa Véliz:Like in the case of the astrologer, one way to know whether somebody is going to be dead in a week is if you are willing to kill them. And so when tech executives and technology companies amass so much data and they use it to predict, it's not only that they're figuring a pattern that is out there and independent of them, is that they're creating that pattern. So when a technology company uses all the data about you, about your personality, your habits, your history, your connections, and then nudges you to vote for this candidate or not go to vote or buy this product, it's not that they're figuring out who you are, it's that they're partly influencing who you are.

12:18I was also struck by your comment that startups shouldn't use Google Docs and that you should be very wary of using e-books because they're informants. Can you explain that?

12:31Carissa Véliz:Yeah, I'm endlessly surprised by how, on the one hand, we seem to be at a historical moment in which there's a lot of doom and gloom and cynicism. But at the same time, we are endearingly naive about technology and about these companies. And one example is how, yeah, Google is well known for buying off companies that might become competition. and if the companies are not bought off of replicating that same technology in their own brand. And yet I meet up with startups all the time that are using Google Docs. And it doesn't take a genius and just read the privacy policy. Google has access to that data.

13:15Carissa Véliz:And so it seems to me very surprising that it wouldn't occur to them that it's a risk. We've become so used to using these products that we have stopped reflecting about what are the power threads behind them and how are they influencing my life individually, my company and my country. Okay. Now, you were talking about how data is absolutely critical, clearly, to predictive algorithms, but the data that we produce and the patterns that we derive from it can become self-fulfilling prophecies. So, you give the example, for example, companies founded by women receive less than 3 % of all investments, and only 10 % to 15 % of investors are women.

14:00So, how do we break out of that kind of pattern recognition cycle?

14:06Carissa Véliz:Algorithms are sold as being very disruptive and very cutting edge and new, but in fact, they're unbelievably conservative. And that's partly why they're so sexist and so racist. They're just rehashing the past because they're using historical data, projecting data of the past onto the future, assuming that the future resembles the past, but also creating the future such that it resembles the past. So the first thing is, if you want to do something different, If you actually want to be disruptive, then don't base your decision on past data because you're just going to perpetuate whatever pattern is there.

14:39Carissa Véliz:And it's a self-fulfilling prophecy and a kind of vicious cycle because there are examples, and I cite one investor which puts it bluntly. They think that the startup by a black woman is a fantastic idea, but they actually tell this woman, I'm not going to fund you because you're black and you're a woman and so other people won't fund you. And of course, that person is creating the exact reality that he's assuming to just describe. And so what do we do? We do things differently. If investors truly want to be contrarian, then invest precisely in the people who will never get selected by the algorithm.

15:18Carissa Véliz:And one of my favorite examples in history is Seinfeld, because obviously I'm a Seinfeld fan. I think comedy is genuinely important for democracy and for critical thinking. I don't take it lightly. ironically. But Seinfeld was a show that nobody wanted to watch, that the algorithm would have never selected because it was extremely unpopular. And part of its brilliance is that it changed the audience sensibility. It cultivated its own audience. And we need that kind of art, comedy, literature, entrepreneurship, that kind of innovation that changes the world for the better. It's interesting you say that because a lot of AI researchers talk about comedy or humor being the final frontier of AI.

16:01It's one of the most difficult things to predict, isn't it? You also have a wonderful quote from Karl Popper, which I'd love you to explain to me. You say, well, Popper said, for strictly logical reasons, it is impossible for us to predict the future course of history. As a philosopher, can you explain that to us non-philosophers?

16:23Carissa Véliz:The first thing to note is that Popper was a philosopher of science who lived through the Second World War and the Holocaust and everything that happened and reflected a lot on those matters. And one of the things he noted, as well as Hannah Arendt, was that totalitarian systems tend to pass as scientific what is not scientific. And they use that as a tool of power. And so that is what Popper is writing about. And that's why it's so important to learn that lesson. And he makes the point that totalitarian regimes tend to come up with grandiose predictions about the future of humanity and to call it science.

17:02Carissa Véliz:But for strictly scientific reasons, you cannot predict the course of history because the course of history is greatly influenced by the development of science and technology. And we can't possibly predict what we're going to learn from science in a year or 10 years or 20 years for the very simple reason that if we could predict it, we would already know it. And if we'd already know it, then it would already be the present, not the future. And so it's a great criticism of the kind of claim that passes as science when it's not. That reminds me a bit of Margaret Bowden, the kind of AI researcher, talked about different types of creativity.

17:42And a lot of them is just kind of incremental creativity, which machine learning and AI is often quite good at because it is just kind of recombining knowledge. But transformational creativity of the kind that you're talking about is the final frontier, as it were, for a lot of AI systems as well, isn't it? So is that where you think the limitations of AI are?

18:05Carissa Véliz:Yes, I think that's quite likely. And in some cases, it might be that AI can happen upon a transformational discovery, like, for example, maybe a new molecule that will supplant, I don't know, plastics. It could be. But at the end of the day, the AI cannot tell what is gold from what it's not. And at the end of the day, it's human beings who recognize it for what it is and who need to validate it and then make it into something else. One of the other dangers both of human prediction and machine prediction is the phenomenon of spurious correlation. And you have a great example of this, that the correlation between the chocolate consumption in a country and Nobel Prizes per capita.

18:54And Switzerland comes top of that list. I was hoping that you were going to say that eating more chocolate would help you win a Nobel Prize. but that is obviously a spurious correlation. But how do we guard against them?

19:09Carissa Véliz:Well, by knowing they exist and by having methods of inquiry that depend more on causal reasoning and logical reasoning and empirical findings than in just mere correlations. And that is perhaps one of the weakest links of AI that currently machine learning works only on correlation. And when we fix it to not only rely on correlation, it's because we plug it into something else, like a calculator. But we haven't figured out how to bake in causal reasoning into a machine learning device. And that has very serious implications for areas in which justice should be the paramount value. If you only care about profit, that's one thing, or efficiency.

19:53Carissa Véliz:But if you care about excellence, truth, or especially justice, that is a huge problem. Because if I deny you an opportunity, whatever it might be, it might be an apartment, it might be a job, it might be a loan, it might be bail, on the basis of a clear criteria that is based on a requirement that you either meet or don't meet, then if you meet the requirement, great. If you don't meet it, then you know what to do in order to meet it. Or if I say you don't meet it, but you do, we can have an argument and there can be somebody who arbitrates and says, yeah, this person does meet this requirement or doesn't.

20:28Carissa Véliz:But when I deny you an opportunity on the basis of a prediction, you cannot contest that. You cannot challenge it because a prediction is never a fact. And that means that we are creating essentially Kafkaist systems. And what is incredible is that we are using AI in the justice system. It's not like a future possible risk. something we're doing today. When I was reading your book, it struck me that, you know, as you say, predictive algorithms can be useful and good in some circumstances and very harmful, as you've just described, in others. What is the criteria that we can use to distinguish one from the other?

21:08Carissa Véliz:The interesting thing is that there's no checkbox, as it often is in ethics. Ethics is It's not a calculation. And so there are certain pointers to bear in mind, but we need more reflection on a case-by-case basis than we need public debate. But a couple of pointers is, one, are you making a prediction about people or are you making a prediction about things? Predictions about people are much trickier. Another thing to bear in mind is, is this a context in which the objective is fairness? If it is, then stay away from predictions, because predictions can be many things. At best, they can be an educated guess.

21:43Carissa Véliz:Often they are a power play in disguise, but whatever they are, they are never a fact. And that in itself is a philosophically important point that doesn't get made enough when it comes to AI. A third thing to bear in mind is, are you making a prediction at a population level? So are you calculating how many people in our country are going to get heart disease for the purposes of planning your medical resource allocation? Or are you making a prediction about this particular person? And the more individualized the prediction, the more dangerous it is because the easier it becomes for it to be a self-fulfilling prophecy.

22:24Carissa Véliz:You change people's expectations, they get treated differently, and you get a different outcome. I think you've previously written, particularly in the justice system where algorithms are applied to determine sentencing or bail and so on, that the algorithms should be testable and contestable. In other words, that as someone who is subject to the outcome of these algorithms, you should be able to test them to understand how the algorithms have been derived, and they should be contestable. You should be able to present kind of counter evidence to suggest that they're wrong. How can we enforce that as a principle?

23:07Carissa Véliz:So I have written about that and I think I've gone a different way in this book because I used to think that it might be enough to have randomized control trials like we have with with pharmacological drugs. And therefore, you would have a sense of how the algorithm is affecting reality. But I've come to think that that's not enough. That's enough in some areas of life. But in the justice system as the extreme case in which fairness should be the primary aim, I think that's not enough. Because even if you test it, that doesn't make it contestable. Because testing it in the sense of a randomized control trial might tell you how it's affecting people, but it's not clear criteria that we agree upon.

23:53Carissa Véliz:And one fundamental element of democracy is due process. And you do not have due process when you use a machine learning algorithm. One other philosophy that's very prevalent in the tech community, as you write about, is effective altruism. You have a pretty scathing critique of this. Can you rehearse that argument? Effective altruism is a movement that was born, I'm afraid to say, in Oxford. And that essentially they're the latest iteration of utilitarians, even though they claim they're not utilitarians. They come from utilitarianism, they share any of the tenets, and the criticism applied to both philosophies.

24:35Carissa Véliz:And it's important to point out the connection because utilitarians have been here for a while and they have been incredibly effective in influencing policymaking in ways that have been helpful and in ways that have been, I think, very unhelpful and very toxic. And the first tenet of effective altruism is very intuitive and it's hard to disagree with. You should, if you can, be altruistic and you should be effectively so. So if you have two choices and one is more effective than another, in the sense that it takes less resources and has a bigger effect, then you should go for that one. And that seems great.