Sir David Spiegelhalter: Statistics, Communicating in Crises, and Living with Uncertainty

0:00Hi, everybody. I'm Nikolaj Tangen, the CEO of the Norwegian Southern Wealth Fund. And today I'm in really good company with Sir David Spiegelhalter. He is, well, I would say he's the world's best statistician and for sure the best communicator of risk that I've ever seen. Written lots of fantastic books and is particularly known for making statistics, which is difficult, really accessible for most people. So big thanks for joining us. Great, great pleasure to be here.

0:40having worked with risks what what's the main thing that you have learned about human psychology well first of all that i'm not a psychologist so you know i am a statistician but i i have done my best to learn from psychologists that i have worked with and i suppose just from observing how people react to risk and uncertainty. I tend to think of the broader idea of uncertainty rather than just risk, everything to do with not knowing about what might happen in the future, or even not knowing what's going on at the moment or what's happened in the past. All these things we are uncertain about. Some of them, usually things do have an upside and a downside, and so they could be considered risks depending on how they occur.

1:23But I think what I've learned is that people have to live with uncertainty. When you ask people, they say, oh, I don't like uncertainty. But some people like it, right? Well, some people like it, and some people are a bit more bold than others. But the point is that when you actually then go a bit further and say, well, do you want to know what you're going to get for Christmas? Do you want to know how a match is going to end, you know, if you've recorded it or something? Do you want to know, you know, do you just jump to the end of your series on TV to see the last episode and see what happens?

1:54Also, the thing I ask is, do you want to know when you're going to die, if I could tell you? No, they all say no. Some would like to know when they're going to die. Some are so, in a way, uncertainty-averse that they really like to have everything planned. But most people have realized that you've got to live with uncertainty and think of a life without uncertainty, without some risk. What are the people who hate risks the most? Oh, there are some people, I think, who are very cautious, who would like to have everything planned out, who want to feel that they can control all contingencies and have mapped out the possibilities.

2:30And of course, this is impossible. And this is, you know, when I talk to audiences, when I say the first thing is not only that we have to face uncertainty, but we have to face actually deeper uncertainty that we can't even list the possibilities of what might happen to us in the future. We have to deal with that cloud of unknowing, and that's part of human life. And I'm interested in strategies that people use personally to deal with that. How is it linked with the Big Five? So my impression is that introverts like risks less, Americans like it more, old people perhaps less than young people.

3:11Just how does that happen? Yeah, I haven't actually looked at that. I mean, people will have looked at that because people have studied risk and proneness and risk aversion. What they found is that there's not a single characteristic. I think this is why it's not part of the big five. A single risk characteristic in people's personalities. Some people, I've known people who are incredibly sort of what I would think consider reckless physically with what they did with their bodies. They were extremely cautious with money. Other people may be very bold socially and take all sorts of risks in changing jobs, in changing friends, going into new environments, but again, may be very cautious about their physical health.

3:52And so there is not a single risk scale where you can put everybody on. It's much more multidimensional than that. Why do people fear the unknown so much? oh i mean i suppose if i was an evolutionary biologist i mean i mean i mean the we we fear the unknown risks more than oh yes more than the known right i mean it's like covid covid was so scary because we hadn't seen it before exactly exactly and that's been known you know for ages the unknown unknown risk ellsberg paradoxes that people invest you know looking in the 1950s that if you couldn't actually say how big the risk was people were much more averse to being exposed to it.

4:31And that's been known for ages, the well-known idea of risk aversion to a well-known, to uncertainty about the risk has been 70 years ago. I think Daniel Ellsberg did his original study of that. And I think quite reasonably, because if you don't know what the possibilities are and roughly how likely they are, then all sorts of other, perhaps rather deeper attitudes to caution and precaution come in, and people might start being really hedging themselves against, and quite reasonably, against major losses. And we can see that going on, you know, in all parts of our life. What kind of habits can help us interpret risk in a more rational way?

5:16Well, first of all, I don't like the word rational. So in my book, I hardly use the word rational at all, because I think this claim that's been, oh, yeah, we can look at risks rationally and deconstruct them. Actually, in real life, I think that's pretty nonsensical. And I've taught this stuff for decades. I've taught decision theory. I know how you should be doing it. And I know you can't do it in practice because for the theory to work, you have to be able to list all the possibilities. You have to list all the options. You have to look at the probabilities of the possible outcomes and their value to you.

5:49And then according to economic principles, you should maximize your expected return and things like that well it just doesn't work it would be it fails at the first step that you can't even list everything that's going going to happen let alone except in really simple circumstances and put numbers on everything so I think nobody can be rational it just doesn't exist I think that's why I really don't like thinking and this is some objective no I think one can try to be in a sense reasonable in a much broader sense of, you know, I suppose how I would say is think is first of all, you have to use as much imagination as you can.

6:30Things might happen that you didn't think of, but really, you're really opening yourself up to problems if you haven't at least made a big effort to envisage the possible futures to consider quite extreme scenarios. And that requires diversity of inputs. I'm hopeless at this. I have no imagination at all. Absolutely disastrous. But I know that if I were, God forbid, making important decisions for society, I'd want advisors with a real range of different inputs. The example I like is Barack Obama, when he was faced with the decision about whether he was sending the seals when it was suspected that Osama bin Laden was in Compound in Abbottabad.

7:12And he had a very diverse team of advisors who didn't speak to each other. And some were, you know, they may have been set up as kind of red teams. They were really pessimistic. 30 to 40 % chance he was going to be there. Others were real gung-ho. 80 to 90 % chance he's there. And he had to put all that information together. But I think that's what someone who actually has to take the rap, the real decision maker, should be open to, a diversity of opinion when there is no clear, correct answer. So I think the first thing is we have to acknowledge there's no correct way of doing this. We have to have a diversity of opinion.

7:46And we have to work in a combination of an analytic approach, which I love, you know, based in maths, I've done maths and stats. I love trying to deconstruct uncertainty, looking at the sources, analysing data, building statistical models. And it's great, but it's never enough. It never tells you what to do. But increasingly numbers are emotional, they're weaponized in debates and so on. Just how do you counteract that? Oh, I think it's one of the numbers. It's a complete myth that they're cold, hard facts. Even before they're weaponized, we know that someone has made the decision to collect that particular data.

8:25There's lots of judgments that's gone into every analysis, every sort of management. There's always judgment behind every statistical analysis. And it's good if that's made very explicit. And so other judgments can be added to it. People may disagree about the fundamental tenets, the fundamental assumptions that are always underlying any statistical analysis. So it's a mixture. It's not thinking fast. It's not thinking slow. It has to be a combination of the two.

8:55Now, you started out in medical statistics. Just how did that come about? Oh, very naturally, just because there's a good job going. And curiously, the job I first, you know, a real job after university. I taught in America for a year and then came back. And it was 1978. And it was working on artificial intelligence in medicine, late 1970s. It was a booming idea. It was then largely called computer-aided diagnosis and computer-aided prognosis. and it was building statistical models to enable diagnosis, but it included computer interviewing of people with stomach complaints and things like that.

9:34It was really advanced. The tech was terrible, but the ideas were absolutely modern and the problems, the issues about how to integrate this in with medical practice were in there. So I was working on uncertainty in AI for much of the 1980s and we thought we'd solved it. How wrong could we be? But because it's still a massive topic, obviously, the machine learning techniques that people use now have developed beyond all imagination. And they really are incredible. However, they are really struggling with uncertainty still. What were some of the strangest stats that you've seen in the medical sector?

10:13Oh, I don't know. Some of the mind-boggling situations. The mind-boggling ones, yeah. Yeah, I suppose it's the ones I've been involved in actually four major public inquiries into health scandals in the UK. That's kind of where I developed quite, you know, a slightly higher public profile, I think. So ones where, you know, over 30 babies died with heart surgery at a centre, more than you would expect to have died. And then, of course, the second one was Harold Shipman, the mass murderer, who murdered at least 250 and possibly 400 of his patients over a 20-year period. And we were brought in. He had been caught by then, but we were brought in to say, could he have been detected earlier or not?

10:57Could he? Yes. Yeah. And we conclude he could have been detected after a few years if somebody had been looking at the data because he had so many excess deaths, but nobody was looking at the data. Are people now looking at data properly across hospitals? It's got better, but it's still slow. I'm on a group in the NHS that's only now building a really rigorous statistical monitoring system for adverse events in maternity units. And there's been endless maternity scandals in the UK. And finally, we've got a system based on – it's essentially a statistical process control system that we applied to shipment.

11:37And then people took what we'd done, which was adapting industrial quality control to medical outcomes, and then applied it to, for example, children's intensive care in the UK has got a monitoring system based almost precisely on the work we did for shipment for early detection of problems. Now, you are a leading public communicator of statistics. Why is it important that people have a grasp of this field? Whoa. What kind of wrong decisions are people – what are the big wrong decisions people are making? So we only have to look at some of the – without mentioning any names. Well, do a few. Do a few.

12:21Okay. We only have to look at what's happening in America at the moment to see what happens when high-level public discourse is not based on evidence. It's not based on numbers. It's just based on people saying what they feel like saying and regardless of the evidence behind it. Give some examples. What are the most horrifying ones in your mind? Oh, well, I think what RFK is saying about vaccines at the moment because he's got a built-in buzz. I mean, vaccines are not perfect. They're not perfectly safe and they're not perfectly effective. So I'd be the first one to say the term vaccines are safe and effective is actually misleading.

12:58However, they are of enormous value. And he's got his particular, I think, biases there. And of course, I'm not going to talk about Trump and his way he was setting tariffs and things like that. So it seemed to be how he originally did that seemed to be based on what a 20-year-old intern might do on a spreadsheet. And so it just upsets me when I see the enormous – oh, apart from, of course, sacking the head of the Bureau of Labor Statistics when he doesn't like the numbers. So all these things are deeply upsetting to a nerdy statistician who – I don't want to tell anyone what to do. I don't want to tell anyone what the right policy is.

13:41All I want to do is say, please respect the evidence that we've got. just respect it and it doesn't tell you what to do or whatever but just just try to respect it and and it's not just because politicians it's social media it's conspiracy theories everywhere the um lack of concern or the almost i think deliberate lack of understanding of what good evidence is and how it should be used is deeply upsetting to a to a nerd why uh why is it happening Oh, God. This is beyond my – look, I'm a statistician. I'm not a great big sociologist. Okay, but give us some more kind of examples of where you think the world is going totally bananas and moving away from facts.

14:26Yeah, yeah. I think, obviously, there's enormous blame on social media, on the algorithms, on the recommendation algorithms that mean that something – oh, wow, that looks impressive. And it's almost certainly wrong. And so often they are based on numbers. You know, people love numbers and they kind of think, as we said before, they kind of think they're cold, hard facts. But no. And often the numbers are actually not completely wrong. It's just that they're grossly misinterpreted and exaggerated and one sided, cherry picked. And, you know, a wrong number that's blown up and people making some bold claim is around the world.

15:04You know, it's just triggered by the algorithms everywhere. way. And it looks good. It looks impressive. And trying to backtrack on that is really, really, is really difficult. And that's why one of the things, you know, I'm on the board for the UK Statistics Authority. And one of the things I try to hammer all the time in the communication of official statistics, boring old official statistics, is to try to preempt the misunderstandings that people will make. Because once everything's out there, it's really difficult to counter the misinformation, the misclaims, which might be made accidentally or deliberately.

15:39You can't stop people saying everything. You can't stop misclaims. But if you can preempt them, if you can understand by knowing your audiences, by listening to people's concerns, even the people you don't like, to know what might be said, you can get in there and actually say, this data means, I think we can interpret it to mean at least this, But it does not mean this. And that, I think, is becoming a more common trend in the communication of official statistics to say what things don't mean. What was the most important thing we learned from COVID? Oh, the importance of data. The importance of data and its clear communication.

16:21Which data in particular? Oh, everything. Good. We wrote a whole book with every chapter on a different data source. There was so much. I mean, I was working around the clock, really, analyzing data and communicating about it because I didn't have an official role, which was great, which meant I could get out there with the media and try to explain things. And again, I never said what should be done. It was only trying to explain the numbers. And you had everything. You had the vaccines. You had the rollout. You had the testing. You had, of course, the disease, the infections. But what a fantastic time for a statistician.

16:51I mean, it must be paradise for you. Well, paradise isn't quite the right word. But it was very exciting, very challenging, and incredibly rewarding. And important. I mean, statisticians have hardly been that important before. Exactly. And I keep on – even now, I get people coming up and saying, well, thank you so much for the work you're doing during COVID. And the media had to learn. It wasn't just me, other statisticians who were out there talking about the numbers. And yet we always, at the beginning at least, asked, well, who's to blame and what's going to happen or what should be done? And we'd have to say, I'm not going to say, no, no, that's not my job.

17:27You'll have to ask somebody else. All we're doing is explaining the numbers. And after a while, the media learned that this is what the audience actually wanted. They wanted an unbiased, unagendered discussion of the numbers. And they loved it. And so, you know, I was really popular, to be honest. Well, in your book, you talk about the five five-con rules. You know, you tell people what you know. You tell people what you don't know. Tell us about these five things. Oh, yeah. Well, this is for communicating evidence in a crisis. This is all derived from John Krebs when he was head of the Food Standards Agency in the UK.

18:00When he faced crisis after crisis, he had foot and mouth, he had mad cow disease, he had everything, one after the other, total disasters. And he developed a playbook that afterwards he wrote and said, this is what I did when I was talking. Five points. And everyone should have these written, tattooed on them, I think. first what you know so you know be really clear about what we know and then you say what you don't know you say where the areas of uncertainty are you admit them straight away second not first but what is second and then you say um you say what we're doing about it you know we are learning more we're doing experiments we're finding out we're collecting data we are learning we are learning then you tell people what they can do in the meantime that you may want to be cautious you may not want to eat beef you may not want to do everything except you so you give people advice advice, self-efficacy in the meantime.

18:48But the final one is the most important. You say, we will come back to you and our advice will change as we learn more. So you emphasize the provisionality of what you're saying. Now, this is both deeply trustworthy, because it's true. It's also, as far as I can see, absolutely impossible for politicians to do. It's just not in their vocabulary at all, this idea of provisionality. Why is it so difficult? Well, they think they have to be absolutely confident about it. They say, oh, if we're not certain about everything, nobody will believe us. They'll just listen to somebody else. We have to be absolutely certain about everything.

19:25And I think they believe it. And our research with psychologists, and not just our work, but other research, we've done randomized trials for different ways of messaging, strongly suggests, this is a complete myth, that if you actually do, you're in a position of authority, you do actually admit some uncertainty, that there are pros and cons, et cetera, et cetera, that you are trusted more. And what's more important, you're trusted more by the people who were initially skeptical, by the very people you're trying to reach trust you more, because you're finally listening to their concerns. You're finally acknowledging that there are issues out there that perhaps vaccines aren't completely safe and effective.

20:05So what that means is the common political way of communicating, which is, I think, put into practice by communication departments in government, which hammer through the message, bam, bam, bam, are actively decreasing trust in the group they're trying to reach, those who are most skeptical. People who believe them already, there's no point. So they're making it worse by their attitude. And since we did these trials and actually saw data on thousands of people showing that trust was improved in the most skeptical way, if you gave a balanced, trustworthy message, including uncertainty, It totally changed my mind.

20:38It absolutely convinced me about this. I also believe it's correct on an ethical point of view, because it is correct. But it's also, purely from a practical point of view, it should be more effective. No, I totally agree with you. I've done a podcast with Rachel Botsman, who is a specialist on trust. And indeed, this is very, very important. But it's a bit, in the public sector generally, they never apologize either, right? It's kind of tied into the same thing, I think. But what did we not learn from COVID? Oh, a lot of it was not being flexible enough, getting tied into. We were told to wash our hands and wipe surfaces and things like that.

21:17And within about a month, we knew this was pretty useless scientifically. And yet nobody ever said, oh, well, actually, this is not the point of point. It's fresh air that's more important than ventilation. Nobody said that for a year. And so I think what we didn't learn was that you need to be agile and flexible and take people with you by acknowledging the uncertainties and that you change course. So the guy we worked with who was most impressive, I thought, was Jonathan Van Tam, the Deputy Chief Medical Officer. And we worked together when the UK did admit that the AstraZeneca vaccine was causing these very nasty, which was actually, I think, pretty well first detected in Norway, causing these nasty blood clots, particularly in young people.

22:05and we worked on the communication of that and where we showed that the the benefits of the vaccine went down massively when you got younger but the risks went up and so there comes a point you just shouldn't give the vaccine stratified by age and he then said to the public he explained all this used our graphics went through the numbers taking treating the audience with respect admitting the evidence had changed and then said we're changing policy and everyone said well fine and he said oh we're adjusting our course it's not a u-turn it's adjusting our course and um and there was no pushback from the media there's no accusations people really accepted it because he he actually showed the evidence to the public as he was explaining it to them and he could understand it a politician would have been hopeless at it because he wouldn't have understood what's going on he wouldn't be able to explain it and so that to me showed that um you know if you can get good scientists actually doing the communication and they're good and they're reliable and trustworthy, this can have an enormous impact on the public and on public trust in authority.

23:18The world is totally overflowing with data. How do you distinguish the signal from the noise, so to say. Sorry, that's my entire career. You're asking me to explain what being a statistician means. That's the statistician's job, trying to split the signal from the noise. And of course, you can't ever do it. And what's a signal, what's noise is never absolutely a black and white thing at all. But it's by trying to understand, and this is a standard statistical thing that would have been said for the last century, the sources of variation. Just like in pre-war, all the statistics developed in Rothamsted breeding stations for plants.

24:04And it was understanding the sources of variation, what led to the variation between the crop yields, what factors led to it. And of course, there is, in a way, unavoidable variation, which tends to be called noise or random error. And then there's, in a sense, predictable variation, which is due to factors that you might be able to control and that's what statistics has worked on for about the last hundred years and um and it's it's not been bad it's got some pretty good techniques for you know largely regression methods and so on um when and so it's actually done quite well but you notice that that is different really from a strict machine learning black box approach which just throws the data in and tries to extract a prediction, a classification, possibly with some uncertainty or a prediction of what's going to happen, where there's no real understanding there of where it came from.

24:58Again, people will have real difficulty for explaining why a piece of AI came up with its conclusion. Again, people are really working on that now, just like they are with uncertainty. But if you have rather slightly more basic statistical methods, that you do, which that's why I support them, you can then generate a much clearer explanation of why you came to that conclusion and what are the important factors. In which area are you the most impressed by the improved predictions? Oh, well, I mean, AI is in terms of what you might call rather tightly controlled areas. It's done brilliantly. I mean, the people at Google DeepMind who started on games like chess and Go and things like that, and then moved into, well, obviously protein folding and then also sort of medical diagnostic from images in terms of breast cancer and eye problems.

25:55They worked with the Moorfields Hospital. It's tremendously impressive in that way, in that they can take a really quite a big area. But these are all tightly constrained problems. You know, there's a block of data. You've got an image, you've got a set of data, and then you produce an outcome. And it's just brilliant at that. Where I'm much more skeptical is about, I think, very unfounded claims that, oh, well, we can just put your medical record into AI and it'll tell you X, Y, and Z. So people have made the jump from these kind of quite tightly constrained problems, which are just brilliant, into much more general problems.

26:36And it's hardly surprising they do that when we look at large language models, how effective they are at coming back with what is quite often a reasonable response to very generic issues because they've been able to mine vast amounts of stuff on the web. Now, how good that would be about – and, of course, it can come up with a list of medical diagnoses and things like that. Yeah, it's just – it's fine. It can make suggestions and it can be extremely effective on that. But if you take another way, I mean, you've been around for a while. I mean, just during that period, the accuracy of weather forecasts, for instance, is just mind-bogglingly different, right?

27:12Oh, weather forecasts are fascinating what's going on because there's a real competition going on. I think a fairly friendly competition about the totally different philosophies for weather forecasting. Because traditionally, it's been kind of applied mathematicians and physicists, you know, building huge weather models based on maybe Stokes equations and third order differential equations. and they build massive models of the atmosphere and then make a prediction, you know, six, 10, a week, 10 days ahead. But they don't use any data apart from the initial conditions, really. And then they vary the initial conditions, and that produces an ensemble model, and they produce a probability of what's going to happen.

27:51The alternative approach, which people like Dean Mind have taken, is to throw out all our knowledge of physics, the atmosphere, everything that's been learned in the last 300 years, and just throw it all out and take a massive amount of data and to do a pattern recognition, essentially, and make a prediction, which has no ability to explain why it's come up with this conclusion, particularly a pure black box, and they're doing really well. Which one is going to win eventually? Well, I'm really pleased that the UK Met Office has got both teams working and collaborating because I think it's going to be complementary in the end.

28:26But I'm kind of, because my background is statistics rather than applied maths, I'm kind of secretly on the side of the black box machine learning people. Because I just love the thought of just putting the data in and out it comes with a prediction. Without any ability to say why it's saying, it's just saying, well, in the past, when this pattern was there, this is what happened. Well, actually, maybe that's as good as you can do. So I think it's a fascinating competition going on at the moment, which I'm watching with glee. would you have liked to be a weather forecaster? Because I kind of think if I had to study something, meteorology would have been, I mean, what a barrier.

29:09Yeah, but what I'm interested in is the, I don't care about the meteorology. That's why I quite like the data-based approach. What I'm interested in is the first main nature paper from DeepMind on this didn't have uncertainties in it. And I think it's really important to have uncertainties. When I, you know, all the time I look at probabilities of rain and things like that, I use those uncertainties. If I don't have them, I really would feel a bit lost. And so what I'm interested in… So when you look at the forecast, you don't look at, is it going to rain or be sun? You look at what's the probability of rain?

29:41Yeah, I do. And what probability do you need to bring your umbrella? Well, exactly. I don't know. It depends. That's very personal about how it caught. It depends what I want to do, if I want to have a picnic or not. So now I need the probabilities. And so this goes back to the 1950s with Glenn Breyer developing a scoring rule for probabilistic precipitation forecasts. And it's terribly exciting, I think. And the scale of these probabilistic forecasts is growing all the time, and it will continue to grow. Do you buy a lot of tickets? I bought one a few years ago when the expected return was higher than the ticket price.

30:19There'd been so many rollovers. It's still almost completely impossible to win, but I thought I've got to have a go at a gamble where the expected return is higher than the stake. In which cases do you look at probabilities where other people would look at yes or no? Oh, I got prostate cancer. And so I'm really interested in forecasting effect of people with cancer. We've been involved in algorithms for breast cancer and prostate cancer and building software to demonstrate those to people. And they're all in terms of probabilities, you know, roughish, but not bad in terms of 10 years survival and how those will change depending on the different treatments you've got.

31:00And I think it's absolutely essential when somebody says, oh, my doctor told me I had six months to live. I think, what? I never believe they said that anymore. Maybe there's some doctors who'd be so stupid as to say that. But I can't believe that. You have to show we don't know how long anyone's going to live. We can put some broad bracket on it because it's a survival curve. But first of all, I want to say I'm really sorry to hear that you have prostate cancer. But given that you are a mathematician, what are your stats? Oh, mine. Oh, yeah. That's the problem. I've got locally advanced prostate cancer.

31:36So I've got some minor metastases, oligometastatic it's called. I've got a few of them. And so I know, but the drugs now, the new drugs, the new hormone drugs on abiraterone, and they just are so effective. Now, I've got my PSA is essentially non-measurable. But the problem is we're trying to get the use of data from survival in clinical trials. I can't, it's really depressing. When I go back on the trials for abiraterone, they're terrible. You get sort of median survival of 18 months or something. And I think, what? No, I'm going to live longer than that, I'm sure. Because, you know, in the trials, they were trying this on very sick people.

32:18And things have improved so much. And actually, I'm at an earlier stage than the people who got the trials. So there's no good data, really, on what are my survival prospects. It's very difficult. It's a shame. And I wish there were, you know, better, you know, database. I really wish I could just tap in and find out, well, out of 100 people who are most similar to me, what happened to them. But for a start, we don't know. We don't know. It's only been given to people like me for a few years. So we've got no long-term follow-up. We don't know. I mean, some people got it earlier. My oncologist said, oh, I've had someone on this for 16 years.

33:01So that's the problem with something that's fairly rapidly changing. When you're asking for a long-term prediction, you can't say. so uh well all i can say all i can say is uh fingers crossed yeah yeah i mean it does seem a bit you know a bit pathetic for a statistician just to say well i you know i hope i'm lucky but i hope i'm lucky why are we so bad at predicting elections oh elections oh well for a start because when you go and ask somebody they let all the election things are just ask people what would you vote if you had to vote now? I mean, that's the question. So you're not even asking them what they're going to vote in the election.

33:42You're asking them, if you had to vote today, what would you vote? And that's a very biased measure of what you're trying to estimate of what that person will vote in three weeks' time, two weeks' time, one week's time, three months' time. People change, people are not necessarily honest. And so people may vote, they may not vote at all. So I think that the basic data source is always going to be biased. And so it's not like predicting weather. Weather, it changes, but in a sense, it's not changing because of what people feel. It's not like changing. So I think they're never going to be great because they're trying to estimate something that you cannot predict.

34:30You can't observe. No. You've been bringing statistics into some new areas. So, for instance, anti-doping, the World Anti-Doping Agency. What kind of things were you doing there? Oh, then, I'm not sure what's happened. Then they had the idea of an athletics passport because they wanted to – actually, a passport which kept a complete record of their drug testing history. And it was to try to allow to a certain extent for the fact that, you know, there is individual variation between how people do respond to drugs. And so when you take a measurement from somebody, they were particularly interested in people getting blood transfusions just before an athletic event.

35:16And so you'd be wanting to look at someone's red cell level or something like that. It was remarkably high. Well, people vary anyway. So to know whether it's been pumped up high, you have to know something about their past history. So essentially, you have to have a model for someone's variability and their natural baseline level for their hemoglobin level. And so it was quite complex. No, it was fascinating statistically. And I haven't – actually, I would be interested to know whether it's – what the current situation is about that. But it was, in a sense, trying to make the drug testing regime a bit fairer in order so it could adapt to the individual biology of the athletes.

36:01From drug testing to the financial industry, is there anything in the financial industry that puzzles you? No, I've kept well away from that. I mean, for a long time. And why are you keeping well away from that? Oh, I'm not interested. For a long time, of course, I was funded from the charitable arm of Winton Capital Management, a hedge fund, David Harding. And he was great. He gave us money and let us get on with what we wanted to. And I think – I hope he felt it was a good investment. But I did nothing to do with the hedge fund business. And I just never had any interest in money. I hope it just puts me off.

36:34I just got an aversion to it. But why didn't Winton, who's established a very successful firm, why didn't he ask you, hey, David, come and check out our things and make sure we make some money? No, no, no. That was not part of the deal at all. No, he had plenty of really good people. He's a mathematician, right? Yeah, yeah, yeah. So he had the whole place stuffed full of mathematicians and really competent people, much more competent mathematicians than me for a start. And so for a start, I couldn't have contributed anyway. And it would have been really inappropriate, I think, for that to happen.

37:09And I just wasn't interested. I couldn't care less. Do you think people are worried enough about the really, really bad outcomes? So, for instance, bioterrorism, the comeback of smallpox, that kind of thing. Nuclear war, climate change. Well, nuclear war is relatively less dangerous than bioterrorism. Yeah, well, it could be on a larger scale. But yeah, it depends on the scale. But nuclear war would not be great. And so it's tricky, I think, because in the end, we're all going to die within a very finite period. From a personal point of view, I would understand, because I think I do that, that people do not want to spend their time obsessing against about all the terrible things that could happen in the world.

38:14It seems to me that this is not beneficial to your mental health, shall we say, to be really obsessed. I kind of hope there are people studying it more professionally and who are trying to counter it with appropriate regulation, appropriate policing, and so on. But I personally do not want to spend my time waking up in the morning worried about smallpox and bioterrorism. And if people are – so in other words, I think I can understand, because I don't do it, why this is not full of top of the agenda in people's concerns. Catastrophic existential risks. For a start, I – but I'm hopelessly optimistic.

38:55I think, actually, they tend to be overrated. and so but that's just maybe because i my particular personality is far too optimistic was norway lucky to find the oil oh that's interesting i don't know nothing i think norway i don't know about how finding it but it was extremely sensible and how it dealt with it once they found it compared with the uk um who so and uh you know which is why you why you've got your job at the moment to some extent and so um i think that norway dealt with this in the absolutely brilliant way of seeing it as a as a national resource to be in a sense of heart you know um you know to be nurtured for the whole entire community rather than just in the uk to sell off the rights uh in order to raise it raise some money at the time in short term How do you look at climate risk?

39:54It's difficult. So you have people talking about it like some places in America. You sit there in the middle of ash, rain, forest fires, and you're kind of in the middle of it. You're sitting in the middle of it and you're saying, there is no problem with the climate. I know. It's just happening and that's it. You could just tell by the events, which are just going to carry on. There will be ups and downs, but there are going to be more extreme events. I'm interested in attributes. What is brought to the fore is attribution studies, which is not so much about climate risk, but about looking backwards and say, to what extent was this caused by man-made climate change, which is a really big growing area in research.

40:38And it's going to be a big growing area financially when people start suing fossil fuel companies for events that happen. and so the uh you know but it all requires models you have to have a model of uh what we would you know how the climate has developed and with man-made uh you know man-made interventions and how we think it would have developed had we not been throwing all this muck into the air um since the 1700s and uh and you see see how likely these events were under these two different scenarios and the relative risk can the our meteorological office do have got an attribution center and they just they will give you a relative risk technically that can be converted into a probability of causation to say the probability that this hot weather event that this tornado was caused by man-made climate change was x percent and um officially uh once that gets above 50 % by the balance of probabilities on a civil court case, you could say that man-made climate change was responsible.

41:44Now, trying to then attribute it to particular companies, I think, is rather more difficult. But it's brought, from a technical point of view, it's brought this fascinating idea of attribution. And of course, for that, you have to have climate models, and the climate models are used to make projections as to what's going to happen. A lot of uncertainty, and the models take a lot of time of dealing with that. They also have what I think is really good, independent teams coming up with different climate models, which then they pool, and then they make it even more uncertain. They broaden the things out.

42:16And, you know, we're never going to know what's going to happen. And one should not state too confidently about what's going to happen. But we know bad things are going to happen. and what's to do about it again not my job not my job i'm afraid talking about big things uh in 2021 you said that uh ai poses an extreme risk but that it perhaps is overrated what do you think now oh i think it poses an extreme risk and i think it's probably overrated what uh could you explain yeah i mean it's amazing how strong it is and it turns what you mean about extreme risk it's certainly going to take some jobs and certainly we're all going to have to adapt our jobs our work to it it's also incredibly useful and valuable i use of course use it every day and um that i but in terms of sort of risks well people when they talk about this they're talking about sort of existential risks, about the self-aware AI that's going to start having essentially a will of its own and deciding that these people get it right, get it in the way of what it wants to optimize.

43:31And I think, of course, that is a possible scenario. And I think it's quite reasonable then that people respond by wanting additional overview of and guardrails on AI. So it's like a lot of these things people say, it's a bit like COVID. Before COVID, the modelers in the UK said, oh, there could be half a million deaths. And that still gets quoted. Oh, they said there were going to be half a million deaths. No, they said there would be half a million deaths if nobody did anything about it. If we all just sat there and let it wash over us, there would be half a million deaths than there would have been.

44:09But they're now being accused of saying, oh, they said there would be half a million deaths than there weren't. So the point is that one could talk about there being risked, but I think it's limited the value of that if we're all going to assume we're just going to sit here and allow ourselves to be taken over by killer robots. So I think that what it does do is, of course, call for far greater scrutiny of what's being done, in openness about the guardrails put in, and so on. I mean... But are you seeing the necessary guardrails and so on being in place? I don't know enough about it. Again, it's not really my area.

44:50But when you look at AI, what are the type of things you look at in order to gauge the risk? Oh, again, I don't know enough about these sort of existential risks and how those could occur, the super intelligence and things like that. I really don't know enough about it, and I wouldn't want to claim to. And how do you use it? You said you use it all the time. Oh, I use it myself. What kind of things do you use it for? Oh, I use it in writing my book. I use it for researching, and I use it for coding, and I use it for personal things like trying to work out where I'm going to go on holiday. So I'll use it all up for all sorts of stuff.

45:20Where does it tell you to go on holiday? Oh, well, I don't just say, where should I go on holiday? It's not quite that broad. But no, I use it all the time. And of course, we're almost forced to use it now because it comes up number one thing when we do a Google search. So I think it's incredibly valuable and it's unbelievable what it can do. But the guardrails, of course, are already in there in many ways in terms of violent speech and racism and all sorts of stuff that it can't do that are built in. and the crucial thing I feel is that these should of course should be open and public and and they should be they should be a regulator to make sure they're being adhered to so but lots of people are saying this I mean there's a massive AI safety is such a massive thing because all the tech people are going on about it as well so so I'll leave them to it and hope that there's some some decent people involved in it but it is a crucial area not my job

46:22Now, if you do look at your job and kind of your legacy, what is the one kind of idea or principle that you hope you will be remembered for? Oh, I don't know. I've moved around rather a lot. Yeah. Oh, one thing. Oh, I think it's the stuff I've been doing later. I think it's on trustworthy communication of evidence. In a way, I've done quite a lot of technical stuff. That's where I get all my citations from, et cetera. And I get loads of those, which is lovely. But in the end, what I'm just obsessed by is the need for trustworthy communication about evidence. As we said, not that it tells you what to do, but unless, I mean, we're sort of doomed if we don't use evidence in an appropriate way and don't communicate it properly.

47:12We are just left up to thinking fast in Danny Kahneman's term. We're just left up to gut reactions and emotional feelings in order to for everything. And I think, wow, that's just disastrous. And where is the next step of this trustworthy communication? Where is that kind of part of the science going? Oh, yeah. Well, people are concerned about it. They're obviously concerned about the quality of what's published in the scientific literature, which is a massive problem because there's so much junk out there, unrupped from paper mills and so on. I think that's all I can talk about in the UK, where actually this is now, you've got a very high level.

47:54There's a new, there's a code of practice of statistics in the UK, which is a pretty dull document, but it's incredibly important. There's a new version coming out, which is really putting down, you know, the way in which all official statistics and even non-official statistics should be communicated to the public and based on these ideas of preempting misunderstandings, of not being misleading, of being open about limitations and so on. And that's all down there. And government departments have to adhere to it. They are actually bound by it. And so it's, I think, setting quite a good... Because in the UK, we've got an Office for Statistics Regulation, which I think might be fairly unique around the world, an actual body that is there as an inspector, as a regulator for statistics.

48:42And I'm a huge believer in that, obviously. And I would love to see that model developed elsewhere, that you have got a body who can really tell people off when statistics are being misused. My God, they'd have the work cut out in the US at the moment, wouldn't they? Oh, that's for sure. What's the big unanswered question that you still want to tackle? oh well i'd like quite like to know understand consciousness and whether there is such a thing as free will but that's again going somewhat outside my um my my professional expertise shall we say um but it is in my book and i do discuss it because i think it comes quite important when you start talking about whether to what extent is the world genuinely stochastic at random or whether to what extent is it actually deterministic but staggeringly complex in a way that renders it unpredictable, I think is an interesting issue.

49:40Oh, oh, interesting question. I suppose, you know, broadly it comes down a bit to what we're discussing before about whether... COVID was actually quite encouraging because in a crisis situation, good communication did rise to the surface. There were obviously people on each side arguing. I was right in the middle getting attacked by everybody. So I thought, yeah, doing the right thing. And it wasn't just me, but there was an enhanced respect for the mainstream media. And largely people were interested. And there was a whole body. Everybody was discussing the data. It was a very active community.

50:21And I think it was, as we discussed, an exciting, positive time. Now, once a crisis has gone, everyone goes back to their normal stuff and loses interest in all of this. And I suppose my question I'm interested in is whether that kind of interest and attention and wanting trustworthy information can be retained in societies that are becoming increasingly popular, you know, follow populist politicians. who object to authority, who are distrustful of what they call elites and experts and so on. And as that happens, well, first of all, this was big things. Can that be counted by having trustworthy experts, people who do know something?

51:11I think it's so interesting. According to Bill Gates, it would take roughly a billion dollars a year to make sure the world is really ready for the next pandemic and the world is not spending them on it yeah and i the next pandemic will be different um i i know and these people will certainly wouldn't respond i don't think in the next pandemic people would accept lockdowns i just don't think they'll be politically acceptable so i think uh so we all so so we'll all be swedes in the next pandemic i think well the swedes as they as they said the guy the minister said well We practice social distancing anyway in Sweden.

51:50So I thought that's great. So, yeah, I think we would be. I think we would be more Swedes in the future. But the crucial thing about that is what it says is that, you know, you said you can spend a billion dollars, but you can't change people. What are you going to do about people? I mean, how people react is absolutely crucial. And you don't necessarily change the way they react by just spending money on things. So I think that, particularly in pandemics, the role of human reaction to the situation is, which is the most important and the least predictable aspect. And actually, it's quite difficult to research, very difficult, I think, to learn from the past pandemic about exactly what worked and what didn't, because things were so different across every bit of society, within and between societies.

52:41so I'm not quite convinced that just throwing money at something is going to necessarily you know protect us against it what do you read outside statistics oh I read some crime novels but I'm interested in I really like history and biography and especially military history I'm obsessed with the second world war so I I spend my time visiting you know war sites around Europe and further and in India. So that's actually what I'm interested in. Why are you so interested in that? I've been asking myself that. Why am I so interested? I think it's growing up in this generation, born in 1953 in the UK, the war was around us all the time, in the films and the culture and the experience of the adults around us, things like that.

53:27We grew up with it as small kids, absolutely obsessed with it. And I never really have quite lost my interest. I think partly because every time I read anything about it, I just sort of thank heavens for my, well, what's known as constitutional luck. The fact that I was born when I was born into a society I was born into, which was staggering constitutive luck. now if you were to apply or if you were to put on your professor in statistics hat and give advice to young people using some numerical math or whatever what what what is your advice to young people how should they think about the life in statistical terms yeah well i do i mean some of them even read my books which is i don't i had a 17 year old come i nearly burst into tears because he came up and he said, I really like your book.

54:19And I said, well, it's not actually aimed for you. I really, I was so moved that he liked it because I actually think there is some stuff in there about, you know, we have to face up to uncertainty. We don't know what's going to happen. I didn't know when I was 18, so I had no idea what was going to happen in my life. As I said, I had enormous constitutive luck where I was at that time. Lots of opportunities, you know, very secure situation. Everything paid, healthcare, university, everything all paid for. I was in a really privileged situation. But there's still, of course, massive unpredictability.

54:53But I didn't mind. I felt that I'd got a good upbringing, which gave me resilience. So the crucial thing, and of course this is as relevant to corporations as it is to human beings, I think is resilience. It's the number one priority because that's how we deal with the deep uncertainty in all situations. The fact that we don't even know, we can't even list what might happen, particularly somewhere in the future. So what we have to do is cultivate resilience, which is an ability to deal with, if not learn from, benefit from anything that can happen, whether it's good or bad. And for some reason, I think I developed quite a lot of it.

55:36I'm not quite sure. Maybe it's a nice, secure upbringing I had and the good fortune I've had. But for some reason, I think I got it. I don't know. And I think, though, again, I would say this for young people that you have to go out and take risks. Don't be reckless. I say this. I do. I say I give talks in schools. I say you've got to take risks. Don't be reckless. Cover yourself for the major downsides. You know, just be careful, but take risks. So go out, you know, camping on your own in the middle of a moor. But don't be stupid. Let people know where you're going and make sure you've got the proper kit in Norway, because everyone knows about that stuff, but not in the UK necessarily.

56:13so go and tell you go and have those adventures but don't be stupid and so and it's through having you know those sorts of adventures and taking some risks and being in situations where you're not quite sure what's going on that you develop resilience so i just say to young people go out there and have adventures but don't be stupid david i had a 90 probability on this uh podcast being very good but it's even better than i expected i just absolutely love talking to you So big thanks for everything you do to society and increasing the knowledge of stats and just for being such a wonderful communicator.

56:50Big thank you. Well, thanks so much. This is somewhat outside my comfort zone, a lot of this. But anyway, so I suppose I had to take the risks of being – there's the biggest risk I've done for quite a long time is being on this podcast. I tell you that. Great. Big thanks. Thank you. Bye-bye.

57:12Thank you.