Should we use AI to make legal decisions?

0:00This BBC podcast is supported by ads outside the UK.

0:30all in one place, so you can invest your way. Visit schwab.com to learn more. Hello and thank you for listening to this edition of The Law Show, the podcast that helps you understand the law. In this week's podcast, the famous Irish author, her publisher, and the UK's anti-terror laws. Is it now illegal to buy Sally Rooney's books? We'll be finding out later in the podcast. But to begin, we're looking at artificial intelligence and its place in the courts. Now, if you're of the view that robo-judges are still a thing of science fiction, I'm thinking of Futurama or the Mechanismo judges in Judge Dredd, then it may surprise you to know that in some countries, AI is already being used to assist judges with sentencing.

1:22Whether you like it or not, Artificial intelligence is increasingly being integrated into all aspects of our lives and work. So how could AI be used by judges and the courts in England and Wales? Legal pilots have been held across the UK, where AI has been used to speed up admin, case management and transcription. But how would you feel about it being used to make judicial decisions? There are still chronic court delays across England and Wales, and AI could speed up justice and reduce costs. But what safeguards need to be put into place to protect our rights and the credibility of the legal system?

2:06These are questions which have been occupying Sir Geoffrey Voss, who until recently was the second most senior judge in England and Wales. He's just retired as Master of the Roles, which is the Head of Civil Justice and President of the Civil Division of the Court of Appeal. And in a rare interview, I asked him how he sees technology being used within the civil court system. I think it's very important that we adopt AI within the judicial system and within the legal system. Actually, what's happening at the moment is nearly every lawyer uses AI to answer legal questions. Nearly every litigant, everybody who has a legal problem, will go on ChatGPT or Claude or Gemini to ask their legal question before they go anywhere near a lawyer.

2:58It would make the legal system very strange if it did not embrace and adopt the use of AI. But the big question is how we're going to use AI, how we should use AI, and where the boundaries should be drawn. You've seen AI being used in a lot of different legal systems across the world. Should we take their lead in any context? Well, I certainly have. And I had the opportunity recently to visit the Shenzhen Intermediate People's Court in China and to see the program they're using to help them resolve small claims in their court. And they have used that system to resolve 670 ,000 cases in the pilot stage.

3:48So in context, that's a lot of cases. That is a vast number of cases. 670 ,000 contested cases is more than we would ever decide here in England and Wales in a year. But the important thing about what they're doing in Shenzhen is that they are using the programme they've created to assist the judge in three main ways. It summarises the facts, it lists the issues the court has to decide, giving the judge the opportunity to say, yes, that's the issue I want to decide and this is how I want to decide it. And if the judge says yes or no to the answer, the programme then produces reasons and an order for the judge to sign up to.

4:40Now that, we were told by one judge in Shenzhen, makes the process six times quicker. So the question is, is that an appropriate way to decide cases in other jurisdictions? And my answer to that is it depends. In jurisdictions where they have massive backlogs, Where the choice is between either using AI to decide the cases or not deciding the cases in a sensible time frame at all, then the answer might well be that AI is a reasonable thing to use. But the problem, in my view at least, is that our constitutional settlement is that cases have to be decided by an independent and impartial tribunal, which to the people in our country normally means a human judge.

5:38And for that to change, it would be necessary for there to be a political decision, possibly also legislation, to give authority to judges to start using machines to help them decide cases. Were there to be legal change, acknowledging that there is a right to a fair trial, and this currently doesn't seem to include AI judgments, but were there to be legal change, what areas would you see AI being used? Yes, I think that's a very good question. There are many decisions made by courts which are fairly algorithmic in nature. There are many cases in our county courts which decide on whether a claimant is entitled to damages for personal injuries suffered in a road accident.

6:28And the law on that is fairly simple. There are many cases and you make an algorithmic determination of where your case fits on a scale of damages and then a sum is awarded. and it would be entirely possible to create a dedicated programme which would reach a conclusion as to the amount of damages which people would get quickly and cheaply rather than waiting a long time for a judge to decide the case on the evidence. To my mind, it would be appropriate if there were legislation in place allowing it to happen and if the parties consented. If the parties didn't consent and there was legislation allowing it to happen, then it would also be appropriate that there might be a backlash and parties might say, no, I really want my case decided by a human judge, even in that area.

7:24Do you think that there's a line that should be drawn between AI assisting judges and AI making judicial decisions? I think there's a very important line that needs to be drawn between AI assistance and AI decision making. But I think that that line is extremely difficult to draw and it's extremely difficult to gain consensus on where the line actually is. and the reason for that is that most people would think that it was the judge's job to read all the papers, to hear all the evidence, to read all the law and come to a conclusion on the outcome that was appropriate in the case. But if you use AI in the way I described in Shenzhen then at least a part of that process is short-secreted.

8:20Then a part of the decision is either made by the machine or indicated by the machine rather than by the human judge. Now, of course, a human judge using the machine is capable of saying, no, I want to see all the documents, I want to read all the documents, I want to hear all the evidence for myself, but that would save no time at all. So you have this very difficult interface between saving time and money and using AI and not saving time and using AI. One of the reasons that is motivating AI being adopted in the judicial system is that we have astonishing and worsening backlog. Is this a good reason to adopt AI?

9:08Well, first of all, we don't actually have so many backlogs in the civil courts at the moment. But it is a reason why some jurisdictions will certainly want to use AI, because delaying the delivery of justice is obviously unacceptable. And if you can clear backlogs and decide cases very quickly, then you are providing much better access to justice. It's also important to understand that delayed justice has an economic cost. because when people are concerned about a legal dispute, they are not so productive at work. Many people will have the experience of seeing colleagues come into work, having received some unwelcome letter from the gas board or the electricity board, asking for more money than they think they owe, and spending two hours on their mobile phone tapping away, trying to sort it out, and not working and also becoming psychologically damaged by the experience.

10:18So dispute is very costly for the economy and it's therefore extremely important that we do get disputes resolved and one way of resolving them, of course, is to use technological means. People accept that disputes can be decided by machines if they are using AI dispute resolution on eBay or Amazon or other programs like that. People accept that it's perfectly reasonable to consent to having a dispute resolved by a machine. And therefore the gap between consenting to have it happen and not consenting because it's being imposed on you by legislation by the government is not quite so great. And also people will use AI programs to get the answers to their problems.

11:15And if a claimant gets the same answer from the AI as the potential defendant gets, it's likely that they will settle the dispute because they'll both know the potential outcome, even if there is no machine-made decision-making if you actually go towards a court. So I think the answer about whether it's going to happen is that it will, but we have to have the debate because it is extremely important that we deliver justice in the way that is accessible and understandable for the people we are serving. Do you think there are areas that AI should never go into in the law? I personally do. I think there are many types of decision which people would find impossible to imagine a machine should be involved with.

12:10So the example I always give is a decision in a family case as to whether or not children should be removed from a parent. But I know that that is not a universal opinion. I've heard academics tell me that, no, that is precisely the kind of decision that people would want a machine to decide because it would take the emotion, the difficulty out of the decision if a machine decided that. For my part, I think that's wrong. I think that people would expect a human decision to be made by a human judge. But I can understand there has to be a debate. and in that debate it's a political decision and it should be an opinion for the people, an opinion for the democratic process and an opinion ultimately for Parliament.

13:01How do you think it would work in practice if it were adopted in England and Wales? Do you think there would or there should always be human oversight at the end of an AI decision? It's absolutely imperative that any decision is actually taken by the judge even if it's assisted by AI. But the problem with human oversight is that if you have human oversight and human checking of every process undertaken by the machine, then you don't save any time or any money. So you have to work out where the machine can assist without actually taking away the independent decision-making of the human judge. That's until legislation says that machines should make decisions.

13:50Now, I'm very aware of how much I've used AI today to tell me where to eat and how to get here. But do you think that the public are ready for this kind of AI use in the judicial system? Well, I think you have to be very careful to create a justice system that is relevant for young people as well as old people. And young people, those under, say, 18, have mostly been brought up with access to AI. They expect every question to be answered by ChatGPT or Claude or Gemini. And therefore, for them, a system which uses AI would be far less strange than for people of a much older age. I don't think you can assume that everybody in our society would be horrified by machine-made decisions.

14:44Do you think that it should be a legal right not to have a decision taken by AI? At the moment, people have a human right under Article 6 of the European Convention on Human Rights to have their legal cases decided by an independent and impartial tribunal established by law. And therefore that is their right. There is a question as to whether a machine judge could ever be an independent and impartial tribunal. My own opinion is that it could not be, but there are others who disagree with me. I'm very curious. Why do you think it could not be? Because I think that independence and impartiality are qualities that when the principles were laid down and even now they are human qualities and I think that the article 6 right is directed at the concept of human judges and I don't think that can be changed merely by saying we've now got a new technological advance that enables decisions to be made by a machine.

15:56So I think the current constitutional settlement in Article 6 is that decisions must be made by a human judge. And I also think that that is entrenched in many constitutions in countries around the world. So we're not alone in that, even if the countries are not governed by the European Convention on Human Rights. So many people now, when they first are faced with a legal problem or a legal question, will ask AI. Of course, one of the problems with that too is that it may not be correct. Is there a danger with the public using AI as their first port of call for legal advice? Well, of course, there are dangers.

16:39There are several dangers. The first danger is that they treat the AI as an oracle. They ask a single question, get a single answer, and don't question it, even if it's fairly obviously wrong. That happens a lot. And we will, in due course, train our young people to use AI much more effectively than that, so that they can interrogate and question the AI and get a better quality of answer. The second danger is, of course, that they don't understand the answer or they misunderstand the answer. And I think that as time goes by, people will use AI to give initial answers to their legal problems. But because they need to have absolute confidence in the answer before they act upon it, they will still want to have a human lawyer explain the answer and also confirm the answer and also help them accept the answer where it's adverse to their interests.

17:42Because if you are going to, for example, decide on whether you're going to lose your house or your job or your home and you're going to ask AI for the answer, you're going to require a lot of persuading if the AI says that you are going to lose your house or your home or your job and there's nothing you can do about it. And so the question of what trust and confidence people have in AI is a really difficult and important one. Lawyers sometimes fear that they're going to be put out of business by AI because the grunt work, if you like, can be done by the machine. I think they're wrong because I think actually humans generally have more confidence in other humans than they do in machines.

18:31and if an unwelcome answer is given by a machine, then the humans are likely to want to check it with another human and that other human is likely to be a lawyer. So we'll always rely on expert barristers and judges to still give us good answers, or the right answers maybe. No, I don't think human judges or human lawyers are necessarily more reliable than machines. In fact, I think it's quite likely that in time the machines may come to be objectively more reliable. But that's not the question. The question is, in whom do people have trust and confidence and on whom are they prepared to rely when something goes wrong?

19:18And the answer to that is more likely to be a human than a machine.

19:28Welcome to the Haunted Library. Join me, Colin Morgan, as I read Frankenstein by Mary Shelley. A story of science and ambition, loneliness and longing, and what it truly means to be human. Listen to The Haunted Library wherever you get your podcasts or find us on YouTube.

19:57You're listening to The Law Show and my thanks to Sir Geoffrey Voss for joining us. Now, it's currently not possible in the UK to buy the e-books of normal people, conversations with friends, intermezzo or any of the other novels written by the Irish author Sally Rooney. Her publisher, Faber, has stopped selling them and has stopped any more printed copies being sent to bookshops across the UK. Why? It's because Sally Rooney has publicly declared that she wants to donate her UK royalties to the prescribed group Palestine Action. The group's founder is currently challenging that prescription in the Supreme Court.

20:34But for now, Faber risks being prosecuted if we're to transfer sales earnings to an author who intends to fund a banned organisation. But what does this mean for people buying her books? Let's look at the law around this case with Jonathan Hall, King's counsel. He's the government's independent reviewer of terrorism legislation. Jonathan, remind us, what does prescribed mean? It means that the Secretary of State has made an order under the Terrorism Act 2000 and the effect of that order is that the organisation has become a terrorist organisation and that means that it cannot be funded, you cannot be a member of it, you cannot invite support for it and a tonne of other offences and in effect it triggers some of the strongest powers and some of the most extensive offences that are known to terrorism legislation.

21:32What do you mean the most extraordinary powers there? Well, you can arrest someone just for being a terrorist. If you're talking about normal criminal law, you can't just arrest one person for giving money to another person. You've got to say, well, I think that you are assisting or encouraging crime. But for prescribed organisations, it's different. It's enough that I give money to a prescribed organisation. The prosecution wouldn't have to say, well, that money is going to be used for this particular action or attack. It's enough that any money is going to the prescribed organisation. So that's what I mean.

22:06Let's imagine, like many normal people, I go to a bookshop today and I find a copy of Normal People by Sally Rooney. Is that bookshop committing an offence? This will be an annoying answer, but I think it does depend. And I would make a strong distinction between the letter of the law and the likelihood of prosecution. What the law says is that if I know that the money that I'm giving is going to work its way eventually to a prescribed organisation, I risk committing an offence. Now, most of the time you have cases where people have reasonable grounds to suspect that might happen. So let's say I'm giving money to Gaza, for example, I might have reasonable grounds to suspect the money might end up in the hands of Hamas, which is why these organisations try as hard as they can to avoid that happening.

22:57And there are all sorts of mechanisms they use. This is different, though, because you have someone who's saying specifically and openly, I want to give money to a terrorist organisation. So the question would be, am I, by buying a book where the sale money will go to the shop, where the shop, as I understand it, will then give some money to the publisher, and the publisher will give some money to the author, am I entering into such an arrangement? And I think the answer is theoretically yes, but the prospects of the police arresting someone and the prosecution prosecuting you for this are vanishingly low.

23:35And the reason is that it's quite remote. I think it's quite hard to say that someone who's buying a book is definitely doing it deliberately in order to fund a prescribed organisation. I mean, I suppose it's possible, but it seems to me very unlikely. There haven't been many cases about this, but there was a case in 2002 where a man called O'Driscoll was driving his car back from France and he had loads of magazines in his boot. And the police reasonably believed that he was going to sell these magazines in order to fund a Turkish prescribed organisation. He said, hang on a second, I'm just someone with a load of magazines about human rights violations in Turkey.

24:15So what are you talking about? And the court said, well, it doesn't really matter what the content of the magazines are. If you are going to use those in order to sell them, to generate revenue to give to a prescribed organisation, then that's sufficient. So if I buy a book by Sally Rooney or the bookshop sells me a book currently available by Sally Rooney, it seems prosecution is unlikely more than anything else. I think that's right. And that's because, quite rightly, terrorism law is broad enough to catch the bad guys, but you want to use it against people who you think are causing harm to national security.

24:50Now, I mean, to take a sort of contentious example, so the arrests of the sign holders, people who are holding signs saying, I oppose genocide, I support Palestine action. I suppose what the police and the prosecutor would say is, well, that is causing some sort of harm to national security because it's very vocal and open support, often in the heart of Westminster, on television, supporting an organisation which, according to the Home Secretary, has caused very serious damage. And in fact, we know some injury to a police officer. So I suppose you might say that that has some public interest in prosecuting.

25:21I think it would be quite difficult to construct a similar public interest argument in relation to someone who is simply going to buy a book. So let's imagine that Sally Rooney had never made her intentions public. She just put all of the royalties that she gets from her publisher away and gave them to Palestine Action. Would the publisher still be breaking the law in that instance? No, terrorism law quite rightly depends upon either knowledge or at least having reasonable grounds to suspect. Sally Rooney is an Irish citizen and she said that she cancelled a promotional tour in the UK for fear of arrest.

25:56If she did come here, could she be arrested for supporting or funding a prescribed organisation? Yes, she could be. What the police would ask themselves is, do we need to carry out an arrest in order to pursue our investigation? So I suppose it would depend upon do they have evidence that she's made payments, whether she has actually paid any and whether there's any evidence that would be available to the authorities. But in principle, let's imagine that the Met Police had a copy of some bank statement. They could show that it was going to Palestine action. Then I suppose, yes, she could be prosecuted.

26:28Do banks have a duty to disclose funding to prescribed organisations? Banks have got very heavy duties. if the bank reasonably suspects, doesn't even have to suspect, but has reasonable grounds of suspicion. And if they don't tell the authorities, they commit an offence. What are the potential penalties for funding a prescribed organisation? Heavy. Up to 14 years. There have been some Hamas funders and some Hezbollah funders quite recently. And you'll be looking at sentences in around sort of five, six years. So Sally Rooney has said that money from adaptations of her novels, these are called residuals, would also be given to Palestine Action.

27:06This would mean broadcasters like the BBC would potentially be breaking counter-terrorism laws by paying her. Is that true? Yes, because if I provide money to someone and I know or have reasonable grounds to suspect that the money I'm giving will ultimately end up with a terrorist organisation, then I could commit an offence. So I don't think it's sort of outrageous. to imagine the BBC having to think carefully as well. On that note, the BBC press office has provided us with a statement which says, The BBC currently makes no direct payments to Sally Rooney and she is not contracted by us. Matters relating to prescribed organisations are for the relevant authorities.

27:52All BBC residuals for writers, that's a TV word for royalties, are paid via a company called Writers Digital Payments Limited. They are given an annual lump sum by the BBC, and they make payments to each author. We've approached them for comment, but have had no response. Jonathan Hawking's counsel, thank you. That's all for now. If you have a question about the law, you can contact us at thelawshow at bbc.co.uk. And our WhatsApp number is 0330 678 4770. We don't give legal advice, but we can help explain how the law works in future podcasts. The Law Show is presented by me, Dr. Joelle Grogan, and produced by Ravi Naik.

28:42The editor was Richard Fenton-Smith. If you enjoyed this podcast about the law, you can subscribe to learn more.

28:56Welcome to the Haunted Library. Join me, Colin Morgan, as I read Frankenstein by Mary Shelley. A story of science and ambition, loneliness and longing, and what it truly means to be human. Listen to The Haunted Library wherever you get your podcasts, or find us on YouTube.