Black Box: episode 5 – The white mask

0:00This is The Guardian.

0:30But should we trust them? Black Box the Chatbots is coming very soon on the Guardian Investigates feed. And because Today in Focus is off this week, we're bringing you the first season of Black Box instead. These episodes are exactly as we broadcast them in 2024. And just to note, yesterday we played an episode partly about the love story between a man named Lee Johnson and his wife Yoki, who was diagnosed with cancer. We found out this week that Yoki passed away on Wednesday morning. Lee was with her to the end and he asked us to let you all know. We're sending our love to Lee, their daughter Rose and their whole family.

1:09May Yoki rest in peace. Today's episode is next.

1:19Melissa Williams lives about 20 minutes outside of Detroit in a neighbourhood called Farmington Hills. Very suburban neighbourhood. A lot of young families. Well, there's a lot of young families and there's a lot of older... Until a couple of years ago, she had a pretty quiet life there with her husband, Robert. They were childhood sweethearts. Me and Robert met at my first job when I was 15. So we've been together a very long time and we've been married for 14 years. We have a nine-year-old and a six-year-old girls and they take up most of our time. Alexa, play Reed Flute. Here's Rich Flex by Drake and 21st.

2:02No, Alexa, stop. Not Rich Flex. It was a Thursday in January 2020. Robert was finishing up work and Melissa was at home with the kids. And then she got a phone call. And it's the police. They're calling about her husband, Robert. They just went right into it, like very quickly, that they were looking for Robert and they needed to talk to him and that he needed to turn himself in, but wouldn't say for what. They would not give her any details. They did not say what his supposed crime was. This, by the way, is Guardian US's senior tech reporter, Johanna Buyan. She's the one who first told us this story.

2:44They just said they needed him to turn himself in. She has no idea what they're talking about. It was just confusing. The whole thing was confusing. I said, why don't you call him yourself? He's at work. And I gave them his phone number. And then we hung up. And she immediately calls her husband, Robert, who is at work. So when she calls me, I'm like, what are you talking about? He's just as confused. And she's like, no, be serious. I said, I don't know what's going on, but I just got a really weird phone call. And I'm like, I don't know what you're talking about. While Melissa was calling Robert, in the middle of the call, Robert started getting a call from a number that he didn't recognize.

3:25Click over to talk to them, and they told me the same thing. And I'm like, so where are you calling from? So he said the third precinct. And I'm like, that would mean that the crime would have had to happen in that area. And so I'm like, wow, I'm never in that area, so you got the wrong person.

3:44Robert hangs up, tries to get on with his day. and drives home to Farmington Hills. But when he turns into his street... There's a police car parked outside of his house. And when he pulls into his driveway, they pull up behind him and block his car in. You have a walk for your car first degree. Go ahead and put your hands behind your back. I'm not going to drive all the way here from Detroit. No, no, no, no, no, no, no. We're the police. We got the police body camera footage from that day. Let's not make a show in front of your wife and kids because you are under arrest for your warrant, for your outstanding warrant.

4:19Can I see the warrant? It's a suburban street. The sun's going down, there's snow on the ground, bare trees. Robert's standing beside his four-wheel drive, looking tense but holding it together. His wife, Melissa, comes out, and she goes straight to his side. It didn't feel real. They didn't introduce themselves. They didn't show their badge or anything. It just felt like, are these even real police officers? I don't know. they kept saying they can't tell me they can't tell me what i what i was being arrested for he's like well when you put the handcuffs on i'll show you the warrant i have it in my hand and we having like a discussion but this is like hey you need to stop because if they're the police you know this is 2020 it's a dangerous time to be um resisting let's be civil about it let's not make a show.

5:11Okay, so I don't get to go in my house? No. No. Just go home. Settle down, bro. His oldest daughter, Julia, comes outside. Robert Williams and Melissa Williams call her Juju. She's stepping around the snow. She doesn't have any shoes on or a jacket. Why are you looking at, they made a mistake, baby. It's okay. I'll be back in a minute. She was still five. I said, Juju, go back in the house. I'll be back in a minute. I got in the car. I'm going to move back a little bit, OK?

5:50The officers who arrested Robert Williams that day were acting on a tip, but not from a witness, nor from an informant. In fact, not from a person at all. From The Guardian, I'm Michael Safi. This is Black Box, Episode 5, The White Mask.

6:14And if you didn't have the camera, do you think you would have known the truth? No, absolutely not. We aren't just talking about cameras. The technology goes a lot further than that. Someone is always watching. Surveillance on steroids. Because these days your face is no longer just your face. It's a data set that machines can use to ID you. And often, you won't even have a code.

6:48That day in January, more than four years ago, Robert's ordinary family life was upended. He found himself in the back of a squad car and then being processed at the Detroit Detention Centre. Yeah, it's a dirty building. They have you get up and you'll get patted down. All of his valuables are taken off of him. Watch, cell phone, wallet. He gets his fingerprints taken. They're not really explaining how or why they're taking any of this information, what they're going to do with it. By this point, Robert had seen the warrant. It said that he was under arrest for felony larceny, stealing. but what he had supposedly stolen and when.

7:32Nobody would tell him. So I see a court-appointed attorney and I tell him, I don't even know why I'm here. And he's like, really? Like, he didn't know what happened. And I'm still trying to figure out, well, why am I locked up for it then? That's when he first learned something about the actual crime he's being accused of. Another detainee in the same cell overhears some officers saying Robert's been arrested for stealing watches. And I'm like, nah, couldn't have been. I don't know, it just didn't make sense to me because why would I still have watched? You know, he's left in the dark, both literally and figuratively.

8:10The next morning, Robert was still in custody, and his wife Melissa was at home with the kids, just trying to make things seem as normal as possible. I just got up and I had to take Julia to school and had to go to work. Julia lost her first tooth that next morning. Just tried to make it regular. I didn't know, like I was hoping that the fact that she lost a tooth would overshadow the fact that her dad went to jail the night before and that would be her story of the day at school and not. Because we didn't know what to say to anybody because we didn't know what was happening still.

8:49Eventually, Robert was taken from his cell for questioning. We asked the Detroit Police Department for an interview for this podcast, but they declined. So here's Robert's version of events. Two detectives sit him down. He asks them to explain what exactly he's done wrong. What's he being accused of stealing? And the detectives say, we'll tell you if you sign this form waiving your right to an attorney, which Robert does. And then they start asking him about this watch store, Shinola. He tries to picture the store and remembers he's been there, but like six years ago. He can remember specifically when he was there because Julia had just been born.

9:31It was the summertime and he was pushing her in a stroller. And Shinola, I think, had just opened up. And my brother-in-law thought that Shinola was the most coolest store in the world. And I'm like, OK, let's see about that. And then we went in there and I was not impressed. And I didn't want to watch from there. And I didn't, I was like, it's all right. Then, Robert says, the detectives pull out a CCTV screenshot from the day the store was robbed. Johanna's seen the footage. The man is in the corner of the store. He's wearing a red hat and a black leather jacket. He starts grabbing watches off the shelf.

10:15There's no employees around him. and he's shoving them into his pockets. This is not a master criminal. The man looks nervous. He keeps looking over his shoulder, freezing, like he expects the staff to come around the corner any second. And then he leaves the store with thousands of dollars of goods in his pockets.

10:37The police show up to the store and they really don't have a lot of evidence about who stole all those watches. There was no one who really saw the person who did it. There were no fingerprints. You can really only make out that he was a black man, that he was wearing a St. Louis Cardinals hat. And that's pretty much it. So when the police take out this CCTV still and show it to Robert. As soon as they pull it out, it really does not look like Robert at all. Yeah, they showed me that and I had to tell the guy that's not me. Robert also says, you know, I hope you don't think all black men look alike.

11:18And then he had to look at it again and say, hold on, let me talk to my partner. And they conferred and said that they agreed that I wasn't the person they was looking for. It's towards the end of their questioning when the detectives say something that gives Robert this little glint of a clue as to why all of this is happening. In that conversation, they mention something along the lines of, oh, well, the computer must not have gotten it right. Yeah, he told me that. He said, so I guess the computer got it wrong. And I'm like, what? He was detained for a total of 30 hours. And after that, he's released.

12:01Because they couldn't keep me any longer. Then the realisation hit that he was out, but it wasn't over. It wasn't just like a, sorry. It was like, no, now you need to get a lawyer because you have to appear and you still have to defend yourself against this mistake.

12:20Even though two detectives had appeared to acknowledge this was all based on a mistake, they didn't do anything to stop the process that was now in motion. Robert was on bail, facing charges that carried a maximum five-year prison sentence. He was due back in court in two weeks. And in the car, driving home from the detention center, he and Melissa started trying to make sense of what the hell had just happened. Robert was recounting to his wife everything that he had went through and told her that in his interview with the detectives, they mentioned something about a computer not getting it right.

12:57And it was basically Lisa's analytical part of her makeup that kind of figured it out because... At that point, Robert hadn't heard much about facial recognition technology, but Melissa had. At the time, I had just known that it was something new that Detroit was doing and that... And she was the one who started suspecting that the police had used facial recognition technology to determine who burglarized the Shinola store. I started reading more articles from the ACLU and others that mentioned facial recognition and the flaws that it has. And we reached out to some local attorneys and I was like, I'm going to try the ACLU.

13:42I don't know if they'll get back to me, but they did. They realized that this was much bigger than just trying to prove that Robert didn't do it. There was technology involved that they might need some help and support in defending against.

14:06Facial recognition technology, the kind that Melissa and Robert now suspected might be behind his arrest, is a type of AI system powered by neural networks. Feed one of these networks enough images of human faces, and eventually it starts to know what it's looking at. A mouth, or a nose, or an eye. Give it some more, and it can begin to distinguish between different faces. The unique curve of someone's mouth, the shape of their nose, the colour of their eyes. The dream, with enough training, is a system that you can show a picture of someone's face, anyone's face and have it tell you with perfect accuracy the specific person it's looking at.

14:46At least that's the dream.

14:52And it was a seductive one for police departments across America, including in Detroit, where around July of 2017, the police department signed a contract with an AI company. The police also built a database of everybody in Michigan's driver's licenses, without their explicit permission, by the way. They could upload the faces of anybody they suspected of a crime, and the AI would compare it to this database and return a list of names. The hope was that it would make Detroit safer. But there was a problem. I've been seeing the problem for some time, actually, in my book, Unmasking AI. One that had first become apparent to a researcher called Dr.

15:36Joy Bualamwini, back when she was a computer science student, trying to teach a simple AI robot named Simon to play a game. Peek-a-boo, so cover your face, uncover your face, that kind of thing. The point was just to get Simon to recognise when it was looking at a human face. And the robot had no trouble recognising the face of her white classmates. But when Joy went peek-a-boo and dropped her hands... And my robot did not detect my face. And so that's when I started to have a notion that, OK, there might be something going on here. A couple of years later, Joy was at MIT doing her master's and playing around with a different facial recognition system.

16:17And once again, this thing just couldn't make her out. It was like she was invisible. And she realised she was carrying something in her bag that might be useful. I actually had a white mask with me since it was Halloween time in the United States. And I put on that white mask. That white mask was detected before my actual dark skin face was detected. And so it was that experience that pushed me to really start looking into these types of technologies a little bit more. This phenomenon became Joy's master's thesis. She took four of the biggest facial recognition systems and tried to figure out, were they biased?

17:02She showed them more than a thousand different faces and asked, is this a man or a woman? And looked at where they made mistakes. And she discovered, if you show them a picture of a white man, they got it right nearly every time. When you showed them a picture of a woman or a black man, they started making a pretty shocking number of mistakes. And there was this one group that the systems were absolutely terrible at identifying. Black women. The best performing system was 32 times more likely to make a mistake with a black woman than a white man. The National Institute for Standards and Technology, after my research came out, they did a more comprehensive study looking at facial recognition.

17:50And they found that in some cases, systems were 10 to 100 times worse when it came to recognizing the faces of individuals who were Black or Asian as compared to Caucasian faces. AI systems are not neutral. Yes, they can have gender bias. Yes, they can have racial bias. There's even colorism, skin type bias, right, baked into AI systems. The reason for this is not because AI is biased, but the data that we use to train it is. Those data sets, the millions of pictures that you feed into an AI system to teach it to recognize faces, they're assembled by people making choices. And that's where the bias creeps in.

18:43What happens is these training data sets generally don't reflect the real world. And so you might have what I call pale male data sets, data sets that are 80 % male, maybe 70 % lighter skinned individuals. And so it's not so surprising that these highly biased training data sets, when they're used, actually then result in systems that work best for the types of faces they were exposed to. According to Dr. Joy, there was a lot of scepticism when her research was published. Oh, the initial response was a combination of bemusement and eye rolls. Like, are you serious? You're telling me the algorithm is racist now?

19:26You tell me math is sexist? I had those types. Joy formed an organisation, the Algorithmic Justice League, specifically to raise the alarm. A few other researchers around the country were doing the same. But as they were warning about these kinds of systems, police departments were embracing them. And to Joy's horror, starting to use them to investigate crimes. I continued to see problem after problem, and the stakes were higher and higher. So even when I gave my TED Talk, I think that was back in 2016, I predicted and warned we were going to get false arrest. Police departments can currently look at these networks unregulated using algorithms that have not been audited for accuracy.

20:16But misidentifying a suspected criminal is no laughing matter, nor is breaching civil liberties. It was predictable and preventable.

20:32Predictable and preventable. So then how did Robert Williams end up in a jail cell? So I spoke to Robert's lawyer at the ACLU, Philip Mayer. We knew that the city had taken a photo still from surveillance footage at the Shinola store. Very grainy, really hard to make out and poor quality. So this photo that was used for facial recognition is one that really anybody with common sense should have known not to use for facial recognition. And yet, it was. The police ran the picture through their facial recognition system. It came back with hundreds of possible matches for the man in the Shinola store.

21:16And one of them was a guy who hadn't been anywhere near that store in six years. Was this the only evidence that police were relying on when they arrested Robert that day in his front drive? For the most part, yes. The only other evidence, so-called, and we're in an oral medium, so you can't see me putting scare quotes around evidence, is that the detective took the photo, put it in a lineup with five other photos, and showed it to a security official associated with the Shinola store. Who actually was not present at the time of the burglary. The only thing she had done was watch the same surveillance footage from which the still was taken.

22:02And so it wasn't, you know, an eyewitness interview where someone had seen the person who was committing the crime and then was able to then try to match that person to an image. It was a person was looking at the two separate images that the police had come up with and said, yes, I think those two people look alike. The Robert Williams case was the first example to come to light of someone being wrongfully arrested in America because of AI. And you might not be surprised to learn, it wasn't the last. And I'm Glenda Lewis. Tonight you'll hear the story of one man who was wrongfully arrested due in part to that technology.

22:40Michael Oliver was being investigated by the same detective, was similarly arrested based on nothing other than a facial recognition match followed by a photographic lineup. I'm asking him, like, how was my face even brought up? And that's when he told me, like, your face, they put you in a lineup. The face recognition said it was you. The exact same sort of defective procedure that was done in Mr. Williams' case. It happened again. It wasn't any evidence. It was just a picture, a computer said we look alike. And again. Facial recognition technology was the only source linking him to the crime.

23:16And again. Hennepin County claimed his arrest was the result of faulty facial recognition software and not human error. Since Robert Williams, there have been six other cases that we know about. All but one of them have involved people who are black. One of the most recent was the frankly mind-blowing case of Portia Woodruff. who was eight months pregnant at the time, was arrested and accused of carjacking. Based on precisely the type of shoddy use of facial recognition that we saw in Mr. Williams' case. There was just no clear evidence tying her to that crime, but she was accused of carjacking.

23:58Six officers showed up at her home when she was getting her little daughters ready for school. She appeared at the door, you know, fully eight months pregnant. It was a female police officer at the door. She asked if I was Portia Woodruff. I confirmed. I said yes. She said, I have a warrant for your arrest. And the police still said that you're under arrest for carjacking. In the midst of the conversation, you know, I opened up my door a little bit wider so she could see so that they could see, you know, I was eight months pregnant. And I also pointed to my cars that were in the driveway at the time.