Revisited: Dr. Joy Buolamwini wants AI to see everyone

0:00Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential. a non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up. Every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no preset spending limit, they were able to spend more and earn more rewards while bringing their vision to life.

0:36Today, Raz's success is proof that with passion and the right support, it's possible to make your dreams a reality. Learn more at CapitalOne.com slash business cards. Hi, it's Rana, and today we wanted to share with you one of our earlier episodes. It's an episode that's near and dear to my heart. I speak with my friend, Dr. Joy Bulamwini, who's a computer scientist, poet, and one of the leading voices on algorithmic bias and how we can mitigate it. This is a must-listen, and I think you'll enjoy it too. Pioneers of AI is made possible with support from Inflection AI. It's not just enterprise AI.

1:17It's your enterprise AI.

1:26In the winter of 2020, two Detroit police officers arrived at Robert Williams' home in Michigan. Without any explanation, they handcuffed him on his front lawn while his whole family was watching. Williams gave the details of his story to a House Judiciary Subcommittee. He pulled up. He said, are you Robert Williams? I said, yes, I am. He said, you're under arrest. I said, whoa, you can't just arrest me for that. What am I under arrest for? He said, don't worry about it. and proceeded to put the handcuffs on me. I told my wife and my kids who were coming out the house at the time, don't worry about it, you're making a mistake, I'll be right back.

2:15Unfortunately, I wasn't right back. Williams spent the next 30 hours in a Detroit jail. Detectives said he was arrested for federal larceny. The proof? Images captured by a surveillance camera and analyzed by a computer algorithm. He said, so that's not you? I said, no, that's not me. He turns over another piece of paper. He said, so I guess that's not you either. I held that piece of paper up to my face and said, I hope you don't think all Black people look alike. He turned over another paper and said, so I guess the computer got it wrong. The computer got it wrong. As in faulty facial recognition technology, wrongfully identified Williams as the culprit.

2:59This was the first known case of its kind, but it wouldn't be the last. And there had been others who had been arrested by that same police department. And part of this is these are the stories we know about. The AI systems are acting as silent witnesses with flaws. That's Dr. Joy Bulamwini, computer scientist, poet, and national bestselling author of the book Unmasking AI. And these flaws that she's talking about have a name. algorithmic bias, and it can have major consequences, including wrongful arrest. But it can also drive any number of computer-based errors that could directly affect you.

3:40Bias could mean a medical misdiagnosis, or a denial of your mortgage application, or being skipped over as a qualified job applicant. I'm an AI optimist, but if we're not mitigating its pitfalls, we'll never see its full potential. Which is why in this episode, we're taking algorithmic bias head-on. Why does algorithmic bias happen? And how can we avoid it?

4:10I'm Rana El-Khalyubi, and this is Pioneers of AI. a podcast taking you behind the scenes of the AI revolution.

4:33Dr. Joy is one of the leading experts on algorithmic bias, a hero of mine, someone who, like me, is a big believer in AI and a staunch advocate for how we can make it better. But our conversation didn't start there. It started with us talking about what inspired us to get into tech in the first place. Dr. Joy, I am so excited for our conversation. I'm a huge fan of your work and I admire all you're doing to make sure AI is just and inclusive for all of us. I'm so excited to be here. I've been following in your path, watching you from afar, being inspired by the work you've been doing and so excited to be here.

5:13So early on, I guess you met the MIT robot Kismet, which for our listeners who are not familiar with Kismet, Kismet is a, I guess, gizmo-looking robot with like blue eyes and furry eyebrows, very 90s style. But I don't know if you know that, but Kismet inspired my own research because it was the very first social robot that had social and emotional intelligence. And so I've really looked at Kismet as an inspiration for a lot of my work. But how did you meet Kismet? Okay, I think we were similarly inspired. So when I was little, I had a very strict diet. And by diet, I mean media diet. So I could only watch PBS.

5:55Love that. I should have done that with my kids. It's a different time. You know, we were immigrants. My parents weren't exactly sure what else was on TV. So PBS seemed safe. So I would, you know, I would watch Nova, Scientific American Frontiers, all of this. So I was watching one of these shows and they had an episode about robots.

6:21This was, in fact, the PBS show Nova in 2006, when Dr. Joy was still in elementary school. And I remember seeing at that time a graduate student, Cynthia Brazil. And she was the one who had created Kismet, the social robot. If we want to build a robot that can interact with people, communicate with people and learn from people, it has to also be social because that's the way that people are going to try to teach it and the way they're going to try to communicate to it. I hadn't seen a robot that was actually developed to interact with humans. And so the big expressive eyes and the ears and all of that, I was so captivated and I was curious if I could make something like that.

7:09And so from then on, I said I wanted to be a robotics engineer and go to MIT. I didn't know there were requirements. I think also seeing that it was a woman behind it as well. I didn't even question it. And I was just like, oh, yeah, this is, of course, something I could do. Growing up, I had a similar experience. My mom was one of the first female computer programmers in the Middle East, and she paved the way for me and so many other women. For Dr. Joy, her childhood dreams of going to MIT would come true. She would even end up working with the same code used to program Kismet. But her first stop was Georgia Tech, where she was admitted as an undergraduate in computer science.

7:52Unsurprisingly, her class did not look like her. The student body was, and still is, largely white and male. But Dr. Joy's attention was turned towards the non-human residents on campus. You know, the cool robots in the computer science lab. When I was working on one of my first robots, Simon the Robot, I was at the time trying to get Simon to play a turn-taking game with me called Peekaboo, right? Peek-a-boo, you cover the eyes, you uncover the eyes, peek-a-boo, I see you. Peek-a-boo doesn't really work if your robot doesn't see you. Right. And my robot wasn't seeing me. And so that's when I really started thinking, huh, okay, what's going on on the computer vision side?

8:36Because it was detecting the face of my roommate, who was fair skin, green eyes, red hair, you know, not necessarily detecting me. This was Dr. Joy's first experience of being unseen by a robot. And she let it slide. But the second time came a few years later in graduate school at MIT. This time it was too blatant of a mistake to ignore. So I had taken this course, Science Fabrication, and I wanted to explore the concept of shape-shifting. We had six weeks, so I was, all right, not changing any laws of physics anytime soon. So maybe instead of shifting my physical appearance, maybe I could somehow alter my appearance in a mirror.

9:20And so I worked on this project called the Aspire Mirror, and I found this material, half silvered glass, which essentially has a really cool property. If you have something black behind it, it behaves just like a regular mirror. But if there's light behind it, the light will shine through. So I thought, huh, if I put a black background and then add a digital mask, then it would make it seem as though that mask were on my face. And so this is all digital first, right? And it worked. I was like, okay, this is so cool. But now let me see if that filter can follow me. Great. Right? So I needed to get a webcam, and then on the webcam, hooked it up to my laptop.

10:01Now I needed some face tracking software to achieve the effect. So nothing to do with social justice or anything like this. You just wanted, like, some facial recognition technology to, like, just track your face, basically. Yes, I could have this cool Aspire mirror thing. And so I wanted to look like a lion or look like Serena Williams. Now maybe Roger Federer or just whoever. And so when I was trying to get that to work, that's when I noticed it wasn't really picking up my face consistently. So I draw a face on my palm, hold it up to the camera. It was not the best face. It was very much the smiley face sort of thing.

10:39And the face on my palm was detected. So after that, I was like, OK, anything is up for grabs. It was around Halloween at the time, and Dr. Joy had a white plastic mask by her desk, the kind you find at the drugstore. As an experiment, she reached out and put it on. It's not even halfway on my face before it starts being detected. I take it off, my face is not detected. I'm just like, you can't even make this up. Can you talk a little bit about that? Like, what was your first thought? Like, did you think it was a bug in the system? Were you angry? Like, what was the emotion? Well, I mean, people with dark skin have been seeing the limitations of cameras for a while, right?

11:21So before we even get to computer vision, if you're thinking film, you know, the early film kind of technology, the way the chemical solutions were created, they were optimized to expose lighter skin. So if you see photos from back in the day, you might just see dark-skinned people and you see the eye whites and maybe the teeth. And those are actually design decisions. It didn't have to be that way. And in fact, I remember reading about the Oprah Winfrey show actually getting specific cameras from Philips. I think they were LDK series that had microchips that were optimized to actually show darker skin better.

12:03So that's like, OK, these are design decisions. And then Kodak changed their film when chocolate companies and furniture companies were complaining. You can't see the fine grain of my mahogany. Some of us got a windmill. Now they're marketing it so good it can shoot a black horse at night. If it can shoot a black horse at night, it might be able to show. So in some ways, it wasn't so surprising that in the world of vision, right, that there are these issues. But what I was surprised about was I had all the lights on. And so that's when I was like, hmm, I think there might be something more here.

12:43Something more than just technical limitations of cameras failing to pick up darker skin tones. I was curious, you know. And then it also did make me think of the book is called Black Skin, White Mask from Fanon. That's Frantz Fanon, the political philosopher. He was talking about the ways in which people of color have to mask who they are in order to be rendered visible or seen by society as well. So it almost seemed too literal. I was like, I can't make this up. So I responded with curiosity. Why is this happening? Is it because of this specific environment or is there a broader pattern here?

13:29So it really became that launching point for exploring more about computer vision, but also AI more broadly. Like, what else could be going wrong? The programs that power facial recognition are built by people, which means they are built with the same biases, unconscious or otherwise, that are already baked into society. Technology is meant to be the great equalizer, bringing access and opportunity to everyone. But if the computer systems we built can't even see us, we're just reproducing the same inequities that already exist. As a computer scientist, Dr. Joy wanted to get to the root of this coded bias.

14:09How is this happening and how widespread was it? That's in a minute.

14:21If you've spent any time building AI products or leading technical teams, you know this. Transformation doesn't fail because of ideas. It fails because teams can't move together. Enter Atlassian's teamwork collection. It has planning in JIRA, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at Atlassian.com slash TeamChanger. That's A-T-L-A-S-S-I-A-N dot com slash TeamChanger.

15:11Dr. Joy wanted to know, how are we training our machines to see? So our human fingerprints are all over. AI and automated systems. So this engineer hat comes on. What's going on? Let's debug it. Let's figure out what's happening. And so this led down a whole exploration where I started looking at a little bit of how we even train machines to detect human faces. And so it turned out they were using this concept of machine learning, right? So let's provide a data set of examples of faces and use that as a way of training an AI system to detect a human face. So I started looking at those data sets and I started looking at the most popular data sets and I started saying, oh, I think I'm seeing where the problem is coming from.

15:58So I would look at a gold standard benchmark, right? This is the one we're using to judge progress in the field. Benchmark data, as in the standard data set of sample faces. Now you'd want your benchmark to be universal, to represent the diversity of the world's population. But this benchmark data, the go-to sets of faces, did not do that. Over 70 % male, 80 % lighter-skinned individuals. And then I would look and I was like, oh, and they're taking it from images of public figures and oftentimes politicians. So I was starting to see what I was calling power shadows. So you have the power shadow of the patriarchy.

16:40When you look at women's representation, when it comes to political positions, you know, you tend to be in the 30 % or less zone so that men were showing up 70 % or more in these data sets wasn't too surprising once you start considering the source. Similarly, when you're looking at the prevalence of lighter skinned individuals, even though the majority of people on Earth would be classified as people of color, who's more likely to have their photos available online. And then there was another technical piece about it. So for computer scientists to collect large sets of data, we ain't got time and we're lazy.

17:23So what we're going to usually do is run some kind of automated system. To scrape data off the internet, basically. Absolutely. And so to get face data sets, people would create face detectors. That way you're not scraping every image, just the image that has a face, but a face that has been detected. So I also found out that the face detectors themselves were especially more likely to fail on darker skinned faces. Not only were the standard data sets overwhelmingly skewed, the tools being used to build those data sets were only further adding to the problem. So I'm like, okay, I start peeling it back.

18:02I start peeling it back. Our measures for success themselves are misleading. So it made sense to me why I would read research papers about breakthroughs that were happening with different types of facial analysis tasks. And yet I was having a different kind of experience in these one-off instances. And then, you know, scientists hat. Okay, let's run an experiment. One of one is not enough to make a generalized conclusion. So that's what led to my research. Again, I think that's a very important concept. The training data is really important, and it's not just the quantity of the data, but the diversity of the examples you're using in the training data set, whatever problem you're solving with machine learning.

18:43But it's also the validation. It's what we're testing on, right? Absolutely. Because if it's not representative of humanity and people who are going to be using the system, it's going to end up being biased. So at Affectiva, the company I started around emotion recognition, early on, we also kind of faced this idea of data and algorithmic bias. We did a lot of work in China, and we were recognizing people's facial expressions and smiles. And we got a call from one of our biggest clients in China, and they were like, this technology doesn't work. Like, you're not detecting any smiles across our Chinese kind of customers or users.

19:16And we looked at it, And we basically realized that we didn't have enough representation of Chinese population. But also, we were putting everybody in the same bucket, right? Like, as we were testing these algorithms, we weren't really, like, paying attention to subpopulations. Absolutely. I used to talk about so much of this data being pale male data, and that's destined to fail the rest of the world. So there's an undersampled majority that was implicit in some of these data sets that I was seeing. And I started to think, what does this mean outside of the face arena? Dr. Joy coined the term coded gaze.

19:54It's her way of describing algorithmic bias, and it's based on feminist theory. The male gaze is this concept of who is prioritized and whose perspective influences the choices of what we see in visual representation. And so when it comes to, let's say, art, and they talk about the male gaze in art, you would look at how women were being posed as objects of desire for a male viewer. Also, if you're thinking about the white gaze as well, Toni Morrison used to talk about this in relation to her writing, where if you're writing a book for a particular audience and you're talking about a Russian, no one's saying, well, no one's going to relate to the Russian character because of this, that or the other.

20:44You're seeing the universal humanity of that person. Yet if you're talking about somebody who was having an African-American experience, suddenly you're getting all of these questions about, well, what about the perspective of this white person? Pointing out the fact that some groups were so used to being centered that when they were no longer centered, it felt destabilizing. But that itself was showing the white gaze and this expectation of whose perspective matters or should be prioritized or even what is considered worthy. I'm seeing this in AI all over the place, right? We need more diversity of people who are building different AI technologies, but also solving different sets of problems.

21:29I agree with that. But the other part to add some nuance to the conversation that I realized was it's important to have diverse people and diverse perspectives. But I was the black computer scientist who built the Aspire Mirror that did not detect my face. Right. And so there you also it's looking at the people involved, but looking at the processes. So in this case, what was I doing? OK, I need a system that can track my face. GitHub. So I go online. I go to open source place to see if I can find some preexisting code. So just like if you're building, I don't know, you have a backyard fence project.

22:09You're not going to go chop all the wood. You go to Lowe's, you know, and you get the pre-made parts. So I'll also say, oh, what are some of the pre-made parts that we use and what's baked into that? Because if you change the people, but you don't change the processes, you're still going to be able to perpetuate that kind of discrimination. So I was realizing we had to go back to the roots even more. So the process and the people are both important. In a research project, Dr. Joy set out to quantify how accurate facial recognition technology actually was. To do this, she created her own data set of over a thousand diverse faces.

22:46Her plan was simple. Process this data through facial recognition technology that some of the biggest tech companies were making. And see how well they did. But to test these companies, she first needed to figure out how to classify her data. She tried to organize the data by race. But that got messy pretty quick. I never felt race was more constructed than when I'm going through ethnic enumeration across different countries, across different times, even looking at the U.S. census, the way labels change. Right. Talking to my Egyptian friends, you know, and they're like, so do we put African American?

23:26Do we put what? The next U.S. census in 2030 will include a Middle Eastern or North African category. But until then, people like me don't have a clear choice. It's just one example of how flimsy racial enumeration is. Some of the justifications for doing ethnic enumeration in different countries is supposed to be around discrimination. But we know Arab Americans get discriminated against in all kinds of ways where that's not even coming through. And then I learned there are all kinds of whites. You know, so looking at ethnic enumeration in European countries and also specific moves that were made to kind of paper over ethnic distinctions.

24:11Yeah. Right. And in efforts to have a sense of national unity. And so when I was going through all of this, I was like, all right, let's move away from race. Let's move away from ethnicity. And let's look at skin response. Yeah. A close objective. Right. Right. She decided to classify the data using the Fitzpatrick scale, a classification of skin pigmentation based on reaction to UV exposure. Then she used her data to evaluate how accurate facial recognition technologies really were. But so then you basically find that all the face ID systems out there, especially those kind of published by the big tech companies, were biased.

24:55More or less. So I was specifically looking at gender classification and binary gender classification. So I didn't test every kind of task they could have done. So whether it's, is there a face, right? Face detection. I was in the Wakanda era of what kind of face, you know? And so for gender classification, I did open source systems for my master's work, as well as systems from IBM, from Microsoft, from Face++, which was a company based in China that had access to over a billion face photos. So I wanted to know if this data part mattered or was it the type of data? So I included them there. And then later on, we included Amazon and Kairos.

25:40Kairos is an anti-fraud company that uses facial recognition technology. And so what we found was for gender classification, binary gender classification. Here's a photo, guess male or female. All of the systems perform better on male-labeled faces than female-labeled faces. They all perform better on lighter-labeled faces and darker-labeled faces. But then this is where it got even more interesting, just like you were saying with your own work with Affectiva, the subpopulations matter. So if we had stopped the story there, it's okay, we have this gender bias, we have the skin type bias, but then we did an intersectional analysis.