Best of Design Matters: Dr. Joy Buolamwini

0:00Debbie Millman: This episode of Design Matters originally appeared in July of 2025. It wasn't as easy as saying, okay, let's make more inclusive data sets. And when we have more inclusive data sets, we'll have more accurate facial recognition. But accurate systems can be abused. And so the analysis had to be not just how well does the technology work, but what kind of technologies do we want in society in the first place?

0:31Debbie Millman: From the TED Audio Collective, this is Design Matters with Debbie Millman.

0:38Debbie Millman: On Design Matters, Debbie talks with some of the most creative people in the world about what they do, how they got to be who they are, and what they're thinking about and working on. On this episode, computer scientist and digital activist Joy Boulamwini talks about her career and about facial recognition technology in the airport. You actually have the right to opt out, but most people don't know. And it's not surprising because you go there and they say, step up to the camera.

1:12Debbie Millman: Healthcare can feel complicated. That's why Optum uses technology to connect the people and processes that make healthcare easier, more affordable, and more effective. we're making it clearer for you to know exactly what your benefits cover and to help you better manage your health we're coordinating care between your doctors and your technology we believe better simpler health care is always possible that's healthy optimism that's optum visit optum.com to learn more investing with schwab is like spending a saturday at a great farmer's market you can fill your reusable tote with a bit of everything maybe you go for some free-range self-directed investing.

1:52Debbie Millman: Or perhaps you pick up a few farm-fresh trades while you peruse. You can even get help from a dedicated advisor. That's full-service wealth management. Mix, match, and change your mind whenever you want. Because at Schwab, you can invest your way. No matter your goals or appetite for investing, Schwab has everything you need, all in one place. Visit schwab.com to learn more.

2:20and manage the timelines, while our human creative director made sure it all made sense. Easy. Create your own AI agent today on Monday.com.

2:32Dr.

2:33Debbie Millman: Joy Bulamwini is a computer scientist and a poet of code who uses art and research to illuminate the social implications of artificial intelligence. She founded the Algorithmic Justice League to create a world with more equitable and accountable technology. Her MIT thesis methodology uncovered large racial and gender bias in the world's largest and most powerful technology companies. Dr. Joy's journey is depicted in the critically acclaimed documentary Coded Bias, of which she is now on a five-year anniversary world tour. And the documentary sheds light on threats artificial intelligence poses to civil rights and democracy.

3:19Debbie Millman: And she is also the author of the bestselling book, Unmasking AI, My Mission to Protect What is Human in a World of Machines. Dr. Joy Bolamwini, welcome to this very special live episode of Design Matters at the WBUR Festival. You know, sometimes dreams come true, and I'm living one right now. So I'm so happy to be here. Thank you. Thank you. Let's begin with baby joy. Baby joy. Okay. You were born in Edmonton, Alberta, just as your father was finishing his PhD, and you've described yourself as a daughter of art and science. What do you remember most from that unique collision of your mother's paints and your father's pipettes?

4:10That's a great question. In the book, I say how my mother asked questions of colors. My dad asked questions of cells. And in that exploration, I started asking questions of computers. And so for me, I literally grew up with art and science as companions through my parents. And in Oxford, Mississippi, that meant going to my dad's lab and feeding cancer cells, looking at squiggles on computers learning later these were graphs and so forth flow cytometry and that kind of thing and then my mom I just thought every weekend you go to art galleries and pitch paintings I didn't realize that's what really was going on and so I grew up with those worlds and it felt very much like an invitation to be creative whether through scientific inquiry or artistic inquiry, or for me, playtime, right?

5:02You know, so I think that was a true gift that I see now in the way that I do my work as a poet of code.

5:12Debbie Millman: You spent your early years in Ghana before moving to Oxford, Mississippi. What were the values that shaped you most in those formative years of in-betweenness? I think the first thing is my first language being chi. And so I used to have all kinds of accents. I code switch often. But when I first came to the United States, and I was in Oxford, Mississippi, they actually put me in speech therapy. They didn't realize English was my second. They should ask more questions. You know, and maybe I really did need to be in there, but we can deconstruct that later. I recently returned back to Ghana last year after 30 years.

5:58And I don't look that old, but it was three decades since I'd been there. And it was the best homecoming I could have imagined. In what way? One of the things that I noticed is all of my relatives were looking at me and I was looking at them, but none of us wanted to be rude or stare. So we would like look at each other through reflections and like just catch an eye. So it felt like I was time traveling. When I would see my aunts and uncles, I was like, Oh, that's what my brother's gonna look like in a few years. Or Oh, that thing I thought my dad was just my dad, I see it in all of his siblings.

6:35So that kind of way. And then also connecting with my young cousins, I think they range from about age eight to 26. So I'm on the older end of the cousins within a more immediate family. And so it felt like a very warm embrace, a long awaited embrace.

6:55Debbie Millman: Given you were so young when you left Ghana to go to Mississippi, did your family share stories about Ghanaian excellence or legacy that helped shape your own sense of possibility? Yes, I'm third generation PhD. So my grandfather was a dean of a school of pharmacy, Kwame Nkrumah University Science and Tech. And so we have some Ghanians in the audience here as well. They went to good schools too, even if it wasn't science and tech, which is all solid. And so growing up, it wasn't like, oh, be excellent or that kind of, it was just around us as this is our family legacy. There's this value of education.

7:41There's this curiosity. Got a library card very early, and we used to live within walking distance from the Oxford Public Library. So my brother would roll me there in a wagon and he introduced me to the boxcar children and Hardy Boys and that kind of thing.

7:59Debbie Millman: I have this vision of him driving you back in the wagon with stacks of books around you. Well, it was so fun because right now the documentary Coded Bias were on world tour. And we just did a screening in Oxford, Mississippi at that same public library. And on the stage, I used to watch magic shows and puppet shows. And then they were showing Coded Bias. And my nieces were there. My brother was there. One of my childhood neighborhood friends was there as well. So kids I grew up playing with and eating honeysuckles and, you know, that kind of deal. So it was really a nice full circle moment. And then a friend I had in Oxford, England, where I later studied, she had now become a professor at Ole Miss.

8:42And she was there with her baby and her dog and her husband. So it was this kind of culmination of many strands, different elements of my life there in Oxford, Mississippi. So the Southern Twang had come out just a little bit, you know, to remind where I came from.

8:58Debbie Millman: But it was all good. It was a good time. There's a lot of kismet in your life, and we'll get to that. As a teenager in Memphis, you were building websites to cover your basketball team's uniform costs, writing Java games, pole vaulting and skateboarding. What did technology represent to you then as you were forging a path as an athlete? Yeah, well, it was a means to cover my dues. So instead of paying for the sporting dues, it was, OK, let me make a social media, a social network for the track team. Let me make a website for the basketball team because I'm going to be warming the bench. Right.

9:42I might as well contribute in some other kind of way. My first website was really for my Latin club. Of course it was. As one does. And I spent so much time on that website. It had animations and I was coding in Flash. And it was, I used to always say it was the top 10 in the country. So probably meant it was number nine, but whatever, you know, so that was a little bit of encouragement. And then I was so fortunate now that I look back at it. At the time when I was in Memphis, Tennessee, I had the opportunity to take three different computer science classes as a high schooler. And part of the reason was because of my teacher, Miss Jill Connell.

10:26And she had this one room kind of schoolhouse where on each three walls and then her wall was on the main wall, her desk. She would teach the first version of computer science, the first class. She'd do the second one and then you'd have the AP class. So I circled those walls, right, for three years. And that really gave me a great basis. And I was looking back at some of the data. And I think at the time I took the advanced placement computer science test, only five kids in the state took it. And so that I was in that specific school with a young teacher eager to make it work, you know, even if it meant having to come up with three different lessons in that same period was really, really fortunate.

11:18And I used to spend the nerd I am, I would spend my lunch period there coding Lego robots. And later when I would come to MIT, I would work in the lifelong kindergarten group that helped create and establish the Lego Mindstorms as well. So I always shout out to Jill Connell and follow her Facebook posts and her kids now and all of that.

11:43Debbie Millman: Well, you just said that you were a tech nerd or a computer nerd. Definitely. But you were also a coder, a creative and an athlete. So how did how how did your friend group view you? I was always in between spaces. And so for my parents, as long as I got straight A's, I could do what I wanted. Right. So I was like, cool, I'll get the grades. And let me try basketball. Let me try track and field. I didn't actually want to do cross country. I was recruited to cross country. and then I remember running my first mile throwing up in the coach like I think you got this do I though so that's how I ended up doing three different sports in high school but I really fell in love with pole vaulting I know the most of all of those things and didn't you have aspirations to go to the Olympics I mean if you don't do it you've got to do it right so I could evolve higher You know, I was at you don't start doing a full pole vault.

12:48You start in the grass, then the sand, then you move to the pit. So before I'd even cleared anything in the grass, I was already saying Olympics. Right. So that just tends to be my mentality to aim high.

13:02Debbie Millman: You attended the Georgia Institute of Technology for your undergrad degree. Yellow jackets. But you first considered majoring in international affairs and biomedical engineering. You eventually transitioned to computer science. What did you envision doing professionally at that time? Oh, I mean, there were so many things I wanted to do. My first career aspiration was to be a professional skateboarder. Did that come before or after pole vaulting? Before pole vaulting. So I started skateboarding when I was around 11 years old. I watched the Goofy movie and then the Goofy movie they made skateboarding look so cool.

13:41So I got like the cheapest skateboard you could ever get from Walmart. It had this purple cobra on the back. Later, I got a much better skateboard, but that's how I started. So when we moved from Oxford, Mississippi to Memphis, Tennessee that summer, I started learning how to skateboard, which meant scraping my ankles a lot, mainly.

14:02Debbie Millman: So you wanted to be a professional skateboarder, but you were majoring in computer science. So that's when I was 12. And then the reason I decided not to pursue that career path is I was introduced to the gender pay gap. So I was looking at skateboarding competitions and I was looking at what the men got paid and what the women got paid. So the women's prize was like what the 13th place guy got in terms of first place. I was like, you know what? I might need to try something else. So that's when I got off the skateboard path for a little bit. I returned to it later in grad school. And do you still?

14:41Every so often. These aren't the right shoes, but if you have the right shoes, I'll try to pull out a 180 boneless. I can still do that from time to time.

14:49Debbie Millman: In your third year at Georgia Tech, you were working with a social robot named Simon. Yes. And your assignment with Simon was to see if you could have the robot engage in a social interaction with a human. You created a project called Peekaboo Simon. Can you talk about that project? Yes. So the idea with Peekaboo Simon was to program the robot to do a simple turn taking game. Right. So you cover your face. you uncover your face, you say peekaboo. So that's what I was doing. The problem is peekaboo doesn't work if your partner can't see you. My robot was not detecting my dark skin face. So I'm looking at my code.

15:36I'm like, I think the code is right. So what's wrong? So I didn't have that much time. And I had a light skin roommate, red hair, green eyes, pale skin. Oh, you're perfect. But so I used her to test it and it was working on her. So when it came to do the demo, I just made sure somebody with light skin tested it. And so this was around 2011 when I was a junior at Georgia Tech.

16:04Debbie Millman: Your senior capstone project at Georgia Tech was an experiment with the Carter Center. And you traveled to Ethiopia to do that work. And there you piloted a health data system in Ethiopia that eventually reached 17 million people. 17 million people. What type of data system and how did you do that? Yes. So that year, I think I kept going around saying, I want to make an impact in African nations as kind of my little tagline when people would ask, what do you want to do? I'm Joy, I want to make an impact in African nations. Oh, that's cute. So I was doing that. And there was an invitation at Georgia Tech to come to some, I think it was a tech and health summit at 8am.

16:55Like 8am? I don't know about this, right? But that year, I had my show up, speak up stand up. It was just like my little mantra for the year. So I said, okay, I'm going to show up. So 8am, I show up. I'm going to speak up. So I speak up, I talk about my little African nations thing. And it turned out there was an epidemiologist from the Carter Center. And they were looking for a software engineer to help them with some of their global health programs. And in particular, they did a program called Maltra in Ethiopia. And so that was their malaria and trachoma program. And they'd been doing it for a few years.

17:34And now it was time for monitoring and evaluation. So they wanted to go in and take surveys and things like that. The problem is their paper-based surveys had some limitations, especially when you're trying to put in GPS coordinates. So if you get one of those digits wrong, right, you might be in a lake instead of the space you're supposed to be in. So at that time, open source was very helpful because Android had been released. And you now had these Android tablets we could program. So the opportunity was to replace their paper based way of assessing the effectiveness of these programs into a tablet, right into this digital form.

18:16And so that's where I came in with my computer science skills. So I was like, hmm, surveys, you could view it that way. Or we're transforming data collection, right? So we can have real-time insights on the trachoma program at the time. So that's what ended up becoming my senior capstone. And we looked at their current system, transformed it to something that had taken about 30 to 90 days we could do in one day. And so that's why they wanted to continue to adopt it. But I realized global health wasn't for me because while I was in the field, I was making up songs about pit latrines and hygiene and things like that.

19:01And that just was not the culture of the global health people at all. I might need to try something else. But that's yeah, that was one of the formative experiences for me, because at that time, I was at Georgia Tech and I was gaining all of these technical skills. And yes, I was working on robots and so forth. But I also knew those robots weren't going to be in people's lives in a major way anytime soon. It might be sooner now, but then a decade ago, it wasn't happening. And this desire to make an impact with the tech skills I was gaining at Georgia Tech, that's why that year I kept saying, OK, I'm going to speak up, you know, show up, stand up, all of that.

19:45So stood up in Ethiopia.

19:48Debbie Millman: You then applied for and were awarded a Fulbright Fellowship, which took you to Lusaka, Zambia. Yes. There you taught young Zambians to make mobile apps. Yes. You partnered with iSchool Zambia and worked on the Zidu Pads. Yes. Did I pronounce that correctly? Yeah. Yeah. Tell us about that. Yes. So after the experience in Ethiopia, I started asking some questions. Like, sure, I came in with this technology that I had actually coded most of it in my childhood bedroom in Memphis, Tennessee. So it was during summer break. And because of that, I made all of these assumptions about the technology and the context that weren't true.

20:33Like for what, for example? So, for example, I assumed that the internet speeds would be comparable. And so one of the features of that data collection process was you could upload it to the cloud and then the people wherever they needed to be could get the data. But because it was very slow, we ended up realizing we needed to download the data actually on SD card. So I needed to, here I am under mosquito net, like in Ethiopia, changing the code because we hadn't accounted for the difference in internet speeds and things like that.

21:07Debbie Millman: Is it true that the students thought that you weren't an instructor? They never thought of you as an instructor? They still don't think I'm an instructor. I take it as a compliment now. So given that had happened in Ethiopia, when I had the opportunity to do a Fulbright fellowship, I was thinking, okay, is one thing for me to come in? Yes, I am of the continent, but I'm also in some ways a Westerner kind of parachuting in. What would it look like to actually equip people to create the systems, the tools that address their own local context? And so that's where the XamRise initiative came to be.

21:46And that was my Fulbright project. And then along the way, we met local skateboarders and poets. So there are a lot of side projects. I also couldn't be directly paid due to Fulbright rules. So we would barter certain things. So with the founder of iSchool Zambia, I bartered a house for the time I was in Zambia to then do training and be compensated for the apps that the students made through the training. And then they provided office space for that. So this whole creative bartering because of the constraint of not being able to be directly paid. We did an Indiegogo campaign and raised money to get the laptops for the students to learn how to code because I asked.

22:35Debbie Millman: What's an Indiegogo campaign? Oh, does anyone? Good question. So it was an alternative to Kickstarter. So crowdfunding platform. And I wanted to raise, I think we needed$10 ,000 or something like that. And so that allowed us to get enough laptops to then work with the students. What I had noticed with the Zedupads when I was talking to the founder of iSchool Zambia, I said, well, who developed the software? Who developed the apps? And they're like, oh, well, we used Eastern Europeans. And I asked why. They're like, oh, we're not able to find the talent we need here. So I asked, OK, if I did a program to train, would we then be able to have apps on the Zedupad?

23:23And so that's what the Zamrise program ended up doing so that when it actually came out, they could say it was also built by Zambians as well.

23:34Debbie Millman: So you've talked about the poetry scene and the skateboarding community you found in Lusaka. What was the connection for you between creation, culture and code? And for me, it was like a way of life. And even now I ask myself, how can I live poetically? And how can I be expressive? And so while I was living in Zambia, and I was making friends and so forth, I was like, okay, yeah, what do people do for fun? Where do you go? I remember going to a slam poetry session about subsidies. It was really interesting for me, right, just to see how people were exploring their creativity and things like that.

24:16So none of my friends were surprised when they saw the skateboard shots while doing the Fulbright. And we're also delivering for iSchool Zambia and things like that. So I never have felt that I've had to separate those worlds. You can have the impact. You can do it with a little flair, hopefully a little bit of style and connect with the artistic communities that are there. And skateboarding is another type of art.

24:44Debbie Millman: Did you teach anyone in Zambia to pole vault? No. See, pole vaulting requires some equipment. You need the pit. You need the poles. The poles are very expensive. Even at skateboards, I learned was tricky. So they didn't actually have a skate park at the time. And I was asking them where they were getting their supplies from and they were getting it from South Africa. You then went to Oxford, where you made history by proposing the first Rhodes service year. What gave you the audacity to reimagine what the Rhodes experience could be? Um, so all the work I did in Ethiopia meant that I had the qualifications to get into a global health program.

25:32So I applied. And the Rhodes Scholarship covers two to three years. And so I'd used my first year to do a master's in learning and technology. And the second year, I wasn't exactly sure what I wanted to do. So I applied to global health and I got in. And I realized I really don't want to do global. I told you the culture, like my pit latrine wraps were not hitting. It had not improved. The situation had not improved since then. And so I wrote a 40 page proposal for this year of service and shared the type of visa I would need to have. I really went in and how it would build on the work I had done in Zambia and so forth.

26:18And so basically, when I sent to the Rhodes Trust, they said, don't get your hopes up, kid. We don't change much here. Okay, I'm gonna do it regardless. And you guys should get the credit, right? Anyways, they decided to agree to it. And so that became the first official Rhodes Scholar year of service. But many senior scholars told me they'd been doing years of service unofficially well before me.

26:43Debbie Millman: Have there been official years of service that have come after you based on that change? There have been, actually. And they tended to be other scholars who are really entrepreneurial. And I remember talking to some of the members of the trust, and they were curious why not as many people had applied for it afterwards. And I was thinking, I mean, the profile of a Rhodes Scholar tends to be those who have mastered the academic game, but aren't taking many risks. So I'm not so surprised that if you had the option to get an MPP or an MBA from Oxford, pretty much guaranteed, or do a year of service, most people would go for the guaranteed option.

27:26So it tended to be the crazy entrepreneurs who were open to that kind of risk anyways, right, who were doing it at the time.

27:34Debbie Millman: What did you do during that service year? Oh, so that service year, I started something called Code for Rights. And so Code for Rights was thinking through the curriculum that had been created in Zambia and actually adapting it to an Oxford context and looking at what kind of problems or issues might we apply that process to. And we ended up focusing on sexual assault on campus and creating this first responders app as well so that survivors would know what their options were. Thank you for doing that work. It's an area that I have particular interest in. You were then awarded a research assistantship at the MIT Media Lab from Ethan Zuckerman.

28:20Yes.

28:20Debbie Millman: The director of the Center for Civic Media. Now, is it true that he told you that if what you're thinking of making already exists, go elsewhere? Oh, so that was one of the Media Lab professors. That wasn't Ethan. Our group was kind of the everybody come sort of group. We had a big wide open table. And every Thursday when we met, we would always have random guests and some regulars, you know, retirees where they would come to our table as well. So Ethan was very open. But the summer before I was in the area for my friend's wedding and I thought, let me go check out the Media Lab in case I apply there.

29:03And so I met another professor who was like, yeah, if what you're thinking of already exists, this isn't the place for you. So what was really interesting to me about Ethan's group was we were in the future factory, but we were a bit of an oddball because we wanted to talk about problems now. Right. The Center for Civic Media. So we were doing things like creating systems that would allow kids to track school lunches to see if the government was delivering on what they had promised, the Promise Tracker app, which was deployed all over Brazil. So it was a group that was a bit out of the mainstream.

29:43But then in the time I was there, 2015 to about 2022, there was also quite a bit of change happening in the U.S. where people were saying, wait a minute. But maybe the research we're doing, maybe we should be thinking about its immediate real world impact, right, with the different elections and so forth. So I remember in 2016, November 2016, after Donald Trump was elected, it was really interesting in my own circles because I'm from the South, right? So some people are like really rejoicing and other people are devastating. And I'm seeing all of that happening. And at the Media Lab, a lot more people were asking questions of, okay, if this is happening, should I really be painting walls with my smile as my project?

30:35Or maybe there's something else I should be doing. And so I remember the day after the election, people were suddenly looking to our group, the oddball group that's talking about problems now, became the mainstream. And that's around the time I was also starting to explore the work that became the Algorithmic Justice League. And so that was a transformation from being at the margins in the future factory to being a group that was viewed as a leader within the future factory.

31:05Debbie Millman: You signed up for a course that changed the trajectory of your life. Yes. Science Fabrication. Yeah. This was specifically focused on building fantastical futuristic technologies. What did you begin to build? Yes, this is where I started to explore this idea of shape-shifting, right? And so we mentioned earlier, I'm from Ghana. I was inspired by stories of Anansi the spider, trickster spider that could also shapeshift. And I wanted to do the same, but I had a six-week deadline. And so instead of shifting my shape, right, I decided to shift my reflection in a mirror. And so in that process of figuring out how do I make it look like there's a mask on my face through a mirror, that's when I started to experiment with computer vision.

31:57So first you have to detect the face. So once I got that going, I started putting different people's faces on my own. So Serena Williams was one, looked like the greatest of all time. Then I thought, okay, this is kind of giving me theme park where you have the cutout hole and you put it through. Wouldn't it be cool if when I moved, it moved just like that? Anyway, so when I was moving, it's moving. So to do that, I needed a camera. So I put a camera on top of the mirror. And then I needed software that would actually follow my face in the mirror. So I went online, downloaded some software that was supposed to help me do that.

32:37But it wasn't working. Just like my robot, my peekaboo robot wasn't working. I was like, I think something might be off. and at the time I was experimenting it was around Halloween time and I had a white mask for a Halloween party so when it wasn't working I started just experimenting with things in my office I even drew a face on my hand held it up to the camera and it detected the face on my hand so that's when I was like okay anything is possible so I reached for the white mask and before I even had it all the way over my face it was already detecting the white mask so here I was at MIT, this epicenter of innovation that I dreamed of coming to since I was a little girl.

33:22And I'm in white face to be seen coding in a white mask at MIT. And so that was when it switched from can I shapeshift like a Nazi the spider to hold up, side quest, what is going on here?

33:38Debbie Millman: You've said that while the white mask episode was disheartening, you didn't want people at the time to think you were making everything about race or being ungrateful for rare and hard-won opportunities. And you felt that speaking up had consequences. What kind of consequences did you fear? Retaliation, being blacklisted, being kind of marginalized with the way my research was perceived. And so I remember even when I started exploring the research, grad students warned me, they're like, you know, XYZ, they try to study bias, didn't go well. It's like, these are the bones of grad students past.

34:25That shadowy place, it looks like studying discrimination. You're right, like places the light doesn't touch for a career. So I was highly discouraged from doing this sort of work because it was touching on bias and discrimination. So that was part of one thread of being discouraged. But another thread of being discouraged was like, this involves AI. AI involves a lot of math. I'm like, and? Right? You know, engineering background, all of that. So the math wasn't an impediment to me, but others' perception of the type of work I was capable of, I was also seeing that come out. in some of other people's perceptions.

35:08And so for me to actually pursue this research was going against all of the wisdom that was being passed down. My supervisor, he wasn't against the exploration, but he was really practical. You spent a year working on a completely different project. This is a two-year program. You want to do a new project halfway through. That might be difficult. Maybe this is a side project, right? And so I think all of the advice was well meaning and everything they warned me about did happen.

35:46So I guess this is where it helps to be stubborn. But also, I think the other thing for me was, even though the people around me at the time didn't quite see the vision it still felt important to me and I have to commend the media lab for creating a space where I could explore a sandbox even if they didn't understand the shapes I was building they're like you do you right we don't know what you're doing but figure it out and there aren't there are many spaces that I wouldn't have even been able to explore that at all.

36:21Debbie Millman: You stated that in some ways you went into computer science to escape the messiness of the multi-headed isms of racism, sexism, classism, and more. And those signs indicated otherwise, you wanted to believe that technology could be apolitical. Oh, 100%. So what changed your mind about speaking up? When I saw the reaction to the 2016 election, it was kind of a all hands on deck moment. And this questioning of what can I do to make a difference in the world? And so when I got to the Future Factory, there were two missions. One, get the PhD, family legacy, third generation. I only applied to, I didn't really want to go to grad school again.

37:16Like, you know, so I kind of did the, well, I'll apply to one place. And if it doesn't work out, I did try. So I applied to one grad school. It worked out. So this was the price for getting it. Did you really think you weren't going to get in? To the media lab? Yeah. And you never know for sure. I thought I had a good chance, but it wasn't 100%. And if it didn't work out, I could continue my entrepreneurial dreams. So I was not I was not heavily invested in getting in. In fact, I was going to be working on an entrepreneurial project until I got a call from my father. It's like, remember who you are?

37:54Oh, no. I was like, Rhodes Scholarship Fulbright Fellow Daddy, this has to count for something. Right? You know, it is not a terminal degree. Okay. Okay. All right. All right. That is true. You know, so I applied to that one place and got it. Yeah.

38:14Debbie Millman: You think you'll ever go back to that entrepreneurial project? It's a whole other area of questions that I'd love to ask you about. But again, the time is slipping away. Yeah, I mean, I was working on a hair care technology company, a CTO of an ed tech company, the hair technology company, we were about a decade early on hair analysis. But now if you go to my Avana, we can actually do a hair analysis where you get a camera, it looks at your hair strands, it can tell you so many things about it, but also give you unique product recommendations. So I was a CTO for that before I went to Zambia. So I had a lot of side projects.

39:15Debbie Millman: And change your mind whenever you want. Because at Schwab, you can invest your way. No matter your goals or appetite for investing, Schwab has everything you need, all in one place. Visit schwab.com to learn more. Healthcare can feel complicated. That's why Optum uses technology to connect the people and processes that make healthcare easier, more affordable, and more effective. We're making it clearer for you to know exactly what your benefits cover. And to help you better manage your health, we're coordinating care between your doctors and your technology. We believe better, simpler health care is always possible.

39:52Debbie Millman: That's healthy optimism. That's Optum. Visit Optum.com to learn more. This Monday.com ad was created by a team of people and AI agents. The agents wrote the copy and managed the timelines, while our human creative director made sure it all made sense. Easy. Create your own AI agent today on Monday.com. There's information bombarding us everywhere all the time on podcasts, social feeds, and all of your group chats. I certainly feel it too. So what actually needs our attention and how can you trust that you're getting the full picture? At TED's big conference, TED 2027, you can go deeper into breakthrough ideas that matter right now.

40:35Debbie Millman: Research reshaping how we understand ourselves and technology that's opening doors we never thought possible. It all unfolds with curated discussions, unforgettable parties, and a community of thoughtful, ambitious people. You might just meet someone who changes the course of your life. Right now, you can secure a First TED membership that's just for newcomers. Don't wait. It's only available until September. Join us at TED 2027, March 1st through 4th in San Diego, California. out. Learn more by going to ted.com slash t27.

41:08Debbie Millman: Your thesis, which you titled the Gender Shade Project, demonstrates the priorities, preferences, and biases of those who create code and the algorithmic bias from companies, including IBM, Amazon, Microsoft, and more. With over 3 ,400 citations, you exposed how AI facial recognition systems had 100 % accuracy for white male faces and near coin flip results for dark skinned women. Some labeled Michelle Obama as male. Others were labeled as gorillas. What were some of your other findings? Yes. So there are a whole combination of findings when it comes to the mislabeling of faces. So with the Gender Shades Project, as I started looking at how computers read faces, I was asking three different kinds of questions when it comes to facial recognition technologies.

42:13So the first question is, is there a face? This is face detection. So when I'm putting on a white mask, that's a face detection fail, right? Another kind of question I like to say you might ask is what kind of face? So what's the gender of the face? What's the age of a face? Maybe what's the emotional expression on the face? And so that's what I focused on for my master's work with gender shades. I was looking at companies guessing binary gender. And in that, we tested a number of different companies. So in the case of Microsoft, it was that there was perfection for one group, the lighter males, the pale males, affectionately called, right?

42:57And it wasn't so great for other groups like darker females. But it was also interesting because we tested like a company from China and we found that it actually had best performance on darker male faces. And this was really important because in that research, we were doing intersectional analysis, borrowing from Kimberly Crenshaw's work on discrimination, right along multiple axes. I was like, Oh, this is being applied in the legal space. But maybe there's something for computer scientists to learn. So instead of just looking in this case at gender, I also started looking at skin type as well.

43:36And so it wasn't just the story of, okay, it works better on male faces than female faces, which was the overall trend, or it works better on lighter faces than darker skin faces. But when we did that subgroup analysis, right, lighter males, lighter females, darker males, darker females, that's where we got the stark contrast you were just mentioning. And Microsoft was the good results, right, With IBM, their error got between their best performing group, lighter males, and their worst performing group, darker females, the highly melanated like myself, was around 34%. So those were the findings that really got me to start exploring what are other ways in which computer vision has failed.

44:22Right. And so you have the Gorilla Gate example that you just brought up with Google Photos, where what I now call an evocative audit, people in the wild, regular people were interacting with these systems and seeing issues. Google fixed the problem not by making a better system, but just by removing any label of gorilla. So gorillas were also not labeled gorillas, just to be extra safe. I don't know if that's quite the solution. But so we've seen an evolution of different approaches to addressing some of these misclassifications and mislabeling that comes on. But as I was also doing the work, I realized even if these systems were perfectly accurate, whether we're going from guessing the gender of a face, which that's difficult, right?

45:13How does a person identify? To figuring out the unique identity, right? With facial identification. If you have perfect facial identification and cameras everywhere, we have the infrastructure for a surveillance state that tracks your every move, where you go to worship, who you see at night or in the morning, whatever your preferences are, right? Where you go and protest. And so that was very interesting tension for me to hold as I was doing this research, because it wasn't as easy as saying, okay, let's make more inclusive data sets. And when we have more inclusive data sets, we'll have more accurate facial recognition, but accurate systems can be abused.

45:59And so the analysis had to be not just how well does the technology work, but what kind of technologies do we want in society in the first place?

46:13Debbie Millman: Amazon dismissed your findings. How did you find the confidence to stand by your data? It was interesting because we were talking about my parents earlier, right? So we did our first research paper, Gender Shades. And when it came out, we had tested IBM, Microsoft, Face++, this company in China. Our second paper tested those companies, but also included Amazon and another company. So when the first paper came out, it was actually fairly well received, right? Like, okay, bias is an issue. We're working on it. Or some companies like we've been known and we've been working on it. Okay, okay. But this was the approach.

47:01So when the second paper came out, the reaction was surprising to me and to my research team. because imagine having a test where the answers have been known for a year and still failing and you're a company as big as Amazon and you're going for a billion dollar contract with the Pentagon which was happening at the time when this research came out and so we're showing their competitors that are also bidding right are performing better than they are on this computer vision task. And so I was surprised at the Amazon attack because the pushback I had expected from the first paper didn't come. But this is what the grad students of the past, right?

47:46You know, those, the places you shouldn't go and the bones, right? This is what they had warned of. And oh, yeah, it came down hard. I remember I was, I was spending my birthday in Switzerland, But I was at a conference. I was at the World Economic Forum. And the paper was going to be released about a week later. And so I remember flying from Zurich to Honolulu. I'm in this big red jacket. And I come, okay, I need some Hawaiian shorts, right? Right. And I had sent the pre results to all of these reporters and Amazon was claiming they had never seen the paper, even though I had a you know, I had the email evidence.

48:30Maybe it hit spam. I don't know. Benefit of the doubt. Yeah, maybe. Right. So they did all of these delay tactics. And then when the paper came out, they had a corporate vice president, Dr. Matt Wood, come out and try to discredit the paper. So much so that we had the research community rally around us and say, this paper is valid and important because the types of issues it addresses, we have to all be thinking about as AI becomes more entrenched in our lives. So we don't get better as a field by ignoring the problems. We have to actually confront it. And so this coming from a Turing Prize winner, you know, our Nobel Prize for computer science, and a former principal AI researcher of Amazon really had a major impact.

49:22What I found interesting, though, is when the book Unmasking AI came out, it was an Amazon best books pick after all that time. And even as this was going on, I separate individuals from institutions. There were people inside the company sharing what they were able to in terms of the opposition that was actually inside the company about the use of facial recognition. And I am happy to say that all of the U.S.-based companies we audited stopped selling facial recognition to law enforcement.

50:02Debbie Millman: Thank you. Thank you. Very important work. Let's talk about the other side of facial recognition that you mentioned and the whole sort of way in which we're now being monitored. How are you? What's your feeling on all of this? I think we have to be very careful now. I was just looking at the recent announcement with Sam Altman and Johnny Ive and thinking about what future computing devices might look like. And if you look at companies like Limitless.ai, Limitless.ai, which is a pendant that's always on, always recording. I'm thinking, where's the consent piece? So some years ago, Google experimented with Google Glass, right, where you could record and so forth.

51:00And people didn't like it. They're calling it glass holes and things like that. I was reading on LinkedIn, an experience from Ali Miller, who's this top voice in AI. And she shared how she was backstage having a conversation with somebody and said, Oh, we said so many good things. I wish we had written it down. And the person said, Oh, don't worry, I have an AI recording it right now. And so that level of violation of privacy consent is going to only increase if we don't push back. One of the ways we've been pushing back with the Algorithmic Justice League is our Freedom Flyers campaign. So you have the TSA has been putting facial recognition at the checkpoints in airports.

51:50They have plans to expand it to 400 airports. and you actually have the right to opt out. But most people don't know. And it's not surprising because you go there and they say, step up to the camera.

52:04Debbie Millman: There's no signage that says you have the right to opt out. It's usually very difficult to see. So we've been doing surveys to see if people are actually finding the signs. And oftentimes people don't even see the signs. Last year, we talked to the Department of Homeland Security. We showed our data. And so they committed to the spring actually adding language on the kiosk to tell people to opt out, which is a good step. But we really have to think about the power dynamics. You're in line. I don't know if you're early or late to the airport, but usually I have time pressure. You might not have time pressure, but I have time pressure, right?

52:40There's social pressure, people behind you in the line. There's financial pressure. I don't know how much your tickets cost, but the prices seem to keep going out, right? So you're doing all of that. You finally made it to the airport. Now they got the scan thing going on. And a TSA officer tells you to step up to the camera. They don't tell you have the right to opt out. You've seen everybody else in front of you step up. The signs are there. They're usually turned around in a different language. We've documented all the things. They managed to put the part that says you opt out right where the pullout piece goes.

53:15might have just been a coincidence you know we've been documenting some of these things right so it's not designed so you're actually aware and one thing they know how to do at airports is design signs right no weapons formed against TSA shall prosper it's a close to$15 ,000 sign real ID you need to have that the signs they want you to see TSA is hiring you will be able to see those. So, you know, there's all of that going on. So the big thing we've been doing is one, letting people know they have the right to opt out because we're getting the story that, well, no one refuses it. They don't know they can.

53:55But the other thing we've been doing is also collecting travelers' experiences. And some of them have been really disheartening. I remember reading one about a son, the son of a man, and he had autism and his son was scanned against his will, You know, I told my parents to opt out because of the work I do. And I remember them being humiliated in line. Right. And so when I sat down with Secretary Mayorkas at the time, I shared that story. but the stories of many other people who talked about intimidation tactics or just being dismissed as like, well, we have cameras all over the airport anyways.

54:38I just came back from Greece and this was the very experience I had when I was crossing over when I said, oh, I want to opt out. Smug look, they're like, we have cameras everywhere. This doesn't matter. But this actually does matter. and why it matters is if we don't opt out and if we don't resist then that narrative persists that people want this which they'll use and they claim right now that they only put the facial recognition on some of these checkpoints and that they don't have it on the overall system but all of this is just one line of code from being mass surveillance if we don't actually resist and so that's why it's important even if you haven't opted out in the past that you continue to do it in the future.

55:24And so then going back to that situation where you're backstage, and someone has their AI device, right? If we want a consent culture, we have to practice a consent culture and also demand a consent culture. Because what we're what we will see, if that doesn't happen is listening devices everywhere, which we are already set up to do, right with our cell phones, but even more ubiquitous and even more intimate.

55:53Debbie Millman: Aside from not providing consent, what else can we do to combat mass surveillance in our society? Yeah, I think this is where there's individual action and collective action that's needed as well. At the end of the day, it does come down to laws, right? And so seeing, for example, the EU AI Act, where they have a restriction on the use of live facial recognition, shows that there's alternative ways. Now, it does depend on the administration. What we saw with facial recognition, we still don't have federal laws around it. But we were able in 2019, 2020, to get quite a bit of traction at the municipal level and also at the state level.

56:41What's going on in Congress right now, and now at the Senate, is there's a push to say any kind of regulation of AI at the state level is going to be on pause for a decade. That's huge. That's saying the protections that people fought for, because at the federal level is a bit harder, but at the state level, we can get traction. All of that gets rolled away. So that's what we should be putting our eyes on right now.

57:11Debbie Millman: Given the current administration, what are your hopes and fears about what is possible from a collective public response? I think my fears right now, especially with the gutting of people in the administration, or people at the IRS, people at different federal agencies is this assumption that humans are so easily replaced. So I think about this is why stories matter so much. It's not even the power of AI, but the power of the stories we tell about AI. So I think about ANETA, the National Eating Disorder Association. They bought into the hype, right? AI is going to be better than humans. It can do call center work, etc.

58:00And at the time, they had their workers were trying to unionize, you know, for the reasons you do, better pay, better conditions, all of that. They decided to fire all the workers and replace them with AI. So you had a headline, right? Netta fire staff replaces with AI. I don't think it was even a week before you had the next headline, which was the chatbot that the people were replaced with had to be shut down. And that chatbot was actually giving people with eating disorders advice that's known to make eating disorders worse. And so here we're looking at the actions of FOMO, fear of missing out.

58:38We have to adopt these AI solutions and also how easy it is to dismiss the work you're not proximate to, to think it's so easily automated. So now coming back to the government, right? We're seeing this clear out of humans and then we're going to see this rollout of AI systems as a replacement to the humans. And then we're going to find out, oh, we didn't need humans after all. And we're going to be backtracked. Surprise, surprise.

59:05Debbie Millman: Tell us about what you're currently doing, because you are on the front lines of helping society manage all of these issues and protecting society from the issues. Talk about what you're currently doing. I'm going to ask you one last question, and then I'm going to have to close out. Oh, it's been so much fun. I know. Okay. So what I'm currently doing is going global. I mean, not that the U.S. isn't a great place to be right now. Sorry, allergies. So last year, Oxford University, where I did the Rhodes Scholarship and did a master's in learning and technology and didn't do that master's in global health.

59:50Debbie Millman: Two master's is enough, right? Not for my day. You don't have me out here, out here working. so they reached out and they started a new institute focused on ethics and ai and they had a fellows program so they asked if i'd be an inaugural fellow i was like oh what does it entail well tell us what you want to do when you want to do it and what resources you need is that a blank check sound like a blank check i'll take i'll take the blank check and so long story short what ended up happening is i proposed doing a five-year anniversary world tour of the documentary Coded Bias. Coded Bias premiered at Sundance in 2020.

1:00:31And when I watched it in 2025, it was as relevant as ever. We're talking about issues of bias and discrimination. AI still persistent and people see it more now, right? Because if you are applying for a job, you know, you might not get it. If you're applying for a loan, all of these areas now, Now AI is becoming a part of the decision making system. If you're applying for college, if you get the right medical diagnosis, if the transcript of what happened when you were talking to your doctor is even accurate, all of this is now being mediated by AI systems. And not to even talk about, I mean, there's so much to talk about regarding the current potential situation of the administration looking at social media to allow you into the country or not.

1:01:22Absolutely. And you've already had issues in the past, right, where context collapse happens. So someone might say, she's the bomb. Was that bomb? Was that a bomb threat? Right? Like, this is real. And that is, there are two parts, right? One when it's mistaken and one when it's intentional.

1:01:40Debbie Millman: Right. As well. We've talked so much about your science, your innovations, the way in which you're working to protect society. We haven't talked very much about your poetry. And for those that are interested in seeing more about Dr. Joy, you can go to poetofcode.com. Before we close out the show, you have some really beautiful poetry in your book, Unmasking AI. You've done extraordinary performance art. I'm wondering if you can share one poem from your book with our audience today. Only one? We don't have time for more? I'm up if you are. I don't know about everybody else in the production team.

1:02:22Okay, I'll do two if they permit me because I want to end on a happy note. But before I end on the happy note, that's the note I think we should all hear at the moment. So this poem is called Unstable Desire. It's in the epilogue where I talk about my roundtable discussion with President Biden. Unstable desire. Prompted to competition, where be the guardrails now? Threat in sight will might make right. Hallucinations taken as prophecy. Destabilized on a Midland journey to outpace, to open chase, to claim supremacy, to reign indefinitely. Haste and paste control altering deletion. Unstable desire remains undefeated.

1:03:12The fate of AI still uncompleted. Responding with fear, responsible AI beware. Prophets do snare, people still dare. To believe our humanity is more than neural nets and transformations of collected muses. More than data and errata. More than transactional diffusions. Are we not transcendent beings bound in transient forms? Can this power be guided with care, augmenting delight alongside economic destitution? Temporary band-aids cannot hold the wind when the task ahead is to transform the atmosphere of innovation. The android dreams entice the nightmare schemes of vice. Poet of Code, Certified, Human Made.

1:04:05Debbie Millman: Thank you. We said you're going to read another one. So this is one I wrote in the chapter called Intrepid Poet. And this is when I start flexing the storytelling muscles, because for so long, I felt that if I let my artistry out, my research might not be taken as seriously. And it was such a relief to find that it was moving from performance metrics to performance arts that really moved this message around the world. So to the Brooklyn tenants and the X-coded, resisting and revealing the lie that we must accept the surrender of our faces, the harvesting of our data, the plunder of our traces.

1:04:49We celebrate your courage. No silence, no consent. You show the path to algorithmic justice requires a league, a sisterhood, a neighborhood, a podcast, hallway gatherings, sharpies and posters, coalitions, petitions, testimonies, letters, research and potlucks, dancing and music. everyone playing a role to orchestrate change to the Brooklyn tenants and freedom fighters around the world persisting and prevailing against algorithms of oppression automating inequality through weapons of math destruction we stand with you in gratitude you demonstrate the people have a voice and a choice. When defiant melodies harmonize to elevate human life, dignity, and rights, the victory is ours.

1:05:51Dr.

1:05:52Debbie Millman: Joy, Abrol, and Weaney, thank you so much for making so much work that matters. And thank you for joining me for this very special live episode of Design Matters at the WBUR Festival. To know more about Dr. Joy, you can read her book, Unmasking AI, and see more of what she's doing on her website, PoetofCode.com. This is the 20th year we've been podcasting Design Matters, and I'd like to thank you all for listening. And remember, we can talk about making a difference, we can make a difference, or we can do both. I'm Debbie Millman, and I look forward to talking with you again soon. Design Matters is produced by the TED Audio Collective by Curtis Fox Productions.

1:06:35Debbie Millman: The interviews are usually recorded at the Masters in Branding Program at the School of Visual Arts in New York City, the first and longest running branding program in the world. The Editor-in-Chief of Design Matters Media is Emily Weiland.

1:06:52Debbie Millman: Investing with Schwab is like spending a Saturday at a great farmer's market. You can fill your reusable tote with a bit of everything. Maybe you go for some free-range, self-directed investing. Or perhaps you pick up a few farm-fresh trades while you peruse. You can even get help from a dedicated advisor. That's full-service wealth management. Mix, match, and change your mind whenever you want. Because at Schwab, you can invest your way. No matter your goals or appetite for investing, Schwab has everything you need, all in one place. Visit schwab.com to learn more. Monday AI agents took over my work and I absolutely love it chasing deadlines, writing status reports updating stakeholders agents handle the daily grind now they live inside Monday.com so they see the full picture my work, my team, the whole company and I don't have to worry about the data it's safe, which means I'm free to focus on the big stuff, knowing everything runs smoothly in the background it's completely shifted the way we work create your own AI agent in minutes on Monday.com There's information bombarding us everywhere all the time on podcasts, social feeds, and all of your group chats.

1:08:02Debbie Millman: I certainly feel it too. So what actually needs our attention and how can you trust that you're getting the full picture? At TED's big conference, TED 2027, you can go deeper into breakthrough ideas that matter right now. Research reshaping how we understand ourselves and technology that's opening doors we never thought possible. It all unfolds with curated discussions, unforgettable parties, and a community of thoughtful, ambitious people. You might just meet someone who changes the course of your life. Right now, you can secure a First TED membership that's just for newcomers. Don't wait. It's only available until September.

1:08:38Debbie Millman: Join us at TED 2027, March 1st through 4th in San Diego, California. Learn more by going to TED.com slash T27.