The Dangerous Bias Shaping the Future of AI

0:28So good, so good, so good. Women are being erased from AI technologies. These technologies, which are already having a transformative effect, for better or worse, on all of society, are being designed almost exclusively by men. While a relative absence of women and various other underrepresented groups is nothing new in the technology sector, the problem and the harm it could cause are about to be turbocharged as governments and companies worldwide rush to adopt AI and bring it ever deeper into our lives. That's the problem we're talking about today on The World, the Universe and Us from New Scientist.

1:02I'm Dr Penny Sarchet. And to discuss this, I'm joined by New Scientist editor Kat Delange. Kat, you've just been at a conference about women in science at the Royal Society in London. What was the general atmosphere there? I'd say the atmosphere was pretty mixed. So the conference was about women and the future of science. And it coincided with 80 years of women being allowed to be fellows of the Royal Society, which is something to celebrate. A lot has changed in that time. So we heard from some really well-established scientists who kind of experienced some awful things at the beginning of their career that we wouldn't expect people to experience now, which is great.

1:37But having said that, the bar was pretty low. And for instance, we heard about one scientist, Hertha Ayrton. Do you know about her? She worked on the electric arc. She was a physicist and published this pioneering text in 1902 and was put forward to become a fellow of the Royal Society. I think she was the first one to be nominated, the first woman to be nominated. And they got in a big bind about it because they were like, we can't possibly admit her because she's married and married women have no rights. So, no, so we can't do that. So, you know, when you think about that, it's pretty depressing.

2:10And a lot of the topics were, you know, depressing. And I think the conversation I'm about to have was one of them. But having said that, it was absolutely packed full of brilliant scientists, inspirational leaders, you know, who many of whom just happen to be women. And so it was also very inspiring. So depressing and inspiring in equal doses, I would say. So shall we focus then on what we mean about a lack of women in AI? Because that feels quite a new subject. But it's not this kind of now familiar idea that AI algorithms have been trained on data sets. And those data sets have biases in. So we know, for example, that potentially if you're asking like a health AI or an AI health questions, And especially as a woman, you might not get very good advice because there's just not as much data on women's health.

2:56That's not the problem that we're talking about this time. No, exactly. I think that's quite familiar. We know these data sets are biased and so the models that they're trained on end up having the same biases. And I think people are quite familiar with that. But no, what this session at the conference was looking at was something much more fundamental, which is the fact that these transformative technologies that are supposed to be changing society through and through are very predominantly being designed by men. Yeah. And we know that's a problem. I mean, it sort of goes without saying. I wish it went without saying, but that is a problem.

3:31And we've seen that throughout all sorts of technologies in the past. And we know what happens when technologies are developed for one group of people by another group of people who might not understand what their needs are. Yeah. Shall we drill into that a bit? Because I'm just thinking some people may not see the problem if the men training AI, designing AI are just great at their jobs, does it matter that they're men? But we know from the long history of the gender data gap that it does matter if only men are doing the research or the training. Can you explain? Yeah, there's some really quite well-known examples now.

4:04But actually, I was thinking about this this morning when I was in the gym, and it's one thing that annoys me. Every time when I go to the gym and I try and do a chin-up or something, the bar is too big for my hands. It's designed for men, so men tend to have bigger hands. And so it's just this really annoying thing that every day, every couple of days, I'm just like, oh, I can't do this thing that I'm trying to do today because it's been designed for somebody else. That's a sort of very superficial example, and it's just a little bit annoying. But often it is a matter of life or death. So things like drugs, trials, crash test dummies, and a lot of...

4:38Car seats and cars just don't hit women in the right place. Cars in general have been designed with these dummies that are, so they're all based on Reference Man. I don't know if you know about Reference Man. Remind me. So Reference Man was developed for, I think like radiation to assess the risk of radiation. And Reference Man is a sort of young white man who weighs like 70 kilos and is about 170 centimetres tall. And that has become the reference for so much of the research that's been done on things like safety. Yeah. And then if you think about cars, it's a great example because if cars are developed for reference man or the crash test dummies are based on reference man, women who tend to be smaller sit differently in the car because it's not it's too big for them, essentially.

5:23And then they're sitting in a less safe position and all of the other safety features are designed in a way that doesn't work for them either. So it can become a matter of life or death. And then safety clothing for people doing jobs where their clothing or their uniform, bulletproof vests, for instance, for police officers, again, developed for this sort of reference, which is a man. So we've got all these examples where you just can't think about the needs of somebody else if you don't bring in that experience right at the beginning at the design stage. It's very trivial compared to cars, but one example I think about a lot is phones, which have just got bigger and bigger to match the average male hand to the point where it hurts my thumbs to use, and I drop it all the time.

6:04Yeah, a lot of women say, I'm always dropping my phone. What phone have you got that you don't drop? Oh, no, it's not the phone. It's just that it's too big. So that's technology, but that's a very physical example. How does that translate then to this kind of gender data bias gap in AI? What does that look like when things aren't being designed with women in mind? All you have to do is think about the kinds of things that AI is being designed for. It's being designed to help with health care, diagnosing diseases, coming up with drugs, education, how we raise our children. We think about the kind of jobs that AI is impacting, replacing.

6:44And then you have to think about are women and men different in these aspects? Do we have different bodies? Yes. Do we do different jobs? Do we have different roles in society? Let's not get into why, but the fact is that, yes, we do. And women tend to have more of the caring roles. I think that just answers the question. Like, we have different needs, and these technologies are being developed that will impact all these elements of our life. And if our needs aren't being thought about, then we're going to get into the same situation as the other kind of problems that we've already heard about.

7:18I think one of the speakers on the panel was saying that she's been working with a startup using AI to tackle endometriosis, which is a great example of, you know, if it's probably quite unlikely that a lot of men are thinking about endometriosis because it's not something that affects them. Maybe a silly example, but think about sex robots. Okay. Well, if we're developing, you know, AI enabled sex robots, I think the needs of men and women are probably going to be quite different. And the statistic that kept coming up was that only 2 % of venture capital funding goes to women, to 2%. Is that AI specific or is that all kind of investment?

7:56No, that's in general, but it's just the kinds of startups and the kind of technologies that women are thinking of developing are not getting the funding. Kayak gets my flight, hotel and rental car right so I can tune out travel advice that's just plain wrong. Bro, Skycoin, way better than points. Never fly during a Scorpio full moon. Just tell the manager you'll sue. Instant room upgrade. Stop taking bad travel advice. Start comparing hundreds of sites with Kayak and get your trip right. Bad advice? You talking to me? Kayak. Got that right. One thing that might surprise some people is that this isn't like the rural society.

8:40This isn't like science where it started off men and it's just gradually getting better, maybe not as quickly as we'd like. is actually, in some respects, seems to be going backwards. Yeah, so all of the panellists, and there was also a man on the panel, I should say, who was quite vocal about this. David Leslie from the Turing Institute said that there's definitely been a regression and that we've heard all about Donald Trump's executive order that targets so-called woke AI, recommending that the National Institute of Standards and Technology in the US revise its AI risk management framework to exclude any reference to misinformation, diversity, equity and inclusion and climate change.

9:28So it's this kind of... Throw that one in there as well. Yeah, just pop it in. Yeah. And one of the panellists, Ruman Trowdhury, who used to work on ethics at Twitter before Elon Musk took over and then it turned into X, was saying that actually this whole idea of woke AI doesn't come from Donald Trump. It actually came from Silicon Valley in the first place. And that is really the atmosphere. It is very misogynistic. And it's not, as you say, a question of things were historically sexist and misogynistic and they're improving actually is getting worse, has got worse over the last few years. I actually caught up with Ruman Chowdhury after the conference and asked her a bit more about her experience from Twitter and what she makes of the whole situation.

10:13There has been an absolute backslide in the time that I've been in tech where we did see it ramp up to be a little bit more inclusive. You know, it was, we could at least talk about it. And now it has regressed significantly. Just as an example, I will tell you, a woman founder colleague of mine said that a VC specifically told her that he liked her company, but he didn't invest in women. No explanation. But the fact that he thought it was okay to tell her. And, you know, to the point on a previous panel, maybe it's kind of better this way. Because before they weren't giving us money anyway. Yeah.

10:46But they just pretended it was other things and invented new hoops for us to jump through. You know, I will say, like, I quoted that 2 % of VC funding. That number was from, like, 2018. That number is not a new thing. It's been abysmally low. It remains abysmally low. I don't even think it's really gone up or down. It's actually remained the same. So it's still 2 % of women getting VC funding. And if you look at sort of where it's distributed, women tend to get funding in sort of lifestyle and healthcare and wellness. And men get to do the quote-unquote hard, hard technology things. And as a woman founder raising to do one of the quote-unquote hard technology things, it is literally a wall you run into.

11:25So this is clearly a mess. AI models have already been trained almost exclusively by men on biased data. What can we do about it now? Well, there are a few things that came up in the conference that gave me some hope. One thing that Ruman Chowdhury mentioned is a kind of narrative around AI, which is everything is an emergency. AI is coming for our jobs. AI is going to become intelligent and kind of, you know, take over the world. And so, you know, it's this idea that the house is on fire. And when the house is on fire, you don't stop to go and get your valuables. You just it's life or death you need to get out.

12:08And so her analogy there is that if within this narrative, you know, people will say, well, yeah, you know, I'd love to I'd love for there to be more inclusion. I'd love to to think about what women want. But, you know, the house is on fire. Like, we've got to go, guys. We've just got to get this stuff out there. And all of those things become extraneous and you can just drop them. And she says that's a false narrative. And so her approach is to not engage with this narrative and just sort of slow things down and not kind of succumb to those sorts of pressure tactics. There was a lot of talk about the next generation.

12:39If we want to get younger people interested in AI, in AI for the greater good, AI that's actually going to help society and have more positive impact. we kind of need to reframe the incentives around it and the economic incentives around it because at the moment everything is very based on making as much money as fast as possible that's the narrative and we're not really thinking about how can we actually employ these tools to make a bigger difference and there was this one really great quote uh from one of the panelists rachel cold who said you know we need to develop this technology for eight billion people not eight billionaires that's great who by the way are all men he said so to serve those eight billion people how do we adapt the ai that we have that's based on these biased models so the sentiment from the panel was that actually a lot of these models probably are just too biased and we need new models right um chowdhury's co-founded a not non-profit called humane intelligence which helps companies to make ai systems more accountable and fair and so there are these things that we can do to to try and sort of address fairness and you know we've seen what's happened at twitter she worked on accountability there and that got a raise so we're going from a sort of baseline of zero um and also i think it's exciting to think about are there completely new models that are focused on bigger problems like climate and society and more kind of um human-centric kind of applications and approaches.

14:11So this is quite a major point then, isn't it? Is that we need to actually go backwards, go back to basics and almost start again? Well, not start again, but do it better. I think so. And, you know, if you go right back to the beginning, you look at what is the definition of artificial intelligence and what is the definition of intelligence that we're even talking about here? Because, you know, the origins of AI come from a very influential meeting in the 1950s at Dartmouth College in New Hampshire. And yet again, that meeting was a meeting of men who came up with this definition of intelligence in the first place.

14:48So on definitions, OpenAI in their charter, they make an interesting link between AGI, they almost define it in terms of economically valuable work. Yeah, that's right. And this is something that Wendy Hall, who was chairing the session mentioned, which is that, you know, open AI define artificial general intelligence as AI that surpasses human cognitive abilities in most economically valuable tasks. And by definition, that actually excludes most of the so-called feminized tasks in society, which aren't actually seen as having economic value or having very little economic value. So these very definitions of intelligence and artificial general intelligence are, again, based around sort of milestones that exclude the kinds of things that women tend to do.

15:34It's interesting because who decides what is economically valuable work? And women on average in society do a vast amount of work that contributes to GDP and productivity that is unpaid. And that's very well established. So I guess it's carrying over when you have men deciding what is valuable in the economy, and then you've got men deciding what AI is going to do for that economy, you start seeing the scale of the problem. Exactly and I think it's a really good example of you know why I just don't think we need to ask this question of why does it matter whether people are not represented in the design of these tools because if we're taking a society that sort of has baked in problems and baked in biases and then we're reinforcing them through this new technology then it's only going to expand those problems and if we want to try and apply the new technology to tackle a lot of these issues women have to be in the room right from the beginning.

16:26thanks so much Kat for joining us thank you for having me I'm Penny Sarté and this is the world, the universe and us do follow us for more episodes and we'll see you soon bye

16:56From craveable sauces to satisfying textures, they're designed to keep you going without slowing you down. So put that fork down. Try the new wraps today in app or at order.sweetgreen.com. Available at participating locations only.