Women not included: Artificial Intelligence

0:00This BBC podcast is supported by ads outside the UK.

0:30at Whole Foods Market. Get more with BBC Podcasts wherever you listen. Be the first to listen to your favourite shows like Evil Genius, Good Bad Billionaire and You're Dead to Me with a subscription to BBC Podcasts Premium on Apple Podcasts. You can also enjoy a range of our podcasts ad-free with an Amazon Music subscription. I have a smart speaker at home which I use pretty much every day I talk to it and it does handy little tasks for me like set timers, play music and tell me the weather. It is a very useful device but for one pretty annoying quirk. Sometimes it simply just does not understand what I'm saying.

1:15While with my husband it seems to have no issues at all. We have very similar accents so I know that's not the problem. So I did a bit of investigating. And I found that in 2016, a linguistics researcher from the University of Washington found that speech recognition software from one of the largest companies using AI today was 70 % more likely to accurately recognise male speech than female speech. And look, things have improved since 2016, but it's still far from perfect. And what's been a minor inconvenience for me can have serious safety implications for others. Voice recognition software in cars is supposed to make driving safer by removing the distraction of having to physically type something in.

2:01But that's going to have the opposite effect if you are fighting against something that doesn't understand you. I'm Ella Hubber, and this is Women Not Included for Discovery on the BBC World Service.

2:18In this six-part series, we will be exploring the many ways the world has not been designed with women in mind. Today, we're talking about the tools which run on artificial intelligence that are working their way into every corner of our lives. And looking into this with me is health and science journalist Katie Silver in Sydney, Australia. Hi Ella, lovely to be here. So Katie, have you personally come across any kind of gendered problems with AI? Yeah, I use AI sometimes to transcribe interviews, for example, in Spanish. And I do notice it often defaults to almost like a male setting rather than a female one.

2:54For example, the person who led the project, la persona que lideró el proyecto, the AI will start using he when no gender was provided. Okay, that's interesting because in my very limited understanding of Spanish, it is a gendered language and the la in la persona is feminine. So if anything, I would have thought the AI translation would default to she. That makes sense. And I guess in this case, the assumption that someone who leads the project is a man always wins out. It's not just me that's noticed this. For example, in 2018, Dr. Munera Bano, who's a lecturer in software engineering at Swinburne University of Technology in Melbourne, basically took phrases in English, translated them to Turkish and then back again into English and found that the AI corrected to conform to gender stereotypes.

3:43Ella, why don't you have a go? So you can basically go to a website and try this phrase, she is the president. I've opened the website. Let me type it in here. She, she is the president. She is the president. Now change it back. Shocker. He is the president. This is even despite the fact that Turkish is a gender neutral language. So they even use the same word for he and she, and yet you still have this kind of sexist autocorrect. Very deliberate, very gendered assumptions here. Exactly. And this issue has been found for many languages a number of times since, especially for things like leadership or technical or high status occupations.

4:22There was a review paper last year that basically described this as one of the most persistent problems in machine translation. I had perhaps naively assumed that removing humans from the equation would mean greater objectivity and less gender bias in applications like this. But I'm starting to think that I was wrong. So the question now is how far does gender bias in AI go? Katie, are you happy to go away and do a bit more research on this? Absolutely. It's fascinating. Fantastic. We will meet back here soon. Bias in the data that these AI systems are being trained on. They are based on this kind of idea that women serve.

5:07physical or mental health issues of women are being downplayed. This is where we are going wrong to consider that AI can be objective. Hi, Katie. Welcome back. Hi, Ella. What have you got for me? So I've talked with a few experts who have explained not only where there is gender bias in AI, but how and why. To get to the bottom of this, I spoke to Micah Harbers, who is a professor of Artificial Intelligence and Society at the Rotterdam University of Applied Sciences in the Netherlands. And to start us off, she explained how AI is initially designed. A lot of AI systems basically work like they are being exposed to a lot of data.

5:49And then they're being trained on this data, which means these models perform all kinds of calculations to sort of recognize patterns in these data. And what they can then do is predict the outcomes or words that you as a user would most likely want to hear. So for example, if we talk about large language models like Claude and Chachypt, they predict the most likely next word. The model doesn't really understand what you're saying, but it's just predicting based on all the text it has seen to follow up. And so at this level, training, prediction, how does AI become gender biased? There are different reasons, but there is a bias often in the data that these AI systems are being trained on.

6:40Again, if we talk about large language models in the text and you look at jobs, there's more a stronger association between, for example, nurse and woman or female. And then for leader, for example, it's more strongly associated to men. So that's why if you ask for an image of a leader, you often get a white male. And obviously, AI goes much bigger than just large language models. Are there any areas that particularly alarm you when it comes to this gender bias? An important area is healthcare. There's a study that shows that AI tools in healthcare are being used to generate summaries, for example, of notes or conversations of a consult.

7:26and in these summaries health problems physical or mental health issues of women are being downplayed compared to those of men so for example pain is being downplayed you know we've we're talking about data here and what goes in but the bias isn't just the data right this is a reflection of other things so it's a reflection of inequalities in society and i think that turns out in two ways So I explained one way in which the data are biased, but there is another issue, and that is that women are less represented in data. You can see that particularly in healthcare, where there's just, has been done way more research on man's health, man's body, how they respond to medicine.

8:10So then if you use AI based on symptoms, predict what is a possible disease, then it's harder to recognize certain diseases like cardiovascular or heart diseases because there's simply less knowledge, less data about that for female bodies than for male bodies. So Katie, Micah tells us that AI trained on real world data will reflect the real world biases present in that data. It seems almost shockingly simple when you put it that way, but it wasn't something I was aware of before. I also thought the healthcare stuff was really worrying. I've done medical consults where it's written down, like I'm a health journalist, so I'm very, very clear about what's going on medically with me and the fact that it could be downplaying that, I would never have thought.

8:59I'm also wondering, women are notoriously underrepresented in tech fields, particularly AI. So would bringing more women into these fields improve things? So I did actually ask Maika this. Let me just play what she said for you. I think more women in the field can contribute as research shows that women are more likely to recognize ethical issues, particularly regarding gender bias. And they're also more likely to work on them to solve them. However, I think that simply adding more women is insufficient because in a lot of companies, they're often male dominated, but you also have a very masculine working culture.

9:39In a lot of companies, if women raise these issues, they really have problems being taken seriously. So adding women would help, but what is needed is also a cultural change. So there you go. Even when women are present, they are not necessarily listened to due to the culture. But culture is often, you know, of course, that top-down problem. It starts with leadership and the people making the big decisions. So a 2025 report analysed the boards of 141 California-based AI companies and found that 43 % didn't even have one woman on their board. Wow. I know it's crazy. Perhaps more women at the top might help to change some of that culture.

10:22Maybe, you know, you'd hope so. But I think another aspect of this is that women are also less likely to use AI tools and that may also reflect their involvement at higher levels. So maybe women need to be more involved and more active in using AI, but I will say it's a double edged sword. I went back to Micah later on because she had found a new study concerning women who do use AI at work. New research has come out that shows that women who use AI are more often seen as incompetent, more than men. And men who use AI are more often seen as pragmatic. Stop it. That's ridiculous and depressing.

11:11It's a lose-lose situation. Damned if you do, damned if you don't. Something else that jumped out to me talking to Micah was this idea, the AI, it can only make what it gets, you know. There is no capacity for nuance here. Even if we correct the research data to include more women or different nationalities, the problem is the original AI models may already be built on biased data. So, for example, while adding that more diverse data helps, it doesn't automatically fix the problem. So if an AI system is built on biased data or assumptions, it can still produce biased results even when you feed it better information.

11:50It's almost like a bit like trying to correct a wrong recipe after the cake is already baked. Very good point. You know, it's kind of like it's trying to train an AI to recognize all the colors of the rainbow, but then only ever showing it blue and yellow. It's not going to find red, you know.

12:34Choose one entree and two sides. Shop smarter, not harder at Whole Foods Market. Get more with BBC Podcasts wherever you listen. Be the first to listen to your favourite shows like Evil Genius, Good Bad Billionaire and You're Dead to Me with a subscription to BBC Podcasts Premium on Apple Podcasts. You can also enjoy a range of our podcasts ad-free with an Amazon Music subscription.

13:02you're listening to women not included for discovery on the bbc world service where we explore the many ways the world is not designed for women this time we're talking about tools run on artificial intelligence we've discussed how the ai technology we use is a direct reflection of our unequal society if biased data goes in biased results come out now let's take a look at the real scope of the problem, Professor Pragya Agawal is a behavioural and data scientist and author of several books exploring unconscious bias around race and gender. When she began looking into bias in AI, it was in facial recognition software.

13:41We found that facial recognition apps are tools that had huge amounts of sexism but also racism and biases at the intersection of both. For example, research has shown that the error rates for some of the facial recognition apps, as much as 34 % for darker skinned women, but even less than 1 % for lighter skinned white men. Also, there was a big company that launched a card, a credit card. And there was a huge amount of like uproar about it and discussion because we found that the card algorithm was actually giving women lower credit limits than men, even when they had the equal score. And so this was automating financial discrimination.

14:24Okay, so this is it's pervasive, it's everywhere. And one area you've talked about extensively is voice assistants chatbots. So data from 2024 shows that up to 8 billion people are using voice assistants many times, several times a day. And we find that most of the voice assistants are feminized. So they have feminine names and feminine voices. And they are based in this kind of idea that women serve, that women are subservient. Data from 2025 finds that actually 50 % of human machine exchanges were verbally abusive, and 44 % were actually using sexually explicit language. So there was a report done by UN, which was titled, I'd blush if I could, which analyzed what kind of response was built in or programmed into these voice assistants if they were confronted with sexually explicit or abusive messages.

15:20And they would often be either flirtatious, they would go silent, or they would say, for instance, one of the voice assistants said, I'd blush if I could. There was nothing built in them or programmed in them which retorted or challenged this kind of behavior. And so we see these entrenched norms, the culture that these voice assistants are being built in is being transferred from the real world into technology. It's interesting because I think a lot of people think this is just a design quirk, but it seems like it is really perpetuating real life biases. It absolutely is. And it's a two way process.

15:59So on one hand, it is reflecting our societal kind of gendered biases and norms and stereotypes, but it is also reinforcing them and exaggerating them as well. For instance, while women chatbots serve, men instruct. And so are there instances where people don't like to hear women's voices as voice assistants? Yes, there was a study done in Germany. So I was looking at a satellite navigation system and the people there did not like a feminized voice because they did not like a woman giving them instructions or they don't trust the instructions that are given by women. So this area of chatbots, voice assistants, it's perpetuating an implicit bias.

16:41But I'm also really interested in more tangible and immediate effects on women. And one of the areas you've talked about is how AI has been used as a hiring tool. What do we see here? Yes, the recent data shows that up to 70 % of organizations are now using AI tools for pre-screening, which looks for keywords. And there was an example in 2018 when one of the big, big companies launched this automated hiring algorithm, which was supposed to actually eliminate the human biases which go into hiring. It was found that very quickly, this algorithm started to penalise CVs, which had the word women in it.

17:22Even with anonymised CVs that didn't have names, anything, if they said they were part of a women's basketball team, they didn't consider that CV. It has a huge real world implication for how women are hired, what kind of opportunities they get. And this is where we are going wrong to consider the AI can be objective or neutral, but it's not neutral. Thank you to Pragya Agarwal. So we heard from Micah before that AI trained on certain data reflects the biases in our world. But Pragya is adding here that it also amplifies and scales them. And that can be implicit bias, like the way we treat female coded machines like chatbots or explicit impacts on women in areas like hiring and finance.

18:08This is a big scale problem. I was absolutely shocked by the idea of people using sexually explicit language for their bots. Maybe I'm sheltered, but I mean, what? I guess it's like almost like this early stage of now, you know, so many people with these AI partners or girlfriends. Yes. Oh, I didn't even think about AI girlfriends. Well, they're submissive as well, by all accounts, I think. And unlike with chatbots, the girlfriends are explicitly trained to be submissive. And so that is concerning. It is concerning. I've actually deliberately optimized my CV for AI. So I basically put my CV into a large language model and said, can you optimize this for an ATS or applicant tracking system that uses AI?

18:54So then your CV is basically tweaked almost like my AI meets their AI and eventually a human might hire another human. There's something incredibly sad about that. But what about, did you find that it removed any gendered language if you had it? I'd have to check that. I didn't notice it, but it's an interesting idea. I mean, hiring practices are a really good example of the immediate way women are affected by AI bias. But I have seen much more than this. So one study from the Berkeley Haas Centre for Equity, Gender and Leadership found that about 44 % of the AI systems they analysed showed gender bias.

19:37And this was in, yes, hiring, but also advertisement, college applications, surveillance and policing, even public benefits like eligibility for food stamps. So basically at every step of the way, that's terrible. What about how gender intersects with race? 25 % of the systems exhibited both gender and racial bias together. If you're a woman and you're of a different ethnicity, you're really, really staffed. And AI can affect inequality much more broadly too. Towards the end of last year, I went to Bangkok for the release of a report by the UN Development Program. And it did extensive research and modelling into all the ways that AI could widen inequality.

20:22And the people who were most likely to be left behind were poor women. How so? So the report found that women tend to be in jobs that are twice as exposed to automation. So I think things like textiles, back office processing, retail. Also, women are potentially less likely to use AI, particularly rural women in poorer countries. So they're not getting that early mover advantage of using AI. This is what one of the report authors said. AI affects the most vulnerable person. And in most of our countries, that person is a young woman. If you protect her, you protect the entire population. But how do we protect her?

21:02Let's talk solutions now, Katie. Professor of Artificial Intelligence and Society, Micah Harbers, told me that there is no quick and easy fix here. But let's start with the most glaring issue, the data. If one of the problems is biased data going in, can we just start with better data? It depends on the AI application. For more narrow AI applications, for example, voice recognition, voice recognition systems used to be worse for female voices than for male voices. And that is because these systems were trained with more examples of male speech. So then you add more data and it starts to perform better for different kinds of voices.

21:47OK, so we can in theory and have, it seems, improved in some areas like voice recognition, where we can just put more representative data in. But then what about areas like large language models, like chatbots, which are trained on the entire Internet's worth of data? There is much harder. You could, in theory, say, add more data, but how do you do that? Or select the data that is non-sexist, but this is such huge amounts that is really impossible. So in those cases, you can mitigate sexism in AI models somewhat by training a model based on data from the Internet and then adding rules on top of that.

22:29An often used technique for that is reinforcement learning from human feedback. But still, when is something non-sexist? That's really complicated. Do you want outputs of AI models to reflect current inequalities? Do you always want to make it 50-50 %? So if you talk about a leader, a housewife, a breadwinner, how do you represent it as a man or as a woman? Or you could argue, well, in order to compensate for the inequalities of society, you should actually try to correct that. And for things like large language models, before we can even make a change, there first needs to be a cultural shift in order for people to actually address these things.

23:12Exactly. Because then it's also the question, are companies that currently develop these models going to want to invest in that? Because it will require a lot of efforts. And yeah, personally, I don't think the big tech companies are currently willing to invest. What else needs to change? Currently, AI is being very under-regulated. The largest regulation of AI in the world is now in the European Union. They are introducing the AI Act. What kind of things does it outline? It depends. It makes a difference between AI systems with high risk, minimal risk and limited risk. And depending on the risk category, it gives certain regulations.

23:58But for example, if you interact with a chatbot, as a user, you have the right to know that you're speaking to an AI. As far as addressing the gender bias in AI goes, what kind of action are we seeing here in the regulation? For the high risk level, which are systems, for example, that decide or give advice about access to things like employment, education, public services, or that do credit scoring. Companies applying these kinds of systems have to do all kinds of checks. Gender bias is not explicitly included, but for example, is the system that you're using fair? But it's also hard to check whether companies are actually doing this.

24:43And we'll really have to see how that will develop. Are you optimistic about this? Well, I think it's a very good step in the right direction. Personally, I don't think it's enough. Thank you to Maika Harbers. So fixing bias isn't just a technical data-led challenge, although that is a huge part of it. It's also about stronger guardrails in regulation. But it's not quite there. We need deeper cultural shifts, it seems like. What do you think, Katie? Cultural shifts and kind of, as she was saying, I guess company buy-in. And if you've only got that small amount of board representation of women, I don't know, this kind of like tech bro culture in Silicon Valley just seems to have kind of gotten worse in recent years.

25:28The other part that I thought was quite interesting was when it comes to regulation. And you've got this fundamental issue where countries don't really want to miss out. The EU is perhaps one of the few jurisdictions that might be big enough to kind of move the dial on this. But looking at smaller countries here in Australia, there's been attempts to crack down on AI. But at the same time, you don't want to stifle investment. It's a tricky line to walk for these governments. We've heard about the EU from Micah there. But I know you've been looking into this from a more global perspective, Katie.

25:58What kind of changes are experts calling for out there? Particularly when it comes to developing countries, they're talking about targeted programs. for women's digital and AI literacy. They're often missing out, particularly because they may not have internet on their phones, those sorts of things. And also women's groups also need to be consulted in the model development. There is some good news in the area that I've seen. Thank God for that. What is it? So the Women for Ethical AI platform was launched by UNESCO in 2023, and it aims to directly support governments and organisations in implementing a global ethical framework for artificial intelligence.

26:33And amongst other things, basically, they're trying to ensure that women are equally represented in the design and use of AI. That's good. That's great news. I have seen one other tangible solution. It's called the Universal Trusted Credentials Initiative. It's in Singapore. And it uses different data points to approve loans. For example, if they consistently pay their utilities bills or they maintain good relationships with suppliers to rate businesses that would be previously excluded from conventional lending processes. And this has the potential to increase loan approval rates for women-owned businesses, as well as rural enterprises.

27:14That is all we have time for today. As we've heard, artificial intelligence is not some neutral, all-seeing technology. It is a mirror reflecting back and even amplifying the inequalities already present in our world. And from hiring tools to financial decisions, the consequences for women are far from trivial. But there is also momentum for change in new regulation and ethical frameworks, although it will take some time to get there. A huge thank you to Katie Silver for exploring all of this with me. Thanks, Ella. It's been fascinating. Women Not Included for Discovery on the BBC World Service was presented and produced by me, Ella Hubber, with additional production support from Elliot Prince.