#142 Arjun Subramonian: The Intersection of AI, Diversity, and Ethics

0:00They were using block lists that contain words like gay and lesbian. and you know we kind of were like hmm this seems like this is like very meta like we're here we're talking about like bias and equity but the acontextuality of that is very stunning right it's like how can you just take a sentence and see the word gay and say immediately that's going to be a bad thing we are a very decentralized organization so of course i'm here talking to you about the organization but i wouldn't consider myself a leader or an organizer or anything anyone can become an organizer at any point anyone can take a step back anytime they want it's all about realizing that there is some kind of initiative that you want to put on, or there's a program that you want to help out with, or there's just something you want to learn more about.

0:40Hi, I'm Craig Smith, and this is Eye on AI. This week, I talked to Arjun Subramanian, a PhD student researching AI fairness and ethics. Subramanian discusses the work of Queer in AI, a decentralized community promoting inclusion of LGBTQ plus people in artificial intelligence, research, and deployment. Subramanian explains how queer and AI runs education programs, advocates for equitable policies in AI, and makes visible the diverse challenges queer people face in tech and academia. While optimistic about improving AI, Subramanian notes achieving true inclusion requires expanding awareness of intersectional identities and listening to marginalized voices globally.

1:41Overall, Queer in AI aims to shape AI positively by surfacing harms, fostering representation and embracing flexible epistemologies. I hope you find the conversation as enlightening as I did. This episode is sponsored by Salonis, the global leader in process mining. AI has landed and enterprises are adapting, giving customers slick experiences and the technology to deliver. The road feels long, but you're closer than you think. You see, your business processes run through many systems, creating data at every step. Solonis reconstructs this data to generate process intelligence, a common business language.

2:31With process intelligence, AI knows how your business flows across every department, every system, and every process. With AI solutions powered by Solonis, enterprises get faster, more accurate insights. a new level of automation, and a step change in productivity, performance, and customer satisfaction. Process intelligence is the missing piece in the AI-enabled tech stack. Search Celonis, C-E-L-O-N-I-S, to find out more. So my name is Arjun Zubramonian. I use they, them pronouns. I just started the second year of my PhD at the University of California, Los Angeles. and my research focuses on the intersections of machine learning on graph structure data as well as things like algorithmic fairness, bias, social justice, and ethics.

3:25So looking at the ways we're doing machine learning on things like social networks but in a way that's very inclusive and equitable for everyone that's being impacted by these algorithms. I grew up in Cupertino in the Silicon Valley and I think when I was growing up I was just like absolutely inundated by technology surrounded by a lot of very famous companies. In fact, walking to school, I kind of walked through Apple campus adjacent areas, and there were self-driving cars being tested. And so it kind of was, for better or worse, has pushed in that direction effectively of going into computer science, of being very interested in artificial intelligence, intrigued by these things.

4:01I have to ask, were your parents' academics or in the tech space? They were also in the tech space. So my mom is a software engineer at Oracle, and my dad used to be an electrical engineer at Texas Instruments and then he just recently moved to a startup. So they're all very involved in tech. It's kind of like a tech family, tech neighborhood. And I went to UCLA for computer science. I was like, this stuff is cool. But then at the same time, this is when I was coming, figuring out my identity, especially with relation to queerness. And I think like when you're marginalized in certain aspects, you start realizing that there are ways in which technology doesn't work for you.

4:33There are a lot of other communities that are also being potentially negatively impacted by the technology that we're building. And so as much as I was interested in artificial intelligence, I wanted to bring in that equity aspect as well into my work. I was doing a lot of diversity and inclusion organizing. Like, for example, at UCLA, we organized with some other amazing people, the first LGBTQIA plus hackathon in the U.S. and at the college level. And then I was doing some other advocacy work just around like the UCLA computer science department. And that really fed my desire to add equity to my research.

5:06right like I was thinking about all these ways in which I was working directly with people who were unable to afford even just applying to college or who felt unsafe in their computer science classes and I was just like I need to this is very important to me so I'm going to bring responsibility into my work and yeah and then I noticed Queer and AI existed actually I didn't I'd never heard of NeurIPS until 2020. NeurIPS 2020 was my first virtual conference and I attended the Queer and AI workshop actually that's the only reason I went to NeurIPS and I was just amazed by everything that was being done.

5:36It was like a community education and empowerment event. We were learning about what it's like to publish academically as like a transgender researcher. There was stuff, there was like a social space as well. Like I was meeting all these really cool queer academics who were like role models. I realized I hadn't had many queer role models before. And then, yeah, that was like the last thing I needed to be like, I'm going to go into grad school. I'm going to go become an academic. I really love this work. I love this community. And then the rest is just like this history. Like I'm now organizing with Queer and AI and I still talk more about what Queer and AI does, but that's a little bit about my background.

6:09Yeah, that's fascinating. When you talk about equity in technology, when you were first thinking that you want to get involved in that, I mean, certainly everyone understands the inequality of access to technology. But But what's interested me about the affinity groups in general in AI is I assumed initially that they were just social clubs. I mean, you're black, you come to a conference, there's an organization where you can meet other people of color, or you are gay, there's Queer and I, AI. It's like a networking club, you meet other people. But then the more I've listened, and particularly on the bias issue, I interviewed Tinmut Jabru for a piece that I did for the Times, and I began to see that this is a field that's critical to the future of humanity, and it's being developed and controlled primarily by white men.

7:14and that there are biases in the data that might not be recognized. Certainly, there are all those early stories that we've all heard of AI systems doing things. But maybe there are areas that one group, like white males, are unaware of or wouldn't see. And so the technology doesn't develop in that direction. So when you started saying that you wanted to work on equity in tech or in AI, were there specific issues that you could see? Or were you assuming that there were issues? Or were you talking more about the access question of who gets a computer, who gets educated? Yeah, that's a great question.

8:00I think it definitely is a combination of the two. I think when I was growing up in the Silicon Valley, like it's very much a bubble. Of course, even there, I realized there are dramatic differences in access to technology. If you're outside the Silicon Valley, it's a completely different place. The Silicon Valley, everyone goes to school. They have access to science classes. They have access to actual computers. You can go study AI if you want to. In fact, a lot of people study it from a very early age. And so studying at UCLA, you meet people from so many different backgrounds. And if people are interested in these things and they want to study them, everyone should have the opportunity to.

8:33And so that's partly where I was coming from. Like, I met people in computer science classes, women, black computer science students, etc., who just, like, were like, I don't see anybody who looks like me here. And that's also, like, an access issue, right? Like, even if you have the tech now, it's like, how do I access education if it's not, like, if it's a very toxic environment to be learning? The other thing is, like, yes, like, I'm sure Tim Neat probably talked about this a lot, and she would probably say things a lot more eloquently than I can on this topic. But I think the first thing I saw was actually Joy Bulamwini and Team Neat and Deb's work on the commercial face recognition technologies and how there were disparities, intersectional disparities between like white men and black women, like whose faces were being recognized.

9:14And I was like, wow, this is really bad. In fact, like black women could put on a white mask that looked nothing at all like a human. And that would enable the technology to recognize them, but it just would not recognize them. And then as you learn more and more, you realize it's not just about these biases. It's like, how do these biases translate into real-world harms? You see a lot of queer people online because of how acontextual and how, I guess, not very granular and there's not a lot of consideration of societal context and also representation that people who are building content moderation systems, queer people are being censored online.

9:49You see that trans people are being harassed and bullied through things like dog whistle transphobia, which is where you don't say explicitly, like, you don't use curse words, you don't use anything explicitly profane, but you're still saying very hurtful and harmful things. Right now I'm talking about social media, but of course this happens in tech communities. Tech workers face this stuff all the time, where there are kind of like these microaggressions or like snide comments that like just make it like an inhabitable space. But yeah, just like in the context of like social media, for example, like these content moderation systems don't catch things for queer and trans people.

10:20And then, if anything, they over-police the language that queer and trans people use themselves. If you use the word, like, actually even in Europe's, the Rocket Chat system, we noticed during the first Queer and AI workshop, you couldn't use the word lesbian in the Rocket Chat system. And that made the workshop hard because, like... Is that right? It blocks that word? Yeah, they were using block lists that contain words like gay and lesbian. And, you know, we kind of were like, hmm, this seems like, this is, like, very meta. Like, we're here. We're talking about, like... Bias and equity. And that's a third-party app that they're using, right?

10:54And that app had these block lists just in their product, not specifically for this conference. Right. But the acontextuality of that is very stunning, right? It's like, how can you just take a sentence and see the word gay and say immediately, that's going to be a bad thing? And it's not just limited to RocketChat. There are these amazing papers on these large data sets, like these huge text corpora that we're using to train our large language models. and the kind of filtering processes that go through that data before we train our models on them. And like about 40 to 60 percent, my belief, I'm quoting the statistic correctly, of documents that have the word queer or gay or lesbian, homosexual, are removed from the training corporate, even though they don't contain anything like explicitly sexual or profane.

11:37And the rules are being set manually, or is that an effect, a knock-on effect of some other policy? They are being set manually, but I also think it's about tendency in this community to not inspect our data or think about our practices super critically. It's not like people are going in, not everybody at least is going in and making these lists and saying like, yeah, I'm going to remove like the documents with the word gay or lesbian. It's like we have these repositories of block lists that just get recycled and repurposed over and over again. And people just take them without being critical about their contents or like the downstream effects of using that kind of block list yeah and even you know the documents that you contain where it's like queer and lesbian it's almost never like the data we scrape just because of how vile the internet can be is it's over-representing like cases where queer people are hyper sexualized and there's not a lot of cases where queer people are just treated as like normal human beings that lead like very complete lives it's all about this kind of almost fetishization of that moment where people are coming out or like people are transitioning and that's the focus like there's this mysticism about it that is like they call it like the cis het gaze like the cis gender like heterosexual gaze yeah it's like the people who are being centered here are like the people who are the perceivers of the queer community and what matters to them is like what they find like to be centered in queer people's life just coming out whatnot which is almost never true obviously kind of have had to come out multiple times but that's not what i think about on a daily basis but when a language model sees something like this it's gonna perceive that like this is kind of what goes on in queer people's lives and i could keep going like social media you can have recommendation algorithms that are outing like queer and trans people because people are have these are private accounts where they post like more personal content that they share with a small group of people but then if you recommend someone's profile to like someone else and now like these people have discovered that you exist and there's this aspect of identity that they weren't comfortable sharing with you that's not particularly good in we know in like egypt and some other countries the police use like grinder to identify where queer men live and come after them so yeah you realize like it's not just bias like of course bias is one part of this but there's so many like tangible harms that result from just the ways in which these systems are being deployed that kind of go beyond just bias it's like the infrastructure in which we're deploying these systems are often so agnostic to social context.

14:02And so maybe like, they're not, we're not considering the like the very complex social inequality faced by queer people. There's so many ways that the people again, because it's predominantly cishet white men who are building these systems, they could not even fathom like that the system would affect people in this way. And the other aspect of it is even when these harms do become apparent to people, even like I've been part of a lot of efforts to voice harms, people still don't do anything about it because they treat queer people as some kind of negligible minority, like there's a utilitarian perspective that kind of comes in.

14:34That's fascinating. So the queer and AI is an affinity group. I mean, and I assumed as much so of my saying that I thought it was a social group. I kind of assumed there was some something more purposeful there, but I just haven't spoken to anybody. So forgive me for that. So there is a real purpose to having the group. And that is not only to raise awareness among the AI community, but to work on how does the group work? Are there projects or is it a place for people to talk about these and learn from each other and then go back to your organizations where they might be and spread the word about, hey, we should pay attention to this.

15:17So how does it work? Yeah, again, another amazing question. It's a little bit of everything. And I think that very much is the result of it being a ground up, community driven effort. We are a very decentralized organization. So of course, I'm here talking to you about the organization, but I wouldn't consider myself a leader or an organizer or anything. Anyone can become an organizer at any point. Anyone can take a step back anytime they want. It's all about realizing that there is some kind of initiative that you want to put on or there's a program that you want to help out with or there's just something you want to learn more about.

15:50You know, as a queer person, like it's not like the whole organization is not monolithic by any means. There's just so much diversity and intersectionality that kind of comes in. And I'm constantly shutting up and like listening to other people's experiences, people who are like queer and Dalit. So like you're like you're oppressed on the basis of caste or if you're like queer and black. Like I have privilege along so many axes. And so it's about listening to all these people, realizing there are ways in which I can support and uplift their voices and then putting my time towards organizing these programs.

16:19And anyone can do that. So I can talk really quickly about some of the programs that we do run. Of course, like you said, there's a lot of socializing. And this is great because we actually run our we run a demographic survey every year to get like a pulse of like the queer community within artificial intelligence. 85 % of people said that they don't feel like they have any role models. And 80 % of people said that their mental health was really bad in the past year. And those are like considerable, very high numbers. A lot of people say they don't feel like welcome at conferences, but relatively, they feel very welcome at Queer and EI events.

16:55People describe it as pure joy to be there and just to even talk to other folks. But the workshops also double as community education and empowerment. It's an opportunity for you to listen to people from the community who are very well-versed in certain areas, like the experiences of trans people in academic publishing or what it's like to be intersectionally marginalized based on queerness and caste. And in the past, we've done things like non-binary people in the global South. So we're really taking the effort to not just focus on cis gay white men. It's about the true complexity of the queer community and for people to have the opportunity to learn about these issues.

17:31And I really like those events. And I think the one thing that you actually even the NURPS board is not fully aware of is the breadth of activities that we do outside of the conferences. The conferences, like we are a full year organization always running. We have a graduate application fee aid program. A lot of people in our community, because of how you might get disowned or financially disenfranchised, do not have the funds to pay for just their grad school applications. Forget like actually living and actually attending grad school afterwards. But just the fee to apply to school is so much.

18:05And this is even worse for people from Ethiopia. The fee to just take the GRE is three times the average salary of a person. Absolutely ridiculous. And so we have this program. We provide people with, it changes every year, but like up to$1 ,000 for people to take their tests, to apply to five to six schools. And this has prevented people from, one third of people from skipping buying groceries, half of people from not being able to apply to certain schools. So it's very impactful work. We also do a lot of research and advocacy. So on the research end, we try to talk about biases and harms and try to make it like have these bodies of work papers that kind of talk about all the ways in which queer people are marginalized in the design, development, deployment of artificial intelligence systems.

18:52On the advocacy side, we work with a lot of organizations to make their policies, their work a lot more queer inclusive. So NeurIPS is actually something that I did a lot of work on. If you registered for the conference in 2019 and possibly even 2020, the form asked you for your gender, which to me is I'm ordering a coffee. Like, why are you asking me for my shoe size? What do you need to know? And then the options were actually like male, female, trans male, trans female, which is, again, like that distinction is weird. Why are you not specifying cis? Like, why are you even asking whether someone is trans or not?

19:25So we removed that question. We gave them the opportunity to ask more about pronouns, which is how you're going to refer to other people at the conference anyways, and make those options inclusive. If Semantic Scholar, they now give you the opportunity to include pronouns as well. And that's something that we worked on them with to make sure it was very queer inclusive. And like we have a guide, we have a diversity and inclusion, making your virtual conferences more queer inclusive guide, where we just put all our recommendations there. everything from how to have more queer speakers, how to ask about gender and pronouns in your forums.

19:58And this has been helpful to so many different organizations. They've emailed us and been, thank you, like we were looking for our answers to this and they're all here. Other advocacy workers might have seen the stuff with Google Scholar and the dead naming of trans authors. If you change your name when you transition and you no longer, of course, want anybody to use your dead name, your publications don't get the signal that they need to change their names as well. And in fact, it's really hard to get like a name change on your publications. People view it as like a threat to the quality and quotes of publications, but this is really dangerous.

20:29If you don't want to be out to your colleagues, if they go in and find a different name on your past publications, this could put you in a situation where your privacy is compromised. You may be subject to more violence. You're stripped of dignity and also credit, right? Publications matter. Your citations matter in this field. And if you're not getting due credit for the work that you do that's not fair and now you're put in this position of ongoing epistemic labor ongoing corrective labor to go and fix your name in fact there's a whole spreadsheet for each publication where people record trans people have transitioned change their name like record the steps that they had to take the people they like emailed and they're like this person actually doesn't even respond to emails so like maybe this route is better and on the one hand i love how ground up and community driven this is it's an act of care for each other like you don't want everyone to go through the same problem.

21:17You're working to communicate, do this well. But at the same time, it makes me so sad that it's so hard to do this. And Google, of course, does not let you update the names on your publications. So we've had to meet with Google over multiple meetings. They've been very stubborn. They still don't have a good solution. This is a taste of the advocacy work we do as well. And the last thing I'll quickly touch upon is at the conferences, you might have seen the Affinity Workshops poster session, which as Affinity Workshops share this year. I was very happy to help organize because I think like that's wonderful.

21:47Like a lot of the people who are coming from these communities have been sidelined for so long and a lot of them are junior folks who are just getting involved in the field. That was my experience in 2020 and the poster session gives them the opportunity to have a low stakes environment, a very chill environment to present to people who share their identity. If you've been to the main conference poster session, it's a little bit like overwhelming, a lot of folks. And so if you're just starting out, it's nice to have this space and to also just get an easy win right i wrote this paper i'm not sure if it's any good but i want some like confidence it'd be nice to have a little boost from going and having people tell you yeah like this is amazing work your work is valuable especially a lot of work that you wouldn't see at the main conference on the intersections of queerness and ai like blackness and ai this is i think honestly some of the best research is there and their posters and papers on technical aspects so both it's like talking about identity and like the humanities and social sciences and artificial intelligence, but also a lot of very technical content, like scientifically technical.

22:44And so there's a good mix of everything. And actually, at least for a queer AI, they don't even have to be papers. It can be like, we allow everything from like videos to like posters to poems. People have written really nice poems in the past. It's just, it's supposed to be very laid back just to get your ideas out there. Can you talk about trying to address some of these problems that exist in the tech now, or as the tech is being developed and how do you do that? So in ensuring that or trying to find ways for large language models, for example, to not absorb the kind of toxic biases that exist out in the internet, can you talk about how Queer and AI addresses some of those things?

23:28Yeah. So our view on this is like a lot of the conference has like a lot of technical solutions. There are quite a bit of like bias and fairness papers that you see. I think there's a lot of discussions on how the efficacy of these methods, because they work on our nice curated data sets and nice benchmarks and how they translate into actual bias and harm reduction in the real world is a little bit questionable. So I think we focus on making the harms visible to people rather than necessarily providing like technical solutions. And there's this really nice quote from this researcher, you don't see harm that's done to people who you don't see as people.

24:02And so if you're building this, there are a lot of people, unfortunately, that you're just not going to see as the people, like you're going to ex-nominate some particular group of individuals using your technology. And our goal is to expand that vision just by surfacing a lot of these harms, making it more apparent and making people more aware of them. At the same time though, I think one, maybe still like less technical, but way we believe in which we can improve the quality of our tech. Actually, it's a little bit twofold. One is just increasing representation of queer people in research, in industry, who are the people who are building these systems.

24:36I want to emphasize that's not enough. We need to be creating inclusive environments for them as well and also giving people the opportunity to comment and be more critical. Maybe sometimes it's just best not to build something. And if a queer person says that, maybe it's a good opportunity to have a discussion and have more opportunities for education. I think generally in this field, we write a lot and produce a lot, but don't read enough. We don't, we're not diverse enough in the kind of literature that we consume in the conversations that we have. So I think that representation can really help broaden people's just like breadth of knowledge.

25:08I think it's super important to have knowledge from other people. We're constantly like talking about ourselves and our identities and that generation of new knowledge based on our personal experiences is so valuable. Like having these queer people in design, development and deployment is essential because I have a paper about the harms of being exclusive with respect to gender, like what the treatment of gender is binary within machine learning and how there are challenges to actually representing trans and non-binary people in our language technologies. And we see a lot of times that there's both fear from the community, but also in reality, when people, trans people who are more inclined not to disclose their gender, like their data just gets thrown out, right?

25:47Like you get erased. We see a lot of, as I mentioned earlier, association of highly sexual and not representative content associated with trans people. We see like a lot of systems failing to recognize trans people as even entities, failing to tag them, failing to recognize like singular they as a pronoun that could be used. And I think having more queer people building these systems, like you're just inherently more alert to these things. Like you are like, that's definitely a possibility. I've actually probably even experienced this before. Let's take a look and run these tests to see how our model is doing on like, for example, recognizing singular they.

26:22And then I think like, on the other hand, I think we're so focused on like how not to harm people that we don't think about the unique ways in which we can benefit certain communities i had a wonderful friend and z built this transformer model that writes stories but using neo-pronouns for different people and that brings a lot of joy to see other people who use your pronouns and just like a nice story and something that's not hyper sexualized something that's just this person went for a walk they were like going to look at birds z really enjoys birds I think that kind of euphoria that comes from just thinking of the ways in which these technologies can have a positive impact that can make people happy is also good.

27:02It's not enough to just strip things of toxicity. I think everyone deserves to be happy when they use AI technologies. It's interesting when you talk about queer people not feeling safe at conferences or in the more broadly computer science community. And I always thought that the academic community was much more inclusive and a much safer space for people than, I mean, simply because of the level of education that community has. Certainly a lot safer than rural America, where maybe there isn't that level of education. Do you think that the level of inclusiveness is higher or lower in the computer science AI community than other academic communities?