Making Machines Mindful: NYU Professor Talks Responsible AI - Ep. 205

0:10Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. On July 5th of this year, a new law went into effect in New York City. The law, believed to be the first of its kind in the world, states that hiring software that relies on machine learning or AI to help employers sift through candidates must pass a third-party audit to show that it's free of racist or sexist bias. Companies that run AI hiring software must also publish those results. The law is important on its own, as more and more employers use AI in their recruiting and hiring processes. But it's also important as a real-world example of what regulation looks like in a world where high-profile decision makers are appealing to governments around the globe to regulate, or not regulate, advanced technologies like AI.

0:56Our guest today played a vital role in the creation of New York City's new law, and that's just one example of the kind of work she's been doing throughout her career. Julia Stojanovic is Associate Professor in the Department of Computer Science and Engineering at NYU's Tandon School of Engineering. She's also an associate professor of data science at the Center for Data Science at NYU, and she's the director of the university's Center for Responsible AI. Julia's work focuses on developing and deploying AI in a responsible and ethical manner. She focuses on the practical issues of AI's impact on our lives, including hiring processes, loan approvals, and other decisions that directly influence human agency.

1:34Julia served on New York City's Automated Decision Systems Task Force. She's worked with the Queen's Public Library to develop courses to help citizens understand and advocate for AI ethics. And she's here now to talk about how responsible AI can and should fit into the real world concerns of, well, anyone who's listening right now. So let's dive right into it. Julia, welcome, and thank you so much for taking the time to join the NVIDIA AI podcast. Thank you for having me, Noah. It's a pleasure to be here. So the introduction there, especially at the end, was a mouthful, but I think it speaks to all the different things that you are involved in and working on as it relates to the technology itself, your roles in academia, and then obviously working to help everyday people, to put it that way, understand and advocate for their own rights in a world that's increasingly governed by AI.

2:25So why don't we start with the Center for Responsible AI? Maybe you can tell us a little bit about what the center is, what your role is there, and And then I'm sure that'll open the gates for us to get into talking about all the specific projects you've been involved with. Yes, sounds great. So the Center for Responsible AI that I have the pleasure of directing at NYU aims to make responsible AI synonymous with AI in a not so distant future. And of course, we all perhaps want to live in the world in which this already happened. Sure. That we don't have two kinds of AI to think about and deal with.

3:04One that is irresponsible and the other that is responsible, right? So the goal for us is to make true that this convergence happens so that we end up building an ecosystem of AI innovation that is socially sustainable. That we can continue to innovate, that we can continue to develop and deploy new AI-based systems that are improving things for people individually and collectively, that are improving things for society. And this is in contrast with the types of systems that we very often see deployed today, unfortunately, that are certainly improving things for some, but for a very select few and based on some questionable, let's say, ethical considerations.

3:49And so the way that we do this work at the center is, of course, through basic research in computer science and data science, But then also at some point, we realized that at this point, perhaps to move into a regime where we use AI responsibly, we don't necessarily need more algorithms or even more data. But rather, what we need is better guardrails, better informed people at every level, and also better laws and better regulation. And this is why the center engages in all of these types of work. Of course, we do research. Of course, we're very interested in making sure that the research insights that we develop are made relevant and brought into everyday practice.

4:33But we also do a lot of work on education, including public education, as you mentioned. And we hope to participate in deliberations on laws and regulation around AI. So before we get into some of the specific work and concerns that you and the center have been working on, maybe we can set the groundwork for listeners who might not be familiar or even sort of confused by some of the terms that have been floating around in the media, especially this year as we're talking responsible AI, AI ethics, and then something you didn't mention, but I've been hearing a lot in the media, AI safety. Maybe you could speak a little bit to what those things are and how they're similar, but maybe more importantly, dissimilar.

5:19Yeah, this is a great question. And it's a question that's very difficult to answer because like with everything in language, right, there is syntax and then there is some surface level semantics, the actual meaning of the words, the colloquial meaning. And then some of these terms, they take on political overtones, right? Yes. And all of these terms have been used for good and they have been used for less good and for kind of manipulation purposes, right? So even if we think about the term AI itself, which is a component of all of these responsible AI and AI ethics and AI safety, that term was coined so as to signal something that is far beyond human capability, especially at the point when that term was coined, right?

6:06So artificial intelligence, it sounds grandiose, it sounds magical. In a way, this kind of hype and grandiosity and magical thinking, they're built into the term AI. And so part of the work that we all need to do as people and as professionals is to unpack what power we allow ourselves to delegate to AI. And what are we comfortable with in this? And how much magical thinking do we want to allow ourselves here? And this is why we often couple this term AI with terms like safety, like ethics, like responsibility. So responsible AI specifically, this term has been questioned also in terms of its appropriateness, because it sounds like we are saying that the AI itself is responsible for the consequences of its use.

7:01But this is not what I mean, and this is not what most people mean when we utter this phrase, responsible AI. The responsible refers to people taking responsibility for the decisions that we make individually and collectively about whether to build an AI system, how to build it, how to test it, how to deploy it, and how to keep it in check. So this is the most important thing here to think about is that really responsibility concerns are those that people share that we don't delegate to AI and that allow us to embed some legal requirements, some ethical norms, some safety requirements, some sustainability requirements like environmental sustainability into the design development and use of AI systems.

7:51Now, AI ethics is a related concern. And in some sense, it's perhaps more specific than responsible AI, at least in the way that I use these terms. And so I usually use the term ethics loosely to denote the study of what is morally good and bad and morally right and wrong. And then AI ethics is when we refer to the embedding of these moral values and principles into the design, development, and use of AI. Right. Okay, so, and if this is not a good segue, jump in and maybe take us on a better path. But it makes me want to ask about the hiring law, the New York City law. And it's kind of a practical example for people of how these things play out in motion, everything from designing the system, what it's trained on, how it's used.

8:41But then, as you mentioned, the political machinery involved, let's say. As you were speaking, I was thinking about the sort of not just the bubble that I, you know, exist in working in the tech industry or being in the San Francisco Bay Area, for that matter, but even within the podcast itself, then thinking about, you know, as you said, the magical thinking. So talking a lot to people who are working in the field and using these terms in a sort of grounded way, let's say, and then thinking about everything going on in the media and talking to friends of mine or family members and sometimes hearing them discussing AI as though it is this already this autonomous thing operating on its own, when in fact it's, you know, a system that's designed and deployed by human beings, as you were saying.

9:32Right. So maybe you can walk us a little bit through what the what the law is, what your role has been and in on the task force and otherwise. So maybe before talking about the New York City law, I'm going to, you know, since I'm an academic, I'm allowed to try and give a definition, right? So that we are somewhat precise, right? So there does not exist a definition of AI. But I like to take a definition that works for us when speaking about responsible AI. And that is that an AI is a system, right? It's not just an algorithm. It's not just data. But it's a system in which algorithms use data to make decisions on our behalf.

10:11or help us humans make decisions, right? So these are the types of systems that I am interested in in particular. And responsible AI is about figuring out how to use AI for good while controlling the risks, right? And here we need to think domain by domain, use case by use case. And so a very important use case, one where there surely are benefits to the use of AI, but there are also plenty of harms that we need to think about, is when we use AI in hiring and employment. And now here again, when I say AI, I mean it in the most general sense in terms of algorithms, right? I'm not saying it's just these, you know, fancy large language models or generative AI.

10:54More generally, it could be any type of a rule-based system. And this is important to think about because people are going to push back and say, we don't need regulation because we're just using these really standard, simple systems like, you know, your Roomba, the smart vacuum. So why are you starting to interrogate us now? Now, it turns out that in hiring and employment in particular, we use AI broadly construed at every stage of the hiring process. So even before people apply for a job, some system somewhere decides which ads for jobs they see. Right. And these decisions are made with the help of personalized information targeting, personalized targeting.

11:36are getting that platform scary, like Facebook, like Google ads, like LinkedIn, right? So here already, of course, AI is at play. And at this point, very important decisions are being made by job seekers. And these decisions very often are implicit. If they don't see an ad for a job, they're never going to apply for that job, right? So if it turns out to be the case that ads for specific jobs are shown disproportionately to particular population groups based on gender or race or disability status and are targeted in a way that has nothing to do with either people's preferences or their ability to do well on the job, then we have a problem, right?

12:17Because we're now systematically changing things in society, sometimes inadvertently, that are impacting things downstream. And then there are tools that are used much more directly. So we refer to people who apply for a job already as candidates. And so when candidates are screened, very often this is done with the help of AI. For example, there could be a resume screener that looks at the text of the resume and the formatting choices that people make. And it constructs something like a personality profile. I'm sure I'm using air quotes. You're not going to see me do this, but note that personality profile is in air quotes, meaning that this is a questionable practice, right?

12:56As a person, I can say that it's questionable to me that somebody could extract my level of dominance, conscientiousness and neuroticism, for example, from the way that my resume is put together. And I also doubt that this is something that is job relevant in any way, right? I'm laughing, but I've experienced it, you know? So, yes. Right, we all have experienced this, all of us at least who've looked for jobs recently, right? And it's one of these sort of funny, not funny things, right? Exactly. That you would think that nobody would seriously believe that you can extract the level of dominance from a resume and that it's relevant for the job in some way.

13:34And yet these tools exist. They are built by commercial entities. They are sold to companies. And the vendors of these tools claim that essentially the majority of the companies, of the Fortune 500 companies, for example, use their tools. And so it's a kind of an arms race also in a way where, you know, employers maybe are under pressure to start using the latest technology as part of their screening, a job placement function, because others in their area are also using AI tools, right? So they are afraid that they will be left behind. Right. I have some friends who work as recruiters, and I empathize with the wanting to use tools to help sift through data sort of broadly.

14:17So, you know, there's utility. But as you say, what exactly are you working with and how exactly does it operate? Are you able to, willing and able to take the time to understand these things before you start using them and affecting people's lives? Absolutely. Right. And Noah, this goes back to one of the terms that you used in the introduction and that I really like, and that is agency, right? So the main thing in my mind that we need to think about is that we are the people who are building and using these tools. And we're making decisions or are being subjected to decisions that are made by these tools.

14:52And so we have to have a say in how these tools are used. And now think about an HR person who is under tremendous time pressure to fill positions, where there is this environment that has been created in large part due to the proliferation of technology, where in order to hire anybody for a position or two or three positions, you have to sift through thousands of resumes potentially. And similarly for job seekers, right? They have to apply to lots and lots of jobs these days to even get an interview with a human. So there's this environment that has been created where there is an overabundance somehow of data, both on the demand and on the supply side.

15:32And then you have to use technology in order to find that needle in a haystack, right? And then also because of these time pressures, because of the volume, HR folks are not actually given enough either time to reflect or enough information really based on which they would make an informed choice about whom to invite for the next stage of that sourcing and screening and interviewing process. Right. And so it's a very badly set up environment in terms of just this lack of thoughtful and responsible human AI interaction. And so maybe that can take us to the brass tacks of the New York City law. Right.

16:13So this is Local Law 144 of 2021. It was passed. This was a proposal for a law for a while, a couple of years, I believe. And then was finally passed into law in December 21. And we're now in July 23. And the law finally came into effect. Right. So this tells you that deliberations around this law have been going on for a number of years. It's been a while. It's been a very contentious process. It's been a very interesting process. And I am really very happy that we were able to pass this law and to turn it into rules ultimately. Because to the best of my knowledge, this is the first law of its kind in this country and possibly even more broadly.

16:58Right. Where we are attempting to do two things, really. One of them is to make sure that vendors of tools that are being used in hiring and employment subject these tools to an audit for bias before these tools can be used and deployed, and that they redo this audit periodically. The second set of provisions of this law, in my mind, is actually even more important, and it's where people are told before they apply for a job, before they become candidates, while they're still job seekers, that their application, if they apply, will be screened with the help of a tool. They will be told what data about them this tool will look at.

17:43Is it going to only look at the resume that they submit explicitly, for example, or will it also look at their Twitter feed? And then also they will be told roughly how they will be screened, and they have an opportunity to request an alternative screening. Now, there are lots of things that the law in its current form after the rulemaking phase doesn't have. And that is the disclosure to job seekers is fairly limited. The original proposal for this law said that they would be told what features and characteristics of their application the tool will use in the screening. And this is much stronger than just telling people, oh, you will be screened by a personality predictor.

18:25Right. But it's a good start, right? Because at the moment where we are, people don't even know that they're screen-banned. There was, I forget which article it was, but there was something I was reading last night where you were quoted as, I'm paraphrasing, forgive me, but saying something about, we can only learn how to regulate these things if we try regulating them. You said it more eloquently, but the idea is saying, you know, it's not perfect, but there was nothing before. And the way you make something good is by starting. And so you're learning, we're learning, as you said, it may be the first law of its kind in the world regulating these things.

19:01So I can only imagine that the original bill was much more strongly in favor of transparency and disclosure. And then it got negotiated down to what could actually be passed. Yeah, I mean, it was more ambitious in some ways. In other ways, it was less precisely specified, right? Because during rulemaking, we actually try to agree as to, you know, how to do a bias audit, for example, who does it. One of the things that is limited now much more so than in the original proposal is that we only are going to be auditing based on gender and ethnicity and race and the intersections of these categories.

19:35So not age, for example, and not disability status. And that's a pity, but we'll see. And of course, you know what I said about us needing to try to then learn from this as an example, right? Either learn from a success or from a spectacular failure. We don't know really how this is going to play out. this is something that an engineer would tell you, right? And of course, you know, some people would object to me saying this because it reminds them of let's move fast and break things, right? So this is actually not what I mean, right? No, no, no. I really don't think that we should be passing laws arbitrarily and trying to see how things will play out without any requirements, right?

20:15For what it's worth, nothing you've said to this point implies anything other than we should be thoughtful and reflective, so. Yeah, that's the intention, right? But of course, you know, the law is not perfect. It can't be perfect. We don't really know what all the harms are of technology of this kind. And we really need to start this conversation very concretely with people at the table, with people being impacted, right? Job seekers. So what you described sounds a little bit akin to, I believe the term is a model card, things that people have been talking about using more broadly to try to introduce greater transparency to AI models in general and get away from the black box effect, as it's called.

20:57And so that people of all ilks, right, whether I'm a job seeker or an engineer or a politician, when I encounter an AI model can get sort of, you know, something like I've heard it described as a nutritional label, kind of explaining, you know, basically what you just described, like, this is what the model does. This is what it was trained on, you know, et cetera, et cetera. Is that along the same lines? And is it something that you can see ideally, you know, maybe a ripple effect, a follow on of something like the New York City law being applied to other industries just in the name of greater transparency?

21:33Yeah, this is a great question. And thank you for bringing this up. You've mentioned model cards and nutritional labels. So in the research community, people have been calling these transparency devices by different names. I have been running with the nutritional label metaphor myself for a number of years, but it's really syntax, right? I mean, I think that we all are in agreement that we need to have ways to explain whatever needs to be explained to a particular person in a way that they would understand and that they would be able to act upon, right? So a model card is usually something that you would show, let's say, to a developer of a model that is going to contain some information that perhaps is at a deeper technical level that explains what were the data sets that you trained and so on.

22:21What are the properties of your performance? How did you measure it, etc. Nutritional labels, in my mind, are kind of a more consumer-facing metaphor. Before we think about nutritional labels for food, where after a long deliberation, now producers of food are compelled to tell us, consumers, how many calories the item contains, what types of calories these are, what's the trans fat, as a percentage of maximum daily value, etc. And we've come through this deliberation to a particular way in which we explain this information to consumers and, you know, even, you know, arguing about what should be the font size of the various components of the label so that people could gather useful information from them.

23:08And in domains where we have AI systems impacting people, I think that we should have similar consumer-facing methods where we're very clear about what information we want to surface and how do we check that the people to whom we're showing this information, in fact, are able to understand it and to act upon it, to exercise agency, right? So in this case, if you show a nutritional label for food to somebody who doesn't know what a calorie means, this disclosure is ineffective. If you show it to somebody and they understand what the calorie means, but they come from a place of food insecurity, then the action that they would take is to consume food that gives you more calories per dollar.

23:49So we should not discount that as nutritional labels not working. They actually are working for these consumers in a way that they choose to make their choices. in the way that they choose for these labels to work. So similarly with nutritional labels for, let's say, job ads, and this is something that I wrote about in an article in the Wall Street Journal, actually a couple of years ago, is that we should be attaching nutritional labels to job ads. We could call them posting labels. And in these labels, we would tell people what data about you will be used, what tools will be using that data?

24:28What features will they look at? For example, what matters really for you to be screened in or out? Is it your GPA? Is it what college you went to? Is it what degree you have? Is it the number of years of experience in, let's say, C++ that you have, right? Right. Is it what I posted on Twitter yesterday to your point earlier? Exactly, right. So this is very, very important. People should know we don't have a data protection law in this country. So we actually don't have any control over our data and its use. So when we put something, when we utter something publicly on Twitter, it shouldn't be assumed that because it's public, it's okay to use this to make decisions about whether or not I'm employable for a particular position, right?

25:05That's not the purpose of that data. And then importantly, also people can then, based on this information, decide whether or not to apply for the job. So this gives us an informed consent mechanism. And then also if people apply and they are told why they were not selected, let's say, for a job interview, a nutritional label would be attached to a decision as well. And then that would give people a way to challenge the decision or to exercise, take a recourse action in some other way. For example, go and pass a test so that they're more competitive for similar jobs in the future. Our guest today is Julia Stojanovic.

25:43Julia is an associate professor in the Department of Computer Science and Engineering at NYU's Tandon School of Engineering. She's also an associate professor of data science at NYU's Center for Data Science. and she's the director of NYU's Center for Responsible AI. Julia, we don't have a lot of time left to chat, which is sad for me because I could see us talking for, well, for as long as you're willing. I have plenty of questions, but I wanted to use at least some of the time we have left to step back a little bit and ask you about some of the other work that the Center for Responsible AI is working on.

26:19And then I would imagine you would talk about this anyway, But to be sure, maybe if you can also speak to the listeners a little bit about things that individuals can be doing now in a world where we don't have these labels that are appearing on job postings. And you're lucky just to get a rejection note from many recruiters these days, let alone an explanation of why you weren't chosen to move forward. So what are some things that an individual can do to advocate for their own agency in a world governed by all of these sort of opaque algorithms and systems? This is, once again, a great question, Noah, and I don't have a really good answer.

27:05Yeah, so what can we do individually and collectively? The main thing is that we need to step back and remind ourselves about the following. if something gives you a promise of knowing what you mean, of being magic, right? Of solving the kinds of problems that we haven't been able to solve up to now in society, like being able to predict who will do well on the job and who will not do well, or being able to predict who will commit a crime in the future and who will not commit a crime in the future. This sounds too much to be true, right? I don't want to say too good there because it's actually not a great thing to be trying to predict someone's behavior.

27:47But if something like to you as a human, if it doesn't sound plausible that this is something that anybody would predict, that it's like looking at the crystal ball, right? Then probably an AI cannot predict it, right? There's no magic here. So let's just all inhale, exhale, and put on our common sense hats. And very often this means put on our skeptical hat right now because there's just so much hype. We can't expect technology of any kind, even AI, as magic as that sounds, to solve all of our problems, like get rid of structural discrimination in hiring, be able to predict who will do well on the job.