The EU AI Act and Mitigating Bias in Automated Decisioning with Peter van der Putten - #699

0:00There's elements of GDPR that clearly didn't work, you know, like that you need to accept cookies, and everyone is like click, click, click, click, click, click, click. But what did work is actually not just data scientists, but anyone in business, at least started to think a little bit more like if we're gathering data, it's not just our data, it's also our customers' data. What is the purpose for which we are gathering this data? Did the customer give consent for that? And I think there will be a similar kind of Brussels effect of the EU regulation here, because, yeah, like for starters, it doesn't matter if you're like an American company, or a Brazilian company or whatever, and you're operating in Europe, then you need to follow these rules.

0:55All right, everyone. Welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. And today I'm joined by Peter van der Pooten. Peter is Director of the AI Lab at Pegasystems and Assistant Professor at the Leiden Institute of Advanced Computer Science at Leiden University in the Netherlands. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Peter, welcome to the podcast. It's great to be here. It's great to connect with you again. We had an opportunity to collaborate actually quite a while ago on an AI program at Pega.

1:34And we've been wanting to catch up and talk about some of the work that you've been doing on bias mitigation and in particular, where academic fairness metrics tend to fall short in how to close those gaps for automated decision making. I'm looking forward to that conversation. and I'd love to have you start us off by giving us a little bit about your background. Yeah, awesome. I'm a bit of an old-timer in AI, and I started to study AI in, believe it or not, 1989, and graduated from a master's in the mid-90s, and first did a lot of interesting work in, let's say, commercial-type research, so with all kinds of different forms of data, like structured data, images, video, semantic search engines, etc.

2:25So that was all in the 90s. But then at some point, as maybe the typical kind of data scientist, let's say five years into your career, I became a little bit almost frustrated that all the interesting data science stuff that you build, you do a project, you lift your heels, and then you come back later. And back then, it was really, yeah, it was all on a project base. It was almost like the data science then evaporated and everyone went back to working the old way. So that's when I decided to join a startup, which was more focused on decisioning. So as a way to make AI actionable, you know, how you can make automated decisions in marketing and service and loan applications, claims.

3:17And because then, yeah, the focus is really on embedding AI into a process, not just I'm an AI machine learning guy, but not just the AI machine learning, but even the good old fashioned AI, you know, so that's also where then ultimately ended up at Pega, you know, in the interim, a bunch of new hypes have happened, deep learning around 2010, generative AI now. It's been quite a journey. Yeah, exactly. But that whole topic of how to make AI actionable, but also to make sure it creates responsible impact. Yeah, that's been kind of a bit of a trail throughout. And one of the interesting points of context for that that has been particularly important for you being in Europe is some of the recent regulation that's coming down the line through the European AI Act.

4:17Can you introduce us to that and tell us what you see there? Yeah, no, absolutely. So, yeah, of course, we've been haggling about that EU style for like almost six years. 2018, it started with a so-called high-level expert group on AI, and they came together, tried to define what kind of definition of AI should we pick, but also what are some of the key ethical principles like transparency, accountability, robustness, but also fairness. And that then evolved into kind of a proposal for EU AI Act already a couple of years ago. And that has been under negotiation for a couple of years. But the final tax got accepted by Parliament, European Parliament in March.

5:08And now there's only one final hurdle, which is pretty much only a formality to get that legislation into effect. and the key idea of the EU AI Act. So it's covering these ethical principles that I spoke about, but higher level it looks at also, it takes a risk-based approach, right? So it looks at an AI system and not AI in general. That would not make sense. You know, you can use it for good things. You can use it for bad things. So you need to look at a particular AI system, a particular use case, and then look at what is the risk to do harm. And there are certain things that are not allowed.

5:49It's a very small list, like whatever. Social credit scoring, like you have in China, for example. But then there's high-risk AI systems and, let's say, regular AI systems. And the higher-risk AI systems, for instance, that's, yeah, come with more scrutiny. I'm curious how it even defines an AI system. What I really liked about the EU AI Act is that people didn't start haggling about what is the difference between machine learning and statistics and things like that. It was taking the point of view of, let's say, the average consumer or citizen who gets exposed to automated decision making. From that perspective, it doesn't really matter if you're being denied a loan or investigated for fraud or whatever it is.

6:47It doesn't matter that much whether that's through a machine learning model or through a regression model or through even a combination of models and rules, good old fashioned AI. In all of those cases, it's an automated decision in an AI system. So from the start, actually, that definition on purpose, the technical definition was kept quite broad because even purely technically, mathematically, there is no fundamental difference between, let's say, a small neural network or a regression model. If you have a neural network without a hidden layer linear transfer function, that's your regression model, right?

7:28But it was good that they took that kind of broad approach and then looked more at the use of that AI in a particular system. The current definition is a definition from the OECD that has been accepted. And actually, it got me even more excited because it's using a definition which very much looks like a field that almost predated AI, which is cybernetics, right? So a good old Norbert Wiener, because it's talking about AI systems that indeed are driving automated decisioning in an environment that can have an impact on that environment as well. And so it's really taking this, it's really trying to move away from, let's say the pure technical definition of what algorithms do you use towards more like what is the use of an AI system as, let's say, a real-time decisioning system or a generative system that's operating an environment, that's interacting with users, that has an impact on the environment.

8:31And so is the act of making the decision or making a decision that impacts the users of the system, is that kind of the key element that defines the scope of this act or is it broader than that? So if I get your question correctly, it's the highest level, apart from the general level principles of transparency and fairness and accuracy and robustness and safety and privacy, the higher level thing is, what is the objective of my AI system? Is that a good objective? Because certain objectives are forbidden. And other objectives for these other objectives, it's what's the risk to do harm? as they they put it right so and then you have these higher risk systems that you need to put on there like higher levels of of of scrutiny yeah uh like like you know providing access to financial services that that that is actually uh uh providing or blocking access to uh for example is a uh such an example or or providing access or or blocking access to education or or whatever it is so um but yeah so but it's really in that sense um not some theoretical law it really looks at like oh you know what do you use it for you know like what is the risk to do art and it's it's a very i think sensible approach is risk in this context uh defined abstractly and uh it's up to the implementer to um figure out if their system is risky or not or are do they explicitly list you know these specific uses are considered high risk medium risk low risk it seems like it will be very difficult to catalog every possible use case and assign it a risk it's it's a miss it's a mix so uh yes indeed they leave a lot of it to yeah ultimately ultimately to yeah uh i don't know certain cases will be brought to court or whatever um uh to really fill in fill in the details of what constitutes a high level of risk or if there are certain requirements around whatever, transparency or whatever.

10:59Did you actually meet those requirements? But what they did do is list a bunch of things that for sure fall under forbidding use and for sure fall under high risk. Now, here in the U.S., one of the ideas that's coming up around AI acts and AI safety are constraints around training large language models, if it's going to be larger than so-and-so parameters or require more than so-and-so compute. Those are forbidden or require some degree of regulation. Does the European AI Act get into that type of regulation or is it primarily focused on consumer protection or, you know, citizen protection in terms of decisions and interactions?

11:55Yeah, no, it does actually get into it. Honestly, the history of this type of law in Europe is a little bit from a product protection point of view, right? So and that still shines through a little bit when you bring a product to the market. Well, with AI, you don't bring a product to the market, you know, like you're you're, you know, many of the whatever. If you're a bank and insurance company, whatever, you're creating your own models. Right. So so it's slightly different than bringing a product on the market. But they did actually address these points around generative AI. Because when I said, like in the beginning, technology can be good or bad, nor is it neutral.

12:41We need to think about an AI system. That was actually a very kind of convenient position a couple of years ago. But then, of course, the NAI came around the corner and this whole concept of foundation models that everyone can reuse. And then it was like, oh, you know, like, should we treat it as a completely different thing? type of thing, or should we apply exactly the same rules? And, you know, there's a little bit of a midway solution. To be honest, I do think it's healthy to not just jump to the conclusion you need to do something completely different. I think the general framework, they haven't changed the general framework and it still holds, but they did make like additional kind of proposals around how to deal with foundation models, because foundation models, they're a little bit in between this general technology and an actual AI system or product, because you can say it's generic, so it's not a system, and it's ultimately people that use the model in a particular context.

13:42That's right. But on the flip side, people typically don't change the foundation model. So it's almost like a half fabricate, right? So and that's where indeed, you know, like there's additional requirements that were put in. also taking into account the relative size of the model, you know, like how complex or large it is. I forgot the exact requirement, but it has to do with how many petaflops to train the model or something like that. So it's not parameters, but more actual kind of training costs, I believe. So that's similar in that sense as in the US. Of course, in the US you also have, I think, the Policy level, there's frankly, yeah, there's a lot of commonality across different regions, right?

14:34So even in China, where they were first to put out AI law, of course, there's maybe elements in that that don't resonate maybe more with values in the US or in Europe, like how closely they monitor their citizens, for example, a little bit too close, maybe. But on the flip side, there's also very, like all these elements of transparency, fairness, et cetera, they're also part of the Chinese legislation. Likewise, the thinking when you had the Biden exec order on AI last fall, or previously there was also a policy document coming out of the White House around AI. It has these similar concepts of the general base layer of things, of these ethical principles, but also what is the type of use.

15:31It's just not legislation. I think at this stage, I think the political climate is not ready for that at the moment in the US with elections coming up and things like that. But in that sense, the White House did take like a prudent approach also by messaging more to government, different government agencies and departments. This is how we want you to use AI, but also how we want you to use AI systems. And similar how there's a trickle down effect, the Brussels effect from EOAI legislation, these type of policies that are directed at US government agencies, they do trickle down into the rest of the economy.

16:18Right. Yeah. When I think about European regulation, you know, clearly there's lots of European regulations. But, you know, when I think about it and its application in the tech space, I can't help but think of GDPR, which, you know, I just remember having a lot of conversations at the time with folks about how it would impact the way data scientists and practitioners approach privacy. and it had global impact for organizations. Do you expect the EU AI Act to have a similarly broad footprint? Or are they talking about things that people are already doing and less about prescriptive things that people need to do?

17:15Yeah. No, I think it's a great remark. It's always good to look back to some other forms of regulation. How did that go, right? So from a GDPR perspective, this is a little bit of an opinionated comment, right? So just to take a position. And then there's elements of GDPR that clearly didn't work, like that you need to accept cookies. And everyone's like, yeah, click, click, click, click. Just give me my stuff, right? So that didn't work. But what did work is actually exactly what you say, that data scientists, not just data scientists, but anyone in business, at least started to think a little bit more like if we're gathering data, it's not just our data, It's also our customers' data.

18:03What is the purpose for which we are gathering this data? Did we kind of give some level of consent? Or did the customer give consent for that? So I think actually the biggest impact is almost like self-regulation in a good way, right? Where not self-regulation, but how do you call it? That you adapt your behavior in a way. I think that's actually the biggest impact. Not so much the big cases that were brought forward and where people, where companies or public organizations need to pay big fines. And I think that's actually quite healthy. I hear you saying that the big impact is establishing standards of care more so than enforcement regime that's going to come and police the market.

18:52it yeah and i think that worked and i think there will be a similar kind of brussels effect of the eu regulation here because um yeah like for starters it doesn't matter if you're like an american company or a brazilian company or whatever and you're operating in um in europe then you need to follow these rules yeah it doesn't matter whether you're american brazilian or Chinese. And then if you're a global company, then yeah, you're kind of going for the most common denominator in a way, right? We don't see that entirely. There's similar legislation like a Digital Service Act in the EU, and then you can see that whatever, the likes of Meta and Google, and they release certain things later in the EU than in other areas.

19:38But yeah, the emphasis here is also a little bit on later. They're releasing it. They do they do release it and then they, if they make sure that they tick the boxes and put in the extra safeguards, right? So, but I think like at Pega, we are a big fan of, I'm personally a big fan of ethical use, trustworthy use of AI. At Pega, we are that as well, which is maybe not what you would expect from a big tech vendor. But we really support it because I don't think regardless of whether you have regulation or not, I don't think there's any sustainable future for irresponsible use of AI. That's a very simple way to think about it, right?

20:22And it may work in the short run, but in the long run, customers will vote with their feet and they will take their business elsewhere. So I think in that sense, that's karma aside, you know, karma-wise it's important. But even practically, you know, like to use a fancy word from a utilitarian perspective, ethical perspective, forget principles and karma, it's not going to fly, you know, like bad, bad uses of AI that might fly in the short term, but it will never fly in the long term. People will just not accept the technology in the long run and they take their business elsewhere, right? So I think, so yeah, I think it's very healthy if you create a level playing field in that sense.

21:11In the lead into the conversation, I referenced some of the fairness and bias metrics that organizations have started looking at to understand the impacts of the way that they're applying machine learning. Does the act specify metrics like that? Is it prescriptive to that level of detail? Right. Yeah, no, it, it, it, exactly. It specifically doesn't tell you what, how you should do it, but it tells you, you should do it. And specifically for high risk applications, uh, you should, you should do it. Yeah. So you need to prove that, that you took, uh, that you, well, uh, that you at least made a conscious effort to, um, to look at fairness and, uh, to look at fairness and bias.

22:06And so one of the challenges that you point out is that the metrics that are often talked about don't really work in the real world scenarios that you see with enterprises. Can you give us some examples of where the disconnect lies? yeah like i think the disconnect there because i'm also an academic i love doing research but sometimes this is not just for fairness and bias but in general for many fields and then there's a grand idea about the field and then everyone is kind of focusing on some subtopic where it's extremely crowded yeah and and i think what you see a lot is in in you know this is a gross generalization so i apologize for anyone who's not doing this but but there is a lot of focus on yet another fairness metric to measure uh bias in a machine learning model at design time yeah so and you hear yet another suite or ensemble of fairness metrics to yeah choose from yeah yeah and i'm not saying that these fairness matrices metrics are not important but to some degree there's other, you know, there's also not bigger fish, but at least other fish to fry as well.

23:20So, well, you know, like if, if I say we measure fairness in a model, like going back to that regulation, if I don't get my loan, right. So that's not just based on one model, you know, that's a set, you know, probably a default model, maybe a fraud model, but also a whole bunch of rules that go into the mix to decide whether you get a loan or not. And yeah, for instance, UAI, but it's similar with lending legislation in the US. The point is, do you get a loan or not? Not like just looking at one particular model in a set of models and rules. So you do need to be able to look at a different level, not just a level of model, but at the level of a full automated decision, which is a combination of models, rules, data going into these models and rules.

24:23In what ways is that technically interesting? When I think about the bias metrics that are applied to models, it's typically you are kind of profiling a data set before and after, or a data set before and a decision after and trying to make inferences between the two, it seems like in the case you're describing, you can just create a larger or broader definition of model and assume that the model is the whole decision. In other words, a lot of these fairness metrics don't really apply to, you know, they're not kind of intrinsic properties of the model. It's more about the data and the decision.

25:05Yeah. I mean, that's a good thing. That's an opportunity. It means that a lot of your fairness metrics, if you have disparate impact or rate ratio or Gini index, or like we use those, for example, they acquire equally at, let's say, at the decision level than at the model level. of course where it becomes interesting then is is where it becomes algorithmically interesting is well how do you break it down you know like if you have an entire decision that consists of many rules and logic like how you know how do you trace fairness through such a logic for example or what's also interesting is maybe looking at well, you're going to, you might be generating a lot of different alerts, like, I don't know, and this is, by the way, largely an unsolved problem.

26:01And we, I don't know, some of our customers are using decisioning, real-time decisioning and one-to-one personalization. And then you're talking like deciding across a library of 3 ,000 next perception recommendations with a model behind each of those and then tons of rules on top and weightings and whatnot. Yeah, so if you're just by the law of large numbers, you're bound to find certain levels of biases in those models. So how do you make sure that, yeah, you're not creating a downstream problem by saying, oh, we have our metrics, we run our simulation, great, fantastic. And then you spit out thousands of alerts where you go like, what do I need to do now?

26:51And I think that problem gets aggravated, which is great for research. We like bigger problems. If you don't just look at design time, but why should you only look at design time? You also need to look at runtime as in when decisions are being made or maybe at orbit time. post decision you want to have the ability to replay maybe different versions of that of that logic so those are all interesting kind of still pretty much open ended research problems that moving from design time to run time monitoring of fairness for example or how to deal with all these alerts that are being generated or Or as data scientists, we have a tendency to, you know, when you have a hammer, everything looks like a nail.

27:45So you go like, oh, if I detect some bias, oh, I'm just going to create a bias correction algorithm that automatically corrects for that bias. Yeah, that's, of course, as a data scientist, what we do. But that's not always the right thing to do because you could be covering up the root cause of the bias. Yeah, so there's some great examples of that in healthcare in the US, where it's important to get into the root cause of what's behind the thing that created the bias. So these are just a random sprinkling of research problems and implementation problems, which are not typically, well, they could deserve more attention, I think, in research.

28:45And also, yeah, and that's not to do even more research, but I think from a practical perspective, it's really important. I think these are the real world problems that people who are implementing AI systems in the real world are having to deal with. And so from a practical perspective, like what do you say to someone who has to deal with these problems? How do they go about trying to address them? Yeah. Now, like, so when people hear compliance, they go like, oh, we need to be compliant everywhere. Yeah. Of course, you need to be compliant everywhere, but it does mean also that you do need to picture battles and you do need to look at like, okay, but where could some real harm be created, right?

29:39So I think that would lead you towards focusing on the right areas. For example, if I have whatever, let's say I'm a bank and I'm selling credit cards and loans. Yes, I can look at my marketing models. Yes, you want to have some level of fairness checking in there. But the models that determine your limit on your credit card, that's already more interesting. There was this case of someone using an Apple card and claiming like, hey, that's strange. The limit on my credit card is 20 times higher than the limit that my wife gets, right? So what's causing this? And then some other person retweeted that.

30:35And then this was about Apple Card and it was Steve Wozniak, right? So I have the same thing. The original was David Hedemeyer Hansen. This was shortly after the Apple Card came out. Yeah. Yeah. Yeah. And they investigated later on, turned out not to, they couldn't confirm there was a real issue. Just to set the record straight on that one. Yeah. But like do focus. So in this example, I started to say, hey, you sell credit cards and loans. Yes, you need to look at your marketing decisions, but the decisions that set your limits or the decisions that decide whether you can have a credit card or not, or a loan or not, or whether we're going after you for fraud, those are, of course, even more material decisions.

31:23And so I think it's healthy in that sense also to take this risk-based approach and see like, hey, where do we need to prevent, where can we prevent damage from happening? So that's something that data scientists are kind of, if you don't think too binary, then you're actually quite open to it because you can, it's a little bit of a quantitative approach. And we say, well, where do we expect the largest risk? And let's focus on there and then build it all out. But yeah, I think that would be a good approach. So what specifically is the approach you're suggesting? Well, the approach is to look at this risk to do harm, for example, and then focus on those decisions first.

32:23Like if I need to prioritize looking at marketing a particular loan versus providing actually access to the loan, can you get the loan or not? The loan acceptance. the loan acceptance is more important than a marketing i'm not saying there should be that we should reduce uh bias in marketing but there is a difference here right so i think that could help focus uh uh where you want to look um also there should be kind of a a culture um uh it should be embedded in in in the yeah at risk of sounding fluffy yeah but in the culture of the organization and the organizational process as well. So if I need to do that next to my day job and then the moment I find a problem, I'm the one causing trouble.

33:17Yeah, that's not a culture where issues are being found and fixed. So whereas if it's a culture like we recognize there's bias in these decisions, there's no way you can eradicate bias completely. There is no way. If you reduce bias for one group, it may impact another. But we want to keep it within certain set levels. And it's okay to find areas where models have drifted or rules have changed or the population has changed. And we're getting more from the green zone into the orange zone. It's okay to find such an issue and solve it. That's a good thing. So I think that's also important. So the first thing is find the areas where you can cause most harm.

34:14That's your strategic focus. Second is indeed more from a governance operational point of view. There needs to be this culture that it's okay to fix these things. And there's a recognition that, It's fine to find issues because at least we can fix them. And there will always be some level of bias. And then it's more into the methods. With the methods, think broader than just a model. Think about automate decisions that are models and rules. Think broader than just doing lots of testing at design time. And then you launch your models and logic and you never check them again. That's wrong. You need to monitor at runtime as well.

34:55So think a little bit out of the box in that sense. And we started talking earlier about kind of this disconnect between the fairness metrics and real world application. does that disconnect get closed by you know A looking at systems in their entirety and not just models and B by looking at runtime in addition to design time or are there other gaps that teams need to be looking at? Yeah like so I do think that um it's it's interesting for me to have one leg in industry and one leg in academics hey if i put my academic set on uh we we suffer from let uh like i'm doing all this call i apologize let me explain this to you one more time because you you probably are not getting it yeah but if you flip it around i think if you almost like uh an anthropologist would do field research if you would do AI ethics rather than talking about it, you'll find out what the real issues are.

36:15So I do think that actually by getting into the field, only then you will find out the bigger issues. And these bigger issues, sometimes they have these things like, oh, culture, organization, blah, blah, blah. But sometimes they're actually a nice data science research puzzle. So like how to deal with tons of alerts, that's a nice data science research puzzle. Or how could I, is there a way to continue to check for fairness bias continuously? Nice research puzzle, yeah? So, or data science puzzle. Yeah, so I do think it's healthy. There's other aspects as well. I don't think we need even more fairness metrics, you know, like we have enough of them.