Suspicion machines ⚙️

0:28Welcome to Practical AI. and database close to your users. No ops required. Learn more at fly.io.

0:42Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder and CEO at Prediction Guard. We have some really exciting stuff to discuss today. So thankful to have my guests with us today, because there's a lot of talk about the dangers of AI or potential risk associated with AI, which we've talked about on the show. But I think maybe that kind of misses some of the actual real world problems that are happening with deployed machine learning systems that maybe have been going on for longer than some people might think. And maybe we can learn some things from those deployed machine learning systems that would help us create better and more trustworthy AI systems moving towards the future.

1:32So I'm really pleased to have with me today, Justin Braun, who is a data journalist at Lighthouse Reports, and Gabriel Geiger, who is an investigative journalist at Lighthouse. Thank you both for joining me so much. Thanks for having me. Thanks so much for having us. Yeah, yeah. Well, like I mentioned, I kind of teed us up to talk a little bit about maybe risks or kind of downsides of deployed machine learning systems. And you both have done amazing journalism related to what you've kind of titled here suspicion machines. and I think it would be worth, before we kind of jump into all of the details of that, which is just incredibly fascinating, if you could give us a little bit of context for both what you mean by suspicion machines and how this topic came across your desks and you started getting interested in it.

2:26Sure, I can start with that. I mean, the reason that we chose the suspicion machine as a title for our series is it's kind of a driving metaphor for what these specific machine learning models are doing within the welfare context. So a while ago, we wanted to investigate the deployment of machine learning in one specific area, but we're not sure which one yet. So in the US, there's been a lot of reporting about the use of machine learning or predictive risk assessments within the criminal justice system. Also in facial recognition and us over in Europe, looked at that reporting and noticed that there's a big lack of it over here in Europe.

3:03And so we were exploring different realms and settled on looking at welfare systems, which is a sort of quintessentially European issue, if you want to say. And in the last decade, welfare systems have become this sort of polarizing political battleground within Europe. How much welfare should we be giving out? Are people defrauding the state? How much money? And so we wanted to hone in on this one area to make it sort of manageable. And we decided to investigate the deployment of predictive risk assessments across European welfare systems. And basically what these systems do, I mean, they vary in sort of size and color, but the basic sort of mechanics remain the same, is that they assign a risk score between zero and one to individual welfare recipients and rank them by their alleged risk of committing welfare fraud.

3:52And the people with the highest scores are then flagged for investigations, which can be quite punitive and where their benefits can be stopped. So we landed on this metaphor of the suspicion machine because we felt that these systems were oftentimes essentially laundering or generating suspicion of different groups who were trying to receive welfare benefits that they needed to pay rent every month. And when you all started thinking about these suspicion machines, these deployed machine learning systems, were there existing examples, like concrete examples of how these suspicion machines were being punitive, maybe in either biased ways or just like in the kind of false positive error sort of way that's creating problems for people that it doesn't need to create?

4:43Was there actual evidence at the time, or was it just a big question because there wasn't any sort of quantitative measurement? So there was signs of it. So in the Netherlands, specifically, there was a case where 30 ,000 families were wrongly accused of welfare fraud. And it turned into this huge scandal called the childcare benefits scandal and eventually led to the fall of the government. And it turned out that the way that these parents were wrongly flagged for investigation was because of a machine learning model that the agency had deployed. But there was no sort of quantitative measure of what that model was actually doing.

5:18Nobody took it apart and actually looked inside and saw, okay, well, why was it making these decisions? Which is, you know, a huge reason why we as Lighthouse decided and were so adamant about the idea of we're not just going to investigate these systems or the classical journalist methods, you know, call people up sources, you know, getting contracts, but we actually wanted to take one of these systems apart. And that was sort of the big challenge or hurdle in our reporting. And I think Justin can maybe talk to what's sort of the existing literature on these predictive risk assessments. Yeah, I think my interest in the topic comes kind of from the broader discussions around AI fairness that really started after ProPublica published its machine bias piece six or seven years ago.

6:04And in the aftermath of that, there were a bunch of systems that worked in a similar way that were kind of discovered in various contexts. I myself worked a little bit on predictive grading systems. So during the COVID pandemic, some school systems replaced their previous, you know, handwritten exams with an algorithm that tried to predict based on previous exams, how well somebody would score in their final exam. And with each of these systems, the issue that emerges is essentially similar. Once you try to classify people according to risk and you have a training set that's not a perfect representation of the true population, you'll start running into issues like disparate impact for different groups, which is kind of the most hot button issue.

6:48But you'll also start running into how representative is your fairness data. In general, you'll start running into issues with, you know, where do you set the threshold? What values are you trading off when you set the threshold higher or lower? And so I was generally interested in that and then kind of joined Leidos at a point when Gabriel and some others had done a lot of the groundwork already to see whether there was something there in welfare risk assessments and then kind of took on the technical work from there. One question I have, just like even as a data scientist, just thinking like, okay, where do I start with this?

7:18This model is deployed by some entity. In theory, it's been developed by some group of technical either engineers or data scientists or whoever it is, where do you go about actually starting to find out, like, where does this model exist? Who has the serialized version of this model sitting on some disk somewhere in some cloud? Or like, yeah, where do you even start with something like that? So now there's a sort of trend of having algorithm registers where public agencies across Europe publish what different types of algorithms or models they're using. But that didn't exist when we started this reporting.

7:55So what we did was we made use of freedom of information laws in the US, I think they're called sunshine laws. And we started sending in these requests, trying to figure out at least where are you using predictive modeling within the sort of welfare system? Because, you know, you could be using it to look for fraud, but you could be using it for other things as well. And we started sort of slowly building out this picture of which countries were using predictive modeling at different places in their welfare system, and then sort of start slowly building a document base. So maybe we'd ask them for, we didn't start by asking a lot of times for like source code or final model files or training data.

8:33We'd start by asking for, can you give me like the manual for your data scientists for retraining the model every year? And that would allow us to ask for more specific documents and more specific questions like, okay, we know that there's a document called performance report 2023.html because we see it referenced in your manual for your data scientists. So we can request that. And then sort of built up to this place of, okay, now let's request the final model file, the source code to train it, ask for the training data, which we can get into because there's some prickly things there around data protection laws in Europe.

9:07So we kind of tried to do this tiered approach to sort of build for that final ask for asking for the model once we could make sure that our request was specific, because oftentimes agencies would try to resist our requests, saying they were too broad or we weren't being too specific enough, or trying to argue that disclosing certain documents could allow potential fraudsters to game the system. I've got to ask, as you did this sweeping look at how predictive analytics was actually deployed across Europe, even before we get into the specific case that you studied, are there any takeaways or trends that you saw in terms of how machine learning is actively being deployed by government entities or by welfare entities across Europe?

9:54Yeah, so I think it started essentially a bit later than in the United States. You kind of have this trend in policing, in kind of risk analysis. I would say that begins in the early 2000s, where you kind of have, I mean, semi-governmental organizations doing, credit risk scoring, the first kind of instances of predictive policing, also more serious thinking around big data mining for some risk analytics in the welfare context. And then I would say there's a bit of a bifurcation. So you kind of see some instances where big industry players, Accenture, the volunteers of the world, right? Like these big, companies hype up the case for big data analytics to be deployed across different sectors.

10:43And at the same time, you have a lot of failures when those tools are deployed. They often don't work very well. People who have to use them in the agencies don't know how to use them. You see some agencies that drop those systems. And at the same time, you see other agencies that kind of build up internal capacity and build those tools themselves, sometimes in collaboration with universities or smaller startups, but you kind of have these two pathways that continue to coexist at the same time. I would say in terms of the systems that we looked at, most of them were developed kind of from the early 2010s onwards.

11:19It's definitely gotten a lot more in the last five or six years. And across the eight or maybe nine countries now, I'm not quite sure how many we've looked at, but I think we've only seen a single country where we did not see evidence of predictive analytics being used to assess risk and welfare. Interesting. So I guess on the other side, I asked the question about evidence of these systems prior to your reporting evidence or cases where these systems maybe behaved in ways that caused harm or issues. On the other side, you mentioned this kind of hyped perception, potentially hyped perception of what these systems could do in a positive way.

12:00I mean, the main case for using these systems, as you mentioned, is to kind of catch fraudsters, from my understanding. On that side of things, is there evidence that, hey, yes, this type of fraud is a huge problem that we need to invest kind of advanced technology in solving? Or is that also kind of up in the air in terms of the, I guess I'm getting at the justification for using these types of systems on this sort of scale? This is one of the questions we try to address in our reporting a little bit. First of all, distinguishing between deliberate fraud and unintentional error is really messy and difficult.

12:38I mean, how do you prove intent? How do you prove that someone intentionally didn't report something? I mean, there's clear-cut cases where it's like criminal enterprises defrauding the welfare state using identity fraud. Okay, that's pretty cut and clear. But when it's individuals or family and they didn't report 200 euros, you know, is that intentional? Is that not intentional? How do you prove it? So that's already a challenge. What we did see is evidence of a lot of the larger consultancies tending to overhype the scale of welfare fraud and these estimations being criticized by, let's say, like academic studies.

13:08And, you know, when national auditors like the National Audit Office of France, for example, actually did, you know, random surveying to try to estimate the true scale of welfare fraud, they estimated at about 0.2 % of all benefits paid, whereas consultancies will estimate it at about 5 % to 6 % of all benefits paid. So there's a little bit of this situation where they're hyping up estimates to sort of sell the solution. At the same time, fraud does happen within the system and our reporting isn't meant to try to dispel the notion that fraud doesn't exist. But I think it's definitely still unsettled science on what the actual scale of welfare fraud is.

13:49and whether these systems that are being deployed in places like the case study we looked at are actually catching fraud or just catching people who have made unintentional mistakes and that these unintentional mistakes are being treated as fraud. To add on to that a little bit, I think the added justification that is often being used is that actually these systems are more fair than analog equivalents, that by using a machine you get rid of biases, and that they're better at detecting fraud than people are. And I think, as we'll probably get into later, there's good reasons to doubt both of those propositions.

14:24All of that was really good setup for this particular case study that I think you've highlighted in some of your recent work. I'm wondering if you could kind of set the context for the particular case study that you focused on, the particular model that you focused on, in light of what you were just talking about, about kind of scanning the environment, I guess, through information information requests and freedom of information requests to understand where things were deployed all the way down to like getting your hands on a model. So how did that transition happen? And tell us a little bit about the use case that you studied more deeply.

15:05As I mentioned earlier, we started by sending these freedom of information requests across Europe, eight or nine countries, and we started receiving a patchwork of responses back. So some places just said, no, we're not going to give you anything at all. Some places would be like, okay, we'll give you the manual. But then when you tried to ask for anything like technical, like code or a list of variables, they shot it down. But there was this kind of one exception in all of this, and that was the Dutch city of Rotterdam. And Rotterdam had deployed one of these predictive models to try to flag people as potential fraudsters and investigate them.

15:40And right off the at Rotterdam sent us the source code for the training process for their model. And we got really excited at first. We were like, wow, this is great. We started looking through the code and we noticed that when the scoring function in the code goes to load something called the final model.rds file, and we go looking through the directory and we notice, huh, wait a second, this final model.rds file, the actual model file that can be imported to the score isn't in the directory. So we emailed them back. We say, hey, guys, I think you made a mistake. There's this final model.rds file missing in the code directory, so we can't actually run anything.

16:20And they go, oh, well, yeah, psych, but you're not getting that one. And their justification for this was that if this was made public, potential fraudsters would be able to gain the system. So long story short, we went on this year-long battle with them to attempt to get this model file. and eventually the city, to their credit, decided to disclose this model file to us so we could actually run it. And what does this model do? I think Justin can do a good explanation of what this model actually does and how it works. Yeah, so it's a grading boosting machine model. It's a pretty standard machine learning model.

16:59It ingests 314 variables and it outputs a score. The issue that we ran into very quickly once we had access to this model is, well, what does this actually tell us, right? Okay, we can make up a bunch of people now and score them, but how do we then know what that means for those people? And so there were kind of two things that became important to figure out at that point. One was what do realistic people look like? And the second was what is the boundary at which a person is considered high risk. The second one was relatively easy to figure out. We kind of had some broad estimations of how many people are flagged each year.

17:42We could run some simulations and kind of see the distribution of risk scores. And at that point, we could take a good guess for what the threshold would be. Getting access to realistic testing data was a lot more challenging. And for a while, we thought we would have to just simulate a bunch of people, you know, take guesses. But But actually, Gabriel had requested some basic stats about the training data at an earlier stage. He essentially asked, look, can you tell us, give us a histogram for each of the variables so we can see what the broad distribution in the training data for ages, for instance, or for gender and so on.

18:19And our idea was to use those basic distributions to sample new people. But when I was meant to type all of this stuff down into a file so we could then run those simulations, I got lazy and I wanted to just scrape the document. And so it was an HTML file. So I opened it up and inspected it. And it turned out that the entire training data was contained in this file, which happens when you create plots with Plotly quite often. So if you want to leak something to a journalist, that's a good way to do it. There you go. Yeah. So we got a backstone and got access to the entire training data. At that point, the question became, okay, now we know what realistic people look like.

18:57What tests can we actually run in terms of figuring out who does this model flag at higher rates? Does it have justification to do so and so on? And the one thing that was missing from the train data was that we didn't have access to the labels itself. So we knew, you know, your age, your family background, your job history, that kind of stuff. But we did not know if you had actually committed fraud or not. And that meant that, and this is the big limitation of our story, but that meant that we could essentially only understand which characteristics lead to higher or lower scores, but we wouldn't know if those scores are erroneous at higher rates for one group rather than another.

19:33So I just want to be very open about that. That is a limitation of the design. But having access to the train data, having access to the source code, being able to see how the train data is constructed, having access to the final model file, all of that allowed us to investigate a bunch of aspects with the system, which I think still made for a very valuable story, both in terms of explaining how this stuff works, but then And also in terms of showing that there are likely consequences which seem to be discriminatory against certain groups. Probably a lot of our listeners will be familiar with what a gradient boosting machine is.

20:04And sort of like maybe this is one of the tutorials that you ran on a Jupyter notebook when you were first taking your kind of data science 101. So the model, I think, is very familiar. I think a lot of the interesting things here are related probably to the model features and that sort of thing. Did anything jump out to you maybe even before you kind of ran a kind of larger scale analysis in terms of like the features that were included in the data set and how those may or may not like intuitively be connected to this sort of welfare fraud situation? Did anything jump out when you were kind of doing your initial discovery and kind of exploratory data analysis on this data?

20:51Yeah, for sure. Though I think it's maybe important to preface this with saying that including features that seem discriminatory does not automatically lead to discriminatory outcomes. And I think that is sometimes being confused, right? You can get discriminatory outcomes without features that look bad, like, I don't know, racial or gender, racial background or gender or something like that. But it also works the other way. You can include a bunch of these features and not get any discriminatory outcomes, right? Both of these things are possible. That being said, there were a bunch of features that seemed perfectly reasonable.

21:25You know, contact with the welfare agency. How often have you been there? Have you missed any of your appointments? That kind of stuff. There were a lot of demographic features. And I think those get into trickier territory. Some like age are maybe justifiable on some level. Gender gets a bit harder. And then a lot of features measuring through proxy, but measuring ethnic background through language skills. I think there was 10 or 12. Gabriel, correct me if I'm wrong, but definitely a lot of variables on language skills. No, I think 30 or something. Oh, 30. Yeah. Yeah. Because it measured everything from like your Dutch, like spoken Dutch fluency, writing Dutch fluency, the actual language you spoke.

22:03So there was like a categorical variable with like 200 values or something. So it got as granular as the specific language you spoke, whether you speak more than one language. But anyways, continue, Justin. Yeah. And then I think in some way, the weirdest set of variables were essentially behavioral assessments by the caseworkers. So we actually got access to some of the variable codebooks. And in there, it said that there was a variable that essentially where people were meant to judge how somebody was wearing makeup, especially for women. So stuff that just seems really sexist. So those variables were included, which is problematic in and of itself.

22:42But then the way they were transformed in the preprocessing steps was that essentially this textual data was just transformed into a zero one variable, depending on whether there was anything in this field or not, which is also, I mean, you just lose a bunch of maybe the more interesting information if you do that. But I think that set of variables, because it's just based on individual caseworker assessments, if your claim is that the system should lead to reduction in bias, and then you include these variables that are so obviously subjective, I think that kind of undermines your claim right away.

23:15And in the data set, like in terms of the label and the output, were you able to understand at all, like, oh, these are investigations that happened that actually were verified to be fraud or not, essentially like a one or zero type of label? Or how is that set up? Yeah, so we did not have access to the label, which is, again, the big drawback. So we could only score people who we know they had labels, but we didn't have that label ourselves. Gotcha. But we did a bunch of ground reporting to essentially work around that. And maybe Gabriel can speak a bit to that. Yeah, I mean, two things first.

23:52I mean, just to talk about how the training gate is constructed first. It's over 12 ,000 past investigations that the city has carried out. And these past investigations are not a random sample. So there's some subset within there that's random, I think about 1 ,000. But all the rest of the cases are just where investigators have looked at the past, either through anonymous tips or through these kind of theme studies that they do where they say, this year, we're going to check every man living in this neighborhood. So it's not a random subset of people that they're training this model on, which is problematic in the first place.

24:23The second thing is that this label, yes, fraud, no fraud, doesn't distinguish between intentional fraud and unintentional mistakes, right? So these are flattened into the same thing when labeling the training data set. So those are, I think, two problematic things right off the bat. I think an even third, more complicated thing is that the law for what is considered fraud has actually changed over time. And this training data spans back 10 years. But all that aside, one of the things that we wanted to do with this reporting was to look at the impact of being flagged for investigation. What does that mean for a person?

25:01And how are they treated by the system? And so we did a bunch of ground reporting in Rotterdam, and we sort of used the results from our experiment to build profiles who would be considered some of the most high-risk people. And we saw that it was, you know, one of them at least would be like single mothers of a migration background who don't have a lot of money, financially struggling, living in certain majority ethnic neighborhoods. So we did a bunch of ground reporting in those places and found people, and it was quite challenging. People were quite afraid to talk. People who had been investigated in the time span that the model was active.

25:36And what we found was that they were treated incredibly punitively by these investigations from the city where fraud controllers are empowered to raid your house at 5 a.m. in the morning unannounced, count your toothbrushes, sift through your laundry, go through all your bank statements, and that even the smallest mistakes, like forgetting to report 100 euros, could leave you landed as an alleged fraudster. So I think there's even, you know, based on reporting, there's reasons to even question the validity of the label and the consistency of the label. But beyond that, I think what we established for the reporting is that the consequences of being flagged, even if in the end, you're found to be completely innocent, just having people, you know, raiding your house at 5am, asking you questions about your romantic life in front of your children.