1013: Weapons of Math Destruction, Ten Years On, with Dr. Cathy O’Neil

27:15I watch it like once a year. I watched the entire series again because I never actually remember. I have this wonderful memory that like fails to remember who killed who and why, but, um, I just love Columbo so much, but like, I love all those scenes where he walks into it, like an, like a business, like a, you know, a magazine office. And there's like 40 secretaries in a room and each executive has at least one secretary, but maybe more than one secretary, like you never see that anymore. Right. So what happened to those people? Um, this book explains it.

27:53Jon Krohn:And, and so, uh, the book I found it, it's, it's called labor and monopoly capital. So degradation of work in the 20th century, Harry Braverman. Um, but interestingly, it actually builds on the book that I first found called monopoly capital. Uh, so, uh, Yeah, so the first book, Paul Buran, Paul Sweezy, Monopoly Capital, and then this Labor and Monopoly Capital book by Braverman. Apparently, I mean, I'm literally just reading the opening paragraph on Wikipedia, but according to that opening paragraph, it's somehow related. And the Labor and Monopoly Capital book is obviously newer, 1974. Yeah, and the Labor and Monopoly Capital book, which I'm looking at on my shelf right now, so you'd think I would have been able to remember it, is itself like a neo-Marxist.

28:41So it works, it relies on some of the notions that Marx himself came up with, which many of them were of his time and didn't really make sense in this world. So in some sense it was like, okay, let's extend this. And by the way, John, like the book I wanted to write that I decided would take too long because it's four years, you know, is basically like, I want to take Braverman's book and just rewrite it for the AI, the world of AI. I want to extend it another 50 years. He wrote it in 75, by the way, it came out in 75. And in 75, he was talking about what was happening in 1925 with Taylor. And I wanted to be like, okay, 2025, you know, what's happening now.

29:24I still think that's a book that could be written and it could be written in a way that doesn't age as quickly as the world is. I mean, you know, because a lot of these ideas are very evergreen. Like, you know, workers being degraded, humiliated, dehumanized, and replaced by themselves, but paid less and with fewer skills, right? Like we're not really being replaced by AI and that's not going to happen, but I do think we're going to be degraded.

29:56Jon Krohn:you know i think my perspective on this podcast the the line that we generally toe is this uh techno optimism but i do also definitely uh acknowledge there's lots of negatives well what is your techno optimistic take on this let's have a let's have a discussion yeah i mean it's tough to it's tough to come directly out of you know these kinds of things that you're describing i i don't disagree with anything uh that you've said in the episode um and i definitely i mean i experience personally the way that uh so we recently did we recently crossed over a thousand episodes and for episode 1001 the guy who founded this podcast and hosted it for the first four years so the show's 10 years old uh for the first four years and a bit um the host was kirill arimenko he founded it he still owns the the podcast business with me but I've been hosting it now for about six years.

30:52Jon Krohn:For episode 1001, Kirill interviewed me as kind of this like role reversal where I'm the guest for the first time, or actually, I guess I was, I was back when he was host before there was any discussion of me becoming host. I was, there was an episode in the 300s where, um, where I was interviewed, but you know, so 700 episodes later, I come back on, I'm interviewed by him again. And the kind of headline thing that I think ended up being the YouTube thumbnail And the title of the episode on YouTube is how I spent 20 years getting a PhD focused on machine learning applications and then in industry, in financial markets like you as well, then in digital advertising and in startups.

31:40Jon Krohn:And so had this kind of moat around my technical skills that I developed over all these years. And now it makes, it would be crazy for me and it would be crazy for a lot of people to not be using a tool like Claude Code to rapidly be able to test different machine learning ideas. There's very little point in me today kind of hand typing keystroke by keystroke my Python library imports and a training loop. I can do that if I enjoy it kind of for fun, but not really on the clock, if that makes sense. And I can definitely say that a lot of the technical moat that I built up around myself for many years has disappeared.

32:23And I'm going to be completely honest with you. I am kind of a techno-optimist. I, you know, I believe technology is absolutely amazing. And, but, but we, we have to, we have to be very careful about what we're optimistic about. I'm optimistic about the power of technology to do amazing things. I'm not optimistic about our choices, political choices, really like geopolitical choices on how to treat people as this stuff is happening. You know, I'm not a fan of China, but I heard, and maybe you know more than I do about this, John. So just tell me if I've got this wrong. But I've heard that the Chinese government has told people, do not replace workers with AI.

33:09And a few workers were replaced by AI, sued, and won. They got their jobs back. And I'm not saying we do that because that's, it's very much like not free marketing. But it, it, it points to something, which is that this is a choice. Like we are choosing how to treat people. And it's, it's a political choice and it's a cultural choice. And we don't have to make the choices that we are making. But unfortunately, the people who are in charge are just like very immature, selfish people who do not feel at all obligated to a social contract. And so the question for me is like, how do the workers who have solidarity, hopefully by this point or soon, how do they like press their power towards the very small cabal of tech executives who are making all the decisions?

34:12So at the end of the day, it's like a question of power and it's a question of like voice. It's a political question. but it's not really anti-technology. Like, I mean, imagine, um, imagine like if we had really great, this is kind of the question of like, do we want, do we like democracy or do we want like Kings? And the answer is always like a really great King would be better, you know, because Congress is, Congress is always mired in indecision and infighting and stuff like that. There's all sorts of terrible inefficiencies by design, really, in our government. We'd rather have a really good king, but we don't have a good king.

34:55Or you can't depend on a king to be good. And right now what we have is, you know, we have is like a very small group of non-good kings in the world of technology.

35:06Jon Krohn:But if we had, so it's, again, my point being is that it's almost neutral to the question of technology itself. The argument that you're making there allows me to believe that there's comfortably a book, you know, a Weapons of Math Destruction 2. It's kind of interesting. I guess the title maybe doesn't even work as well now in the 2020s or 2030s, because it really plays on the Weapons of Math Destruction thing, the WMD thing. It was a big news item in the Bush era, which maybe some of my younger listeners are not even really like on top of. Yeah, exactly. No, it's definitely an old pun. You know, a Weapons of Math Destruction 2 style book that, you know, updates the Taylorism concept from a century ago, as you say.

35:49Jon Krohn:And by the way, I'll have links to articles on Taylorism as well as the Labor and Monopoly Capital book for our listeners, for sure, to follow up on that. In case people are curious, right off the bat, the Taylorism comes from someone, Frederick Winslow Taylor, who has passed away more than a century ago. But yeah, exactly. The resources I'll point to, they talk a lot about the management style. And actually, this is a bit of a tangent, but do you know Kurt Vonnegut's first novel? It's called Play Your Piano. Don't know it. So Kurt Vonnegut imagines a future, maybe like the 1980s, the 1990s, the distant future from the perspective of the 1950s, where the kind of computing technology that was new in the 1950s.

36:41Jon Krohn:And so using magnetic tape to store information and having punch cards. And so he basically, he has factories where it's just magnetic tape and punch cards that are running everything in the factories. And you just have a few people with PhDs. So even, you know, executive assistants in his, in his imagining have PhDs because there's so few jobs available for people to have anymore. I love the idea that executive assistants still existed, but like, yeah, It's like he saw data centers before anybody else. That's great. I mean, by the way, a lot of, I mean, Kurt Vonnegut is a positive example of this, but a lot of the problem, I think, I mean, this is just me projecting, but like a lot of the problem, I think coming from the political decisions of, or the sort of moral obligations that aren't being felt by the tech overlords comes from the fact that they read a bunch of dystopian science fiction when they were young and they thought that was like, cool.

37:47Jon Krohn:You know, they're like, let's make that happen. Like, cool. Let's Terminator. Yeah, let's do that. You know, it's almost like they're just like, they watched Star Trek or so they watched Star Wars and they were on the side of the empire. Like they, you know, they just didn't get it. they didn't understand the prime directive. They didn't understand the spirit of the prime directive. By the way, when I was, I have three sons and when my older two sons were little, I would make them watch star Trek, the next generation with me. And then we went on to like Voyager and like deep space nine. We went to all, we would did all of them, even the enterprise, which I love.

38:24I still love that. Especially the, um, the theme music is excellent. But, um, And we were just, for some reason, last night around the table reminiscing about how wonderful the lessons learned there were about, you know, I remember because we were talking about how Riker was always like you could count on him to like, if needed, if needed, have sex with an alien. You know, if that was going to get us out of a bind, he'd be like, OK, I'll take one for the team. Sometimes he did it literally because he's like, this is how we're going to save the crew. But the other times he just fell in love. And we were talking about this like one planet where people weren't allowed to have a gender, but like this one alien was like, I secretly feel female.

39:08And like, he was like, let's do it. And they like went off to the, you know, holodeck or wherever they do that kind of thing. um anyway i'm just like it just i'm like circling back because i like on a tangent of a tangent but um yeah we kurt vonnegut saw into the future as many science fiction writers do um and the question is do they have enough like signage in their in their um in their imaginings and their short stories or or novels to say like this is not a good idea

39:44Jon Krohn:don't do this yeah and i mean that is that is the whole premise of player piano and of course the player piano name of course comes from you know recording somebody playing a piano and then the piano just playing by itself and the idea is that these factories are full of like you were describing the metal workers where you find the best metal worker in the factory is the idea in player piano and in real life and you use at you know to his imagining the 1950s magnetic tape and and punch cards to remember exactly how the metal worker did what he or she was doing and have that be done perfectly. And now, and so then you have this like, there's this town of people, they basically have something like, they have universal basic income for everyone.

40:28Things break down.

40:29Jon Krohn:So there's all these people, the vast, almost everybody's on UBI and they're not happy. They're very unhappy. They have no sense of value. They feel degraded. and they come up with some schemes, which I don't want to ruin any plot points, but basically, you know, the people, you know, try to come up with a plan. Is it Luddite-esque? Exactly, a Luddite-esque, yeah, without trying to spoil anything. But something that's interesting in the book is, so you talked about executive assistants because basically in the book, Kurt Vonnegut, he mostly fleshes out this idea of blue-collar work being automated.

41:06Jon Krohn:But there's an interesting chapter, which I was hoping he would get into more later in the book, where the main character in the book, he's a manager of one of the biggest factories, and he is an outstanding backgammon player. And he can beat all the new recruits at backgammon. And so whenever new people join the company, there's this kind of ceremonial thing of him beating them at backgammon. But this group of young engineers who are new to the company create a backgammon machine, which of course is a huge thing that has to be rolled in. And it appears that it's going to beat this backgammon star, the main character of the book, at backgammon.

41:55Jon Krohn:But then somebody kind of like literally throws like a wrench into the machine or something to cause it to go haywire. and there's a line around that time about how it's that guy, the main character, this engineer saying to his executive assistant, oh, how they're lucky that their work could never be automated, that only the blue collar work could be automated. And so I feel like he was kind of like touching on that with the backgammon thing, but it never gets fully fleshed out. He was still brilliant. I actually had the pleasure of seeing him speak at Cody's Books in Berkeley when I was a college student there.

42:32Jon Krohn:that is amazing i'd you know he's he's my favorite fiction author and uh i think i've read i've read i think i've read all the novels i haven't read all of his essays yet but um uh or all of his yeah his shorter form content yet his short stories but um i yeah i think he's he's really funny and one of the things this is now we're on such a weird tangent here just basically talking about kerfona again just hang it one of the things that i really like about him is that he doesn't really ever have bad guys. That's a really important point. Someone actually asked me the other day, if you had three bullets and you can kill three people and you're not gonna get in trouble for it, who would they be?

43:13And I was like, nobody, I'm not killing anybody. These people are all products of the system. They're products of this terrible shit system that we need to get better. We need to improve the system. I actually, having been at a hedge fund, having spent time around very rich people, billionaires, like they're totally socialized to be absolute jerks. And look at me editing my words here. Um, you know, they're, everybody tells them how smart they are. Everybody agrees with everything they say. Can you imagine being in that environment for longer than a week without becoming a flaming jerk? Well, it happens.

43:54It happens to every single one of them. They also stop caring about other people. They, they're megalomaniacs. And you've seen this. I'm sure, I'm sure we've all seen this. We've seen this happen in front of our very eyes to the people who were just like one time Harvard undergrads with like a techie nerd idea. And then they become who we see today. You know, um, it's, it's the system. It doesn't, you don't need bad guys. And that's, what's actually kind of scary about it. It's scarier that way. Bad guys are like defeatable. And that's one of the reasons I'm not a superhero fan. Like I don't watch those superhero movies.

44:33It's just none of it works. It's not how bad shit really actually goes down is like where Superman could intervene and put an end to it. Just not true.

44:44Jon Krohn:You and I are on exactly the same page in terms of our perspective of how the world works as well as film and fiction novels. and then it actually ends up tying really nicely to weapons of math destruction. So we ended up being on long tangents, but we circled all the way back. Wow, look at us. Because it is basically this idea that we have just over time created systems, Vonnegut-esque systems that are kind of not doing the vast majority of people a favor. So Vonnegut, a lot of it is about war, but a fair bit of it is justice at work and other kinds of things like that that weapons of math destruction tackles head on.

45:26Jon Krohn:But in all of it, in the real world, in weapons of math destruction, in Vonnegut books, it is a system that kind of gradually evolves into what it is that is creating injustices and recapitulating injustices, having LLMs memorize injustices and regurgitate them. Yeah, so I guess it's kind of nice that we kind of come full circle. Now is maybe a great time to talk about what you're doing to try to change the system a bit. So you have an algorithmic auditing company called ORCA, which stands for O 'Neill Risk Consulting and Algorithmic Auditing. And then you also have a nonprofit, Ocean, which kind of recursively has ORCA as the first O.

46:10Jon Krohn:You like that, right? Yeah, exactly. ORCA Collaborative Expert Assistance Network. So yeah, so tell us about ORCA and Ocean and how you're using them to try to make a difference. So that hopefully, as you were talking about how in four years, the technology will change, maybe saying a word like chat GPT or Claude will feel weird in four years and we'll be like, you can't have that in a book that's published four years from now because it'll look so antiquated. It's kind of sad that some of the systemic issues will probably be the same in four years, if not worse. Absolutely. But so in what ways are you hoping that Orca and Ocean could at least chip away a little bit four years from now and we can be closer to a Star Trek Enterprise world?

46:53The Star Trek vision is really, yeah, very dear to me. And because it's abundance, you know, it's an abundance, like mind frame mindset. And it's like sharing with other people. Of course, they have infinite resources. So that helps. Um, so Orca, I started when my book came out because I was like, oh, people are going to be really worried once they find out that their hiring algorithms could be, um, against the law because of the anti-discrimination in hiring laws. And they're going to want to hire me so that they don't get in trouble. And like, not really. I was imagining, and by the way, I'd also, because I'd worked in finance, seen how bad rating systems make things worse.

47:39So like the AAA ratings and mortgage-backed securities made things worse. They were mathematical lies, but they propped up a loaded housing market. It made it into a huge bubble. We saw the financial explosion of that whole system. And I was like, we got to do better than that. So it was like a dual goal of setting better standards for algorithmic auditing and like helping people address their risk of getting regulatory pushback or getting sued. I started this in October 2016. I think you can see where this is going. Like pretty quickly after that, I was like, oh, like maybe people aren't going to be so worried about that.

48:21And by the way, I'll just back up one second. I was an independent consultant while I was writing weapons, trying to make a little money as a data scientist. And I started working for the attorney general of Illinois on a payday loan case. So it was like consumer fraud. And we forced them to settle. They had this fake insurance product. I'm allowed to talk about this. It's one of the very few things I'm actually allowed to talk about because it was actually settled and it's public, but it was this payday loan company that forced people to pay for fake insurance products. And so everybody who worked there, like so-called quote unquote forgot how their business ran once they were deposed on the stand.

49:02But like, because I had all the data, I could sort of reverse engineer the entire business. And that was really exciting. Right. So I was like, okay, now I can figure out like, how do they treat people and how, like, how much this, like the usury rate in Illinois, by the way, is like 36%, but the effective APR of these loans was more than 500 % after you added this non-optional insurance product. I could also prove that it wasn't really acting as insurance because it was never used, blah, blah, blah. Long story short, there was this killer little graph I built for the judge. And the judge was like, okay, you got to pay these people back.

49:39And then I helped them figure out restitution. And that was also very, very exciting. But one of the takeaways there was like, well, if you have the data, you can do a lot. You can really figure out a lot. And by the way, that wasn't even an algorithm, but it was essentially an algorithm because it was like a company that followed rules. And so it was a bureaucracy. So when I, by the way, when I say auditing algorithms, I'm really auditing bureaucracies. Like how does this bureaucracy work? How does it treat people? Does it treat these people differently from that, those kinds of people? Like how does, you know, that kind of thing.

50:09So number one is like, yeah, this can really work. This, this is, this is a neat thing to do with my skills as a data scientist. That was first. The second thing was like, oh my God, I just went to that trouble of building that awesome graph with that beautiful metric of measurement of harm. It was completely convincing and persuasive to the judge, but now it's gone. It's just completely gone forever. Until I get hired by another AG for a similar kind of thing, and then I could resurrect this idea, build a new graph with the new data, see if the judge is convinced, blah, blah, blah. But really what we should do is make this a rule.

50:49And like every lending company should be forced to, um, like a, like almost like a, like a public filing, like quarterly statement. Here's, here's this, the measurement of this metric. Um, that wasn't happening, but I was like, you know, by the time I started Orca, I was like, this is the kind of thing that we should be doing in the, in the context of using a hiring algorithm, how do you measure whether the extent to which this is not sexist or racist or ageist or whatever? You can measure that. And why don't you just report that on a quarterly basis? Why aren't regulatory agencies who are in charge of anti-discrimination law enforcement doing that?

51:29Like, well, that's because they don't know how to do that, but I know how to do that. I'm going to show them how to do that. That was the idea of ORCA. Didn't really happen. It didn't play out as quickly and it still hasn't played out as quickly as I would hope. But I just want to say that just this week, John, I don't know when this is going to air, but just this week, Illinois passed a law that requires auditing of maybe their AI systems, but, you know, so maybe not all algorithmic systems, but so it's happening. Connecticut recently made a law, passed a law as well. And it's, these things are going into effect in 2027 and many of them require auditing.

52:11So I'm, I'm very excited about that. So that's ORCA. And we've, we've been working with lots and lots of law firms, lawsuits, like class action lawsuits, lots of AGs, some federal, federal agencies, but not now because for some reason, the federal agencies are no longer working on that stuff with us. But in the meantime, last year after Trump got reelected and our private clients disappeared, because usually we get private clients who are like worried about it, regulatory enforcement are worried about lawsuits. I got the third email from, from a representative of TikTok asking me to like help represent them in a lawsuit that they've been harming young women.

52:55And for the third time, I was like, hell no, I'm not going to represent TikTok because I actually do think they're harming young people. What I really want to do is work for the other side. And then I was like, oh, they're never going to approach me because they don't have the money to pay for my services because I'm not cheap. Although I'm not that expensive either, but it's just, you know, it's a lot for an individual who's been harmed to be like, hey, Kathy, can you do the data on this? So that's what ocean is. Ocean is like a reaction to be like, wait a second, we need to be reaching out to advocacy lawyers who are advocating on behalf of somebody who's been harmed and offered to do the data on this.

53:38And so we've been working with like Vitaly Jane, who, um, who's representing the parents of kids who's killed themselves after talking to chatbot AI chatbots. We've also been that part of Ocean's work is the podcast, AI Skeptics. I was on a, I was deposed for a lawsuit against an education company that was stealing student data and scoring them for various things against their parents' knowledge or permission. So that's the kind of work where like, just like lawsuits that really need to happen, but, but nobody's doing the data stuff on it. And when I say doing the data stuff on it. What I mean is this, somebody's harmed.

54:22They know they've been harmed. They're trying to make the case they're being harmed. A lot of the times the pushback from the tech companies are like, well, you probably were just getting harmed anyway. Like we didn't harm you. How do they prove that they were harmed? And the answer is I come along and I say, ask for the following data over time, the trends, blah, blah, blah. What I want to do is get the evidence together that this is happening not just to this kid, but is happening at a systematic level to an entire class of kids who are in a particular situation or whatever. The point is that when you do, and I know you know this because you're a data scientist, but most of the harm that comes about with respect to algorithms is statistical harm.

55:08It's actually really hard to prove a particular person was harmed by a biased hiring system, right? Like it could be biased against women, but I could be looked over for a valid reason. So to prove that I'm the one that's being harmed by this biased system is really difficult, but it's not that statistically difficult if you have the data to prove that it is a biased system. And then to estimate restitution, which is usually to the entire class of people who could have been harmed. Anyway, that's the, I hope that made sense. And that's what I'm doing with Ocean. And it's really exciting, except I'll tell you what, I've been hustling for 10 years now.

55:49I'm a like professional hustler for gigs, for like algorithmic auditing gigs. And it turns out it's really easy to offer free work to lawyers who don't have enough resources, but it's a different kind of hustle to get paid for by charities, like foundations and stuff. So that's what I'm learning now. It's like, that's hard.

56:15Jon Krohn:Regardless of how hard bits of it are, hopefully it'll continue to get easier over time. I don't know. I mean, it's, I have, you know, some combination of societal factors somehow getting better. And I realized I earlier said Star Trek Enterprise World. I actually meant to say Star Trek The Next Generation. uh which i mean star trek enterprise yes yes yes yes for sure but i think you started with the next generation uh on purpose and i would as well most of the ships are called the enterprise aren't they they are yeah yeah exactly not all of them like voyager wasn't yeah but i understood what you meant it's really it's really about the prime directive exactly yeah and i don't know like i probably naively to some extent you know like i understand you know individual issues but i maintain this, I guess, like, you know, I, in the nineties growing up, I had the next generation and would watch new episodes as they came out and still love it recently have been rewatching all of it, um, to try to have more, um, kind of optimistic ideas, like trying to have the opposite of what you described earlier with, you know, Silicon Valley people having, uh, consumed all of this.

57:29Yeah.

57:30Jon Krohn:Dystopian. That's exactly what I was looking for. Thank you. all this dystopian content and there's very little utopian content. The next generation is the, is kind of like the only, one of the very few examples. You know, we still live in this largely kind of dystopian world with algorithms kind of acting against us. And your latest book, which I want to give you just a few minutes to talk about before we wrap up here, your latest book, which came out in 2022 is called The Shame Machine, who profits in the new age of humiliation. And in this book, you talk about how shame has been weaponized into a destructive profit-driven industry.

58:12Jon Krohn:And it might kind of sound at first glance like that has very little to do with weapons of math destruction or algorithms, but in fact, it doesn't because it's the same kind of idea. Like you were talking earlier, and I agree a hundred percent around this idea that algorithm doesn't need to be confined to math or computer science ideas or data science, we can have algorithms effectively encoded into a business process or into a law, into a political system. And so in that sense, the same kind of thing. It's like shame is a tool, or rather than serving as a tool for community building, institutions and algorithms exploit public humiliation and personal insecurities for those classics of financial and political gain that you've been talking about now for over a decade.

59:03Jon Krohn:So I want to give you the chance to talk about the shame machine. And I came up with that notion because I just experienced it personally. And so did my children. So did so many other children. I saw so much, you know, resonance with this idea when I talked to young people, especially about their experiences on social media. This is before the chatbots came out. And I will talk about that in a second because it's related and things have changed since 2022 when this book came out. But the idea was for me, like, it's actually not new to make money off of shame. Skin cream has been shaming women for having wrinkles for centuries, makeup, you know, it's like, and it's not just that it's like, you know, weight loss products or, um, uh, are you getting old and senile?

59:58You should get this brain medicine. Um, are you smelly in your, um, particulars? You should buy this, um, you know, deodorant. So there's all, and that's old school. I would call that old school shame profiteering. But what I noticed was that the social media companies in particular had done something really tricky where, whereas with the old school, they would shame you so that you would buy from them and try to solve a problem that was actually not either not a problem or insoluble. So that was like a direct shame, like the direct tactic. We're going to shame you. You're going to buy stuff from us.

1:00:36Right. But social media did something really tricky, which is like, we're not going to shame you directly. We're just going to set up an amazingly perfect platform that is designed exactly so that you'll shame each other. And you'll sling mud at each other through shame. It becomes addictive. It becomes all consuming. And by the way, we'll also exploit you because we'll let those old school shaming products like target you with targeted ads. So it was just like an amazing, you know, just a framing. I guess really I wanted to do that at the very least is to make a framing that people would remember that when we were on social media, we're working for the social media companies when we shame each other, you know, when we're like giving into that design, which was made to exploit the way that we skewer each other.

1:01:26And, you know, to be clear, backing up a second, like shame is not a bad thing. We need shame. And so we spent, you know, like there's, you know, We would still have apartheid in South Africa if we didn't have shame because we couldn't appeal to law. The laws were unjust. So the only way you get through, you get past an unjust law is through shaming people internationally in that case. again, because I'm a thought experiment type of girl. Like I was like, what is, what are the requirements for shame to be valid? Like, when is it a good idea to shame? And when is it a bad idea to shame? And I came up with, for me, something that's been very useful.

1:02:07Um, since I came up with, and since I wrote the book, which is like voice and choice, like, does this person really have a choice about what they're doing? Like, do you have a choice to have wrinkles? If not, don't shame the person. Um, also do they have a voice to defend themselves? like it might look on this two, two minute, two second video that they did something terrible. Do you know what actually happened before that? Can they defend themselves? What's, you know, what's the voice. And if you think about it, like in the context of the constitution, if we accuse somebody of something, they should have defense.

1:02:38They have the opportunity to defend themselves, you know, meet their accuser. None of that seemed to be true for most of the shame trains that we were watching sort of develop. And so that was part of the book. And then, and also just to kind of try to urge people to stop doing the invalid shaming, which is like shaming people without voice or without choice and trying to get more into the, the valid shaming, which is shaming people in power who should know better and should be making better choices. So people in power have the choice and they have the voice because they're in power. So those are the people that we should be pressuring.

1:03:17And by that, I mean, you know, our representatives, our policymakers. And in general, like, I'm a very, very pro-protest. I said, you know, like, it really is exciting to see, like, what happened to the ICE agents in Minnesota and Minneapolis. Like, that was great. And that was shame. And thank God for those people. You know, they're heroes. And I'll just, one last thing I'll say, because I did kind of write that book for young people, like for their, you know, mental health. Um, one of the things that I find so disturbing about AI chatbots is that they are in some sense, refuges, um, from, uh, from social media, like where it's social media, you're like put on the spot, socially vulnerable, constantly battling and being, you know, shamed or being shown other people being shamed.

1:04:12So that's like a marker that you should feel bad. Like, it's like, Oh my God, I don't want that to happen to me. How do I conform? How do I conform? How do I conform? That kind of anxiety that it's provoking. It's almost, it's almost like the big tech companies built that. And then, and then they made everyone super neurotic. And then they built these little cocoons of, of like, you're safe with me, AI chatbots that do almost the opposite, but also in a harmful way. Just a different, but you know what I mean? They created this terrible place so that they could create this wonderful place. Neither of them is a good place, right?

1:04:50Neither of them is the, is the right place to develop into a normal person. You know, and I go, I sit on these panels all the time about AI safety. And I just get so frustrated to hear the AI folks defend the use of AI chatbots as friends or coaches. I'm like, oh, some kids need to practice asking their friends over for a sleepover with somebody who's not going to judge them or whatever. And it's like, no, actually they need to practice doing difficult things with their friends when they're 13. So that by the time they're 15, they're not as worried about that. Like the idea that you, that, that, uh, that a chat bot is going to, you know, help their anxiety is let me put it this way, John, when I get my hands on that data, it will not be borne out.

1:05:44Jon Krohn:That's what I'm hoping to do. A week ago, prior to recording on social media, I announced that you would be a guest on the show. And of course, it's a very popular post and we had lots of questions for you. And so we're going to actually, I've got three questions that I think we can handle pretty quickly and they're transcontinental. So the first one's going to be in North America, then we've got a European one, and then we've got an Oceania one. And the North American one is actually from Minnesota you were just talking about. And so this is from Sanjeev uh, Wyge. I hope I am pronouncing Sanjeev's, uh, family name correctly.

1:06:20Jon Krohn:He's a principal architect at a company called Optum in Minneapolis, uh, St. Paul. And he wants to know what we can do about these problems at an individual level, or, or even at a group community collective level, like what can listeners do to tackle the kinds of problems that you're tackling, um, through Orca and Ocean and that you discuss so much in Weapons of Math Destruction, in The Shame Machine and on your podcast. If I hadn't heard where he worked, I would have been like, yeah, that's hard. But like I actually went to Optum like many years ago when I first started Orca and I was like, you guys got to be worried about your algorithms being like racist in healthcare because of the history of black people not getting good health care in this country.

1:07:12And the executives at Optum were like, oh, we're not going to worry about that. We're not going to pay you to help us think about that. And then there was this huge New York Times front page story about racist Optum algorithms, which is one of the case studies that everyone refers to. And I don't know where this guy works, but is he a senior data scientist?

1:07:33Jon Krohn:Yeah, it says he leads teams in architecture, design and delivery of network infrastructure projects. So probably not, but I guess my point is you could start a, um, uh, you could start an algorithmic auditing group or lunch meeting at your company. Um, and you could say, what, what could be going wrong in the algorithms that we use at this company, especially when, as it, as it pertains to, to customers and to, um, patients in the healthcare system? That's a big question and it hasn't been addressed by this one New York Times headline. In general, what we need is accountability. I think the pushback against data centers is a good thing, but it's not directly related to the kinds of algorithmic harm that I typically think about.

1:08:21I wrote an essay which hasn't been published yet, but it's like, I'm still looking for a placement, but I'm like, the basic idea is like, juries of angry parents are our last resort because like right now it's, you know, well, maybe Illinois, maybe Connecticut, there's, there's, I'm sure there's constituency pushback against the AI kinds of harms that we've been seeing, especially among young people. But that stuff is pretty slow moving. It happens at the state level. It takes a year or two to go into force that the enforcement takes another few years. Like what I do see happening in a faster pace and all over the place, which is very exciting.

1:09:01And one of the reasons I started Ocean is lawsuits. One of the, I don't know if you remember a couple of months ago that the lawsuit against social media companies for a particular girl's anxiety and depression, like that won. and the companies had to pay up. Why did it win when regulation is at a standstill at the federal level? Because lobbyists didn't infiltrate the jury. It's a bunch of parents of kids who they see what's happening to their kids or their kids' friends and say, oh yeah, I believe this is happening. I believe this social media company is designed this to hurt young people, to addict them, to keep them on the social media all the time.

1:09:44And I believe that has a deleterious effect. So I guess what I'm saying is that like, um, as a, an individual in this country, what you can do is go to go do your jury duty, you know, talk to you, talk to other folks, um, in your PTA meetings or other parents, or, you know, if you don't have kids yet, like start getting engaged at a community level about what is this happening to us? We don't actually have to accept this. We can actually change the rules and we can get people in trouble, like the companies in trouble. We can make them accountable because going back to the original discussion we had, John, it doesn't like an algorithm can be terrifying if it's just an unaccountable, unfair system.

1:10:32It doesn't have to be complicated, right? So how do we insist on accountability? That's through our court system and that's through our elected representatives. So be involved. This stuff isn't going away. It's all going to happen. It's only growing. This stuff is only growing. Why do I say that? Because algorithms have replaced every bureaucracy. So all of the things that seem unfair as systems are going to end up being algorithmic questions and algorithmic accountability is a big thing in our future. It's pretty wild that question asker

1:11:09Jon Krohn:works at a company that you know so much about. I also wrote about Optum in Weapons, by the way, is one of the companies I wrote about. I hope these question askers don't mind that I'm mentioning their company name. I hope they told you. Exactly. This is all public information. So our second one, It follows on really nicely from everything you were just saying. So this is someone named Mila Vasuk, who is at Deutsche Bank in Berlin. And she's the vice president of group strategic analytics for technology and banking at Deutsche Bank, obviously. And she points out, she says, we evaluate AI models for bias, fairness, and accuracy.

1:11:50Jon Krohn:Probably not as much as you'd like, Kathy, but there's some of that happening. and she says, we rarely measure the quality of the human decisions that AI replaces or augments. Do you think we're holding AI to a higher standard than humans? It's a fair question. And I am all for doing both. As I've been saying, I audit algorithms, I audit bureaucracies. And one of the things that puzzles me and bewilders me and not in a good way is how you'll see a company just replace their human bureaucracy with an algorithmic bureaucracy without testing it. Like, why are you doing that? At the very least, you should have an ongoing A-B test to make sure that it is replicating in the ways that you like the things that humans are doing.

1:12:46Like the example I have from my book is the, um, from weapons is like the, the, the teacher value added model, which was a random number generator pretty much. And so they didn't test it. They didn't, they didn't compare it to how they used to evaluate teachers. And one of the reasons they didn't do that is because they didn't agree on which evaluation of teachers method was the best. So they basically as a way of avoiding a difficult conversation. They're like, let's just do something stupid. My opinion is that you should always be, you should have, before you implement an algorithmic solution, you should be running parallel testing of the human version and the computer version.

1:13:28And of course, that's going to force you to decide what does it mean for this to be running well, whether it's human or or algorithmic. But ultimately the way I build, the way I do auditing at my firm is we build what I would call cockpits for systems. Like you wouldn't fly in an airplane without a cockpit. Don't fly a massive algorithmic bureaucracy without a cockpit, like measure everything that you can think of that could go wrong to keep track of it and monitor it and to see if it's going wrong. And that means you have to define what you mean by wrong. So that's the tricky part. But, But of course, you absolutely need to do that.

1:14:08Jon Krohn:Great answer. And I had a feeling you tied into the kind of bureaucracy auditing points that you've been making earlier in the episode. Final question for you is from Miriam Kakpur, who is actually a PhD student that I'm co-supervising at the University of Auckland in New Zealand. And so she's a robotics PhD student. and she says that she and her friends who have read Weapons of Math Destruction were discussing this upcoming interview that I have with you. And they had a number of questions, but I'm just going to pick what I think is the one that has been least addressed, which is who audits the auditors?

1:14:49Jon Krohn:So yeah, if we build a system to check for algorithms for bias, what stops that system from having its own blind spots? Oh, yeah, it's absolutely a thing. And that's one, you know, I was just on a call earlier this week with folks in Massachusetts who are thinking about writing a law that requires auditing third party auditors. And I was like, listen, you got to make the language in that law really strong because there's a bunch of my competitors who will just literally write whatever you want to hear for a fee. And the fee will be low because it's not very hard to do that. and I will charge more because I do a thorough job.

1:15:30So if you're just going based on cost, you're going to go with a cheaper person that's going to give you whatever you want to hear, right? So like we need a better system. Like I didn't just start Orca because I want to get paid. I started Orca to like improve the actual standards or to actually improve makes it sound like they have, we have standards. We don't have standards. We have to invent standards and we have to make sure they're good. So one of the things I've been calling for is the methodology has to be, open. Even if the answers are private, like, you know, and to get in the context of like legislation saying like, you need third party auditors.

1:16:05Every time you have a third party auditor, the third party auditor has to say, here's my methodology. I posted it here. Like, here's the, here's my Python code. You know, like I'm using this way of measuring the extent to which this is biased or this is working or this is a fair. So you have to define your terms. So So that's not the end of the answer because we're not going to agree. Me and my competitors aren't going to necessarily agree on what's a good methodology, but at least we're going to have the conversation, which will be an ongoing conversation about like, why this methodology? Why not that?

1:16:40What makes this like more robust? And I'll finish by saying, if you really want robustness, you don't just ask one question. you have that whole cockpit panel, a dashboard kind of perspective on a system to make sure it's working. Because you can game a system to make it look good in one way, which is exactly what happened, by the way, in finance with the VAR, value at risk system that was absolutely gamed. And I saw that up close because that's what I worked on in finance for a couple of years. And I was like, oh yeah, if you just rely on one number, it's easy to stuff all the risk in the last top 5%.

1:17:23So you need more than one way of looking at risk.

1:17:30Jon Krohn:Brilliant answer, Kathy. Dr. O 'Neill, thank you again for an amazing episode and answering all the questions and all the random tangents that I've gone off on. Thank you for all of your hard work in preparing for this conversation. It's been really fun. I'm glad you enjoyed it. I'm super lucky to have an amazing researcher named Serge Massis, who's a great data scientist in his own right, who makes it super easy for me to prepare for episodes. So big shout out to Serge there. Before I let you go, how should people follow you after this episode? Obviously we have the AI Skeptic podcast that they can listen to you on.

1:18:06Jon Krohn:Is there any place that they could also find you on social media or anywhere else? No, I got off of all of it because I don't like working for those people. Um, I do have, I still have my blog on my blog, math, babe. Um, and I sometimes write on it, but mostly I just post my, um, podcast, um, links and I do really love feedback, um, to my podcast. So, um, that's the best way of probably interacting with it. Well, there's a email address, which I wish I knew on the top of my head, um, but I will send it to you, um, for feedback on AI skeptics. But yeah, that's the best way of doing it. But thank you.

1:18:47Thanks for having me.

1:18:49Jon Krohn:Perfect. Yeah. We'll have that in the show notes as well as your MathBay blog. Thank you so much for taking the time. It's been seriously an honor for me, someone I've looked up to for a really long time, for at least a decade, that I've been aware of your outstanding work. And yeah, I really enjoyed this conversation. Thank you. Yeah, me too. Thanks. Wow. What an episode in it. Dr. Kathy O 'Neill detailed why an algorithm doesn't need to be complicated to be terrifying since the real dangers are secrecy, unaccountability, and having no way to opt out. How white-collar workers whose keystrokes are now tracked are getting a taste of the surveillance, degradation, and dehumanization that truckers and teachers have endured from algorithms for decades.

1:19:26Jon Krohn:How auditing an algorithm really means auditing a bureaucracy. Why she wants every algorithmic system to fly with what she calls a cockpit. Parallel testing against the human process it replaces, plus a whole dashboard of metrics because any single number can be gamed. And she talked about why algorithmic harm is statistical harm, nearly impossible to prove for one individual, but very provable across a whole class of people. Exactly. The data work her nonprofit Ocean now contributes to lawsuits against big tech. As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Kathy's social media profiles, as well as my own at superdatascience.com slash 1013.

1:20:10Jon Krohn:Thanks, of course, to everyone on the Super Data Science podcast team, our podcast manager, Sonja Breivich, media editor, Mario Pombo, partnerships manager, Natalie Zajski, researcher Serge Massis, and our founder, Kirill Aromanko. Thanks to all of them for producing another exceptional episode for us today, for enabling that super team to create this free podcast for you. We are deeply grateful to our sponsors. You can support the show by checking out our sponsors links, which are in the show notes. And if you yourself are interested in sponsoring an episode, you can get the details on how by making your way to johnkrone.com slash podcast.

1:20:45Jon Krohn:Otherwise, help us out by sharing this episode with folks that would find Dr. O 'Neill fascinating. Review this podcast on YouTube or on whatever podcasting platform that you listen to your podcasts on. If you write a review on Apple podcasts, that is especially helpful for us. So bonus points to you if you do that. Subscribe, obviously, if you're not already a subscriber, but most importantly, just keep on tuning in. I'm so grateful to have you listening, and I hope I can continue to make episodes you love for years and years to come. Till next time, keep on rocking it out there, and I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.

1:21:31We'll be right back.