


19 Nov 2026 · 22m · 13 chapters

A paradox in multistage high-stakes decision systems: an AI that greatly improves stage-one predictions (Michigan CPS hotline screening) yields only small system-wide gains when inserted into the real workflow.
No guest names or bios are provided in the transcript; only two hosts discuss the research.
Stage-one AI can reduce subsequent maltreatment among screened-out calls by 11.5 percentage points (nearly a third) versus human screeners, yet system-wide improvement is only about 1.1 percentage points when investigators still make stage-two decisions. The advantage “evaporates” due to downstream anchoring bias on prior CPS history and aggregation correction/double counting in single-stage evaluations.
AI flags high-risk cases with zero prior history; investigators often close them because they expect familiar patterns. High-performing investigators outperform the AI by seeking exculpatory evidence (contradictions, physical safety, corroboration via school). Best design uses an AI “defer” option for ambiguous cases, improving welfare by 18.5% versus conventional AI.
Written by AI. May contain mistakes.
Discover how an advanced AI designed to predict child endangerment fails in real-world applications.
“Imagine an AI that, you know, predicts child endangerment so accurately that it drops abuse rates by almost a third.”
Learn about the complexities of multi-stage decision-making systems in child protective services.
“We are taking a deep dive into a really massive paradox today.”
Explore the extensive reach and impact of Child Protective Services in the U.S.
“And we should probably set the scene for you here because the scale of this system is just immense.”
Unpack the dual stages of the Michigan Child Protective Services investigation process.
“And, you know, we constantly hear this narrative about AI replacing human jobs entirely, but this research shows something way more nuanced.”
Examine the effects of implementing AI in the initial screening stage of CPS calls.
“And the researchers wanted to see, well, what if you automate that first stage?”
Understand why improvements from AI screening do not translate to real-world safety benefits.
“Doing that barely moves the needle for the system as a whole.”
Dive into the limitations of AI when faced with human biases in child protective investigations.
“Wait, wait, I really need to process that.”
Discuss the potential and limitations of fully automating the CPS process with AI.
“The downstream human vetoes the supercomputer because they don't share the same mental model of what danger even looks like.”
Examine what human investigators bring to the table that AI cannot replicate.
“So what are the human investigators bringing to the table that the AI is missing?”
Learn how human investigators gather crucial contextual information that AI lacks.
“And to test it, the researchers used large language models to read through all the unstructured narrative case notes written by the investigators.”
Discover how top human investigators seek disconfirming evidence rather than just risks.
“They're basically gathering physical context that just doesn't exist in any database yet.”
Explore how AI can defer complex cases to human investigators for better outcomes.
“It can admit what it doesn't know and just hand it off.”
Consider the implications of advancing AI technologies on the human investigative role.
“Algorithms process available, structured information perfectly.”

Get the gist, explore an idea or ask about something said.
On Best AI papers explained, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00Imagine an AI that, you know, predicts child endangerment so accurately that it drops abuse rates by almost a third. I mean, mathematically, just completely outclasses human decision makers. Right. It's totally superior on paper. Yeah. But then you take that exact same AI, you drop it into a real world system to fix things. And well, that massive improvement just vanishes. It completely evaporates. You take this perfect algorithm, put it in a human workflow, and the actual final result is barely a fraction of a percent better than what you started with. Welcome in, everyone. We are taking a deep dive into a really massive paradox today.
0:36Our mission is to look at some brand new data to figure out why exactly injecting this highly accurate AI into a human assembly line often just fails to solve the problem. Yeah, because we tend to evaluate artificial intelligence in a complete vacuum. You know, like a tech company builds a model. They test it on one super specific task and they just ask, does this beat a human? And lately, I mean, the answer is usually yes. Exactly. Yes. But in reality, very few high stakes decisions are just made by one person sitting alone in a room. They're multi-stage systems. Multi-stage systems. Let's ground that for everyone listening.
1:14So think about your company's hiring process, right? You have a resume screener, human or AI, whatever, and they pass a short list to an in-person interviewer. Right. Or medicine is a great example. Your primary care doctor is basically a screener. They decide if you need to be advanced to a specialist. Exactly. It's an assembly line of sequential human decisions. And to really understand what happens when you force a predictive algorithm into one of those assembly lines, we're diving into this really high-stakes setting using recent data from the Michigan Child Protective Services, or CPS. And we should probably set the scene for you here because the scale of this system is just immense.
1:55I think people assume CPS is this fringe agency dealing with a tiny fraction of the population. Oh, not at all. It's incredibly common. Yeah. By the time kids in the U.S. reach age 18, 37 % of them will have been subject to a CPS investigation. Over a third of all children. I mean, it is a gigantic sprawling system that's just trying to predict and prevent future harm on this massive scale. And, you know, we constantly hear this narrative about AI replacing human jobs entirely, but this research shows something way more nuanced. The real drama is basically how the humans and algorithms interact down the assembly line.
2:35Yeah. So let's actually look at the mechanics of that Michigan CPS process to see where the friction starts, because it's a classic two-stage system. Stage one is the central hotline screener. Right. So someone just calls the hotline, like a teacher or neighbor, maybe a police officer. Exactly. The human screener takes the call, and it's super brief, usually about 15 minutes. They get whatever info the caller has. They quickly review prior administrative records in the database. And then, well, they have to make a binary choice. Just yes or no. Right. Do we screen this call in for a full investigation or screen it out and just end the agency's involvement right there?
3:13That is an unbelievable amount of pressure for a 15-minute phone call and just a quick database check. It really is. Now, if they do screen it in, it moves to stage two, which is the local investigator. Okay, so that's the person on the ground. Yeah. They have up to 30 days to actually go out into the real world. They visit the house, interview the caregivers, talk to the kids privately, maybe call the school. And after all that, they make the final call on opening a formal case. Which is huge. I mean, you're putting a family on a registry, mandating state services. It's a massive intervention.
3:46Absolutely. So you've got this rapid-fire screener at the front door doing triage, and then an investigator doing the deep dive. And the researchers wanted to see, well, what if you automate that first stage? So they built an algorithm. And they actually handicapped the AI, which is important. They trained it using only the information that was available during that initial 15-minute phone call. So no unfair advantages. Nope. It had the exact same info as the human screener. Nothing more. And they gave it one single objective. Predict the risk of subsequent maltreatment. Basically, if we don't intervene today, will this kid be the subject of another hotline call in the next six months?
4:25Okay, and here is where the data gets crazy. When they put the AI's predictions up against the actual historical decisions made by the human screeners, the AI didn't just win. It absolutely crushed them. Yeah, to put a number on it, if you swapped out the human screeners for an equally selective AI, meaning it advances the exact same number of cases, just picking its own priorities. Right. Subsuppant maltreatment among the screened out calls drops by 11.5 percentage points. I mean, that's nearly a third of the baseline maltreatment rate. You're eliminating a third of subsequent abuse just by letting the machine answer the phones.
5:02It's staggering. And I was looking at the breakdown. They tracked 291 individual human screeners. Every single one of them underperformed the algorithm. Literally not one human beat the machine at stage one. Wow. Which honestly makes logical sense when you strip the emotion away. The algorithm is perfectly consistent. It doesn't get decision fatigue at 4 p.m. on a Friday, you know. Yeah, or get distracted by an angry caller. Exactly. It doesn't care about emotionally charged but statistically irrelevant details. It just calmly crunches historical risk variables. It's almost like having a flawless spam filter for your inbox.
5:39It catches all the bad stuff and never accidentally deletes a real email. So, okay, let's unpack this. If I'm a tech CEO or a policymaker looking at this, my first thought is automate the hotline tomorrow. Fire up the servers or we're done here. And a lot of organizations do exactly that based on stage one testing. But, and here's the plot twist, when they modeled deploying this super filter AI at the screening stage in the real world. Meaning the AI does the initial triage, but human investigators still do the home visits. Right. Doing that barely moves the needle for the system as a whole. Total subsequent maltreatment only drops by about 1.1 percentage points.
6:20Wait, wait, I really need to process that. The AI drops the error rate by nearly a third at the front door, but overall safety only improves by 1%. Just about 1%, yeah. Where does all that AI advantage go? It can't just evaporate. But it does, because it gets completely swallowed up by the human bottleneck downstream. Think about what the AI is actually doing at the hotline. It's reversing about half of the human screener's decisions. Okay. So it's screening out cases the humans historically thought were dangerous, and advancing cases the humans thought were safe. Oh, I see where this is going.
6:56It's handing the downstream guys a totally unfamiliar batch of files. Exactly. The human investigators are suddenly getting these high-risk cases that do not look like the high-risk cases they're used to seeing. And when the AI passes those unfamiliar families down the pipeline, the investigators mostly just ignore them. Really? Yeah. The AI floods the zone with genuinely risky cases, but the actual case opening rate downstream barely ticks up. It rises by only 1.4 percentage points. I got to push back on that, though. Are the investigators just being stubborn because it's a machine, or do they honestly look at these files and think, you know, this AI is hallucinating, why reject it?
7:34Well, it comes down to anchoring bias. The data shows that human decision makers in this system are heavily, heavily anchored to a family's prior CPS history. Okay. When an investigator picks up a file, the first thing they check is, have we been to this house before? If there's a thick file of past investigations, they're way more likely to open a new case. But, I mean, that feels so intuitively human. We're wired to look for precedent, right? Past behavior predicts future behavior. If someone has 10 complaints, you assume, or there's smoke, there's fire. But if it's their very first complaint, you might just think it's a misunderstanding.
8:09It totally feels right to a human. But statistically, relying that much on prior history is a massive blind spot. The AI looks at the exact same data and realizes some of the absolute highest risk situations involve families with zero prior history. Oh, wow. The algorithm finds hidden danger in much more subtle combinations of variables. Can you give me an example of what that looks like in practice? Sure. So the AI might flag a case as extremely high risk because even though there's no prior history, the person calling the hotline is law enforcement officer and they're reporting a very specific combo of allegations.
8:46Okay. So the math says that cocktail of variables is super dangerous. Right. Yeah, the math says it almost always leads to severe future harm. But the human investigator gets the file, drives out to the house, looks at the paperwork and says, no, there's no prior record here. The AI must be overreacting. And they just close the case. And they close it. Because they're insensitive to the AI's new criteria. They're looking for familiar patterns. So they keep opening cases for the families they're used to, the ones with long histories, and leave the AI's new, genuinely high-risk cases totally unopened.
9:20It's like the new titanium gear is just grinding against the old, rusty gears downstream. Exactly. That is wild. The downstream human vetoes the supercomputer because they don't share the same mental model of what danger even looks like. This feels like such a massive trap for anyone selling or testing AI. Like, if I ran a pilot program testing this AI just at the hotline, I'd throw a party. I'd give my engineers a medal. And that is the trap of single-stage testing. It is for linked tech companies and policymakers all the time. If you test the AI in isolation at stage one, it's an absolute genius.
9:59And if you run a separate test at stage two, replacing the investigators with AI, it performs totally fine. Matches human performance. Okay, so both stages look good independently. Right. But if you do this flawed math and just add those two single stage improvements together, you'd project an 8.2 percentage point drop in maltreatment. And an 8.2 point drop system-wide would be front page news. huge news. But the reality, when you actually deploy them together, is only a 1.3 point drop. Okay, hold on. If stage one is massively better, and stage two is at least equal to a human, how does adding them up equal almost zero?
10:40The researchers call this the aggregation correction. It's a mathematical reality in these multi-stage systems, and it's mostly driven by double counting and redundant screenouts. Redundant screenouts. Okay. Does that mean the AI is basically just taking credit for fixing a mistake that the human investigator was going to fix anyway? That's exactly it. Let's trace a specific file. Imagine a very low-risk call comes in. The human hotline screener misjudges it and mistakenly advances it to stage two. Okay. The human investigator goes out, does the home visit, realizes it's totally safe, and dismisses it.
11:16No harm done. Right, the system worked. The second human caught the first human's mistake. Now introduce the AI at the hotline. The AI correctly screens that low-risk call out immediately. It never even goes to the investigator. So the AI gets a point on the scoreboard for being smarter than the human screener. Yes. But did it actually change the final outcome for the kid? No. The human investigator was going to dismiss it anyway. So the AI's correct decision was entirely redundant. Ugh. I mean, it saved the investigator a drive across town, which is great for efficiency, but it didn't actually prevent any additional harm.
11:53But single-stage testing counts that as a massive win for the algorithm. Oh, man. It's totally like a dysfunctional corporate pipeline. Like, imagine the marketing department is the screeners and the sales team is the investigators. Perfect analogy. Marketing generates a lead and says, hey, we found this amazing prospect. Mark it down as a win. Then sales goes and actually closes the deal and says, we closed this amazing account. Mark it down as a win. Right. They're both taking credit on their own spreadsheets for the exact same deal. Yeah. And if the CEO just adds the marketing spreadsheet and the sales spreadsheet together, the company looks twice as profitable as it actually is.
12:31C-suite executives fall for this aggregation trap constantly. They double count the successes because they don't realize humans are already naturally correcting a lot of those errors downstream. Okay, so the AI is brilliant at initial triage, but it gets bottlenecked by human bias later on. Which begs the question, why not just automate the entire assembly line? If the AI is so good and humans are anchoring to the wrong data, just let the AI run stage one and stage two. You'd think so, but the data shows that doesn't yield huge gains either. Really? So what are the human investigators bringing to the table that the AI is missing?
13:04Well, this is the silver lining of the human element. The hotline screeners were universally beaten by the AI, right? But the field investigators are a completely different story. The data shows that about a third of the human investigators, 33.9%, actually outperformed the algorithm. Wait, I totally assumed the machine just won everywhere eventually. A third of them consistently beat the AI? What makes that specific group of humans so special? The researchers wanted to isolate that exact superpower. First, they looked at demographics. Was it the older, more experienced workers? No. Interesting.
13:43Was it race? Or maybe how heavy their daily caseload was? Nope. None of that correlated. My first thought is that maybe they're doing things off the books. Like, they aren't officially opening a state-mandated case, but while they're in the living room, they're referring the parents to a local food bank or setting up a temporary safety plan, stuff the algorithm just can't track. That was actually a major hypothesis. And to test it, the researchers used large language models to read through all the unstructured narrative case notes written by the investigators. Oh, smart. Yeah, they had the AI scanned for any evidence of informal services, counseling recommendations, parenting advice, temporary care.
14:21But it turned out the high-performing investigators weren't providing any more of those informal services than the low-performing ones. So if it's not age and it's not secret off the book social work, what is the advantage? Their advantage is private information. It's the unstructured data they gather by physically leaving their desk and walking into a complex, messy human environment. Oh, wow. Think about it. The AI only has structured intake data, a 15 minute call and a prior record. But the human physically crosses the threshold of the house. They smell the kitchen. They see if the kid's clothes actually fit.
14:59They interview the child privately in their bedroom, away from the parents. They're basically gathering physical context that just doesn't exist in any database yet. Exactly. But the researchers went a step further because they wanted to know what specific context actually mattered. So they used large language models again, this time to analyze the actual text of the notes, looking for patterns that differentiated cases that got open from cases that got closed. I love that. They basically let an AI play detective on the human detective's private notes to see how they were beating the other AI.
15:32What did the text analysis find? It identified five recurring themes. Two were exactly what you'd expect, finding physical evidence of harm or discovering a caregiver was involving the child in dangerous conduct. But the really revealing themes were exculpatory. Exculpatory, meaning evidence that tends to prove innocence or safety. Yes. The top performing investigators were actively seeking out and documenting things like an absence or direct contradiction of the initial abuse allegations. Oh. Or they were doing the legwork to corroborate that the child actually had adequate care, like going to the school and having a teacher confirm, yeah, this kid always arrives clean, happy and well fed.
16:15That is a massive paradigm shift. So you're saying the best human investigators aren't the ones acting like bloodhounds desperately sniffing out hidden dangers. They're acting like defense attorneys actively trying to prove the initial hotline allegation wrong? That is the core insight right there. Think about the system flow. By the time a case reaches an investigator, a hotline screener has already decided there's a credible risk of danger. The working hypothesis is already guilty. Right. So the high-performing humans are the ones who go into the field and actively hunt for disconfirming evidence.
16:50They look for real-world nuance that proves the sterile database wrong. Because an algorithm is, I mean, it's fundamentally designed to look for patterns of guilt. It matches risk variables. It's way harder for a machine to weigh a completely novel, unquantifiable piece of physical evidence that breaks its pattern. Right. Right. It's that human ability to say, look, I know the spreadsheet says this family is high risk, but I'm standing in this warm, fully stocked kitchen and this kid is safe. And when they find that contextual nuance, they override the initial risk profile. That is a capability the AI just does not possess.
17:27OK, so we've got two incredibly powerful but fundamentally different tools here. We have an AI that's vastly superior at initial triage and processing structure data. And we have a subset of human investigators who are vastly superior at gathering physical context and finding exculpatory evidence. So how do we put this back together into an assembly line that actually protects people? Well, the research modeled an optimal division of labor. They designed a new screening algorithm, but instead of forcing it to make a binary yes or no decision on every single case, this optimal AI has a third option.
18:03It can defer. Oh, nice. It can admit what it doesn't know and just hand it off. It handles the clear-cut extremes. It screens out the definitively safe cases, saving the agency massive amounts of time. But for the complex, ambiguous cases, the ones where the AI mathematically recognizes its own limitations, it defers. And sends it to the human. It sends it to the human investigator. But crucially, it only sends them when the mathematical value of the human's physical investigation outweighs the financial and temporal cost of sending them out. So it stops treating the human like a rubber stamp and starts treating them like a highly specialized physical sensor.
18:42Exactly. And when you pair this deferral algorithm with those high-performing human investigators, the entire system's welfare improves by a staggering 18.5 percent compared to just letting a conventional AI run everything. 18.5 percent. Now that is the leap we were looking for. It kind of reminds me of the dynamic between like an MRI machine and human surgeon. Oh, that's a good comparison. Right. The AI is the MRI. It can scan millions of images, flags a suspicious shadow perfectly, never gets tired, scales instantly. But you do not start cutting a patient open just because the machine found a shadow.
19:19Right. Definitely not. You still need the expert human surgeon to open the patient up, physically look around, and use their real-world judgment to determine, oh, actually, that's just a benign cyst. Close them up. You need the machine's immense scale, but you absolutely rely on the human's physical context to make the final call. That is the optimal design. When you optimize a multi-stage process, you have to respect that division of labor. Algorithms process available, structured information perfectly. Humans gather new unstructured information that changes the paradigm entirely. So what does this all mean for you listening right now?
19:56Whether you're prepping for a management meeting tomorrow, designing an HR hiring pipeline, or building a new tech workflow for your startup, you have to remember this dynamic. Yeah, it applies everywhere. If you just drop a shiny new AI into step one of a multi-step human process, do not expect miracles. Expect the downstream humans to act as a massive bottleneck unless you completely redesign how they communicate, what they anchor their decisions on, and what they are uniquely responsible for. You have to design for collaboration, not just replacement. Very true. But as we look to the future, I actually want to leave you with one final thought to mull over.
20:34Okay, let's hear it. This data proves that the human advantage right now, the thing that keeps us beating the algorithm, is our ability to physically walk into a room, observe nuance, and gather private, unstructured field data. Right. Our physical context superpower. But AI systems are rapidly becoming multimodal. They are plugging into live video feeds, police body cams, smart home data, real-time audio analysis. Oh, wow. So if the algorithm no longer has to rely on a 15-minute phone call, if you can just analyze the body cam footage of an officer in the living room in real time, will that human superpower of gathering physical context eventually belong to the machine too and if it does where exactly does the human fit in then yeah that is a fascinating and um slightly terrifying question to end on a lot to think about there until next time everyone keep diving deep
This paper evaluates predictive algorithms within multistage decision systems, focusing specifically on Child Protective Services (CPS) screening and investigation processes. Traditional algorithm evaluations typically examine a single stage in isolation, which fails to capture how downstream human actors respond to algorithmic recommendations or how later-stage workers gather new information that alters outcomes. To address this limitation, the authors develop an econometric framework using large language models to analyze unstructured investigator case notes, successfully identifying crucial narrative themes that capture significant predictive signal beyond standard administrative data. Ultimately, the research demonstrates that deploying algorithms in multistage systems requires understanding human-algorithm complementarities, revealing that human investigators add distinct value by collecting new field evidence rather than merely replicating automated risk predictions.