1. Podcasts
  2. Best AI papers explained
  3. Uncertainty Quantification Needs Reassessment for Large-language Model Agents
Best AI papers explained

25 Jun 2025 · 19m · 11 chapters

Episode artwork

Uncertainty Quantification Needs Reassessment for Large-language Model Agents

In short

The episode argues that uncertainty quantification for large-language-model agents needs reassessment because hallucinations are inherent to generative models, and the traditional aleatoric vs epistemic split is often inconsistent, entangled, and fluid in multi-turn conversations. It proposes three directions: handle under-specification (task and context ambiguity), use interactive learning (agents ask clarifying questions to reduce uncertainty), and improve output uncertainty communication (richer explanations than single confidence numbers; possibly speech/hesitation cues). It cites Mutsani et al. (2024) showing aleatoric and epistemic estimators are highly correlated (rank correlations ~0.8–0.999).

Notable examples

“dragon scales” recipe hallucination; “Which bank?” (river vs money bank); Google Natural Questions where 56% contain ambiguities; GPT-3.5 Turbo 16K detects ambiguous questions with ~57% accuracy.

Guests

No guests are named; it’s a host-led discussion with no identifiable guest backgrounds.

Key claims

uncertainty labels are subjective in interactive settings; definitions of epistemic uncertainty can conflict (max vs min uncertainty) and aleatoric uncertainty depends on model-class assumptions; numbers may still be preferable for machine-to-machine abstain decisions (e.g., threshold-based “don’t answer”).

Written by AI. May contain mistakes.

Chapters

0:45

Understanding Uncertainty in LLMs

Explains the concept of uncertainty quantification in LLMs and its importance.

“So if we can't just fix it easily, what's our mission today?”
1:23

Types of Uncertainty: Aleatoric and Epistemic

Describes the two main types of uncertainty and their implications for LLMs.

“So to really get where LLMs are going with this, we first need to understand the traditional view.”
2:45

Conflicting Definitions of Epistemic Uncertainty

Discusses the paradoxes and conflicts in defining epistemic uncertainty among different schools of thought.

“Yeah, things with clear inputs and outputs.”
4:20

Debates Around Aleatoric Uncertainty

Explores the debate on whether aleatoric uncertainty is truly irreducible and the differing perspectives.

“What our sources called the density view.”
6:00

The Entanglement Problem in Uncertainty

Introduces the entanglement problem, highlighting the correlation between aleatoric and epistemic uncertainties.

“The data uncertainty proponents come along and argue, hold on, that model bias is reducible.”
7:34

Interactive Nature of Uncertainty in Chatbots

Explains how the dynamic nature of conversations can alter perceptions of uncertainty.

“And what's really interesting here is the crossover.”
9:23

New Avenues for Addressing Uncertainties

Discusses innovative approaches to effectively manage underspecified and interactive uncertainties.

“It reflects which uncertainties they intend to try and reduce through interaction or model improvement.”
11:20

Communicating Uncertainty Effectively

Highlights ways LLMs can better communicate uncertainty beyond simple probability scores.

“Directly to the second new avenue, interactive learning.”
14:01

Exploring Uncertainty in LLMs

Discussing the nuances of aleatoric and epistemic uncertainty in LLMs.

“Phonetic features in speech could add another layer.”
14:59

Interactive Learning and Decision Making

Examining how LLMs can learn through interaction and the role of numerical probabilities.

“Couldn't LLM just learn to ask questions naturally from his training data, especially for like standardized things like customer service bots?”
Show all 11 chapters
16:33

The Future of Uncertainty in AI

Reflecting on the implications of LLMs expressing uncertainty through language.

“They just serve different purposes and different environments.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Best AI papers explained, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00Welcome to the deep dive. You know that feeling, right? You're chatting with an AI, maybe trying to get some quick info, and then, ah, out of nowhere, it just makes something up. Oh, yeah. I remember once asking for a recipe, and it told me to use dragon scales. Hey, dragon scales! Seriously! Yeah. Amusing, sure, but definitely frustrating. Or it gives you an answer that's just wildly off. We've all been there, and it can be, well, pretty disorienting. It truly can. And what's fascinating and maybe a little concerning is that recent research confirms this tendency for large Lankrid models, LLMs, to hallucinate.

0:38It isn't just some random bug you can patch. It's actually inherent to their very generative nature. It's sort of built into how they create responses. Okay. So if we can't just fix it easily, what's our mission today? Wow. It's all about something called uncertainty quantification, which basically means helping LLMs not just generate text, but also understand what they don't know. And then, crucially, communicate that uncertainty to us, you know, transparently. So this deep dive is going to challenge some of our traditional ways of thinking about uncertainty. Maybe explore how we can push LLMs towards becoming more trustworthy, more intuitive.

1:16Yeah, more transparent in how they work with us every day. We've got a whole stack of sources here to explore this, so let's jump in. Okay. So to really get where LLMs are going with this, we first need to understand the traditional view. How machine learning usually looks at uncertainty. Right. The old way. Typically, it's split into two main types. The first one is aleatoric uncertainty. Aleatoric. Okay. Think of this as the irreducible uncertainty. It's like noise that's just fundamentally baked into the data itself. No matter how perfect your model is or, you know, how much data you throw at it, some ambiguity just can't be removed.

1:53It's fundamental. So it's like trying to predict a coin flip. Like even if you knew everything about the coin, the flipper, there's always that inherent 50-50 chance. You can't reduce that randomness. Precisely, yeah. You've hit the core idea there. Then there's the other type, epistemic uncertainty. This is the reducible kind. It comes from the model's lack of knowledge, basically. Okay. So if you give the model more data or maybe improve the model itself, you can decrease this type of uncertainty. It's about what the model doesn't know yet. But could learn. Exactly. What it could learn. So traditionally we've seen uncertainty in this pretty clear, almost black and white way.

2:31It's either unfixable noise or a fixable lack of knowledge. Right. That's been the standard view. Okay, so what happens when we try to apply this lens to something as dynamic and interactive as an LLM? Our sources argue that while these concepts have been super useful for, say, structured tasks like classifying images. Yeah, things with clear inputs and outputs. They're simply too limited for the multi-turn conversational nature of these LLM agents. That's where things get really interesting and complicated. Because when you dig into the definitions themselves, you find conflicts almost immediately.

3:06Conflicts? How so? Well, take epistemic uncertainty. There's a kind of paradox in how different, equally valid schools of thought define it. Imagine an LLM that's pulled in two completely opposite directions. It's absolutely convinced the answer is either a definite zero or a definite one. No middle ground at all. So a really clear split, like it sees only two stark possibilities. Very certain about those two, but torn between them. Exactly. Now, some definitions of epistemic uncertainty, often based on the disagreement among plausible models, would say the situation is maximal uncertainty. Maximal.

3:42Why? Because there's maximum disagreement between those two possibilities. The model is completely torn between two extremes. Okay, that makes sense. But other equally valid definitions, often based on the number of plausible models, would see this as minimal uncertainty. Minimal. How could it be minimal if it's torn? Because the number of plausible models has been reduced to just two. The model has narrowed it down almost completely, right, to just those two options. Wow. Okay. That's, yeah, that's wild that something so fundamental can have such contradictory definitions. How does a field even function with that kind of disagreement?

4:15It's a great question. And it really highlights how slippery the concept can be. And it doesn't stop there. There's even a third perspective. Oh, great. Yeah. What our sources called the density view. This ties epistemic uncertainty to how well the training data supports predictions. So how close the input is to stuff it's seen before. Exactly. So uncertainty here depends on how far an input is from seen examples in the training data. For our zero or one example, this view would just say it depends. Depends on the training data near zero and near one. Right. So the key takeaway here is that even in these simple theoretical examples, there's no single universally agreed upon definition of epistemic uncertainty.

4:57Okay. So the lines are already blurry there. Does it get any better with aleatoric uncertainty, the irreducible kind? Not really. The same kind of definitional debate pops up. The question becomes, is anything truly irreducible? And the debate here is mainly between two major schools of thought. On one side, you have folks who say uncertainty is irreducible within a chosen model class. Within a specific type of model. Right. So say you're using a simple linear model, a straight line to fit data that clearly curves, like trying to fit a line through squiggly data. There's a figure in the sources showing something like this.

5:33Yeah, I can picture that. The line will never quite fit. Exactly. That model bias, the inability of the simple model to capture the complexity, is considered aleatoric by this group. Because within that specific linear model, you can't reduce that error further, even with infinite data. Okay, within that constraint, it's irreducible. Right. They argue that the bias persists for that specific model class, making it inherent, irreducible for that system. But then there's the other side. Yep. The data uncertainty proponents come along and argue, hold on, that model bias is reducible. How? You just need to choose a better model, a more complex one that can actually fit the curve.

6:10Ah, so don't blame the data, blame the bad model choice. Essentially. From their perspective, aleatoric uncertainty is only what no possible model could ever reduce, maybe due to inherent fundamental issues in the data itself, not just the limitations of the model you happen to pick. So the consequence is what we call aleatoric or epistemic isn't a fixed truth. Pretty much. Whether something like model bias gets labeled aleatoric or epistemic depends entirely on subjective definitions and the specific model context you're considering. It makes that once clear line seem incredibly blurry. Extremely blurry.

6:48The labels often reflect the modeler's choice, not some objective property of the uncertainty itself. Okay, as if that wasn't complex enough, you mentioned something called the entanglement problem. Right. This is a really fascinating discovery from the research. Well, you know, theory often tries to neatly separate aleatoric and epistemic uncertainties. Keep them in their boxes. Yeah. But empirical studies, like one from Mutsani and colleagues in 2024, show they are almost perfectly correlated in practice. Almost perfectly correlated. What does that mean? It means if you measure both types of uncertainty across different data points, they tend to go up and down together, like really tightly linked.

7:24The paper mentioned rank correlations between 0.8 and 0.999. Wow. That's incredibly high. So if one is high, the other is almost certainly high too. Pretty much, yeah. And what's really interesting here is the crossover. However, aleatoric uncertainty estimator's tools designed to measure the irreducible noise can be surprisingly good at tasks we usually think of as epistemic. Like what? Like detecting out-of-distribution data. That's data that's really different from what the model was trained on. Traditionally, you'd think that's an epistemic problem. The model doesn't know this new kind of data.

7:58But the aleatoric measure picks it up. Surprisingly often, yes. And vice versa. epistemic estimators can be good at things you might expect aleatoric ones to handle. So they're kind of measuring the same thing in practice. It strongly indicates they capture the same uncertainty in practice, as the sources put it, which raises a big question. If they're so intertwined, can we really separate them meaningfully at all? It really sounds like maybe not, at least not in a practical sense. And it gets even more complicated with chatbots, right? Because of the back and forth. Exactly. The dynamic nature of multi-turn interactions really throws another wrench in the works.

8:35What might start out looking like epistemic uncertainty, you know, a simple lack of knowledge from the LLM. Like, I don't know the answer to that. Right. That can actually transform and become aleatoric if the conversation continues and follow-up questions fail to reduce the ambiguity. Maybe the question itself is fundamentally vague. Ah, okay. And the other way around, too. Absolutely. An initially underspecified question, maybe something inherently ambiguous, which seems aleatoric. Like, tell me about the bank. Which bank? River bank. Money bank. Perfect example. That initial seemingly aleatoric uncertainty can transform into a reducible epistemic uncertainty if the LLM can just ask clarifying questions.

9:15Which type of bank do you mean? Precisely. So this leads to a pretty significant conclusion in the sources. Calling an uncertainty aleatoric or epistemic in these interactive settings is ultimately a subjective choice by the modeler. It reflects which uncertainties they intend to try and reduce through interaction or model improvement. Exactly that. It's about intent and approach, not a fixed state. Okay. So, with all these challenges to the old ways of thinking, this traditional AE split, what are the new avenues? Our sources must propose some better ways forward for LLM agents, right? to make uncertainty actually useful for us.

9:53They do. There are some really exciting directions. The first one is tackling under-specification uncertainties head-on. Under-specification, meaning the user hasn't specified enough. Exactly. LLMs are generalists, right? They're meant to handle almost anything. But the challenge is we users often don't provide all the necessary info or even clearly define the task we want from the start. So two types maybe. Like the LLM isn't sure what task I even want it to do. That's task under-specification, yeah. And then there's context under specification. That's like my Harry Potter example earlier, asking when did the first movie come out?

10:26But not saying which country. Perfect. Exactly that. And it's incredibly common. Our sources cite one data set, natural questions from Google queries, where 56 % over half contained ambiguities like that. Wow. 56%. That's huge. It is. And current models aren't great at spotting these yet. No. Even the fancy ones. Even advanced models like GPT 3.5 Turbo 16K. They only detect ambiguous questions with about 57 % accuracy in one study. 57 %? That's barely better than guessing. It's not great, no. And the crucial point here is that these uncertainties happen at inference time, the moment you ask the question.

11:06So you can't just train them away by feeding the model more text during its initial training. Right, because it's about missing information from the user in that specific interaction, not missing knowledge in the model's training data itself. Okay, that makes sense. Which leads us where? Directly to the second new avenue, interactive learning. Ah, okay. So the LLM can actively try to reduce its uncertainty by asking us questions. Exactly. Asking follow-up questions. That's a huge shift from just, you know, spitting out a single answer, take it or leave it. Definitely. So it's not just about improving the model generally over time.

11:38No, this is different from traditional active learning, which is about improving the model itself. This is about solving the current problem right now by querying the user. Got it. But that raises some practical issues, doesn't it? For sure. Like how many questions are too many before the user gets annoyed? Yeah. Or how does the LMM avoid asking questions it actually already knows the answer to? That would be weird. Right. Or how does it avoid just giving really vague answers while it tries to figure things out? It's a tricky balance. Finding that sweet spot must be key. But is there like an unexpected upside to this interactive approach?

12:14Absolutely. Beyond just getting more accurate or personalized answers, interactive learning significantly improves accessibility. Accessibility? How so? Well, think about it. A tech-savvy user might write a super clear, detailed prompt, but someone less versatile, maybe an elderly person, or a young student using an LLM for homework help might really benefit from the agent guiding them. Ah, the LLM helps them articulate what they actually need. Exactly. It helps everyone get the best out of these powerful tools, not just the expert prompters. That's a really powerful benefit I hadn't considered.

12:50Okay, so we have underspecification and interactive learning. What's the third avenue? The third one is output uncertainties. This is about how the LLM communicates the uncertainty it does have. Instead of just giving us a single, maybe misleadingly confident answer, or just a probability score. Right. Instead of just a number, LLMs can use their amazing language capabilities to communicate uncertainty in much richer, more meaningful ways. Like what? What would that look like? Well, it could involve outlining competing possibilities. Well, it could be X, but there's also a strong case for Y. Exactly.

13:23And explaining why it's uncertain between them. X is supported by this source, but Y fits this other context better. And maybe even suggesting what could resolve the uncertainty. Yes. Suggesting what specific information is missing or would help clarify things. It's about providing a really rich explanation, not just a single score. So it's not just the words themselves, but maybe how it says things, too. Definitely. Using verbal pews like most likely or perhaps or it's possible that in the text. And maybe even in synthesized speech, things like tone of voice or hesitation could convey nuance.

14:01That's a fascinating possibility, yeah. Phonetic features in speech could add another layer. The overall goal, as one source puts it, is to provide a better information base to a human decision maker than a mere number. More context, more nuance. Okay, that all sounds promising. But we should probably consider the counter arguments, right? Where do the traditional views like aleatoric and epistemic still hold value? That's fair. And the sources acknowledge this. First off, you could argue that the terms aleatoric and epistemic, even if blurry, are still useful. Because they're so widely used. Exactly.

14:34Our sources mention Figure 5, showing roughly one paper a day on ARCIF, mentions these terms. They're everywhere. So they are valuable for communicating the intention behind an uncertainty approach, especially during model training. So the advice isn't to ditch the terms entirely? No, just to be really clear about how you are defining and using them in your specific context, define them clearly when you use them. Okay. What about interactive learning? Couldn't LLM just learn to ask questions naturally from his training data, especially for like standardized things like customer service bots? That's a valid point.

15:10For very routine standardized interactions, it's plausible an LLM could simply learn common follow-up questions through standard next token prediction. Just mimicking patterns it saw in training dialogues. Right. But the key challenge remains. LLMs still need to honestly reflect their internal state of knowledge. How do we know if it's asking because it's genuinely uncertain or just because that's the pattern it learned? Exactly. We currently lack really good metrics to distinguish between genuine information needs and just pattern matching. So those human-computer interaction questions about honest signaling are still very much open.

15:45Makes sense. And finally, what about numbers? We talked about rich linguistic explanations, but sometimes a simple probability is useful, isn't it? Absolutely. And the sources agree. For interactions between LLMs and other automated systems, machine-to-machine communication, numerical probabilities are often preferable. Why is that? Think about something like abstain prediction. That's where a system decides to just not give an answer if it's too uncertain. Right, like saying, I'm not confident enough to answer that. Exactly. For that, having a clear numerical threshold, like don't answer if confidence is below 70%, is ideal for automation.

16:21It's easy to program against. So it's not an either-er situation. Not at all. The takeaway is that both numerical uncertainty scores and rich linguistic uncertainty communication can and likely should coexist. They just serve different purposes and different environments. Numbers for machines, rich explanations for humans. That's a good way to summarize it, yeah. Okay. So we've taken quite a deep dive into the sometimes murky world of LLM uncertainty today. Definitely murky, but fascinating. We started by acknowledging that feeling we all get the LLM hallucinations and quickly saw that the traditional sort of binary view of uncertainty.

17:00Aleatoric and epistemic. Is just too limited, really, for the dynamic interactive reality of these LLM agents. Yeah, we explored the inherent conflicts in the definitions, the blurring lines, that surprising entanglement, where they seem to track each other in practice. And how the very character of uncertainty can shift and change during a single conversation. It's much more fluid and nuanced than the old model suggested. But the journey didn't stop at the problems. We looked at these exciting new research directions. Tackling underspecification uncertainties when we don't give enough info. The power of interactive learning, where the LLM asks us questions.

17:37And the potential for output uncertainties, using rich language, maybe even speech nuances, to communicate what the model doesn't know. Yeah, moving beyond just a single number. The promise of all this is pretty immense, really. It's about creating LLM agents that feel more transparent, more trustworthy, and honestly, just more intuitive to work with. Agents capable of navigating these complex, real-world chats with all of us in all our sometimes ambiguous ways of asking things. Exactly. Handling unknown contexts better. So, maybe a final thought to leave everyone with. Considering how much of our own human communication is wrapped up in nuance, ambiguity, maybe unresolved questions, how might a future LLM agent's ability to express its own uncertainty not just with a number, but with the richness of language, how might that fundamentally change how we actually trust and interact with AI?

18:30That's a really interesting question. Does expressing uncertainty make it seem more human, more trustworthy, or just less useful? Something to think about. how that deep expression of not knowing changes the relationship. Definitely something to mull over. Well, that's all the time we have for this deep dive. Thanks for exploring this with us. Thanks for digging in.

From the publisher

This academic paper challenges the traditional dichotomy of aleatoric and epistemic uncertainty within the context of large language model (LLM) agents, arguing that these established definitions are insufficient for complex, interactive AI systems. The authors assert that the existing frameworks often contradict each other and fail to account for the dynamic nature of human-computer interaction. They propose three new research directions to enhance uncertainty quantification in LLM agents: underspecification uncertainties, which arise from incomplete user input; interactive learning, enabling agents to ask clarifying questions; and output uncertainties, advocating for richer, language-based expressions of uncertainty beyond simple numerical values. Ultimately, the paper seeks to inspire new approaches to making LLM agents more transparent, trustworthy, and intuitive in real-world applications.

More from Best AI papers explained

All episodes