In short
Episode topic: Overconfident LLMs—how “epistemic markers” (e.g., “I’m certain”) are learned from human language, how they relate to accuracy, and how they change user reliance; plus a separate segment on voice cloning/style transfer and listener perceptions.
Guest
Kaitlyn Zhou, incoming assistant professor at Cornell (formerly at Stanford). Background includes crisis informatics (uncertainty communication online) and AI/LLM research; linguistics-influenced work via hedges/evidentiality.
Key claims
LLM certainty language is often miscalibrated: in a study, 40%+ of certainty expressions were paired with incorrect answers. Prompting with “I’m certain…” increased incorrect completions (opposite of expectations). Humans rely on overconfident outputs more than hedged ones; even plain answers without confidence cues are relied on nearly as much as high-certainty phrasing. RLHF training can penalize uncertainty, suppressing hedges.
Notable examples
“capital of France” prompts; multiple-choice questions where “absolutely certain” is wrong; “I’m 100% certain” vs “I think” reliance game; voice cloning erases non-native accents and sounds more native, warm, and authoritative.
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Chapters
Tap a time to open that second in VOExploring Overconfidence in LLMs
0:46 to 2:56
Discussion about Kaitlyn's research on overconfidence in language models and its impact on user interaction.
“which had to do with, for lack of a better term, maybe confidence in LLM outputs, and then how we as users interact with them maybe as a function of that confidence.”
Crisis Informatics and Uncertainty
2:57 to 5:18
Kaitlyn discusses her background in crisis informatics and its relation to understanding uncertainty in LLM outputs.
“Maybe some of the crisis informatics was like informing that take, but I think it's an interesting thing to try to even get out of the data that you were probably generating for yourself or using for these studies.”
Measuring Confidence in LLM Outputs
5:19 to 7:43
Exploration of how confidence in LLM outputs is measured and the implications of overconfident responses.
“So what we, one of our experiments that we did is we would prompt language models to say, hey, answer this very difficult question and tell us your level of confidence with it and use a weakener or a strengthener.”
Understanding Epistemic Markers
7:44 to 10:09
Discussion on epistemic markers, their significance, and how they are misused by LLMs.
“Do they even understand what epistemic markers are?”
User Reliance on LLMs
10:10 to 12:34
Exploration of how users rely on LLM outputs and the psychological factors influencing their trust in these models.
“And so you could see artifacts of this in our training data sets where expressions of certainty and uncertainty are used in unexpected ways, and the language model then interprets them incorrectly in prompting.”
Incentivized Decision-Making in AI Interaction
12:35 to 14:01
Kaitlyn explains the design of an incentivized game to study human reliance on AI outputs.
“So what we do is we take generations from these language models and we present them to online crowd workers.”
Understanding Relying vs. Trusting in AI
14:01 to 18:15
Explore the difference between reliance and trust in interactions with AI.
“And trusting someone is different from relying on someone.”
Research Findings on Overconfidence in LLMs
18:16 to 22:04
Discuss research revealing how overconfidence in language models affects user reliance.
“single text with uncertainty because sometimes it is important for models to communicate uncertainty.”
Voice Cloning and Its Implications
22:05 to 27:24
Delve into the ethical and practical implications of voice cloning technology.
“And yeah, voice cloning is a very fraught topic.”
Using AI in Research and Personal Work
27:25 to 28:00
Hear insights on personal AI usage and how research findings influence it.
“Our follow-up work is looking at deploying a similar framework to what we did with the epistemic markers and actually getting humans to make decisions with these voices and see how their behaviors change.”
Show all 13 chapters
Researcher Insights on AI Usage
28:00 to 29:12
Learn how researchers integrate AI into their daily work and the challenges they face.
“And there's some initial findings there, too.”
The Importance of Non-Adopters in AI Design
29:12 to 31:31
Discover how AI development overlooks non-adopters and the implications for society.
“And there's a lot of continued learning to do for everyone.”
Cognitive Labor and AI Opportunities
31:31 to 33:10
Explore how cognitive labor in physical tasks presents opportunities for AI integration.
“And so when you think about people who are older, people who maybe aren't bi-coastal elites, like how are they thinking about using AI and integrating into their daily lives?”
Transcript
Automatic transcript. May contain errors.0:00Linear Digressions Hosts:Hi, everyone. This week, I'm very excited to have an interview with Kaitlyn Zhou, recently of Stanford, now an incoming assistant professor at Cornell. Kaitlyn studies AI, LLMs, from a bunch of different angles, and actually, that's why I'm so excited to talk to her today to explore some of those different facets to her research. From a bit of a linguistics background, I think, but I'm going to have her actually dig in and tell me what she researches here in just a second. Caitlin, we're really excited to have you. You're listening to Linear Digressions. All right, so Caitlin, again, thanks for coming on.
0:37Linear Digressions Hosts:I studied many different things, but as I was reading through some of your papers, the first few that I was looking at had this through line that I want to start by exploring with you, which had to do with, for lack of a better term, maybe confidence in LLM outputs, and then how we as users interact with them maybe as a function of that confidence. So I know you've explored this from a few different angles. I'd love to have you start by just talking about that research arc. Like in general, what have you been studying on that front and what have you been finding?
1:11Kaitlyn Zhou:Cool. Thank you so much, Katie, for having me on this podcast. I'm excited to share some thoughts with you. Yeah. At the end of my PhD, a lot of what we were working on was this idea of overconfidence and language models, and we approach it from lots of different through lines. But it's a funny story kind of how this all started. So I have a background in undergrad working in crisis informatics. So it's a lot about during a crisis event, how do people converge online and make sense of information? And one of the big important threads there is how people use expressions of uncertainty to communicate doubt, to cast doubt, and to ask questions on what is going on.
1:49Kaitlyn Zhou:Is this true? Is is false and so on and so forth. And so I was wrapping up a paper on topics around kind of crisis informatics and uncertainty. And that was at the same time, 2022, when language models were really exploding in use. And so it was a very natural connection to be like, oh, I wonder how language models are also using these very important markers that we use every day in human communication. And so a lot of the motivation is it's critical for doctors and lawyers to be able to accurately assess the confidence of the outcome of a trial or the outcome of a treatment? And what does it look like when language models who at the time advertised to be new state-of-the-art capabilities, having all this knowledge, what does it look like for them to accurately or inaccurately use these very important markers of communication?
2:37Linear Digressions Hosts:And so the first paper I think that I read was one from 2023. It was called Navigating the Gray Area, and it was really taking this focus on the LLMs themselves and the production of these very confident answers. And by the way, I think that's, it probably turned out to be a very prescient research line for you, because that's a big feature, I think, of what we are grappling with some of these models today. So as you were starting to think about this, even as a concept that you needed to define as a research entity that you wanted to see, like, how did you think about confidence as something that you could even measure in those LLM outputs.
3:18Linear Digressions Hosts:Maybe some of the crisis informatics was like informing that take, but I think it's an interesting thing to try to even get out of the data that you were probably generating for yourself or using for these studies.
3:29Kaitlyn Zhou:Yeah, definitely. And it was like, it was at an earlier time when we thought about confidence in language models, a lot of it was focused on like token probabilities. Like if I prompt a language model, say the capital of France is Paris, you're looking at how much probability is assigned to that last token, Paris, and you want that to be calibrated with how accurate that token is actually going to be. Right. So if you assign 95 % probability on those predicted tokens, then in expectation, 95 % of those tokens should be actually correct. Another way to think about it is people would do probability and then output some ordinal value.
4:06Kaitlyn Zhou:So like low, medium, high as a way to express how confident the language model is. And those were all nice ways to communicate language model uncertainty, but there was something that wasn't quite organic with that, in that if you're a user, you just have to probe the language model to tell you its probability at the very end, rather than just naturally reading its output as it's given to you on the screen. And so thinking about uncertainty through this lens of expressed uncertainty, like what it's actually going to generate to you, and can it automatically encode these expressions as it's giving you responses, was this new shift that we were taking.
4:41Kaitlyn Zhou:And luckily, in the field of linguistics, there's so much literature on this. Hedges is long-studied, evidentiality throughout lots of different languages and such. So my advisor, Dan Juravsky, is a linguist and computer scientist by training, and so he sent me to the stacks of the Stanford Library, and there were a lot of old books and textbooks that were read to try and develop an initial framework and taxonomy of how historically we thought about expressions of uncertainty, epistemic markers is another word for them. And so we built a taxonomy that we felt comfortable using on language model generations.
5:16Kaitlyn Zhou:And from there, it became a lot easier to automate and quantify expressions of uncertainty in language models.
5:22Linear Digressions Hosts:And so just to make this land for folks who have almost certainly seen these, but maybe never knew to call them epistemic markers, it's maybe, yeah, what are some examples of, when you see it, you know, the phrase that you're like, ah, epistemic marker.
5:37Kaitlyn Zhou:Yeah, definitely. So what we, one of our experiments that we did is we would prompt language models to say, hey, answer this very difficult question and tell us your level of confidence with it and use a weakener or a strengthener. And so the language model will respond to a multiple choice question and do all this calculation and be like, I am absolutely certain that the answer is A. And because it's multiple choice question and these are well-known data sets and benchmarks, we know that the answer is not A. And so you can literally calculate how often it is going to incorrectly be overconfident.
6:08Kaitlyn Zhou:And at the time when we were doing it, aggregating across the models, there was something like 40 plus percent of these expressions of certainty were coupled with incorrect answers. And so that's very jarring when there's like a objectively true answer that you know the language model is incorrectly producing.
6:24Linear Digressions Hosts:So what you're measuring there, just if I can distill this a little bit, is you mentioned 40 % as one of the research results that you got, but that was like what you actually measured in this study for the confidently wrong answers that I think we all intuitively have that vibe of where it says something like, here you go, but we know it's wrong.
6:42Kaitlyn Zhou:Yes, yeah, exactly. And of course, the better models have a lower rate of that because they are more accurate. The models that perform really well in these data sets, there's fewer answers that they get wrong, but there's still a great proportion of them where they are incorrect and they're going to very confidently assert that they know the answer.
6:59Linear Digressions Hosts:And as I was reading one of your papers, I think there was one finding in particular that leapt out to me, which was, tell me if I'm getting any of the nuances wrong here. I want to get it right. But if I recall correctly, it was, in some cases, you measured higher statements of confidence were actually less likely to be accurate than if it was neutral or expressing underconfidence. Is that right?
7:24Kaitlyn Zhou:I think when we started the project, we wanted to know how language models were going to generate these epistemic markers and how they were going to express certainty and uncertainty. And we wanted to know what the error rates were and so on and so forth. But before we could even answer that question, we first had to deal with the idea of do language models even understand what these markers even mean? Do they even understand what epistemic markers are? And so we did a much easier experiment to start off, which was that first paper in the series of three papers we wrote, which was if I prompt a language model with saying, what is the capital of France?
7:58Kaitlyn Zhou:I'm certain the answer is blank. And we have the language model fill in the blank. We wanted to know, could it recognize that if it says, I'm certain the answer is blank versus I'm not sure the answer is blank, that when it says I'm certain, it's more likely to produce the correct answer. And when we lead it with, I think the answer is, it's more likely to produce the incorrect answer. And that was the hypothesis, right? And this stems from persona research that if you say, oh, you are a professor, to the chat model is more likely to produce very complex and nuanced answers versus I'm a toddler and I'm explaining this, you're gonna maybe expect more simplistic responses.
8:37Kaitlyn Zhou:And so the same intuition applied to these initial prompts, which is prompting it with, I'm certain the answer is blank versus I think the answer is blank. And what we found was the opposite effect. We found that when we prompted the language model with, I'm certain the answer is blank, they were more likely to complete that prompt with the incorrect answer. And this has great ties to Dian Kruger effects in psychology, where people who are absolutely certain in what they say may actually not know what they're talking about, and that the real experts are the ones who are more likely to be hedged and more nuanced in their responses.
9:10Kaitlyn Zhou:So we actually traced this back to training data sets. So we were looking at stack exchange blog posts, where you would have paired data sets of someone's asking a question, and then there'll be responses. And we found is that in questions, so people seeking information, people were a lot more likely to say things like, I'm sure it's. And it was in very unexpected ways. They would say, oh, I'm sure it's not the Wi-Fi, or I'm sure this is just a silly problem. So people were using certainty in these very unexpected ways when they were looking for information. And then when it comes to uncertainty, we found a lot more uncertainty in their responses.
9:48Kaitlyn Zhou:So this is usually when you had the answer, you had the information, and you're trying to help someone out. And you could see that people were using uncertainty, but it wasn't because they were uncertain of the answer, but because they were trying to be polite. So using uncertainty as a way to express politeness or to correct someone is very common while studied in linguistics. So they'll say something like, oh, I think this is your error, or I think this is the problem. And so you could see artifacts of this in our training data sets where expressions of certainty and uncertainty are used in unexpected ways, and the language model then interprets them incorrectly in prompting.
10:24Linear Digressions Hosts:Interesting. So maybe to run this back based on my understanding then is what your research, it sounds like, suggests is that epistemic markers reflecting some inherent confidence that we have that we're actually correct. What you're finding is that maybe where they're coming from or how they're being used by the language models more mimics like the linguistic patterns that we have and how we talk about stuff, not necessarily the actual inherent correctness of what we're saying.
10:52Kaitlyn Zhou:Yeah, absolutely. Epistemic markers in human communication are used for multiple different purposes. Sometimes it's to communicate confidence, but sometimes it's for other discourse acts such as politeness or to assert confidence or to try to convince you of something. And what we expect from language models is that they're always objectively and correctly using these epistemic markers and not in the way that's fuzzy that humans use it as. But that's not the case. These language models are learning from human data, and so they bring some of these artifacts in where they're using these, I'm 100 % certain it's mainly as a way to exaggerate or to convince you of something rather than asserting that they are actually 100 % certain.
11:32And then one
11:33Linear Digressions Hosts:of the other papers in this trio is really focusing on, okay, we've got these language models and they're generating these epistemic markers, they're talking about things with different levels of confidence. How does that end up impacting the way that the user, the human who's on the other side of the chatbot window, actually interprets or you're in particular looking at the reliance that they have on what the model is telling them. So it's not just about here's what the model is saying, but it's starting to introduce the other part in that like human LLM interaction and understanding how the confidence ends up impacting this as a conversation.
12:10Kaitlyn Zhou:Yeah, we really liked this project. A lot of what we found was, oh, these language models are overconfident. They're using these expressions wrong. And then the natural question from a very human-centered perspective is, oh, like, how do humans respond to that? Are humans in some ways vulnerable to this? Or maybe they're smarter than we think, and they know that a language model is you, and they know when they say language models are 100 % certain, they know to be, like, skeptical of that. So what we do is we take generations from these language models and we present them to online crowd workers.
12:41Kaitlyn Zhou:We take a lot of inspiration from frameworks in HCI where we have self-incentivized games that people are playing where they need to make decisions based off of what a language model says and then these language models use I think it's or I'm certain it's and we measure rates of reliance based on that.
12:58Linear Digressions Hosts:I thought this was an interesting nuance and I wanted to ask you about it. So you explicitly introduced this idea of a game and asking the human to make a choice in the context of this game about whether they rely on what the LLM says as the answer to this trivia question, or they can say, I'm not going to rely on that. I think it's wrong. And then based on whether they rely correctly or incorrectly, can get a point or lose a point. I'm just curious why you took that framing of it rather than the simpler, more straightforward version of do you think it's right or not? Do you trust this? What pulled that angle into it?
13:33Linear Digressions Hosts:Yeah.
13:34Kaitlyn Zhou:Okay. Yeah. No, that's a really good question.
13:36Linear Digressions Hosts:In the incentivized game setting, what we're trying to do is force people
13:40Kaitlyn Zhou:to make reliance decisions given the resources that they do have. One nuance I'd point out is we're very careful to say in this game-like scenario, we're asking humans about their reliance behaviors, not about their trust behaviors. So trust is a more complex concept and it might involve a relationship with an entity. And trusting someone is different from relying on someone. So relying is just, maybe this is the best option that I have given the circumstances, or given all the information I provided, this is the best decision I can make. So you can have reliance without trust. And so a lot of this game-like scenario is focused on these reliance behaviors.
14:19Kaitlyn Zhou:Given this constraint, this kind of time scenario where you're trying to get through the task as quickly as possible, and you don't have a lot of information or external resources, are you going to rely on what the AI is providing you or not?
14:31Linear Digressions Hosts:And so what did you find? Are people more reliant, for example, when it's more confident? That might be my naive guess. Was that borne out in the research?
14:39Kaitlyn Zhou:Yes, and it was very clear that was the case. So when language models are hedging, so they say something like, I think that's relied on very infrequently. But if a language model says, I'm 100 % certain the answer is, like 98 % of humans are going to rely on that statement rather than looking up the answer themselves. And so it is very striking that when language models are overconfident, which we know they are and they're incorrect, humans are going to be reliant on that. What's, I think, more striking in some ways is that we also put in plain statements into these reliance tests. And these plain statements are just the answer is blank or it is Paris is the capital, like something where there's no expression at all.
15:22Kaitlyn Zhou:And we just wanted to see these plain statements where they would land on this kind of range of reliance. And it turns out these plain statements are relied on nearly as often as these expressions of certainty. And this is very concerning because most of the time when you interact with language models, they're not doing this meta-level work of giving you their degree of confidence or certainty, but they're just plainly stating the answer. It turns out these plain statements are not neutral statements. they're not seen as uncertainty, but rather they're seen nearly as confident as these expressions of high certainty.
15:55Linear Digressions Hosts:I had one of your research colleagues, Chris Potts, on this show not too long ago. We had some really interesting conversation about thinking about LLMs as things that users can delegate to. You just hand something off and don't maybe introspect or think as critically about what you get back versus augmentative, something that you're really going back and forth with the AI you're iterating and thinking about augmentative being a signature of more fluent AI usage, like probably a better way for us to generally aspire to interact with AI, you know, not that we like, we aren't outsourcing our brains to it.
16:31Linear Digressions Hosts:And so I was thinking about that some in the context of some of your reliance research, and that there's generally maybe some nudging that these models are getting from several different parts of their training process toward giving confidently wrong answers, maybe more than you might guess. At the same time, those confidence markers are, you're also finding manifesting is more reliance in the users on this. So there's, there strikes me that there might be a nasty little feedback loop that we have here where you're being nudged towards being more accepting of some of the, the lower quality answers that these systems might be giving you.
17:12Kaitlyn Zhou:So we were looking at these RLHF data sets where you can have annotators mark which text do you prefer? So do you prefer text A or text B? And what we had hypothesized was that people really liked these markers of certainty, and that whenever they saw certainty in text A or text B, that they would choose the text with certainty markers. And so we would look through these RLHF data sets, which we know are important in model post-training alignment. And what we found was not that people really liked certainty. In turns, I was just people really hated uncertainty. So whenever there was uncertainty, there was a great penalty towards that text.
17:48Kaitlyn Zhou:And so once that is embedded into these annotations, then when we put them through post-training processes, it becomes very evident that the language models will then suppress expressions of uncertainty moving forward. And we found this bias because we were explicitly looking out for it, but we would hypothesize that when the annotators were doing these annotations, they probably weren't queued in on, oh, check your implicit biases around uncertainty and make sure you don't reject every single text with uncertainty because sometimes it is important for models to communicate uncertainty. So I think it's one of those unfortunate things where had we known about this earlier, or if we were to retrain some of these larger models on annotator data sets, if we could queue in on this dimension and provide additional scaffolding and training to annotators, that could help.
Read the full transcript
18:36Kaitlyn Zhou:But then again, this is one of many implicit biases that will rise. It's like whack-a-mole. If we get this one, something else will emerge. And it speaks to the kind of long-term limitations of mimicking human data and trying to build these models that are mimicking human behavior without fully understanding why humans are acting in this way. It goes back to kind of these epistemic markers as well. Humans will say, I think it's to be polite. Maybe language models haven't cued in on this politeness thing as much or why we would use it in that way.
19:06Linear Digressions Hosts:And one last thing on this topic. I do wonder about the specific circumstance that we've all had and most people really hate where you're talking with Claude is the one that people always attribute this to chat GPT. You say something, it says something back, you correct it. It says something wrong back in whatever sense that means. Confidently wrong in all likelihood. You say no and you correct it and you're and it's like you're absolutely right.
19:35Kaitlyn Zhou:and I was like, here's blah, blah, blah.
19:37Linear Digressions Hosts:And I'm very confident. It's this weird juxtaposition. It's very confident on both sides.
19:42Kaitlyn Zhou:Exactly.
19:42Linear Digressions Hosts:And as far as I can tell, this makes people just like, it short circuits your brain about like, I don't know whether to trust you or not trust you and I hate this. And yeah, I don't know. There's some kind of weird little feedback loop that's happening there. And I really just wonder like, yeah, what happens? I think we all like intuitively feel that is something we struggle to grasp with in our interactions with these models, the confidence and the I'm wrong in the same thing together. But like, boy, oh boy, does it make people react? Yeah.
20:12Kaitlyn Zhou:Yeah. Like, how can you be so confident and then be so flippant at the same time? Exactly. Yes. Yeah. Yeah. That's very troubling. So I think overconfident, the way we've described it, is the most direct, visible way to see overconfidence, like when it's literally written in the language. But there are so many other implicit forms of overconfidence that I've been thinking more about. So one thing is if I say, how do you build a car and give me step-by-step directions on how to build a car and do it in less than 500 words? Claude will attempt that. Claude will attempt to give you step-by-step directions on how to build a car in less than 500 words.
20:48Kaitlyn Zhou:And no rational human being would try to do that because it's just not possible to like describe something so complex in so few words or like, oh, how do we solve cancer? How do we cure cancer and give me step-by-step directions and very concrete steps on how to approach this. And just nobody would try to answer that in an enumerated list. But language models, as another form of overconfidence, will try and give you a response. They are game for it, yeah. Yeah, they are totally game for it. They will totally try to tell you how to cure cancer. Yeah, and so it comes up in lots of different ways.
21:22Kaitlyn Zhou:It comes up by them not asking clarifying questions. It's by them making assumptions about what you want. It comes up by just even assuming that they should be producing 10 ,000 tokens for you. Yes, you will read all the text and I'm going to have this huge text box that I'm sending back to you. So those are all the ways in which human AI interactions do not mimic human-human interactions. That would not be a normal conversation you would have with someone. It's a good reminder that we are not interacting with humans as anthropomorphized as they may seem. But they are kind of tools and they have these limitations and design affordances.
22:00Linear Digressions Hosts:some of your more recent research now switching topics a little bit you have a paper voice cloning is style transfer and i thought this was a really interesting actual extension of some of your earlier work i think that the title of the paper itself is already pretty dense the whole idea of voice cloning is style transfer i wanted to talk about it a little bit but wanted to start with just the general concept that you're exploring in this paper.
22:26Kaitlyn Zhou:Yeah, definitely. And yeah, voice cloning is a very fraught topic. I think it has a lot of security and privacy risks. And I think we've been grappling with these ethical concerns as we work on these papers of like, what is voice cloning for and who benefits from it?
22:42Linear Digressions Hosts:And voice cloning, as I understand you mean it here, is where you have synthetically generated vocal text in the style, like the actual like vocal tones of a particular person. Is that the right interpretation here? Okay.
22:57Kaitlyn Zhou:Yes. Yeah. Yeah. The voice cloning technology, kind of how it works now is there are text-to-speech systems that will take a target text and be able to generate sound that sounds like someone reading out that text. And historically, that has been done by training a lot of data on an individual's speech. And then and they will be able to produce speech in the style of that individual whose training data you had hours and hours of data for. Voice cloning takes it a step further and is more generalizable. So instead of needing lots of data for a particular individual to generate speech that sounds like that individual, you can take a snippet of someone's voice, pass that into the language model in addition to target text, and then it will produce speech that sounds like this snippet of this voice that you've passed in.
23:43Kaitlyn Zhou:And this is very popular technology that's rising. And yeah, and there's a lot of, yeah, there's a lot of ethical concerns. And there's a lot of important questions that we have to ask here. And there's a lot of legitimate use cases for it as well. I think we were in your email back and forth about using it for podcasts or for recording. I think it's very needed for a lot of professionals where for me, like every time I present a paper to a conference, I have to do a video or an audio recording of the paper. and sometimes it's just a few words that I've tripped up that I would love to have polished and so if I had a high fidelity voice clone of my voice I could then like just do a few edits and have this perfectly smooth professional sounding voice be the audio recording that I have.
24:28Kaitlyn Zhou:There's obviously a lot of benefits in that voice recording is very expensive. Having microphone set up, having a quiet environment, having all these conditions for you to create really high quality voice, synthetic voice obviously doesn't need that it can always produce noiseless sounds there are no ambulances in the background but it's going to be perfect and smooth every time so you were studying
24:49Linear Digressions Hosts:again how cloned voices are interacted with also like what is the perceived interaction on the other side the listener basically and I think you found some interesting things here would love to hear the highlights of your findings from your perspective but in particular that like in general people like cloned voices, I think, or they like is probably not exactly the right word, but they generally rate them higher on a number of maybe desirable attributes. Yeah. A more precise way to say it.
25:20Kaitlyn Zhou:Yes. Yeah. So kind of the one of the big findings of this paper is that we had a lot of non-native English speakers record audio text and then voice clone that audio and then compared kind of what these two audio clips sound like, and we had a bunch of monolingual English speakers listen to both versions of it and rate them on features like how warm does this voice sound, how authoritative does this voice sound, does this sound like a native English speaker. One of the major findings is that these cloned voices were seen as significantly more native sounding, native English sounding than the original voices.
25:55Kaitlyn Zhou:And so there's kind of this erasure of these non-native English accents that are being erased in this process of voice cloning. In addition to that style transformation, there are other transformations like the voices sounded more warm, more authoritative, more customer service-like. And so that obviously has potential implications for human agency and human self-disclosure. If you're more willing to trust this voice, what does it look like if you're on the phone with a synthesized voice and you are trying to get out of a extra phone bill charge or something like that? What does it mean when these voices can be synthetically seen as more authoritative or more trustworthy?
26:33Kaitlyn Zhou:So those are like future questions that we're actively trying to explore and trying to make sense of.
26:39Linear Digressions Hosts:And so then when you were studying this, just a quick methodological question. With epistemic markers, for example, you have just the text on the page and you can see if there's something that's like, I am confident that or I'm not sure. But for warmth, it seems definitely more qualitative. are you primarily asking listeners for kind of their impression you play them the clip and they rate it for you or are there ways that you think about analyzing the audio itself for some kind of some notion I don't even know how you would do this but yeah for warmth or for other tonal I'm sure there's like audio terms for this but yeah the tone of the voice as it's recorded yeah we can
27:18Kaitlyn Zhou:do both so in this paper we're primarily focused on human perceptions of the voice as humans rate it. Our follow-up work is looking at deploying a similar framework to what we did with the epistemic markers and actually getting humans to make decisions with these voices and see how their behaviors change. So can this voice be more or less persuasive? And can you figure out why it might be more or less persuasive? And then at the same time, you can do all this quantitative analysis on the audio snippets themselves. And so you can calculate how quickly the voice, words per minute, You could calculate pitch frequency.
27:55Kaitlyn Zhou:You could calculate prosody, like how the voice goes up and down. I'm trying to remember. Yeah. And there's some initial findings there, too. A lot of these voice cloning methods will produce faster speech. So if it's more fluent speech, you're not really hesitating saying ums or likes. You're able to get more words out per minute. There's, yeah, some natural transformations there. Cool.
28:17Linear Digressions Hosts:And then the last thing that I wanted to ask about, I find myself wanting to ask this of everybody. So forgive me. I always like to hear about how you as a researcher think about AI in your own usage, partly because you have all these insights about how to think about AI as this thing that you're interacting with as a user, but also you're also just trying to use it to get stuff done. And so I'm curious, like for you, are there any like interesting ways in which you use AI or don't use it perhaps in your day-to-day work? And especially if you don't use it in particular ways, if any of those are like inspired by some of the research findings that you have, or I think at least for me and maybe for a lot of people right now, like figuring out where the line is something that we're all trying to figure out where it's helpful, but augmentative, but not delegative, maybe as mutual friends would say.
29:06Linear Digressions Hosts:So what does that look like for you? Yeah.
29:08Kaitlyn Zhou:And I think it's just going to be an ongoing process. I think we're all going to continue to learn how these technologies are going to stabilize and become a part of our lives. And there's a lot of continued learning to do for everyone. Maybe some things off the top of my head is, I think when you're, yeah, I think when you enter something in Google search, always being very careful when you are reading the AI overview and when you choose not to. I think most of the time I choose not to, unless it's really for something trivial. And so I always try and be good and scroll past the AI overview and actually go to the primary sources.
29:42Kaitlyn Zhou:when it comes to writing code in workflow and using quad as agents to complete tasks. There's differing opinions on this, and I find younger students are more likely to trust the AI systems and let them run for really long tasks, long horizon tasks. And I feel like I'm in the middle where I like to have the AI write code for me. Not that I don't trust it, it just feels more reproducible if it's going to write code for me that I can execute and I can reproduce as as many times as I need to, and I can write the code and edit the code as needed. So I usually don't have Claude do computations and just trust the numbers that it would give me back in a table.
30:22Kaitlyn Zhou:I almost always make it write the code that I could see, interpret, and run that code that way. I, yeah, like I mentioned earlier, I try really hard not to anthropomorphize the language models, even though it's, there's past theoretical work on this, that it's human nature. And so I try really hard to not. I'm the worst.
30:39Linear Digressions Hosts:Yeah, I try really hard not to say I'm sorry. I try really hard to say not to say thank you.
30:44Kaitlyn Zhou:Like all those polite things that we would do in human-human interactions, I try to avoid doing that. This is a good time maybe for a plug for there's another thread of work that me and some colleagues at Stanford have worked on, which is this idea of attention to non-adopters. So I think one thing that we've been thinking about there is that currently the way we develop language models is really focused on how current adopters and current users are using these systems. So when you talk about interaction logs, when you talk about user studies or online crowd workers, our study found that most online crowd workers are early adopters of AI.
31:20Kaitlyn Zhou:this is actually a small minority of people in the United States that the majority of Americans actually have not used chat GPT and don't use chat GPT in their daily lives. And so there are important kind of design questions to be asked here of if we continue to design for this population that has adopted it into their daily lives, do we maybe forego other opportunities to expand to a wider use case. And so when you think about people who are older, people who maybe aren't bi-coastal elites, like how are they thinking about using AI and integrating into their daily lives? And what would be helpful for them to leverage some of this very kind of impressive technology as well?
32:00Kaitlyn Zhou:So yeah, that's very top of mind too.
32:02Linear Digressions Hosts:Yeah, thank you for that plug. I think that's actually really interesting and important work. And I'm glad folks in academia are thinking about it. In some of what I've read, it definitely suggests pretty strongly that the types of folks who are non-adopters are very much concentrated in certain pockets of society that are, for example, older, less affluent. They might be disproportionately working in jobs that have a physical labor component to them, so they're not sitting in front of a computer all day, basically. And you can see how that probably correlates with a whole set of concerns that then are just blind spots for the AI is this getting developed.
32:41Linear Digressions Hosts:Yeah. If they're not like participating in the, in like the training loops, basically.
32:45Kaitlyn Zhou:Definitely. Yeah. One thing that became obvious was that non-adopters prioritize a lot more physical tasks, but it was also true that physical tasks are not just physical tasks. They also involve a lot of cognitive labor. So there's a lot of planning and reasoning and organizing around physical labor tasks. And that is perfect opportunity to offload to language models. And so there's just missed pockets of new tasks, new benchmarks that we can think about if we looked broader.
33:12Linear Digressions Hosts:So let's talk about that again sometime, but we're at the end of the time that I was able to ask from you this afternoon. I really appreciate the wide ranging discussion here. These are certainly like super interesting topics and I'll have links to all of the papers that we discussed. And there were a couple more that we didn't have time to cover here, but I think are also really interesting from you that we'll toss in and make available. for any of the listeners that we have. So Caitlin, I want to thank you again for coming on. I want to wish you the best of luck in your, at the beginning of your career as a professor.
33:46Linear Digressions Hosts:I'm going to talk to you again sometime because I think this is only going to get more interesting.
33:50Kaitlyn Zhou:Yeah, thank you so much, Katie.
From the publisher
When a language model tells you it's absolutely certain, is it actually more likely to be right? Kaitlyn Zhou's research says: not necessarily — sometimes confident phrasing correlates with *worse* accuracy, echoing a very human Dunning-Kruger effect. In this conversation, Kaitlyn (soon an assistant professor at Cornell) walks through why LLMs talk this way in the first place — tracing the tendency back through training data and the RLHF annotation process, where it turns out humans don't love confidence so much as they punish uncertainty — and what that does to the person on the other end of the chat window, who turns out to rely on confident (and even flatly-stated) answers far more than they should. We also get into her newer work on voice cloning, and how a cloned voice can sound more "native" and more trustworthy than the real one it's based on.