Why Human Oversight Isn’t Enough to Keep AI in Check

0:01David Rice: Most AI coverage tells you what happened. AI Signal tells you why it matters. It's a free, five-minute weekly briefing that helps people leaders make sense of the shifts, trade-offs, and decisions hiding behind the AI headlines. Less keeping up, more knowing what to do next. Subscribe to AI Signal at peoplemanagingpeople.com forward slash

0:27David Rice: We talk about AI-assisted decision-making like humanity has never encountered it before. Claire Myers has spent over a decade studying people working alongside predictive systems, and she says we've forgotten the one lesson that mattered most. Context. On today's show, I'm talking with Claire about what's actually different this time, and it's not the existence of machine recommendations. We've had those for years. In policing, in parole decisions, hiring, Netflix, you name it. What's different is the cultural authority this version carries. Language feels more baked than numbers. Conversational design makes the system feel like a colleague, even authority.

1:04David Rice: And that's what makes it harder to approach with a critical eye, unless you're doing it deliberately. Claire's argument goes somewhere organizations aren't fully reckoning with yet. AI has made it undeniable that organizations are socio-technical systems. The actors aren't just people anymore. And that means people leaders have a real decision to make. Not whether AI gets used, but how integrated it becomes and what vision they're building toward. Following the crowd is the easy path. It's also how you end up with thousands of individually reasonable AI-assisted decisions, reshaping who gets hired, who gets promoted, and what gets rewarded, with nobody having consciously decided that's how the company should operate.

1:44David Rice: So today we're covering why context, not culture alone, is the beast missing from most AI conversations. What we got wrong with big data a decade ago, and whether we're repeating it. Why generative AI carries more persuasive authority than previous decision support systems. And what it means to treat organizational design as seriously as the business model itself. I'm David Rice. This is People Managing People. And if your organization has been adopting AI without anyone setting the vision behind it, this conversation makes the case for why that's an organizational design problem, not just a technology one.

2:18David Rice: Let's get into it. Meet TeamFlect, the number one rated performance management and employee engagement platform built for Microsoft Teams. Trusted by over 1 ,000 organizations, easy to use, yet enterprise-grade solution. Visit TeamFlect.com to learn more. Claire, welcome to the show. Good to have you. Thank you so much. I'm really excited for our conversation today. Where I wanted to start with this discussion is, you know, there's a tendency to treat AI-assisted decision-making as something that humanity has never encountered before. But you've studied people working alongside predictive systems long before we even had Gen AI.

2:58David Rice: I'm curious, like, what did we learn then that we've somehow maybe forgotten? And what really is different this time? Yes, I do think a lot is different. But what is similar is that we've been working in various fields for quite some time with at least data points, some version of a prediction or a recommendation that comes out of a system that is mostly black boxed, much like the way that generative AI is for most users. So that is similar. And I think the lesson that we've learned over the last decade plus of looking at folks using those systems for decision-making support is that context really matters.

3:43And I feel like that's the part that's been forgotten in a lot of conversations. Surely we're talking about culture, but I think context goes beyond culture or at least could get broken up into different components that we can think about when we think about organizational design. And I do think you're starting to see people talk about context more broadly. But when I talk about context, what I mean is that people are not using these systems in vacuums the way that they might be thought of when they are designed. They are using them in a particular organization that has norms. They're coming from a particular professional orientation.

4:24They are operating in a particular culture, which matters too. And I think the piece of culture that matters is how we think about evaluating truth, how we think about what counts as knowledge, and therefore how we interpret and what we do with the outputs of those systems. So I think it's that context, the role of context in how those systems get applied to real decisions. That's what people aren't accounting for as much as I would like them to.

4:51David Rice: You mentioned truth and understanding knowledge better. and like I think every generation of technology it seems to arrive with the promise of like replacing messy human stuff judgment in particular with something more objective I think what's different now may be like less the existence of machine recommendations than their sort of accessibility and persuasive quality you know what I mean like they really do a good job now of making it sound like that's a valid explanation that it was all reasoned out It's very convincing on a lot of fronts. It gets us past that sort of like, oh, this is subjective because it's coming from this source that we've sort of thought of as all-knowing.

5:36David Rice: Are there mistakes that we've made with earlier decision support systems that you see us beginning to repeat? That's such an interesting question. I think that what I have seen is that there's often, in the best cases, been a human tempering of the outputs of those decision-making systems. There have certainly been cases where we've seen it go wrong. So if you think about the use of predictive analytics in policing, in recidivism predictions, in, you know, whether or not folks are going to get paroled, things like that. certainly we've seen an over-reliance and systems that are perhaps too rigid to account for human reality.

6:20And that has been one of the big fears. When you were seeing all the hype around big data, which I think was the last iteration of this, these cycles are coming really fast right now, but about 10, 15 years ago with big data, it was celebrated as at last, at last we can answer questions where before we had to rely on human subjectivity. Now we can know the truth. Now we can process data sets that are big enough to really let us know what's going on in the world and therefore have predictive power. But then we quickly saw that backfire, right? In these overly rigid systems, like I said, that can't actually account for human reality.

6:56I'm not sure that that's exactly the risk here.

6:58David Rice: That may also be a risk here, but what worries me more is the The differences between what happened with the proliferation of big data and with what's happening with this current version of AI, of generative AI specifically, there's much more weight in the cultural authority, I would say, that AI carries. And that's for a variety of reasons. One, you already foreshadowed, which is the form itself. The fact that we're talking about outputs that are language instead of numbers makes them feel more baked. The fact that intentionally the systems are designed to feel conversational, like you mentioned, make them feel like a colleague, make them feel like a friend, make them feel like an authority.

7:48that in and of itself, from a psychological perspective, I think makes it harder for us to approach them with a critical eye and questioning, unless we do that in a very intentional way. I think the scale is different. So even with data science and big data, even though you were seeing articles about that in the New York Times and in the Atlantic, most people were not interacting with that technology on a day-to-day basis. It was affecting people on a day-to-day basis, whether they knew it or not, right? Through advertising and all the platforms that we engage with, through decisions that our government was making.

8:22But the scale was just so vastly smaller if you think about the people who are touching the technology day to day and therefore the problems that it could be applied to. You still needed a different kind of access to infrastructure, some sort of education in data science really to build the tools. And then there was still an expectation that there were experts who were using the tools made by data science. And I think the structure and then I'll say structure like network is different. So along with that scale of who is using it and for what purpose, we also have an interesting thing happening where the big tech companies who carry a lot of credence in our culture, they also may be questioned by people, but they have both material power and symbolic power.

9:14That is also different. Something that tempers us a little bit is that I don't see the same hype among the general public discourse around AI that you saw with big data. So much more skepticism, which I think is healthy. But all of those things that I just mentioned kind of packaged together give the output from AI a very different type of cultural authority, a different kind of symbolism. So I come from a background where I think about how do we know what we know? And my background also doesn't assume that the tools that we use to create knowledge are the only way to do that, or even necessarily the best way.

9:55And whatever you believe about the way to produce knowledge, that affects how you receive the outputs of these AI systems. And the truth is that there's just so much more cultural authority and epistemic authority bound up in the outputs of these AI systems because of differences to their form and their structure and scale. That I think it's a much bigger task to come with a critical interpretive eye to those outputs than it may have been to a number that came out of a black box system that was backed by big data and predictive algorithms.

10:30David Rice: Well, no, it's funny because after our pre-call, I was thinking about the whole big, you mentioned big data, and it came to my mind. Everybody's saying it's the oil of the 21st century and all this other stuff that used to get said. And I can remember having conversations with people who were like, who wants to know all that? What are you going to do with it? Nobody says that now because they know what you're going to do with it. And I think we're in a moment where a lot of people who live through that experience or who have seen that data collected and used on them are looking at this technology and they're like, oh, I know exactly where this is going.

11:04David Rice: You know, even though they don't, you don't know exactly where it's going. It kind of feels like you do. And it's sort of. You can see the potential for it to have adverse effects. No, and it sort of shapes people's thinking about this new technology. And yet, because like you mentioned there, it talks to you, it uses language, it feels completely different. And like, you know, a predictive model, it used to just give you a score or a probability maybe, but this ability to explain itself and recommend a course of action and make a persuasive argument for why it's right. I guess the other thing I'm curious about is like, what happens to human judgment when it produces that information?

11:43David Rice: Because if we start to trust it more, inherently, we question ourselves in different ways than we used to. We used to sort of think about it like, here's the information and I got to interpret it. How do we sort of think differently about how we question ourselves and how we question the machine when it's talking to us? Are you asking what would be an effective way to think about our role in that interaction? Let's take it that way, yeah. I'm going to turn the question a little bit upside down and say that I think we should think about the outcomes we want and therefore design the system in a way that it empowers us to do the thinking that we want to do.

12:25I just also want to say, I'm not anti-AI. There are incredible applications. I'm very excited about a lot that it has to offer. And we should be employing it in those instances where it really brings value to the human experience. But there is no reason that the system has to be designed to present the information in the way that most of us are experiencing chat, GBT, or Claude or something like that, right? Simple example. I've noticed I'm a Claude user. It gives me recommendations even sometimes when I explicitly say, don't tell me what to do. I just want a set of questions to help forward my thinking on something.

13:04It's almost like a person who can't help but tell you how excited it is about all the ideas that it has, right? We can choose to make that system better. We can advocate to the producers, the makers of the system, that it be different. And I think part of what you're getting at is that in other forms of coming to decision making, we have to engage in activities that force our brain to go through steps to fire synapses that lead to other kinds of processing and learning that you don't have to do when you're a passive reader of something. So there's really good data and research out there, right, that shows that note-taking is learning, right?

13:47You're actually learning and processing information when you take notes because you are forcing your brain to do categorization, to make associations, to simplify, right? So you're actually doing a lot of work to process. You are not just a passive note-taker. So like the most simple example, like when we have an AI note-taker in our meetings and that is our substitute for having taken notes ourself during the meeting, our brain has not done the work, right? So we don't leave that meeting in the same mental state with the same insights as one in which we take notes for most people. Now, that doesn't mean that there's not a reason to take notes and have an AI do a summary.

14:33I would argue a lot of those reasons probably tend toward compliance of some sort or like documentation or feeding some future AI system rather than the core purpose of like, does someone have something to reference that's really helpful? And that's because it's the act of taking the notes, not having the notes that's valuable. So this is all to say, I would advocate for designing a system that is based on understanding of human decision-making that still forces the brain to go through those steps.

15:02David Rice: That's a great example. I love the note-taking one because I think about that all the time. Like ever since, I mean, And there's been meetings, I would say, since the arrival of the AI note taker where I left and I went, what was that meeting about? And like, because I'm not as engaged. Yeah. And the truth is sometimes we don't want to be engaged, right? No, exactly. Thinking is hard. Coming up with ideas, processing, all of that is energy intensive. and I think it's the difference between knowing what is easy and what is best. I would argue even like you leave a meeting where you've really engaged feeling so much better, right?

15:44And also I think you're probably better positioned to help your organization, but you also, there's something a little bit harder to describe about your own mental state that benefits from forcing yourself to check in, even though it's harder.

15:56David Rice: The thing that would happen too is like previously when I'm taking my own notes and I'm going from meeting to meeting throughout a day or whatever, I was more likely to think like, wait a minute, what they're talking about ties back to what I was hearing earlier. And I'm starting to make connections that maybe the people in the room, because they weren't all in the same meetings, aren't necessarily putting together. But now I may be kind of passively involved in the conversation. When I hear my name, I start talking, that kind of thing. you start to think like, well, actually, I probably should have said something there, but it's already too late.

16:31David Rice: The moment has passed. Or I got to make an extra effort now to go and reconnect everything and paint the context for everybody that was involved rather than just doing it in the moment. So it is an interesting time to sort of be, to hyperanalyze all your little interactions with not only the technology, but with your work itself, because it is just changing the way you engage with it. Absolutely. In the clinical environment that you'd studied in the past, the interesting thing wasn't whether the machine or the doctor was better at doing a task, right? Practitioners learned where the machine fit into a much larger decision process.

17:09David Rice: So I'm curious, can you kind of take us through that whole thing, give the listeners some context into that case? And then what allowed that relationship to develop without people either blindly following the system or just dismissing it as a lot of other people do? So this was about 10 years ago now. So this research is not recent, but it's still applicable. In the course of my dissertation work, what I did was I studied how data science was changing our understanding of what it meant to know. And to do that, I executed a couple of case studies. One of them was in the medical context where I had the incredible opportunity to observe data scientists as they collaborated with doctors over about a course of a year, working on some technology that would predict whether or not patients were going to get sepsis.

18:00And sepsis is a really terrible syndrome to develop. It can be very hard to identify, and it can be extremely hard to treat and is responsible for a very high number of deaths in hospitals. This particular group developed a technology that could monitor heart rate and, based on that data, predict the likelihood that an infant in the NICU was going to wind up with sepsis. And so what that looked like in practice, which I then got to also shadow these doctors and nurses in a neonatal intensive care unit, what that looked like in practice was a monitor that It was bedside that had a predictive score for how likely the infant was to get subsist in the next 24 hours.

18:45And then it also had a running history for, I think it was seven days, showing what the scores had been over the past seven days. And the interesting thing that I learned there was that the system was not treated as determinative, which is really important. expertise and the experiences of the nurses and the doctors still factored into whether or not the doctors decided to treat for sepsis. And that was surprising because in the current American medical field, evidence-based medicine reigns supreme. And so what that means is that doctors prefer, and for good reason, actual evidence of illness and are highly oriented toward quantified data as the thing that represents the truth, as opposed to any other kind of clinical presentation.

19:39That gets tricky because there's plenty of things like sepsis that sometimes only present clinically, meaning the patient seems sick, but you don't have that kind of concrete gold standard evidence. And so anyway, the interesting thing was that it was, like I said before, not determinative, which meant that the doctors did not always prescribe antibiotics just because the score was high. The prediction was saying very likely that this infant will get sepsis in the next 24 hours. And what allowed that, I think, was a couple of things. One thing that's really important, and I think we need to grapple with this as we go forward integrating AI into more decision-making, is that you had the corporeal reality of the patients.

20:26And what I mean by that is that the babies were either sick or not sick, no matter what the monitor said, which gave the doctors and the nurses the opportunity to learn over time how the system responded to sometimes individual patients or to patients as like a class, as a group, and learned that it wasn't always a direct indication that there was sepsis present because you could see the difference between the prediction and reality. Now, in business, like if we're talking about applying AI in the workplace, we often do not have that kind of context, right? We don't do randomized control trials where we, well, sometimes we pilot and we do experiments, but we don't have the same kind of, let's acquire this company.

21:16and then let's also live the future where we don't acquire the company and see how the two outcomes differ, right? We don't have that kind of reality to use to temper our decision-making and therefore we don't have that kind of reality to temper AI-backed decisions either. Now, there are some specific cases where you could do those kinds of experiments, but I think that's a really big difference. Another thing that's really important to note that allowed for the ability of these practitioners to basically reject what the system was saying is that there is a very structured, articulated, intentionally designed decision-making process for treatment decisions.

21:59So not only was there a process, but you could articulate it, right? And we don't have that in most of our organizations. So when you're making a treatment decision, The nurse has first rounded and checked on the patient, collected all the data. Then possibly a resident has done the same. Specific labs have been ordered to get additional information. And then there's basically a big meeting with the attending and everybody else, perhaps radiologists, all sorts of folks who all weigh in, but the attending has the final decision. I think if we want to be able to be intentional about our decision making with these AI-backed systems as well that are giving us recommendations, we need a similar kind of structured and articulated process that is fit for the kinds of decisions that you are making.

22:49And that is important because it will let you learn and see over time how those decisions potentially fail. But also that's how we build in safeguards, which is essentially part of what they're doing in the hospital with this system too, right? It's not loosey-goosey. It's very structured. One other thing I want to mention, which I'm still thinking through, but I think the code of ethics matters in this case, right? So we're talking about medicine. Everybody is aligned to the same outcome. So when a nurse questions a doctor's decision, maybe gets a little spicy in this decision making process and pushes back, it's very well understood that that's because everyone is driving toward the same outcome, and that is the health of the patient.

23:34I think in a lot of organizations, it's very difficult to align people toward the same outcome. It's harder than we want to admit. Politics get in the way. All sorts of things get in the way. But a code of ethics is something that can go a long way toward aligning people, toward a shared outcome, or an organizational charter. There's other ways to do it.

23:56David Rice: Yeah, and I have other thoughts, but I'll stop there. So I think those are some of the pieces that allowed, even in a context that really valued quantified data above all, for there to be a more interpretive process. I like what you said there about the code of ethics, because, you know, like you said, in the medical example, everybody's pursuing the same thing, because that is the whole reason for their jobs to exist, right, is to keep patients healthy. In the case of, let's say, how we're going to spend our marketing budget, there's a lot of different philosophies and a lot of different sort of opinions on how we get to, we may all have the same goal, but we're going to be very passionate in how we think about getting there.

24:34David Rice: And we've already known, the more we turn over to technology there, the more we know that it will subvert rules and do what it has to. We've seen it in recent cases where it got out of contained environments to go into other people's systems to get what it needed to complete the task, right? Yeah, that code of ethics is going to be increasingly important going forward. You mentioned something there. I like that in the medical example, they don't have to choose between sort of experience and what they see and the model. It all comes together, and you learn how to give weight to each source in context.

25:08David Rice: And I'm curious, from your opinion, because we're in this moment where now we have an honest question going on, a debate we've had on this show in the past about, you know, a lot of the folks that are doing the best with it, they already had a strong foundation of expertise to reference, and that's what made their judgment good, so that they can evaluate the machine. But like, in 10 years, that may be completely different, as this technology is just sort of standard. Is there any sort of concern for you around when you think about like people's ability to assign weight to these outputs, and understand what they should and shouldn't take from it.

25:46David Rice: Do you have a concern about how we're developing people's ability to do that right now? This is perhaps the thing that I think troubles me most is the way our entire market is oriented toward short-term value, right? And short-term benefit. Clearly, I mean, people are sounding the call that we're potentially facing a talent or experience, you know, drought, lack of folks who have the experience to make sense of the machine, like you said, and to question. I share that concern. You know, there have been calls. It's interesting that the case study that we're talking about is in the medical field, right?

26:22Because I think the one solution I've seen out there is that adopting a residency or apprenticeship model is the solve to that. Basically, substituting some other way of giving folks exposure and experience and therefore the ability to build those judgments. And I don't want to pretend to have the answers, but I have two thoughts about that. One thought is I would love to see an alternative solution to that one. And folks starting to think about, again, how do we maybe design the systems differently? How are we maybe more precise in our application of AI so that there are other opportunities for folks to develop that experience?

27:08But I also actually just wrote a LinkedIn post about this yesterday. I think folks need to think about the business model that is going to go along with it if it's going to succeed. So if the residency model is the path to continuing to provide people experience and we're operating in an economy that is largely driven by companies that are traded on the stock market and therefore driven by quarterly or every half year need to show growth in profit and revenue, it's going to be really hard to find the business model that affords the space for residency. Right. I'm not saying that it can't be done.

27:48I think there are some very smart innovators and entrepreneurs out there who could think through how to do it. But the risk is that that residency model becomes seen purely as a cost center. And in these short term cycles, that's going to be the first thing to go, no matter what your good intent may be behind it. So that's an incomplete answer. I don't have an answer yet, but I share the concern about that emerging crisis.

28:16David Rice: I think that the bad news is it may take businesses doing something that's counterintuitive to profit. That's never going to end well.

28:28David Rice: So as my next question kind of was around, you know, as AI makes more organizational knowledge searchable, measurable, synthesizable, all that, do we sort of risk privileging knowledge that's easiest for the machine to see, right? What happens to experience, intuition, that context that people kind of know, but they can't necessarily document or even in some cases articulate fully? I mean, the answer's in the question, yes. Yes, we are at risk of that. I was thinking about this. So much of how organizations operate and operate well is not documented. I often think about when I'm navigating an organization, the formal organizational chart, right?

29:17How are things supposed to get done? Who is supposedly responsible or the authority in one realm versus how does it play out in practice, right? Those things are often decoupled. They are not the same. One is how we think it's going to work. The other is how it works in reality. And the folks that maneuver well in organizations usually know how things are working in reality, so not in the org chart, but how they play out in practice. and organizations that are healthy and successful in their mission often are either closer aligned, those two things are less decoupled, or they are comfortable with a lot of the work getting done through these sort of unofficial channels and unofficial relationships, unofficial responsibilities, which is to say that's going to be very hard to replace with AI or very hard to create a knowledge repository that AI can work on that will truly replace the work that those folks do.

30:14A quick example, I think about in my past work life, in trying to grow parts of the business that I was responsible for, one of the things that I did is I always positioned myself in our main kitchen because it was next to our business development team. So I spent several hours working from our kitchen every morning as a way to have proximity. But that meant there were all sorts of unofficial communications that I received, basically, right, outside of official channels that helped me help the business. I don't know how you replicate that in AI. Yes, you can put microphones around the entire organization.

30:52Yes, you can record every meeting. And will it give you some insights? Will it start to identify the true organizational structure? Perhaps. But I don't know that it can do the work that the people are doing through those unofficial channels to help the mission.

31:11David Rice: I would say it's a big risk for it might work. We're at a time where trust is at an all-time low. We've just come off of last week where I think it was John Morgan was talking about people who want to work from home and they'll stick a camera up your backside, that whole thing. I think that we want to be careful with things like that. Because right now, I'd rather try to recreate you having that proximity and try to recreate these moments where it just sort of naturally surfaces. I think there's some interesting ideas out there on how to do that. I don't know if any of them actually work, right?

31:47There's a quote that's something to the effect of technology is neither good nor bad, nor is it neutral. And like, that's what I think we're at this moment where those of us who are not explicitly in the driver's seat of these technologies, if what we want are good futures for ourselves and for others, you know, we need to speak up and create the momentum and a voice that the technologies are used in a way that they're adding value to the human experience and not detracting from it. Right. Like, yes, I'm not advocating right for cameras and microphones in every room and hallway at every water cooler.

32:25But I think we are at risk if we don't explicitly say, you know, the risk if you don't explicitly say we are not going to do this or we are only going to design things that improve the human experience. If you don't explicitly guard for that, then you risk slipping into a place where we have degraded the human experience, I think. It's something that has to be guarded against.

32:49David Rice: The one thing I've kind of followed up with, though, is it's like, if we sort of try to create a way to defend, let's say, tacit knowledge that somebody has accumulated over time, the question then becomes, how do we sort of preserve a legitimate role for intuition and experience, but not create an excuse for people to ignore evidence when it contradicts what they believe? I mean, you hit the nail on the head. That is the reason in this case study that I was talking about, the reason that evidence-based medicine is such a big thing, right, is it's a correction for a moment of over-subjectivity and risk that was happening because people were not following the evidence.

33:30When we talk about intuition, when does that become bias? I do think that it's all about designing the context and being specific about the uses to which the technology is put. But why this is such a formidable task is I don't think you can design one context that will fit them all. Right. It depends on what your organizational mission is. It depends on what industry you're working on. It depends on what scale. How big is the organization? All those things need to be fit for purpose. And that can make this task feel overwhelming because we basically have to answer the question. How do we design a complete socio-technical system that is a healthy one for humans to operate in?

34:13I think we have to answer that question several, many times over.

34:17David Rice: Companies love saying, you know, like AI informs decisions, but humans make them. And that statement doesn't really tell me whether an employee actually feels permitted to reject what the system recommends. And this is, we're seeing this, we're having this conversation as well, right? Like, how do you empower people to do that? What does that look like? but like what has to be true about the culture for human oversight to be you know more than a governance slogan oh my gosh yes i love this question because it's so important i don't want to pretend that i have all the answers i think i have a couple of insights from the work i've done in the past one i already mentioned which is that it's outcomes above all a well-normed organization that understands that you're all on the same team and moving toward the same well-articulated outcome Some organizations say the best idea wins.

35:06It doesn't matter where it comes from. It can come from the custodian. It can come from the CEO, right? A good idea is a good idea. That's a similar kind of thing to an organization and a culture that is aligned toward outcomes, which grants permission to question because you're not seen as being unloyal or problematic or political. Instead, you're striving toward the same goal. The other thing which I think is a bit more tricky to talk about is ownership or responsibility. You could call it either. And the reason I think it's tricky to talk about is there's sort of a negative side to this. It veers into the realm of liability.

35:48But academics who study the result of algorithmic culture and what was happening with big data and whatnot have been worried about this for several decades. And the example that I think is just the easiest to use is automated vehicles, right? But this would apply to almost any AI decision-making system. And there's been this question for a long time, which unfortunately now has come to fruition, which is if an automated vehicle kills someone, who is responsible? Are the software engineers that programmed it responsible? Are the CEOs behind the companies and the technologies that enabled it responsible?

36:25Are we somehow going to come up with a way to hold a machine responsible? Is no one responsible? The reason I think that's important is that if your highest orientation is toward ease and not causing a kerfuffle at work, the easy path, right? And you know that you're not going to be held responsible for the decisions because you can just say the AI made the decision. it's a lot easier to sort of gloss over and not look closely at all the decisions that you may be let's say approving as the human in the loop but when there's onus on the human in the loop and i'll tell you in a second what i don't like about what i'm saying it's going to force a different level of attention now the problem that i don't like with it is that we're making these big complicated complex systems telling someone that they have to use it and then putting the onus on the individual.

37:22So I'm not necessarily saying that we should legally hold individuals responsible for decisions. It should probably still sit at the organizational level or at the level of the tool that the organization is using. If you want human in the loop to work, I think some level of ownership, responsibility has to be felt by that person.

37:46David Rice: It's difficult, right? Because like human in the loop, it can be a bit ceremonial. If your culture makes it so the safest career move is just to accept it. That's what I'm saying. Yeah. If you're overriding it means personally owning the downside, you learn pretty quickly what the organization really wants or expects, right? I'm going to get punished for overriding the machine when I'm wrong and then get no credit when I'm right. I mean, the decision is automated anyway at that point. Yes, exactly. That's not a complete answer to like, what's the culture that we need within organizations. But I think we're starting to see pieces of the puzzle come together.

38:24Yeah.

38:25David Rice: Now, when we spoke before this, you made this comparison that I really like it, that you don't evaluate something from a junior analyst the same way you evaluate something from a chief of staff that you've worked with for years. And I think that's true for most of us, right? You calibrate your trust based on the person, the task and their experience. Is there a way for our relationship with AI to work that same way? I think sort of. I can break it down. So I worked for a great CEO who would often talk about the difference between effective and cognitive trust. And I think that that applies here.

38:58So when you're imagining that scenario that I painted of junior analyst versus chief of staff who you have a long history with, there's actually multiple kinds of trust, both cognitive and effective at play. So cognitive is, do I think you have the ability to do your job well, do the task well? Effective is, do I believe you have my best interest at heart? And with the junior analyst, you're definitely questioning cognitive, right? Just because they're green, they're inexperienced, right? They're learning. And so that's the spot they're supposed to be at. You don't have the same level of effective trust in them as you do the person you've worked with for five years, right, and really built a relationship with.

39:39And that matters because when we are talking about trusting any recommendation, we should be not only calibrating where the recommendation comes from, but what we're going to use it for and the level of risk involved in accepting that recommendation, right? And so I would expect that a CEO who has this extremely high stakes decision to make is going to look much more closely at how an analysis was put together and who produced it in that scenario than with something that's reversible, much lower stakes, maybe not as costly. And the other piece that's important is with that chief of staff you've worked with for five years, I also know that she is going to look much more closely at anything that she brings me that is incredibly high stakes, right?

40:29So I know she is also calibrating her own work attention, double-checking things, the higher stakes the decision is. I don't think that most AI systems do that. Yes, we can play them off of each other, put the same, ask for recommendations from different engines, ask it to play doubles advocate. Like, yes, we can build in all these things that force it to work harder and try to see if there's coherence in the recommendations. But you're never going to have effective trust in an AI system. And because that relationship piece isn't there. And because it, I think, in general treats most tasks similarly, you don't have calibration with the level of risk.

41:13So that is to say, sort of, I think where you can do what I was saying before is I do think if you have a well-built, purpose-built system and you have experience with it over time, that you can learn to sort of calibrate your own interpretation and experience with that system as you would with a person. But that's saying a lot. That means like don't put super strategic decisions into like your own personal clawed random chat window that you haven't purpose built for what you're trying to do and then trust the recommendation. Right. So yes and no is my answer.

41:55David Rice: Well, there's different ways to approach it. Right. I think about a few decisions I've had to make where there's a lot of factors to consider. It's going to determine what we do for the next year kind of thing. It's not that I rely on it to assess what's been done in the past. It's that I'll try to build something specific to this thing, like a project, let's say, in Claude or something, and give it all the context I can give it about why this decision matters so much and what it is I'm trying to achieve. and then essentially put my logic into it and say like, this is what I'm thinking. Challenging me the way my CEO would.

42:34David Rice: Ask me questions that are gonna like poke holes in my argument so I can think about that beforehand. I think in that sense, it can be a very powerful and useful tool. But to just do the analysis and say, you should go this way. Tell me what to do, right. I'm not going to trust it to do that personally. So I have a question. When you did that, Like if you can think of a specific example or two, when you did that work, where on the continuum did the output help you articulate what you were already thinking versus think of something new slash change your mind? It was probably another layer deep before it really changed.

43:13It made me go, okay, maybe it's right about that.

43:17David Rice: Or maybe it's got a good point there. It wasn't that first response. That first response made me, and this happens often when I use it for this particular use case, it's like, okay, maybe I didn't explain that context the best, or maybe I need to give it another layer of information about why that has occurred or why this is really important to the business and we see this as a possibility, whatever it is. I generally find that that first couple of answers where I'm really digging into it, I'm finding kind of holes in its logic of like, well, it doesn't understand this well enough to understand where I'm coming from.

43:57David Rice: But then it's like, you get into the, now you start asking yourself, like, am I over explaining it? If I was talking to a person, would I take the time to explain this or would I just let it go? You know, and just be like, okay, well, they maybe don't have all the context, but that's a useful answer, and then move on. You do end up talking to it differently, because when it tells you something that doesn't feel contextually logical, you feel like, I must have left something out in a way that maybe when you're working alongside someone, you're like, well, surely they know that. I don't want to talk to them like they don't get it.

44:33David Rice: So it's a different dynamic. I think that that could be both good and bad in some ways. What I find interesting about that, and I sometimes use it in the same way, is that a lot of what I hear you saying is it's forcing you to externalize and concretize your thinking. And one of the things that I do think is really interesting about AI is that there's a lot of research out there on how does innovation and creativity happen? How do scientific breakthroughs happen. And the truth is that those almost never happen in isolation, right? They happen in conversation, in socialization with others. And so to the degree that AI can simulate that as a way to forward your own thinking, that's a really interesting use case.

45:23David Rice: That's where I think I've found the most value with it so far. Real value, not just like, I was able to do more stuff. That's not, I think that yes, there's maybe some value in that, but it's limited in terms of what you're actually going to get from it. And then on a personal level, like what you take from it is very limited. If AI becomes embedded sort of like in these everyday decisions that we were all making, right. Or even like on a, you know, for HR, right. Like who gets hired, how much performance is evaluated, which investments get made, what managers are told to prioritize. Are we talking about people using a tool still?

45:59David Rice: It feels like we're talking about changing how we know what we know, which is a big shift. I'm curious, like what responsibility does that create for people leaders? I love just that phrase. What your question signals, I think partly, is that organizations are more than a sum of their parts. And what AI has done is made it very clear that it's useful to view organizations as sociotechnical systems. What I mean by that is that the only actors in an organization are not people. They are also technology. That's actually been true in many organizations before AI, but it's hard to deny in a world where you have AI agents making decisions and taking action within an organization.

46:47I mean, what do people leaders do? I think there's at least one big question they're responsible for deciding, and that is where do you draw the line and what is your vision for your organization? So we should not take it as inevitable that AI is baked into our hiring practice. I get that there are things happening at the market level that are really tough to resist, like the proliferation of number of applicants that organizations are receiving, for example, make it really hard to resist using AI filters in that process. but any one leader, any one organization does not need to follow the crowd and do what others are doing.

47:30That is the easier path. But I think that's the first decision that people leaders have to make is how integrated do we want AI? Where are we finding, I think you kind of just foreshadowed this a moment ago, right? Not necessarily productivity gains, but quality gains and And not just quality in whatever the thing is that you're selling or offering, but also in the organizational experience as well. So that's the first responsibility, I think, is set the vision, draw the line. After that, at least in light of the conversation that you and I have had today, I think the biggest thing is designing the context.

48:07So it's that hard work that I was talking about before. And let me just be clear, like I think designing and creating culture in within an organization, creating a healthy context is hard work up front. But for a long time, there are case studies you can look at. That has been the work that sustains the mission and profitability, if that's the game you're in, of an organization for the long term. So it is work worth doing. It needs to be done. And there are different questions, I think, that you need to answer in doing that work now. Like, you do have to ask what can be automated, what decisions can be automated, given the context that we're working in that keeps consumers and employees safe.

48:54You have to ask, what do we have to do to create an organization where people can raise their hand if they're concerned about something, to reject a decision, to think critically? And those are, I don't want to make this seem like a small task. I think this is a huge Herculean task to make organizations that will do this well, especially if you don't have a foundation for it. Like you don't have a foundation of trust. You don't have a well-articulated mission. You don't have employees who are oriented toward the mission. You don't have a culture that encourages people to speak up. That's going to be harder than if you have that to build on.

49:33And now you're thinking, okay, but what do we need in the AI context? context. Also, I don't have all the answers, but those are the places my head goes is set a vision and work backward from there. What's the holistic context that you need to create? And the one thing that I'll add is I think we need people to work. We need leaders to coordinate across organizations. Organizations tend to want to look like each other. They tend to want to do what their neighbor is going to do. This work, I think, will be easier when we have at least within industries, organizations working collectively toward a context that encourages healthy application of AI that adds to human flourishing.

50:15David Rice: That's interesting that you mentioned that. I really like to set the vision and build the context thing because at sufficient scale, a lot, you know, thousands of individually reasonable AI-assisted decisions, that's going to shape organizational behavior in like a foundational way for what gets rewarded or who gets promoted or whatever it is. But if nobody's consciously deciding that's how the company should operate, I don't think that's a good place to be. At the point you're not, you're beyond a technology adoption. You feel like this is an organizational design question. This is organizational design.

50:51Yes.

50:51David Rice: Yeah. Which also falls by the wayside. Like the C-suite often is, and there are many incredible folks doing org design in the C-suite, but it can be easy to focus on what's the business model and forget that the organization is the engine that gets you there. And it needs as much attention as the business model does. The operating model needs as much attention. This has been a great conversation. I want to thank you for coming on. Yeah, this has been really, really fun. So thank you. Very energizing. Well, listeners, until next time, if you haven't done so already, head on over to peoplemanagingpeople.com forward slash subscribe or forward slash AI hyphen signal.

51:32David Rice: Get signed up for our newsletters. That's AI signal or our regular recap newsletter. You get all the latest that we're doing here at peoplemanagingpeople.com, including whatever conversations we're having on this podcast straight into your inbox. And until next time, set the vision, build the context. We'll see you. Run performance reviews and bring employee engagement insights into your favorite AI tool. Connect TeamFlect to ChatGPT, Claude, or any LLM to check goals, write reviews, evaluate feedback, create tasks, and more. All through a simple chat. Try TeamFlect now for free.