Best of Pre-Mortem Analysis: Gary Klein

0:06I'm Ted Seides, and this is Capital Allocators. For the second year, our summer series will share some of the best episodes of the past, curated from our favorites and yours. You can pull down many of the great ones on our Spotify playlists. This series pushes them to your regular listening feed. Our 2026 Summer Series focuses on improving the investment process. Over the next six weeks, we'll share 10 gems from interdisciplinary skill sets that will help you level up your investing across decision making, interviewing, storytelling, negotiations and habit formation. This week's Summer Series covers the premortem analysis with its founder, Gary Klein.

0:55Gary discovered premortems while working with fighter pilots and wrote a white paper applying the tool to investment analysis alongside Paul Johnson and Paul Sonken. Before we get to the interview, we're excited to introduce the next chapter of Capital Allocators University, an online on-demand curriculum with an annual in-person gathering for investment professionals committed to continuous learning. One of the great ironies of investing is that the skills that matter most over a long career are often the ones least formally taught. Things like decision-making, communication, leadership, time management, interviewing, and building relationships.

1:32That's why we've evolved Capital Allocators University into an online on-demand experience, making those lessons accessible to investment professionals everywhere. The best careers don't compound through knowledge alone. They compound through knowledge and relationships. If you'd like to explore the curriculum or someone on your team comes to mind, visit the link in our show notes or the university page on our website. Thanks so much for spreading the word about the next chapter of Capital Allocators University. Capital Allocators is brought to you by AlphaSense. Here's something for you. Most AI tools today are very good at sounding right.

2:10But can you actually trace it back to a filing transcript or specific passage that drove the answer? Or are you just trusting the confidence of the output? For allocators, that's not a minor concern. A missed filing, incorrect source, or context that gets lost somewhere in a retrieval chain aren't edge cases. They're how decisions go wrong. AlphaSense is the AI platform built specifically for this. They own the content. Over 500 million curated documents from broker research and expert transcripts to filings and earnings calls. And they own the retrieval layer on top of it. That means every answer can link back to an exact verifiable source.

2:57Because the answer is only as good as what's underneath it. And with Alphasense, you know exactly what that is. See it for yourself. Try a free trial at alpha-sense.com slash capital. That's alphasense.com with a hyphen in the middle slash capital. Capital Allocators is also brought to you by Intap DealCloud. You already know Intap DealCloud. It's where your firm keeps its pipeline, its relationships, and years of deal history. Now meet Celeste, Intap's new agentic AI and an expert coworker that works right inside DealCloud. Together, DealCloud with Celeste changes what your firm can do. Celeste runs on your firm's own data and your own playbook, how you source, diligence, and decide.

3:49It works inside your compliance rules from day one, so it screens your inbound deals against your thesis far more than your team could reach on its own. It tells you which LPs are most likely to re-up in your next race. Hand it a sim, and it builds your first LBO model in minutes. Every firm can only chase as many deals as its people have hours to work. Celeste lifts that ceiling. Now the deals you walk away from are the ones you meant to. DealCloud with Celeste is generally available now from Intap. Learn more and book your demo at intap.com slash capital allocators. Capital Allocators is also brought to you by Admired Leadership.

4:33Back in April on episode 497, I sat down with Randall Stutman, the executive coach behind Admired Leadership, who's advised more than 500 CEOs, including some of the most respected names in asset management. Randall introduced me to Alex, an AI leadership coach's team built on 40 years of proprietary research into what the best leaders actually do. For investment professionals, that means your entire team gets on-demand coaching grounded in the behaviors that drive results and build the kind of followership that retains your top talent. We use Alex, and our team at Capital Allocators swears by it.

5:18Try Alex for yourself at the link in our show notes, tryalex.admiredleadership.com. Please enjoy this discussion of the proper use of premortem analysis and investing with Gary Klein, Paul Johnson, and Paul Sonken from 2019. Gentlemen, thanks for joining me. Nobody wants to respond. There's too many people in the room, huh? Ted, it's always a pleasure. Thank you for inviting us. I know Paul and Paul have been on the show before, so I think it'd be fun given that they were bringing you to the table for this paper we're going to talk about. But why don't you go through your background and areas of interest?

5:59Sure. So I'm a cognitive psychologist. My primary area of interest is expertise. How do people become highly skilled? And that started in the mid-70s. And when all of a sudden the Arab oil embargo hit and you had all these pilots. I was working as a research psychologist for the Air Force at the time. All these pilots used to fly as much as they wanted. Now they couldn't because jet fuel was too expensive. And so they had to learn in simulators, which they always had considered as toys, but now they couldn't anymore. And so my job was to figure out what is expertise and how do you develop it and how do you develop it most quickly.

6:35How did you first get interested in that field? It was a requirement of the job. We had to help these pilots. They were sort of desperate to find out how do we design simulators that are going to allow us to become as skilled as we used to because we can't fly nearly as much. And what did you find in that initial research? So I'll tell you a story. I was doing a project and was down at Tyndall Air Force Base, and there were these two fighter pilots, F-106 pilots, and we were working on a simulator during the day, and at night, one of the evenings, we went to a bar, because that's where they usually went after we finished in the day, and they said, can we ask you a tough question?

7:12We're supposed to be developing training, and there's a system called Instructional System Design, and it's very procedural and takes a task, break it down into the components, and that's the way you teach it. And I said, okay. And they said, we're the ISD specialists for our group. And I said, okay, that's nice. And we're building the IOD specs for how to perform a lot of our flying tasks. And I said, okay, so what's the problem? And they looked around to make sure nobody was overhearing them. And they said, well, we've written all the specs, but we don't follow any of those rules. That's not the way we fly.

7:49So I said, okay, just write down the rules you follow. And they said, we don't follow any rules. We just know what to do. And that's what I've been hearing time and time again from firefighters and others, that there's a belief that you just follow rules and procedures. For complex tasks and ambiguous situations with lots of uncertainty, there's no set of rules that are sufficient. And expertise is not about just incorporating all the rules. Expertise is about building tacit knowledge. And that's the core. That's the primary thing that I've learned, is how to go after tacit knowledge. We hear a lot about checklists and Atul Gawande's checklist manifestos.

8:30How do you blend the necessary tasks with the intuition that comes on top of that? Right. And that's an important part, is that it is a blend. And Gawande, you know, comes up with these great checklists. But in the book, he describes how much work it is to come up with a reasonable checklist. And you can't have too many items. You can't make it too complicated. And so there's a real art to coming up with a checklist. How do you blend the other part? You break people out of the mindset. My job is simply to learn the rules and learn the procedures. And a lot of people do have that belief. And you've got to get them out of that mindset and get them into accepting that life is too complicated and they are going to be encountering situations that none of the checklists are prepared them for.

9:16And they better be adaptive and resilient in order to handle them. If we go back then to you're doing this research to try to distill how to help these fighter pilots and simulators, and then you learn that, well, the real way that they do it, the real way they develop expertise is in their heads. What do you do with that as a conclusion to research? Well, there was no conclusion for the work with the fighter pilots because we were building simulators. So we just had to make the simulators as effective as possible. Now the fighter pilots and the whole community had the belief you had to make the simulator look as much as possible like the real thing.

9:55That was wrong. It doesn't have to look like the real thing and you don't have to waste a lot of money on that. You have to have really good training scenarios that put people through their paces. And so that was the big aha from that project. But where the expertise work came in was much later when I started studying decision-making. And up to that point, there was a belief that a rational decision-maker lays out all the options, lays out all the evaluation dimensions, rates each option on each dimension and so forth. When's the last time you I mean, people sometimes do it, but it's very rare, and it's not an adaptive strategy.

10:33But that's what the field thought was the gold standard, because the field had mostly studied college sophomores handling unfamiliar tasks in laboratory environments, and so the field hadn't studied experts. And so because of my interest in expertise, I was able to look at decision-making and say, where's the expertise here? And that was the formulation for my recognition prime decision model and the discovery we made is that people don't generate lots of options and use lots of standard evaluation dimensions. They use their experience to size up situations very quickly and to identify what's going on, what's likely to happen next, what can I accomplish, and what should I do about it?

11:18And then to evaluate it, that's the intuitive part, but there's two parts to the model. the evaluation part was mental stimulation. If this is what I think I should do in this situation, let me imagine doing it in this context and see if it works. If it works, I'm fine. If it almost works, I can improve it. And if it doesn't work, then I'll go to the next option in my list. And so we came up with a different model of decision-making, and that's probably the accomplishment in my career that I'm the most pleased with. As you discussed that and looked at different applications, were there areas where it was highly effective and others where it's less effective?

11:58We haven't found many areas where it's not effective. And we looked. We said, we're not teaching this strategy. This strategy describes what people naturally do. So we said, maybe it doesn't get applied when there's less time pressure. So we looked at non-time pressured situations. We looked at designers. and they still were predominantly using a recognition strategy. And then we said, maybe if you don't have much experience to begin with, it's not a strategy you'll use. So we looked at novices. We looked at soldiers at Fort Knox, tank commanders who hadn't really had much experience in tanks.

12:34And we said, what about them? They were the group we've seen that did not use recognitional decision-making. They did look at options. They tried to identify the best option from a set. It didn't help them. They still did a terrible job, but they had no basis and expertise to use recognition strategies. Once you identified that the recognition strategies are the way that people go about making decisions, how did you then study different ways that people can improve their decision processes? That's where it really gets important, isn't it? What can you do about it? And so for maybe 10 or 20 years, we were looking for how do we turn this into training and into preparation so that we can help people become better decision makers.

13:17And we came up with nothing. And so it was very discouraging. And then about six or seven years ago, I was exposed to a technique that we call the shadow box, which is a technique that we've been using it for the last five years. And it wasn't developed by me or my team, developed by a guy named Neil Heinz, who's a retired New York Fire Department battalion commander. And he had developed this in the wake of the 9-11 attacks. And how do people handle situations that no training could have prepared them for because nobody could foresee them? And the way Shadowbox works, it's a scenario-based approach.

13:58So I give you a scenario and we walk through the scenario. And then at a certain point, I'd stop and I'd say, okay, Ted, here's four options of what you can do at this point. Rank order them and write down the rationale why you rank ordered them. But it's not just about the options. We may go a little bit further and then I'd stop and I'd say, okay, Ted, here's three goals that you could be pursuing. Rank order them in terms of what's the most important to achieve to the least and write down your reason. And then we might go a little bit further and I'd say, here's five different kinds of signals and pieces of information that you might pursue.

14:34Rank order, which you would go for and why. And on the side, we'd have a set of three to five experts who've gone through the same scenario. They've done their ranking. They've written down their rationale. So after you come up with your ranking and we say, here's what the experts ranked. And you want yours to match the experts. but more important, you look at what the experts saw. So this is really how to see the world through the eyes of experts but the experts aren't there because we already collected their data so we don't have to gather them and schedule them or anything like that and you see what the experts noticed in the same scenario you read and you see the inferences they drew and all of the warnings that they spotted and things like that that you just went right past and you didn't notice.

15:21So that's how shadow box works. Think about how you could revolutionize is the Harvard Business School case studies. And it has enormous implications for training security analysts. And Ted, you can see why from what Gary's been talking about that I think that, and that's why I was attracted to Gary's work, because it's directly applicable in so many different ways to the jobs that portfolio managers and security analysts do. You've now set up this framework with the shadow box where someone can go through a specific scenario and then have an expert way of assessing it. Is that just a specific training tool for certain scenarios?

16:02And where have you seen that applied? Okay, so we've applied it to panel operators and petrochemical plants. We've applied it with caseworkers doing trial protective services. We just did a project on systems integration specialists, trying to get younger engineers up to speed faster in that area. or I'm going to be talking to a hedge fund they're interested in in shadow box. So we've applied it in a variety of areas. Interesting. And what are some of the other tools that you've seen along the way? We have a method for cognitive after action reviews. Cognitive after action reviews, right. So it's called cargo, cognitive after action review guide for observers.

16:49And the idea here is a lot of times after a situation, after you've had something go well or go poorly, you do an after action review. And frequently the after action review is you did X and you should have done Y. If you had done Y, things would have turned out better. But it doesn't get into the cognitive part. When you chose X, what were you seeing? What were you interpreting? in hindsight, what could you have been looking at? And so trying to get into the person's head to see what they were considering at the moment that they made their judgment and decision, rather than just saying X is better than Y.

17:27So that's a method that we've been using now. Then there's also the premortem method that we developed, gosh, about 30 years ago. And it's probably the best known of all the techniques that we're using. What was the first insight in premortems? So let me just tell you what the premortem is. So the premortem is sort of a risk assessment method. And the idea is to try to anticipate what could possibly go wrong with a plan. And when I had my company, I had my research company for about 27 years till I sold it in 2005. And we had grown to 37 people by that time. And at a certain point, we said, we have a lot of projects that go well, but not all of them.

18:13And some of them don't go well. And we do a postmortem after it's finished. And it may help us in the future, but it doesn't help us with the project that finished. Why don't we try to move that up front to the kickoff meeting? And the analogy would be like in medicine, in health care. You do a postmortem after a patient dies to find out why. and then the physician can discover what killed the patient and let the family members know so they also gain. And if it's something interesting, they can write it up. So everybody benefits from a post-mortem except for the patient. Of course, the patient is dead.

18:52So the idea of a pre-mortem is let's anticipate what could happen at the very beginning of the project. So we started doing it with our own projects. We would add this, and it doesn't take very long. we'd added to the end of our kickoff meetings for a new project. And then in around 1993, we had an Air Force customer that wanted us to lead an effort to build a complex piece of software for them. And there were a number of Air Force engineers and there were other team members who had algorithms that were going to be included in this tool. And we said, okay, at the end of this kickoff meeting, we're going to do a pre-mortem because we had really gotten to like the method.

19:33and our contract monitor said, absolutely not. He hated the idea. He said, it's going to be too depressing. I don't want to do anything that might reduce people's confidence and enthusiasm. And we said, this is the way we do things. You know, you hired us to run the project. This is the way we do it. So we have the meeting. The meeting goes for a couple of hours. And it's the end and we're going to do the premortem. And we said, all right, now let's go through the process. and the way the pre-mortem works is you have a plan. Everybody knows what the plan is and it's the team that's done the plan and is going to do the execution.

20:11They're all primed. They're all ready to spring into action. And if I was doing a pre-mortem with you, I'd say, okay, now, now let's just step back a second. And I'm looking into a crystal ball and I might sort of imagine there's a crystal ball on the table in front of us all. I'm looking at a crystal ball and it's now six months in the future, maybe a year. Oh, I don't like what I see. I see a disaster. This plan has failed. It's been a complete and embarrassing failure. And the people on the team don't even make eye contact when they pass each other in the hall. It's been a really big disaster.

20:51All right, we know the plan has failed. Now, everybody around the table, take two minutes and write down all the reasons why this plan failed. And everybody starts writing madly. And then at the end of two minutes, I stop them and I say, now let's find out what people wrote. And I go around the table and I start with the project lead to see what that person came up with. and then we just continue around the table like that until we get all of the items that people had and that was fairly depressing so we added another step after we finished compiling it which is now that we've seen all the things that might go wrong that could go wrong everybody take another two minutes and write down what you individually could do to prevent some of these things from occurring.

21:44And so people go through that. And that sort of is a mitigating factor. So we did the pre-mortem on this project, this Air Force project. And we got to one person. He was, I think, a captain. He was the most junior person at the table. His specialty was information technology. And we got to him. He hadn't said a word during the entire meeting. And we said, what do you have? And he said, well, I'm looking at the algorithms that people are describing, and they're really powerful algorithms to do the job. And they run on a supercomputer and they take a day or two. And we're talking about putting this in the field, like during Desert Storm or something like that, to be used by a history major who graduated from Northwestern and is now in the reserves and he's now thrust into this job.

22:34And at that point, he might have had a 486 machine or something like that. It just won't work. The software won't run on that machine. And everybody looked at him and realized how ridiculous the quest had been up to that point. And had we pursued the vision that was the sponsor's vision, it would have been a disaster. And instead, people said, you know, I have a back-of-the-envelope approach that doesn't require a supercomputer. And it's not as good as a supercomputer, but it's like about 90%. And we said, that's what we want. Interesting. So there were a couple of things you mentioned as you laid out how the pre-mortem works.

23:14I want to see if you can talk about the significance of it. The first is you said there's two minutes for everybody to go around and write down. Is that just because you just threw out two minutes or is there significance to that timeframe? Trial and error. When we did this for a number of projects in my company, one minute was leaving people frustrated because people were still writing when they said time is up. When we tried three minutes, a lot of people would be finished and they'd be twiddling their thumbs and it would reduce the energy of the room. So two minutes seemed to be the sweet spot.

23:45Okay. And then you said when people start discussing what they wrote down, the leader comes first. Right. And the reason for that is people don't know how candid to be. And the leader is going to set an example. And if the leader picks something that's frivolous or trivial, that tells people not to take it seriously. So it's really important for the leader to come up with a real problem that hadn't been discussed before. Now, if you are conducting a premortem on the premortem process, what are the things that will commonly go wrong that makes a plan of a premortem fail? Well, that's one of the reasons that I've been writing this piece with Paul Johnson and Paul Solkin.

24:37We see people making mistakes. There are ways that the premortem will fail. First of all, if the leader doesn't have buy-in, then the leader will come up with something trivial and everybody knows they have to not surface their prime concerns and choose something that's safer and minor. So that's one part. We see premortems fail when the person who's facilitating will say something like, okay, we've seen the plan. Let's do a premortem. What could go wrong with the plan? That's the wrong approach. What can go wrong with the plan? That's a typical critiquing approach. you can't say what could go wrong.

25:15You say, I'm looking at a crystal ball. The plan has failed. There's no doubt. That changes the mindset that instead of just sort of musing about it, now you're seeing a disaster and you're now in a mode of what caused that disaster. It changes the kinds of issues people identify. Why is it that it changes it? There is a study from 1989 about perspective hindsight that giving people the perspective of a future situation where something that is counterfactual has occurred all of a sudden opens up access to kinds of thinking and speculation that people ordinarily wouldn't engage in. And the metaphor I have is like an eclipse.

26:03Right now, if we stare at the sun for like a quarter of a second or so, it's really bright and we look away. But during the eclipse, and I had a chance to see the eclipse a year or so ago when it happened, when the eclipse occurs and the moon blots out the sun and you look up and you see the corona, you see all of these trails streaming out of the sun and it's just gorgeous. And I had these issues I wanted to study during the eclipse and I was just paralyzed by the beauty of it. But it was all invisible because the sun is so blinding under normal conditions. The premortem is a sort of a way of blotting out what you normally think about and your enthusiasm for carrying out a plan.

26:46And so you need to cover that up so that these other issues can emerge and become visible. That was what we tried to do in the paper, was kind of tie some of these concepts back to research that's been done. But as Gary said, you're reframing the problem and then it's easier to find a reason for why something has happened, why the project has failed, as opposed to developing different scenarios of things that might happen. So it just kind of shifts the mindset. It's very nuanced and it's very subtle. And that's why we feel as though the premortem is misused because they'll just say, okay, what could go wrong?

27:28and they think they're doing a pre-mortem, but just that subtle framing kind of kills the effectiveness. I think there's one more thing that we sort of swept under the rug is that Gary says at the end of the meeting, you pause, take a deep breath. We're talking 15 seconds. And then you say to people, you have two minutes to write down why the plan failed and no one has a problem writing down two or three reasons. And that to me is so important because we've given them five seconds to think about it and then two minutes to write and everybody is scribbling things down. So that knowledge is in the group at the tip of their tongue.