The Insight: Conversations – Special Episode with Annie Duke and Howard Marks

0:08Hello and welcome to a special edition of the Insight by Oaktree Capital. Today, Howard and I are thrilled to be joined by Annie Duke, the former professional poker player, bestselling author, and expert on decision making. Howard's 2020 memo, You Bet, was partly inspired by Annie's 2018 book, Thinking in Bets. And Howard recently highlighted the book in Oaktree's annual book recommendations piece. So today, we're going to be talking about ideas from Annie's book. From Howard's memos, we're going to be exploring luck, decision quality, uncertainty, and much more. Let's jump right into it.

0:46Howard and Annie, thank you so much for joining me. Thank you, Anna. It's good to be here. Thank you, and I'm happy to see you again, Anna, and you too, Howard. Thank you. I think I'm going to begin with you, Annie. I'd like you to explain the main idea behind your book, Thinking in Bets. The main idea behind Thinking in Bets is that there's a particular thing about decision-making that's very hard, particularly trying to figure out why something happened, which is that there is an interference of both luck on how things turn out and hidden information. In other words, when you make decisions, you don't know everything there is to be known because you're non-omniscient.

1:26And after you make a decision, there is a strong influence of luck on what outcome you actually observe. And this creates a really big problem with trying to look backwards at things because we get an outcome and we don't actually know what the influence of luck was on why we observed that outcome. So we tend to bind outcome quality and decision quality much too tightly together. So the book's main concept is, look, you're making decisions under uncertainty. We need to embrace that and start to think about how can we improve the quality of our decisions and how can we learn better from the outcomes that we observe given the uncertainty and the environment in which we're decided.

2:07And now, Howard, as I mentioned, you cited Annie's book extensively in your memo, You Bet. You also blurbed her book when it was originally published. So what initially appealed to you about the book? Well, you know, there's an old saying, he had me at hello. Annie had me at the title. I think it's a terrific title. I think it's the way one should think. You should think about the things that you face that involve uncertainty, like a bet. You bet on things because you think you're going to win. What are the chances? And so forth. Betting on sports or investing requires you to cope with incomplete knowledge, live with luck, randomness, and hopefully bring some skill to bear.

2:50And you have to understand the interaction of those three things. Well, I just want to say that the next time somebody asks me to describe what my book is about, I just want to throw it over to Howard every time because that was a much better description than I gave. Now, Annie, as you mentioned, one of the key things that you focus on in the book is decision quality. This idea also features quite prominently in Howard's memo, You Bet. Can you explain exactly what you mean when you say decision quality? Yeah, so basically what we're trying to do when we make a decision is maximize the probability that we are going to gain ground toward whatever goal we set.

3:36Okay, so your goals might be different than mine, which means that a good quality decision for you might be different than a good quality decision for me because we're trying to gain ground toward our goals. Our goals are informed by us. Our values, the things that we value are informed by us. So when we think about a term like expected value, which is exactly how we would calculate, are we gaining ground on average toward our goals? Again, that's going to be personal to you and me. A simple way to think about it is if you're trying to make money, we can calculate expected value pretty easily, obviously, because then we've decided what the currency is that we value.

4:10So given that that's the situation that we're trying to gain ground toward any goal that we have set, decision quality is going to describe the accuracy with which we select options that will allow us to do that. And how do we separate when we're thinking about decision quality, the logic behind the decision and the outcome? Well, in the short run, you kind of have to ignore outcomes, honestly. Like in the short run, there's just way too much luck. So as an example, like if Howard and I were betting and he said, I'm going to lay you two to one on a coin flip, a coin being 50-50, I'm going to make 50 % on every dollar that I bet there.

4:50If I call heads and it lands tails, it means nothing. Now, if we were to do it a thousand times and I kept losing, then I could start to draw some conclusions from the outcome, like maybe Howard isn't using a fair coin. But in the short run, we have to really try to separate those things out and concentrate much more on what is it that we don't know? How do we model the facts? So two people can look at the same facts and model those very differently. How well are we getting to a perspective outside of our own? Because our own perspective tends to be pretty biased and incorporating that information.

5:25So it's a combination of what's the analysis, what's the quant work, how eager are we to get to other people's perspective on the information that we get. And if we do that, then we can assume that decision quality is going up, but it's going to take quite a few outcomes over the long run that's going to get us there. That's one of the reasons actually why I really emphasize that decisions are bets. Because what do you want to bet on that does is exactly what I'm talking about. It starts to make you think, well, how probable do I think this really is? How certain am I? What do they know that I don't know?

6:00If they're willing to bet against me. I have to consider that they have information that I haven't considered, or they're modeling this set of facts differently than me. And so that allows us to increase decision quality, even in a situation where short-term outcomes aren't particularly informative. And Howard, I know that that idea of separating outcome from decision quality was something you also mentioned, you cited quite a bit in your memo. So what are some of your thoughts on that? Well, I said at the very beginning of the memo that the first book I remember reading when I went to Wharton 60 years ago, September 1963, was Drilling Decisions Under Uncertainty by C.

6:37Jackson Grayson. The most important lesson I took away, because it made the biggest impression on me, was the statement that you can't tell the quality of a decision from the outcome. Poor decisions end in success often. And we all know people, especially in the investment business, where we say, well, he was right for the wrong reasons. That's what that means. Or I always say that their industry is full of people who got famous for being right once in a row. Being right once says nothing about the decision-making ability of an investor. And in fact, another thing I say, not so often, is that for success, there are three ingredients, aggressiveness, timing, and skill.

7:19And if you have enough aggressiveness at the right time, you don't need that much skill. But you also have to understand the difference. There's an old saying in the investment business, never confuse brains with the bull market. You do something risky, it works. Smart or dumb? Can't tell you. Maybe smart because you made a lot of money, maybe dumb because you took big risks that really could have subjected you to intolerable outcomes. And yet it worked. And you tend to pat yourself on the back, as do many others. So if you think that every success is because of your brilliance and every failure is because of your ineptitude.

7:55As Annie says, if you reach that conclusion in the short run, based on one trial or five trials, it may be the wrong conclusion. But once you do it over many trials and it holds, then you start thinking, well, maybe it's reality and not noise. This reminds me of a couple of things. One is I remember many years ago when I still played poker, I've been retired since 2012, there was a new player who had come on the scene who had been doing amazing for six months. And someone came up to me and said, don't you think that so-and-so is the best player in the world right now? And I said, I don't know.

8:31And I think that they thought that I, as a veteran, was acting out of jealousy or something, envy. I don't know. And so I could see that they thought that I was saying something mean. And I followed it by clarifying, I don't no, six months isn't enough for me to know. Because to Howard's point of what he just said, in poker, aggression can be enough for a while until people figure out the aggression bit. And then they'll start to do things to counteract it. And what I said to them was, you need to come back to me in a couple of years. I need that much time to see how they're doing. How do they handle losses?

9:09Have people figured them out? If they figure them out, are they adjusting back and figuring them out back. And then let's talk in a few years. But I'm not saying anything mean about them. I'm just saying after six months, when you come and ask me if they're the best player in the world and they've only been playing for six months, I have no idea. And I think that that's very hard for people to get in practice. I think that theoretically they understand that. But when you see someone winning everything over the course of six months, it's just very hard not to drink their Kool-Aid. And I think that that's what we have to become impervious to.

9:45So that's the first thing that what he said reminded me of. The second thing I just want to go back to that he said is this idea of it's filled with people who win once in a row. By the way, I'm just totally going to steal that. I'm just completely admitting to I'm going to steal that. One of the things about winning once in a row, and it's something that I think really deeply about, is that that ability to ride off of that reputation of winning once in a row depends on the market that you're playing in. So let me try to explain. This is a huge problem in venture because in venture, the outcome of the eventual exit is so far from the initial decision that if you get one big winner, you can ride off of that reputation for so long absent any decision quality because you're not getting, say, the speed a feedback loop of an options trader, for example, where an options trader winning once in a row is not going to be particularly helpful for them.

10:41And then I think that we can think about something in between like a long short trader. To Howard's point, if you're trading in a bull market and you have a bull market that lasts a pretty long time, that becomes a once in a row win. And you don't know until times get bad in the same way that I would say for that poker player, I want to see what happens when they go on a losing streak. How do they handle that when all of a sudden things turn against them? Because we can have bull markets that last a long time and it creates that once in a row because you're not getting that feedback of what are you doing when all of a sudden the base rate, what's happening with beta isn't in your favor.

11:19It isn't allowing you to take the kind of risk that you've been taking on that might not have been so smart. So I just wanted to key on those two things because I thought there was a lot to unpack in what Howard said. Something you've both touched on now a few times here is this idea of luck and why people will have a good outcome and not acknowledge that it's luck and then maybe make decisions based on the fact that they don't think it's luck. So I'd like you both to speak about this importance of acknowledging the very large role that luck often plays. My 2018 book, Mastering the Market Cycle, ended up with the discussion of what I call the cycle in success.

11:52And I said in there that you learn no lessons from success or maybe bad ones. Success teaches people to say, I can do this. It's easy. I can do it with more money. I can do it in different areas. I can do it anytime. And I can do it without a team. It was me. And those are all terrible lessons. So for us to draw important conclusions, the thing we're testing has to go through a trial. Buffett says it's only when the tide goes out that we find out who was swimming naked. If you look at a period in which the tide never went out, that requirement is not satisfied. And by the way, I was really enjoying what Annie was saying before.

12:32The person who makes the big successful bet is the person that gets their picture in the paper or uses the aggressive strategy in a game or in a sport and wins. Or the investor with the highest return gets their picture in the paper. They don't run any articles saying that this person made a great decision that didn't work, or this person, their investment results were not good, but it was reasonable for them to expect the actions they took to work out. It just didn't happen this time. Those articles never get written, because for those to be written, you have to look deep, and you have to say, what was the truth in what I just went through, not what appeared on the surface?

13:08Celeb, in Fooled by Randomness, which is an important book for the Annies and me, talked about alternative histories, which I define as the other things that reasonably probably could have happened but didn't. And before you assess the quality of any decision, you have to say, well, not the outcome. At the time the decision was made, what were the possibilities? What were the probabilities? And under those circumstances, did the decision maker make a good decision? When I talk about the role of luck versus skill, I say you invest$100 and at the end of the year, you get back 200. Was it risky? Can't tell.

13:46It might have been a risky investment where you got lucky or a sure thing that was brilliantly fashioned. And from one outcome, you don't know. That's why you need the long trial that Annie described. Yeah. So I love the discussion of alternative history. Just for full disclosure, it happens to be what I wrote my dissertation on. So what Howard is referring to is something called counterfactuals, which is things that did not occur but could have occurred. And I think that that type of counterfactual thinking is so incredibly important because in order to be a good counterfactual thinker, you have to understand two things.

14:23One is what are the base rates? And the base rates are just what generally happens in the situation that I'm considering. I really highly recommend people read Michael Mobison's work. He goes very deep into the importance of base rates. I think that when we're trying to tease out luck from skill, base rates become incredibly important. But the other thing that counterfactual thinking really makes you go very deep into is what are the causal drivers of the base rates? And I think that this is something that we lose as we're looking back to Howard's point about, okay, well, I had$100, I turned it into$200.

14:58What were the causal drivers of that? Because that's what really matters. If you had some kind of sure thing arbitrage, that's very different than making a huge bet with all of your capital on a very risky investment that happened to work out. So when we take all of these together, let me just give you an example of why this is so important for teasing outlook from skill. We all know that in venture a year ago, from about six months or a year into the pandemic till last year, there was just a huge bubble that occurred. So now we can think about how we bring this together to start teasing out luck from skill.

15:36So there were companies that were getting 40 to 60 X multiples. This was very unusual within venture. And so we might ask there, is that a reasonable price? Is that a price that we would expect to persist? Or should we expect that that's going to collapse? Now, there's lots of people who might have invested two years before that 60x multiple and thought, wow, I'm a genius because look at how it went up. But now we want to ask what's going on there. And essentially, the first thing we can do is go and look at the base rates. So that helps us to level set. And then we ask, what's the causal driver?

16:11So these companies are being priced really far away from the base rates. Has something changed? Is there a paradigm shift that has occurred that might cause this to have just a higher multiple? A way that people could think about that is, is this a Walmart and Amazon situation, the difference between those two companies in 2001? One has a retail imprint, one has an online imprint. We might say, okay, there's a paradigm shift. So at that time, you can then now use that to say, is this a reasonable multiple? Would I be willing to invest at this multiple? So it helps us to start to tease out what's weird in the world, what's not weird in the world.

16:51And it's actually the only way to be a good counterfactual thinker as we think about changing antecedents, changing conditions of what would have happened if, how could the world have pulled it differently? That digs us down into the causal drivers, which is where all the skill lies. This discussion of alternate histories of the role of luck, it also makes me think of something that you've both touched on here and both written about quite extensively, which is this idea of accepting uncertainty and accepting that there are things you cannot know. So I'd like you to speak more about why it is so important for people, whether they're a gambler, whether they're an investor, to just accept that there are some things they don't know.

17:31I have a one sentence answer to that because it's a fact. Really, that's it. If you're going to make good decisions, you have to have an accurate assessment of what you know and what you don't know. Because your beliefs are the foundations of the decisions that you make. And it's just a fact that you don't know everything. So if you can't embrace that, your decision quality is going to go down. Yeah, and mastering the market cycle, I wrote that when you try to think about what's going to happen in the future, What most people do is they spend all this time coming up with a forecast. It's not enough.

18:07You need two things. You need a forecast and you need a judgment of the probability that you're right. Because there are things which are highly dependable, like tomorrow's weather will be similar to today's. And then there are things that are absolutely unpredictable, like what's going to happen in Ukraine. When will that war end and how? And you have to be rigorous in saying, this is a decision that I think it's reasonable to act on, and this is a decision which probably has no information value. Mark Twain said it best. It ain't what you don't know that gets you into trouble. It's what you know for certain that just ain't true.

18:47Let's see. If I want to drive up to Albany today, I will get a map, ask directions, turn on the GPS, drive slowly, so I don't rocket past my exit. But if I think I know the way, I won't ask directions. I won't get a map. I won't turn on the GPS and I'll go as fast as I can. And it'll probably take me about three or four times longer to get there. No sentence starts with, I could be wrong, but, or I don't know, but ever got anybody into trouble. The ones that do are the ones that start with. I'm absolutely certain that such and such, because there's no room in our world for certainty. And when you think you have it, you might tend to bet too much.

19:26And if it turns out you were wrong and you get a negative outcome instead, you could be out of the game. Yeah. So what Howard pointed out so eloquently is this difference between your certainty for very short-term outcomes versus your certainty for very long-term outcomes. Because as we think about luck, the influences of luck that are going to cause that future to branch in different ways obviously gets greater. You create many more branches as you go toward the future. But what we have to remember, I think that's so important is that even if I'm predicting something that's going to happen an hour from now, we still have to embrace the uncertainty.

20:00So if we take the weather, what the weather is going to be like in six months, you know, on that particular day, it's kind of hard for me to predict. I have some good base rates. I'll have some pretty good ranges on the temperature and things like that, but I'm not going to know that much. But what the weather is going to be like in an hour, I'm going to have a lot of certainty around. But we've all gotten caught in an unexpected rainstorm. So even in the short run, there's quite a bit of uncertainty. And I think one of the problems that we have is not just that we're overconfident in the forecast that we make, but I think on a meta level, we're also really overconfident that we're going to notice in the future the signals that tell us that our forecast was wrong.

20:41And we're actually quite bad at that. So when we think about what is one of the reasons that when we have good outcomes, we're downplaying the role of luck? Well, it's because we want to feel good about ourselves. We want to think that we get to have ownership over that good outcome, that that's because we're great decision makers, that we're good at what we do. Okay, so now we make some sort of forecast that's a long-term forecast. We're probably overconfident in that forecast in the first place. But then we have the second layer of overconfidence that we say, well, obviously, if conditions change in the future, I will change my mind.

21:16But we won't because we get into the same problem of, well, that means that I have to admit that I made a mistake. Because we don't really understand the word mistake. So if we can get off the confidence of the initial forecast and says, this is what I believe for now. But obviously, if these things changed in the future, I would change my forecast. We would be a lot better off. One of the things that I think about is I knew of a lot of people who invested in Bitcoin because they believed that it would be discorrelated with market chaos, that it would help be a hedge against inflation. And at the time, I didn't invest in Bitcoin.

21:50I didn't feel like I knew enough about it, but these were smart people. It didn't seem like a totally unreasonable thing to say. I wasn't really going to argue with them. I didn't feel like educating myself enough to have an argument with them. But then when it turned out, it was actually quite correlated with market chaos and it was not a good hedge against inflation. I didn't see those people selling. What I saw is them having thesis creep. I think that this is a good example. I think we all experience this with a lot of people is that we're overconfident in our long-term forecast, but then we're also overconfident that we'll change our mind when the world goes against what caused us to forecast that in the first place.

22:27You have to say like, this is my best educated guess for now. And I'm trying to educate the guess as much as I can and make good decisions based on those educated guesses, but they're educated guesses and that's okay. This reminds me of something I was recently reading where some of the people who had predicted that inflation would be transitory were now coming back and saying, well, what I meant by transitory was multiple years. Right. Right. And that just goes to bear out the old adage that being too far ahead of your time is indistinguishable from being wrong. Right. One of the interesting things about forecasts, Anna, that that brings up is that people like language that has a broad target area.

23:08Let's put it that way. So we love to say that things are likely or probable or will happen more often than not, because it just allows us to completely get out of that problem. But by transitory, I meant something totally different than what you said. I think it's interesting because one of the ways to embrace uncertainty that might be a little bit counterintuitive is to actually make real point forecasts. Don't say likely. Say this is 63%. Do that across a lot of different forecasts. And you can find out when I say 63%, how close to 63 % of the time does this thing occur? So it's like when we say something like likely, it's a way for us to say, well, I know it's uncertain, but I also don't want to be wrong.

23:54And I don't really want to communicate what I'm saying precisely because I'm afraid that I'll be wrong. It's understanding that there's some uncertainty, but trying to get out of that bind. Whereas it's much better to say, I believe that there's a 63 % chance that inflation will resolve by November of 2024. And this is the way that great forecasters speak. And then they update their forecasts as they go along. And it sounds weird because it sounds very certain, but it's not. Because you're saying there's some percentage of the time that I believe this is going to occur. Obviously, we can create upper bound and lower bounds to that.

24:32There's all sorts of things we can do to incorporate into those types of forecasts what you do and don't know. I can say, Anna, if you have a dog, I could say, I'm guessing your dog is 45 pounds, but I'm going to put a lower bound of 10 pounds and an upper bound of 90 pounds on that. So this is a way for me now to express with some precision what my certainty is. I'm telling you what I know and what I don't know. I'm telling you that I know a lot about dogs. I know a lot about what they're likely to weigh, but I don't know exactly what your dog weighs. And I think that this is a very helpful way to speak, much better than it's transitory, or it's likely, or it's going to happen in the near future.

25:08I don't even know what the near future means. There's another very interesting topic under this heading, but if you think about it. The only forecasts that can be said to be wrong are 0 % probability and 100 % probability. The weatherman says, I think there's an 80 % chance of rain tomorrow. It rains. Does that mean he was right? No, that only means that it rained. The probability yesterday of rain today could have been almost anything. And yet it could have been 1%. Exactly. And yet it rained. But if he says zero, then I know where to look. But if I say something's going to happen and it happens, maybe I was lucky.