Charley Ellis on Why Most Investors Fail (And What to Do Instead) (EP.191)

0:28We all need to make smart decisions with our money. investing Charlie Ellis. If you've ever felt like investing is harder than it used to be, or that the odds are stacked against you when trying to beat the market, then this conversation is a must listen. Charlie is probably best known for his wildly important bestselling book, Winning the Loser's Game. But we also talk about his newest book that just came out called Rethinking Investing. He spent decades advising institutions, studying market behavior, and uncovering why even the smartest investors struggle to outperform over time. If you're not familiar with Charlie's work, he is just an absolute master at distilling complex financial truths into simple, actionable advice.

1:13And so whether you're a do-it-yourself investor or you work with an advisor, this episode is really, really great. Now, before we jump in, I do want to invite you to subscribe to my free newsletter. Every two weeks, I share insights on investing, personal finance, and decision-making delivered straight to your inbox. It's the best way to stay up-to-date on the strategies and frameworks I use to build long-term wealth. And so if you just open up the episode description in your podcast app, you will see a link right there to subscribe. Or as always, you can go visit thelongterminvestor.com. All right, let's get into it.

1:49Here is my conversation with Charlie Ellis.

1:55Charlie Ellis, welcome to The Long-Term Investor. Thanks, Peter. This is just a thrill for me, a career thrill. So much of your work has shaped my thinking on investing. Anytime you write anything, I feel like I absolutely have to read it. When I see that you're speaking, I have to see it. When I first read Winning the Losers Game, that really kind of opened my eyes in understanding what it is I'm up against as an investor. And so I was hoping today, before we get to your new book, just to have you talk a little bit about that. Do you mind defining what the loser's game is? Sure. It comes from a wonderfully intelligent man named Simon Ramon, who was TRW's leader when it was the dominant factor in our space program.

2:45And he also played violin and a quartet of professional musicians plus him at the Los Angeles Symphony Orchestra. And he was a gifted amateur tennis player. And he figured out that in tennis, most people don't actually win the competition. Somebody else loses the competition. And that if you watch most matches in tennis that are amateurs, and he pointed out there are two different games, one by professionals and won by amateurs. And the same rules, same equipment, same court, same way of counting, but the games are completely different. Experts win points just barely beyond the reach of the other player.

3:33Amateurs lose points. And if you want to win the match, what you have to do is just keep the ball in play and let the other person make the mistakes that inevitably will happened. And pretty soon you will find that you were, quote unquote, the winner. But you did not control the outcome of the game. The outcome was controlled by the loser. So interesting to hear the background. Now, tell me, Charlie, why is money management a loser's game? Well, it used to be a winner's game years ago when most of the investors, 90 % of trading on the New York Exchange was done by individuals who had absolutely no access to research, had not studied or trained how to be analysts, did not understand investment theories, just they would occasionally buy or sell for whatever reasons they had.

4:27That environment made it pretty easy, if you had anything like good information, to do much better than the market. And when I first got involved in the investment field in the 1960s, that's a long time ago. But in the 1960s, if you wanted to make a meeting with a senior corporate executive, you call them on the phone and say, I've done my homework. I've studied your reports for the last several years. And I've got several questions I'd like to ask you. And it would take at least half an hour, maybe an hour. May I come and meet with you? Oh, I'm delighted to have you come. because the corporate executive knew that they were trying to be sure they were getting a fair price for their stock.

5:11And the more people they had investing in their stock, the more they're likely to get a fair price and also a little bit higher price than otherwise. So that could be a corporate assets if you were acquiring other companies or defending against being taken over by other companies. So corporate executives thought they were doing the right thing to give information to investors, particularly professional investors. So you could get good information and then take that and go into competition with people who had no information. And then in the proverbial, it was like stealing candy from children. Well, that changed a lot, massively changed, so much so that you wouldn't even recognize.

5:54And the world of investing today is candidly completely different. And I can give you a whole bunch of different illustrations if it would be helpful. Yeah, let's do that. Well, first of all, it used to be that 90 % of trading was done by individuals. Now more than 90 % is done by professionals. And those professionals are formidable. They typically get very well paid to be very, very good. And if they're not very, very good, they get displaced. Their clients will say, Now I'm going to shift to somebody else. The quality of competitors is really up there. The second thing is technology has changed massively.

6:36When I first got involved in the investment field, slide rules were the technology. Pretty exciting to have a slide rule. Yeah, that's fine. But how about computing power? Now everybody has more computing power than anybody had 60 years ago. You carry it around your pocket. small cell phone. And you can do more calculations with that device than other people were able to do years ago. So big change. Another big change is the law. It used to be that there was no law against corporate executives giving information to individuals. Now, SEC has made it clear that any publicly listed company or publicly owned company that gives any useful information to anyone must make a concentrated effort to get that same information to everyone.

7:29So the obvious answer is you have a quarterly call. Everybody in the world can call in, listen to the same information, and then try to figure out what they could do that might give them a comparative advantage. But the leveling of the competition by giving everybody the same information at exactly the same time, as a matter of law, is really, really powerful. Then you think, okay, what else has changed? You're kidding. Mike Bloomberg's wonderful devices, the Bloomberg terminals, they are unbelievable in terms of what you can do to gather information, process it virtually any way you want to. And there are hundreds of thousands of those terminals all over the world.

8:11So anybody with anything like ambition to be really competitive has got one at the office, might have one at home. And there are many people that have one in their car so that as they're being driven to work, they can operate the terminal. Well, okay, the internet would be another technology. Instantaneous distribution of information all over the world. Another change that's really important to keep in mind is it used to be that if you were a Canadian investor, you invested in Canada. If you were a Mexican investor, you invested in Mexico. If you were a British investor, you invested in Great Britain.

8:47Now, everybody invests all over the world. And the access to information from really superb research analysts has gone up and up and up and up. And there's now just extraordinary resource available all the time to anybody who is in the field. So we have really changed the nature of the market. And it's become a better and better market in the sense that What markets are supposed to do is find a correct price. And the markets that we have today are so endowed with information and so endowed with talented operators and so competitive that the price discovery process, which is what markets are theoretically all about, are in play in a very, very effective way.

9:38So they're not that the prices are exactly right, but they are so close to right that chances that any one of us could improve on that best estimate being made by experts all over the world with money at stake is very unlikely. And to be able to do that on a regular basis by enough to cover the cost of trying, just not there. I want to reemphasize a couple of things I heard you say so important for our listeners and our viewers to understand is that we're not saying that prices are perfect. We're saying that for somebody to identify a mispricing and profit from it, those are two different skills continuously year after year is very, very difficult.

10:21And the competition is so high that the relative difference in these investors is so small. So you talk about it in your new book, Rethinking Investing. You talk about it in your past books. These are all brilliant people. We're not saying active managers aren't brilliant, but they're all brilliant. I think if you're a sports fan, it's easy to see in something like baseball or everybody's throwing 100 miles an hour and everybody's a good hitter. And when the competition and the skill level gets so high and the dispersion gets so tight in terms of skill level, well, then luck starts playing a bigger role.

10:54And so you're a big advocate, and you have been for a long time, of not trying to play that game, not trying to win, but just not losing. And I think index funds is typically where you've pointed. Is that a fair assessment? Well, yeah. Well, in all fairness, if you look at index funds versus active managers, if you choose to be an indexer or an ETF investor, you're virtually sure of being the top quartile in terms of investment results of reasonable time period, probably be in the top half of the top quartile. And, you know, that's a dream come true. Who wouldn't settle for that? Give me that.

11:34How about give me that on an assured basis with lots and lots of data to support that? That's the way it's going to continue to be. Well, I am going to dig a little bit more into this idea. But before I do, you spent many years at the Yale Endowment where David Swinson changed the way the institutions thought about investing. And so it's hard to think of an investor. And I tell people that if you're using index funds, I'm not going to fight you. Can you use other low cost systematic approaches? Yes. But indexing is never really the wrong answer for anybody. You could maybe make the argument that somebody like David Swenson, who showed ways in an unusual manner to deviate from the broad market exposure and add return.

12:21Maybe he's someone who shouldn't have, with the benefit of hindsight, needed indexes. But even he said that that's something you should do. Let me ask you this, though. All the time that you spent at Yale, what is it that you think that people misunderstand about the skill that's applied and things that institutions like that are doing when they choose not to index? Wonderful question. And I can't tell you how much I feel strongly about the answer. First of all, David Swenson was a really, really nice guy. And he had the ability to develop relationships with other people who were just special.

12:55I was devoted to David. I would do anything to help him at any time. And I think there must have been a couple thousand people in the investment world that felt exactly that same way. The second thing is he was very disciplined. disciplined in every way. And that discipline focuses the mind on doing exactly what you have figured out is the right answer. The third thing is David was extraordinarily creative. Most people have never even begun to think about this, but he is the first person to execute a transaction in a derivative. Now, derivatives are larger than the cash market. What was the derivative that he executed?

13:37It was an interest rate swap between the World Bank and IBM. One had fixed rate debt. One had floating rate debt. And David believed that the interests of the two organizations were such that he could do a swap from one to the other. Oh, no, David, you can't do that. Nobody's ever done that. I know nobody's ever done it, but I really think it's the right thing for these two organizations. And I think I could set it up. You're never going to make it. Can I try? Sure. Waste of time, but go ahead. $100 million transaction. And there's a rumor that says he got 1 % on each side. But that was a brilliant, creative move.

14:20Nobody had ever done a derivative transaction of any kind in securities. And of course, today, that's the largest part of the total market. So you have to understand how lovely a nice guy David was, brilliant enough to do that transaction, brilliant enough to do a superb job as a PhD student at Yale, working with Jim Tobin. And Jim Tobin was a Nobel Prize winner. The two got along really well because they were both brilliant and creative and disciplined and hardworking. It's a lovely example you give because if memory serves me, just as an example, when people think about what institutions do and the best institutions and what David was a pioneer in is doing things that others weren't.

15:08And even in venture capital, they were very early in their venture capital exposures relative to their competitors, other big university endowments. Being first can be a terrific advantage if you're first and right. And one of the great things that David brought to the process was the courage to be first and the knowledge and understanding that he had done such rigorous homework that he knew what he was doing. This was not speculating. This was not gambling. This was not taking a chance. It was very focused, disciplined execution of smart moves that he understood all the dimensions of. And let me stay on this track just a little longer on Yale and institutional investing as a whole, because you've spent time at a lot of institutions.

16:01Let me tell you, Charlie, I am in front of charitable endowments, university endowments, healthcare endowments, and they range anywhere from, say, like$5 million to$500 million. And a lot of times when I hear their investment committee and they're a group of people who've been donors and, you know, maybe they have experience in some portion of the finance world, a very small specific place in the finance world. But they sort of feel like if they aren't using alternatives like Yale does all in kind of air quotes, then they're not doing it right. I just don't feel like they understand how much talent and effort and thoughtfulness goes into a lot of these endowment operations.

16:45Do you think it's reasonable for any given endowment, let's say even just below$200 million, to be veering into some of these spaces that a place like Yale does? No, I don't. In fact, I would urge anyone who is under$2 billion to say, I just don't have the resources to be able to design a competitive strategy that would allow me to be a serious competitor. And look at it two different ways. One, as a buying organization, will you be able to afford the quality of staff that would be required to have expertise in each of those different categories of, quote unquote, alternative investments? The odds are very slim that you would have full-time capacity to have first-rate talent competing on your behalf for the best relationships with the best managers.

17:38It just isn't going to happen. You think about that. Okay, that's perfectly fair. It's not that a five-year-old boy can't play professional football, but he can't play professional football at five years of age. He's going to have to wait until he's 25 or 30 in order to be big enough and himself enough. If you look at the dispersion of returns in each category of alternative investment, real estate, venture capital, private equity, so on, look at the dispersion of returns. The very top performance is wonderful. The near top performance is good. The not so near top performance, and you're still talking about much better than the crowd, is not the market average.

18:25And below that, it gets painful. The average investor loses by going into alternatives. There are some people who go into alternatives who have the skills to make it into something really remarkable. But the alternative managers who have those skills are very careful in their selection of which investors they're going to allow access to their skill set. And the answer is they're going to concentrate on large funds that are very expert and they're quick to make decisions and have lots more money coming along for the next fund and the next fund and the next fund. And so you find, sure enough, ideal customers and ideal providers get together and others should stay out of trying to compete because it's very hard to do.

19:20And yes, and I feel like there's a lot of people selling access and selling expertise. But if you have 50 clients or even 15 clients that you're selling that expertise to, the best funds may not really want your money. That's not really the type of long-term capital they're looking for. And honestly, it's very tempting to keep going down this rabbit hole. But I want to hit some of the things that you're talking about in rethinking investing, as well as some of your past works. We were talking earlier about index investing. clearly a form of low cost rules-based investing. But I also referenced how there are other forms of low cost rules-based investing that people refer to as factor investing.

20:01I'll just be pretty plain. And so instead of just taking the size of the company as the only rule, it might assign weightings a little bit based on their price relative to some fundamental. What are your thoughts on this form of systematic rules-based investing like a factor investing? Well, first of all, I really like the idea of being disciplined in your thinking. And secondly, I do respect those people who have studied segments of the market and found ways in which they can, in certain periods of time, get better results by using a particular discipline or factor. The problem is, for most of us, the problem is really serious.

20:39At the time that the results look most compelling, and the time when we're most likely to say, I'm going to do that, is after the good results have been achieved. And the future results are not going to be anything like a replication of the past. So I'm very skeptical of factor investing, except for those who are very skilled at understanding what the factors are and knowing what they're doing and when to go in and when to come out. And even then, it's going to be a relatively modest amount of incremental benefit. And for those of us who are not expert in the particular factor, I think that the net result over time will be we will pay a penalty for doing the factor investing.

21:23We'd be better off to do something as plain vanilla as index investing. And is the key, it's just a matter of indexing, it's just an implementation strategy, and you just have to know what can you live with for 10 or 20 years. You know, and the time to be investing in a fund that is factor-driven is when the results in the past have not been at all good. Then you might very well get a surge of, the tide comes in, the tide goes out, the tide comes in, the tide goes out. But for most of us, we would be making a mistake if we went to factor investing, because most of us being human beings would tend to get excited when we could see there was a lot of evidence in support of the factor.

22:07But that's about when the time has come all the way in and it's about to turn and start going out. Well, Charlie, you make a great point because I feel like I spend half of my time when I'm speaking with our clients, just coaching them through anything that doesn't perfectly track the market. We have a lot of factor investors at PlanCorp. We have a lot of index investors at PlanCorp, but you're right. It's these behavioral things that make such a big difference. And actually, in your new book, you dedicate an entire chapter to behavioral economics. And in my opinion, you just can't take the human out of human nature.

22:43So my question to you is, how would you suggest investors protect themselves against these well-documented behavioral financial tendencies that tend to hurt our returns? The first thing is to pay attention to what the behavioral economists are teaching us and study the data that they're producing. You're right, a whole chapter on behavioral economics may sound like a lot, but actually it's the biggest single problem for most investors, and the penalty cost is huge. So one chapter seems like actually too little rather than too much. If you really want to get into it, Thinking Fast, Thinking Slow is probably one of the great books ever written about behavioral economics.

23:26And anybody that wants to be serious about investing should read that book at least once every year to remind them that as human beings, we are our biggest problem, which is the way it's going to stay for a long, long time. So understanding the reality of behavioral economics and that we're all in that place. We're all doing those sorts of things that we know we shouldn't do, at least when we're reading it in the book. Fine, but it is the biggest single problem. There is a solution, and it's really quite wonderful. We talk about indexing on ETFs as they are terrific because you have lower fees.

24:05Correct. They also are a fabulous protection against the perils and misbehaviors linked into behavioral economics because indexing and ETFs are boring. They're not interesting. They're no fun. There's no excitement. So you kind of leave them alone. And it's a little bit like your mother taught you, don't pick at a scab. It won't heal as fast as if you just leave it alone. And if you do leave your scab alone, it does heal faster. And the same thing is true if you index or use ETFs. You are not all that excited by the market. You're not doing about responding to individual stocks, because it's the same old, same old, same old.

24:50And you probably just benign neglect is actually a big positive. And let the market do what markets are doing when they're left alone, which is producing a terrific long-term rate of return. So I think anybody who's serious about investing needs to become proficient in understanding behavioral economics and what the problems are. And the easiest and most powerful data we've got is over a long period of time, the average investor penalizes himself or herself by 200 basis points or two full percent on average, year in and year out. If you think, oh, no, the market rate of return is going to be something like 7%.

25:37No, wait a minute, minus 2%, 2.5%, 3 % for inflation. Hey, wait a minute, this is starting to get pretty squeezed. Taking another 2 % out because of behavioral economics? Wouldn't it be great if you didn't have to take that extra 2 % out? If you could leave it in there, going for compounding over the long term would be terrific. To me, that's the core value of this really short book. It's only 100 pages long, and it is a complete rethinking of investing, why you don't need to have bonds because you can have a spending rule that smooths out the fluctuations, and how you can therefore emphasize equities in a major way, because it really will work for you in the long run.

26:20It's just so much in the way of a real opportunity that we tend to miss, but it's all sitting there. And in a very short book, that's what Rethinking Investing is all about. Well, I love that it's short. Actually, my most gifted book, I didn't tell you this before, is a book you co-authored with Burt Malkiel, The Elements of Investing. And I think sometimes the brevity and just focusing on those really key points are what is going to keep someone who's not really ready to read 400 pages on investing. It's what they need to see and hear and engage with. And I think your book did a great job really of just emphasizing that successful investing is simple, but it's not necessarily easy.

27:06There were a couple of different things you felt like were required of an investor to have success. What are some of those things? Well, the first is to understand who you really are as an investor and what are you trying to accomplish. And that leads you to realizing, hey, wait a minute, long term is not six months. That's what the tax man says. Long term is I start investing in my 20s. I'm still investing in my 80s, maybe my 90s if I'm lucky. That's 50, 60 years. That's long-term for me. And if I am successful enough to leave something to my children, they're going to extend that 50 or 60 years out to 70, 80, 90 years.

27:51That's really important. And if you think about it, that gives you the freedom to do compounding, which Warren Buffett teaches all of us is the greatest force for good in the investment world. And if you compound and double every 10 years, which would be assuming a 7 % return, you double every 10 years. In 10 years, you go from 1 to 2, then from 2 to 4, then from 4 to 8, then from 8 to 16. Now we're starting to get your attention from 16 to 32, from 32 to 64, from 64 to 128. All of a sudden, you realize, hey, wait a minute. Those last doublings are really something. call it the snowball, but they really are something.

28:37The only way to get there is to get started and let time do its magical wonders on an overall basis. It's just unrelenting the power of time. And if you think about it long-term, you do have time. If you don't think about it long-term, you will chop your investing into little bits and pieces and you will never have the power of compounding working for you. Since you mentioned Warren Buffett with compounding, the fun fact that maybe you know, but listeners may not realize is that the vast majority of Warren Buffett's wealth accumulated after he qualified for social security. Now that is compounding and doubling on the back end.