Ben Carlson on Risk, Reward, and Building a Portfolio You Can Stick With (EP.255)

0:02We all need to make smart decisions with our money. The Long-Term Investor podcast shows you how by distilling complex financial matters into easily digestible lessons. And now, here's your host, Chief Investment Officer at PlanCorp and the author of Making Money Simple, Peter Lazaroff. Welcome back to The Long-Term Investor. Today, I'm joined by Ben Carlson to talk about his new book, Risk and Reward, and a deceptively simple question. Why is investing so hard when so much of the best advice sounds so simple? You probably know Ben from his blog, A Wealth of Common Sense, or from the Animal Spirits podcast, where he has built a reputation as one of the clearest and most thoughtful voices in personal finance and investing.

0:50What I've always appreciated about Ben's work is that he just has this unusual ability to combine market history, data, common sense, and a really honest understanding of human behavior. He doesn't pretend that investing is easy. And in fact, one of the big themes of this conversation is that every investment choice comes with trade-offs. Sitting in cash has risks. Owning stocks has risk. Trying to time the market has risk. Even doing nothing can feel risky when headlines are loud. So in this episode, Ben and I talk about why more effort doesn't always lead to better investment results, when complexity can't help or hurt, why inflation feels so painful, and why the perfect portfolio only exists with the benefit of hindsight.

1:32As always, you can find detailed show notes at thelongterminvestor.com. And if you haven't heard of Ben before, I link to him all the time in my newsletter, which you can also sign up for in the show notes or using the link at the top of the episode description. And now, here's my conversation with Ben Carlson.

1:54Ben, thanks so much for joining today. Glad to be here, Peter. During my workout today, I was thinking there may not be somebody with more published words on this topic than you. And on one hand, like if people regularly follow you, there really isn't anything I could ask where I could generate an answer that would catch them off guard. But for those who are not familiar with your work, they're going to love this conversation. I think they're really going to like the new book. Let me just start right at the beginning. you kind of open up saying, Hey, there's no secret to investing. Um, you also are saying right out the gate, I'm a long-term investor, but people push back against me.

2:30So as you started to write the book, as you started to compile the information, like what were some of the things that you were trying to poke holes in most with, uh, with what you were doing here?

2:38Ben Carlson:I think the biggest one is just that it's easy, that it's ever going to be easy for anyone. there's i just this idea that like whatever you do with your money whether you're buried in the backyard you put it on t-bills you put it in the most wildly speculative things you can there's always a risk with whatever you do so it's kind of like there's never an easy stance to take because whatever you do there there's some sort of risk and so i want to just i just wanted to outline all the risks because i do get a lot of pushback in terms of well what about this and what about that people love to do the what about game right which i you know i think it's it's fair because there's exceptions to every rule that there is.

3:18Ben Carlson:And so there's no like investment philosophy or investment strategy that is foolproof, that works all the time. And that's kind of what I wanted to show people is just that like whatever you do with your money, you're gonna have to make some sort of trade-offs and it's never gonna be easier. You can have the simplest strategy in the world. You can put it all in one index fund or one target date fund or one stock or whatever it is, but there's still gonna be risks involved in that and it's never gonna work all the time. So it's like, how do you handle those times when whatever you're doing doesn't work well.

3:45So if you're an advisor, you're trying to showcase value through some of the investing. And so maybe there's like a little bit of a smoke show, a light show involved where it isn't really necessary. Do it yourself investors, I think, are always looking for a deal, you know, whether that's something cheaper or like a trick or a loophole. You have I won't like totally give away everything that's in the books of people actually go and listen. But you give an example that I think I've seen you write about before, like with a study on goalkeepers, like in penalty kicks. And generally speaking, when you're a goalie, you have to just guess like, is it going left?

4:17Or is it going right? Or is it coming right at me? And the data shows that, hey, it's better just to stand there and do nothing. But as a goalie, you feel like a moron. If someone just launches balls left and right, there was a couple of narratives in that chapter I really liked. But the main point being like, making it look easy requires plenty of work. And more often than not, like more effort doesn't equate to better results. You know, what are some ways in which you see that transpiring with people who read your blog or listen to your podcast or clients you work with?

4:46Ben Carlson:Yeah, I think there is always this illusion of control because with other areas of life, if you, you know, if you work harder, you should improve and get better, right? If you're playing a musical instrument and you practice it all the time, you should be better at that musical instrument. If you're working out all the time, you can see some changes happen in your body. If you study more in school, hopefully your grades improve. And investing is not, there's no like correlation there. for most people. And I think the hard part these days is because there's just so many smart people that are involved in the markets.

5:16Ben Carlson:In the past, I don't know, in the 50s and 60s, there's these little stories of Warren Buffett knocking on the door at Geico on the weekend to talk to the CEO, right? And you have to send away for the reports. Now, the information is everywhere. There's literal rocket scientists and code breakers that are trying to beat the market. And so I just think that part of it is just like, I'm going to try harder. I'm going to be the smartest person. Like it's almost impossible to outsmart people out there because there's always going to be people smarter than you. So I think there is this like illusion of control, this bias towards action where it's like if I'm doing something and I put my hands on the steering wheel, it's going to make me feel better.

5:52Ben Carlson:Like a release valve. Like, oh, at least I did something, right? I bought something or I sold something or I made this change or I changed the allocation. I think just sitting on your hands is extremely difficult. And it's kind of interesting because I think that there's never been a better time to be an individual investor. The barriers to entry have been just like bulldozed, right? There's no more trading fees. You can trade on your phone if you want. It's never been easier to jump in and out. Like there's strategies that you can put in that for pennies on the dollar, fractions of a penny, where if you would have tried to put this strategy together 20 years ago, it would have been so cost prohibitive.

6:25Ben Carlson:It would have been impossible, right? These baskets of ETFs and stuff. When I was first coming up, it would have been impossible. So all those barriers to entry being down are great for individual investors. But the lack of barriers makes it way more tempting to do something, right? To dive left or dive right. I think that's the hard part, is there's never been more products or strategies or vehicles or the ability to make a change when you want. And if we get like 24-7 trading with most of the stock market, which it seems like that's where we're going, it's just going to be more temptation. I think that's the hard part for people that that the ability to do nothing is as a decision is way harder than it was in the past.

7:02You know, one of the things that's hard for me, at least I'd be curious on what you think about when you publish something is sometimes like the absolute simplest thing is not the best thing for an individual. Like I myself have a ridiculously simple portfolio because it's really what works for me. but it that even though the philosophy is the same as what we do for our clients you know our clients has it have a few more moving parts you know not everybody should just own the s &p 500 like warren buffett once said annoyingly in my opinion um you know when you think about maybe where people can are there places where people can get too crazy and oversimplification or over fee adverseness is adverseness a word i don't know but you know what i mean

7:47Ben Carlson:Yeah, I think there is some room for complexity. If you do have all your money in one fund, it's one fund that has all these different allocations with it. It's all under one umbrella. It's easy because it's tax efficient, but maybe it takes away some optionality. You don't want to sell out of a little bit of everything when things are going down and you want some money. You want to have a liquidity source that's moving in a different direction. You want to be able to you know, rebalance your portfolio in specific ways. And I think if you just have one fund that holds everything, it's much harder to do that.

8:21Ben Carlson:So I think that having more steps of complexity, it's not always bad. If you think about and this is kind of stuff that's just really big in the wealth management space in the last couple of years. It's funny how we've moved from like, you know, portfolio alpha was the big thing. Now it's like tax alpha. And a lot of these new tax strategies, if you dig into them are very complex. And we've put in front of our clients, And a lot of times we preface it with, listen, just so you know, this strategy still fits with our philosophy, but it's way more complex than anything you're probably used to dealing with.

8:54Ben Carlson:And you have to think about the complexity and the stuff that can be done these days that couldn't be done in the past. Again, on that side of things, it's kind of mind-boggling. But you have to make a choice. Is that complexity worth it? For some people, it really is because they hate taxes more than they like making money or something. like, like, like there's, there's, again, it kind of comes down to trade-offs. So I do think that there are people that have the ability to introduce more complexity. I think you just have to have some sort of like goalposts along the way to make sure, you know, like transistors or whatever to make sure you're not going too far with it.

9:29Yeah. I mean, there's, there's gotta be good complexity versus bad complexity, future blog posts. You write every day. So you run with that one. I don't have time to write with every idea I have, but you know, when I think of good complexity versus bad complexity. When you're doing it on your own, I think complexity is a real problem. When you have an objective third party, like, yes, you and I both work for investment advisors. So like there's, there's bias in what I'm saying, but like anytime I add complexity to my life outside of my finances, if I can have someone who's objective and understands the trade-offs that can be helpful in general, I just feel though, that like most investors who watch this stuff closely or who are following like your work or following my work, there are pieces of the markets they misunderstand.

10:15But I also think they sometimes misunderstand themselves. Like, do you feel like there's a bigger one way or the other misunderstanding markets versus misunderstanding themselves?

10:24Ben Carlson:Yeah. And I think the hard part is that I think some people in the wealth management field take it for granted. Like you and I pay attention to this stuff all the time. We're in this every day. We're reading about it. We're following it. I think there is always the the risk of someone who's just dabbling in it, you know, getting in over their head. It's like a little knowledge can, can be a little hurtful, I guess. I think that that is where some people, you know, get tripped up. And so I think that can be a problem, but you're right. It's like, I'm just going to dip my toe in the water into this and see what happens.

10:56Ben Carlson:If, you know, if you don't understand how these things work, I think that's the hardest part about introducing complexity is just that when it's time to rebalance into the pain, because it's not working, I think a simpler approach is easier to rebalance into. And if you, because I, you know, my previous life in the nonprofit space was all these hedge funds and private equity and venture capital. And it was always really difficult in like these board meetings for these investment committees to think through, you know, they'd have their, this one's green, this one's red, this one's yellow. Should we get rid of this and punt it?

11:29Ben Carlson:And usually the reason to punt on a strategy was the performance stinks. But if you really trust a strategy, it should be the opposite. You should be leaning into the pain and rebalancing more into that. But with a complex strategy, it was always like, we don't know if this is coming back. And I think it's easier to rebalance into the pain with something that's simpler, right? It's not like this discretionary asset manager who's going through a hard time. I like to say like no index fund has ever shut down the fund because it's going through a divorce and needs to spend more time with its family.

11:59Ben Carlson:But that happens like hedge fund people all the time, right? So I think that's where it can be tricky. And to your point, like outsourcing, yeah, sometimes you need someone to help explain to you the potential pitfalls of these things who really understands what's under the hood and not just like the shiny sales pitch. Yeah. And I think that as I listened to you speak, I think about, well, maybe it's like simplicity is just that you, the end investor, as well as the advisor themselves can actually explain it. So like you wrote something in one of your books forever ago. You have a number of books about like, hey, like investment philosophy is sort of like what it all rests on when it looks all broken.

12:36And if you can't explain what you own and why you own it and what it might do throughout different scenarios, then yeah, you're more likely to chase performance and bail. I mean, I think the other thing is that for people who are real students of the market, they are somehow able to separate long-term investing versus something else that is not such. Maybe it's not quite trading. I don't know what that like place between trading and long-term investing is. Maybe there isn't a middle ground, but basically like the long-term is an eternity to live through in the moment. And the big theme that you talk about throughout the book is loss aversion.

13:12And just generally speaking to say like, you have amazing data tables throughout. Like, look, if you're watching us on YouTube or Cheddar or listening to us on the podcast, like Ben's book has so much data in it. Like that alone, if you don't read a word of his writing and you just look at all the data tables, you're gonna get a ton out of this stuff. And when you look at like past 20 % drops, you can say like, sure, I can handle that. But in the moment, it just seems so much scarier.

13:36Ben Carlson:Right, yeah, sure. I can look back and know that bear market came back, but how long is this gonna go? Is it gonna go from 20 % to 30 or 40 or 50? And that's the hard thing. And the loss aversion point, I think a lot of advisors, like the one of the things they hand you on day one like the advisor 101 is just tell your clients just ignore the noise right just ignore the noise don't pay attention to it it's easy i think that that is just not effective advice anymore because it's impossible to ignore what's going on unless you live in a cave and even then you might have like cell phone reception and still be able to pay attention in the cave you live in so i think you just need a way better filter in place whether that's information or opinions or stuff to pay attention to.

14:18Ben Carlson:Like you said, it could be outsourcing to an advisor. We have plenty of clients who say, like, listen, I've done this my whole life. I'm a DIY person. I'm ready to just, I don't want the brain damage anymore. I want someone else to help me. And I think it's also can be for stuff you will or won't invest in. I think advisors get into trouble with this stuff too, right? So Josh, who I work with, always talks about a good advisor is like a bouncer at a club who holds the line, right? Keeping most, and when it comes to your portfolio, it's like keeping stuff out. I think we've learned, you know, in this cycle, especially with something like the private credit interval funds that's going on right now, that a lot of advisors maybe didn't understand like their clients' tolerance for risk enough with this stuff.

14:58Ben Carlson:And I think they even got into, even the private managers have been saying, like maybe this stuff wasn't explained good enough because I think they're even shocked and how many people are trying to get out of this stuff. So I think these, they don't have to be like ironclad rules, but I think having some sort of like limitations in place is helpful too. Listen, this whole subset of products or services or whatever, like that can be helpful for certain people, but for 95 % of our clients, we just won't touch it, right? I think having those kinds of filters in place, I think that's how you deal with this stuff is just having a lot of filters and limitations and make it so there's higher hurdles to get into some of this stuff.

15:36Ben Carlson:I think that can be helpful. Yeah, the filtering information is key. It's, I don't know how often we can tell people and have them really listen. I mean, there's just absolute garbage all around us. And how do you find what is a good resource versus a bad one? And for those listening or watching to us, who says we're a good resource? I mean, naturally, we think we are. But I just, you know, I've come to a point where I realize and I try to educate clients, try to educate listeners and followers, like losses are normal. market's going to go down, different things are going to win, don't chase them.

16:09And I just I think people are no longer worried about the actual data loss itself or the volatility. It's the story attached to it gets them so nervous. And generally speaking, I will talk to someone who says like, hey, I'm not a market timer, but like dot dot dot and then they want to do something or like, I know we can't tie. You actually have a series of them. I wrote them down from the book, like when people start getting nervous, start getting nervous, they'll say like, let's just sell everything and wait for the dust to settle, or I'll go to cash and then buy after the market dips, or, you know, like, what if I put money to work in stocks, and then the market crashes even more, you know, these are things that people go through.

16:44It's just, it is, we've been doing this long enough, you just can't tell when there's going to be a top or a bottom.

16:51Ben Carlson:Yeah, because you have to be right twice. And I know, I know people who sold in like the fall of 2008, before things got even worse, right. And at the time, they looked and felt like geniuses. And then so many of them, We had so many clients come to us in like 2013, 2014, 2015, being like, I sold. I got out relatively early. The market fell 10%. Things seemed like they were going to wobble. I got out, and I never got back in. And the market has left me behind because I was waiting for a 30 % crash that didn't happen or whatever. My own father sold the Friday, all of his stocks, the Friday before Lehman Brothers failed.

17:28Ben Carlson:He sold them all. He got kind of a tip from someone at work. he did it and I said all right fine you know you missed the six like maybe the one of the worst six months period we'll ever have in the as far as the economic system goes but please start slowly but surely legging back in and it took him it took him a couple years to get back in and even then I'm like you know sure you slept at night but then on the way back up then you were anxious the whole time and that's the hard part is is like if you're gonna ever try anything like that you need you need like specific set in stone rules in place that you're going to follow, right?

18:05Ben Carlson:Some sort of trend following or whatever. And even that is really hard to do because a lot of time you're going against the grain. And so I think that if you're just trying to guess here, like how far this will go and I'll get back in when things feel better and the headlines change. And I give a lot of examples in the book that like it's never, the market doesn't wait for you, right? The market bottoms way before earnings, way before the unemployment rate, way before the economy, all this stuff, the market doesn't wait around. but sometimes the market bounces, then it falls again. And so it just plays so many head games with you.

18:36Ben Carlson:And that's kind of the point I wanted to bring across is that trying to figure this stuff out in real time is so, so hard. And I think it's just way easier to have an allocation that takes this into account, right? That there's going to be these periods, there's going to be the volatility, like just find an allocation that you can stick with and live with. That's the whole thing. You bring up an example you've used in the past, Bob, the world's worst market timer, where he like keeps putting in money at market highs and you could take a couple different angles of it. But one of the things that strikes me, um, as you were talking about your dad, getting back in the market and just like living through the financial crisis in the period after it was that I don't feel like the all clear bell was rang until like the late teens, you know, people were worried about European debt.

19:21They were worried about double dip recession. They're worried about U.S. debt, all these different things. And I don't know how this feels for you, but every time we've had a bear market since then, I'm kind of like, meh, this isn't that interesting. Like the financial crisis, I was lucky enough to be working in and that was my first real bear market. And everything since 2022 is really slow and painful. But even then I was kind of like, you know, for a bad market, this just isn't that eventful or exciting or scary.

19:49Ben Carlson:I agree. I kind of came up at the same time as you. And it was my first real, I took a new job in July of 2007. So I worked for a few years and I came to this, this new firm and it was right when the credit market started really like getting disrupted. And you had this like quant meltdown and all this stuff happening. And I think it was a real eye opener for me about like, yeah, what real risk is like and how freaked out people can be. And that's one of the reasons that I started writing because coming out of that, I just, the pessimism was overwhelming. And it's kind of funny because the 2020 seems to be different.

20:20Ben Carlson:Like it's the other side of the coin. There seems to be more optimism among market people. But I saw all these nonprofit institutions managing, you know, hundreds of millions or billions of dollars trying to get all their money into hedge funds and black swan funds, like after the crash had already happened. It's like, no, you don't realize the stock market just fell almost 60%. Like I wasn't out here pounding the table for a bull market, but I'm like, you know, if this has happened in the past, this is like, you know, better things come from this. The prices are way lower. The valuations are lower.

20:50Ben Carlson:The dividend yields are higher. I'm reading this stuff in books because it was my first time experiencing it. But I'm like, this is the stuff they tell us about, right? These are these once in a generation system resets. And yeah, you're right. It never felt clear after that. I do remember the double dip recession stuff was huge. 2011, we almost had another bear market and people rarely felt, you know, like, is this okay now? And no one would have possibly predicted like a 16 year bull market from those levels, right? No, impossible. Like we're not going to have another real recession for that long.

21:21Ben Carlson:It was, it's, it's kind of hard to believe that it worked out like it did. We went years without even a 10 % drop. And I remember, um, you know, at PlanCorp, I was working and there were a number of employees who had been at the firm for years and never seen the market go down 10%. And they were really nervous. And I was like, wait, what? Like guys, just like, wake me up when I'm when we're down 30 % and then we can send the panic email. But in general, you know, I think being a student of history helps the fact that we pay such close attention helps the fact that you I mean, again, you're publishing every day, I publish probably every week, but documenting your thoughts really helps give you a clear vision of what it was you believed at the time about the future about the past.