Episode 16: Barry Ritholtz - Chairman of Ritholtz Wealth Management

0:00This week, I have the pleasure of speaking with none other than Barry Ritholtz, renowned investor, author, and the chairman, co-founder, and chief investment officer of Ritholtz wealth management. One of the fastest growing wealth management firms in the United States with$5 billion assets under management. With a career that blends market insight, media, and thoughtful commentary, Barry has become one of the most influential voices in modern finance. Barry is the creator of The Big Picture, a financial markets blog that has amassed over 275 million visitors. He is also the host of the Masters in Business podcast, the first mainstream, long-format finance podcast, having hosted guests such as Ray Dalio and Howard Marks.

0:44Barry is also an accomplished author, having written Bailout Nation in the new book, How Not to Invest, where he breaks down the most common mistakes investors make and how to avoid them. We spoke about his path from law school to Wall Street, the origins of Ritholt's Wealth Management, key lessons from his new book, and his advice to the next generation of investors. I hope you enjoy. Thank you, Barry, for joining me. I really appreciate it. Thank you for coming. Oh, thanks so much for having me. So I want to start your childhood and your upbringing. So tell me a little bit about your childhood.

1:16What was it like? And how do you think your kind of upbringing shaped the person you are today? Sure. You know, a pretty middle class upbringing. My mom was a real estate agent. My My dad started out as a personnel manager, getting jobs for engineers, scientists, people like that. When the company he worked for kind of blew up, when the founder died, he decided to open his own business running a sporting goods and sneakers shop. So I always had the latest, greatest Air Jordans and the like and wore them before we knew they had any value. Every now and then I'll wear like an old pair of Kobe dart frogs and people will on the subway will look at me like, you know what those are worth?

2:06I'm like, dude, they're just sneakers. But a pretty normal upbringing. I went to a state school. I was told I'm going to be an engineer because I was good at math and science. And so I did that for a couple of years. Wasn't really my favorite thing. and ultimately switched to political science and philosophy. Didn't know what I really was super enthusiastic about. There were like a lot of things I really liked. And so, uh, went to law school and ultimately kind of stumbled ass backwards into wall street and finance, a dumb coincidence of having grown up, um, you know, 45 minutes away from wall street.

2:53Yeah. And And so your mom was a real estate agent and your father was a placement officer for engineers and scientists, but eventually on a sporting goods store. So what were each of your parents like and what traits do you think you kind of inherited from each? So my father was very logical and rigorous and my mother was very creative. She had gone to music and art, played the piano, the drums, the saxophone, you know, had a perfect pitch. I could play any song for her and she can immediately play it on the on the piano. So a little bit of a little bit of both. You get some some of both your parents.

3:34My generation was kind of the latchkey generation. Like forget today's helicopter parents. Like you kind of learned how to do your own laundry and cook because, hey, they were both working. And if you want something to eat, you better make it yourself. probably what led me to a love of books and reading and eventually writing. And so, you know, I could spend years and years in therapy whining about my parents, or I could just say, all right, I adapted and made the best of it. You know, all that freedom that I grew up with, that your generation didn't grow up with, hey, it has its pluses and its minuses.

4:19But it meant that, you know, I fell down the rabbit hole of of Tolkien and Heinlein and science fiction. And and nobody told me, go do your homework. It's like I'm reading The Hobbit for the fourth time. I'll get to calculus eventually. I love that. And so you went to law school, which is not the most common background for someone on Wall Street. So how do you think your time at law school influenced, first of all, the way you think, but also applied to your career later as a researcher and strategist in the markets. So first of all, you would be surprised at how many people on Wall Street have a legal background or legal training.

4:59The stat that I remember was seven years after graduation, half of law school graduates aren't practicing law. But but here's a really fascinating thing. And I kind of joke with my business school friends. I think business school teaches you what to think and law school kind of teaches you how to think. And let me unpack that a little bit. So you have the starting documents, the Constitution, various other legislation, state and federal. And then each time a case comes up, it's a completely different fact pattern. And you have to figure out the classic syllogism. here's the rule here's the fact apply the rule to the fact that turns out to be really helpful when you're looking at things like markets and companies and investing and so here's what the market has done historically when a b and c has happened here's what the results look here's the range of results so so the first thing that you kind of learn is what what are the facts what are guiding principles.

6:09That's a great thing to learn no matter what field you go into. And then perhaps the most interesting thing to me anyway, is the concept of moot court. Moot court is second year of school and you have to argue a case from start to finish. Everybody is given the same legal research, the same case law, the same fact pattern. So it's not like somebody's research skill is going to pull a rabbit out of a hat. You all have the same tools to work with. But here's the tricky part. You don't know which side you're going to be arguing for. And so what you learn is you can't prosecute a case or prosecute a crime if you don't understand how to defend that case or defend that crime.

7:01And it teaches you that in life, there are really very few absolutes. There are two sides to everything. And when there are two sides to everything, it makes you consider things from a much more comprehensive perspective. So if you think that stocks are overvalued and they're going to go down before you make that bet, you should be able to articulate what happens if stocks aren't overvalued, what happens if the economy expands, what happens if there's this policy change or the Federal Reserve does this or consumers do that. And it forces you to not think of the world in binary up or down terms.

7:44It makes you think in terms of not just two sides to each issue, but in terms of probabilities. What's the high probability outcome? What's the low probability outcome? What's the big fat middle probability where most unknowns tend to fall? You know, when you think about the bell curve, typically how a lot of schools do grades, the tails, the outliers, those are really rare, very few F minuses and A pluses. But there's a lot of, you know, C pluses, B minuses is a lot of that middle part. Well, the world is like that. You flip on television and everybody is so sure of what's going to happen next.

8:33Everything is described in black and white terms. And, you know, one of the things you learn from moot court is there are lots of shades of gray. The world is complicated. Things happen in all sorts of ways for all sorts of reason. And if you grossly oversimplify things, you tend to end up with a false model of the world. And so that was to me, my favorite takeaway from law school is to really, you know, we all live in our own little bias bubbles. You have to be able to get out of that. You have to be able to see all sides of the argument, whether it's this stock or that or recession or no recession or impeach this judge or president or that.

9:20There's multiple multiple sides to everything. Just looking at the world from a very narrow, if we're talking politics, tribal, if we're talking investing, are you a stock guy? You a bond guy? You an active guy? Are you a pass? Are you an index guy? Are you a value? Are you a growth? Like everybody breaks the world up into these very binary categories. It's not how the world really works. I've never really, I've heard a lot that law school teaches you how to think, but I've never really appreciated that specific perspective. And especially I like that example of moot court. But I want to move on a little bit.

10:02So after law school, you went into corporate law for some time, and then you were a trader at what was the predecessor to E-Trade. So take us through your first job on Wall Street all the way to right now, being the chairman, CIO, and co-founder of holds wealth management kind of your job profession sure so um starting starting out as a uh a trader you know the the training is kind of modest give you a rough overview they show you how the technology works and kind of throw everybody in the deep end of the pool if you don't drown hey congratulations, you're a trader. And so I spent, I don't know, the first couple of years kind of figuring out why certain sectors or stocks would move, why the whole market moved at a time.

10:56This was in the mid-90s when everything was going up. There was a lot of enthusiasm for technology and software, particularly online and internet. And you ended up with a situation where on a trading desk, you have to execute orders for the house, but you also have the opportunity to trade whenever you saw an opportunity came up. And I tried to learn from, you know, you're on a long desk. All these people are mostly guys are on this desk. And I tried to learn from the people I sat next to. And what was fascinating was one month, the guy on my right is making a lot of money and the guy on my left is losing a ton of money.

11:39And then the next week or the next month, suddenly the guy who's losing all that money is making money. And the other guy who is winning is losing. And I tried to figure out, was it their style? What changed? I mean, it seemed almost random. And I kind of eventually stumbled into the world of behavioral finance. I got a book. The first book on Wall Street I ever read was Market Wizards by Jack Schwager. And even though the book talks to bond traders and currency traders and stock investors and all these different areas. It's all about managing risk, controlling your own behavior and making sure that whatever you're doing squares with reality, that you aren't, you know, sort of stuck in your own little make believe world.

12:33And then that was kind of interesting. And then the next book, I don't remember who gave this to me, but a professor up in Cornell named Thomas Gilovich, how we know what isn't so. It was one of the first popular culture books on, I want to say behavioral economics, but it hadn't quite gotten there yet. It wasn't until the 2000s that really the mainstream was talking about that. And that forced me to, instead of looking outwards to figure out why people were doing what they were doing, to turn around and look inwards and say, what's going on in my mind, my cognitive processes, the model of the world we all create, my own emotions, biases, et cetera.

13:24Every person has that. And you don't really think about them until you're watching your profit and loss each day, week, month swing. And you're trying to figure out what's going on here. And eventually I kind of figured out that, oh, this isn't a strategy or a tactic or something I'm doing wrong. My entire operating system is not designed to manage this at all. And that's really fascinating. And so you eventually started Ritholtz Wealth management with Josh Brown in 2013, if I'm correct. And when you started out, you didn't want to deal with the back office or the reason you didn't start your own wealth management firm is you didn't want to deal with the back office or with the clients.

14:13But you agreed that Josh Brown would focus on the back office, the administrative work and with the clients, and you would do the investment management. So you guys, in turn, built what is now one of the fastest growing wealth management firms in the United States. So I ask this question to you, what makes a great partnership. So let me just roll back the timeline a little bit. I had met Josh about two or three years earlier at a conference and immediately recognized him as talented. And he wanted to get away from the sell side to the buy side, meaning management, not sales and brokerage. And, you know, the financial crisis had come and gone.

14:55I had been fortunate enough, mostly because my mother was a real estate agent and I was looking at the real estate side. So I was able to see that coming. Josh joined me at the firm we were working at prior to the launch of Ritholtz Wealth Management. And people were just throwing a ton of money at me at the time. And, you know, we had a conversation once. He said, do you have any idea how hard it is to raise capital? You're like, you're turning this money down. And my answer was, we're not in the right place to manage money. I don't want to put them into house product. I don't want to put these people into a transactional non-fiduciary sort of situation.

15:41It's not in their best interest. So the conversation eventually reached the point where, Barry, you manage the money. decide how it goes, keep it away from the broker dealer side. We'll manage it on the RIA side. And once we get to$100 million, we could leave and launch our own firm. And pretty much that's the genesis. Chris Venn runs all the advisors in the firm. He was one of the first hires. Michael Batnick is a managing partner at the firm and really is where a lot of the rubber meets the road in terms of being the liaison between the investment committee and the advisors. He's a huge, huge help.

16:29What was so important with us, the four of us when we launched, is everybody more or less gravitated to their strength. And we each did what we did best. And, you know, that was 12 years ago and still going strong. And there are thousands, maybe tens of thousands of wealth managers out there. What makes Rit Holt's wealth management different? So there's there's a couple of things that makes us different. But there are two in particular that really stand out. One is all of our clients self-select. They all come to us and say, please help me with my asset management. I know that sounds a little odd, but most of the industry is outgoing, meaning smiling and dialing, reaching out for clients, working the circuit, doing what they have to do to raise money.

17:27From inception, we had literally a giant fire hose of people reaching out. So we didn't have to spend a lot of time and effort finding new clients. We got to spend all our time and effort creating informational content. The model is very much here. You could do this yourself. Here's a whole lot of great content and information how to do yourself. By the way, if one percent or less of you want someone to help you, we're happy to charge you a fee to to do it for you. And so that, you know, became the business model. So I think that's a little unusual in Wall Street. But the second thing was what I was hinting at earlier is the behavioral side.

18:14We were very conscious. I have been very conscious from the early days of my career. All the mistakes people make, the bad information they consume, the lack of understanding about numeracy, exponential compounding, some just really basic math concepts like framing and denominator blindness. And there's just so many things that we were not built for and that finance is very involved in. The vast majority of human history is adapting to evolving to adapt on a hostile savanna. You know, humans are soft and squishy and tasty and we don't have fangs or claws or armor. So we've evolved to adapt to be a cooperative species.

19:11And that has created our operating systems, which work great. Humans are are all over the planet. We're one of the top 10 species. You know, we're just right behind bacteria, viruses, worms, crabs, mosquitoes. Then us. I mean, that's how dominant of all the millions and millions of species over the past few billion years. We've managed to take over every corner of the planet just about not quite as much as as as the worms and the crabs, but pretty close. And so we're all born with this operating system. We all have this wetware. It doesn't always work to our advantage. Understanding that and understanding how to use that in terms of helping clients understand the vicissitudes of the markets, the economy, that stocks go up and down, that you there's a challenge in in planning long term.

20:17And, you know, just using all of the knowledge that has been amassed, especially over the past 20, 30 years, I think that's something that's really significant. in. And especially in today's age, many students know exactly what they want to do as a career starting out as early as their early teens or mid teens. And your career has been pretty unconventional as far as it goes in the financial markets world. And you're also a part-time podcast and part-time blogger. And we'll talk about that later. But how important has been trying new roles and seeing what works for you been? And what do you say to those who, students like me who think they know exactly what they want to do.

21:01I'm jealous. I wish I knew exactly what I wanted to do when I was your age. I had no idea. And it was definitely a source of stress. I will tell you, I don't think of myself as a part time anything. I think that the writing and blogging, the podcasting, the asset management, they're all sides of the same. I don't know. Let's call it dice, not coin, because there's so many different sides. I think in the book, I quote Daniel Boorstin, the librarian of Congress. I write to figure out what I think. There's something to be said for organizing your thoughts and putting them together in a very logical, coherent written piece, sharing it publicly, dealing with the sort of response you get from others about it.

21:52Like, I appreciate people challenging the ideas, forcing you to think through what you're saying to to defend certain positions, to reverse certain positions and say, oh, maybe I'm not thinking correctly on this. Maybe I need to adjust my thinking about this. I have found that to be really the writing part to be really helpful. The podcasting part is kind of amusing because I've been podcasting for so long. When I started podcasting, there were really no finance podcasts. I was writing for some bail. My first book, Bailout Nation, came out about 15 years ago. And I was writing after that. I was writing for The Washington Post and Bloomberg came knocking.

22:42and um when there was we're giving me a tour of the facility um part of the tour was hey we have these television studios how do you feel about a a tv show i was kind of lukewarm and they're like why not i honestly i don't feel like i have a whole lot to say every day oh the market went up half a percent the market went down half a percent you know it's all hindsight bias if you can't tell me why the market is going to do this tomorrow. Telling me tomorrow afternoon is you're just you're just telling me what happened. And, you know, I remember during the start of the Iraq war, a mosque got bombed and oil prices spiked.

23:27And the Wall Street Journal headline was oil rises as allies accidentally destroy a mosque. And I'm like, huh, I guess they're saying that those two things are related. By the end of the day, oil prices had reversed and fallen, but the mosque was still destroyed. So the same exact article, but the headline changed. Oil prices fall despite mosque getting bombed. And it kind of was like a light bulb went off and oh, so they have no idea why these things are happening. It's almost random. And you start researching narrative fallacy and the hindsight bias hey after we know how something ended when you know who won the masters tournament when you know how the movie ended oh it was the butler who did it right i won't spoil white lotus for anyone who might not have seen it wait what happened with walter goggins and and uh mary lou they what so once it ends you can't recall what it was like when you You didn't know what happened.

24:39And so you just rationalize that, oh, this was inevitable. This has to happen. I used to give a presentation not long after the financial crisis called This Is Your Brain on Stocks. And I wanted to demonstrate for people how the various biases that we're all every one of us are afflicted with and are wholly unaware of. So I like to ask an audience back in 2010, 11, 12, how many of you saw the financial crisis coming long in advance? Raise your hand. Two thirds of the hands in the room go on. That's great. Now, everybody put your hands down. Let me ask you that same question differently. How many of you saw this coming and you shorted Lehman and you shorted Bear Stearns and you shorted AIG?

25:27Anybody short those companies? No hands go up. All right. Forget those companies. How many of you saw the risk to the brokerage and bank stocks, the home builders and the mortgage and drives? How many of you shorted those sectors? No hands go up. All right. So forget the stocks. Forget the sectors. You all saw this giant crisis coming. How many of you shorted the market? Bet the entire stock market was going to fall. One or two. All right. Forget shorting the market. How many of you at least moved out of stocks into bonds and cash? Still, no hands go up. So now let me ask you again, because I reminded you of your actual behavior during that time.

26:09How many of you saw the crisis coming, shorted AIG Lehman, bank stocks, moved out of stocks into bonds, and otherwise people kind of are shocked? Because at the moment, of course, of course we saw the financial crisis. How could you miss it? It's giant. So once you start to realize how how influenced we are by our own software, by our own operating system, it's very challenging to make good decisions without taking that into account. There are lots of different ways you can do that. I'm happy to go over all of them. But be aware, there's an entire field called behavioral economics. And it's a grad level course that you can take at a lot of schools that spending two, three, four years studying that.

27:07and i want to take kind of a step back um talk a little bit about your book so very recently published a book called how not to invest i have it right here with me and i took a read before this interview and i very much recommend it to especially students who are listening this to this podcast who want to who are just testing their feet in the water of the markets and kind of want to understand what to do by learning what not to do um so you focus on three categories in the book, bad ideas, bad numbers and bad behavior. Can you kind of break down those three main themes and some of the common mistakes investors make in each theme?

27:43Sure. So let me start out by giving a shout out to Charlie Ellis, who in the 1970s wrote a short paper called The Loser's Game, which 20 years later, he expanded into a book called Winning the Loser's Game. And I play tennis. I don't know what your game is. So tennis is a perfect example. Tennis is two games in one. There's a winner's game and there's a loser's game. The winner's game is how the professionals play, how the 0.01 percent of all the millions of tennis players in the world. The winner's game win by they hit with power and accuracy. They serve aces. They kiss the line with the ball.

28:28They use drop shots that are impossible to hit back. They keep the ball away from their opponent's sweet spot. The professional game is a winner's game because you win games and matches by winning points. But, you know, you and I are not Rafael Nadal and Roger Federer. We're amateurs. Amateurs don't win usually by scoring points. Most of the time, amateurs lose through unforced errors. We double fault on the serve. We hit the ball long. We hit it wide. We hit it into the net. We hit the ball with not enough topspin. It comes right up to the sweet spot of our opponent's swing. And they're able to put the ball where we can't touch it.

29:16And so Ellis drew the comparison between investing and tennis. Investing turns out to also be two games in one. There's a winner's game and a loser's game. The professionals win by generating alpha. They score returns over and above what the market gives us. Amateurs, which includes most nearly everybody, because even foundations and endowments and institutions, they're still run by people and they still have the same wetware the rest of us have. We lose by making mistakes. We overtrade. We either have too much risk or not enough risk. We're not aware of taxes. We think we can market time. We think we have the ability to not just pick sectors, but pick stocks.

30:08And it turns out that the parallels are so perfect because in tennis, if you're an amateur, as most of us are, just make fewer errors and you do much better in investing. Everybody is trying to hit that next. I'm going to mix metaphors to hit that next home run, but don't try and hit a home run. Just wait for your pitch. Get some wood on the ball. You know, make have a good swing. If you as a as an amateur just make fewer decisions, make fewer mistakes, you're just much better off than everybody else who's just making one error after another and hurting themselves. So that's the broad overview.

30:51We could drill down into bad ideas, bad numbers, bad behavior. Tell me where you want to go from here. So I want to focus on bad behavior for right now. And you kind of cite emotional decision making as a part of bad investing. So how can you investors, investors new to markets mitigate the role of emotion, bias and human error in their investing? Sure. So so fear and greed is what's notoriously known for driving markets. You know, there's an old joke. There's nothing more infuriating than seeing your idiot neighbor down the street become wealthy. Right. So stocks begin to run up. It doesn't matter what it is.

31:37It could have been Pets.com in the 90s. It could have been Bitcoin last decade, Apple and Amazon before that, NVIDIA now. and it doesn't matter. You missed it. You're not participating. And there's FOMO. There's, you know, FOMO is nothing more than greed by another name. And so you jump in, you chase the hot stock and that rarely seems to work. Um, so your fight or flight response is, is one of those things that, um, uh, your, your amygdala is part of your limbic system. It's where all the, uh, fear and greed and reward and and all the the pleasure centers are. And, you know, I had a really interesting conversation with Bill Bernstein, former neurologist turned investor.