Market Simulations & Financial Planning | #411 (John Yang)

24:29And then we also publish the expected return of the portfolio. Our paper has everything in geometric return. So it's simple for people to interpret and to do their own projections with if they want. but there's kind of this intermediate step you have to convert the asset class geometric means back to their arithmetic means and then you can take the weighted average of the arithmetic means to come up with the expected arithmetic mean of the portfolio but then in order to get the geometric mean of the portfolio you need to know what the standard deviation of the portfolio is and in order to do that you need the correlations of those asset classes along with the standard deviations of those asset classes and the weighting that they make up in the portfolio.

25:11So once you're able to do that, then you have the standard deviation of the portfolio and you can easily convert back to the geometric mean of the portfolio. You may think that this is just unnecessary. It's just complex. It doesn't really add any value, but it provides, well, based on our current data, we see about a 20 to 30 basis point increase in the expected return of the portfolio, depending on the asset mix, when we do it the proper way, which is converting to arithmetic means, combining the arithmetic means, and then converting back to the geometric mean of the portfolio. And the difference that 20 to 30 basis points is the benefits of diversification and the correlation benefits that you get from holding those different asset classes in a single portfolio.

25:52Ben Felix:Yeah, it's really interesting. If you're not doing that, if you're not going through those steps to calculate the portfolio level expected return. You're underestimating your expected returns and you're ignoring the effects of the positive effects of diversification. That's right. And 20 to 30 basis points that can exceed fees in some cases. So it is a meaningful number that I think should be included. And it's worth noting too that one, Conquest will do this automatically for you. It has the proper portfolio math baked in so that if you provide the asset class expected returns, that it will do that conversion properly.

26:30And two, previously, we also were making that same mistake. So we were not immune to this. We found the mistake, corrected it. And I think that also kind of speaks to the client testimonial that it's nice to have a whole team of people, a financial advisor to diagnose that issue and fix it as a bit of a tall task because of all the other work that they have going on. But having Ben and I and other people to think through this stuff and to take care of it in conquest, I think is pretty powerful.

26:54Ben Felix:Constantly nerding out, looking for little things like that. It matters. As you mentioned, it does increase the expected return that we're using as an assumption, which changes the advice on like, how much can you spend in retirement? How much do you need to save? Like this stuff, now the future is super uncertain, but I would still say that this stuff matters. Definitely. So real quick, right now when we do our simulations, returns are modeled as random. We're defining the mean, the standard deviation, the correlations, and then we're just pumping out a thousand simulations. It's a pretty basic simulation process.

27:27Ben Felix:John Yang, we're going to go to our interview with him in a second. He's going to talk about what him and his student group did for their industry project and how it changed, how they came up with an improved way to do the simulations. And then once we've talked with John Braden, you've got some really interesting analysis on how using their simulation process instead of ours changes long-term projections and what they specifically change and why it's anyway, super interesting. So John Yang is a financial engineering student at Columbia Engineering with experience in quantitative modeling markets and applied analytics.

27:59Ben Felix:The class has now ended, but his project with professor Micah Robbins was on a synthetic data generation project for PWO Capital, which was focused on building more realistic market scenarios for financial planning and risk analysis. John's work on this project examines how advanced simulation methods can better capture asset behavior, tail risk, and periods of market stress than traditional models. All right, let's go to our conversation with John.

28:28Ben Felix:John Yang, welcome to the Rational Reminder podcast. Thank you. Great to be here. We're very excited to be talking to you. So real quick for listeners, I'll give a little bit of background. Professor Michael Robbins, who's at Columbia University, reached out to me. He runs an industry project with his class every semester. And the industry project basically takes a group of students and assigns them to someone in an industry or a company in industry. And they're given a problem from that person or company or whatever it may be to solve a data-based project. So Professor Robbins reached out, asked if I thought I had anything interesting for students to work on.

29:09Ben Felix:And I was like, actually, yeah, there's something that we have been trying to solve and we just haven't dedicated the time to do it. And so we landed on that as a project. But Braden, maybe you can real quick introduce the problem that we asked John and his team to solve. It comes back to our expected return assumptions model. We still are using a Gaussian distribution to sample our expected returns for each asset class. And it's not a perfect approach, but it was simple and it wasn't extremely obvious to me how to improve it or what needed to be done to improve the model. We know that returns don't follow a normal distribution in reality, but we knew that it was also going to be a big undertaking to improve that.

29:54When this opportunity came up to engage an academic partner, it seemed obvious that this was a great opportunity to improve our model. So John, why don't you tell us a bit about how you try to tackle that problem? So first of all, it's great to be working with you guys for the past two months. And Professor Michael Robbins actually helped us a lot in the process. And how this is going to work is I'm just going to share my screen and I will talk about the approach we took in improving the model. So like Braden have talked about before, we were essentially trying to simulate test cases for wealth planning using synthetic data generation.

30:31And in past, it hasn't been done through Goshen or normal distribution-based models. What we are trying to do is to improve that. And I think a good way for the audience to grasp what's going on here is we're essentially translating the forward-looking expected returns and volatilities into market scenarios that can be used as test cases. Expected returns and volatilities are very useful in looking at how your portfolio is doing or how the market is going to perform. but they're one-dimensional. There's still summary statistics. They don't actually tell us that sequence of returns that you might live through.

31:08And when planning, let's say for your retirement, that sequence matters a lot. You're unlikely to experience, say, 6 % return year over year, like in a straight line. You experienced some good years, some bad years, some recovery years, some drawdowns. And during crisis, assets that crash together when they're expected to diversify each other. The question for us is once we have the forward-looking assumptions, how do we turn them into realistic return paths that can be used to test the resilience of a wealth plan? This is why we are generating synthetic data. I think one thing that I need to clarify before I start to talk about approach is that synthetic data is not randomly generated data.

31:49It's very mathematically grounded estimation that is designed to represent historical scenarios. In our case, each synthetic data point is a part of possible return path that a portfolio could experience. And the goal for us is not to say this exact path will happen, but rather we're trying to create many realistic paths so we can ask better planning questions, such as what happens if bad returns arrive early or what happens when a crisis hit?

32:15Ben Felix:So it's basically like we know returns aren't random. Imperically, you can see that by looking at real data. we had been using randomly generated returns to do retirement modeling and stress testing. So you guys were taking, if we look at real historical data, what has actually happened? So we know things like there tends to be a little bit of mean reversion, like you were talking about. Volatility tends to cluster. Correlations tend to go up during bad times. And so you guys were taking all of those empirical realities and trying to figure out a way to generate synthetic data that reflects those properties.

32:50Exactly like you said, once we think about the problem that way and think about synthetic data generation, the workflow becomes, I think, more intuitive. So we take the expected returns and volatilities and correlation assumptions as an input to our model. These inputs are fed into our simulation engine. Its job is to take those assumptions and generate a portfolio return path that can be used for long-time horizon planning. The simulation engine is not trying to create new market forecasts. It's like grounded in historicals and try to generate something that's more realistic. We are not the first ones that are doing this.

33:25Like Braden said, PwO had a baseline approach before based on Gaussian distributions. This basically utilize a multivariate normal distribution and run a Monte Carlo model on it. And for listeners who are not familiar with what Monte Carlo simulation is, it is a common framework for generating correlated data. You can basically take expected returns, volatilities and correlations, and the models will generate simulated returns from that normal distribution. There are a lot of advantages to use it, like Braden mentioned. It is very easy to implement. It is also very easy to collaborate. Also, on top of it, it's also very easy to explain.

34:02So I think it's definitely a great baseline to start from, but it also has its issues. At the end of the day, assuming a normal distribution, a normal distribution is smooth and is symmetric. we're thinking about the bell curve that look exactly the same, perfect on both sides. And real markets don't do that at all. In reality, the left tails can be ugly, which means when crisis happens, it goes really bad. Losses can cluster. Like when you have one bad day, it doesn't stop there. Like it goes on a strike. And those are the issues that we're trying to address with our new simulation model. More specifically, we have three challenges that we're trying to have our model to address.

34:43First, each asset need to have its own personality. In the Goshen baseline, you're kind of assuming everything follows a bell curve, but that's not true. Bonds behave very different from equities. That also behave very different from, say, oil futures. Each of them has to be fitted independently instead of just assuming a bell curve for everything. And the second challenge is that the asset classes also need to move together realistically. You can't just model equity without thinking about bound, because think about it when equity drops, when it's like reacting to one macro event, asset classes don't move in isolation.

35:21So we're trying to model that correlated behavior in our model as well. And the third challenge that we're addressing is that the model also need to take into the bad outcomes and the tail event seriously. For now, a normal distribution has a relatively thin tail, and it's not really giving the tails the attention that it deserves. That was my setup for the problem. I don't know, Ben, if you have any questions about that or anything you would like to clarify.

35:48Ben Felix:No, I mean, I thought that was great. We're simulating each asset individually. As you said, they each have their own personalities, their own characteristics. And then we care about how they behave relative to each other through time. it's the correlation but it's also stuff like everything going badly at once which can happen sometimes even with stocks and bonds which we've seen in pretty recent history and then the tails which as you said in a normal distribution they really don't get as much attention as they probably deserve exactly those are the exact steps that we're taking in our simulation process to tackle the first thing you've said about like the individual asset classes return distribution We first built the historical return series for portfolios that represent asset classes.

36:31Then we model out its distributions based on its historical probability of having different returns. It's also important to mention that we are building those proxies for those asset classes using indices. For example, we're using the MSCI EAFE Investable Market Index for approximating international equities. And we combine them into portfolios to make sure it realistically represent one asset class, for example. And once we do that, we learn the historical distributions. We model out the correlated structure between asset classes. Through doing that, we have our shocks for the market that we are trying to generate.

37:14At that point, those data are still normalized. They're in standardized space. and what's good about this model is it allows you to impose the expected returns that you believe to be true and that can be adjusted from time to time when you're planning.

37:31Ben Felix:That's really cool so the way that you set it up the shape of the distribution and the correlation properties and the tails and all that stuff gets defined using historical data but then once the shape of those distributions and the way that they work with each other are defined we can put in whatever market assumptions we want. Yeah, exactly. I'm also going to go through that as well and how we do that. But I think we can first look at how we are modeling out the correlation structures and the distributions of the return of every single asset class. Essentially, we are not only thinking about modeling those two things, the correlated structure and the marginal distribution of each asset separately.

38:12When we're implementing it, we're also like dealing with them separately using two separate steps. I think essentially we're asking two questions here. First is how does each asset class behave on its own? For example, what does a typical year look like for Canadian equities? How often do we see a strong year? Second question we're asking is how do the asset classes behave together? How do they move together? For example, when US equity is down, are Canadian international equities also down? Or when equity is down, does bond help offset the loss? When we're modeling it, we also do it separately.

38:48First, we handle the dependent structure actually. And we do that through a T copula. Without being too technical in getting to what copula is, an easy way to explain it is it's just a part of the model that links the portfolios, linked asset classes together. It helps you generate the correlated data by encoding it with essentially a matrix. And the reason we're using a T copula rather than the Goshen copula that PwL was using before is the T copula captures the code movement at the tail better. That was one of the challenges that we were talking about earlier. And I think what is more interesting is how we're modeling the distribution of each individual asset classes.

39:35Originally, the method was just looking at asset classes as a classic bell curve. You're assuming perfect symmetry, smooth distribution of returns, but that is not realistic. So instead of forcing our portfolio into the same bell curve, we let the historical data describe the shape more directly. For the distribution that we're modeling, we also divide this problem into two parts. For most of the distribution, we use the actual historical pattern of the returns. We first standardize the returns. So we're looking at the shape of the return pattern rather than the absolute return level. And then we use that standardized historical pattern as the empirical distribution.

40:15We're basically replicating the historical probability of each return level happening in the simulated data. That works for the majority of the distribution because we have enough observation there, like in the top 90%. But in the bottom 10%, that's like when this asset class is doing the worst, that becomes a problem. This is because these events are rare and bad outcomes. Think about 2020 when COVID happens. U.S. equity was down, I believe, 34 % in a month when it was like the worst. So those are sort of outliers if you purely look at it from a data science perspective. However, from a wealth planning perspective, it is a completely different story because those crises are actually the most important challenges to your wealth plan if you're trying to conserve wealth.

41:05For the bottom part of the distribution, we use extreme value theory. More specifically, we're using a generalized Pareto distribution. What it does is it is a specific distribution that is designed to fit those extreme events. So by combining those two methods together when we were estimating the marginal distribution of the return of each asset class. We're essentially using historical data where we have enough data, and then we are using like tail specific model where data is sparse.

41:36Ben Felix:Very interesting. So we don't have enough data to properly capture the tails. And so you found a way to sort of manufacture tails that are more realistic. Yeah, exactly. And I wouldn't say manufacturer because it is still based on historical, instead of forcing it into a specific bell shape or forcing it to look like historical exactly, which has a lot of zigzags because the frequency those events happen, we fit it with a more appropriate distribution. And that's how we model the individual asset classes return. And once the individual return distribution and the cult movement structure are defined, we now use Monte Carlo simulation to generate scenarios.

42:19Well, Monte Carlo, what it was simply doing is creating many possible future paths instead of pretending there's only one future. If you're only looking at summary statistics, you're essentially assuming there's only one possibilities. And that is what I think is going to happen. But Monte Carlo generated many possible paths. And on average, they match the expected volatility and expected return that you specify. One analogy that I really like when I'm explaining this is that expected return and expected of volatility is the destination that you set. And Monte Carlo simulation tells you many different passes that you can get there.

42:57Ben Felix:Yeah, that's very cool. The next part is going to be very useful, especially for wealth planning when the market condition is changing all the time. I mentioned earlier that we're learning the shape of distribution from history, but we're not necessarily assuming the historical return is going to be the same as what it is now. Instead, we can impose a forward-looking assumption on our simulation model. And up to this point, the model has learned the pattern of risk, the pattern of return from the history, but we do not want to simply replay history. So the model first created standardized shocks, but now you can re-standardize everything into a standard space, and then you impose the target return and target volatility on the shock sizes.

43:44Think about the simulation as a generator that's generating shocks, but it only tells you how bad is the shock relative to normal, but you haven't defined what normal is. And by imposing those mean and volatility, you're defining what normal is.

43:59Ben Felix:Yeah, I love that. That was one of our big concerns with starting this project is that we change our expected return assumptions twice a year. We always restate them based on the way that we calculate them. And we need to be able to take this return generating process and input what our current mean expected return or current standard deviation are. You guys solved that in a really nice way. It's actually a lot of fun doing that in the process. When we were trying to model it in the first place, we kind of assumed the historical mean is going to be the same as the future because we didn't think that would be an issue.

44:33But when we were doing testing, we realized it is completely off because let's say you take a set of data from the 1990s, and then you compare it to today's market scenario, the distribution of the return can even look very different. And then not to mention the expected value and volatility. Looking at recent events, the volatility is through the sky, and that is not something that would have happened in the last decade. After doing all of these, the final output that we are getting is a thousand simulated paths, and each of them covering 80 years of returns. Through our testing, we see that our results have preserved more realistic, higher-order distribution features, for example, skewness and kurtoltis.

45:17For listeners who are not familiar, skewness simply means the distribution is not perfectly balanced or symmetric, and kurtoltis means extreme events in the tails. Essentially, we're looking at the tails. If you look at the comparison graph on the right of the slide, the newly simulated distribution using our method are compared with the historical behavior, the historical empirical CDF, the cumulative distribution of the return, and the Gaussian baseline that PWL was using before. You can see that the new method followed the historical shape more closely, especially in the left tail where the Gaussian model is way too smooth.

45:56Ben Felix:Yeah, it's super interesting. You got very close to the historical distributions with your simulations. That's the benefit of using the empirical structure. we're not forcing it into any of the shape it's the same as running a optimizer like when you're trying to optimize there are constraints that's stopping you from getting that exact shape that history has showed us it's also really interesting how different both the simulated and historical returns are from the normal distribution for people who are not watching there's a red dotted line showing the normal distribution and then there's a shaded area and a green line showing the simulated and historical returns, they're not the same shapes.

46:37It's even interesting too how it differs from asset class to asset class. Like if we look at the fixed income, there's a dramatic difference between the Gaussian data and the sample data. Same thing with US equities, but Canadian equities and international equities seem to be a little bit closer. And not just visually, we've also ran tests on it to look at how big the improvement has been comparing to the Gaussian baseline. We believe at the end of the day, the goal for us is not to produce a more complicated model. Rather, we want something that's actually usable. So we look at three things. First, we look at if each asset class simulator return pattern look more realistic.

47:17That is what we mean by the marginal shape score. It considers things like quantile gap, skewness, cartoltis, and downsides loss measures like CVAR. And the second metric we look at is if our model does better on the tail. This measure is mainly driven by the tail quantile gap. And third, we also ask, does the model still preserve how assets move together? So that is the correlation structure that I talked about at the start of this presentation. We are measuring this using a basket of correlation measures like Pearson and Kendall, which some argue are more realistic. We're basically combining all of them together to capture all of it.

47:58So I see like a dramatic improvement in the shape and the tail score, but the co-movement score is relatively similar to what we had before. Is that right? Yeah, that's correct. That's actually like exactly what we were expecting. Because for marginal and tail shape, that's where our method is mainly improving in. That's because we are using the empirical distribution of return rather than assuming a bell curve shape. That's why it does a lot better in marginal and tail shape. But for cold movement, our goal is to be on par with the Gaussian method. That is because in the Gaussian method, the goal is to optimize for that correlation structure that was given to the model.

48:43So it is doing a very good job at it. And we are actually surprised that we didn't have to compromise any of the cold movement score to get a better marginal and tail score in that sense. But yeah, it's great to have. Well, we've only been working on this for the past five months, and this is like far from perfect. There are a lot of things that we still can do and will do in the coming semesters. For us, there are three natural extensions that we're thinking about. The first is dynamic correlation. Right now, we're assuming a statistic correlation structure. Essentially, what we're assuming is that the average relationship between asset is stable across all environments.

49:27But in real markets, the relationship between asset classes changes. For example, think about the recent tension in the Shade of Hormuz. The oil price, inflation, expectation, and bond yields, equities, currencies, those things are essentially responding to the same macro event, and they're interacting in the way they do right now. But that is very different. How they interact right now is very different from more stable time when inflation fear is not a trigger for the market, for example. By using a dynamic correlation, we can model each period separately and better replicate the history with more realistic test cases.

50:08And the second thing is something we weren't able to address with our current model. That is volatility clustering. Ben mentioned at the start of the podcast, at the start of our conversation, that some of the bad days arrive together. For example, in 2008, you don't just get one bad trading day. Once there's a fear in a market, people sell everything and those sell-off continues for a long time. And those clusterings are not modeled in our current approach because we're assuming independent draws when we're simulating. So that is another extension we're trying to work on. Our current idea is potentially using a GARCH style model that deals with time series.

50:51And by incorporating the time dimension, we can solve that problem. And the third thing is not as much of an extension to our model, but rather an alternative path that we will explore in the future. That is gang models. So gang is generative adversarial networks. Those are machine learning models that are the new cool kids on a block situation. people are using that to generate synthetic data. So essentially what it's trying to do is that it is generating a set of samples using a machine learning model and then using another model to judge it. You kind of like go back and forth and find realistic scenarios.

51:28The drawback for that is that there's still a lot of research that need to be done. And like a lot of machine learning models, when you're using it, you might be overfitting. You might discover patterns that are not there at all. When you're simulating it, you're essentially amplifying those patterns. That can be an issue for clients who don't want their wealth to go wrong.

51:51Ben Felix:Very cool. You'll leave these as sort of breadcrumbs for future student groups. And this project will continue with Professor Robbins. And we may be able to implement some of these return distributions in the future. Exactly. I think to close my talk here, I think the main takeaway from this is that forward-looking assumptions are starting points. when you're planning for wealth to use those assumptions in planning. They had to be turned into a path and that's what we've been working on for this project. The baseline Gaussian model that we start with is clean and transparent, but it makes market looks way too smooth than it actually behaves.

52:27And our framework keeps the same assumptions, but it produced paths that are better reflecting the messiness, I would say, of the market. We look at asymmetries, we look at fat tails, as we look at assets that are moving together during crisis. And those are the things that matters a lot for a long-term wealth planner who are planning for their retirement or like preserving their wealth. And for the next phase, we'll continue to work on the extensions that I talked about, like dynamic correlation matrix and a GARCH model, which handles volatility clustering.

52:58Ben Felix:Awesome. I've got to read the sentence that you have here on the key model improvement, because it really does summarize it well. The T-copula plus empirical EVT framework is a stronger simulation baseline than the Gaussian model because it better reproduces historical shape and downside risk while maintaining comparable co-movement. Very nice. Very clean. Thank you. Awesome. That was great, John. Thanks so much for coming on and summarizing the work that you guys have done. No problem. It's a pleasure to work with you.

53:28All right.

53:28Ben Felix:That was cool. It was cool to have John's slides and kind of see what they presented to the class. I thought that was just kind of neat to have a peek into that. I thought John was great. This was his first podcast ever. And I thought he spoke really well. For sure. It was great. Really enjoyed working with them over the course of the past few months and really cool to see what they come up with. The main question left in my mind after hearing his analysis is great. This is real improvements on our modeling approach, but how is this going to impact the financial advice? How may this impact the recommendations that we're providing to clients or the financial decisions that we're helping our clients with?

54:05That's sort of the genesis of this post analysis. I wanted to quantify that impact on the financial advice, but I also wanted to look at their model using the DMS data. That's one thing we didn't really talk about too much with John, but they were using monthly index data with 12 month rolling time periods. The index data only went back to 2003 for all of the asset classes that we were looking at. So it was a relatively short time period. The first thing I want to look at was what happens when we use the annualized DMS data that goes back to over 100 years of data? How will that change, if at all, the outcomes of his analysis?

54:46I'll say upfront, ideally, I'd like to answer this question of how does this new model impact the financial advice? I'd like to answer that directly in Conquest. But at the moment, we can't. We can't send those 4 million data points that I talked about earlier to Conquest directly via API so I can make real-time changes. Instead of doing that, I built a pretty simple life cycle style model where I was able to capture the basics of the life cycle. So I modeled everything in terms of after-tax cash flows, modeled accumulation and decumulation. I modeled different ages, so different time horizons.

55:23So I modeled investors that were 20 years old all the way up to 80 years old in 10-year increments. I fixed retirement age at age 65. So that way we can model different lengths of the accumulation and decumulation and also decumulation only. So all of that was captured. And we looked at different asset mixes as well. So the 60-40 portfolio, 80-20 and 100 % equities. And then in each case, I fixed the spending at the sustainable spending amount. So I solved for the spending amount that would lead to an 80 % success rate using our previous Gaussian distribution. That way we have a benchmark that we can go back to and it's also a realistic benchmark.

56:06It's representative of an actual plan that we would work with our clients on and not something that is someone spending way below their means or way above their means. It's sort of representative of a realistic client scenario. That way we can benchmark the difference between this new model and our old Gaussian approach. So the results, first I'll talk about the differences between the DMS data and the index data. What we found were pretty similar results, but we found the Gaussian represented the DMS data better than it did the index data. This wasn't too surprising for me to see. It was kind of what I was expecting to see because the DMS data goes back a lot further.

56:43It's a lot more independent samples. The 12-month rolling time periods monthly data has data points like the March 2020 drawdown in it, which can lead to these very extreme left tail events. And that was one of the struggles that we worked with over the course of the project, was trying to account for this lack of data and the index data and the over-representation of the March 2020 drawdown, especially when it didn't really impact our long-term financial planning projections. Your clients obviously had to be able to withstand that drawdown, but beyond the behavioral side of it, it didn't really impact our financial planning projections necessarily.

57:23Ben Felix:Well, it came back so quickly, right? That's one of the crazy things about higher frequency of return data is that you can see these crazy events. Even with daily returns, it can be even crazier. There can be these massive short-term drawdowns that bounce back. And it's like, Like if you look at annual data, things just look a lot smoother than if you look at monthly and then likewise with monthly versus daily. So that finding, I think does make sense. Totally. But importantly, the new model improved on both. So even though the Gaussian was a better representation for the DMS data, this new model performs better regardless in both the DMS data and with the index data.

57:59So I think it's a step forward regardless of how you look at it. we can see the same thing too when we're looking at the tails and maybe matt can overlay some of these graphics on the screen for the viewers to see but ultimately we can see the same thing that i just talked about where the tails of the index data were fatter than the tails of the dms data specifically looking at fixed income and u.s equities where the dms data was closer to the gaussian but like i said it's still a better representation of the data than the gaussian so it's a step forward nonetheless.

58:32Ben Felix:For that part, basically what we found was that the new simulation method makes the return distributions look more similar to actual historical return distributions than what we've been doing previously. Is that a good summary? Yeah, it's the same finding that John had. It's just, I looked at it also through the lens of the DMS data and made sure that the results were still consistent and they were. So then I moved on to answering the bigger question, which is how is this going to impact financial advice that we're giving to our clients? I started by looking at the success rates, the Monte Carlo success rate, using the new model compared to the Gaussian model.

59:09And remember, for the Gaussian, I benchmarked the spending rate so that it would result in an 80 % success rate. That's what we see exactly with both the DMS and the index data, we see an 80 % success rate. And then we see the new model improves that success rate by roughly two to three percent on average. Not like an earth shattering difference in terms of the difference in success rates. I don't imagine it would really lead to much different financial advice if you're looking at it through that lens of the success rate only. If we were to do a financial planning update and the new success rate went from 80 percent to 83 percent, we probably wouldn't change the financial plan.

59:50So I think it fits in that same lens where this change isn't going to completely overhaul the financial plans that are already in place with clients. And I personally view that as a good thing. I mean, of course, if we did find something out that drastically changed the financial advice through the lens of the success rate, we would have to implement it, but it would just be a much more difficult conversation.

1:00:09Ben Felix:We could see that when we do our expected returns update, which we do twice a year. So it's not a huge deal. Did you look at the impact sorted by age or like length of the projection? I did, but there wasn't really anything drastic to note. Interesting. Okay. I also looked at that success rate change over different asset mixes as well. And I found pretty consistent results. Using the index data, there was a bit of a relationship. It was still within 1%, nothing to note. I think because the results didn't change very much at all, the average was 2 % to 3%, but the bounds were a 1 % change to a 5 % change.

1:00:49So it's really all within the normal financial planning assumptions, updates could change the success rates by that much. So then I went from the success rates to looking at the change in the sustainable spending amounts. And you would expect that those results to be pretty similar because the sustainable spending, basically it's the same mechanism. The success rate is going to inform the sustainable spending amount. That's pretty much what we saw. we saw changes of spending that ranged within one to 2%. Not a drastic difference. I think one thing to know, like some people, it may be confusing because the tails of the distribution changed so much, but then we're not seeing a ton of change with this.

1:01:30The reason why is it's really the point on the distribution of like, say the bottom 20th percentile. If you're looking for an 80 % success rate, it's the return at that exact point, the bottom 20th percentile that's going going to drive these results. So if there's really not a big shift left or right, then it's really not going to have a dramatic impact on these metrics specifically.

1:01:53Ben Felix:That's an important point. And this is one of the cool things, as John mentioned about the model that they built, is that we are still using the same expected returns. We've just changed the way that we're generating the distribution around those expectations. So as you said, we're not shifting the distribution one way or another, we're just changing the shape of the distribution to be more historically accurate, realistic. That's right. So you wouldn't really expect a dramatic shift here anyways, because like you said, Ben, like we're already capturing the standard deviation and the expected mean, and that's the same as it was previously.

1:02:27Ben Felix:It's worth mentioning, you touched on this briefly because people might be thinking, why did you guys expend all of this effort if it doesn't actually change things that much? One of the big reasons, and we haven't really gotten there yet, but we're moving in that direction. This was a first step. one of the big reasons that we care about those higher moments of the distribution and the shape of the distribution is for other asset classes. So for us to model venture capital, for example, using a mean and a standard deviation and a correlation is just not sufficient to capture the expected effect of that asset class at the portfolio level, likewise for other sort of alternative asset classes.

1:03:03Ben Felix:So that's something we've struggled for a long time because even though they're not things that we recommend, we do have high net worth clients who have allocations of those for whatever reason, maybe they were recommended by a previous advisor and maybe they were done as a passion project or whatever, or maybe they just wanted exposure to that asset class. And modeling that in a financial plan has been a thorn in our sides for a long time. Using this type of simulation method is going to be a huge improvement when we get to those asset classes. I thought it was worth mentioning because we're not seeing huge changes to, small changes, we're not seeing huge changes to the simulation results based on this better methodology, but for other asset classes, it'll matter a lot more than for standard stock and bond portfolios.

1:03:47100%. And I'm going to circle back to this point once I cover this next section, because it's going to become even more obvious why that is important and why that's going to be super impactful. Up to this point, in terms of the success rate and the sustainable spending amount, not drastic changes. I feel like I make this point every time I'm on, but it always comes back to the goals of the client. If the goals of the client are purely focused on sustainable spending, then they're probably not going to receive really any different advice for the most part. They're still essentially on track to the same plan that they were using the Gaussian approach.

1:04:19I think that's a bit reassuring too, to the general community that if you're just using Gaussian distributions, it's probably okay. Like it's not something that you necessarily need to lose sleep over. But the next section is a lot more interesting when we talk about the distribution of wealth outcomes. So for someone with goals that are aligned to their final after-tax net worth or their estate value or their multi-generational wealth, it is a meaningful difference. And I think it's a pretty cool findings that I found. So the first thing here that was a bit tricky to wrap your head around at first when looking at the distribution of wealth outcomes, but the median net worth of the new model increased, but the mean of the new model decreased compared to the Gaussian model.

1:05:06So it's a bit tricky to think about but let me try to explain this. The median which is if we're running a thousand simulations that means the point that there's 500 that performed better and 500 that performed worse that increased and that makes sense because we have mean reversion so we should expect that distribution of outcomes to narrow. We would expect that the median would increase from that but the mean is not calculated in the same way. And because the lower bound of wealth outcomes is zero and at the upper bound, the Gaussian distribution or sampling from the Gaussian distribution can lead to these sort of runaway events where just imagine you have a sequence of returns that is a right tail after right tail after right tail that can really compound wealth.

1:05:51So imagine a 30 % return and a 40 % return and a 50 % return all stacked together. it's really going to compound wealth to this crazy high level that we wouldn't really necessarily expect to see in real life because there's no mechanism to pull those returns back down after a right tail return has been realized. In the Gaussian model, you see these really high wealth values that bring up the mean quite a bit. Regardless, the financial decision should be made using the median outcome. That's how you should be thinking about the expected outcome is in terms of the median, but I'm not sure if that's universally true among financial planners or retail investors that are doing this type of analysis.

1:06:29So I think that's an important distinction to make is the difference between a median and a mean outcome. The median outcome improved using our new model like we'd expect because it has this mean reversion and sequence of return modeling captured more accurately. The mean decreased.

1:06:45Ben Felix:Yeah, it's interesting. That's like Japan leading up to 1990. You did have a crazy right tail event where there were just incredible incredible returns bam bam bam but then you've got a cape ratio at 90 or whatever it was and returns don't last forever right tail returns don't last forever in real life but it is interesting how it changes the mean because that right tail in the gaussian distribution with random returns is totally uncapped but the left tail is implicitly capped because financial plans stop like failure happens when the assets run out you don't get a big negative number that's right what I thought was really interesting about that is through our conversation with John and our work with John we spent so much time looking at these left tail events and how to improve the left tail of the distribution but in terms of the impact of the financial planning distributions it's actually that upper tail that's impacted the most because it really is pulling these runaway events back down to earth I thought that was pretty interesting and the results are pretty meaningful too we're seeing that the median is improving by on average about a half million dollars in final net worth.

1:07:50And the mean is decreasing by a million dollars in final net worth. So these are definitely meaningful numbers that would impact the financial advice, especially for someone who has those types of goals, like estate goals or multi-generational wealth type goals.

1:08:05Ben Felix:People never believe, man. I haven't been in client meetings like this for a while now, but I know advisors still have this issue. When we show that long-term projection, for someone who's 35 or 40 years old, and it says they're going to have all these tens of millions of dollars in the future, people are always a little bit skeptical. So I mean, reining that in a little bit by improving our simulation process is great to show a more accurate number. And I think it's more believable for people. Yeah, exactly. Like you alluded to, Ben, if we're able to model this with alternative asset classes, I think that's been the biggest challenge that we've seen so far is that when we have our concentrated stock portfolios in conquest and we're trying to answer that question, And should I invest in this something that we would think of as like some sort of crazy investment where it's a concentrated portfolio or some sort of bet?

1:08:52It's difficult to show that in Conquest because Conquest will display the distribution of planning outcomes. And there's so many runaway events. You know, obviously, we don't publish this data, but I'm sure our audience can imagine that the standard deviation on these concentrated portfolios would be substantially higher than a broadly diversified portfolio that we would recommend. And so you see these extreme runaway events where there's 60, 70 % returns and the upper bound just goes crazy. So it's very difficult to present that to a client and actually make the case not to do it because it's human nature to look at that upper bound and see like, oh, I'm missing out on all of this potential growth.

1:09:32But meanwhile, we know that that is statistically almost impossible for that to happen because the distribution of a concentrated portfolio or individual stock is definitely not Gaussian. And the DMS portfolios are a lot closer to the Gaussian distribution than these other portfolios of single stocks or concentrated portfolios would be. And then there's also that mean reversion dynamic that wouldn't be captured either. So the runaways become even crazier with that level of standard deviation. So we've always found it really difficult to actually show that in Conquest. So that's why I'm really excited to see what the next version of this project will look like in terms of getting those other asset classes fitted with this model so that we can make a realistic, informed case for clients to look at that scenario through the appropriate lens.

1:10:22Ben Felix:Yeah, that's such a good example. People have trouble thinking about how much risk should I take? Should I invest in this or that? All that kind of stuff. One of the ways that we always come back to is let's put it in the financial planning projection and see how it affects things. People have a much easier time grasping. this will change my expected retirement date or how much I have to save or how much I can spend in retirement. Those are much more tangible data points for people to use to make investment asset allocation decisions. But you're so right that for some stuff like, should I continue holding this individual stock or should I invest in this private markets fund or whatever, those have historically been a lot harder to show people in our planning projection, like using that same type of approach, because for the reasons you've described, they often make things look good, even though we know that's not the right thing to show and the right expected result.

1:11:09Ben Felix:We didn't previously have the capability to model the components or the characteristics of the return distribution for those asset classes. We're moving in that direction, which is very exciting. Totally. And I'd really love to be able to show a visual of the expected outcome down to completely eliminate those higher bounds because it's really not practical for planning purposes. If we were able to show clients what the expected outcome is, and obviously our audience would know that those other forms of risk are uncompensated risk, so we wouldn't expect the expected outcome to improve in that scenario, but we would expect those downside events to increase and overall the financial plan to look worse.

1:11:47If we're able to just show that, then it would be painfully obvious that this is not the right move. But just including those upper tail events makes that conversation really difficult because it's human nature to be drawn to that higher potential for growth.

1:12:02Ben Felix:After going through this and running the simulations, what are your main practical takeaways? So practical takeaways, again, it comes back to goals. So if the client's goals are focused mostly on spending or almost entirely on spending, then it probably won't change the advice necessarily. And I think that's good. It makes that transition smoother for us. We're not completely overhauling a client's financial plan. So from a change management perspective, it makes life easier. But for people that do have any amount of legacy goals or interested in making decisions based on their final after-tax net worth at the end of their life, then this does impact the financial decision.

1:12:42We would expect the median outcome to improve using our new model, which is more representative of reality. and we would expect the mean outcome to decrease just because we're decreasing the bound of outcomes. And the runaway events cause that upper bound to have a higher proportional impact on the mean. And also I think financial planners and retail investors that are doing their own financial projections should be looking at the median outcome rather than the mean outcome. So it shouldn't change the advice. The fact that the median improved should mean that their expectations should improve and might leave a little bit of extra net worth that they can plan for to delegate out of their estate.

1:13:25Ben Felix:Cool. I think we mentioned with John, this project is ongoing. I think the analogy that I use is that John and his team have left some breadcrumbs for the next group of students that will come through. With Professor Robin's guidance, he's got some pretty cool ideas for how we can make this simulation process even better and account for other interesting stuff like correlations that are time-varying when there's a market crash, things tend to crash together, being able to model stuff like that and just improving the overall simulation process further. And then another big piece that we did not get to a John and his team, even though we talked about it quite a bit was figuring out proxies for alternatives.

1:14:02Ben Felix:Like if we wanted to do this process, for example, using individual stocks or venture capital or private equity or hedge funds or whatever the case may be, a big part of that is finding good quality historical data that we can use to benchmark against what should our simulation look like so that's something that we're going to continue working through with the next groups of students but it's pretty cool having this relationship seeing the impact that it's having we've moved a little bit in the right direction but i think once we get through this next iteration we will actually start deploying this into conquest into the way that we're running our simulated data and at that point it'll be having real impacts on people's financial planning decisions which i think is pretty cool it's very cool all right i think that's it for the episode thanks everyone for listening.

1:15:26Thank you.

1:15:45has been derived from sources believed to be truthy, but not necessarily accurate. We really do try, but we can't make any guarantees. Even if nothing we say is fundamentally wrong, it might not be the whole story. Furthermore, nothing herein should be construed as investment, tax, or legal advice. Even though we call the podcast your weekly reality check on sensible investing and financial decision making, you shouldn't rely on us when making actual decisions, only hypothetical ones. Different types of investments and investment strategies have varying degrees of risk and are not suitable for all investors.

1:16:19You should consult with a professional advisor to see how the information contained herein may apply to your individual circumstances. It might not apply at all. Honestly, you can probably ignore most of it. All market indices discussed are unmanaged, do not incur management fees, and cannot be invested indirectly. Which is a shame, because it would be awesome if you could. All investing involves risk of loss, including loss of money, loss of sleep, loss of hair, and loss of reputation. Nothing herein should be construed as a guarantee of any specific outcome or profit. Past performance is not indicative of or a guarantee of future results.

1:16:56If it were, it would be much easier to be a Leafs fan. All statements and opinions presented herein are those of the individual host and or guests, and are current only as of this communication's original publication date. No one should be surprised if they have all since recanted. Neither One Digital nor PWL Capital has any obligation to provide revised statements and or opinions in the event of changed circumstances. See you next time.