Wait But... How Do I Invest Using AI? w/ Hari Krishnan & Vasant Dhar PhD

0:00If you've been considering futures trading, now might be the time to take a closer look. The futures market has seen increased activity recently, and Plus 500 Futures offers a straightforward entry point. The platform provides access to major instruments, including the S &P 500, NASDAQ, Bitcoin, natural gas, and other key markets across equity indices, energy, metals, forex, and crypto. Their interface is designed for accessibility. You can monitor and execute trades from your phone with a$100 minimum deposit. Once your account is open, potential trades can be executed in two clicks. For those who prefer to practice first, Plus500 offers an unlimited demo account with full charting and analytical tools.

0:45No risk involves while you familiarize yourself with the platform. The company has been operating in the trading space for over 20 years. Download the Plus500 app. Trading in futures involves the risk of loss. It is not suitable for everyone. Not all applicants will qualify. Hey, everyone. If you like this podcast, go behind the paywall to get privileged access to the smartest minds in finance. Visit realvision.com slash RVpod and use the promo code podcast10. That's podcast10 to get 10 % off our essential membership for the first year. Join the Real Vision community and learn how to become a better investor.

1:24And now to the top analysis of today's markets.

1:36Thanks for joining us on our panel. Wait, but how do I invest using AI with Hari Krishnan and Vasant Dar? Before we get into today's conversation, I want to mention that back in 2020, you guys both joined us for a down the rabbit hole conversation on machine learning, AI, and investing. Three and a half years later, it seems like everything has changed dramatically, and AI appears to be eating the world. So we're back again today with Harry and Vasant for an update. Why don't you guys reintroduce yourself to our viewers? Sure. You know, I brought machine learning to Wall Street in 1994. I went to work at Morgan Stanley in one of the prop trading groups.

2:15And that's how I got started, you know, initially looking at volatility overlays for some of the existing strategies. And then subsequent to that, just, you know, starting SCT and, you know, just, you know, developing completely systematic machine learning based strategies. So yeah, since 94. So it's almost 30 years, believe it or not. And we should also point out, of course, that you are a professor at Stern School of Business at the Center for Data Science, where you teach on lecture on these topics. I do. I teach a course on systematic investing that I've been teaching for, you know, almost 15 years now.

2:56Really fun to teach. And I basically, you know, people call me a pracademic. And so I take sort of my real world experiences and bring them to the classroom. It's a lot of fun. Yeah, indeed it is. Harry, you, of course, are no stranger to a Real Vision audience either. You've been on with us many times. We've had a lot of conversations. You and I, of course, we should probably mention parenthetically, co-wrote a book together called Market Trends, Quantifying Structural Risks in Modern Financial Markets. You've been on many times. Talk a little bit about your journey, both in the academic sense as well as the professional sense to get to where we are right now and how you're currently using and thinking about AI.

3:37Well, I started as a tourist, frankly. When I was a grad student, I read a few papers. One was by Leo Bryman on, I think it was on bagging and random forests and a lot of machine learning techniques that hadn't been really applied at the time. I also read David Mumford's book, essay, long essay. I think it was called The Dawning of the Age of Stochasticity, which made a bold claim at the time, which was that uh you know the old school aristotelian logic would be replaced by giant statistical inference engines as models for the brain and the way the brain thinks and that was pretty profound for me but i never did anything with it until i joined the sunth and now i'm uh an accidental participant in the space so i'm i'm delighted to be on so let's talk a little bit about what you guys are doing right now, bring a little bit of context to this conversation.

4:34Vasant, talk about what you guys are doing today using AI. Well, so before I do that, let me just provide a little bit of context, right? Because it's been a long journey and I think it's important to appreciate why we're doing what we're doing today. So as you might imagine, I started off using machine learning in the way most people would use it, right? You take a bunch of data, you create features, you run algorithms on that, you come up in patterns, you backtest them, you make sure you do things carefully, scientifically, avoid overfitting, all of that good stuff, right? That's sort of standard machine learning stuff.

5:11So that's how I got started. And over the years, you know, as is sometimes the case, you sort of learn by doing right so um and over the years i realized that um if i took that approach the machine often gave me patterns that i didn't want uh and essentially what what i mean by that is that it was giving me beta and sort of alpha because beta was more stable i wanted to use machine learning to machine learning to extract alpha but most of the time it was giving me beta Why is that? The reason that happens, and this emerged sort of 20 years later, and I feel like a, you know, a bit of a, I don't know, you know, pick your favorite expression.

5:59I feel a little slow that it took me so long to realize this. But, you know, my sort of core realization was that finance is very different from perception where, you know, AI and machine learning have had huge success. And that's because it's highly noisy. There's very little signal in the problem. And what I found was, you know, over, you know, based on a bunch of simulations was that as a problem gets noisier. A standard machine learning algorithm will amplify the bias in the data sort of in proportion to the noise. So if you think about it, if a problem is completely predictable, completely deterministic, your prediction distribution will mirror the actual.

6:41Whereas if it's completely random, there's absolutely no signal in the data, then the prediction distribution shrinks to a point, namely the average. And so that's what I realized. And since finance falls closer to the sort of random end of the spectrum, that was my frustration with sort of standard machine learning algorithms. So I've spent the last, I don't know, seven, eight years sort of rectifying that and getting the machine to actually give me models that have good properties as opposed to emergent behavior that I don't want. And by good properties, I mean, you know, better convexity, right?

7:17Because that's what a hedge fund is supposed to be. Less beta and sort of more, quote unquote, alpha. And so that's where I've ended up. And so these models now that we're building have much better convexity properties. But of course, they can still be wrong. And so that's where Hari comes into the picture for sort of a more explicit engineered solution to the problem where, you know, your machine learning model may not work. Yeah, I'll just make one comment to break it down a bit. One thing that Vasant focuses on a lot is the notion that if you train a machine on slightly different data, it shouldn't make wildly different decisions.

7:59And so model stability is a major factor. And if you think of something like the S &P 500, which has been trending up most of the time for many, many years, let's say since the 1940s or whenever, if you want to create the most stable model to trade long and short in the S &P, you just buy and hold. Because no matter what data you show to the machine, it's going to make the same decision, just buy and hold. And so that's the most stable model. And so you run into a lot of issues that you wouldn't, if you don't think about the problem deeply, which are that you cannot optimize over stability, but you must take it into account.

8:39You want your machine to be repetitive in the way that it looks at things without overemphasis on a particular dataset that it's trained on. Let me ask you a question. How do you deal with that? Because if model stability is the only objective and if that's true, then basically what you get is beta and we all just take an app and don't do anything, right? So how do you try and derive alpha from a model that the longer-term data stability objective states would be essentially buy and hold? Well, Vasanthi will add to this, but model stability isn't the only thing you optimize over. You also want accuracy and you want to come up with lots of predictions because if your accuracy is 52%, let's say, you need to make lots of bets to lock in that statistical edge but i'll hand it over yeah in fact you know thanks for pointing that out uh harry that you know i think what harry was referring to earlier about stability is what i call model variance right so if you make small perturbations to the training set how much does your decision making change and i and and the important thing is decision making not performance right because if your decision making changes a lot then you shouldn't trust the model, right?

9:52So your decision making should be relatively stable. And so this gets me to sort of, you know, what have we been doing, right? So the whole sort of emphasis now that we have is to actually constrain the machine, to guide it, to look for patterns in a way that sort of give you these convexity properties, right? So essentially, so okay, let me sort of step back a second, right? So if you look at standard things like trend following models, right? The thing that is sort of wicked about them is that they have a very unpredictable holding period distribution, right? You may hold for three days, 300 days, 200 days, sort of depending on the speed of your model.

10:35You just don't know, you know, how long you're going to hold. What we're really shooting for is models that have like a sort of a predictable distribution, right and that's what i call a constraint um and so the way we sort of approach the problem now is we want the model to be unbiased right so we don't want to be just long but we also want it to be stable right and so the way we approach it is sort of through this two-step process where we say all right let's just first work on coming up with a model that has the right kind of behavior that we want because we know that we don't want trend following behavior, right?

11:15We want a behavior. And by behavior, I mean, you know, a certain holding period distribution. We then figure out sort of, you know, what sort of complexity we need. And this reminds me of Einstein's sort of maxim, which is that a model should be as, you know, complex as possible, but no more, right? And so that's what we base as simple as possible, you know, but no simpler. yes mangle that one up so that's good advice if you're working at a bank

11:49touche touche touche yeah so yeah so that's the idea right is that we we need we need some complexity for in order to get alpha right you're not going to get alpha without complexity so you need sort of some level of complexity and so that's what we do to get the right kind of behavior And then the next sort of problem is to, you know, rank these models according to some sort of criterion and then pick an ensemble at sort of the last step to minimize variance. Right. Which is what sort of ensembles are supposed to be about. Right. So that's sort of it's sort of a three stage process. First, get the behavior that you want.

12:26Second, rank them. And third, pick an ensemble to reduce variance. Right. So the net result is you want something that's unbiased but still stable. So that's the holy grail. And that's kind of what we've been working on. And the results so far are really interesting and intriguing. We get sort of behavior that where our out-of-sample performance looks like quite similar to in-sample, which is something to celebrate. We're going to take a quick break and be right back with more of the day's top analysis on the Real Vision daily briefing. Have you ever wanted to trade Bitcoin but haven't dared try?

13:02With Plus500 Futures, you can trade crypto without the hassle of opening a wallet. With just a few clicks, you can register and start practicing with their free and unlimited demo. See a trading opportunity? You'll be able to trade it in just two clicks. Feel ready? You can move to real money with as little as$100 once your account is approved. And the great thing is that in addition to crypto, Plus500 gives you access to a wide range of instruments. S &P 500, NASDAQ, gas, and much more. Explore equity indices, energy, metals, forex, and beyond. With a simple and intuitive platform, you can trade anytime, anywhere.

13:36Experience the fast, accessible futures trading you've been waiting for with Plus500. With over 20 years of experience, Plus500 is your gateway to the markets. Visit us.plus500.com to learn more. Trading in futures involves the risk of loss and is not suitable for everyone. Not all applicants will qualify. Plus500. It's trading with a plus.

14:02Yeah, I think it's our suspicion, and Vasanth would know more about this than I do, but our suspicion that machine learning methods will probably lead to long carry, short left-tail risk strategies, because those are the sorts of signals that emerge from the sea of data. And so avoiding that, where you're just loading on the same risk that everyone already has, is vital. And so making or constraining the systems in a soft way so that they're more dynamic, that they trade both sides, long and short, and they don't stick in a position too long, we think is very accretive. creative. You know, guys, in order to make this much more concrete and to bring it down to earth, without giving away the secret sauce, can you give us a simplified, stylized example of how you would actually implement this with regard to a specific model, just so we can get a sense of how the actual implementation of the strategy works?

15:05I actually thought I just, I gave away the secret sauce, but let me actually provide some more detail behind the sauce because I'm not concerned at all about giving it away the secret sauce because it takes a while to recreate it. So secret sauce specifically, right, I'll give you the details, is, you know, we basically have a method, right? We have a bunch of complexity parameters, right? We start with like really simple parameters and we say, does that give us the, does that give us this whole inquiry distribution that we want? And the answer is no, it doesn't. So you introduce a little bit of complexity, ask the same question again, iterate until you get the right behavior that you want from the complexity that you infused into the model.

15:49So that's sort of the first step is make the machine generate models that have the right behavior. And that's by sort of gradually increasing complexity. Right. The Einsteinian notion. right that'll give you thousands or tens of thousands of possible models right if you apply that method you can you can get a plethora of models the problem is that most of them won't perform right most of them will perform relatively poorly so then the question you have to ask yourself is under what conditions is it important for you to perform well right and that's sort of the standard notion of like what's the validation set in machine learning right so if it's important for you to perform well when the VIX is high.

16:35Well, that's what you'd pick as your validation set, right, to sort of do model selection. And at the last stage, you put the ensemble together, right? So there's the secret sauce, nicely tied up. And the simple-minded version, which I can give you, is that you can constrain a model to say that in such a way that you say, this model can never be long more than five days in a row. It's got to switch. And so the more days it's long in a row, the more pressure the constraint is applying to force it to go the other way. And most markets have rhythms. They tend to have winning or up streaks and down streaks.

17:16And there's a distribution you can build. And to create models which are dynamic enough to move around without reference to the prevailing trend is very significant. because that's what people want. In the short-term contrarian future space, there has been a lot of interest, at least in principle, because these strategies can be long volatility. They can be long realized vol, whereas longer-term trend following systems tend to be less so. But to build such a system in the right way is a real challenge because it's well known that short-term trends are hard to pick. A lot of them reverse. So it's very important to do things in a slightly more sophisticated way.

18:03Short-term trends are only hard to pick if you want to get it right. Well, you can pick them. Yeah, you can definitely pick them. Hey, let me ask you guys this, a foundational question. We've used the terms AI and machine learning here kind of almost interchangeably. Talk to us about the definition there and how you guys understand the distinction between the two. And we should probably also throw in deep learning as well. So AI, machine learning and deep learning. Yeah. So, you know, I view machine learning as being a subset of artificial intelligence, right, which is more broadly about, you know, understanding intelligence in general.

18:43Right. So that's so, you know, so that's the goal of artificial intelligence is to understand intelligence, whether it's human, animal. Right. So that's the sort of larger goal of AI. Machine learning is a subset of that in that we focus on learning from data, right? Whereas in AI, you can learn from anywhere. You can actually extract knowledge from humans and represent it as we used to do in the expert systems paradigm, right? That was also AI. In deep learning, okay, so one more thing. So in machine learning, what we do is we take raw data and typically you featurize it, right? You build features from the data and then you learn based on those features, right?

19:19So if you have medical data, you may say, how often has the person had X in the past? That's a feature that you actually construct from transactional data. Same thing in finance. With deep learning, essentially what you do is you try and eliminate the whole feature construction process. You take the data as is. You take an image as is. You take numbers as is. You take text as is. And you let the machine do that feature construction for you. So instead of having to engineer the features, the machine essentially does the feature construction for you on its own, right? So it's like directly raw input data to prediction, and the machine does everything, including feature engineering, and then, you know, learning the weights of the network to basically minimize overall prediction error, right?

20:11So that's how I view AI, machine learning, and deep learning. Yeah, there's an interesting side comment here, which is that if you're looking at images, which of sort of big matrices or arrays um it doesn't really on pixel it might be so outliers can be kind of smoothed away they're not significant in the financial markets outliers are very significant this is kind of the nasim taleb world and so that's one of the reasons that um we partnered because as someone who focuses on volatility for managing uncertainty in portfolios it's a good it works well in conjunction with something that aggregates, a system that aggregates large amounts of data, doesn't focus too much on the extremes, and tries to find or identify finer patterns in the data.

21:03So I think that's an interesting aside relative to image processing. And just to sort of add to that, right, I think the way to sort of think about the progression of AI is in terms of sort of these paradigm shifts, right? We started off, when I got into AI, it was all about specifying knowledge by extracting it from the human brain. Figure out how people do things and then represent it, and the machine would then follow those instructions, the rules that you specified. That sort of stalled at some point because it was too complex. Intelligence is too complex to be specified in terms of rules.

21:38It's just heterogeneous, it's subtle. right um and so the next sort of progression was well let's let's get the machine to actually learn automatically from data but we still had to do feature engineering um and then that ran into bottlenecks because how do we know we're engineering the right features right if a medical diagnostic system is looking at an image let's say of a lung you know the human in the old days had to describe the image oh there's a dark spot on the right side you know on the right periphery of the lung that's You lose information. Now you just feed the image and the machine figures out the features from the image itself.

22:13So that's the sort of progression that we've seen in AI through these paradigm shifts. We're going to take another quick break and be right back with more of the day's top analysis on the Real Vision Daily Briefing.

22:30I'd really like to just take a moment to pause here and open up this conversation to our viewers. If you have questions for Harry and Vasant, please drop them into the chat. I'd love to ask those questions from our viewers and listeners to this conversation. And by the way, if you're watching this live here, it's about 2.20 Eastern time. The Fed has held constant on rates, still 500 to 525 basis points. So now you have no excuse to switch away. You can keep watching this show and this great conversation that we're having here right now. Let me ask you this. And this is a question that we've been thinking about here at Real Vision.

23:01And the question is this. So if AI can give an edge to certain investors, does that mean it will immediately be arbitraged away due to the egalitarian nature of AI or even regulated away? If so, in a world of AI, do we need a new edge? Does it basically level the playing field or not? Yeah, great question. So let me start with the second question first about the regulation because I've been the victim of that. So I traded a high-frequency strategy for quite a few years in the early part of this century. And it worked like a charm, like double-digit sharp ratios. And then overnight, the edge disappeared, right?

23:49And that's because of Reg NMS, right? So Reg NMS came and just kind of wiped out the edge from that strategy pretty much overnight, right? So - This is the national best bid, best offer regulation that you're referring to here. Exactly. Exactly. Like basically making, you know, the entire market a fast market. So that totally wiped the edge. And so high frequency strategies are particularly susceptible to sort of these changes in market structure that might happen or changes in the microstructure that might actually happen because of regulation. So I've witnessed that firsthand. about your other question about whether the edge will get armed away that really depends right there's two ways of looking at that one is that there's a finite set of opportunities in the market right for versus gm pairs trading etc right so if you look at the world that way yes you know alpha gets discovered other people discover it you know word gets around people move around and disappears but there's another way to look at the market which is you know that patterns emerge before reasons for them become apparent, right?

24:55That the market is not stationary, it's not static. So as opportunities are being, you know, arbed away or discovered, new patterns are arising because of new instruments, new behaviors, you know, and all that kind of stuff that are coming into the picture. So this is not a static kind of game. It's like being on a treadmill, right? You have to be constantly vigilant for new sources of alpha, right? And And machine learning is great for that. I've always argued that machine learning is great as a theory building tool because it can help you uncover patterns faster than other people. But the other side of it is that you have to be vigilant about alpha really eroding as well.

25:36And that's not an easy problem, especially with strategies with sharp ratios of one or in that neighborhood. It isn't obvious that alpha has eroded right away like it is with a high frequency strategy that has a sharp ratio of 10. Harry, I saw you nodding. Jump in. I agreed with everything. There's very little for me to add on that point, so I'll sit back. Let me ask you this. As people who are watching this space, for all of the obvious reasons, I'm curious what you think, sort of more generally and colloquially, you see happening. Obviously, AI was, it's interesting. I was having this conversation with someone the other day where I said, you know, for the last 10 years, AI was whispered to be just around the corner.

26:24And then it was going to be six months later. And then another six months, another six months. And we went through this period where AI was always the technology of the future. It never seemed to happen. And then one morning we just woke up and everybody in the world, it seems simultaneously, who was an early adopter, was obsessed with chat GPT. What do you think is this sort of your general interpretation about the sort of mass market adoption of AI? How do you guys see it? How do you guys think about it? Or not at all? or are you just too focused doing what you're doing to really have a whole lot of bandwidth to allocate to those kinds of thoughts?

26:52Oh, no, I've got plenty of bandwidth to allocate to that. In fact, I've used these models. I've used pre-trained models to actually build systems that predict various things based on text and images. But I think what really got people's attention is the fact that for the first time, you could actually talk to a machine, right? And it could converse with you, right? To me, that was like a complete game changer. It doesn't matter that GPT hallucinates, right? It doesn't matter that it gives you the wrong answers sometimes. It doesn't matter that it's not, you know, great at doing math, right? All of these things will come.