LLMs for Equities Feature Forecasting at Two Sigma with Ben Wellington - #736

27:17Um, or you could take the output and, you know, do what you, embed that, right? You, Like you can do so many different, there's so many ways to interact. So I think the very simple way is, hey, LLM, give me the sentiment of this thing. And it gives one through 10. And like that, I could do that. That's very straightforward now. That wouldn't have been straightforward, obviously, years ago. But that's the thing about financial modeling is that we, it is so incredibly noisy, right? In machine learning, I think people don't fully think about the fact that in most machine learning tasks, there is an answer that you could get given the information you have.

27:55So if I had to say, hey, is this a cat or a dog? You know, you could figure out the answer. Usually, hey, you know, what should I do in this car? Should I slow down or speed up? There's usually a good answer. There might be like something crazy happens or not always. But in finance, you actually, it's not clear that no matter how much information you have, that there is enough information to predict with any real certainty what's going to happen tomorrow, right? And so we're making this prediction to the future, which means that the signal to noise ratio, the noise is just so much bigger, right?

28:31And so if we can predict an R square, if you see an R squared of like 0.05, you know, something's broken because that's way too high. Like, you know, like there's no, these things that you're trying to predict are very small signals, right? the nose touching might be real, but it's not gonna be predicting most of that stock price. It's gonna be a tiny, tiny, tiny, tiny, tiny portion. And so, yeah, that's one of the challenges of the financial markets. And also - And so from that perspective, the signals that you're operating with are so noisy. The fact that the LLM, the fact that the difference between a five and a six and a seven sentiment is kind of random, doesn't really matter.

29:15Maybe, right, yeah. Or it matters very, very little in a way that's hard to leverage. You know, it's hard to tell. And so are these, we talked a little bit about the hierarchical nature of these signals. I'm curious about how you put that all together. Like I'm envisioning, you know, at a certain point, And, you know, particularly if you've been doing this for 17 years, like you've collected a crap ton of data, like you're trying to update, you know, a large portfolio of features across that raw data. You're trying to enforce some degree of consistency across that. Like I start to think of like, you know, what platform, you know, what the platform looks like and what the tooling looks like and that kind of stuff to do that at scale.

30:08It's funny, so one thing about Two Sigma is that we are a, we call ourselves a platform company because ultimately, you know, it's a very tech heavy investment company. And we have a lot of different teams looking at a lot of different types of data, but they are doing that together into sort of a centralized set of portfolios. some other financial companies have these competing pods that sort of fight each other, you know, to try to have the best performance. And we're not like that. We're all centralized. So we have a shared, we have a shared platform that, that kind of, we all can feed on. And so what does that look like?

30:50We try to break up the investment process into abstractions that have, you know, good APIs, right? So I talked about this model is something that predicts something, but in terms of a platform, right? What does that mean? It means I have a set of inputs, right? Which are going to be timestamped information. And the API, well, my job is to, you know, send out a prediction for every time cycle, what I think is going to happen to everything that I'm looking at, right? And I'm going to do that in a loop. And so if someone, if you provide a modeler with access to data and this API on the output of what they can predict, then their job is to connect the raw data or the features that they've read in to that prediction.

31:32And what happens in between those two points is sort of the magic of that particular model or the hypothesis or the machine learning or whatever's happening. Obviously, you need a lot of storage. You need good search. And a lot of tools to run these back tests are very complicated. If you're trying to predict what would happen had I made this decision over the last 15 years on the market, there's a lot of technology that goes into replaying the set of stocks and what would have, you know, creating a virtual world where you're backtesting this new idea and see how it would have done. So that's all kind of built into the platform where someone likes me, I can just call, okay, here's my forecast.

32:13Go tell me how I did. And I can put that on the, I can put that on the platform and it can come back with the measurements for me without me having to think about how or what's happening. So those abstractions are really helpful as, because, you know, the process of turning raw data into a portfolio has many, many steps. And, you know, my team's responsibility is turning raw data to features, then features to just predictions. And after the predictions, another team is thinking about how to make all those predictions, combine them, and build a portfolio for our company that's going to maximize the returns for our investors.

32:49And so the distinction there is that a prediction that your team is making is whether, well, are you predicting the future of the future value of features or are you predicting the future of an equity price based on features? It sounds like there's a line somewhere between one of those things and predicting, you know, the value of the portfolio given a decision to buy or not. So we're predicting the future price of, if we're doing equities, which is where I tend to focus where I focus and where I work. We are predicting the future price of equities that we trade. Based on a feature set. Based on everything you have.

33:38Based on everything we have. Yeah. And so our role is to build a bunch of different models. One model might look at CEO interviews. One might look at fundamental prices. One might look at what an analyst is doing. And so you end up with literally thousands of these models all interacting and they all have their own view of the world. And the idea is, you know, to keep your signal working, they're hopefully orthogonal enough to one another that when one's not working, you know, and so that's how the company works overall. But the thing is, once you have all these predictions, you can say, all right, we overall, we think that this stock price is going up and this stock price going up.

34:14But how do you turn that into a portfolio that doesn't have certain risks that you're not to expose to a particular industry or you're not, you know, so you have to keep everything neutral so that if something happens in the market, we can stay balanced. And so, you know, that's not the part that I, that's called optimization. It's basically converting the predictions into a portfolio that's been optimized to be kind of balanced well. And so that's like its own research area, their own set of researchers, their own set of technology that then decides what optimal portfolio we want to be in at that particular time.

34:50I think chasing down a, the signal that I'm latching onto and chasing down here is you kind of anthropomorphize the idea, are these models having their own point of view or something along those lines, as you said. It made me think of some kind of agentic extension of that where the models are arguing with one another about the direction that they think something is going. Is that a thing that you're thinking about? I mean, if I wasn't, it would be weird, right? I mean, you're hitting the nail on the head. I mean, we have these different views of the world that have been thoughtfully created by people with hypotheses, with views, encapsulating from a set of features.

35:37You know, some might be very interpretable and simple. Some might be more complex in a black box machine learning way. You know, we always have humans on the end to monitor everything. But ultimately, we have all these different models with all these different approaches and techniques. And yes, do I think a lot about what an agentic approach to that looks like? Like, absolutely, I do. It's terrifying and exciting all at once. But what's your best guess today at what an agentic approach to all that looks like or means even? My best guess is that you have different agents that have, I mean, this is sort of the point of these agents, right?

36:12They have different understanding. They're experts on different things, right? One is in depth, understands currencies and one understands what the Fed is saying. And, you know, I think ultimately their ability to communicate with each other and say, all right, well, I know that this company has, you know, I know that interest rates are important for this company. Let me go ask the interest rate expert what's going to happen. And then I'm going to go ask the, you know, the weather person what's happening tomorrow. So I, and that's, this world of these, these, these, these agents making guesses about the things that they're experts on.

36:44And yeah, I don't exactly know what it's going to look like, but definitely an area where we're experimenting in because it clearly, look, any of our roles feel like they're heading towards automation, right? Many, many, many, and I don't think just because it's research or innovative, it doesn't mean that a good reasoning LLM can't start to fill in some of those gaps. And I find it, yeah, I find it pretty fascinating, but also exciting because, hey, can I have 10 ,000 researchers at my fingertips? That would be pretty powerful. And in some ways, it feels like that's the way we're going and faster than I would have guessed.

37:26One thing I'll say is that what I've learned is that innovation cycles seem to be happening pretty quickly at an increased pace. You know, it is business as usual. NLP has always gotten more advanced over the years, but it feels like the cycle is speeding up. And so you need to be ready with a platform that can adapt nimbly to the latest technology, right? So if you're going to go spend a year training up your own thing and then you come out for air and the technology you use that you started a year ago is now obsoleted, you know, that's not going to work. So you really need to be front footed and look at what's happening in the world and build on top of the ability to take in modern open source approaches or work with vendors and tools that are adapting very quickly.

38:14And your platform needs to be able to jump on and off different things at different places to stay up with this new innovation cycle because there's no time to stop and build something in this space that's going to be able to keep up with what's happening. And so that's kind of fascinating as well. So interesting. Yeah, you mentioned open source and you kind of started to allude to like how you think about build versus buy. I'm sure it's an oversimplification to say that things are moving too fast for you to build anything. Right, that's a oversimplification. I think the things that you build need to be plug and playable with the changing world.

38:54Can you give me a concrete example of navigating that? While there's a lot of interesting things you can do with, say, fine tuning in LLM or something like that, you want to be in a position so that you can swap in the next model, wherever it comes from, however it comes from, very quickly. So that the rest of your stack is not like, oh wait, it's a new model. It's from a new company. Everything now has to be redone, right? So you have to design your inputs and outputs so that you can wrap a new model around something such that it's easily ingestible into your system as opposed to picking something and then customizing on top of that one thing.

39:39Yeah, I'm hearing kind of like the minimum to build is like owning the abstraction. Right, right. So that you can swap things out and then maybe you're selectively, I mean, I've got to imagine that you're, you know, well down the road of fine tuning your own LMs for in areas that it makes sense. Yes? Yeah, yeah. Which, yeah, obviously that's an important aspect of it. But even with fine tuning, the things that you had to fine tune six months ago, you know, the next model somehow is already, you know, risks being better at the thing you fine tune your six month old model, right? Like, I mean, it's moving that quickly.

40:19And so it's kind of wild. You'd think that the level of depth that a generic LLM has on, say, financial accounting statements, I mean, it's incredible. It's in any field, right? We've all seen that. But so maybe I had to fine tune last year's model, but suddenly this year's model seems to have all that ability. So that doesn't mean that you're not fine tuning. It's just that the assumptions that you have about what you need to do might be invalid six months down the line as the technology starts to speed up, is my point. With that in mind, how do you think about where you make those build investments?

41:02like, and I'm specifically trying to get at, you know, it would be easy to say, like to try to project, well, you know, the LLM is going to be able to do this for us. And in fact, it's not just easy, like it's necessary in a lot of cases to say, yeah, I'm not going to, I'm going to force myself not to go down that rabbit hole because, you know, by the time we're done with X, the LLM is going to be able to do that, you know, But sometimes you sit around waiting and it doesn't actually fill that gap for you. How do you think about riding that edge? I mean, it has to do with your priors on how important a particular build is, like how fast you need to get it out.

41:42And if you think that you're in a position where you have an edge to utilize that novel thing you're doing, and especially in an industry like finance and quantitative finance where you're literally competing for prices. Like, you know, it's someone else's machine learning is against your machine learning and it's a zero-sum game, right? So, you know, time matters. If you're six months out the gate before the other group, that means that that can make a big difference because what you find, by the way, is when you trace signals over time, they get weaker and weaker and weaker as your competition starts to come in.

42:19And so it's called alpha. It's kind of efficient market. Yeah, it's an efficient market, right? Any innovation you think you have, well, now everyone has access to data X or Y. And so it slowly decays away over time. So, you know, there is a time component and you have to, it comes back to that return on investment question and your priors as a researcher of how important that particular thing is to get now versus later. You know, when I think about this, again, this kind of hierarchical, you've got these base level features, you're building higher level features. It made me think a bit of kind of the old ideas, quote unquote, old ideas around like AutoML, like a data robot thing, like I'm going to just take all these features and run some statistical programs across them to like combine them in different ways.

43:04Like, are you doing a lot of that kind of stuff? I mean, it's hard not to, right? You're staring at this pile of data. I think it would be, you know, I wouldn't be doing my job if I haven't explored what happens when I use them all. So, yeah, it's a complicated question because the risks of key hacking and overfitting, you know, when you start to take everything at the wall, throw it and see what sticks, it opens up new scientific rigor questions. And we live or die off of, you know, whether we overfit or not, right? If you make that mistake and your model doesn't trade, that's not going to go well for a modeler.

43:40You can have, you know. So we have to be very careful. But if you do it in a rigorous way and careful way, then yeah, I think there's a lot of exciting work to be done in automation end to end. And it's definitely a focus area that we think a lot about and is growing as technology grows. Yeah. In a lot of ways, it touches on a question that comes up in lots of different spaces around like modular versus end to end. So, for example, I've had this conversation with a number of guests in, you know, robotics, autonomous vehicles where like they're, you know, the industry has priors of like you've got, you know, a sensing system, a control system.

44:27you know, if you've got these subsystems and you want to wire them together in ways that make sense. But then there are, you know, that's kind of one pole. And the other pole is, well, yeah, let's just take off the data, feed it into a model. And, you know, if we have enough data at some point, you know, the model should be able to do it. Like, is that a tension that you have to? I mean, yeah, absolutely. It reminds me, again, back to my days in the PhD program where there was this sort of syntax versus enough data view. And there used to be this, competition for who can make the best, you know, machine translation system.

45:03And there were all these fancy algorithms. And one day Google entered and they had, you know, the five-year-old algorithm, but they had this massive model, like 10 times bigger, a hundred times bigger than anyone ever had. And they beat everybody. And everyone was like, wait, like we have all these fancy ideas. Yeah, but we have a lot of data. And it kind of changed, it changed the world. And so I think with enough data, you can start to do really interesting things. And so, yes, even, even I described to you these abstractions, I'm in modeling and my job is to turn data into forecast and someone else does that, you know, what is the word look like when you just hook raw data up to, you know, a portfolio and you, you, you start to take away those abstractions.

45:45What, what new things? And there's definitely a tension there. I think that if you, if you, if you force yourself into a set of chained events, you might lose a lot of the power of machine learning because at each handoff point, you've stripped things down into your assumption of what matters. And that algorithm might disagree, right? I mean, in fact, machine translation is a great example of that. It used to be voice to text, text to text, text to voice, right? So if I wanted to go from English to Spanish, it would go English, you know, I would say it, it would turn to text, then they would use a backend model, then they would generate.

46:21Now it's just, you know, they don't, that's not how these things work anymore. They've hooked the language. They know how to convert it from audio to audio. It's not running any text in the background. It's an exact example where if somebody had forced that hop, you would actually have a less good system than we have today when they said, oh, look, let me remove these abstractions that humans have added and just let the system go with enough data. And it turns out that can work incredibly well. Same time, I'm thinking of a conversation I had with Scott Stevenson, who founded a company called DeepGram that's in like the speech-to-text, text-to-speech base.

46:55And one of the things that he observed is that, you know, while skipping, while staying in the audio domain can be really powerful, sometimes jumping into the text domain is useful because as us humans, we can't look at the spectrogram and like intuit anything about it. But like text is useful for debugging. for understanding the system. Are there analogies in your space along those lines? Yeah, I think interpretability is an interesting thing. And as you go more end to end, you lose more of that visibility into the world, right? And I get asked a lot about how comfortable are we with black boxes in investing, right?

47:40These are big decisions we're making. And what I like to say to people is, well, have you ever gotten under general anesthetic? So yeah, I mean, when I got my wisdom teeth out or something, I said, okay, did you know that, you know, modern science doesn't actually know how that works? We've just tested the inputs, we've tested the outputs, and we've grown comfortable with enough data that general anesthetic has particular properties. And we, in medicine, they're comfortable enough to utilize it by just having very careful measurement on the input and output. And so I see, I love interpretability.

48:11I think I'd rather, by the way, have a medicine that people understand. Don't get me wrong. I think that's a more comforting thing and you'd have higher priors. But if you study something enough, that interpretability maybe isn't as important and maybe our ability to understand, it might not even, even if it could be explained, it might be that the thought is so complex that it would be hard to explain even in the most perfect world. And so I tend to prefer interpretability. I have higher priors when I understand what's happening and it's all about being equal, like Occam's razor, simple is better.

48:46But that doesn't mean that we should be so naive that we can turn away anything that we don't understand. At the risk of bouncing all over the place here, I remember you mentioned open source and kind of going back to that platform conversation. I'm curious, can you articulate like what, you know, like, are you using one of everything or are there specific, do you have strong feelings around like, you know, O3 is, you know, the state of the art for us and we're doing everything with XYZ or we've hard pivoted to Llama 3 and use it for everything like, or, you know. Yeah. So, you know, one of the interesting things about these models is, remember how I told you that we're terrified of temporal leakage, you know, timestamps matter.

49:38so it is kind of scary for us to use a a you know off-the-shelf model that's been trained on data in 2020 to ask questions from a document of 2019 right so if i could say hey here's this enron conference call you think it's good or bad is the word enron going to trigger a negative reaction because you know somewhere deep in the psyche of lm and knows there was a big bankruptcy and so you have this sort of sneaky look-ahead bias that could kind of type in. So, you know, in the build versus buy world, there are risks to using these off-the-shelf things because you can't control the data that's going in and you might fool yourself.

50:18And so that's one reason why we do prefer open source more controlled experiments because we know what went in and we know that what we're seeing we can actually believe in. In terms of the best model, by the time I answer the question, it would probably be different, right? So I guess my quick answer is no, we're not, it's not the kind of thing where, I mean, look, I have opinions, this is working out better than that at this task and that task. But nothing that, it's clear that this is going to be a iterative process where different, there's going to be different advantages to different models are going to be hopping over each other, it's going to be happening rapidly.

50:56And there's not going to be a horse to bet on, you're going to be well suited to have a diversified set of inputs to, you know, build interesting and orthogonal outputs. And you need to be comfortable using a wide array of technologies, not just kind of betting on a single one. But can you say something to the effect of like, given your preference towards open source and understanding the, you know, training environment, data recipe, whatever of a model, you know, you've been able to shift, you know, you know, some significant percentage of your workloads to open source models or, you know, conversely, you know, the model yet another feature in this basket of features and you train all of your data on all of the models and it's just, you know, one level higher that you go to make sense of things.

51:46Yeah. I mean, I kind of, if I could stop my fingers and train all my data on all my models, I think I might. I mean, it's nice to have it all there, right? Then I wouldn't have to run any experiments. So I love that concept. And I think Like, actually, a colleague of mine last week was framing it basically as combinatorial research, right? Yeah, exactly. Right. The ultimate scary place to go and not get buried. And so, you know, I think there's real truth to that. I do think that when you're making a prediction about what's going to happen, much like we have different models that combine to make Two Sigma's forecast about what's happening next, right?

52:24the model looking at this data, that data, so too is their value in having an ensemble of LLMs that each have their own strengths and weaknesses. And it's not, I'm not always, I'm not looking at the best at things. I'm looking for, you know, a group of things that each have their own take that when I average out among them, I'm better off and more robust in the future than had I just picked one. So yeah, I'm not, I really am of the belief that there are pros and cons And that in itself is a strength because it builds orthogonality, which when you combine orthogonal signals, you get a much smoother response than when they're correlated signals.

53:05And you kind of said something that alluded to, you know, just like, you know, all your data is timestamped, you know, maybe all of your models are timestamped and you have like you know, a model with a 2019 perspective or a model with a 2020 perspective, you know, etc. Are you, you know, and again, speaking of combinatorics, like that explodes really quickly. It's yet another dimension. It's yet just, yes, another dimension to add to the factorial. Yeah, it's very, it's, you're exactly right. And so, but it sounds like you are doing some of that. Yeah. I mean, ultimately, look, we've always done that, right?

53:47So if I'm training even a linear model, there's different ways you could do it. If I'm trying to predict the nose touching as a predictor of a company's performance, right? I could, if I take my entire sample and I calculate some beta between the number of times they touch in the company performance, then I guarantee, I just overfit my data and it will work on my, it won't work in the future, but it'll certainly work on the history, right? So what's a better way to do it? The better way to do it is to kind of think about this temporally and say, okay, well, if I had the data from 2000, 2010, what would I have done in 2011?

54:29And, and then, you know, so we, those, those are some small changes you can make to, to avoid. Now you're still fitting because you still have to be careful. It doesn't solve all your problems, but, but yeah, we've always lived in a world where, where you have to be very careful to not get future data in, in your experiment. And the thing about complex machine learning is if you happen to give it a leak into the future, it will find it, you know, like a linear regression might not find it, but a, a neural network that sees, oh, wait a minute, I already know this. you know, it's going to pull whatever weird hole you might have accidentally given it and pull hard and, you know, and be successful.

55:05So I think time is so important. So what measures do you take to prevent that kind of data leakage? We're pretty strict about timestamping our data at the company. And so I say all data is a time series. People say, what are you talking about? I'm like, well, every piece of data has a timestamp on it. Well, what do you mean? You know, I mean, yeah, tell me, well, you know, and if you think about it, all data came to exist at some point. And so you can inherently add a timestamp to almost any data. And if you do that, and if you're careful to make that data, that timestamp, the creation date of that data, then you can easily query, hey, I want to join my information, but only backwards in time.

55:48And so if you constantly are only looking backwards in time, then you kind of save yourself from that leakage. Now there are risks, because vendors don't always, you know, abide by the same rules. For example, let's say a company wants to track the S &P 500 stocks and they're going to give me, you know, the website job postings of S &P 500 and they come and they go, oh, I have all this data on the S &P 500. So they give me a 10-year dump. But the problem is if they give me a 10-year dump of the S &P 500 today, and then they give me 10 years of history, that has look-ahead bias because 10 years ago, we didn't know who the S &P 500 would be in 10 years.

56:26All I have to do is buy all those companies for all 10 years and I will make a lot of money. But I think that I want to do every time someone hire someone just buy stock and I would make so much money. Right. But it wouldn't be real. I wouldn't generalize to the future. Now, the vendor is not doing this because they're trying to trick you. They just don't think temporally in that same way. And so, you know, you do have to be very vigilant in the timestamps people put in and to make sure that that leakage doesn't come because again, in the world of modern machine learning, that leakage can be, if you think that you have this model that's so performant and you're being tricked, then, you know, it's not going to, that's not going to show up in, in its live trading.

57:03And then you're, you're going to notice that and eventually turn that off, but it's going to be a waste of your time. So thinking about forecasting and predicting the future, what do you predict for the space? And what are you excited about, you know, coming down the road? I got to say, I think your, your question on the agentic stuff is quite exciting. It's hard to, for all of us in many fields, to wrap our head around what's gonna happen in the next six months to a year to two years, or five years down the road.

57:35I'm excited about the increase in efficiency that brings and the ability to convert ideas into signals at a faster and faster pace because, you know, then you can just take your most creative people and create data out of them, you know, at such rapid speed. And that's really exciting. It opens up so many different avenues. And of course, it opens up challenges, right? As these tools, as the LLMs get stronger, if they're able to code as good as, you know, your junior engineers or able to do research as good as your, you know, what does that mean for the future of a company like ours and who we hire and training and things like that?

58:16So I think that's the challenge, but I think that the upside outweighs some of those challenges in that the opening up of data is just so profound and exciting that that keeps me excited to come to work every day. Well, Ben, thanks so much for taking the time to jump on and share a bit about what you're up to and what you've learned. Yeah, I enjoyed it. Thanks for the chat today. Yeah, thank you.

58:48Thank you.