BloombergGPT - an LLM for Finance with David Rosenberg - #639

0:07All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington, and today I'm joined by David Rosenberg. David is head of the machine learning strategy team in the office of the CTO at Bloomberg. Before we get into today's conversation, be sure to take a moment to head over to Spotify, Apple Podcasts, or your listening platform of choice. And if you enjoy the show, please leave us your best rating and review. David, welcome to the podcast. Thank you. Great to be back. It is amazing to think that it's just over, what, five years ago that you were on the show?

0:42Yeah, it's been a while. The world has changed in the AI scene. Absolutely. You were just mentioning that at that time we were doing just audio-only shows. Now we're doing this, of course, in audio and video. It can be seen on YouTube. But I think that's probably the least of the changes that we've seen, especially in this NLP space. What do you think? Yeah, I'll say. I mean, I think for me, when GPT-3 came out, it was about three years ago, that was a moment of, wow, this is different. This is something that seems really new. The capabilities were really impressive. Absolutely. So in that vein of GPT, today we're going to be talking about your project, which is Bloomberg GPT, and how you trained and built large language models specializing in financial language.

1:30Before we jump into that, share a little bit about your background just to refresh those who weren't listening in five years ago. Five years ago, right. See, academically, I studied math in undergrad. In college, I was already interested. I'd heard about neural networks. I wanted to do a project on neural networks. Just a funny anecdote. My stochastic processes instructor said, this is in 2000, don't do neural networks. Those are kind of passe. Do these Bayesian networks. So I took a detour into probabilistic modeling for quite some time. Anyway, I went to grad school in statistics, focusing in machine learning, and moved out to New York City, where I was at a startup for many years and eventually joined Bloomberg, where I'm in the CTO office working on machine learning strategy, heading the machine learning strategy team.

2:14Awesome. Tell us a little bit about the background of Bloomberg GPT. I mean, in one sense, it seems like an obvious thing to try to build, but what was the need that drove you to try to create that? It's not necessarily so obvious to build it. So that's kind of one of the things that we do in the CTO office is make these strategic decisions on what to invest in, you know, with both money and time and people's resources, in particular on new machine learning technology. That's what our group would be focusing on. So as I mentioned, I guess it was mid 2020 when GPT-3 came out. And the question was, is this a direction we pursue, we invest in?

2:54Because it was clearly a big investment, right? Everyone knew how much GPT-3, but we didn't know how much GPT-3 actually cost to make, but it was clear that it was a huge investment. And we decided that it was worth making the move. Kind of, maybe there's some risk there, but it seems like the possibilities were pretty great. So that was kind of a decision made back in late 2020 to start building towards this goal of our own GPT-3 style model. And I'm not sure we knew exactly at that time what it would be used for. We're still experimenting to figure out how best to use it for our purposes. Can you provide an overview of it and its capabilities?

3:34Sure. Well, in some ways, it's a general purpose model. It's also, we see kind of purpose built for finance applications. So our training data set was a split between the standard general purpose training data that are used for building, we don't know exactly what chat GPT models use, but things like OPT and Bloom, we know the data sets that they used. And we kind of followed a lot of their training data plan for about half of our data set. And then we had another half of the data set that is based on Bloomberg's data collected over many years, starts in 2007. That's called FinPile. So about half of our training data was finance-specific Bloomberg curated data.

4:16And digging into that a bit more, are we talking about kind of financial reports or articles? Or I think I remember the last time we spoke, that conversation was focused on kind of extracting information from semi-structured documents. You know, did some of that come into play? What was the makeup of FinPile? Very much a mixture of all the things you mentioned. So a big chunk of it is news data that we have, like news that we have acquired. It's financial filings, that's like company filings, press releases, transcripts of various calls that we have. It's kind of a broad collection of financial data.

4:59Got it. Some of those documents have tables and charts in them. So we didn't do any new additional processing specifically for this trained data set. But when that information was already extracted, we used it. Is there anything to be done kind of taking advantage of the semi-structured nature of a lot of that information? You've got a lot of metadata in addition to raw text. Right. So we didn't do anything specifically different for the for structured data. It was tokenized the same way as everything else. But one thing we did have some concern about or some focus on was numerical data because finance data has a lot of numbers in it.

5:39And we were concerned with the way that... I guess a lot of previous work used the GPT-2 tokenizer, which doesn't treat numbers in any special way. So you'll get tokens, long number like 5 ,234 may tokenize into 523 and then 4, or it may tokenize into 52 and then 34. And we're concerned that that makes it, it feels like a harder problem for the model to solve when it doesn't have any kind of consistent representation of numbers. So we took a stab at it following kind of the palm model also did this, where we broke up numbers into individual digits. So maybe the model could then figure out the idea of first digit is the highest order and the second digit is the second highest order and so on.

6:22So that's kind of one thing we did to accommodate this financial data, having a lot of numbers in it. So talk a little bit about the process of training the model. It's for the size that you achieved. There are a few out there who have attempted that, and there are a lot of, I'm imagining there was a lot of kind of new ground to explore. Is that the case? Yes. I think we're still at the stage where it's always kind of an adventure training one of these very large models. When we started our process, there was OPT model for Meta, and they did a very nice job of sharing their kind of training logs.

7:06All the challenges that they went through, they were very detailed describing them and what they did to resolve them. And so we kind of had that as a bit of a roadmap. And then Bloom also published, the Bloom model from Big Science, Hugging Face, also published their training logs, training chronicles. That was about it as far as detailed information on how to handle the types of issues that come up when you're actually doing the training. Kind of because of that, we want to control our risk by making the model as close as we could to something that we knew worked before. And so in our case, we copied the blue model architecture fairly closely with some small tweaks, the tokenizer being one of them.

7:46I mentioned the way we handled numbers being a key piece of that. One or two other spots where we changed, I could talk about that later as well. So the training process starts and we call it V0 because there are subsequent versions. and one innovation or thing we tried in the first version of the model was something that seemed innocuous enough which was what we noted that in our data in the bloomberg data set we call finpile all the documents are timestamped and the data further back is not as good quality for whatever reason there's any less of it and moreover it seemed like the most recent data would have the most correct and accurate data, factually speaking.

8:28And so it seemed reasonable to do something you could call it a curriculum learning or whatever it is. It's not a purely random mix shuffle of the training data. We decided to order the training data kind of sequentially in time for the part that's finpile for the Bloomberg piece. The other data, which is about half of it, was randomly shuffled. So we start training. And as for our validation data to measure progress, we are using kind of the month following all that training data. So the most recent data would be our validation data. So we're training for four or five days, getting started. And we notice like the training performance curve levels off, the validation performance curve levels off, or at least it slows down a lot.

9:09We're looking at it. Is it actually still learning or has it stopped learning? It's hard to tell. And after about eight or 10 days, we decided to stop that training. And we were worried that maybe the curriculum learning line Sorting by timestamp of the document was fine, but didn't seem to be working. And it seemed like that might have been a step too far into the unknown. And so we started over, got rid of that curriculum learning. So we randomly shuffled all the data. And that was kind of the beginning of version one of training. Started off better for about, I think, eight days or so. Nice improvement of the training performance and validation performance.

9:49We also were tracking the norm of the gradient. which is a spot where you often see issues. So the gradient norm will suddenly spike up or gradually increase. And we're looking out for that. And sure enough, after something like eight days, we see the gradient norm almost angle upwards and kind of start linearly climbing and couldn't figure out what it was from. And unfortunately, it also coincided with that validation performance kind of getting worse and spiking. So that's when we turn to the limited amount of roadmap we have from OPT paper, who also encountered these kind of gradient norm spikes.

10:29We started, not only was there a gradual increase in the gradient norm, there were also these spikes in gradient norm, which corresponded with poor performance. And they had all kind of this recipe they used to get past these spikes, which was kind of roll back to a checkpoint, maybe 100 steps before the spike. You reshuffle your training data. maybe lower your learning rate. That's kind of the recipe. We tried that and didn't seem to help that much in the sense that we weren't getting kind of nice, compelling learning curves that are like improving performance gradually, gradually. We dug into the weights.

11:05With these gradient arm spikes, what's happening to the weights? We found something really interesting, which to this day, don't really know the source of this. It's kind of an open question. There's 70 layers in our network. And the only thing that seemed off was in the very first layer, the very first thing in the first layer is this layer norm. And the scale weights of that layer norm, right when the gradient norm started increasing, those parameters went from decreasing to steadily increasing. It's very strange. It's like all the weights were more or less throughout the model, more or less kind of on average staying the same magnitude.

11:41And these layer norm scales were going down and then up. So the down piece we think was a bit of a bug in how we're using our optimizer we we're applying a weight decay which pulls the weights towards zero to scale weights which should be centered around one so that was a little bug that we fixed in version two did kind of a thorough call through our code and we found a few other things like involving mixed precision for whatever reason the bloom model either didn't need to or just didn't use kind of the full 32-bit precision in certain parts of the model where often one does use them because the lower precision weights are kind of not precise enough.

12:22We put that back in. We decided to put an extra layer norm at the very beginning before the first layer is kind of additional protection. And we started over again, this time with the shuffle data. And that was version two. This was finally the version of the model that worked. And we ran for 42 days with nice steady decrease before we ran into some challenges, which was basically that the model stopped learning after about 75 % of our data set. And we tried a bunch of stuff, but really we're kind of towards the end of our budget and nothing seemed to be working compellingly. And so we kind of called it and said, this is performance and downstream task is already good, more than met our expectations and hopes.

13:04We still have a chunk of training data to apply to later when we have some more time and ideas. And yeah, that was the training experience. Oh, wow. So 42 days was around the total time of training? 42 days until we started kind of leveling off and then we ran it for another week or so. Okay. Can you talk a little bit about the team that was involved in pulling this together? You mentioned budget. I'm curious about that. I'm curious about infrastructure, what you use there. Talk through some of those things. Sure. So the team was ultimately about nine people. Some were hands down coding. About four were doing kind of actual implementation work.

13:45Three of those were focused on the machine learning data aspects. One was focused on kind of optimization and compute aspects. The rest of us were in some sense advisory, but it was culling the literature for things that might help fix the models and working on evaluation and those sorts of things. So compared to a lot of these large models, we had a relatively small team. What else? As far as hardware infrastructure, we were training on Amazon SageMaker. We used their SMP platform for optimization. On NVIDIA, we were using 40 gigabyte A100 GPUs. We had 512 of them. What else? Software PyTorch.

14:24Our model was kind of small tweaks from the Bloom model, source code, which I mentioned. And can you talk about cost? So I don't know exactly. The way it worked from a big picture is that we bought in advance some number of hours on these GPUs. I think it was like 1.3 million GPU hours. I don't know offhand. And I think you could find the rack rate for that. I think we had some negotiated rate for a certain amount of hours in a certain period of time. I'm not sure exactly. Okay. So you've got this model, you've trained it up. You mentioned you had some folks working on validation. Talk through validation and how you evaluate the performance of the model.

15:08Right. So there's two pieces of that. There's kind of evaluation while we're training and then evaluation after the fact. The while we were training evaluation evolved a bit over time. It started with just this validation set from the last month of training data, which I mentioned earlier. As we were going, we added in another validation set that was chosen randomly from the training period, which is a little bit more comforting because that would have the same distribution as the training distribution, whereas this last month may be different. So it's a little bit harder to interpret. As we got further into it, we had actual downstream tasks that we'd evaluate on.

15:43There's MMLU, which is kind of like a giant multiple choice question task covering knowledge from like a wide range of areas, pretty hard questions. And then there's BBH, Big Bench Hard, which we took kind of the multiple choice questions from. Those are the things that we were monitoring while we were training. Afterwards, we did kind of a much more thorough analysis of performance. We had, I guess, one category is internal and external tasks. And another is rather like kind of Bloomberg specific. These are our tasks for our downstream needs. and then the general publicly available tasks. And within those, they're kind of financial specific and generic ones.

16:20So kind of our, we consider a cohort of models to compare to, we like that OPT, the 66 billion parameter model, ours is 50. So that was kind of the closest of the OPT family and the Bloom model and the GPT NeoX model from Eleuther. They're open source. Those are ones that we can run on anything. And then other models will publish in their papers, their performance on some benchmarks. So when we could, we pulled in kind of reported numbers. So that's how we got the GPT-3 numbers we compared to in the paper, the POM numbers we compared to. And the results were really good. We were kind of at least competitive with Bloom and OPT on most tasks and probably were better more often than not.

17:02But certainly as far as general purpose tasks, we were in the same ballpark as the others in this cohort. And on financial benchmarks? Financial data, we were significantly better. And one possible reason, there's probably two reasons, and I don't know how to allocate to each, but one is the data we trained on. It was half of us financially sourced. And perhaps the tokenizer approach helped. It's hard to know. We didn't do the ablation studies to determine that. As far as what we would do next, that's one thing that we'd focus on, I think, is more experimentation on these small changes, maybe on a smaller model size, to know what makes a difference and what doesn't.

17:42And did you already have or did you need to create a kind of a financial benchmark for evaluation? The internal. So things external exist. One that I like is a con for FinQA. It's kind of like reading comprehension for financial documents. There's text, there's at least one table of financial information, and then some question that requires some amount of numerical reasoning. So that exists, that we performed very well. And that's one where the gap between our model and other models was maybe most impressive. So we do a lot of sentiment analysis on various sources. So there's news and social media and kind of company statements.

18:25And so we have kind of internal benchmark data sets for that sort of thing for our own internal tasks. Another interesting thing that we'll do is NED, Named Entity Disambiguation. So you're reading a news article. It mentions Apple being able to map that to our knowledge base where Apple is a well-defined company. So, for example, you can map it to its stock ticker or something. So this would be a task that would be kind of finance specific. And that was one of our internal benchmarks that we did well on, that we're pleased with our performance on that. Also, this is not so much a formal benchmark because it's more generative and it is more challenging to evaluate these things.

19:10But internal Bloomberg, there's something called a, it's called BQL, Bloomberg Query Language. You can think of it as an SQL, but maybe much more complicated or capable for combining and operating on all the data internal to Bloomberg, a large piece of that data. It's difficult to use. So could it translate a natural language request into this Bloomberg query language? Now, it wasn't trained on Bloomberg query language at all. Yeah. And so we tried this in kind of a few-shot setting where we gave some examples. So it would see B2L in the prompt. And it did rather well. It did rather impressively that it kind of could pick up the sense of this language from 10 shots or something.

19:51We also played around with news headline generation. Given an article, can you come up with a good headline automatically? So these are kind of things we were experimenting with in a less formal basis because they're just harder to evaluate numerically. Got it. What's the context size that the model is? 2048. Okay. tokens, 2048 tokens. Yep. I'm curious, maybe stepping back a little bit, given how quickly the space is evolving, like what's your, your meta take on, like, is it worth doing an effort like this? You mentioned you spent in excess of a million dollars models, the kind of the baseline against Bloom, the, you know, your baseline, if you were to choose to do this today would probably be different and your, your comps would be different.

20:37Like, was it worth it from a learning exercise? Was it worth it? As a practical, hey, we can use this or we can get a year of use out of this thing and it's better than what we had as an alternative. Who would you recommend take this on? What's your feeling on all that? Yeah, great questions. One question that we wrestled with a lot internally was, should we build from scratch or take an existing model and fine tune it? It turns out there actually aren't that many open source models that you can take the weights, fine tune for commercial use of the scale that we're considering. I think we just saw the first of those last week in the Falcon model, right?

21:19Yeah. So that's an exciting contribution. I have not looked into the details of that. So when we started, there just weren't a lot of options. Presumably more will come up. And so the question will become more and more relevant. Do we fine tune? And I think it's unclear still, you know in our case for example what would we be fine-tuning on we have like 350 billion tokens in finpile that's like a full data set is it as effective to kind of take an existing model and then fine-tune it on essentially i guess you could almost call it domain adaptation where it's it's not fine-tuning it's like completely kind of pushing the distribution somewhere else yeah that may end up being the right approach but i think the science on that is not even figured out it's almost harder.

22:04There's almost more options on how you fine tune than to train from scratch. And if you have a very large data set to fine tune on, it's not clear that there's a massive financial difference. There's also the question of to just train a small model, a smaller model that's just on very domain specific information. So there's certainly examples of that in the literature that people speak about. And maybe that's another approach. To advise on people training from scratch, the route we made. I mean, for us, it was a great learning experience. And we're continuing on this path. There's certain another option that I don't know that you mentioned is kind of using existing models as an API, like sending data to OpenAI or, you know, there's other companies that do the same.

22:50And that's a step too far for us because there's certain data that we just don't want to send away. We want to handle it in-house. We don't want to have that process at this point by other people's systems. So some people are going to have concerns about wanting to keep data in-house on-prem. It's a tough call. It's a large investment to build a model from scratch, but it gives you more options. Yeah. And you mentioned that you followed very closely the Loom architecture and you took a lot out of their training notes, kind of suggesting that the things you run into in training are very specific to the architecture that you're using, do you feel like the things that you learned are transferable to kind of the next model architecture that you might want to explore?

23:36Or do you feel kind of locked into building on this current architecture? No, we don't feel locked in. You know, a lot of the things you learn are what you instrument in your model training and, you know, just kind of building up the infrastructure or knowing what infrastructure you want to build for testing the model as it goes and that sort of thing. I think we have a lot of work to do still on data set selection. We didn't follow Bloom's data set because we brought in a lot of our own stuff. There's questions on cleaning, which seem to be really important. Also, I guess for a while, there is a sense that it's really best to train on every data item once.

24:16You don't want to repeat documents. And it's not so clear how important that is at this point. And so the relative weightings of the different portions of the data is set. Wikipedia seems like a great source of knowledge. Do you run that through 10 times as opposed to, we kind of had it three times because it happened to show up in various collections that we used. Got it. Those are kind of problems that apply to any architecture. Tokenizer is another really interesting piece of the puzzle that's orthogonal to architecture. I mentioned a little bit with digits, but there's big questions on how big should your vocabulary size be.

24:52One other change we made for most models is our vocabulary size was about 150 ,000. Typical English-only model is like 50 ,000. So that's another spot where we deviated from most model approaches. And we did not have time to do extensive studies on smaller models to see like pros and cons. The learnings that we did on that would also kind of are kind of independent of architecture. A lot of the conversation in the broader around LLMs more broadly is around there are two, I guess, two numbers, kind of the number of parameters and the context length. Are either of those things at the top of your list of things you would do differently or things that are lacking, that feel like big constraints for you?

25:36Or are the things that you just mentioned, like exploring smaller models, all these other things, are those more interesting to you to play around with? Is it just scale or is it all these other kind of nuances that you've been describing? Yeah, I mean, there's definitely a lot of stuff going there. It's not just scale. You know, the Lama paper was pretty interesting because their smaller models did great. And it seems like it was largely due to, they continued training for a trillion tokens or something on the smaller models. Way more than chinchilla optimal for those smaller model sizes. So as we think about what's happening next, you are asking, is it bigger models, smaller?

26:18In some sense, we a little bit want to experiment more with the smaller models for a while because they're so much easier to use. You can run inference on a smaller model on a single GPU, whereas at 50 billion parameter model, you need kind of an inference platform to use it because it's got to go across multiple GPUs. So there is some interest in seeing how much we can get out of the smaller models, maybe the llama style training. There's clear performance differences between different models of the same size. And so it's a matter of collecting those different tricks and secret sauces that make kind of get the best use out of the data you have for your kind of compute budget.

26:58So we have a lot to learn there. And that's part of our research right now. There's a relationship between trained from scratch versus fine-tuned versus context, right? And, you know, because a lot of times you want this large context, so you can stick in a lot of context that isn't in the base model. but you've trained from scratch on the kind of data that you work with. Have you found then that you don't need a large context because of that? Or do you still find yourself wanting more tokens to be able to construct your prompts? Right, right, right. Okay, so how I think about that is the fact that we have this fin pile in the data set, that puts in knowledge into the model and maybe kind of ways to deal with certain types of document.