NeurIPS 2023 Recap — Best Papers

27:19The more general lesson is that if you want to predict changes in model capabilities with increasing scale, you need to consider the interplay between known scaling properties, the amount and quality of evaluation data, and the specific metrics and evaluation processes that you have available. So with that, and with gratitude to all my collaborators and everyone here for attending, thank you.

27:46So for the purposes of this episode, we actually tried to do interviews at the poster sessions for each paper, but some we just didn't manage to find. Or for the case of the Emergent Mirage paper, it was just way too popular. There were just so many people crowding out and listening to Ryan explain his paper again and again that we just couldn't get a proper question in. And I have to say, if I'm allowed to be a little bit critical, I'm a bit puzzled as to why this paper was the best paper. I mean, it's a good paper, but it doesn't really deny the existence of emergence. It just pointed out some methodological disagreements, which Jason Wei has also responded to.

28:25In other words, I don't really know if this paper affected literally anything in the field. So I don't know why it's best paper and not just a regular paper. But it's still a notable paper for sure. and it's very well done. Next, we have the runner-up for best paper, which is direct preference optimization, which is a direct challenger to PPO. And you can hear directly from the authors. Hi, everyone. My name is Eric, and I'm here with Raphael and Archit. And today we're going to talk about direct preference optimization, which is this algorithm that simplifies RLHF, which is this algorithm framework that has sort of been taking the LLM world by storm recently.

29:09So to start, why are we even talking about reinforcement learning for language models? Now, it's not the first time people have been studying reinforcement learning in the context of language models, but the sort of simple answer to this question is that a few years ago, GPT-3 came onto the scene, and it was sort of a big deal, and you probably, well, I'm an LLM person, but you probably heard from a lot of your researcher friends, like, did you hear about this new model? And then last year, ChatGPT came on the scene, and it was more like, at least I was getting texts from my grandmother saying, like, hey, have you seen this new model?

29:42And these are just two different levels of permeation in the public consciousness. And so what is the difference between these two models? And really, the main sort of key ingredient is this reinforcement learning from human feedback framework, which lets us sort of align the behaviors of the models more towards what people kind of want or expect. Okay, so to give a little bit of an overview of what sort of the existing RLHF pipeline looked like, kind of when we started working on this project, so there are basically two main steps. So the first step is we're going to start with some reasonably behaved kind of imitation behavior clone policy, what we call pi theta SFT here, so supervised fine-tuned policy.

30:25We're going to sample pairs of responses or trajectories from this policy conditioned on a prompt X. and that's how we're going to gather this data set of preferences. So we'll have an X and we'll have two Ys, and a human is going to just label which Y they think is better. So they're just going to give us this binary preference pair over responses, and we're going to use this data to fit a reward model. And then in the second step, we're just going to optimize a policy to maximize rewards. So that's just RL. So to look at this a little more closely in this first step, we get this feedback. It's these triples of a prompt and two responses.

31:01One is sort of the winner, and one is the loser. And we're simply going to train a reward model with this binary classification loss on the preference data. So this is this Bradley-Terry model of discrete choice in humans from the 50s. But, you know, it has some nice properties, and it's relatively simple to understand, and we use this to fit this reward model. So we're just taking the difference in the rewards, and we have this sort of Boltzmann rational model here that we're fitting with maximum likelihood. Okay. And so now that we're done with this first step, We want to, you know, what are we going to do with this reward model?

31:33Well, we're going to try to find a policy achieving high reward. And so, you know, ideally this reward model, after we've done this supervised learning stage, it should, you know, represent goodness according to what humans want. And so we're just going to fit a policy that both achieves high reward, but also stays close to our original model, our reference model, or our supervised fine-tuned model. And so that means we're going to try to find a policy here, pi theta, that generates samples that achieve high reward under our learned reward model, but also stays close to our original model, our reference model, because if you remember, we actually fit our reward model on samples that were annotated by humans, but these samples were generated by our reference model, our supervised fine-tuned model, right?

32:15So we don't want our policy to drift too far away because we want to stay in the regime where our reward model is actually reliable. So now that we have this objective, We take some off-the-shelf RL algorithm. Typically, it's PPO. And we find a policy that optimizes these rewards. This is a very complicated procedure. So there's this nice figure in this recent paper showing sort of the full pipeline of just the PPO step. And there are a lot of moving pieces here. And so in light of sort of this complexity, we kind of set out to see if there's some way we can sort of use a structure of this problem to simplify things.

32:54All right. So how the heck do we solve this optimization without reinforcement learning or what we call direct preference optimization? Really the key here is that the optimization that was set up for RLHF has a closed form optimal solution. Now this may look a bit intimidating, but it's really just the reference distribution re-rated by the exponent shaded reward. So if you have a good completion, you want to put more probability mass on it. And if you have a bad completion, why? You want to put less probability mass on this. Does me look familiar? It's the Boltzmann distribution that you might have seen earlier, and it's very commonly used across machine learning and physics.

33:30But the key takeaway here is that every reward function r will induce an optimal policy, pi r. But there's a very nice way to view this identity through another perspective, where we express the reward model in terms of the policy itself. So r pi x comma y can be written as beta log ratio of pi by pi ref plus the beta log partition function zx. And this really is the key where every policy pi is optimal for some induced reward model, rpi. And this really is the key to DPO because our key idea here is that you can fit this reward model, parameterized as a beta log ratio, to the preference data and hopefully skip the RL process altogether.

34:13But the problem is that this log partition function is basically intractable as you have to sum over all possible completions for a given instruction. So how do we get away from this? Now, fortunately for us, the reward modeling loss that we looked at, the Bradley Terry loss, only depends on the differences in the reward. Specifically, the reward for the preferred completion subtracting the dispreferred completions reward from that. Now, if you look at the induced reward difference, and if you plug in the DPO parametrization here, you can see that it only ends up depending on the DPO reward for the preferred completion and subtract the DPO reward for the dispreferred completion.

34:54Now, the more important thing here is that the partition function, which only depends on the instruction X, cancels out. As it only depends on the prompt. And this really is the key part here. And if you plug in this difference of rewards in the classification loss, you get the DPO loss function. And really, in its essence, it's just a classification loss with a specific reward parametrization, which will give you the optimal policy for the original RLHF objective. So to go back to what Eric presented earlier, the RLHF is typically a two-step process. You first fit a reward model, and then you do some RL on top of it.

35:38Really what we are doing here is that we choose a specific parametrization, the DPO parametrization for the reward model, and we're still fitting the reward model exactly the same way, but you get the optimal policy in process and you don't have to do the step two at any point of time. It's pretty useful to look at the DPO loss function through its gradient as well. Just to recall, it's still a classification loss, nothing changed in the two slides. And you're trying to maximize the difference between the rewards, but the gradient is really intuitive. Specifically, what we're trying to do is increase the log probability of the chosen completion and we're trying to reduce the log probability of the rejected completion.

36:14The important part here is that we slow down the training on the preference pairs, where the induced reward model is already pointing in the right direction, so you're not overfitting to the examples over and over again. But overall, it's very intuitive, as you're just doing up on the good examples and down on the bad examples.

36:33And finally moving to our experimental results, the first thing we really wanted to evaluate is how good of an optimizer that is for the core objective of reward versus divergence trade-off for these language models. So we started with this synthetic experiment where the goal is to generate positive movie reviews on this IMDB dataset with a small GPT-2 base model. We created synthetic preferences by sampling several times from the base model and using a pre-trained score classifier to construct synthetic feedback pairs. Immediately, the first thing we see is that DPO provides the best reward-KL tradeoff.

37:11And PPO, although improves quite a bit, it doesn't quite match that efficiency of optimization, even when we provide it with the ground-truth scoring model that generated the preference data. And in addition, other sort of algorithms that are RL-free, avoid sort of the RL modeling approach, such as just fine-tuning on the preferred answers or things like that, either don't produce the same level of improvement or are unstable. We then decided to try to scale these results up to more harder, more involved problems. The first thing we did is this summarization task with the goal is to provide summarizations of some Reddit posts and dialogue tasks of the Tropic helpful and harmless dataset, publicly released datasets.

37:58And kind of again, what we see there is that across the board, DPO either matches or outperforms all other baselines. And particularly, for example, in the summarization case, the PPO model is almost twice as big. So another interesting experiment that we ran recently is evaluating the generalization capabilities of the DPO policy. Because essentially the PPO-trained approaches sample a lot of additional data and have the capability to train a lot of additional data, while DPO is fully only using the offline data set of preferences. So what we did here is we took the summarization models that we presented in the previous slides.

38:38Those are the first two graphs on the left, sampled at different temperatures, and evaluated within distribution. As you kind of see, within distribution, they're quite comparable. And then we evaluated them on out-of-distribution data, on particular summarization of news, CNN, and Daily Mail articles. And we do see quite a significant drop when we take these models out-of-distribution. But the interesting thing is that the DPO policy still generalizes just as well or even perhaps better than the PPO trained policy, even though the PPO trained policy is trained on a lot more additionally sampled data.

39:13However, I think the strongest sort of validation of this algorithm and its capabilities are the strong open source models that have been trained by the community. And this is only a selection of those. There are others we couldn't fit on the slide. and if you could go through all of them, you see that especially some of the recent ones do match or sometimes even outperform chat GPT on some broad benchmarks. Another point to mention here is this is only within the language domain, but recently works have done this training state-of-the-art text-to-image models with the DPO algorithm used for vision language models and also using for multi-step control as well.

39:50So this is going beyond languages. It's becoming kind of a paradigm of alignment. So in conclusion, I want to point out that kind of the DPO removes the complicated, expensive RO training loop from ROHF. It's a simple, stable, and computationally cheaper than PPO. I think almost, you know, order of magnitude. And most importantly, it's also principled. You're optimizing the exact same objective. It's not a hack. You know, it's optimizing for the exact same thing. And, yeah, as you've seen, others are training, you know, a lot of state-of-the-art models. We're achieving pretty strong results, so you should do as well.

40:25So if you want to learn more about it, you can come talk to us at our poster. And we have publicly opened our code implementation. You can find it on GitHub, and you can check our paper on Archive as well. Thank you very much. So DPO is interesting because it promises to be simpler than PPO. It's definitely easier and cheaper to train, and there are a bunch of models already emerging being trained on it. The main criticism that people seem to have is that it isn't performing as well in terms of alignments or results or benchmarks as PPO-trained models. But that still remains to be seen, whether that ease of use and cheapness of availability of data or whatever makes it so much better that it doesn't actually matter.

41:07So what happens in Europe is that some papers are selected for oral sessions, and then everyone heads down to the poster hall, where there's about 600 posters simultaneously presenting, including the people from the oral sessions. And this is what we did. We went down to talk to the paper authors after their oral session. So we're going to hear them re-explain DPO in four minutes and then answer a bunch of Q &A. But you can also get a sense of how chaotic and noisy it is in that poster session. It's just a mess and I love it. I'm talking about direct reference optimization here. RLHF is really cool.

41:40You get chat GPT from GPT-3 using RLHF. If you've never heard of chat GPT, you might want to look it up. It's really important. RLHF is complicated. It's really hard. When you start with preference data distribution, you usually have to do some kind of RL process on top of it. And RL is hard to implement because it has a lot of moving components. You have to sample the model a lot. You have to train a value function. You have to do a lot of magic trickery to get it to work. Our hope was that, like, can we make this simpler? And that's where we designed DPO. Just to give a brief overview of RLHF, it starts off with some distribution or some model that you have already trained, which is usually reasonably good.

42:21I'm thinking of GPT-3, which is already pretty good. They like some preference data on top of it. So you have instruction, two pairs of completions, and the human labels which one is preferred and which one is dispreferred. With this preference data, you first fit a reward model. The reward model will give you, it's basically telling you which preferred model should have a higher reward, and the dispreferred completion should have a lower reward. And this is a simple classification problem. It's very straightforward. Now, given this reward model, you want to do RL on top of it. So, like, you want to generate completions, which are good.

42:55And the way you set it up is you maximize the expected reward under a KL constraint to the initial distribution that it started with. Now, why the KL constraint? The models can degenerate very, very quickly. And usually what you want to do is, like, stay close to these models so you don't degenerate and you do not exploit the reward model. The reward models are trained on a very little amount of data. And these are very easy to exploit. So that's why the scale constraint is important. This is a traditional RLHF pipeline. This is what exactly was used for chat GPD, initially at least. And it's very complicated to do with PPO.

43:31It's hard to get it right. Now, our contribution is the direct reference optimization. And the way this works is that it turns out for this optimization, there is an exact optimal solution. This optimal solution, if you've seen Boltzmann distribution before, very simple. You take your reference distribution. you upweigh the good things by exponentiated reward and you downweigh the things by exponentiated reward which are bad. So it's just the exponentiated reward weighted for the reference distribution. Now, unfortunately, this is intractable. Why? Because the partition function is intractable.

44:05So you cannot actually compute this distribution. But as it will turn out, this is not going to matter. So our main contribution is that you can actually rewrite the reward in terms of the policy itself. So simple algebra, you write the reward in terms of beta log pi pi by ref. This is just simple algebra here. Take your time, just look at it for a second. You're just rearranging terms. But the thing is that you still have a beta log partition function, which is still intractable. Now the key thing is we can fit this reward using the same classification loss that we were using earlier over here.

44:41And the nice thing is it depends upon the difference between the reward for the good completion and the bad completion. And the partition function actually cancels out. If you look at the partition function, it only depends on the instruction. So it only ends up depending on this quantity. And this is exactly how you get the DPO loss. You're plugging in this implied reward function into the classification loss, and you get the DPO classification, which is directly in terms of your policy that is being fine-tuned. So you no longer need to do an explicit reward model where you're learning a different reward model.

45:14You do not have to do any RL optimization after that. What you're doing is exactly, you're fitting this reward model and you immediately get the optimal policy for that reward model without doing any RL. And that's like the main pitch for DPO. Any questions, anything I can explain further? So that means you don't have to learn any reward function. You don't have to, but the policy already implies a reward. Yes, exactly. Does that make sense?

45:49What about the data collection aspect of RLH? That's a great question. So PPO usually samples more completions online, and you don't have to do any of that. You only have to sample the preference data set in the beginning, which we use for... How do you know that your preference data set is as good as... We use the exact same preference data set for RLHF and for DPO. It's like a mathematical shortcut. Yeah. Like they set a new loss function. It comes from the fact like you train this model on some data distribution, but when you explore, it might go out of distribution, which kind of like limits the policy.

46:31Yes, exactly. That is a major reason for drop is? In general, PPO also has like a high variance estimator, so the optimization is never perfect, Whereas with DPO, you know for a fact that it's an optimal policy. So it's very, very similar. You know for a fact that it's optimal. But in general, if you have a very well-fine-tuned PPO pipeline, it will usually work reasonably similarly. But yeah, you don't have to do any of that. Essentially, one of the things is that... This is not an assumption. This is the actual solution. This is not an assumption. Right. It's generic enough in terms of mathematical form, but I was wondering because this term reward, for example, does it match the definition of reward?

47:16Because you could write any exponential function here, and it's been called reward, but does it match the reward definition? In this optimal solution, you assume there's a reward function that has been given to you. Oh, I see. The sequence of actions that's asked. Yeah, yeah, yeah. Some constant times the log ratio of some. And, I mean, overall, if I look at the experiments, let's look at the real-world data sets. Like, I mean, we try out, like, summarizations, like, single-turn dialogue, and it all works great. You'd never have to do, like, any online exploration of any form, and, like, BPO relatively works better than BPO or very similarly to it.

47:54I think... Can I... What's the methodology? Like, you take a base model and you fine-chain it with DPO? So, we take a same base model. Yeah. We have the same preference dataset. First, you fit a reward model for PPO, and then you do RL for it. Okay, so completely comparable. Yes. In general, we tried to reuse people's already pre-trained models for RLHF, but we looked at their pipeline, and it was exactly the same. Because if we do it, there's always a case that it's possible that we didn't tune it well enough. So we tried to take models that were trained using RLHF and try to compare to them directly.

48:30But they're trained on the same datasets. Very strong models have been trained using DPO. They're already being used. Yeah, Zephyr is the one I know about. Zephyr, Dulu, Mixtral models, if you ever look at it, were trained using DPO as well. Oh, that's the Mixtral Instructs team? Yes. Okay. They were trained using DPO as well. That came out very recently. Yes, that's why it's not on the poster. I see. You guys, if you're thinking of fine dealing using preferences, you should try to use DPO. How much is the efficiency gained compared to a PPO process? A lot, because you only have to do one step.

49:06It uses the same set of preferences. It's usually the one. So basically, no trade-offs? I think more research... I'm looking for trade-offs. I cannot find any. More research needs to be done. There are arguments to be made that PPO might do better in some cases, but it's unclear. We haven't personally seen any evidence yet. I see. Sorry, one more question before I... Go for it, yeah. I noticed Chelsea Finn's a co-author. Uh-huh. What guidance has she given? I'm curious. I mean, look, we're all in her lab. She's the one who selected us. She's the one who's providing the infrastructure. Yeah. Like, I mean, none of this would be possible without her.

49:45I'm just curious, is there any interesting stories, any good advice that she gave that really inspired you that you want to pass on to others? When we started discussing the idea with her, she was very insistent that you should try to push this because this is a nice idea. But if you sit on it, somebody might do it or it might fade out of an element. So this paper came about in three weeks before the New York deadline. So we had to push really hard. And how did you come up with the idea? I mean, we were looking at this kind of equation before. Rafael did a bit of algebra and said, oh, maybe we can just completely skip the RL part if we look at this thing.

50:24So, like, I mean, we're playing around. Generally speaking, like, there's a reward estimation step. Whenever you're learning three things in a sequence, if you can statistically remove one of the steps... Yeah, you gain a lot. Yeah. So that's where the motivation usually comes from. Has John Schulman commented on this? Yes. What did he say? I mean, he tried it. He said it works. But there's some questions about, like, they might be training the reward models on more than binary pairwise preferences. so like it's not immediately clear how to extend that using DPO like multiple choice unclear they obviously did not tell me like I mean what they're doing but like there's training on more than just pairwise preferences and they might still want to do like RL you can decompose most things into pairwise yeah yeah that's kind of what I assume but like I don't know what exactly they're doing so there's a situation where they might be conditioning their reward model on something more than what your policy is conditioned on so x is a constant.

51:21Yeah. And so this... That's all I got. Thank you. ...is zero. The other best paper runner-up that we'll talk about is scaling data-constrained language models. In other words, the data-blations paper. And this is a scaling laws paper, kind of in the vein of the chinchilla paper, but done with a different assumption in mind. Instead of holding compute constant or holding parameter count constant, here we are running into the real-world problem of data-constrained. So given that you have a fixed amount of data, what should you do to pre-train your models? This kind of paper tends to be a very expensive paper to write, just because you have to do so many ablations.

51:56Here it's notable that HuggingFace has created this and open-sourced it, both models and data sets. So kudos to HuggingFace. Hi, I'm Nicholas, and I'm presenting Scaling Data Constraint Language Models. The premise for this work is that we are data constrained. Here's a plot from prior work that estimates that given their definition of high-quality language data, we're going to be exhausted next year. And what they mean with high-quality language data is data such as papers and books. There's other sources, like code, however it's unclear how useful it actually is for large language models. And for low-resource languages, we are already hardcore data constraint.

52:38The first solution we investigate is simply repeating data. It's important to mention here that, while it's pretty common to train for multiple epochs in most machine learning problems, for large language models, this has been very uncommon. In GPT-3, they write that data are sampled without replacement. In Palm, they say that they explicitly avoid repeating data in any subcomponent. And there was other work explicitly recommending against repeating any data when training large language models. So we ask, is it really that bad? To answer this question, we have three different setups. setups. We start by simply training for a single epoch.

53:17Here, this is your usual training graph where we have the validation loss on the y-axis and the training tokens on the x-axis. And for all of those setups, there's nothing special here. Loss improves as we increase training. Now what happens if we train for two epochs? Notably, the performance is around the same. So here, only half of the data is unique and it has to be repeated twice. So for the setup on the left, 28 billion tokens are unique and they're repeated for two epochs. Three, four, and it's still pretty similar. However, eventually, it starts to diverge. So we shouldn't train for too many epochs.

53:50At 44 epochs, literally just one 44th of the data is unique and repeating 44 times. So that's like one billion tokens, one billion unique tokens for the setup on the left. And that obviously isn't very good. However, for a few repeats, performance is very similar, suggesting that we can scale a lot further with existing data constraints by simply repeating for large language models. This naturally leads to the question, how should we allocate compute when we are in that repeated regime? A quick reminder from last year, Chinchilla told us that when we're not repeating data, so in the single epoch regime, we should scale model size and training data equally, in equal proportions.

54:30How does it look like when we're repeating? To investigate this, we train on 100 million unique tokens and vary the model size and the number of epochs over those tokens. Each model is depicted as one of those dots. And as we go towards the upper right, so more parameters and more epochs, loss improves as indicated by the contours. We put forth scaling equations to exactly predict this change in loss and how you should allocate when you're in that repeated regime. They're depicted on the right. Now if we add in the efficient frontiers, the Chinchilla scaling loss efficient frontier extrapolated to multiple epochs corresponds to the dashed line.

55:11So here's just an equal scaling of parameters, an equal scaling of epochs. However, outfit suggests that data should be scaled faster when we're in that repeated regime. And this is seen by the line branching off below and eventually just fades away because at some point you can't get more value out of your data, especially with just 100 million tokens. At some point you're just running out of value in those few tokens.

55:39Now we test our predictions at scale. Here we have two models, one allocated according to Chinchilla scaling loss and one allocated according to data constraint scaling. The one on the top is Chinchilla and the one on the bottom indicated by the red star is our allocated model. They both have the same number of flops and the same data budget of 25 billion tokens. And we see that by training with fewer parameters for more epochs, so 6.3 and 9.7 epochs or 242 billion tokens, we get a better loss. But not only loss, we also test this in terms of downstream performance and get better downstream performance, as indicated by the column towards the right.

56:24This was repeating, and now we're going to look at complementary strategies to solve data constraints.

56:33One intuitive strategy is making use of that code data that we saw earlier. So can we simply fill up the missing data with code from GitHub? In addition, we evaluate filtering strategies. Specifically, we look at fuzzy to duplication and perplexity filtering. The idea here is, can we use a quality filter and then repeat to get better performance than with the initial data set?

57:02here are the results on the y-axis we have the average performance across 19 natural language tasks on the x-axis is the data budget so towards the left we have 100 % of available data so we don't need to use any of those strategies but as we go to the right our data budget is smaller and smaller and we need to repeat data or fill the missing data with code starting with the purple line we can confirm our findings from earlier that also in terms of downstream performance, roughly four epochs seems like a good trade-off. So at 25 % data budget, we have to repeat four times, corresponding to four epochs.

57:36And then eventually, if you train for too many epochs, it drops quite a bit. So you have to be careful with repeating. The red line corresponds to filling missing data with Python code. Similar to the repeating line, we see that we can make up for a lot of natural language data with code without a drop in natural language performance. So these are all natural language tasks. And it seems like coding data is helpful for some of them. We even see spikes on some of these tasks as soon as code is added. Finally, we investigate the filtering strategies. We find that quality filtering, then repeating, can be much better than the data set to start with.

58:12So here the yellow star at the top corresponds to perplexity filtering and then repeating for two epochs. The orange star corresponds to fuzzy deduplication towards the right. And we find that you have to be careful with too much deduplication because it can lead to a worse model by limiting your available data.

58:38Now I'll go through the takeaways. The first takeaway is that repeating data is generally fine. For many setups, roughly four epochs seems to provide a good trade-off. However, there are diminishing returns and you have to be careful with too many epochs. Next, adding code data is fine even if you're only interested in natural language tasks. We find that 50 % provides a good trade-off for most setups. Finally, quality filtering plus repeating can be a good strategy and is often much better than the dataset you started with. Because the penalty from repeating is often much smaller than the additional gain you can get from quality filtering.

59:15And finally, I wanted to finish off with some other work that has made use of these findings in their large language model training. So at the top, we have FinGPT, a large language model for Finnish, where they only had 38 billion unique tokens. And they had to repeat them for eight epochs in order to be able to train a reasonable large language model with 13 billion parameters. and there are several more that haven't made use of these findings.

59:42The finding that training up to four epochs is almost as good as getting new data is pretty surprising and actually directly counters a very famous paper called One Epoch is All You Need. I actually read into Aaron Komatsuzaki at the decibel party and it's just surprising at this stage in ML that we still don't know some very basic questions around how many epochs we should train on a dataset. I mean, I still think that we are surprisingly sample efficient. You know, the consensus is now between one to four epochs, sometimes in some cases maybe up to eight. But more importantly than that, I think this work is notable because it is the best example of what open source AI research should look like.

1:00:22And of course, it's from Hugging Face. If you go to the GitHub repo, you can see not only their papers, but also very, very well-documented code showing exactly what they did and how they got their results. including the dataset filtering. So just exemplary work of open source AI and no surprise that they won one of the best paper awards. However, I did not manage to catch up with them for a post-presentation interview, but I did go straight to the next session on Qlora with Tim Datmers. I'm Tim. Today I present Qlora, efficient fine-tuning of quantized large language models. Language models have been gotten a lot bigger and a lot more powerful, but they have become so big that it's actually quite difficult if you take a pre-trained model and you want to fine-tune it as sort of a normal researcher, often you need now a big GPU server, and most researchers don't have that.

1:01:11So with Qlora, what we worked on is reducing the memory requirements so that everybody can fine-tune large language models. The main contribution of Qlora is we compress neural networks to 4-bit, and we develop a new data type, 4-bit normal float, that can replicate 16-bit performance, even though we compress the neural network to 4-bit. Before I talk about QLOR, I give you a little bit of background. So this work is about quantization, about compression. So we do, for example, quantization if we have a 32-bit float number, and we want to quantize it to a 4-bit integer. In this diagram, I have a histogram, which is equivalent to an int 4 quantization with 16 different bins.

1:01:55and in red I have the normal distribution and if we want to quantize all the values in the normal distribution to a four-bit integer, we need to reduce all these values to 16 different values. How do we do that? We find the empirical minimum and maximum range of the distribution and then we slice this distribution in 16 different slices with equal width. Each of these slices is a quantization bin and all the values contained of the normal distribution in this bin are quantized to the middle value of the bin. With that, we can reduce all the values in the normal distribution just to 16 different values.

1:02:32And this is an in-four quantization. Now, if we do other quantizations with other data types, we have different ranges. And so what I do in my work is I generalize these data types by normalizing the range the data types take to the range minus one and one. This approach is also called a code book where you map an index to a particular values in the data type And so if we have this code book, there's a two-step recipe how we can quantize any tensor And so we take the tensor X then we normalize it into the range. Oh, sorry And we normalize it into the range of minus one one By dividing by the absolute maximum value and then we go through each element in the tensor and find the closest value in the data type.

1:03:17We do that by doing a binary search on the sorted values in the data type and with that we can then quantize the entire tensor. Just to make this a little clearer, here's an example. This is a very unusual two-bit data type. It has the values minus one, 0.3, 0.5, and 1.0. The input tensor is 10, minus three, five, four. and now let's go through the steps of the recipe. So first we find the absolute maximum value, which is 10. We divide by it. We get 1, minus 0.3, 0.5, 0.4 and then we find the closest value of these values for each element associated in the data type. We get 1, 0.3, 0.5, 0.5. Then we find the associated index of these values and this is now a two-bit representation.

1:04:06Now we can store it and it's compressed. If we want to dequantize these values, we just do all the steps in reverse. So we look up the associated values in the data type, and then we denormalize by multiplying by the absolute maximum value 10. That gives us 10, 3, 5, 5. And so if we compare input and output tensors, what we see is that we have two big errors. The minus 3 turned into a 3, and the 4 turned into a 5. These are quantization errors. And so the main challenge in quantization research is we want to compress a neural network with a low precision data type, but we want to keep all the quantization errors minimal.

1:04:44If the quantization errors are large, we degrade the neural network performance, and we want to avoid that. And that's the main challenge. Let's talk a little bit about fine-tuning. Why is it so expensive? So the best way to look at it is to look at the cost per parameter in fine-tuning. And so the per parameter cost for full fine-tuning is 16-bit for each weight, 16-bit for each weight gradient, and 64-bit if we use Atom for each parameter. That gives us 12 megabytes per parameter. And if you have a 70 billion model, that's 840 gigabytes of GPU memory. 36 consumer GPUs. That's a lot of memory.

1:05:24If we use lowering adapters, we get much more efficient. And so what we do there is we take a pre-trained model, we freeze it, now we put some tiny layers on top of it, some adapters, and so if we fine-tune it, we do stochastic gradient descent through the frozen layers into the adapters, and we just update the adapters, not the main model. And so what that does is the weights still need 16 bits per value, but now all the other values that are updated, they're only a fraction of a bit on average. And so in total we have 17.6 bits per parameter. That adds up to 150 gigabytes of memory, which is eight consumer GPUs.

1:06:03Now with our development of Qlora, we step in and go a step further. So now we take the pre-trained model, quantize it to four bit, and then put adapters on top. That reduces the average footprint to 5.2 bits per parameter, which is 46 gigabytes, and that fits into two consumer GPUs. Now, the main challenge is we want to preserve the performance while doing this four-bit compression, and that is the main challenge. So we have three innovations that improve the memory performance, but then also the precisions to reduce the quantization error. There's one part, paged optimizers I will not talk about.

1:06:41You can read about it in the paper. It's used to prevent memory spikes during fine-tuning if you hit a large document and during your fine-tuning run. The main contribution that we have is the 4-bit normal float data type. This is a data type that's information theoretically optimal. And so you can think about it like this. So in the beginning, I showed you an int 4 quantization where the quantization bins have equal width. In a normal float data type, the bins have equal area. That means each slice has equal probability mass in the normal distribution. And that means the same amount of values are quantized into each bin.

1:07:17With that, each bin has equal amount of values and its information theoretically optimal. Our second contribution is a little bit silly. It's double quantization. We do a quantization of the quantization. And so what does that look like? So in the normal quantization, we take the weight, quantize it, and now we get two pieces. The quantized weights and then the absolute maximum constants. We have multiple constants because we slice the weight into blocks and each block has its own constant. And so we get a matrix of constants. On average, these are 0.5 bits, and that's multiple gigabytes of GPU memory.

1:07:53And now we quantize those constants again. We save about 0.4 bits on average, and that is important if we want to fit large models into consumer GPUs, because otherwise they don't quite fit. And so these are the contributions. Now let's look at the results. So the main thing that we want is to replicate 16-bit performance. That was our main goal. And so what I have here is different LAMA models of different sizes, and we fine-tune on the FLAN2 instruction data set. We evaluate on MMLU accuracy. We have in pink the 16-bit baseline and BrainFloat16, and what we see now that the float data type, the regular float data type, 4-bit float in blue, doesn't quite replicate 16-bit performance.

1:08:42However, if you use our normal flow data type, we get up to 16-bit performance. And so with that, we have now replicated 16-bit performance. In our papers, we have much more experiments that also have the same finding. But with that now, we are at the stage where we can very efficiently fine-tune very large language models with very little resources. And so now we go a step further and ask, can we build a high-quality chatbot now that we very quickly can explore all possibilities with cheap fine-tuning? And so through our experiments, we run over 1 ,000 experiments, we find a very good data set and build a chatbot called Gonaco, which is a 4-bit data set.

1:09:24We create it by just fine-tuning on a single consumer GPU for 24 hours. And now we want to compare how good is a chatbot compared to other chatbots that are trained or fine-tuned in 16-bit. And so we have a tournament-style setup where the setup is we have 80 different prompts from the Vicunia dataset, and we give this prompt to two random Shetbots, and then they compete to generate the best response. Each Shetbot generates a response, and then the responses are judged by the humans or GPT-4, and either humans or GPT-4 say, which response is better? This consists as a game, and so we play multiple games with many random allocations of chatbots, and with that we can determine which chatbot is better than another chatbot.

1:10:14If we do the setup, then we find that humans think our chatbot on these Vicunia prompts is a little bit better than chatGPT. If we ask GPT4, it says it's about the same quality as chatGPT. This doesn't mean that our bot is as good as chatGPT, but for these particular prompts, it is about the same quality. On the right is also a demo. You can scan it and try our Shadbot. And that's everything that I have. So just to conclude, Qlora makes fine-tuning 18 times cheaper. With the 4-bit normal float, we can replicate 16-bit fine-tuning performance. And we have also shown that you can create very high-quality Shadbots with Qlora.

1:10:58So with all of that, it's very simple to now create high quality fine tuned models and it's so cheap that everybody has access to the fine tuning these large models. Qlora is available in the bits and bytes library and it's also integrated in the Hugging Face Transformer stack. And so there you can very easily use it. I'm also on the academic job market, so please get in touch if you're interested. Later this week I will also give a talk on the making of Qlora at the workshop. So stay tuned on Twitter for more information about that. And that's what I have, and I'm happy to take questions. Thank you so much.

1:11:38So we're going to make a bit of a hard pivot now from the world of optimization, fine-tuning, and training methods into the world of multimodality, which is another big theme of this year and probably every year to come. Every previous paper we've covered on the pod up to this point, I've heard of online and, you know, it's relatively well known. You didn't actually need to meet the people to hear about them. But one of the joys of coming to a conference like NeurIPS is finding things that you may not have seen just in case of your filter bubble or just because there's just way too many things out there and you didn't have the time to look into them.

1:12:12And this was definitely true for me for Datacomp, which I never heard of, but also a very legitimate effort. And I actually had to chat with them after their talk. But first, let's introduce what Datacomp is. My name is Samir, and this is Gabriel Iliarco, and this is Alex Fang, and today we're going to be presenting our work, Data Comp, in search of the next generation of multimodal data sets. And this paper was really made possible by a whole team of people, and so we're very lucky and fortunate to be able to share it on behalf of the whole team. Okay, so we want to start with a little bit of a history of computer vision models.

1:12:52so in this kind of traditional paradigm of image classification what we would do is we would create a specialized data set we'll call that a traditional supervised data set with certain class labels for example 10 different labels for the mnist data set and then we would train these fixed models on these kinds of data sets and this was really cool because it led to all kinds of architectural improvements you can think res nets skip connections applications of attention but when you needed to add an additional task say ImageNet 1K you had to kind of create a new data set with a new set of labels and this was kind of a laborious process but then right around 2021 something really cool happened the paradigm a little bit switched to these kind of image text data sets that allowed trading these open vocabulary models And suddenly we could do things like train a unified model that could then downstream do arbitrary image classification tasks.

1:13:58And this is really a sort of data set transition is kind of the takeaway here. So in spite of this kind of transition between data sets, the standard machine learning pipeline actually stayed relatively consistent. So what we're still going to do is create a monolithic artifact, a data set, keep that fixed, and then iterate on model training on that data set. And this is still a really cool recipe, and it's led to progress in downstream evaluations. But what we really ask in DataComp and the center of our paper is how much performance are we actually leaving on the table by adopting the standard ML pipeline?

1:14:41design, can we actually improve models by iterating on data sets instead of on model architectures? And so fundamentally, DataComp is a benchmark for data set development to help the community understand how data set decisions improve models. So specifically, we're going to look at this clip trading regime for these more modern image text data sets, which are popular nowadays. And so we want to give just a brief overview of clips so that we're all kind of on the same page. We roughly have a text encoder and an image encoder, and we're going to train these encoders from scratch contrastively in order to align image and text representations.

1:15:29And then downstream, if we have a new classification task, we're going to do things like write sentences, a photo of a plane, a photo of a car, et cetera, and then query an image feature against all of these text features to retrieve our class label. So kind of recentering things back to data comp now, the picture I think that we should all have in mind is we're actually going to fix this clip bit, which is this middle training diagram and we're going to iterate on the data selection process to create new data sets to train our clip models. And now I'm going to hand it over to Alex. So the data comp workflow consists of five steps.

1:16:15Choosing a scale, selecting data, training a model, evaluating and submitting the results. And the first step is choosing the scale which roughly reflects the amount of compute used. So data comp has four scales. At the small At the small scale, we train a VIT B32 for 12.8 million samples, which is equivalent to fine tuning a model on ImageNet 1K. At the medium scale, we train a VIT B32 at 128 million samples seen, which is equivalent to training a model from scratch on ImageNet 1K. At large, we train a VIT B16 for 1.28 billion samples seen, which is equivalent to training an ImageNet 21K model from scratch.

1:16:55And at extra large, we train for 12.8 billion samples seen on a vit L14, which is equivalent to training an open AI clip model. One key design decision is that there is no constraint on data set size. We build our scale configurations around sample seen because practically speaking, the key constraints are pool size and compute. This means each data point in a data set of 6.4 million samples at the small scale is seen twice.

1:17:25At the chosen scale, participants can then use their data selection method on either a fixed provided pool of raw data or are free to bring in additional data. So in the first option, which is the filtering track, participants filter from a provided raw pool equivalent in size to the samples seen at the chosen scale. Our pool, which we call common pool, comes from common crawl, and then we do minimal preprocessing, such as near-duplicate checking against evaluation and not-safe-for-work filtering. Additionally, we provide metadata to help with potential filtering approaches. This metadata includes original width and height, caption, a checksum, clip features, clip scores, and face bounding boxes for automatic blurring to help with privacy concerns.

1:18:11The second option is the bring your own data track. This allows participants to use additional data sources as well as both edit and generate images and captions from common pool. We hope this track supports participants whose creative approaches do not fit neatly into the filtering track, while also maintaining fair comparison within the filtering track. Next, participants use a fixed training procedure to train a model on their newly filtered data. For training, we adopt fixed training recipes, including hyperparameters for CLIP training, and this was based on prior experience. Notably, data comp participants are not allowed to modify these parameters.

1:18:50therefore focusing investigation on data set selection. In the paper, we show that better data sets are largely consistent across variations in training recipes.

1:19:02Once models are trained, they are evaluated using our provided script. Our evaluation suite contains 38 downstream tasks, which include ImageNet and variants, a subset of VTAB, a subset of WILD's distribution shifts, fairness benchmarks, and retrieval benchmarks. And the evaluations are done in a zero shot manner to remove the need for fine tuning on each individual downstream task. And the last step of the process is to submit your results. We provide an online leaderboard that participants can submit to which we hope promotes participation and collaboration. We believe that many of these individual data filtering approaches should stack and when combined will lead to better results.

1:19:44Next I'll hand it over to Gabriel to talk about baselines and some new results. All right, so let's talk about experiments now. We study many baselines in our paper, but I'll focus on the two most interesting ones in the interest of time. The first one is what we call clip score filtering. The idea behind clip score filtering is simple. We use a pre-trained clip model to compute cosine similarity scores for all image text pairs in our dataset. In this plot, you can see a distribution of these scores in our dataset. set. We then choose a threshold for the similarity, for example, corresponding to the top 30 % scores in our unfiltered pool.

1:20:22We then remove all samples that have similarities smaller than this threshold, keeping only the samples with high score, as a proxy for discarding all samples that we think have low quality. Another filtering baseline is what we call image-based filtering. For image-based filtering, we again use a trained clip model, but this time only to extract image features. We then cluster these image features and find clusters that match images on ImageNet. We keep all clusters that are assigned to at least one image. We then discard all the other clusters. Note that this filtering is purely based on image features and we do not use any labels or captions for this filtering strategy.

1:21:04Our best performing baseline is built by intersecting between the two baselines I just described, clip score filtering and image-based filtering. When we apply this technique to our larger pool, we find a data set with 1.4 billion samples that we call data comp 1b. So let's see how well this works in practice. We conducted over 300 pre-training experiments with many different strategies for filtering our pool. Our best data set is data comp 1b, a 1.4 billion subset of our pool that leads to much higher accuracy than existing datasets, including OpenAI's WIT and Lion2B. This is the first public dataset that outperforms OpenAI.

1:21:47Also note that all these models are compute matched, so these gains come at no extra cost at training time. One key finding from our work is that smaller, more aggressively filtered datasets can perform better than larger datasets coming from the same pool. As you can see on the plot, when we selected samples that have the highest cosine similarity according to a train clip model, there's a sweet spot for the size of the data set that we keep, around 30 % of the original pool. This means that you're better off using a smaller subset of the pool instead of using more noisier data. Interestingly, this doesn't happen when you sample randomly from the pool, as you can see from the dotted line.

1:22:29So you can get away with smaller data sets, but you do need to be a bit more careful on how you are selecting samples. Another key finding from our experiments is that the ranking of different filtering strategies is relatively stable across scales As you can see in these scatter plots These plots show how performance on the small scale correlates to performance on the medium scale And while it's not a perfect correlation These plots show that there is hope for doing research at smaller scales Since there's a good chance that findings will generalize to larger scales And in fact, this is exactly how we proceeded during our experiments by first testing things out at smaller scales and only scaling up the most promising results.

1:23:10This saves us a lot of compute during our experiments. There's much more in the paper, as you can see in these slides, and if you're interested, definitely check it out. We are very happy to answer any questions and talk more about any of these topics in our poster.

1:23:28Since we released the paper, there's been a lot of activity in DataComp. The fun thing is that our best performing baseline, which we thought was pretty decent, were blown out of the water by the community since. And it's just really nice to see that happening in real time. One example is data filtering networks, or DFN for short, where the main idea is similar to clip score filtering, but with a deeper dive into what makes a good model for data filtering. And careful data creation has led to what now are the best clip models, even outside DataComp with an impressive 84.4 % zero-shot accuracy on ImageNet using a VIT H14.

1:24:07The central takeaway I'd like to leave with you today is that careful experimentation with data sets can really pay off and can lead to very large improvements in performance on downstream models. So instead of blindly scaling models up, I think we as a community should start paying more attention to how we design data sets. DataComp is designed to facilitate research in that direction. It's amazing to see what people are already building with it, and I'm super excited to see what comes next. Finally, I'd like to reiterate that our benchmark is designed to encourage everyone to participate, even if you only have a couple of GPUs under your desk.

1:24:44So if any of this sounds interesting at all to you, feel free to check out our resources, including our website, code base, and paper. Everything we do is fully open source, and we hope these resources are useful for the community. Thank you very much.

1:25:01So I quite enjoyed that presentation. And obviously this being a image heavy and multimodal type of paper, you should probably check out the images and the competition at datacomp.ai. But I did manage to catch up with them at their poster session and ask them more questions. It turns out there's some intellectual lineage from Lion with Lion 5B. And I do think that this has a strong chance to become the new ImageNet. So let's give them a listen. Oh, fun fact. They were also wearing Datacom t-shirts. Like most people, when they present their poster sessions, they're in kind of just like somewhat semi-formal attire.

1:25:33These guys, they make custom t-shirts for their posters. So you know how they're serious. My name is Samir. I'm a fourth year PhD student at Columbia. I started working on Datacom like I guess around November of last year. I collaborated with a lot of the folks that are already on the paper on previous projects. like Mitchell Wurtzman, Ludwig, Faisal. And they kind of just kind of roped me in. They were looking for hands to help out with different tasks. And then through the course of time, my involvement just kind of grew because I got really excited about it. How did this become such a big effort?

1:26:12You guys are wearing T-shirts. This is not normal. Yeah, yeah. Yeah, so we really took this project very seriously because we wanted the benchmark to be really good and thorough. And because of that, we were working at kind of a scale that was kind of unprecedented for academics. We generated the pool of like 12.8 billion image text pairs. We wanted evaluations to be very thorough, many, many downstream tasks. And that just took a lot of people to commit to the project. And how do people find out about something like this? Like, is there, you're not from the same university. Is there a community somewhere that you all just gather and coordinate?

1:26:53Yeah, yes. So Ludwig, who's kind of the last author on this paper, is kind of networked all around. He's very friendly and very open to collaboration. And I think because of him, many people from many different universities, corporations were able to join. Yeah. And this is separate from the Lyon group? Yeah, so Ludwig is affiliated with Lyon. Because I've seen his name around. Yeah, yeah. Yeah, but most of the people are not necessarily part of Lion, but we all kind of know each other and collaborate. I mean, wouldn't it be better to make this like Lion 1.4B? Yeah, so we maybe could have done that.

1:27:37Sorry, Lion 12.8B, right? Lion has 5B? Yeah, Lion has a 5B and a 2B subset that people train on a lot. Yeah, we could have done that. I think we were thinking about things more from the standpoint of a benchmark, and that was really our focus. While this 1B dataset that came out of the benchmark is an artifact, we kind of wanted to place emphasis on the benchmark itself. So just to comment on that, the idea of initially it was a dataset, but then we thought also about benchmarking, but then we thought about the real thing is the community. So we thought that the way that you can actually build a community is by opening up the tooling.

1:28:21So a lot of the, in dataset curation, the problems is not about, usually you work super hard, and then at the end of the day, you make a dataset, you release a dataset, and you're done. But the tools that you developed to actually clean up the dataset, filter the dataset, benchmark the dataset, these are often more valuable for other people who want to do the same job. So the central idea was to make a community and to open source the tools in addition to the data set and then allow other people to try different tooling methods, so going in this data-centric AI direction. So that was kind of one of the central ideas around Datacom.

1:28:55So Datacom is really about building community around data set curation. This is the first time I've seen ClipScore filtering apply like this. is there like, and you also mentioned at the end of your oral presentation that there were other methods, like what filtering methods are you seeing that work really well? It's a whole community of people who are trying a gazillion different tricks, and that's the whole point, right? One remarkable thing to point out is that if you picked a benchmark, you will see performance changing across different benchmarks, but we see surprising correlation of ImageNet, zero shot to gazillion other benchmarks so we have 38 benchmarks and we see that if you do well basically zero shot on image net you're very very correlated in predicting in how good your model is across the board for retrieval for all kinds of very useful things and the community is developing a gazillion type of different methods of data curation that's why we have a leaderboard and we're building a community this is not like a paper and we're done you could you could write like 20 projects of different data set curation it's more like a platform for data set curation evaluations.

1:30:01Do you remember other methods that are doing well? Yeah, so that's a great overview. And yeah, I think specifically people have been looking into designing filtering networks. So rather than using Clip to be the filtering network, like what are some other data sets that we might train on in order to create these filtering networks? What are the differences between a good Clip model and a good filtering model? So these are all kind of open questions that, as Alex was saying, the community will answer by trying a bunch of tricks and methods. Yeah, so you can train new clip, but you can also train new stable diffusion from this.

1:30:38I'm sure Stability is interested in this. Unless you're working on your own diffusion model. Yeah, so the problem is that compute to train multiple stable diffusions is needed. But yeah, we're definitely interested in that direction and we're definitely thinking about that. But you could basically include quality of a stable diffusion as a benchmark and evaluate how you would select a subset of data to improve on that benchmark. I want to talk to Luther. I've been talking to Stella Bitterman from Luther. She's around here. She'll come by. Cool. Any other future directions that you're very excited about?

1:31:19Yeah. We're actually really excited about just the concept of data comp high level. So right now we're pretty excited about NLP and what a data comp light effort would look like in that space. You could extend this approach to audio, potentially video. Although video is tricky for me just because it's so data heavy. And there's a lot of orders of magnitude of different dimensions they could go to. So I don't know what that might look like. Let me tell you about that. so one idea is you can make a data comp for mri images or a data comp for this a data comp for what's the idea what does that mean it means you you fix the model like so classical machine learning as was as much mentioned in the talk classical machine learning says here's a data set image net build a million models and tell me what's the best one right now this the data comp idea flips this on its head right it says here is a big pool of data the model is fixed you only select a subset of the pool.

1:32:25So the thing you're selecting is which images to keep in the pool, then the model is fixed. But you're training other machine learning models to select what to keep. And that's very powerful. So that was in classical AI, if you're doing the data cleaning, the data filtering, that's like the shittiest job. That's like, what? But we're trying to make that a first-class citizen and try and tell you that it's worth to do research because it's not that you will manually sit down and select images from 5 billion or 13 billion. You will be building models that do that. So you can do a data comp for X and we're seeing that from the community.

1:33:06Curious how you became involved with data comp. Yeah, so we have this NSF institute. It's called IFML, the Institute for the Foundations of Machine Learning. And Ludwig is part of our institute. so we were you know having lunch and we were discussing about how do we're discussing about lion right and how to make a better better lion and we said okay instead of just making a better lion which is what we started with let's make let's make it a community where we open the tools so everybody can make a better lion so that was the central idea yeah what happened to the original lion lion is still a great data set that's still public but this is basically building the next generation, yeah.

1:33:47Yeah, yeah, very cool. I wish you good luck. I think this is really foundational work. It's basically the new ImageNet, right? Yeah, yeah. That's 10 years after the original AlexNet moment. By the way, that second speaker who was not introduced was Alex DeMarcus, who is a professor at UT Austin, who just jumped in and chatted. And I do find that it's a very charming element of NeurIPS, is that it's effectively a coming-out party slash hiring party, where all the grad students publish their papers. They all have sponsors in more senior researchers and professors as the secondary or tertiary authors.

1:34:19But their name gets first because then they get all the credit and the citations. And the people who are more senior just kind of stand there and support them. And Alex definitely jumped in and supported them. Just like I saw a bunch of other senior authors supporting their grad students and directing questions to their grad students. Because their reputations are already secure, they have jobs, they're just here to help their interns and grad students. There's a very interesting tension between effectively data sets, papers and models papers the data sets people think that their work is more long-lasting and models people think that data sets work is done and i think you just need both so that's my awkward transition from data comp into lava which is probably the single most interesting visual language model this year as much as people are in love with gpt4 vision it's not open source and we don't really honestly know very much about it.

1:35:10But Lava is open and trainable with a whole bunch of open source models. And together with Datacomp, I think Lava and Datacomp together will provide some kind of template for the next generation of multimodal models to form. So let's check out Lava. I'm Houtian, a final year PhD student at UW-Madison, and I'm on the job market. Today, I'm presenting Visual Instruction Tuning, I'll join work with Chun Yuan, Qin Yang, and my advisor, Yang Jie. So as a background, we, as humans, we can see and reason about the visual world, express and interact with natural language. Doctors read the CT scans and explain their findings to their patients.

1:35:51Teachers teach students with conversations. And we share our life and findings on social media and interact with others. It will be great if we can have a visual intelligent assistant that can reason about the visual world and reflect with language. The closest to work along this direction are image-to-text genesis of models, where the model takes in the image as the input and outputs the text, reflecting its understanding. Such models like JIT, Blip2, and Flamingo have basic visual reasoning capability, while they generally lack the ability to follow very complex instructions or engage in very long conversations.

1:36:27Back in March, OpenAI demonstrated GPT for vision with strong visual reasoning capability. For example, given such an image and the user's request, what's unusual about this image? GPT for vision is able to reason beyond just visual facts. It's able to figure out that the unusual thing is actually the man's ironing clothes when standing on the back of a taxi. It's great, but it's not accessible until very recently, and there's no disclosure on how it works. So if we are able to create an open source model with similar level of visual reasoning capability, it will be great as it allows us to have a deeper understanding of how the models behave, and we can have a joint effort from the whole community to make it better.

1:37:12So, as a starting point, how can we create such multi-model models that can actually follow humans' intent? In NLP, researchers find that instruction tuning allows the model to learn to follow the instructions by fine tuning the model on a small set of instruction and answer pairs, like explaining the human's behavior or movie recommendation and creating such instruction just by letting human writing it is very costly and self instruct proposed to use teacher models like chat GPT to create such instructions by expanding a small set of seed instruction all the pairs to million scale using in context learning and it's affordable and has been used to create open source language models like Alpaca based on the base Lama model.

1:37:52So now the question is how can we create visual instruction following models? And let's start with this basic architecture where we have an image first. We have a visual encoder, which can encode it into the visual features. A cross-model connector that can bridge it to the language decoder. The language decoder also takes in the user instructions and perform the reasoning and output its understanding using the text. So the key is, how do we train this model for following multi-model instructions and how do we obtain such data? The straightforward way would be to use a self-instruct and let's find a multimodal teacher and let it expand.

1:38:29However, if we take a look at those existing teachers' models that were used, they are all text-only and there were no powerful multimodal teachers. And in our paper, we propose to leverage a text-only GPT and we provide image context in the textual format to GPT so that it can understand. For example here, we have an image, and we can use the Coco annotations captions so we can have an image level context which describes what's happening in an image. We can also have the bounding box and object category annotations from the Coco so that we are able to get region level context which provides even more details that may not be captured in the captions.

1:39:07So let's take a closer look on our text only data engine. We will have two parts of the inputs. First are in context examples which are the exemplars that we guide the chat GPT on how they should generate the visual instructions. So we'll have example image, we convert them into the image context in the textual format that we just described. We write the instruction and answers about those visual content in the image. These are the examples for chat GPT to learn from. And then we do the actual inference and we for any image in the COCO training data set, we're able to convert them into textual format using the Coco annotation.

1:39:46And ChatGPT will just learn to generate those instructions and answers following about those image context. We gather the instructions, answers, and also the image to create our visual instruction following data, which is a triplet of image, instruction, and answer. To better facilitate learning, we create three types of responses. First is the conversation to facilitate multi-term engagement, detailed description to train the model to focus on visual details, and complex reasoning to allow the model to focus beyond the visual effects. For example, question what challenges do people face? The model not only needs to figure out like there are back luggages, there are bags, there are SUVs, it also needs to figure out that the challenges as they may not be able to fit all the luggage on the back of the SUV.

1:40:33So we created a Lava Instruct 158K and train Lava. It's a model composed of these three simple components. We use CLIP as a vision encoder. Instruction-tuned language model as a Vicuna as a language decoder. And we use linear layer for the projection. And we find it work quite well because the CLIP visual features already carry great semantics. And a single linear layer is sufficient to project it into a space where the language decoder can understand well. For model training, we use a two-stage training pipeline. Where in the first stage, we pre-train the projector only for the feature alignment so that it's projected into a proper space.

1:41:09And in stage two, we perform end-to-end visual instruction tuning on the generated visual instruction following data set. We train the projector, we train the language model, and if you have limited compute, you can try LoRa or QLoRa or even just the projector only. It can give you decent visual chat performance. After we train Lava, we found several interesting emerging properties. Let's quickly revisit some of the data properties first. So our visual instructions are English only, no human name annotation, and there's no explicit OSIR data. LAVA can have strong visual reasoning capability as GPT for Vision does, where we are able to figure out the unusualness is actually the man's ironing clothes on the back of a minivan.

1:41:51And it's more visually grounded than open source baselines like Blip2 and Open Flamingo. It understands this humorously parodied Mona Lisa with a dog in the same pose, and it's definitely out of distribution. It also has a strong emerging OCR capability where it can recognize NeurIPS 2023 from this presentation slide, and it correlates with the pre-trained language knowledge when asked about who will be interested in this. It will relate and say it's related to artificial intelligence and machine learning. Although our visual instructions are just in English, it's able to perform the reasoning and output text in Chinese and other foreign languages, like it recognizes the French quarter and performs a brief description in Chinese here.

1:42:38So, how can we evaluate our large multimodal models? We draw away inspiration from NLP and slightly modify our data creation pipeline to use a text-only GPT to do the evaluation, where we have the image context in the textual format, we have the user instruction, we have two model outputs, and we just feed all of them into the text on the GPT and request it for the feedback. It will give you a score out of 10 for each of the assistants and also provide you an explanation so that you can understand how the model is behaved. We create a challenging benchmark, Lava Bench in the Wild, which requires knowledge beyond training data, multilingual understanding, and also perception of subtle details.

1:43:19We create a very detailed textual annotation of the image context in those images, and we can feed them for GPT evaluation. And it's not only just for accuracy, but also for hallucination. Since the introduction of Lava, there has been great effort from the community, ranging from data, model, modality, and expanding to different tasks, as well as developing benchmarks for us to better understand the model. The development after June are just too many to fit into the slide. And we as Lava team has also pushing the effort to make it more accessible and expanding its capability in terms of RHF tool use as well as visual prompting.

1:44:03Our improved version of Lava 1.5 was just simple modifications to the data and model. we show that it achieves great performance on a range of 12 benchmarks and it's sample efficient that it only requires less than 1 % of the data that other approaches use. We're able to train Lava 1.5 within one day on a single node. Check out our workshop poster on Friday. So, in conclusion, Lava can reason about the visual world, reflect with natural language. Its design is simple and general that we show that it is possible to adapt the language model to multi-model effectively and efficiently, that we can train it within one day on a single node.

1:44:47Because its design is so simple that we are compatible with almost all the optimizations that are designed for language models for both training and deployment. And it's fully open source. And unfortunately, due to the Wi-Fi network issue, we are unable to do a live demo here, but Lava is able to run on MacBook Air, so I will still do a Lava demo here, and let's try to see. This is an image that I took yesterday here, and I will just say, like, what's this event, and is it popular? It's the Tree of Thought presentation. It's really popular. I can just barely stand in the back. So it will just run for a while, and it says, the event seems to be a conference or presentation.

1:45:33there's a large number of people attending and it appears to be popular because some ones are, lots of them are standing, including me. So I will say like, okay, I attended as well. It is NeurIPS 2023 and the experience is great. Help me draft a tweet. And it will think for a while. I hope it will be optimized further for a faster filling stage, and just attended NeurIPS 23, amazing experience. Packed with knowledgeable speakers and attendees, learned so much and made valuable connections. True. And highly recommend for anyone interested in AI-related fields and some hashtags. I hope you like it, and thank you so much.

1:46:21Please come to our poster session at number 229. Our demo, code, data, model, everything is open source. We are so excited to talk with you more about Lava, and thank you all for coming.

1:46:40If DataComp is an example of what a really good benchmarking dataset paper looks like, then I think Lava is an example of what really good kind of state-of-the-art research on visual instruction tuning and visual language models looks like. It definitely has inspired a bunch of copycats and derivative work in the open-source model space, notably Baklava. And I think there's just going to be a lot more work being done here. Like we're just realizing that we can plug and play these models and train them together in all sorts of ways. And Lava is definitely one of the more innovative solutions of that, that also just solves simultaneously a whole bunch of issues with visual understanding.

1:47:17Here's the poster session Q &A with Hau Tian. Basically, we are trying to create a simple like architecture as simple as possible. So we have a vision encoder just to encode those features, a language model to perform the reasoning, and we use a projection layer, which is a linear layer. We find it doing pretty well to project the visual features to a latent space that the language decoder can understand. And we believe this is because that the visual features of the clip already carry great semantics are in a good latent space. So a single linear is sufficient for it to understand. So is the language model GPT-4?

1:47:55Oh, the language model is Vicuna, something open source, not GPT-4. Right. You can take that off the shelf, but you're training the linear layer. So it will be two stages. In the first stage, we want to train the language model to understand those images. So we train the projection layer only, and this is our stage one. The language model and the vision encoder are all frozen. And in the stage two, we will train the model to follow those instructions. so we train the language model and the projector. To my knowledge, this is the first work that is adding the bounding boxes with the captions. Any difficulty in having the language model understand all those things?

1:48:34Our model does not need to understand bounding box because what we provide to train our model is this, visual instruction bounding data. The model just needs to understand the image and give a proper answer when you give a user's instruction. So this is not something our model needs to worry about, although we do find that the model is able to understand those bounding boxes well. And the key point is that does GPT-4 understand those well, and does text-only GPT-4 understand that well? We find it to be true because what we did is that I also work on some image generation model, and we have a work that we can control the image layout by just providing some bounding boxes.

1:49:16What we did is that we give GPT-4 a caption and we say like, can you generate a reasonable layout for me? And it's able to do that pretty well. So we believe that we do not quantitatively evaluate how it's good at doing that. But it does understand those like layout pretty well. And also it can be used to. And also like from the instruction in general, it does know like which is on the left, which is on the right. Yeah. Did you have to qualitatively evaluate the output of the answers that GPT-4 gave you? Yeah, actually we do not quantitatively evaluate, but we did manually go through some of them when we were developing those data engines.

1:49:58Because we do have some factors to consider in this. So we can change the number of in-contact examples. We can change the way we write those reference instructions and answers. We can also change the actual instructions we use to teach GPT on what is the task. So we did qualitatively iterate on how we design those data engines. And we find this process really is quite rewarding because we do, in this process, understand how GPT thinks and what are the information that we do need to provide GPT for. And then all the bounding boxes that you provide, these are kind of ground truth because you get them from CoCo, right?

1:50:47That's correct. Right, right, right. So this actually ensures that those contexts are perfect, if the human annotators are perfect, and the generated instruction and answers are as good as possible. I just want to ask about the training part. So if I were to take Lava 1.5 and fine-tune it, either full-fine-tune or LoRa or whatever, would you recommend also retraining the projector? I guess it depends on your task. Like, are you considering, like, a different domain? So I want to build off the Lava Plus stuff. So I know that goes into, like, using tools. So it's a little out of scope for this project, but maybe, like, for both.

1:51:29So let's say I want to take a different multimodal instruction following data set. For that part, would you recommend retraining? Yeah, that's a good question. So I would say that if you want to, if the domain, like the image domain that you're going to work on, yeah, medical image, if it is too different, then I would recommend actually go with different stage one training or even just do everything from scratch that you have a biomedical clip. right because that may give you even more benefit but but we do observe that if you train pre-train with Lava's instructions we pre-train with those visual like visual information like it learns to do some reasoning about the like the visual contents and it may be crucial for some like it may be crucial for the visual understanding on different other domains so I guess there will be a trade-off and I guess there will be both pros and cons for training from another domain from scratch because you may lose the benefit that you get when pre-training on lava on how to localize those objects.

1:52:42So I guess you would need some more like experimental evidence on making the proper decision. So is it fair to say that unless the domain is super different like x-rays, maybe it's fine to just... Yeah, I think it's totally fine. And I I guess it's better to use the instruction tuned version because it has so many vision knowledge injected into it. Okay, and then, sorry, one last question. For stage 2, like let's say I want to fine tune on my own thing, is the roughly 160k number of examples a good target to hit? Like, do you have recommendations around how big that data set should be? I guess it also depends on how different the task is and also how bad the model is performing on that task because I can give a brief example on one of the experiments we have done.

1:53:33So there's a task that we can train the model to generate stable diffusion prompts, for example. Basically, it's kind of captured in some style we want. And because the LAVA is already able to understand those visual attributes, the content very well, it's just a form of reorganizing the style it responds. So we find even 100 examples is sufficient. Yeah, 100. And we just use 100 examples, and it does the work decently. I see. Yeah, it's just a form of changing stuff. But if you're trying to do some very different reasoning tasks that Lava is not good at, I guess you may need more. Also, I think 10K generally saving is enough.

1:54:19What? 10K, generally saving. Yeah, yeah, yeah. I guess 10K. Or if you want to make it safe, I guess maybe 50k is at most. Can I ask some questions? Yeah, of course. I just want to understand how important this vision encoder is. Have you ever tried to remove the encoder entirely and use the bounding box here as the input of whatever language model you are using and just do the same task? So I guess the key point here is that if you want to remove the encoder completely and just use the bounding boxes as input, there will be one question like how are you going to get those bounding boxes? And second is that what if the user asks you about the text?

1:55:04Like are you going to also have an OCR engine? And what if the user asks about something else? For example, like the attribute. and it just like if you if you think of this like having an end-to-end model will be make it much more easier and much more generalizable to extend to different types of the inputs and the user's instructions so and also like because now you need some other model to generate those bonding boxes tags all of those things like it's i feel that it's good if we can have those models to enhance the capability but you do have a model that can are really trained with vision and you really understand what's happening this image it can better coordinate those information yes yes we have tried to offer is the vision encoder and we find it quite useful for some of the text but not for the other so specifically if it's just asking about what's the attribute what's the object.

1:56:06Those kind of tasks, it does not matter much. But there are two kinds of tasks that unfreezing the vision encoder really matters. One is that it's not necessary about the semantics. For example, I'm asking whether this line is straight. Those kind of tasks which require you to understand the low-level details or the low-level details really matters. it's one of the thing that and we also have another work VIP Lava where we try to train the model to understand the visual prompts so basically the visual prompts we mean that can we just like use some scribble to circle some objects that we want to ask about instead of necessarily trying to describe it very clearly like making the model to understand what we are curious about so for that in order to correctly identify those scribbles and those tiny lines, it requires you to somehow unfreeze the vision encoder to properly unfreeze or use some earlier layers, which still preserves those information.

1:57:11I'm curious about the backstory behind this whole thing. How did you get started exploring multimodality and your inspiration? We have been working on vision language. Our team has been also working on vision language. Your team, is it a lab? Is it his team? I see. Chun-Yuen from Microsoft and my advisor. We have a collaborative effort on this. We have a series of work on vision language. And although I'm not having tons of years' experience on vision language, but we do see that in March, we see Vicuna, which do make us very impressed about the performance it can have. For the size. Yeah, for the size and also for the open source.

1:58:00And we believe that it's possible for us to create a visual reasoning model that is purely open source with similar level of capability. And we believe that with open source, we are able to have a joint effort from the community to make it much, much better. Yeah, it was cheap, right? You trained for like eight hours. Yeah, eight hours or Lava 1.5, one day on a single node. That means everyone else can do it too. Yes, not everyone else, but most people. Most people. Thank you very much. So super interesting and notable work on the lava model. I guess someone should try to hire him. But I guess the next segment we're going to explore is the prompting segment, quote unquote.

1:58:44And there is a surprising number of prompting papers here. I'm not sure that many papers should be represented at NeurIPS but where else are they going to present? I don't really know but anyway, so there was a whole channel or track just to Chain of Thought that blows my mind to me and I do think that that is appropriate and I do think that the techniques here are innovative it's impossible to cover all of them I actually talked to Noah Shin from Reflection remember Reflection? as well as a whole bunch of others but probably the most representative one was the tree of thought paper. So here it is.

1:59:19My name is Shen Yu. I'm from Princeton. I'm very excited to talk about tree of thoughts. It's a joint work with my colleagues from Princeton and Google. So we all know language models and large energy models. Language models were invented to generate text, token by token, and left to right. But now they are used to solve an increasingly wide range of problems using scale-up models and prompting techniques like chain of thought. So here is an example, like you can break down complex calculation into steps and it will make it solve like problems I cannot solve in steps. So the question is, can those language models one day become a general problem solver by keep scaling up and using autoregressive inference?

2:00:03Or there are some fundamental limitations? So to answer the question, let's take a look at a very simple example, this game of 24, where the rule is you are given four numbers and you have plus, minus, divide and multiply operations and you need to combine those four numbers to obtain 24. So one example is if you are given input 2, 9, 10 and 12, what you can do is you can first multiply 12 and 2 get 24, then 10 minus 9 to get 1, then 24 times 1 to get 24. Okay, so it's not a really hard game. Now you give it a new input 4, 5, 6, 10 to GBT 3.5 You let it solve the task It will first try to multiply 10 and 6 to get 60 Then divide it by 5 to get 12 Then 12 times 4 to get 48 Then to make it up it will say it's 24 And then call it a day So it's a hallucination You might argue that if you have better models Or better prompts it will solve this But even if you use GBT 4 with 5 shot 5 example is in the OT prompt you'll only get 4 % task success.

2:01:13So why is this easy task so hard for language models? So if you look at the initial token generation, 10 and 6, 10 times, because those language models are making local and token level decisions, one by one, left to right, those initial decisions are really hard. Even for humans, we don't know whether the first token should be 10 or 6 or 5. We don't have like pre-trained intuition, right? We have to play the game to have a better sense. Worse still, once you generate those run token at the beginning, the task is already filled in that you cannot really complete the whole triadary in the CLT format and be right.

2:01:53So by this very simple example, what I want to show is there is something about auto-regressive inference that is lacking mechanisms for deliberate reasoning. So it's even true for biggest, strongest language models like GPT-4, and the reason is quite simple. Just like Ben's talk mentioned, right? So for the COT to work, you really need strong local signals to guide every step through those local decisions. And just to draw analog, right? Imagine if you have a robot that's trained only on successful navigation trajectories, and it's only trained to predict the next move. and then you put it into a new maze and then it's very hard to explore.

2:02:36So how do we solve this issue, right? So in this work, we took inspiration from human cognition. In his famous book, Thinking Fast and Slow, Daniel Kahnman proposed that our cognition has two parts. We have a fast and automatic system one that's handling everyday tasks, like riding a bike, and we have a slow and deliberate system two that's imposing control and intervention over system one for harder tasks, like designing a plan. So if language models automatically auto-regressive inference is similar to this spontaneous but error-prong system one process, maybe we can impose some kind of control algorithm on top of it to get system two reasoning.

2:03:18And tree search is naturally the choice, which is also one of the oldest ideas in artificial intelligence. For example, the Y. Simmons general problem solver in the 1950s. However, doing search in this reasoning space is not trivial. because traditionally, you know, if we search in classical games like chess, we often have like a small fixed set of next moves so that we can design or learn search heuristics. But if you want to search in open-ended reasoning, the next move can be arbitrary tasks, which is really hard to enumerate or evaluate. So the idea here is now that we have large language models, we can use them to start generating and evaluating next moves.

2:04:01So from the previous two slides, you have seen what's the problem of large language models and what's the problem of classical search. And the hint of combining them might lead to a better result. And that's true. So we proposed tree of thoughts. It's a general method for combining language models and search algorithms for deliberate reasoning. And to solve a problem, you need four parts. So first, you need to define what is the search space or what is the thought space. Then you need to generate and evaluate thoughts using language models. And you need to combine that with a search algorithm to explore and maintain thoughts.

2:04:33So I'll use the simplest example, which is Game of 24, to explain each part. Okay, so what is a thought, right? That's not a question in Chain of Thought because everything is coherent and you don't have to split it. But it's a very critical thing in Tree of Thought. So here we define a thought as a coherent piece of text as the next move in the reasoning game. And if you think about Game of 24, right, there are two extreme choices. On one extreme, you can treat each token as a thought, right? Then it will be very easy to generate each thought. But as I explained before, it's very hard to evaluate whether 10 is a good thought or 13 is a good thought.

2:05:10On the other extreme, you can treat the whole reasoning as a thought. You generate the whole thing, which will be very easy to evaluate. You just look at the end if the number is 24. But if you can generate that, the problem is solved already. So it's very hard to generate. So in this game, naturally, the choice of thought is something in between. We can use each intermediate equation as a thought, so that it's relatively easy to generate and evaluate thoughts. And this is really a problem-specific trade-off design. So for different problems, a thought can be a token, can be a word, can be a sentence, can be a paragraph, and so on.

2:05:46So once you have defined what is a thought, it's easy to generate that with a language model. So here is a simple prompt. You have one example of what's the input and what's the possible thoughts. then you give it a new input and the language model just can generate a new source, right? So here each new line is a new source of how to continue the reasoning.

2:06:07Once you have those source, right, you want to give them a value so that you can search. So here what we do is we give this problem of example where, you know, for the remaining numbers if the language model can simulate within a field trial and reach 24 then a high value is given. If not, depending on whether the numbers look reasonable or not, a medium or a low value is given. OK, so the previous three examples are the in-context examples. Now for this new input, 566, the language model try one round, 524, and sure, it's a high value. For turn 1313, it will try a few rounds, and it will fail.

2:06:47And these numbers look too large, so it's impossible, so a low value. So for something like 559, it try a few rounds. it doesn't work, but the numbers look reasonable, so likely, so a median value. But here, actually, 559 is not actually possible to reach 24. So it's important to know that, just like any search heuristics, here the value does not have to be perfect, and it just needs to bias the search to what problems and directions. Also, here, the prompt uses common sense reasoning and simulation, but you can really design different strategies for different problems. It's really flexible. So lastly, you can combine them together with a tree search algorithm.

2:07:26Here we use breadth-first search, which is the simplest algorithm. You have a depth of three, and you have breadth from one until five. And the idea is very simple, right? You have the input, you generate a bunch of thoughts, you evaluate them. You only keep the top choices. So it's like a thought-level beam search, right? And you keep doing that until four numbers become three numbers, three numbers become two numbers, and two numbers become one number. And you're succeeded if the only number is 24. So while CLT only achieved 4%, TLT with a breadth of 1 already leads to 45, and a breadth of 5 leads to even higher, 74.

2:08:02We can also use a similar idea for different algorithms and for different problems. So for example, for crosswords, right? Suppose you have five clues horizontally and five clues vertically. What you can do is you can generate a bunch of thoughts, evaluate them, and then gets preceded only with the most promising choice. So that's a depth first search, or best first search. And you can keep doing this until the language model realizes this board is no longer solvable. Then what you do is you prune the subtree, and then you backtrack. So you move on to this, but maybe this is still not solvable, so you will move on again.

2:08:38If none of the thing works, then you go back one level back, and then you try again. So it's a very classic depth first search. and here are the results. COT reached 1 % and we got 20%. But if you don't have pruning and backtrack, then it again goes to 5%, which shows pruning and backtracking is very important. So in our paper, we have these two games, but we also have a natural language task that's trying to write creative stories. And the intuition is also very simple. So if you're a good writer, you don't just write token by token. You will deliberately plan what are the possible plots, you will choose, compare between them, and you will select them.

2:09:18So similarly here, the language model will write a bunch of diverse plans, then self-evaluate what is a good plan, then proceed with that. And you can do this kind of search for writing, and then humans will find it more creative than the CLT writing. But the writing is too complex and long, so I cannot display it here. so what I want to say is across those different tasks with very different reasoning challenges the modular design of TOT allows us to have very flexible ways to generate, evaluate and search thoughts across very general and diverse tasks and we're doing so in a very systematic framework and achieve very good performances without retraining any models so it's very convenient to use so we believe this is an initial step toward connecting old insights and new frontiers of AI.

2:10:13So here, TreeSearch, one of the oldest ideas in AI, helps language model do more deliberate reasoning, while language models help search, provide search with very flexible and general purpose, powerful heuristics. So we have these follow-up efforts trying to connect cognitive architectures to language model-based agents. So those are systems that does not just reason internally, but also interact with the external world and learn through such interaction continuously. So it's like autonomous agents. So we have this follow-up paper called COALA, Cognitive Architectures for Language Agents. I highly encourage you to check it out.

2:10:54And I thank my co-authors. Thank you guys for listening. Check out the poster today. And happy to chat. Thank you so much.

2:11:05I do like it when people come with a general enough model that you can customize it and specialize it to recover smaller effects that other people have found so you can, from the Tree of Thought paper recover something like the Backspace Token Model or recover Skeleton of Thought, Chain of Thought whatever a thought, I don't care, I can't keep track anymore anyway, so I caught up with him at his poster session and here's a bit of our chat You can hold it up. All right. So the TLDR of this paper is very simple. Large language models and search, they complement each other. So what's wrong with just using large language models without search?

2:11:42Is everyone familiar with chain of thought? Okay. So suppose you're trying to solve this game of 24, where given four numbers, you try to combine them to get 24. Okay, so you can give GPT-4 this task instruction. You give it a couple of CLT examples. but the performance is really low, it's 4%. Why is it so hard, right? So that's because this problem like intrinsically needs exploration. So let's take a look at the initial example, right? So the model is making local token decisions, right? It first generates 10, then it generates times, then it generates 6. But it's very hard to decide those initial tokens, even for humans.

2:12:29you don't really know whether the first token should be 10 or 5 or 6 or are they equally good. That's really hard to decide. But what's worse is once you decide to run tokens at the beginning the task is already failed. So in this particular example if you generate 10 and times this task is already failed. Because no matter what times 10 you cannot get three numbers remaining to reach 24. for. So the intuition is that autoregressive inference is like you're keeping making those local token decisions one by one left to right without look ahead, without backtrack. And it's not very robust when you don't have good local signals to guide through those kind of process.

2:13:17So another analogy would be suppose you're training a robot that's trying to navigate misses. If you only train them on successful trajectories and only train them to predict the next move, and you do this local imitation and you put them in a new maze that requires exploration, then it probably won't solve the new maze. So obviously some kind of search is needed. But why this is a 2023 work, given that search has been around since 1940, 1950s? That's because classical search problems like chess, they have a small fixed set of next moves, what we call the search space. That makes it easy to define, to design or to learn search heuristics to guide the search.

2:14:11But here, for this kind of open-ended reasoning, the next move can be anything. It could be a token, it could be a sentence, it could be a paragraph. and it's impossible to enumerate this huge space or to design evaluations. So the key point here is you want to really define what is the search space first. You can consider two extremes first. So on one extreme, you can define SOT as the next token. Then you will be searching in a tree of tokens, something like BeamSearch. then the problem is it's very easy to generate tokens, right? But it's very hard to evaluate tokens. You don't really know whether 10 is good or 13 is good or whatever.

2:15:02On the other extreme, you can define thought as the whole reasoning. Then, it's very easy to evaluate the thoughts. You just look at if the final number is 24 or not. but in this bandit it will be very hard to generate a good thought, otherwise the task is solved already so in this case it seems like the right balance is you define each of the intermediate steps of the thought so you can do something like you tell language model, here are some numbers come up with some different ways to combine two of the numbers you can generate a bunch of thoughts and for each of them you can do something like this you can say, try a few runs, can you reach 24 If not, try to design a value based on that.

2:15:50So within three trials, if you can already reach 24, then this thought has very high value. If it couldn't reach 24, but maybe it could reach 25 or 26, maybe a median value is given. But if this is something like 1, 2, 3, and then you can only reach 6 or 4, then maybe a low value is given. So value is not perfect, and it does not need to be perfect, just like any search heuristics. You just need to bias the search towards promising directions. So the point is, once you define this search space, you can generate and evaluate next moves using large-lander models, and then you can systematically maintain them using the trade search algorithm.

2:16:32And we show across diverse tasks, this significantly improves the task performances, and it's very easy to use. You don't need to train any new models. Everything is done with GBT4. Pretty elegant. Good. So I like that comparison to beam search. Yeah. This is like a level of abstraction above that with the unit, the atomic unit being a thought. Yeah. A thought here you illustrated being an equation. But here you have an equation, a clue word, like here are the examples. Yeah. Do you have a planning stage in order to plan out the thought steps, right? Like here you have thought steps of three, thought steps of five to ten, thought steps of one.

2:17:13Like, usually when people design agents, they'll have, like, a planner. But I don't see a planner here. That's a great question. And you will notice here is for those two games, the search steps are kind of homogeneous. Because every step, you're just trying to come up with a new equation or you're trying to come up with a new clue. So in this case, you don't really need planning. You can just use one generation prompt, one inversion prompt, and use that across different steps. But for something more complicated where for each third step you might do different things, then you probably need to plan ahead and maybe design different prompts for different kinds of generation and different kinds of evaluation.

2:17:53Got it. So do you also see this being able to be combined with self-consistency? Because in a way your judge is a self-consistency. That's a great idea and we did that. So the point here is, in this creative writing task, it's like what we do for evaluation is, here's a task instruction. Here are some of the plans. Think step by step, what is the best plan, and come up with an ID, right? So if you just do this one time, you'll just get one vote. It's very noisy. So you can apply something like self-consistency, right? You can ID, do like 10 different votings or 100 different votings. And then the evaluation will become more faithful.

2:18:42And that's kind of hyper-parameter you can choose. It's like if you want better performance, you can spend more money and try to do that more. It's like a post-generation layer. It's like a stepwise democracy, I guess. Okay, so one more question about just in general, Princeton NLP. How is it organized? What should people know about the Princeton program? Because I feel like you guys are very productive. And how are you so productive? What's the backstory to three of your thoughts, maybe? I think one thing that's good about Princeton is it's a kind of small school. I've been there. It's not that small.

2:19:19I mean, compared to Harvard or MIT. You have a lot of interdisciplinary kind of collaborations. So I did this with cognitive science professors. I think this kind of idea across different fields is very important. So usually in NLP, we don't consider tasks like that. Yeah. That's classical search, right? Yeah. So I think it's very useful to combine ideas from different fields, and that could be a way to promote, come up with new ideas. Yeah. Are a lot of people asking you about Qstar stuff? No. No comments? No comments. Okay, well, thank you very much. This is a great paper. So perhaps one paper that made a bigger splash than Tria thought earlier in this year was Toolformer, where we started really considering the myriad number of ways that we can train language models to use tools.

2:20:07So here's the Toolformer oral. Hi, everyone. My name is Jane. I'm a researcher from Fair Labs at Meta, and today I'm super excited to be presenting to you Toolformer. Language models can teach themselves to use tools. And the reason we might want language models like ChatGPT to have access to external tools is exemplified by these three queries. In the first two cases, I've asked who is the current president and what day of the week is it today. And ChatGPT basically says it doesn't have real-time data or access to current time or date information. And in the final query, I've asked to do a simple set of computations, but ChatGPT unfortunately hallucinates an answer that's about 300 off from the real answer.

2:20:47And what we really could have used here is access to external tools. For example, a QA system that has up-to-date information, a calendar tool which has the timer date, and a calculator tool which is designed specifically to do these simple computations perfectly. And so for Toolformer, we have five tools at our disposal. We have a QA system with up-to-date information. We have a Wikipedia search tool which is able to search Wikipedia. We have a calculator tool, a calendar tool which has the current day of the week and the date, and we have a translation tool which takes in text and puts it back into English.

2:21:22And so in choosing these five tools, we really wanted a set of tools that is not only diverse, but is also going to be likely useful to the language model. And what we want to train a model to learn is not only which of these five tools to use, but when to use that particular tool and how to use that tool all on its own without human annotation. And the way we do this is by taking natural language text, like Pittsburgh is known as the Steel City, and augmenting that text with API or tool calls. So for example here, a useful API call would be to the QA system with the question, what other name is Pittsburgh known by?

2:21:59And this is useful because it's useful in anticipating the remainder of the text, which is the Steel City. And we represent an API or tool call with natural language. We should do square brackets followed by the tool name. And then in round parentheses, we have the input to that tool followed by a right arrow, which is followed by the output of the tool with that query. And with that, the steps to creating toolformer is pretty simple. In the first step, we want to create a new training data set augmented with these API calls that I just showed you on the previous slide. And in the second step, we want to fine-tune GPT-J, our base model, on this new data set.

2:22:37And this fine-tuned model is the model that we refer to as toolformer. Now to create that training data set, we have three simple steps, which I'll get into in just a second. But first we want to start out with a standard language modeling data set like CCNET. And the reason we want to start here is because we don't want to disrupt any of the core language modeling capabilities that the model may already have. And so in using a data set or something similar to what it's seen before, we minimize this risk as much as possible. Okay, so let's go into the first step, which is to generate API calls.

2:23:11And to do so, we show the model a simple prompt. We say, your task is to add calls to a question answering API to a piece of text. The question should help you get information required to complete the text. And then we explain the format of the API call that we want, and we show it a couple of examples. I only have one example here, but we would put as many examples as can fit into the context window. And then we show the input that we actually want to do inference on and we let the model generate. And here I'm only showing you the question answering API prompt, but you can imagine that we do a very similar thing for the rest of the four tools.

2:23:48Okay, so let's look at a couple of generated API call examples. For the input, Pittsburgh is known as the Steel City. And here the model has generated in which state is Pittsburgh, what other name is Pittsburgh known by and what is the second city in Pennsylvania. And so from these generations you can see that we get a mix of relevant API calls, non-relevant API calls, and also some that don't make a lot of sense like the last one. And now for the second step where we actually try to execute those API calls. So what we do here is we take that natural language string, we parse it for the input parameters, we send it to the relevant tool, and we get an output from the tool.

2:24:28Now, using those outputs, we want to put them back into the embedded API call, and we indicate this with a right arrow followed by the output. And this is also the step where we would filter out generated API calls that are ill-formatted or don't actually return a result from the tool. And additionally, we also want to filter out API calls that aren't actually useful to the model. So the way we want to think about usefulness, if it's useful for anticipating the remainder of the text, as I showed you earlier. And the way we quantify usefulness is through model-based perplexity. And perplexity is basically the negative log likelihood of the remainder of the text given the prefix of the text.

2:25:07So basically you want the lowest perplexity possible because you want the model to be least perplexed about what it's about to see. So here we evaluate perplexity under three different settings. The first setting is where we don't have any API call. So here the prefix would just be Pittsburgh is known as. In the second setting, we have the non-executed API call, where we have the API call, but we're not actually going to put the result from the tool yet. And then finally, we have the full executed API call, where we have the API call and its corresponding output. And intuitively, what we want here is for the perplexity for setting C to be much lower than either A or B, because not only do we want the generated API call to be useful, but we also want the results from the tool to be really useful.

2:25:58So this is exactly how we evaluate usefulness. It's the minimum of the perplexity of either under A or B minus the perplexity of C. So we want that difference to be as large as possible. And here we have a pretty sizable usefulness score of 1.3, which is pretty good. But to give you more context, Here's another example from the calendar tool. It says the WL will be open on Friday. And the calendar tool tells us that today is Thursday, March 9th. And from this, we can kind of infer that Friday is going to be March 10th. So this gets a high usefulness score of 2.11. On the other hand, we have this example from the calculator tool.

2:26:39The model has seen these two numbers, 85 patients and 23%, and it thinks maybe the ratio is going to be useful. But unfortunately, that's not the case, and it gets a low usefulness score of negative 0.02. So this would likely be filtered out in our final third step. Now here I'm showing you the number of examples that remain after this filtering process for two different kinds of thresholds. We have in light blue, 0.5, and in dark blue, we have 1.0. And obviously you're going to get a lot more examples left over if you use a less stringent threshold, 0.5. But the other thing you can see here is that we have the most number of examples from the Wikipedia search tool, whereas for calculator and machine translation, we have the fewest number of examples.

2:27:21And now what we do here is we cap the number of examples per tool at 25 ,000, and we put it all together in one big data set. And with that data set, we fine-tune our base model, GPT-J, and this fine-tuned model is what we refer to as toolformer. Now, to evaluate toolformer, we want to evaluate on a range of tasks where we think at least one of the tool is going to be useful. So we have fact completion and question answering. We also have math computations and multilingual questions, where the context is given in English, but the question can be in a different language. And we also have temporal questions, like how many days is it until Christmas, where you need to know the current time or date in order to answer the question.

2:28:05Now here are the results for those five tasks. We have three different models. We have GPT-J, which is the base model, Toolformer, and GPT-3, which is a 175 billion parameter model. And what you can see here is that in almost all cases, Toolformer is outperforming GPT-J, but it's also outperforming GPT-3, even though it's about 30 times smaller than GPT-3. And an exception to this is the question answering task, where we actually disabled the QA system. and this is because there's a lot of overlap in the training set of the QA system and our evaluation tasks, so we thought this would be too much of an advantage if we enabled that tool.

2:28:41The second anomaly is the multilingual task where we don't see a lot of benefit from the translation tool and we think this is likely because GBTJ has already seen a lot of multilingual text and isn't getting a whole lot of benefit from actually using that tool. But regardless, we see that toolformer is either on par with GBTJ or outperforming GBTJ. And the second thing that we want to look at is whether or not small models can effectively use tools. So in other words, is there a minimum size requirement with which models are able to effectively use tools? So to investigate this, we applied the same kind of pipeline to the family of GPT-2 models.

2:29:21So there are four of them, and in total we have five different models at various sizes, which I'm showing you on the x-axis. On the y-axis we have model performance. And in blue, we have Toolformer, and in red, we have Toolformer disabled, where we use constraint decoding to prevent the usage of tools. And as you can see, in the smallest two sizes, we don't see any performance difference between Toolformer and Toolformer disabled, meaning that Toolformer is not able to make use of those five tools to its fullest. But once we get to 775 million parameters, we see a performance gap emerging, and this gets bigger and sustained for the rest of the sizes.

2:29:59And this is a similar thing that we see with the math benchmarks. It seems that tool usage is really emerging at 775 million parameters. For the question answering benchmarks, we don't see this as clearly, and we think that maybe this is likely because the QA system and the Wikipedia search tool are easier tools to use, and so you don't need a more capable model to be able to understand how to use it effectively. And finally, we also want to revisit the question of whether or not toolformer is a good language model. We originally used a data set CCNET because we didn't want to disrupt any of the core language modeling capabilities.

2:30:34And so now we revisit that question by looking at perplexity on a held out set of Wikitext and CCNET. And here we have three different models, GPTJ, GPTJ further fine-tuned on CCNET, and to a former, which is further fine-tuned on CCNET, augmented with those API calls. And what we find is that the perplexity is pretty much on par with the base model and the further fine-tuned one. We don't see a whole lot of difference. And so we feel pretty encouraged that even though this data set may look a bit unnatural with these API calls, it doesn't actually harm the core language modeling capabilities here.

2:31:09So thank you for listening to this talk. Please check out our paper at this QR code. We have a poster in the next poster session. We are poster number 332, and I will be there with Roberta. Please feel free to reach out to any of the co-authors and me. I'm happy to take questions now or later. Thanks. When I look at all the relevant papers for AI engineers this year, there's the chain of thought papers and the tool used papers, two of which we just covered. But something that I think incorporates all of them and then adds a few ideas that are unique and notable to them is the Voyager paper from NVIDIA.

2:31:50And even though it was released in the first half of the year, people are still talking about it today. It's still shaping people's mental perceptions of how they want to build their LLM architectures. It was somehow not accepted for posters or oral sessions at this year's NeurIPS. It's kind of a mystery as to why. I did chat with Jim, and I'm still not really sure what's going on there. But it would have been my vote for best paper because it's so foundational and established such a strong baseline for everyone else to build on top of LLMs. And anyway, so there is some workshops, presentations about Voyager with the first author.

2:32:27So here it is. My name is Guan Zhi Wang. Currently, I'm a 30-year PhD student at Caltech. I'm also a research intern at Vidya. I'm very happy to present Voyager, an open-ended embodied agent with large language models. This year, GPT-4 came, a large language model that's so good at coding and long-horizon planning. So we built Voyager the first large-ranger model powered platform learning agent When we set Voyager loose in Minecraft, it is able to play the game for hours on end without any human intervention The video here shows snippets from a single episode of Voyager So it explores the terrains, mines all kinds of materials, fight monsters, craft hundreds of recipes, and unlocks an ever-expanding treat of skills If you want to use the full power of GPT-4, a central question is how do we stringify things?

2:33:25In other words, how do we convert this embodied environment with multi-model observation and action space into pure text? We need a magic box. And thankfully, the enthusiastic Minecraft community already built one. It's called Manflare, a high-level JavaScript API that's actively maintained to work with every Minecraft version. The beauty of Manflare is that it has access to the game state surrounding the agent like the nearby blocks, animals, and enemies. So we effectively have a ground truth perception module as a textual channel. Now that we convert everything to text, we are ready to construct an agent algorithm on top of GPT-4.

2:34:04And on the high level, there are three components. First, a coding module that writes JavaScript to control the game bot. It's the me module that generates executable actions. Second, we have a code base to store the correctly written code and look it up in the future if the agents need to recall the skill. In this way, we don't duplicate coding efforts and achieve a form of learning without grading design. Third, we have a curriculum that propose what to do next given the agent's current capabilities. So we wire them up together, we get a loop that drives the agents indefinitely and achieves something like lifelong learning.

2:34:44So let's do me in the center module. We prompt GPT-4 with documentations and examples on how to use a subset of the ManFlyer API. The GPT-4 writes code to take actions given the current assigned task and because JavaScript runs a code interpreter, GPT-4 can define new functions on the fly and run it interactively. But the code that GPT-4 writes isn't always able to get it right at the first try. We develop an iterator prompting mechanism to refine the program. There are three types of feedback. First, the environment feedback like what new materials did you get after taking an action? Second, the execution error from JavaScript interpretor like variable undefined error.

2:35:31And we have another GPT-4 that provides critiques through self-reflection from the agent's own states. So these components help the agent refine the program effectively. I want to show some examples of how the critique module provides feedback on the task completion progress. In the first example, the task is to craft a spyglass. So GPT-4 looks at the agent's inventory and decides that it has enough copper but not enough amazes Second task is to kill three sheep to collect food So each sheep drops one unit of white wool, but there are only two units in the inventory So one more sheep to go Last example killing a zombie drops a unit of rotten flesh which is in the inventory So GPT-4 determines that the task is successful and moves on.

2:36:25So this critique procedure is repeated until the task is deemed successful or hits the time limit. Now moving on to the second part. Once it implements a skill correctly, we save it to our persistent storage. So think of it as a skill library that's authored purely by GPT-4 through trial and error. Then the agent can retrieve the skills from the library when facing similar situations in the future So it doesn't need to write them again. In this way, Voidre improves itself as it experiences more and more in Minecraft Let's step a bit deeper into how the skill library is implemented. So this is how we insert a new skill First, we use GPT 3.5 to summarize the program into plain English So summarization is very easy and doesn't need GPT-4.

2:37:19So we save some money here. Then the embedding of the summary becomes a key and the program becomes a value which we insert into a vector database. We find it better to embed the description instead of the raw program because it's more semantic and improves the retrieval.

2:37:44Now, when Voyager is faced with a new task, let's say, craft iron pickaxe, we use GPT 3.5 to generate a hint on how to solve the task and combine it with world state as a query content. Then we do the embedding and retrieve the top five relevant skills from the skill library. So Voyager is free to directly use one of the skills as is, or interpret it among the file, or rewrite one from scratch. In this way, we maximally reuse the old experiences. Think of it as an in-context replay buffer in the reinforcement learning terminology.

2:38:27Now moving on to the third part. We have yet another GPT-4 that propose what task to do given its own capability at the moment. The curriculum has an unsupervised objective, which is to maximize the number of novel items that the agent obtains. There are two key insights here. First, it's kind of curiosity-driven exploration or novelty search in prior literature, but implemented purely in context. Oh, sorry. And second, it's a situation where curriculum that naturally gets progressively harder over time or without any manual prescription from us. So let's go through our working example together.

2:39:12The agent finds its hunger bar dropping to one out of 20, so it needs to find food. Now it senses four entities nearby, a cat, a villager, a pig, and some wheat seed. So it starts an inner monologue. Do I kill the cat or villager? Bad idea. How about the wheat seed? I can grow a farm but it's gonna take a long time. So sorry piggy, you are the chosen one. It checks the inventory and retrieves an old skill from the library to craft an iron sword and then starts to learn a new skill called hunting. Now we also know that Voyager isn't vegetarian unfortunately. So putting all pieces together, we have an iterative prompting mechanism that refines the program by self debugging, a scale library as an in context replay buffer, and an automatic curriculum as in context curiosity driven exploration.

2:40:10This is Voyager's no gradient architecture where we don't train any new model or front train any parameters. It allows Voyager to self-bootstrap and perform lifelong learning in an open-ended world.

2:40:25So these are the tasks that Voyager happens to pick up along the way. We didn't pre-program any of these. It's all Voyager's idea. The agent is forever curious and forever pursuing new adventures.

2:40:40We've done a lot of systematic study for Voyager and here is the quantitative learning curve. Well the The X axis is the number of prompting iterations and the Y axis is the number of unique items obtained by each agent. We compare with three baselines, React, Reflexing, and Auto-GPT. All of these are no gradient.

2:41:04All of these are no gradient architecture on top of GPT-4. React is a very simple reasoning and acting loop and Reflexing is built on top of React with self-reflection. we see that both struggle to make progress beyond the basic wooden tools. And AutoGPT is a popular software repo. It combines React and a task planner that they compose an objective into sub-goals. It makes more progress but it's very slow. And this is a voider. We are able to obtain three times more novel items than the prior method and unlock the whole tech tree significantly faster from wooden to stone to iron to diamond. The blue curve here is an ablation without scale library, which plateaus after a while.

2:41:51So basically the scale library is very essential for Voyager's lifelong learning capabilities.

2:41:59Here are two births I've used of Minecraft maps. So these circles are what the prior method explore, given the same prompting intuition budget. You can see that they tend to get stuck in local areas. Voyager is able to navigate distance two times longer compared to prior works. It has to visit more diverse terrains in order to find more novel items quickly. Finally, one limitation is that Voyager does not currently support visual perception because GPT-4 is tax-only when we were developing Voyager, but there's nothing stopping Voyager from using a multi-model model to achieve more impressive tasks.

2:42:34And here we demonstrate that given human feedback, Voyager is able to construct complex 3D structures in Minecraft, such as a house and a nether portal, we basically used a human to replace the critic module of Voyager and provide 3D spatial otherwise. So to build very complex structures, we definitely need some full-blown multi-model models, and I will leave that to future works. This is Voyager's website at voyager.mandojo.org. We open source everything, including the environment, algorithm, prompts, and pre-trained skill libraries. Finally, I want to acknowledge all the team members for either this work would not be possible without their help.

2:43:13So please feel free to reach out if you have any questions. Thanks. I think the last component of agents, apart from chain of thought and tool use that I wrote up in the Anatomy of Autonomy right up in April is the need for better planning. And I think one of the most interesting or challenging pieces, depending how you look at it, of NeurIPS is doing poster diving, where instead of going to all the oral sessions, which have been curated by track committees and all that, you just go and walk the halls and look for posters and look for papers and people that are underrated have been overlooked.

2:43:55And in fact, the original Attention is All You Need Transformers paper was one such paper where they were just a poster only paper, apparently. From walking the halls in the poster sessions, my pick for underrated paper was Ida Momenajad from Microsoft Research with COG eval. Ida was very confident and professorial in her presentation, made it engaging, made it a quiz. Some parts of the quiz are visual, so if you're listening along and you want to solve it alongside us, you should probably pull up the show notes and check out the graphs that I'm going to paste inside of the show notes. But otherwise, she just made it very engaging for people to follow along like i'm not kidding there was a group of like 10 20 of us way back in the halls in the poster sessions where a lot of people don't really end up going and we're just like half an hour while she was giving her impromptu lecture about cog and i do think that this is notable because it is potentially a quantifiable benchmark for reasoning and planning capabilities that currently all the language models don't do very well.

2:44:57And framing it as a graph problem helps us generalize to all sorts of reasoning, planning, and search situations. And I just like that it was really well presented. This is obviously a benchmark paper, so there's no solutions proposed, but she has another paper that she's working on that has some of her solutions. So LLMs are ubiquitous, and a lot of people claim that they can plan or they're going to plan to take over the world. But first things first, can they actually plan? I have 15 years of experience working in reinforcement learning and cognitive science and in neuroscience, evaluating planning in humans and brains and reinforcement learning models.

2:45:31So I thought, okay, let's apply that. In order to accurately evaluate whether a cognitive capacity exists in an agent or in a biological system, there needs to be a systematic protocol to evaluate it. Inspired by cognitive science, we have two contributions here. First, we introduce a systematic protocol for evaluating cognitive capacities. What that means is you need to operationalize a particular latent ability in terms of multiple tasks that can be measured and these measurements need to unconfound or decouple certain confounds from what is actually being measured in terms of that cognitive ability.

2:46:09So for instance if you give it some simple situations it might be that it solves it but you can declare victory unless you show that the tasks that you have created somehow capture different aspects of the cognitive latent ability that you are measuring. Second, you want to operationalize it in terms of different structures, different domains, and different tasks. You don't want to measure one or two things in one or two environments and with an anecdote declare that something works or something exists. So here what, for instance, you have is different graph structures. I have six structures that I'll show you, different domain.

2:46:45I'll show you the spatial domain. For instance, if I ask you for planning, I could ask you how do you go to Hall Seafronter. Or I could give you an information about Ali is friends with Michael, Michael is friends with Mary, Mary is friends with Sue. If Ali wants to pass a message to Sue, what is the path, for instance, right? That's the planning in the social domain. So social and spatial domain, different domains, and also task conditions. We use 15 different tasks. These are inspired by various tasks that I have designed in the past. You can look at these two papers and others. This goes back 100 years ago to the tradition started by Edward Tolman on cognitive maps in rats and men, 1948 review paper, reviews 20 years of research, where it shows behaviorally how to measure whether an entity, in that case he was measuring rats, possessed a cognitive map.

2:47:30It was a revolutionary result at the time because it went against the behaviorist dogma of the time that you need reward to learn structures. It showed that no rats can learn the cognitive map of the environment even if you don't give them rewards. Okay, come back to present day. 15 tasks in five different categories. The goal is to evaluate systematically whether LLMs can extract from descriptions of an environment the cognitive map. And what does that mean? It means similar to Tolman from 100 years ago until now tradition. Can it solve particular tasks? Is it robust to certain tasks? Can it do flexible planning with respect to and in response to different kinds of tasks where you have maybe short or brief local changes to the environment, like a reward location changed or one edge changed?

2:48:17Can it integrate those two accurately plan, for instance? And we have these different graph structures. Just to give you an example of how it goes. So for graph A, domain is spatial and the task is value-based planning. What would it look like? I would describe the graph to the LLM as you imagine a building with six rooms. From the lobby, you have two choices. You go to room one or two. From room one, there is a door to room three. From room three, there is a door to room five. In room five, there is$10. You don't take any money because at the end you only have one possibility to take money. You go back from room two, you can go to four to six.

2:48:52And in room six, there's$50. And then the question is, and this was the description of the environment. Then the question is, you return to the lobby, you have only one choice. You can only take money once. What is the optimal room to choose in order to take the most money? And you should say two because six has the most room, right? So all of these environments are described in that way, either in the spatial domain or the social domain, and the different tasks are prompted like this. For cases where something in the environment changes, you can see how the second prompt, for instance, modifies something.

2:49:21You say, oh, now you learned that the reward in this room changed to such and such. Oh, now you learned that the door to this room has been changed, and it all of a sudden opens to this other, right? Okay, now with that, please don't look here. I don't want you guys to cheat. And I know you guys might have heard things, but forget everything you heard. Between these three, which one do you think is going to be the most difficult? And why? So choices A, B, and C. A, B, and C. Which graph is it going to be difficult or are they going to be the same? In terms of for the LLM to solve. They're similar.

2:49:52So B has more branching. Okay. And C has more length. So which one is going to be more difficult to solve for LLMs? You can say different things and we can see who is right. I don't know the answer. I'll guess B because more branching. Okay. Anybody guesses anything else? Probably C because I guess all M is not able to handle a very long-term sequence. So we have two hypotheses here. Anybody thinks they're the same? So between A, B and C, which one is more difficult or are they the same? I just don't understand. When you say so, what kind of problem are you trying to see? This problem that we just mentioned here.

2:50:25Oh, sorry. There is some money somewhere at the end of them. One of the nodes that is terminal has the most money. C is harder. C is harder. Okay. And then between D and E, which one do you think is harder?

2:50:38more branching so see so E you think E is harder if it's branching okay I don't I don't actually know that I mean I do I do feel like he has a point okay so who thinks okay so you think E is harder anybody thinks D is harder okay why because it has less weight to go from 1.1 it has bottlenecks right okay ready okay right here so take a look at this B is harder than C as you can see it's branching. Right. B is harder, even though C is twice as large as C in terms of the number of nodes. And A, you can see that it was easy, right? So imagine if I showed you this as the planning task and I declared victory and I said, look, LLMs can solve planning.

2:51:20GPT-4, great, near 100%, right? But then you try just a little longer or you'd have the same number of nodes, but with a branching structure, what do you see here? Huge drop, right? And in fact, what do you see for three of the LLMs, it's at almost a 0%. And now between D and E, let's take a look. As you can see, D is much more difficult for GPT-4, which is the blue one. In fact, E is more difficult than B. Sorry, B is more difficult than E. It's not consistent. Well, there is something there. In these two, you have structures where you need to be exact. There is not multiple paths between different nodes, right?

2:51:57So it's very important if you're going from this cluster to this one, you have to path through this bottleneck. So there needs to be an ability to plan accurately the specific bottleneck, correct? Now, what about the different tasks? Let's see. As you can see, they're not robust to the different tasks either. Traversal, which is one step, two step, three step, end step path, and value path. This is easier for these guys. Why is that? The reason is that traversal does not change the structure of the environment or the rewards. However, as soon as you have the local change, the stuff that Edward Tolman was talking about 100 years ago that is required for measuring cognitive maps in rodents, for instance, like detour and shortcut that we have, all of a sudden you see a drop, and you can see all of a sudden it goes to 0 for cohere, for alpaca, and for llama, right?

2:52:45And so, and here you can see this sad thing also. It's almost at 0 % for four of the graphs. So all of these graphs are at almost a 0 % for three of the LLMs and about 20 % for most of them. And it's only GPT-4 that does a little better, and that's about 40%, right? So based on all of these things, robustness to task if you aggregate across graphs, not robust to tasks. And robustness to different graphs if you aggregate across tasks, also not very robust. So you compare these, the general conclusion I would draw is that they're not good at planning. Now let's take a look at some of their failure modes.

2:53:24So can you guys see what is the failure mode that is happening here? There is an edge that doesn't exist. It hallucinated an edge in giving the planning response that doesn't exist. Now let's take a look at this case where you have a direct path from 1 to 7, but it's giving it very long. It says what is the shortest path between 1 and 7, and it says 1, 13, 10, 7. But interestingly, if I ask the LLM, can you list the topples? GPT-4 can easily list the topples, but at the same time still can hallucinate like in this case. Now in this last one, there's two mistakes. I told you one of the mistakes, which is hallucinating the edges.

2:54:01What other mistake do you see? Is it out of order somehow? I don't know. It's hard to tell from this distance. Take a look at the answer. What is wrong with the answer? It revisits a node. Exactly. There's a loop. So a short-term path should not have a loop. Of course. So we found another case, right? So these three failure modes are failures of planning, Even though it knows the one-step topples correctly, it seems to fail at planning. And it can give you some insight into what is going on. So it's not very good at stitching one-step things together. So based on that, why do you think it was better at graph A?

2:54:33Can people give me guesses? Why do you think graph A was easier? Or it showed some apparent success on graph A. Why do you think that is? Smaller steps, fewer choices. Fewer choices. But this one is also very few choices. It's like B. It's a tree, right? This has fewer choices than that, right? But why is this so much more difficult? More ways to be wrong. Say again? More ways to be wrong. More ways to be wrong. So another way to say it is that the things that showed up exact in the kind of the prompt are more likely to work for C and A, basically. So if it just did just memorization of what's going on, right?

2:55:14because it's just sort of a kind of a two tracks here. But there are more branching. For what it's worth, I think a lot of the common sense reasoning benchmarks that these things are specifically trained on are transitive. I don't know what you call this. Yeah. We trained it to be good at ANC. No, that's not true. No? GPT-4 has been trained on a huge amount of text. A lot of that is family trees and structures that are actually tree-like. It turns out transformers, in fact, do have some limitations with tree-like structures and with things that are bottleneck. We are very good at bottleneck. In fact, bottlenecks makes things easier for us, right?

2:55:53You have a few nodes that are basically, they have high centrality, especially eigencentrality or betweenness centrality. And basically what you do is when you're solving a problem and planning, you say, I'm going to find that. No, then from there I'll go somewhere. You have a subway system. You go to 14th Station in New York City, then you can find a train that goes somewhere else, right? So if you get lost, just find the hub. We actually use these heuristics a lot. It's available in human texts a lot, but it hasn't been picking up on that. So this is about the structures that the transformer, for instance, might have been learning.

2:56:28And as you can see, you have here from 7 billion parameters to 1 trillion parameters to the best of our knowledge or larger, right? and none of them is capable of figuring out or like having a high performance higher than like, we have something between zero and 40 % on a simple two-step tree, which is the simplest thing you can give a model-based planner. It's not even probabilistic, it's deterministic. And even that is failing, right? And then we saw these failure modes. Another thing, what if I give it extra instructions? By the way, all of these have been told things step by step. So we give that simple chain of thought.

2:57:04What if we give it extra instructions? For instance, I describe entire breadth-first search and depth-first search. And I say, hey, use depth-first search. How is that working? First do this, then do that, and another one. So you can see in the supplementary material of our paper the entire sort of breadth-first search and depth-first search. Then you see that it improves somewhat for when you are within a cluster. But when you look at a situation in this graph D where you need to find the shortest path between nodes that are a cluster away from each other, What you see is that it doesn't help much.

2:57:37And interestingly, for different temperatures, for temperature zero, it doesn't help at all. It helps a little bit for temperatures that are higher and I guess like take different kind of paths. But it's interesting, only one cluster away. Shortest path, one cluster away, it's not a big deal. The diameter of this network is not that large. There is not a lot of improvement and the performance is pretty low, as you can see. For all of them and for three of them, it's actually closer to zero. So this is the evaluation. we have done together with my summer interns we have a paper where we did a prefrontal cortex inspired modular architecture where gpt4 basically plays the role of these different kind of modules and solves these problems in a kind of a modular way similar to the prefrontal cortex i have like 15 years of working on prefrontal cortex i'm very excited to do this with these models you can see it here and this paper is you can find it on my website web at all 2023 and you You'll find it here as well.

2:58:32I have an archive number over there. Okay, so I had to cut it for time there, but literally, I'm not joking, I had another half an hour of audio just chatting with her and all of us just crowding around her like students. She just was very, very engaging in person. And I love to see that. I love to see when people can not only do great work, but then also talk in a compelling fashion about it, not just passively answer questions about it, but also challenge you to think along the way. so I guess if I were to include one agent's paper from NeurIPS this would be it and for the final talk of this entire pod which is already stretching into three hours I have saved for the coverage of state space models which have been the talk of the town the Mamba model was released a few days before NeurIPS and Albert Gu was there I met him but I couldn't get a conversation with him but Chris Ray was on stage talking about effectively all of hazy research, what Stanford's doing and what Chris Ray is up to and all the people he's associated with, including Tridao and Albuquerque.

2:59:32So if you want a primer or a good entry point on just how Chris Ray is thinking about six-phase models, I think this is it. So as I mentioned, our motivation for getting rid of attention, potentially, is long sequences. That's the practical motivation. I'll come back to my real motivation in one slide. Practically, some data comes as long sequences. Data, audio, DNA is billions of base pairs. We can also cram in tons of few-shot examples, which seems pretty cool. When we started this project, really the standard models couldn't have it. GPT1 had only a 512 context length. And as I mentioned, transformers are scaling quadratically in their sequence length.

3:00:09So we kind of took two parallel paths to this. One is better hardware algorithms. So we tried with, you know, flash attention and now people have followed up to make that path really, really fast. Just optimize the crap out of it on hardware and there's a lot of juice to squeeze there. The other approach which I'll talk about now are new models. Now, as I mentioned, I actually wasn't totally motivated by that. I wasn't, honestly, that wasn't my total motivation. I was really motivated by this inductive bias issue. So the idea here is you give me this image, and I flatten it into one single pixel.

3:00:42And then I ask you, is it a car or a boat, some CIFAR-like thing, sequential CIFAR, if you know the task. And this is really interesting to me because when, you know, a human would do this, this would be hopeless. If you gave me a picture and gave me a one-pixel vector as a flat thing, I would have no chance of classifying it. Machines could do something, but there was a huge gap. And I wanted to understand why is there this inductive bias underneath the covers? Do you really need this spatial inductive bias for the machines to reason? Do they have to reason like us when they do this? So I was fascinated by this problem.

3:01:12All right. So there's another benchmark that came out that was really exciting from the Google folks, which was about how to benchmark efficient attention. It's called Long range arena. It's extremely cool. we found them basically because we were playing around with the sequential CIFAR things, and they had a much greater library of places where they were seeing possibilities to improve attention. This was the leaderboard in 2021 of this attention, and they were basically looking at a bunch of very cool linear attention variants, some of which we still play with. I want to draw your attention to two columns on this thing.

3:01:45The first is image. That is that sequential CIFAR task I was just talking about. It's a really interesting task. You've probably trained CIFAR to, like, you know, 90s or high 80s on your laptop or, you know, on a small GPU. And you see the sequential version was lagging quite a bit behind. The other column is this thing PathX, which were these large images where you had two dots and you're trying to say are the two dots connected. And the reason there are Xs is that every model was basically random guessing at this point. So there are three approaches that we were trying to improve long sequences.

3:02:15Improve the hardware, the utilization on hardware, approximate attention, and this last one which I'm going to talk about most, which is using RNN-based kinds of ideas and signal processing ideas. All of them are great. I just happened to pick the last one. So the idea is we're going to replace just the signal processing box, the signal mixing box, with this new operator, S4, that's based on signal processing ideas. So this was inspired by Albert and Karin. Albert's now a professor at CMU. Karin is now running this company, Cartesia, which is a small company just started. And basically, S4 is a classic states-based model.

3:02:50So if you're an EE person, You've seen these in like your undergrad right away. It's an LTI system. But we're going to tweak it for deep learning. The first thing we're going to get, as I'll show you pretty mathematically and nicely, is that signal processing people are obsessed with stability. They understand bounded input, bounded output, stability like nobody's business. It's simple and it's clean. We can use it right away. This is a challenge when training these models. A second thing which was quite surprising is I've always thought about CNNs and RNNs as quite distinct models. But what I'm going to show you mathematically is these models actually unify both.

3:03:23Now, these are CNNs in a kind of different way than we're used to. They're convolutions where the filters are potentially as long as the input. But we're going to be able to view the exact same weights and operate on them either as an RNN or a CNN, which is quite exciting. And the last piece, of course, is that we're going to make this quite fast. And these are going to be asymptotically more efficient than transformers. We're eventually going to be able to process sequence in like n log n time, which is then a challenge to make practical and I'll share some results there. Now this thing is extremely simple.

3:03:54Very simple, very simple signal processing ideas but I just want to point out it had a large improvement on LRA that surprised me. So here's the improvement on LRA. This is the first of its kind of solve PathX. It was like a 26 point jump on this benchmark that a bunch of folks had played at. I also want to point out that the image task, that spatial bias seems to matter less than I thought and that was really the thing that was interesting to me and since then many people have followed on and pushed these numbers up higher but I just think that's really interesting. I don't know what to do with the observation but I really like it.

3:04:25Okay, so what is signal processing? Well, signal processing people view a signal of d dimensions at n time steps as input and an output is a signal of d dimensions at n time steps. That looks a lot like our X and O matrix that we had in attention. They also think causally. They think that time moves left to right through this and things like GPT are also kind of causal. So, so far, what I want to emphasize is we've really done nothing. It's just symbol pushing that we've been able to move into this model. So what does signal processing actually buy us? Two big ideas. The first is, over 100 years, they figured out a bunch of models which are relatively simple but capture pretty interesting phenomenon.

3:05:05These aren't the best models you could ever use, these LTI systems, but they're a simple and very well understood starting point. So I argue it makes sense to start there. the second piece which I think a lot of machine learners don't necessarily love is that they have this idea that a signal is a continuous object that then is discreetly sampled and that idea allows us to do a bunch of stuff in particular it allows us to use all our discrete tricks which are more common in machine learning and AI but also a bunch of you know 19th 20th century mathematics that knows how to do integrals and solves things exactly and I'll show you at least one of those tricks in the next couple of slides I think it's an incredibly powerful idea, and it was really helpful for us to think about it.

3:05:46And as I said, it's going to teach us about stability in like a trivial way. We're going to use theorems from the 1800s to be able to prove that our models are stable, which I just think is awesome. All right, so what's an LTI system? If you've never played with one, this is what's called a single input, single output system. You have some curve that's coming in, which is typically called UT, that's the input, and some output curve, YT. You have some hidden state, which is much higher dimension, usually than the input and the output. We'll take the hidden state as large as the input when it's discretized.

3:06:15It's going to be a huge thing, okay? Now, I haven't told you how that hidden state evolves yet, but it's going to be constrained. And the LTI people say there's lots of things that can fit into basically letting it evolve according to an ODE, okay? So I'm going to show you that in just one second. So here's what you need for the ODE. You need two matrices A and B, and we're going to learn those matrices. And basically it says that the hidden state can only evolve according to this equation. It basically says the change in the hidden state is proportional to some learned function of the input plus the previous state.

3:06:49The output is then from projection, from this linear projection, from this high dimensional state down to 1D. This is all that an LTI system does. I'm just saying it's something that's surprisingly powerful and well understood. This is not the best model. If you're a signal processing person, you say, oh, you should use X, Y, or Z. You're probably right. But we want to start with something really, really simple that we can understand all the way. Alright, so it turns out that one of the beautiful things is because it has this continuous object lurking in the background, you can use high school calculus.

3:07:17And in particular, you can get out this nice expression. And what this says is the hidden state is exactly this function, x of s, and this convolutional style integral. This is exactly what it is. This is wonderful. You just solve the ODE. Then when we realize it, we have to discretize. We'll come back to that in a second. So the immediate win is, well, this can tell us exactly when the system is stable. Basically, as long as the eigenvalues are in the left-hand part of the plane, which every EE person memorizes, and the reason left-hand part of the complex plane matters is, E to those values goes inside the unit disk, you know that this thing is not going to blow up.

3:07:55The system is going to have bounded input, bounded output stability, which is really exciting. So when we train, we can fix our A's, our representations, so that the eigenvalues satisfy this property. and that's going to be one of the arts. Now to implement this on a machine, we can't use continuous objects. We have to use them as discrete. And integrals are just big smooth sums, basically. They're actually nicer to deal with than functions. And so what we'll do is we'll break that sum down into functions. And what happens in signal processing is you think that you're going to sample at some regular frequency t, and then what I'm denoting here is x bracket k means the kth sample, which is at the point kt.

3:08:31So you're seeing this animation that the integral is just this nice, smooth sum. All right? Cool. All right. So now that we're in discrete land, we can relate it to more familiar machine learning concepts. The first thing is you can view this as a recurrence, as an RNN. So I'll introduce notation G here, which is basically the B times the input. It's all the modifications on the input that we had. And with just a little bit of arithmetic, I can move it out so that I get X of N plus 1, the next hidden state, is T times GN plus some term that's kind of downweighting it. And I'm illustrating the downweighting here and the visualization.

3:09:07RNNs are super fast. So if we did manage to learn the weights, the Bs, the As, all the rest of these things in the filter, then we could run this as an RNN automatically from the same parameterizations. Super cool. With just a little bit more notation, I can take that E term, the exponential there, and put that matrix exponential into this function F. And that becomes a convolution that's probably more familiar to most people which is a discrete convolution. But notice this discrete convolution is of length N. It's a huge long convolution. It's not a three by three convolution like a ResNet. It's actually as long potentially as the filter.

3:09:40So that's going to be challenging to process. But this model says they're basically both the same. So the key technical challenge is to make these SSMs fast. Those long comms are hard. If you think about it, that F, that filter, is huge. And so if you materialize it at every time stamp, you'd be toast. It turns out that you don't ever have to materialize the hidden state. That's a really important observation. That allows you to go fast and allows you to have runtime that's proportional to the input and the output, not the massive hidden state. The hidden state is important for representation, but it's actually not important for implementation.

3:10:14You can check out the blog. The blog has more details about exactly how that works. The second thing which we spent a lot of time on, and Albert did a bunch of really brilliant things, inspired by the LeJean memory units, is how do we make that A have that nice eigenvalue structure so that we know it's stable. Things like diagonal matrices are really easy to keep this structure because you just, you know, they're scalars on the diagonal. You can keep it. But computing matrix exponentials in general for expressive classes is actually pretty challenging. So we had to do a ton of work to get that to happen over the last couple of years.

3:10:45And the last bit is this practical fast convolution that we needed. Now, I love this slide because Dolly 3 made most of the art in this talk or all of the art in this talk. And it made this poster. I didn't give it the tagline. I still think it's hysterical. Too fast, too furious, revving up the equations. I have no idea what that means, but I love it. It's supposed to be furrier, by the way. That's the thing. Any case, the thing is we had to do the same type of operation that we did in flash attention, but now on FFTs and convolutions. If you naively run FFTs, you have terrible memory behavior.

3:11:14If you can somehow group them together in nice ways and be IO aware, you can get back to that kind of nice utilization. Flash attention, if you recall, was about 72 % utilization. Dan and Herman got to 65 % utilization. I would also say that Dan's on the faculty market this year and Herman's on the PhD market, and you'd be smart to hire them. They're amazing. So the point is there's not really a hardware tradeoff after you do a bunch of work. It's really algorithmic. This thing is going to do a lot fewer operations. And this led to what some folks have called, Sasha called an RNN renaissance. And I want to say it's been super fun.

3:11:47I have to say the last year and a half of two years of research I've absolutely loved because you've had a ton of people contributing amazing ideas like S5 and Mega and RWKV on super technical topics that were really exciting for us to do. And there's been so many more that I can't put on here and they've been pushing the state of the art. So now you've listened to my talk and you're like, should we use these models everywhere? And maybe I'm a California optimist, so I sound happy. I know it's irritating, but I'm happy about everything. So I am. So you're like, maybe you should use these things.

3:12:16I say, well, maybe, but there's actually a pretty big gap on language. So it was wonderful on LRA in those signal processing tasks, but when we actually deployed it on language, there was a gap. Now, the standard way you measure a language model is perplexity. This is the score of how predictable the language is. To give you a sense of this measure, S4 was five points worse on perplexity versus transformers. And that's a staggering number because five points is about the difference between a 125 million parameter model and a 7 billion parameter model. It was a big gap. So we started to wonder, why is that?

3:12:52So we went back to work that other folks had done, which was amazing, this associative recall task. So the task here is I give you letters and numbers. The last letter is a query, in this case C, and you have to tell me which number is associated with that letter. It's a lookup task. Attention can crush this because it's a very easy lookup task. These two variants of S4 that came out later that are supposed to be better on language were better, but there was a gap here too. And so without going into too much detail on this piece, Michael Polley came along and did this thing hyena, and he showed he could get 100 % on this underlying operator and did it in a very exciting way while still maintaining speed and all the rest.

3:13:30So this is what the picture looked like as of a couple of months ago, or a couple of weeks ago, I guess, two weeks ago. You had S4, which was a bit worse, but then in quick succession, people were coming down to this very strong attention baseline. All the baselines are released. Eleuther made a wonderful harness. These are all at 350 million. You can start to play with these things. And people are, RWKV has been releasing even bigger models. And so there was this baseline here. These are closing the gap without attention. But part of the reason I love academia is you can worry about tiny problems.

3:13:59It's like, well, it seems like a small problem, but why is it worse? And so we kept asking, we kept poking at it, and Simran and Sabri came in, and they actually came up with this idea. It took us a surprising amount of time, but it was just a small twist. The small twist is what a transformer can do is not one lookup, but many lookups. So what MQAR is, is multi-queries. We don't just look up one letter, we look up many letters. Now we can worry about scaling in the letters, the vocab size, the model dimension. And what we found is that all of these models can, quote, solve the task, but how they do it, their scaling is quite different.

3:14:30And this relates to a bunch of things in parallel circuit complexity that I won't get to, but this is a really interesting thing where we can start to study the scaling. And so what they realized is that attention can solve these things with a small number of dimensions, roughly logarithmic, whereas Hyena an RWKV require, and all the convolutional models as a result of their reduction, require things, model dimensions that scale with the sequence length. And so you get charts that look like this. They'll solve it, but they need more capacity to do so. So when we started looking at these MQAR things in the wild, we started thinking, well, okay, MQAR is a nice synthetic, but does it translate?

3:15:05And this was really insightful. Simran and Sabri did this. They said, we're going to take the pile, and we're going to segment out which ones are AR-like, which sentences are AR-like. So these are things that have repeated bigrams. Common buzzard is repeated twice. There's kind of an implicit lookup the second time you're doing the common buzzard task. That's about 7 % of the pile. The non-AR slice was basically everything else. What they found is that the attention gap, 82 % of it, was explained even though this is a pretty rough proxy for the task by just what's going on here. And this made us think, maybe if we solve this task, we can even close it.

3:15:38But the other observation was actually these convolutional models are slightly better on the non-lookup task. So maybe there's hope to go beyond them. And so we started this kind of architecture, and I want to give another shout-out here to a paper I love. I love the T5 paper. I'm sure many of you do too. I love the vibe of it where it's like, hey, we just want to say what are the common elements that are going on. If you're outside this little tiny sub-community, all the papers look very, very different. But if you're inside, I would say there's a couple of really common themes, and Simran and Sabri tried to boil them down so that more folks can participate and come into the field in a more easy way.

3:16:11The themes are long convolutions, convolutions that are scaling with the input, not necessarily the full input size. Gating is a wonderful idea. That's multiplying in this kind of component-wise way in the sequence. That's an old idea. And data dependence. And Mamba just came out from Albert and Tree, which did this and still kept that subquadratic runtime. Base is basically just simplifying all of the things that people are doing and trying to get to something nice. We don't have T5-level niceness left. but we were inspired by them. One thing I want to point out is that this new convolutional architecture does scale for MQAR a little bit like attention.

3:16:46So it has the same kind of dimension scaling that the others had, which is interesting. So the point is, is very recently, this is in the last, you know, week run up to NeurIPS, both Mamba and Base, and I'm sure five others will come out in the next couple of weeks, are now attention-free and actually getting you lower PPL at 350. Doesn't mean they're going to get you lower PPL necessarily at, you know, 100 billion, but it's interesting to say there doesn't seem to be any fundamental kind of block, and that's, to me, extremely exciting. I did want to point out a little bit that, you know, there is another bottleneck that's lurking for truly subquadratic models.

3:17:20We talked a lot about the signal processing part, but there's also this MLPs, and I've become obsessed with them. There's a whole line of work, check out Dan Fu's talk, about trying to understand what's going on with the MLPs, and can we slim those down? They become a bottleneck at much larger dimension sizes. So the questions that were driving our work really were threefold. I shared with you, I hope, a little bit about how foundation models changed the systems that we're building. I also talked a lot about how classical ideas from signal processing and databases were interesting bits of canon to bring into the field so that maybe we can make these models more efficient.

3:17:51What I thought I would end with is just why I think there's such a bright future in AI for two minutes and systems. The first thing is we weren't using these models really 15 to 18 months ago in the way we're using them now. We knew intuitively that you train them once and use them multiple times, but it's not really clear we were doing that. We were kind of just showing them to each other, if we're honest. Now, people are using them on like a daily basis. And inference has become an unbelievable task. And I would say even the last three or four months, the speed of inference, if you watch on a bunch of the commercial servers, are just going through the roof as new ideas come in.

3:18:24Of course, people were thinking about this. MQA and GQA a while ago. Speculative decoding was an amazing paper. VLM was really exciting. Flash decode, Matt Former. There's a ton of exciting work here. My point is, this really kicked off like six months ago. Wild to think about, but that's a whole thing. Another bit is, there's a big difference between low latency systems and high throughput systems. When you don't care if it returns in a couple of milliseconds, but you want to say run on a hundred different documents or a million different documents. We're just at the outset of seeing that systems pitch as people are actually using these foundation models on all the back of house data cleaning-ish tasks that I think are going to happen the next while.

3:19:01There's new data types. I do want to call out that there's all kinds of things you can worry about from Kunele, about how to program the system, what the right accelerators and hardware, that's just happening. What are the right systems to build that are systems of record underneath the covers? There's tons of stuff. Yep, I gave Chris a little bit more time there because he's such a legend and he covers so many different concepts and updates and models in such a small amount of time. So his time is very high quality and you should watch the whole talk if you get the opportunity. but that's it for our coverage of New York 2023.

3:19:34It's just a ton of papers. We are going to follow up with a lot of the startups that I encountered and met, a lot of which are returning guests. So keep a lookout for that. But also thank you so much for listening in on this. It's an experimental new format. We grabbed a whole bunch of audio spliced in, you know, live interviews of stage talks and some of my own commentary with a little bit of backing music. It's an experimental new thing. Like, did you like it? Let us know. if you liked it and share it with a friend that would help us a lot and also just remember we have a listener survey going on so please come to our website and fill out our survey thanks and see you at the next new york's recap dj qd outro