Scaling Multi-Modal Generative AI with Luke Zettlemoyer - #650

0:09All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host Sam Charrington And today I'm joined by Luke Zettelmoyer. Luke is a professor at the Paul G. Allen School of Computer Science and Engineering at the University of Washington and a research manager at Meta. Before we get into today's conversation, be sure to take a moment to head over to your listening platform of choice. And if you enjoy the show, please leave us a five-star rating and review. Luke, welcome to the podcast. Thanks, Sam. I'm really excited to be here. I'm super excited to have you on the show as well.

0:40We will have a lot to cover focusing on your work in multimodal generative AI, open source and open science, the effect of data on models and more. But before we do that, I'd love to have you share a little bit about your background and how you came to work in the field. Absolutely. So I've been at the University of Washington for about 13 years. So I've been faculty for a while. And before that, I got into AI through grad school and so forth. And then I've been at Meta for about over five years now, kind of looking at scaling things, looking at different applications of things, trying to do larger scale LLM type work.

1:17But really, you know, the interest kind of for me spans across like lots of different kind of use cases. These models are really fascinating. We don't really understand how they work or why they work. And they're also kind of locked away. Can we open them up? Can we give people access to them? Can we study them more? Can we make them more usable? These kinds of things. And I don't know. I've always just been kind of fascinated about like what's the limit of what we can do with models. And that's really what drove me to the field and it's kind of still what I'm doing now, if that makes sense. Nice.

1:43How has the recent popularity of large language models impacted your research agenda? It's really exciting. I think the resources involved to do the research changes. And so you have to kind of figure out what kind of work you're going to do. Like training one of these models is now a gigantic team in a supercomputing effort. And a few years ago, individual researchers could train state-of-the-art models for things and everything has just really changed. But then also, as one of my students, Ari Holtzman, often says, sometimes our whole field has shifted. Like now, whereas we used to be more of an engineering or algorithmic science kind of thing, we're almost now like a complex system science or a natural science.

2:20Like we have these emergent things. We built them, so it's not exactly the same thing as studying the weather, but we still don't understand how they behave. We understand how the individual neurons interact. There's nothing surprising in the math, but the emergent behavior, we don't understand. That's really fascinating to me. That's like a whole new thing that didn't exist before. In what ways does your research focus in on trying to gain that understanding relative to some of the other topics that we'll be jumping into, like multimodal and other things? Do you have projects that are kind of probing into these models and trying to understand how they work?

2:53Yeah, absolutely. I mean, I think we've done a whole variety of different projects over the years. Some places we've done a lot of analysis, for example, is in a multilingual setting. So the fact that you can, you know, every model is multilingual now is kind of shocking. You can try to filter the data. So my student Tara Blevins had this really nice paper. You can try to filter the data, but classifiers have 1 % error. So even if you try to train only on English, 1 % of hundreds of billions of tokens is a lot of data and the models will be multilingual no matter what, right? And that's fascinating to me.

3:24So it's kind of hard. I haven't done a lot of work on like mechanistic understanding. I know there's great work at Anthropic and other places on that, But even just like thinking about data's effect on how the model behaves and what the models can and can't do when you train them different ways and exactly what emerges. I think that's also a type of analysis, maybe more kind of input-output style that I find really interesting. Yeah, you know, when you think about the topics that, again, I mentioned that we'll be talking about and that kind of are driving your current research, multimodal effect of data on models, open source, like kind of articulate how those all tie together.

3:58what's the the unifying thread yeah i mean there's i mean i'm sort of freewheeling i kind of have a lot of threads and i enjoy that like this sort of variety but roughly speaking because i don't know that i really have one perfect thread some researchers are better at that than others i just i love curiosity i love going in lots of directions but one thing i do feel really strongly i do feel really strongly about like the effect of data and so one interesting trend is like we're on this incredible scaling curve right now right so models are getting bigger we're spending more compute. And actually, parameter size probably isn't the right metric.

4:30Probably, like, amount of compute and amount of data is the right metric, vaguely. But one cool thing is, like, as we scale, you have to scale both of those lock-in stuff, right? GPT-N is always going to have more compute and more data, and probably more parameters do. But the interesting thing is to think about where these trends break as a researcher, and you want to, like, think out a little bit. So, for example, like, we're going to run out of data someday. There's no question, right? And especially if you're thinking about text-only models, it's going to be sooner rather than later. Like you could try to guess how many trillions of tokens of text there are on the web and, you know, Google and Bing have way more of that than anyone else.

5:05So there's like lots of extra caveats, but it's probably not much more than an order of magnitude that larger than what we're trading on right now. I mean, maybe somebody else could come up with another estimate, but we're going in order of magnitude every year, roughly right now. So the runway probably isn't that long. So one cool thing with multimodal is kind of just that raw scaling trend. So if you're going to run out of text, you should be training on more interesting data. So that's kind of silly. It's kind of formulaic. Another aspect of it that I find really exciting is like the grounding problem, right?

5:34So the models learn a lot about how the world works, a lot about common sense from text only. But I think most researchers, or at least a lot of researchers think that text only isn't going to get you there. And you need to like have visual grounding and eventually maybe embodiment, but even as a step towards embodiment, you know, maybe video, maybe audio, maybe other things into the model. And I think that's where most of the big model training projects are kind of going over time. I think you're going to see basically all the models move from being text only to be multimodal over the next year or two.

6:03And, you know, it's an exciting trend because there's like a lot more signal, a lot more to learn from. And hopefully the models will be able to do really interesting new things, that extra data. Video and images come up frequently as a potential way to ground text and to make models richer. I'm curious, do you have a favorite example in research, like a favorite paper or something like that, that demonstrates this? And I guess I'm asking because it's still to some degree a future, like it's a direction that folks are going as opposed to, we have demonstrated it to a degree though, and I'm wondering what your favorite example of that is.

6:40I see. Yeah, I mean, I don't have deep insights that you don't have already here, but, you know, Dolly 3 just came out. And like the pictures are amazing and like highly compositional. And they've really, they've got some pretty deep understanding of some of the spatial stuff in the text, right? And a lot of the compositionality of describing objects in the world. And there's other ways of learning about compositional semantics and how things compose, you know, language and code gives you interesting windows into that too. And that is kind of grounding if you ever actually execute that code. But I think the sort of, the next token prediction in words is a very direct relationship between, you know, a piece of text and the next piece of text.

7:14But the text to image is much less direct. And so the fact that these models can produce these beautiful images with these very compositional suggests that there's something really interesting going on there. You're learning things there that you wouldn't have learned from the text alone. One of our longer term goals, I think you were hinting at also in your question, was could we ever, say, add images to a text-only language model, and then it would be better at a text-only task than if it had been trained without those images? That's kind of a holy grail, like showing transfer from images back into the text domain.

7:45And I haven't seen any examples of things like that yet, but I think we will get there. And earlier this year, or around I think January this year, we had a scaling laws of multimodal competition paper where we looked at training LLMs. Armin was the first author on this. We looked at training LLMs on single modalities, pairs of modalities, and then a bunch of modalities all at once. And what we were trying to predict is when would bimodal training outperform unimodal training, you know, for given resource levels and so forth. And the scaling law is not perfect. We could talk about that if you want, but it makes predictions.

8:17And, you know, it does predict that at some point, like training on images and text would be better than training on either alone, if you can get a sufficient scale. And that's one of our research efforts right now is try to test that hypothesis and go bigger scale and see if we can get there. You hit on what I was hoping you would hit on and the kind of the sense I'm getting of kind of where we're trying to go and where we are. Can you elaborate on the paper you mentioned a little bit more? How was that? How was it set up? Yeah, absolutely. So we have a research agenda going on right now. A bunch of folks who had the scaling law paper, and then more recently, we had a chameleon paper that kind of followed up on that.

8:52I think you mentioned at the beginning a little bit. And the mindset is that even though diffusion-based models are kind of the rage for image generation right now, because we want to be able to put more and more modalities all into a single model, we're consciously stepping away from that. And instead, we're going back to the image sort of discretization tokenization approach. And so, you know, you use VQGANs, you tokenize into sequences of tokens. And then basically, I'm an NLP person, right? So when I look at a continuous signal, I think, I wish that was a sequence of tokens, because I would really know what to do with it a lot better.

9:21And so basically, I'm just going to take any data I see, and I'm going to tokenize it, and I'm going to turn it into a sequence of tokens. And eventually, I'd like to go byte level. This tokenization has irrecoverable loss, but that's where we are right now. Eventually we'll push that further. But anyways, now I can look at any data in the world and then I can just say, okay, that's a sequence of tokens and I can apply all my LLM tricks to train models. So this has cool implications in terms of being able to train with any combination of text. So let's just look at text and images, although people have certainly done it for audio and for video, not yet, but it will be soon.

9:52But if I look at any data, I can have any tokens in, any tokens out. So it's not text to image or image to text. It's just, you know, mixtures of text and image to mixtures of text and image. Okay. That's kind of the story. And then the models work. And then you can do interesting things like train a bunch of different models, some on text only, some on image only, some on mixtures, and then look at their scaling properties as you scale the amount of data or you scale the number of parameters, tradition kind of scaling law sort of stuff. And you fit a power law and you see where we think there'll be crossing points.

10:22So where we think for a fixed amount of resource, you'd prefer to train bimodally instead of just training unimodally, and you would get a better model for both of the modalities, if that makes sense. So when you talk about seeing images as sequence of tokens, how is that distinguished from how we would traditionally look at an image? When you say a token, is that a pixel vector or is it something else? Oh yeah, good question. So the way these tokenizers work is there's a discrete vocabulary. It's really the same thing as an NLP model. So an NLP model, you have a vocabulary of 50 ,000 word pieces or maybe up to 100 ,000 words.

10:55In the image tokenizer, you're going to have a vocabulary of maybe 10 ,000 words or 20 ,000 words or even less, sometimes single thousand words. And what those words are, it's just less clear. So they're kind of gen sims, like they're arbitrary symbols. It's not the word the in English, it's just some arbitrary symbol. And then what you've trained as a model that can take an image, map it to a sequence of, let's say, 500 or 1 ,000 of those words. And then you throw away the image, you have another model that given those image words can map it back to the original image. And so you train an auto encoder that way, you get those two models, and then you can tokenize your data.

11:28So literally everything just becomes words. It's just some of the words in this special image vocabulary that we know how to turn back into images. And some of the words are in a BPE that we know how to turn back and it's actually tokenized even on the text side, if you know the BPE details, for some listeners that do, there's no vectors, there's no nothing, it's just discrete symbols. and then we know how to apply all of our really cool NLP techniques very, very broadly. And we get like lots of really interesting functionality and things we can do. Like, for example, a single model that can take arbitrary mixtures of text and input, text and images and input and in the output, for example.

12:01Interesting. And so in this example, I think there's an LLM involved in this work here. Are you training on these tokenized images? Are you fine tuning on these tokenized images or something altogether different? Both. The traditional LLM training pipeline now is basically, like you just said, two steps, right? You've got your pre-training step, which is typically done at incredible scale. It's very, very expensive. And it's all next token prediction for all the documents you can steal off the web or whatever, right? Your second step, which is, you know, either you pay people to label data or you get existing data sets, you do some supervised fine tuning, which is generally a little bit cheaper, but the data is really a secret sauce.

12:38And then you worry about safety and things like that. So with multimodal models, like that same recipe hasn't always been followed, right? So maybe it is, maybe it's not. There aren't that many open papers on how the pre-training works, but it hasn't been explored as much. Like typically you would just kind of, I mean, there's definitely safety work and tuning, but it's not as established. So for example, one thing we can do, and this is what the Chameleon paper was a few months ago, is we just apply that same recipe. We do exactly the same steps. We curate as much data as we can. That's text or images or mixtures of the two.

13:09And then we pull together a bunch of labeled data sets, kind of like Flon style fine tuning. And we do those two stages, but we do it multimodally. And when you do your multitask fine tuning, some of those are text to image tasks. Some of those are image to text tasks. Some of those are other things. It doesn't matter. The Sango model can do all those different things. And so you can tune on them all and you can get a very general, very controlled model by just kind of copying the recipe from LLMs directly over to multimodal. And that's kind of a real advantage of our sort of tokenize everything worldview is that anything that works for LLMs, we're expected to work here.

13:39Another example is we've done versions of it that are retrieval augmented. So I don't know how much you talked about retrieval augmentation in previous episodes and stuff. So we have that and it works. And Mitchie, a student at Stanford that was an intern with us last summer, led that effort. And it was, I think, an ICML paper this year. I can't remember which conference, but it's on archive. And you can do retrieval augmentation, not exactly the same, but similar to like the Bing chat GPT is retrieval augmented or whatever. You know, you go and pull in documents, put them in the context, you read them.

14:07That all works. You can literally use the same code. It's just the documents can be multimodal. And so we showed that there's really cool long tail phenomenon. So like, for example, if you want to generate a picture of a particular architectural style, like an Armenian church that have very specific like steeple styles. Okay. And, you know, it's a long tail thing. It's definitely in the pre-training data that all the image generation models are trained on, but it's not common. But if you can retrieve an example of that and put that in the context, the model can look at it when it's doing its generation, it can actually get it right.

14:37It can produce a much, much better image that actually has the right architectural style, for example. Interesting. Is part of the way that you evaluated this model relative to diffusion models? Do you see this as a demonstration of this interesting effect of tokenizing? Or do you see this as a promising approach for image generation on its own merits? I think that it is a promising approach for image generation on its own merits. And I think that the field shifted to stable diffusion for lots of good reasons. So one of the reasons is efficiency. A lot of the, not necessarily stable diffusion, but just diffusion-based techniques in general.

15:17Diffusion models. Yeah, yeah. So one reason is efficiency, although we're finding you can train transformers much, much more efficiently and do more efficient inference with them. So I think that gap is going to kind of go away over time. I mean, H100s have transformers on chip. So, you know, transformers are going to catch up in terms of efficiency. And then the other thing is that the tokenization, you have some irrecoverable loss. Like I said, like a thousand tokens just isn't enough to perfectly reconstruct an image. But if you look, for example, at Google's party paper, which is a really beautiful paper, you know, that's a token-based approach.

15:48I thought that was an equation. A picture is worth a thousand tokens. Exactly. I'm always not good at landing that joke, but that's one I should probably try for every time. But, you know, Google's party came out at the same time as, I think, ImageGen. I can't remember exactly the two, but one was image-based and one was diffusion-based. And if you look at the images, they're both great. Both technologies do an amazing job of generating images. And most of the trade-off is like efficiency and so forth. And we think we can solve the efficiency problem. So we're hoping to reinvigorate the token-based, transformer-based image generation.

16:18And it has all these extra advantages that you can do these very universal things. You can use the same models for everything. You don't have to have a separate model for image generation than you have to have for text generation. We got to prove that. And we got to show that that actually scales. That's kind of a current research effort for us, but we're pretty confident that it's going to get there. Interesting. Another paper that I wanted to dig into with you is the self-alignment with instruction back translation paper. That's an interesting one. Can you talk a little bit about the motivation there?

16:48Yeah, Shan did a really, really nice job with that paper. It's really great. I think the, again, we talked about sort of in pre-training, there's the two steps. There's the really large scale next token prediction, and then there's the later fine tuning, which tends to be kind of smaller scale, just a little bit of data, or at least in comparison, very, very tiny amounts of data. And, you know, I think my personal prediction is that over time, these distinction is going to get kind of blurred. Okay, so there is lots of data on the web that looks like instruction data. That's why the model can zero shot stuff, because it sort of saw it in its pre-training.

17:19And, you know, exactly what you put where and how you blend these things is, I think, is a really interesting area to explore. So this instruction back translation, in a way, was a first step towards that. And the idea was if you could get enough data that looks like instruction input output pairs, like a prompt you might show to a chat thing like ChatGPT, if you can get enough of that data, you could train on an incredible scale and you could eventually, maybe you could get much better, more instructable models that can really do more things. And then the way you would get all that data is not obvious.

17:47It could be super expensive to pay annotators or do the other things, but we can borrow this trick from the machine translation literature. It's called back translation. And the cool insight here is that if you can get examples, if you're trying to learn like a function from X to Y, whether it's a translation thing or whether it's an instruction thing, if you can get lots of good examples of high quality Ys, it's often easier to have a model right to X, which is the input anyways, and then use a real output Y that you gather. So you go find lots of really good things like essays you wish ChatGPT could have written for you or other documents.

18:20And then all you have to do is kind of come up with a prompt that would actually yield that output. And you've got a training example. And so you can get models to do that very easily. I think we did a few shot with an off-the-shelf model. It's not that hard to write the input. It's much, much easier than writing the output. And then you can make essentially unlimited amounts of data. And then you can study how much does it help to actually scale up that second stage relative to the first stage. or how do you sort of do more interesting ways of blending them together? Now, granted, one big use of LLMs is to create content like you might find on the web.

18:51But do you find that the instructions that are learned or this tuning approach more broadly, is it generalizable to lots of different types of content? Or does it just create a model that's really good at creating web articles? I think that it does generalize. I don't know that we have a perfect answer for how much it generalizes. I do think of it as like unlocking abilities the model already has when you do this fine-tuning. If you want to get a lot of different instruction stuff, what you want to do is kind of cover a lot of different domains. In some internal experiments we did at some point, we found various versions of Lama or whatever.

19:26It wasn't very good at poetry generation out of the box, you know, just from the pre-training without the alignment. But then we found that if you fine-tune it on like three prompts of like write a poem about this and then example poems, maybe slightly more than three, but a very small number, it would unlock poetry very generally. So it could do all kinds of other styles of poetry. It hadn't taught it. It kind of had that knowledge, but it was somehow not exposed to the prompting. But it didn't make it better at writing cooking recipes, but it made it better at poetry more generally. So it's like you get a certain amount of generalization beyond what you exactly show it, but you still want to get lots of examples of different stuff across the whole range of different things you want it to do.

20:02And in fact, one of the examples from the paper was demonstrating that the model gets better at a math problem, kind of like a word problem. If you remember that example, it was something like, you know, John ran, it was like two track times and then it separately gave the length of the track and it asked the LLM to figure out his average pace. And I think without this approach, I think it was maybe raw llama, it was way off. And then with this approach, it was better. It's amazing that that works. Like why, why does this approach make the model better at that kind of reasoning problem? Yeah, so I think the reason is exactly the same thing as I was just saying for the poetry, where poetry is like more of a style of text, whereas math maybe requires more reasoning.

20:45I would say the model has those abilities already, but they're kind of not exposed. They're not promptable. They're not tunable. They haven't been sort of tuned in a way that they know how to respond to the prompts yet. Because maybe on the web, it didn't see a prompt exactly like those word problems, or at least not in the right setting. We have some of those examples in the instruction back translation data. And so that exposes not that exact word problem, but some sort of math things that give it the ability to kind of expose that information already has. That's my intuition. I don't think we have a super hard science to really definitively nail that and say that's absolutely what's happening.

21:19But that's my that's our best intuition right now. Yeah, I think that this is getting into the kind of clearly getting into the we don't understand why these models work the way they do and kind of the whole emergent properties idea. complex system science type angle. That's right. Yeah. It's not that the untuned model, untuned as in not using the approach demonstrated in this paper. It's not that that kind of the before model couldn't follow the formatting. The formatting was plausible. It's just that the numbers were wrong. And I guess the formatting was better in the sense of like it broke down the problem a little bit better and it didn't get thrown off by kind of an easy but wrong approach in that example.