Meta’s Joe Spisak on Llama 3.1 405B and the Democratization of Frontier Models

0:00If I was a fighter right now, I would absolutely adopt open zores. It forces me, though, to look at the engineering complexion of my org, right? And things like, I'm going to need people doing LOMOPS and things like, you know, fit data fine tuning and how to build Ragn things. And APIs, there's plenty of APIs that allow you to do this, but ultimately you want control. Like your mode is your data, your mode is your interaction with users.

0:42Hi, everyone. Welcome to Training Data. Today we're excited to welcome Joe Spisak, Director of PM for Generous AI and Metta, where he leads Lama and third party ecosystem efforts. Joe spent the last decade in AI, leading product at PyTorch and working on initiatives that span protein folding in AI math, many of which have spun out for meta into their own startups. We're speaking to Joe just two days after the Lama 3 .1 405B launch and we're excited to get his view on questions like, where is the open source ecosystem headed? Will models commoditize even at the frontier? Is model development becoming more like software development?

1:19And what's next in agents and reasoning, small models, data and more? Joe, thank you so much for being here today. We're so excited to have you just two days after the Lama 3 .1 405B launch. It's an incredible gift to the ecosystem. We'd love to learn a little bit more about how you, what specific capabilities do you think the 405B is particularly unique at, especially in comparison to the other state -of -the -art models? Oh, thanks so much for having me. This is so much fun. I haven't done a podcast like this in Saudi pre -ckelby, so it's like fun to be in the room and just like, you know, chatting about this cool stuff.

1:57Yeah, I mean, we're like beyond excited and meta. This is something that I think a lot of us have been working on for such a long time, months and months and months. And, you know, we kind of put out that nice little like appetizer, I'll call it in April of like, about three. And like, I was actually like, we're looking at like, be that excited about these models and like, the response was like, through the roof, like, my God. Like, everyone's excited, but they really don't know what's really And so, like, yeah, kind of hold that, kind of had to hold that back for a while and kind of keep it to ourselves.

2:29And then kind of build up for this launch. And the 4 or 5B is a monster. It's a great model. And I think the biggest thing we've learned about 4 or 5B is just a great, it's like a massive teacher for other models. And we kind of had that plan all along, because when you have a big model, you can use it for improving small models or just distillation. And that's how the eight and seventies became and the great models that they are. I mean, in terms of capabilities, like we listen to the community, we listen obviously to our own product teams, right? Because we got to build products for Meta. And I mean, long context was like one of the biggest things people wanted and we have much longer context internally than what we were released.

3:10But we saw just the use cases start to build up. Multilingual, I mean, we're a global company. So we released more languages, many, many more to come because obviously Meta has billions of people on their platform and hundreds of countries. And so I think that was like to me, those are like table stakes things, but they're like really done well on the models. Like I think we spent a lot of time in post training on our different languages and improving them and safety. Just their really, really high quality. So we don't just like pre -training out like a ton of data and say, look at us from multi -lingual.

3:47You know, we actually did a lot of work in our SFT phase and supervised fine tuning and a lot of safety work. I think one of the coolest things that I'm excited about, well there's just a couple of things I'm excited about, but one is tool use. I think models, oh my god, zero -shot tool use, this is going to be crazy for the community. I'm going to show a few examples, like we can show like calling Wolfram or Brave Search or Google Search and it works really great. But it was that tool use is going to be a game changer. The ability to kind of call code interpreter and actually like run code or kind of build your own kind of plug in for things like Ragn and things like that really be state of the art.

4:28I think it's going to be a really big game changer. And I think just the fact that we released the 4 or 5 itself and we changed our license so you can actually use our data. Like that was a big deal. Like, that was a big discussion. We had many meetings with Mark on that. And we, ultimately, like, landing on a place where, you know, this was like this pain point for the community for so long. Like, these closed models, like I can't use the outputs or maybe I can use them, but maybe I'm using them and slightly inscurpulously or whatever. Like, we actually are encouraging people to do it. I'm sure that was a tough decision to make.

5:04Walk us through the things that you did to consider and actually making that leap to open up license. and life and sing in that way. Yeah, licensing is too permissible. Oh, licensing is like a huge topic in itself. Obviously, you've probably spent a little podcast talking about it. I don't want to, but we could. I think we wanted number one, just to unlock new things. I think we wanted to have the 405 and our Lama 3 .1 models differentiate people and new capabilities. Just like we just looked at what people were really excited about in the community, not only in enterprise and products, but also in the research community because we obviously have a research team and you know we work with academia and we talked to folks I mean you know Percy Langan Stanford checks to be all the time saying you know when you get a release it when you know he said can I use it can I use it and Percy like you know stay state -state -state patient but I think we we heard them and we knew kind of what they wanted and yeah I think ultimately we wanted Lama everywhere.

6:08We wanted just adoption, you know, maximal adoption, really the world using it and building on it and I think Mark even used in his his letter he put out like, you know, the new standard or standardized. So I think like to do that, you kind of have to enable so like that or you kind of have to unblock all these different use cases and really look at what the community wants to do and make sure that that you don't have these kind of artificial barriers. And that's what the discussion really was. And so actually even beyond that, we started working with partners like Nvidia and AWS, and they started building distillation recipes and even synthetic data generation services, which is pretty cool.

6:48I mean, you can start to use those and actually create specialized models from it. And the data that you, I mean, we know how good the data is because we used it in our smaller models. It's really good. And it improves our model significantly. So I want to pull on the open star set a little bit more sure and I've read Zux manifesto it was great, but I'm still I've tried to grab my head around like what what's in it for meta This is a massive investment the open source in some ways you're leaving a lot of money on the table because you now have a state of the art Model that you're offering to everybody for free and so I guess Is this an offensive move is this a defensive move?

7:26What's in it for meta? I mean we've So, well, first of all, our business models doesn't depend on this model to make us money directly. So we're not selling a cloud service. We've never been a cloud company. We've always worked, I would say, with a partner ecosystem. All the way back to the five years, I was helping to lead PyTorch and the ecosystem in the community who built around that. Like, we never built a service we probably could have in some way, but it would have been weird. We saw basically going back to PyTorch, we kind of saw it as this kind of lingua franca, kind of bridge, you know, to this like area of high entropy.

8:06It's kind of weird way to say it, but like there's all this innovation happening, how do we kind of build a bridge to it and actually be able to harness all that innovation. And the way to do that is to be open, and it's to kind of get the world building on your stuff. And I think that's, that ethos is kind of carried over in Talaama. And if you look at PyTorch, that was a huge way for us to pull in. At the time, when we really started working on PyTorch in earnest, computer vision and CNNs and all that, if you remember that, old times now. But we actually would see these architectures come constantly.

8:41The people would, in the right code and they'd publish it in PyTorch. And we'd take it internally. We evaluated people with open source models and put them out on model zoos. and we devaluate them and we'd see just how quickly the community was was improving things and we'd actually leverage that. Especially for like integrity applications, where we're released like heat -full memes this way, these other data sets, we just saw the improvements like week over week, month over month, and it was built on something that is like we were using internally. So it was very easy for us to just take it inside.

9:11So I think like Lava is definitely similar in Evergaard where, you know, when we act academia and when companies start to read TV's models or, you know, try and jailbreak them, we want people to do that to art models. And so we can improve. And I think that's a big reason. And it's like, be careful what you wish for. Of course, but like it's the same with Linux, right? The Linux is open source and the kernels open source and people will, you know, it's much more secure when things are transparent. and bugs can be pushed faster. And so that helps us a lot. I think it's, you know, there's also the angle of, you know, we don't want this to turn into kind of a completely closed environment.

9:57Like I think just like today, if you do, if you look at like Linux and Windows, and like in my opinion, there's room for both, right? There's room for closed, room for open, and people use, depending on what they need and the applications, I think that there's gotta be a world of open models and I think there's gonna be a world of post models and I think that's totally fine. What was the primary argument against open sourcing? Was there one? I mean, there was definitely like competitive concerns we talked through, do you wanna give your technology, put it out there and that. And I think we're less concerned about that because we're moving really fast.

10:37Yeah, like if you look back, I mean, I've been, you know, back, I've been in meta like close to what six or seven years now and like in the last You know, you're you're so we've done you know, we got a connect launch. We release purple llama less December. We released llama three 3 .1 before that we released llama two in July Lama one was like in February so like just if you think about the like the pace incredible. The pace of innovation that's coming out of our team and our our company is like just at a crazy pace right now. So I'm not too worried about it. I don't think we're that worried about it.

11:15I'd like to move into your personal views on the broader ecosystem. I think a lot of the questions that people have center around what happens to the value of all these models, especially as meta open sources, more of them at the state of the art level, with Lama 3 .1 with OpenAI launching GPT -40 mini. What is your view on DeModels commoditized even at the seat of the art frontier? This is a great question. I think if you look at just even the last two weeks, I mean, 40 mini is a really, really good model. Input, I think in -put per million tokens is something like $0 .15, $0 .60 out. So it's incredibly cheap to run, but it's also an excellent model.

12:00Like, it's just like they they've done an incredible job and just distilling and getting to something that's like really really performant yet really really cheap. So I think like, you know, Sam is definitely pushing on that. And then if you look at what we've done in like last week and pushing out, I would say like pretty, pretty compelling to their models across the spectrum. I do think like it's rapidly getting to a place where you know, the model is going to be kind of a commodity. I mean, I think there's this frontier of like data where, you know, I mean, we can certainly gather data from the internet.

12:33We can license data. But at some point there is kind of like some frontier of limitations. I think that we're all going to have. And this goes back to our conversation goes, we can kind of the better lesson of data and scale and, you know, compute is that enough. It's probably not quite enough, but it's like compute and data becomes kind of, if you have enough of both, you know, you can kind of get it. like a first -order approximation of the state of the art without anything else is kind of what we've seen. So I do think the models are commoditizing. I think the value is elsewhere. And I look at Meta and I look at our products and I look at what we're building.

13:08That's honestly where the value is. For us, it's Meta AI, it's our agent. It's all the technology that we're going to put into Instagram and WhatsApp and all of our products where we actually are going to monetize where we're actually going to add real value. the model itself, I think, definitely will keep innovating, new modalities, new languages, new capabilities. That's what research is, right? It's pushing in the frontier, the immersion capabilities, and then we can leverage those in products. But the models are definitely pushing in that direction. If that's the case, and all these existing companies that have massive distribution and wonderful applications that they're already out in the wild, can just adopt the state of the art models, what advice would you give to the whole wave of new startups that are trying to make it out there, either building their own models using other city -yard models and then trying to build applications on top.

14:02Yeah, I mean, there's definitely like some model companies or companies that are building you know, they're training pre -training foundation models and it's expensive. It's like I think we're you know, I can't say how much Lama 3 costs but it was very expensive and you know, Lama 4 is going to be even more expensive. And so I to me given kind of the state of play and things, to me, it doesn't make that much sense if I was a startup to try and go and do a pre -training. I think the Lama models are absolutely incredible as foundations to build on. So I do think if I was a founder right now, I would absolutely adopt open zores.

14:40It forces me, though, to look at the engineering complexion of my org, right? And I'm going to need people doing LOMOPS and things like fit data fine tuning and and how to build Ragn things. And APIs, there's plenty of APIs that allow you to do this, but ultimately you want control. Like your mode is your data, your mode is your interaction with users, and you may want to deploy these things onto a device at some point, and have a mixed interaction or something. You might want to have a similar queries running on your device and have very low latency interactions with your users. You might want to split and have a more cloud -based approach from more complex queries, more complex interactions.

15:25And I think like the open source approach gives you that flexibility. It gives you the ability to modify the models directly on the weights. You can run the weights, you can distill them yourself. There's going to be distillation services that allow you to take your weights, distill them down to something smaller. I think that's pretty awesome. We're just now seeing the beginnings of that. So I think like in my mind, like control matters a lot. And ownership of the weights. There are a lot of API services where you'll do fine tuning your mouth. So you're bringing your own data, you're fine tuning, and then you use something like a low -rate adaptation or Laura.

16:02And unfortunately, you don't actually have access to those lower weights at the end of it. You're forced to use their inference. So you're like, hmm, let's see, I'm kind of like hell hostage here. I've given my data. I don't have access to the actual IP that was generated from that data. And now I have to enforce to use their inference service. That's not a good deal. So I think the open source kind of like brings inherent freedom. I think that approach doesn't so. What do you think of Mr. Large was announced I think maybe a day after Lama 3, if you're following? What do you think of them? And I guess more broadly for everybody at the frontier, is everyone kind of pursuing the same recipes, the same techniques, the same kind of compute scale, data, etc.

16:47run. And so like, you know, everyone's kind of going to be roughly similar at the frontier. Or do you think you guys are doing something very different? So, first of all, I'm going to straw. I mean, an amazing team. It was one of my old teams in Fair. Yeah. They were working on through improving and AI mathematics. So, Guillaume and Tim and the team are very, and they're, they're incredible. Joe was just talking about fun, banter. I mean, I'm less than I can. There, there. So, I mean, this was like one of the scrapiest teams that I've ever worked with. I mean, the team, I don't think ever slept.

17:20So it was like basically by day, they're doing like, pushing the less now. Probably the less now. I mean, they would push the state of art and like AI and Thurham Proving and, you know, during the day, we published some work on that. You know, I think what, a couple of years ago, they now, cheese. And but by night, they were basically scrappily grabbing compute to train Lama 1. And so we were building large language models several years ago in Fair. And that team basically just, they were just really ambitious and they were kind of working by night. And that's really where Lama 1 came from. So the team is great.

17:58I mean, I think they're doing really good work. I think they're definitely challenged in that they're trying to also open source models, but also make money. And models like Foro, Mini are not helping them. Because this is why they changed their license, for example, to have a research -related license, which makes sense. Because they were open -sourcing models, and they immediately, their own ecosystem is competing with them in a lot of ways. because they'll release a model, they'll host it, like use this model, but then they have together and fireworks and lepton and all these companies that provide like sometimes a lower cost, you know, per million token, like offering.

18:40So it's really tough business right now. In terms of like large too, I think it's a really good model. I mean, we just on paper, I haven't evaluated it, we haven't looked it internally yet. I think if you look at like artificial analysis, I think they added up and kind of like though it was a little It was under I think like the 70b model In terms of like quality, but you know, that's like a blended, you know, the blend of bunch of benchmarks to make that Distinction but on paper it looks really good. We're gonna evaluate it I think you know to for me anyway like the more the merrier like the more models are out there the more companies are doing this the better It's not like we're not gonna be the only one.

19:19I think that's good that we're not the only one So, and I think like more generally like the Jenny ice base like you wake up every single day and you kind of expect something like this right you expect to Mollody release or something groundbreaking to happen and that's kind of like the fun of being in it so. Totally, totally. Do you think everyone at the frontier is comparable though like are you all pursuing comparable strategies? Yeah, this is actually a good question because you know if you read the Lama 3 paper which was I think 96 pages you ended up at right lots of citations obviously. sharing.

19:50Lots of sharing, lots of like contributors and core contributors and that. So it was a it was a detailed paper and Lawrence and Angela on the team spearheaded writing that and I think that was like part of the hardest things. Like developing the model was like relatively easy compared to writing the paper. It was it was a lot of work pulling that paper together. I think if you look at a lot of three, you know, there was a lot of I would say innovation that happened, but also we didn't, like we also didn't, I would say take on like a lot of research risk either. So I would say like the the primary things we really did with Lama with the 4 or 5B especially was, was really pushing scale.

20:33I mean it was still, you know, we group career attention for example. So we don't GQA and that improves inference time and you know kind of helps solve the kind of quadratic attention computational challenge. We trained on, or 15 trillion tokens. We did post -training. We used synthetic data, which improved the models. The smaller models quite a bit. We trained on over 16 ,000 GPUs on our training runs, which is something we hadn't done before. It's really, really hard to do that, because GPUs fail. And you know. It's not the table. Yeah. I mean, it's everyone's like, oh, I'm just going to train on 100 ,000 GPUs like, good luck.

21:10Right? You better have a really, really great infrotein, a really great ML system. You better be ready to innovate at that level because it's just non -trivial. Everyone says it's easy or says you can do it, it's non -trivial. So I think like, I almost look at Lambda 3 is very similar to the GBD3 paper. So if you were to talk to like Tom, he was a lead off of Tom Brown, now at Anthropic. And there's a reason why Tom was the first author on that paper is because a lot of the innovation was really scale. It was really like how do I take something that's you know like an architecture and like push it as hard as we can push it.

21:47And that involves like a lot at the MLSIS kind of layer and infer layer and like how do I scale the algorithm. And so I think that was really like the mentality we had with like the Lama 3 and Lama 3 .1. And I mean internally obviously we have great research team, the fair, we have research in our org and we're looking at and lots of different architectures and MOE and other things. And so, you know, so I think we, you know, who knows what Lumberworld will be? We're a lot of candidate architectures and we're looking at it. But it's kind of a trade -off. It's a trade -off between how much risk you take on as like for research and potentially how much reward or, you know, the ceiling of the potential improvements versus just taking something that's relatively known and like pushing scale and getting that to improve even more.

22:34So ultimately this becomes a trade -off. I think this is such an interesting point. I actually also think it makes Salama and Metta quite unique in the strategy it's taking. The words that I like to use yesterday were, is model development becoming more like software development? Yeah. I'm curious to hear if you think, I think, unlike what many of the other labs have been doing on pushing more of the research, you guys have been focused on just executing on strategies that you know work. Do you see that representative of the continuous strategy you think as you extend Lama out 4, 5, 6, 7, 8? And then also how do you think the other research labs and maybe some of the other startups in the ecosystem will react?

23:16Will they kind of switch and veer a little bit more to the strategy that you've been taking? I mean it's a really great question. We don't have all the answers for sure. I think there's definitely like some somewhere in the middle right now is kind of where I see things landing where where we will continue to push, and on execution, we'll continue to push models out. We'll continue, it's one of our products to iteratively improve as well. So we want that AI improving constantly. And so there's definitely a software engineering analog here that's happening where you can imagine something like a Lama train.

23:52And new features and new capabilities get on that train and we have a model release. It's actually it's much easier when you start to componentize the capabilities to like we're doing that with safety right now and You saw in the release we released Pump guard and new long guard and and you can iterate on those components externally. It's great Obviously the core model is much more difficult. I do think You know, we'll start to include or start to kind of push on the research side because as well because I need the architecture like he's going to evolve. I mean, you've seen like, you know, what AI2, for example, is done with their Jamba and their, you know, Mamba and everyone kind of thinks Mamba is like a new architecture that's that could have promise.

24:33I think what's interesting though is like to truly understand like the capabilities of the architecture, you kind of have to push the scale. And I think that's what's missing right now in the ecosystem is, you know, if you look at academia, an academia is like a lot of absolute brilliant people there, but they don't have a lot of access to compute. And that's a problem because they have these great ideas, but they have no way to truly execute them at the level that's needed to really understand. Will this actually scale? Because like the the job of paper and and model was really interesting and the benchmarks are great, but they didn't scale it beyond I think like under 10 billion parameters.

25:08So you're like, okay, what happens when you know we trying this in a hundred so like does it actually do still see those improvements or not and no one really the extra least outside of these labs, knows the answer yet. So I think that's like one challenge. So I think like to me, we're gonna get into this hybrid space of, you know, we are gonna push definitely on architecture with a very, very smart and well -accomplished research team. But we also are gonna be like, you know, we are gonna be executing. And I think that's when we start to get like a recipe, you know, we're gonna push it to the limits.

25:40And, you know, we are gonna start, you know, release, we're gonna continue to release more models on it. But in parallel to that, we have to push on architecture. Yeah. And I think it just makes sense because the next breakthrough, you know, at some point you're going to reach like a kind of a theoretical limit, and you need to evolve the architecture. All right, so I see kind of a little bit of an in -between. And obviously, we're really good at execution. I think we're a pretty good execution. But we're also good at research, and we just need to marry those two, so it makes sense. Because like research and products are very different, right?