There's a new Llama in town

0:06Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at Fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

0:43Welcome to another fully connected episode of Practical AI. In these episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We're going to take some time to discuss the latest AI news, and then we'll share some learning resources to help you level up your machine learning game. This is Daniel Whitenack. I'm a founder and data scientist at Prediction Guard, And I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing cool. I'm trying to figure out how did we survive before all these great new models and stuff?

1:22Like, it's changed. Yeah, it's been crazy. Yeah, I just created a post for LinkedIn and I was like grabbing text, putting it into chat GPT, like getting nice rephrasing and then I'm like oh I need an image and in particular we'll talk about it a little bit in this episode but I was like oh there's this free willy model from stable AI which is like whale themed right and then I've got the llama thing so I just went to stable diffusion excel on clip drop and said hey generate me an image with a whale and a llama together and How do I even post to LinkedIn before without these things? It's like a different world.

2:07Yeah. 2023 versus 2022 is totally different. The content generation, the way you code, it's a different world. Yeah. And this week, as most weeks are, it seems like in 2023, had some pretty groundbreaking announcements and releases, which we're going to dive into a bunch of those things. There's just a huge amount to update on. And I think it's a good time for one of these episodes between you and I to just parse through some of the new stuff that is hitting our feeds. So, yeah. Well, I mentioned Llama. One of the big things this week was Llama 2. But I think before we jump into Llama 2, which I think was maybe the main thing dominating at least my world this week, it might be worth just taking a little bit of time to highlight something outside of the stream of large language models, which also crossed my desk this week, which I thought was really cool.

3:09It's this latest version of Nerf. This is work from Google presented at ICCV 2023. So it's called ZIPNERF, Anti-Alias Grid-Based Neural Radiance Field. That's quite a name right there. It is quite a name. It stands for Neural Radiance Field. So NERF, it's like camel-cased in capital N, small E, and capital RF. NERF, these are fully connected neural networks that create unique novel views of complicated 3D scenes based on a set of images that are input. So I don't know if you've seen that video yet. I'm looking at it as we are talking. And when you say the video, I know which video you're talking about because it's amazing.

4:02I've just left it on. It's pretty spectacular. Yeah. So this is a podcast, so it's hard to express some of this for people. If you just search for zip nerf you can go to the page for this paper which is a great summary but there's a video on the page and just to describe what it is imagine this kind of complicated house with a bunch of different rooms and an outdoor patio sort of garden area and the video is actually this kind of almost like a drone fly through of the house and then the outdoor area if you imagine a drone flying through a house there's hats and coats and toys and couches and plants and all sorts of things everywhere but the video is extremely seamless and it's not generated by a drone it's actually just generated by interpolating between a whole bunch of 2d images and then interpolating from that the 3d scene so yeah i don't know what are your impressions chris first of all the the from the perspective, the drone flight, if you will, that you have as a perspective viewing it is like the best drone operator in the history of the world.

5:16Yeah. That'd probably be hard to get one to do that. Yeah. Yeah. You're not going to get a real drone operator that could fly that amazingly, you know, and get those things. It's just phenomenal. And the house is like, for a moment you look at it and I mean, it is, it looks real, but I have noticed it's cluttery, but it's immaculately clean at the same time as well. The clutter is cleanly distributed and stuff. So I wish when my house was cluttered, it looked as beautiful as this house. It doesn't. But yeah, I mean, just like if you didn't know, if you weren't listening to the Practical AI podcast to go look at it or something like that and you just stumbled upon it, you'd think it was a drone video if you didn't have the education and go, oh my God, this is just really cool.

6:02I wonder what they're doing here. But it's indistinguishable from real life for all practical purposes. Yeah, and so it's based on 2D images, and then there are these generated interpolations, which maybe gets to, there was something that we were talking about prior to hitting the record button, which was this whole field of generative AI is sometimes conflated with large language models or chat GPT, But there's a whole lot going on in generative AI that's not language related or maybe even based on language related prompts. So I mentioned that image that I generated for my LinkedIn post that was still a text prompt into a model that generated an image.

6:48But here what we're seeing is we've got static 2D images that are input to a model that's actually generating a whole bunch of different perspectives that are synthesized in a 3D scene. So this is, I would say, still fitting into our current landscape and world of generative AI. But it's not a text in, text out or text in, image out model. Right. And I think people, there's so much coming at people right now. I think, you know, we keep talking about that this year. In the five years we've been doing this podcast, we've never had a moment like the last few months where things have been coming, new things have been coming at people so fast, new terms, new models, and people are trying to distinguish.

7:36So it's pretty, I think it's pretty fair that people are trying to make sense of how they relate together. And there's, there's a lot of connecting between, you know, the, the idea of generative and the idea of large language models overlap in a lot of areas. And you have models that are both and you have models that are just one and stuff. But I think it's a brave new world right now in terms of the, the amount, every show, we're just trying to figure out what matters right now. Cause there's a lot we're not hitting. Yeah. And this side of things, maybe like the 3D or video or image based side of things, I know has its own set of kind of transformative use cases that are popping out.

8:11I even remember a little while ago there was some technology, I think from Shopify, but others have done this as well, where maybe you have a room in your house and you want to see how you can transform it with new furniture or something that, of course, you could buy. This is a real kind of e-commerce or retail sort of use case for some of the scene technology of a different kind. If you think of this sort of technology that can take 2D things and create these 3D scenes, certainly there's use cases within game development, for example. But even other cases where maybe AI has never impacted the process as much like in real estate, for example, you know, how expensive is it to literally have a person come out with specialized camera gear?

9:08I know that we've had this in the past where it takes a special person to come out with special camera gear to capture the kind of 3D walkthrough, essentially the street view walkthrough of your house and map that onto an actual schematic of your house. And here, if you imagine someone, maybe I'm now selling my house myself without a real estate agent, and I can take an app potentially and go through my house just taking 2D images and create this really cool kind of fly around 3D view that's interactive. That's really, I think, a powerful transformative change for a number of different industries.

9:51I came across a company called Luma AI in one of the posts about this technology. I don't know exactly how much of the, if they're even using the Zip Nerf stuff, but certainly some things related to Nerf to take these 2D images and they have an app that will create 3D views, which is pretty cool to see some of this kind of hit actual real users. We keep talking about the fact that we've hit this inflection point where it's hitting all the, you don't have to be in the AI world, you know, for this to have a big impact. And so, you know, it's very easy looking at the ZipNerf video to imagine walking around with your cell phone on an app and you're, you're just kind of like walking around and the app takes care of whether it's video or whether it's still images or what, and it just uploads it to this and produces this, you know, amazing, you know, so it's not your walk around that it's doing.

10:46It takes that as raw video, but then it produces this super high quality thing. So yeah, I mean, I think this is another case where there's this one technology with thousands of use case possibilities, you know, where it just changes everything. Yeah. And maybe also in the, it'd be curious to know your reaction to this also with respect to kind of the industrial use cases where I've been thinking about, of course, Just like capturing 3D scenes is very important, for example, for simulated environments where you're trying to maybe train an agent or you even kind of an industrial training for human sort of sort of scenario where you want to kind of take someone into an environment that it's physically hard to bring a lot of people into.

11:36Or there could be safety issues and such. Yeah, safety issues. I don't know if that sparks things in your mind. I think in the industrial sense, this could have a more B2B sort of impact than just a consumer app. Sure. I mean, a simple thing, and I'm making something up in the next thing I say, but it's very easy for me to imagine intelligence agencies that are, you know, like if you go back some years to when Osama bin Laden was found and they had various imagery and stuff. But with stuff like this, they might take all those images that they're getting from various sources and produce, you know, a high.

12:15Like a flyover. Yeah, a flyover and very photorealistic of certain parts of the compound where that kind of imagery and that can be used in a military operation subsequently. Now, I'm making that up. So nobody should take that as a thing. But it's not hard to imagine that. It's not hard to imagine a lot of factory uses and other industrial things where you have safety issues, you have limited access kind of concerns where you're trying to convey that. But there's a lot of mundane things. There's a lot of home-based things and small business-based things, as you pointed out the real estate one earlier.

12:48So this is just one technology that we're talking about so far. Yeah. And I think what you're saying, it illustrates how this is impacting very large organizations all the way down to small organizations. Yeah. So proprietorships. Yeah. And it's interesting how like if we just take this use case, for example, these kind of 3D scenes, kind of large scale organizations that maybe their bread and butter was either the compute associated with like rendering videos and 3D scenes or their hardware providers that are creating specialized kind of 3D type of equipment. like their whole business model they've got to be thinking similar to other organizations that are dealing with maybe language related problems that are thinking about these things with respect to llms there's a fundamental shift in maybe how their businesses will operate but then at the same time it provides an opportunity for the kind of small to medium businesses to embrace this technology very quickly and actually make innovative products that can be widely adopted very quickly and actually be competitors within an established market.

14:09So there's an established market for 3D things that has been quite expensive over time in terms of access to that technology. So now that whole market's going to change. I think a lot of the players will be these kind small to medium-sized businesses. I agree. I think there's a moment here, kind of ironically, because people are so worried about the impact on human creativity because of all these models and stuff like that. But on a more positive note, there's this huge opportunity that you're just now alluding to for people that if you can connect the dots as things are coming out and you can stay on top of it, it's a great equalizer.

14:47And so it will clearly change many, many markets that are out there and many, many industries. And so there's huge opportunities for those who want to surge ahead at this moment and take advantage of that. And so I think that the message we tend to see in the media tends to be a little bit doomy and gloomy on that, but it kind of discounts the fact that change isn't always a bad thing. People are afraid of it, but there's huge, huge opportunities here as well if people choose to go find them.

15:32Well, Chris, there is a new llama in town. I know. Llama 2. Llama 2 basically destroyed all of my feeds and concentration this week when it was released because it is quite, to me, an encouraging thing, but also another transformative step in what we're doing. So LLAMA 2, for those that maybe lack the context here, Meta or Facebook or however you want to refer to it, Meta had released a large language model called LLAMA, which was extremely useful. It was a model where you could host it yourself as opposed to like OpenAI. You could get the weights and host it yourself. But the original LLAMA had a very restrictive licensing and access sort of pattern, even though you could kind of download the weights from maybe like a BitTorrent link or something like that.

16:37And those propagated. Technically, if you got those weights, you were still restricted by a license that prevented commercial use cases specifically. And now with Llama 2, Meta's released the kind of follow-on to Llama and we can talk through some of what the differences are and what it is and some of what went into it. But I think one of the biggest things, which is I think going to create this huge ripple effect throughout the industry is that they've released it with a commercial license. As long as on the day that Llama 2 was released, you as a commercial entity don't have greater than 700 million monthly active users.

17:28You can use it for commercial purposes. So maybe if my company maybe later on has 700 million monthly active users, which would be great, probably never. There'll be something past Llama 2 by then, though. Yes. If it does, though, I could still actually use it because it's only on the release date. So on the release date, which was this week, as long as you didn't have greater than 700 million monthly active users, you can use this in your business for commercial use cases. And I think that's going to have a huge ripple effect downstream. And we can talk about the model itself here in a second, but maybe just I'll pause there to get your reaction on that, Chris.

18:09It made me smile when I heard that because it's kind of like saying, so long as you don't compete with us at Meta, you can use this for commercial. Oh, it's totally true. Yeah. Like, who is that? Right. So that's Snapchat. Yes. TikTok. Right. Like you can think of. Yeah, you can think of who this is. And I guess one way to put this is it's not totally open source, quote unquote. We wouldn't call this maybe open source in the kind of official definition of open source. But it's certainly commercially available to a very wide set of people. Yep. You know, one of the first things I noticed when this came out on their page and they're taught, you know, there's there's and I'm diving into like the specifics of the model here is we had an episode not too long ago and you were describing about kind of the I believe it was the seven billion limit, you know, in terms of hardware usage and stuff.

19:05And having been taught that by you, I immediately locked in on the smallest being$7 billion there. And I thought, ah, this is what Daniel has taught all of us about that limitation on accessibility and who can do it. So, you know, it has the$13 billion and the$70 billion size. But I definitely picked up on the$7 billion, which I'm assuming is going back to what you were teaching us a few episodes back. Yeah. And so just to fill in a little bit on that. So the Llama 2 release includes three sizes. So again, thinking back to what are the kind of characteristics of large language models that kind of matter as you're considering using them.

19:48One is license. We've already talked about that a little bit here. We might revisit it here in a second. Another is size because that influences both the hardware that you need to run it and also its kind of ease of deployment. So LAMA 2 is released in 7 billion parameter, 13 billion parameter and 70 billion parameter sizes. And then there's also, of course, the training data and that sort of thing that's related to this and how it's fine tuned or instruction tuned. So LAMA 2 is released in these three sizes, both as a base large language model and a chat fine tuned model. So there's the 7 ,013 ,070 ,070 ,020 Llama2s.

20:37And then there's the 7 ,013 ,070 ,020 Llama2 chat models, which we can talk about that fine tuning here in a second. But yes, you're right, Chris, in that 7 billion, I could reasonably pull that into a collab notebook and maybe with a few tricks, but with certainly with the great tooling from hugging face, including ways to load it in even 4-bit or other quantizations, I can run that, you know, on a T4, for example, in Google CoLab with some of the great tooling that's out there. So not needing to have a huge cluster. The 70 billion, even with that, that's kind of another limit where using some of these tricks, I've definitely seen people running the 70 billion parameter model on an A100.

21:30Again, loading in 4-bit with some of the quantization stuff and all that. The 70 billion is certainly going to be more difficult to run. It might require multiple GPUs, but that's kind of that sizing range for people to have in mind and how accessible things are. And yeah. How might you, I'm just curious if you're looking at these, you're a business out there or data scientist and can you make up a couple of use cases that you might target with each of these where you might say oh I want to go 13 on this not 7 not 70 for something like this can you imagine something like this I'm putting you on the spot yeah I think I mean there's certainly innumerable use cases but I think maybe two distinctions that people could have in their mind is if you want like your own private chat GPT right or like a Another way you could think about it is a very general purpose model.

22:23You could do anything with this model, any specific prompt, whatever. You're probably going to look towards that higher end, the 70 billion parameter model for that kind of almost chat GPT-like performance. You're going to have to go much higher. But as we've talked about on the show before, most businesses don't need a general purpose model. They need a model to do a thing. And so or a task or a set of tasks. And so in that case, I think businesses, because this is open and commercially licensed businesses that could take those seven and 13 billion parameter models and fine tune them for a task in their business, which also is increasingly has amazing tooling around it.

23:11again from from hugging face and others with the peft library parameter efficient fine tuning and the laura technique which is the low rank adapter technique which basically only adapts an existing model it's kind of an adapter technique rather than retraining a bunch of the the original model this opens up fine tuning possibilities in these smaller models where that fine tune for an organization is going to perform probably better than any general purpose model out there. And because it's that smaller size, you can run it on a reasonable set of hardware that's not going to require you to buy your own GPU cluster to host the thing.

23:54So that's kind of maybe a range of use cases that people could have in mind. I have one more question for you before we abandon this. 7 billion to 70 billion being an order of magnitude jump on that. Why would you have something fairly close to that at 13 billion parameters? Like what's the difference in 7 and 13 when the next step is all the way up to 70? What's the rationale you think? Yeah, so it is interesting actually if I'm understanding right from some of the sources that I've been reading. There was actually, I forget if it was 30 or 34 billion parameter. parameter model that they were also had in pre-release and were tuning.

24:37So there was another one that kind of fit in that slot that is kind of missing that gap like you're talking about. Like if you think of MPT, MPT has a 30 billion parameter model that fits in that kind of gap. My understanding and you know if our listeners can correct me if I'm wrong please do but my understanding is that they actually did test that size of model and found it to not pass their kind of safety parameters around harmful potentially harmful output or not truthful output that sort of thing so they decided actually to hold that back so it could be possible as they instruction tune and get human feedback potentially more iterations of reinforcement learning from human feedback.

25:24There may be a model that they release in that parameter range. So that was one thing that that happened, I think. It is interesting, you know, several different things here that are unique about this model specifically, or maybe the release as well, other than the license, is they were fairly vague on the data that went into the pre-training. So they talked specifically about some very intense data cleaning and filtering that they did on public data sets. And it was trained on more data than the original LAMA, but they're fairly vague on the mix of that data and all of that. So that may be related to feedback they got on the data sets that were used in the first LAMA.

26:14I don't know, but the technical paper was mostly related to the modeling and fine-tuning trickery and methodologies that they used, which was interesting. And one of those interesting elements of the way that they fine-tuned this model was, I think, the reward modeling. So if you remember, like the GPT family of models, the MPT, Falcon, these different models, one of the things that is often done with these models is this process of reinforcement learning through human feedback, which is this process, and we covered this on a previous episode, which we can link in the show notes, but actually using human preferences to score the output of a model and then actually use reinforcement learning to correct the model to better align with human preferences or human feedback.

27:09They actually use two separate reward models in this fine-tuning of the chat-based model, one that was related to helpfulness, and then the other one which was related to safety. And one of the interesting things that they talked about in the paper was how sometimes those things can kind of work against each other if you're trying to do both of them at the same time. So they actually separated out the reward models that they used for the chat fine-tuning into these two reward models, one for helpfulness and one for safety, which is quite interesting, I think.

28:01So Chris, maybe just a couple other things related to Llama. And then I want to see your feedback on Code Interpreter as well, because we haven't talked about that yet on the show. and maybe Claude 2 if we can get to it. Yeah, we got to mention Claude 2 as well because they were both big releases. Yeah, so just one maybe other note which I find quite interesting and actually I love our previous guest, Damian's thoughts on this who was in our last episode about the legal implications of generative AI. But one of the interesting things about the Lama license in addition to it allowing this commercial usage is that there is technically a restriction in the LAMA license that says you will not use LAMA materials, which includes the model weights and et cetera, or any output or results of the LAMA materials to improve any other large language model, excluding LAMA 2 or derivative works thereof.

29:02So essentially what this means is if you're using LAMA 2 and you want to fine tune a model or you're fine tuning a model off of llama to outputs, you're stuck with llama to basically llama to is your model and that you're going to stick with llama to. So you couldn't, for example, technically take the llama outputs from llama to and fine tune, say, Dolly three billion. Right. That would not be allowed by the license. And of course, that's something that people are doing all over the place. They're taking outputs from GPT-4 and fine-tuning a different model or taking outputs from a large model like, you know, maybe Lama 270 billion now and fine-tuning another model that's smaller based on a certain type of prompt or something.

29:56So this is restricting that family of models that you're allowed to do that sort of thing with, which is the first time I've seen that. I think it's kind of interesting. Yes, it strikes me as another Mark Zuckerberg anti-competitiveness thing, which he's fairly famous for. I mean, that's kind of even before this. Yeah. And how could you enforce such a thing? Yeah. That was my next question to you is, is there any possible way that you could conceive of to actually know that from an enforceability standpoint? I have no idea. I don't either. So it seems it's like it's a license thing and it will concern the lawyers, but it's hard to imagine.

30:36I mean, going back to our conversation last week, once you have output and that output is input to more output and, you know, there's a point where it becomes very, very, very difficult to know what the sourcing really was. And the fine tunes are already appearing off of Llama 2. So the most notable probably is Free Willy, which is from Stability AI and is a fine tune of the largest 70 billion model. But there's other ones coming out as well. And so I think we're about to see just a huge explosion of these Llama 2 based models for a whole variety of purposes. and who knows how they will fit into that licensing restriction or how open people will be about that.

31:24But it's about to start. The fine tunes are already coming. Yeah, well, you know, to your point earlier, they weren't terribly clear about the data that they were sourcing from their own standpoint. Yeah. And I find it interesting, a little ironic. It's a bit of a double standard maybe. Yeah, a little bit of a double standard right there in terms of like, we're not going to tell you everything about how we're doing input, but by the way, you better not use our output for your, you know, for something. So yeah, a little interesting. Do you think there's any risk of a walled garden kind of concept happening in large language models if others were to follow this lead on anti-competitiveness?

32:01Yeah, it will be interesting. I think it is a notable trend that the first llama from meta was not open for commercial at all. And now they're opening it up for commercial purposes. And, And, you know, maybe there's a separate trend that will happen with some of these use based restrictions that people are importing into their licenses and how useful those things are over time that will may shift and we'll see those things die off. Or maybe if they're enforced and there's precedent, maybe we'll see something go the other way. I'm not sure. But speaking of models that you might get their output and use it to train other models, that is these large scale proprietary closed models from people like OpenAI and Anthropic and others.