Open Source Generative AI at Hugging Face with Jeff Boudier - #624

0:07All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. And today I'm joined by Jeff Boudier. Jeff is head of product at Hugging Face. Before we get into today's conversation, be sure to take a moment to head over to Apple Podcasts or your listening platform of choice. And if you enjoy the show, please leave us a five-star rating and review. Jeff, welcome to the podcast. Thank you, Sam. Thanks for having me. I am really looking forward to our conversation. We're going to be talking about open source and generative AI and Hugging Face, of course, recent partnership with AWS, a bunch of things on the agenda.

0:46But before we dive into that, I'd love to hear a little bit about your background. Oh, for sure. Well, I'm a late bloomer to AI. I joined Hugging Face two and a half years ago. I've known Julien and Clem for some time. And my first foray into AI was about like, how can you automate the editing in videos? And so there's lots of early applications of AIs in there, like trying to transcribe the speech, trying to identify key moments through audio, through computer vision. And these things have come a long way, man, since then. But the last couple of years have been really amazing. And I feel like last week could have been a year in regular time.

1:28Yeah, that's one of the things that I wanted to maybe spend a little bit of time on was all the news from last week. But you mentioned that you've been at Hugging Face for two and a half years. We were chatting before. There's like the startup multiplier that multiplies that by like five or seven. But then another AI multiplier on top of that. It's been a crazy couple of years, I guess. Yeah, it feels like it's compounding. And we've been saying that for a long time. There's been an exponential increase in model size, in compute needs, everything everywhere, all at once. And the past few weeks have been super interesting, not just in the rate of new models, new releases, but also how the whole landscape of AI has been evolving.

2:15Yeah, I've been mentioning to folks recently that when I started the podcast six and change years ago, big part of the reason why I started was because there was just something in the air at the time. This was like three years past AlexNet. Folks were really starting to do interesting things with deep neural nets. I'd end up every week with hundreds of tabs of things that I wanted to learn more about and explore. And it's been an exciting six years, but it feels like that same energy right now that I was experiencing back then, almost making me feel like I need to start another podcast or something.

2:57Well, you started it at the same time. I'm sorry. You started it at the same time that Hiking Face started. At the time, like Julien and Clément and Thomas, they saw that things were starting to work that were not possible before, but it wasn't yet fully working. Like they had this crazy idea at the beginning of Hiking Face that you could actually create AI that would be fun to have a conversation with and you would interact it like you would text to your friends, like kind of a crazy idea. But I guess now, six years later, it came true. But I'm super excited to talk to you because, you know, three years ago when I started getting deeper into ML, like I needed to catch up and yours was my source of inspiration and learning.

3:40So yeah, super happy to talk to you. Well, that's awesome to hear. You mentioned this past week, We are, of course, referring to the release of GPT-4 among a ton of other things that are going on. But I think GPT-4 maybe provides an interesting backdrop for our broader conversation in that it contrasts the open source theme that we're going to be spending some time chatting about. Any reflections on the GPT-4 launch and how you're seeing that impact the broader market? Yeah, I mean, it was kind of a fireworks, right? It was like this Pi day, like everybody in the AI community decided to go on release mode.

4:21And so, of course, you had GPT-4, but you also had Google Palm APIs and you have Anthropic Cloud and you had Stanford coming out with the open source Alpaca. I don't know how to say that. I always get it wrong. Alpaca, yeah, the Alpaca model. Yeah, the instruction-tuned llama or yama. How do you say it? I'm confused. Anyways, everything everywhere happened all at once in the world of AI. But I think like to me... Who is it that also released competitor to Whisper that, I forget the name of the company that just announced that, Assembly AI announced a model that supposedly is 43 % better than Whisper or something like that.

4:59Just a ton of really amazing news this past week. Yeah. And also on the open source side, right? Together Compute came out with Open Chat Kit. So that's Neo GPT-X 20 billion that's been fine-tuned on like 40 plus million of instructions to get an instruction fine-tuned models that like fully open source. Like Apeche 2.0, it's like on the hub, you can use it. So yeah, it's been firing up. Mid-Journey 5 also? Yeah, that's right. And apparently it does a good job with hands now. That's big news. How many fingers do you see? Right. Before you jump in, I saw someone tweeted a tweet where they asked Midjourney 5 to create an image of 100 raised hands.

5:41And it did. And all of the hands had five fingers, but it was like 500 hands. So the model still can't count, but it can do hands now. That's progress. So sorry, you were saying. Oh, no, I was saying, yeah, through this milestone of announcements, like what became clear to me is there is a shift for our fields. where sort of six months ago, AI machine learning was very much a scientific field with researchers building upon each other, publishing papers, reproducing each other results, improving everything. And with the release of GPT-4 and Google and Anthropic announcements, we're in a different reality where the new models are released kind of like Apple style.

6:28You have like the iPhone 10, and it's got this new feature, and it's like a cool demo. And you're going to bring on stage some people who are going to tell you the story about it. And it's available today at this price. And that's like new to me, right? And it's been a shift. And for us, like our mission is to democratize good machine learning. And the way to do that is through open source. So the availability of the models, the training data sets, the model weights, the training, the code. All of that is super important for the field to progress together and make sure that everybody can build upon machine learning.

7:05And so, yeah, that was like the bittersweet sort of part of those announcements. And yeah, we... And in particular, sweet in that you're seeing this spread and acceleration, but bitter in that for the most part, there's a lack of openness in the major large models. Is that what you're referring to? Yeah, it was quite stunning. Like if you look at the paper, quote unquote, of the GPT-4 release, like the most notable thing was the absence, right? It was like, yeah, you know, for competitive commercial reasons, we won't tell you how big the model is, what it was trained on, how it was trained, like nothing.

7:40So that's a new turn. Yeah, it is. It was interesting that they're not trying to position it as a safety concern as much as for competitive reasons, like you said, this is our core asset and we're going to hide it behind. We're not going to be as open as maybe our name might suggest. Yeah, I think Ilya put it like very plainly in follow-up interviews saying that open source is not the way forward for them for commercial reasons. And so, yeah, for us, that triples our commitment to make open source models, open source foundational models, state-of-the-art models available to the community and enable the open source community to contribute.

8:21I think it's super important for everyone involved. So how do you think about the open source landscape of models in particular? Is there maybe kind of broadly at first, and then we can dig into some examples of some notable things on your radar? Yeah, I mean, the ecosystem is as vibrant as ever. The open source community uses the Hugging Face Hub as a central place to come and share and contribute and discuss all the latest models. And I have a slide where I say, hey, we have that many models available on the hub and I have to redo it every other week. Right now, there's 150 ,000 models that are free and openly accessible on the Hugging Face Hub.

9:03You can try any of them right there on the page. And the community keeps contributing and building upon each other work to offer more alternatives in any task you can imagine, any language you can imagine. And yeah, as I said, a recent sort of commercialization of AI is sort of building a lot of momentum behind the open source community to accelerate their work. So, of course, at Hugging Face, we are building new foundational models to provide open source alternative. We're doing this with AWS, building them on our cluster there. That's part of the partnership that we recently announced. But also, we are enabling the community.

9:46Eleuther structured themselves to do this work as well. Lion is building Open Assistant. I mentioned Together Computer and just many other projects right now will create new open source models. Maybe connecting back to LLMs in particular, to what degree do you believe open source LLMs are kind of long term viable given the immense cost associated with training, cutting edge models? Do you feel like that cost is going to be or become insurmountable for open source communities? Or do you think that those communities will find a way to stay competitive with closed commercial models? Yeah, I think there is a lot of research right now into making models more performant on a model size basis.

10:38The scaling laws that were sort of the main takeaway from the GPT-3 paper and release have sort of been challenged in a way or improved upon through new developments from the Chinchilla paper to the latest models like the Alpaca. Yes, the Alpaca model from Stanford, right? So you saw that. So they trained that thing on$500 of compute, right? So they started from a 7 billion parameter, not a 100 plus billion parameter, a 7 billion parameter. And then we're able to fine tune it with instructions to produce Alpaca with just$500 of compute. So I don't think we're going to be in a place where the practical way of doing machine learning is going to require millions of dollars to train models and exorbitant amounts of compute for like every type of application.

11:32Of course, if you want to do a Bing chat, that's going to be expensive. But you should take a look at all the efforts. So the GDM from Gregory, I forgot his last name, that did C++ implementation of the deployment of Llama. We've done the same on Bloom. First, it was whisper.cpp, then Llama.cpp. That's right. Whisper, then Llama, and then we added Blooms to the pile. So that allows you to run those models on the edge. So that's super exciting. But yes. Yeah, we're seeing reports with Llama in particular. Someone ran it on a Raspberry Pi doing 10 tokens a second. It then was like converted over to a Pixel 6, I think, at five tokens a second.

12:13Pretty amazing, amazing work happening out there. When Yunus and our team are saying that he got 16 tokens a second on Bloom on a local machine, I think it was a Mac, I was blown away. Wow. So I guess one other thought that occurred to me in thinking about the broader hub and open source models and what's happening with LLMs now, you know, on the one hand, LLMs are kind of demonstrated themselves to be this like Swiss army knife of machine learning in the sense that they can do classification and a bunch of other tasks. On the other hand, I think one of the things I'm seeing is the excitement about LLMs is kind of causing folks to want to treat them as the first tool or the de facto tool, as opposed to often for cases where there are use case specific models that probably already exist on the Hugging Face Hub that do a better job.

13:11Are you seeing that kind of thing happening? And how are you when you're talking to folks, like how are you addressing that? I'm glad you bring it up because I think the hype around generative AI and LLMs is creating a lot of confusion in the market. So you're seeing that as well. Yeah. Like I see customers come in and say, hey, I tried on the playground with like GPT-4. It's amazing. It's able to parse HTML. I'm like, that was solved like 10 years ago with like super cost efficient algorithms. Right. And we have so many task-specific, domain-specific, language-specific models on the hub that have been contributed by our community so that when you have a specific task, you can have a very efficient way to do it that maybe you can run on CPU, maybe you can run on a single machine.

13:59You can apply it to all of your data, all your customer tickets coming in, a whole Twitter feed, whatever, at scale. And that's the way to approach machine learning. and do data science in a more pragmatic way. So in a way, like, yeah, you have this Swiss army knife, but if you want to hang a painting in your wall, like, are you going to use a Swiss army knife? Like, nah, you're probably going to use a drill. I don't know, right? So there's something about picking the right tool for the job and super happy to provide that service to the community to have like this place where all the tools for all the jobs are.

14:36And just like Apple was saying, there's an app for that. on the hub, there is a model for that. So that's, yeah, we don't use very, very expensive models to do simple, repeatable tasks. Yeah. I spoke with your colleague, Thomas Wolfe, actually a year ago. It is amazing how quickly that year has flown by, but about kind of big science. This was just as I think the research phase around Bloom was kind of coming to an end and the the productionalizing phase was starting. So that model's been released. But there's recently been, I think you mentioned this, the Bloom C, the instruction tuned version of that.

15:20Can you talk a little bit about that model and what do you think its contribution is or will be? Yeah, I think with the big science, the biggest deliverable of big science was to show that you can build a large scale collaboration where you can bring all the leading experts from every corner, every company, every organization to work together, thinking through all the ethics from the ground up and produce something that's like a meaningful improvement. And that was Bloom, right? So it's 176 billion parameter that remains today, probably the best multilingual open source base LLM. But I think the main contribution was to show like as a field we can collaborate together scientifically to really advance all boats like rise the tide for all boats and to me like that's even more interesting than the actual model itself model checkpoint itself yeah and that was a big theme that came up in our our conversation and it was particularly interesting because it i guess it's a natural consequence, but it kind of arose out of in this time when there was a lot of question around, you know, can a non-Google, non-AWS, non-Microsoft research team compete in NLP and contribute in NLP, you know, given that they tend to not have the investments that are required to train these massive models.

16:53I think big science was a great experiment for that. I do think small open source teams, given some amount of compute, right? You do need millions or tens of millions of compute are able to provide meaningful improvements to the state of the art. But today it's not only about advancing the state of the art, it's also about just making it accessible to people. And that's why like our efforts today are really centered around the open reproduction of closed source models, right? We have big code that sort of took the torch from a big science to produce a code generation model. We're doing this with ServiceNow, fully open source, the way Big Science was fully open source.

17:35We got some checkpoints already, and there's more to come. We talked about our Flamingo reproduction effort, where much like GPT-4, we're training a new model on both text images. So there are many efforts ongoing, and I don't think it's out of reach for small focused open source organizations to make meaningful contributions. That's why we're backing the great folks at Eleuther and we're collaborating with Lion, Stability and all the other guys. And is there a code focused model that you're working on or backing? Yeah, that's the big code effort. And there are already some checkpoints out. You can find that on Hagen Face.

18:20It's the big code organization. Everything is out there. It's kind of analogous to big science. It's a separate organization that's going after code generation model. Yeah. It's really a collaboration between ServiceNow and Hugging Face to build this thing. It's more focused in that way. But you talked about BloomZ and I think it's cool to mention because a lot of people don't know about it. So the same way that you have a T5 as a base model and then Flan T5 as an instruction tuned model that can respond to instructions like describe in one sentence the following paragraph or translate this type of prompts and requests.

18:55So the same way we instruction tuned Bloom into this Bloom Z checkpoint, that's today the largest open source instruction-based model. So it's, yeah, it's 176 billion parameter. And I think not enough people know about it. How do you evaluate and characterize the performance of models like that? Well, the thing is with these general capable models is that you need to develop a new kind of benchmarks. Thankfully, that's a domain that's still very much a scientific domain where everybody's sharing a result and different benchmarks from Helm to other things. But yeah, for Bloom Z, what's very important to us is inheriting from Bloom like all the multilingual components of them.

19:41So yeah, you have to really look at a wide variety of benchmarks. And then as a user, ask yourself like what's important for my use case? You mentioned Helm. Tell us a little bit about that. Just one of the reference benchmarks that's being used for evaluating LLMs. To be honest with you, I don't know too much about it. It's not necessarily something that we applied to BlueZ. Oh, got it. One of the things that came up when we were chatting was this idea of tend to think of open versus closed as kind of this switch or binary thing. But there's, in fact, a spectrum of different ways to make models available.

20:17Tell us a little bit more about how you think about that. Yeah, for us, open source machine learning is very important because that's core to our mission to democratize good machine learning. There's various components of that. There is, of course, the open source code, which is the implementation of the model. But there's also the accessibility of the training data sets and the accessibility of the model weights and the transparency of the research. So all of this goes into what can be closed or open. And there are various sort of approaches to releasing new models along that spectrum. Probably the best researcher in the field is Irene Suleiman, who used to work at OpenAI and worked on the GPT-2 release.

21:06And it's now at HikingFace. And she published, I think it was two or three months ago, a really cool paper called The Gradient of Release for Models. And she really breaks it down very well, like the whole spectrum between a fully open release and a closed release. It was interesting to see Meta finding the cursor within that gradient along the last few releases, right? From Galactica to Llama. So here it's a different approach where you gate some things, you approve on a case-by-case basis, you release in open source the code, but not the weight. So yeah, there's like many different approaches.

21:46And then you just leak everything to torrents. Allegedly, I don't know. In any case, like on a hugging face, we're all the way on the fully open spectrum, right? We want everything to be public. We think that AI is too important not to be a common good. And we want the whole field to progress together. So how does your collaboration with AWS fit into that? Well, we've been working with AWS for quite some time. It's the number one cloud that HikingFace users apply our models in. But recently, we decided to really extend and deepen our collaboration. And I guess, two main aspects to that. The first one is, as I say, there is renewed urgency around making sure that the community has access to fully open source models that they can use.

22:37And so toward that, we needed to build our own capabilities to do those training. And that's what we did with AWS. We benchmarked a whole bunch of different solutions and we found a great solution for us to build that capability. So we have a supercomputer cluster that's running right now on training new open source models. That's one important part. And we want to make sure those models are available and easy to use to AWS customers. And then the second part of it is how do we drive the adoption of machine learning within companies? I think we do a pretty good job at making things accessible to practitioners.

23:18But how do you take that to production? How do you make sure that you can use machine learning in a way that your production costs don't go out of control? So that's a big focus of this collaboration. And it's a very deep engineering collaboration. Like we work day to day with the engineering teams from the hardware to the platform layer. So from the hardware with teams that build these hardware accelerators that are designed from the ground up for machine learning. So it's Tranium and Inferentia. And we work day to day closely with the engineering teams at SageMaker, which is how data scientists and machine learning engineers can use these models to deploy them and fine tune them, etc.

24:02To build very easy experiences using open source to take a model from Hanging Face and then build with it directly in SageMaker, controlling costs along the way. One of the things that I struggled a little bit with reviewing that blog post about the banded relationship was compared to a couple of years ago or 18 months ago or so when you announced the initial relationship, it was a ton of detail published. Like, these are the things that we're going to do. These are the modules that we've created to integrate Hugging Face and SageMaker. Whereas more recently, it was higher level. And I'm curious, maybe a way to make the way you're planning to work together or the way you're working together more tangible for me.

24:47I guess I have my analyst hat on now. In a year or the next 18 months, if you look back, what are the things that you will have achieved that will let you know that the past 18 months was successful? So there are a couple of things to that. So the first thing, as I say, we built a supercomputer cluster to train new foundational models. And we're not going to announce the models until we release them. And we're not going to release them until they're not just ready, but also they work really well. So I think that's going to be one of the good ways to look at what the impact of our partnership will have been a year from now.

25:29When we see new big models coming out, we know that those were taking advantage of that supercomputer cluster that was built as part of this. So that's one thing. Yeah. The open source contribution. And then the other thing is the developer experience that we are building between Hugging Face and Amazon SageMaker and AWS. And some of that we had been working on for quite some time. So already, if you go to the Hanging Face Hub, like go look at any of the 150 ,000 models that are out there, we provide an easy way to deploy them or take them to SageMaker for fine tuning with like examples and et cetera.

26:14And so we already sort of have that platform to build upon. What you're going to see is that we're going to be expanding like all the use cases that you can do this way. And we're going to integrate more closely with the hardware accelerators so that can take advantage of the cost savings for your model. So all of the sort of practical, how to get started information that we put together when we first announced our collaboration with SageMaker, it's sort of already there, right? We have a full-on documentation about using Hagen Face on SageMaker on HagenFace.co. We have deep learning containers that are available and open source with the latest version of PyTorch, TensorFlow, and Hagen Face that are available.

27:03So a lot of these things already exist. And for me, the way that we're going to be measuring the success a year from now is by seeing how many companies have been enabled by all these things that we've been building. That's awesome. You mentioned the hardware accelerators. These are Tranium and Inferentia. Can you talk about the enablement process? Like what does it mean for those accelerators to better support Hugging Face or Hugging Face models or transformers or what specifically needs to happen there? Yeah. So in order to take advantage of the acceleration on Tranium and Inferentia, you have to bring your model through the Neuron compiler.

27:49And so part of the work is to build the open source bridge between our models that can be in PyTorch, Terselflow, et cetera, to bring them down in a way that you're going to get all the acceleration. And so we haven't talked about it yet, but we just released an open source package, Optimum Neuron, that's going to be instrumental in enabling those experiences. In our testing, if you take Tranium head-to-head with a comparable price GPU instance, like an A10G, and you try to do your typical out-of-the-box training, you get up to 5x better throughput. So in terms of cost savings, really, really big.

28:34And then on Inferentia, we got early sneak preview to the next generation. Inferentia 2 is not yet generally available, but it's out there in preview. And again, we did out of the box, take bird base and run that thing for various sequence length. And the acceleration is crazy. It's like 8x faster for some sequence length. So it's really compelling. So for companies that want to really apply machine learning at scale, right, using the right tool for the job, don't use an LLM to classify emails, you can really take advantage of that acceleration to reduce your costs by a lot. So I think for us, that's important for our goal of demarketization, making things not just accessible, but also affordable.

29:27This open neuron may have missed the name that you mentioned. optimum neuron. It sounds a little bit like an Onyx type of competitor or alternative, I guess, more specifically. Is it doing a similar kind of thing? Well, I guess at a high level, right, it's bridging the language of the model to the language of the hardware. So there is some notion of... Kind of graph compiler. Right, exactly. And Onyx allows that kind of intermediate representation of your model graph so you can lower it to various types of hardware. Neuron allows you to compile your model so you can run super fast on Trinium and Inferentia.