Data synthesis for SOTA LLMs

0:28Welcome to Practical AI. link in the show notes. Thank you to our partners at Fly.io. Launch your app close to your users. Find out how at Fly.io.

0:43Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the CEO and founder at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing great today. It was nice seeing you a few days ago in person. In the flesh. In the flesh. Yeah, that was great. I think you posted a picture on LinkedIn. So if anybody doesn't know what we look like and has some crazy reason to want to know, there's a smiling mug of us on Daniel's profile. Yes, yes. And the reason we met is I was on a client visit on site and we were prototyping out some stuff like chat over your docs and natural language to SQL stuff and all sorts of things with PredictionGuard.

1:33And one of the models that we were using was from Noose Research. And that works out great because we have Curran Mahotra here, who is from Noose Research, co-founder and researcher there. So welcome. Glad to have you, Curran. Hey, all. Thanks for having me. I'm extremely excited to chat with you guys. Yeah, like I said, I'm a huge, well, this is our first time meeting, but I feel like we're already friends because I've had so much of my own benefit and interaction in working with models from Noose Research. a lot of amazing models that you've posted on Hugging Face and research that you're doing.

2:13I'm wondering if you could just give us a little bit of a background about Noose specifically and kind of how you came together as researchers and started, to me, from the sidelines, it seemed like, oh, all of a sudden there's these amazing models on Hugging Face and I don't know who these people are, these Noose research people, but they're amazing. So give us a little bit of the backstory there. Absolutely. Yeah. So just as a general overview, we are one part like open source research organization. We put these models out for free. We put a lot of research out for free, some data sets so people can build on top of these open models.

2:51On the other hand, we're very recently a company as well, a C-Corp. So we've been working pretty hard after getting some seed funding on building together some exciting stuff. I won't go too into during the overview point, but we're continuing to do our open source research and development and release of models indefinitely. The way we started is very interesting, and it would be pretty out of nowhere to the outside for sure. It was extremely fast for us. We're a collective of people who have been playing around in the open source language model space for a while, ranging from like GPT-2 release to Llama release to like the first Transformers paper.

3:33We've got people from various eras of Gen AI of when they came in. And for myself, it was GPT-2. I stumbled upon a CoLab notebook and started fine tuning, made some Edgar Allan Poe and Lovecraft tunes. I've done the same. That's awesome. And we just got pulled into this world of look at these next token predictors that are just managing to smatter together the most wonderful and amazing stories that slowly turn into a deeper and deeper dive of, well, how can I use this for learning information? How can I learn to use this for production and automation, right? It's evolved over time. For us, we started off just working with different open source collectives, actually.

4:16Once OpenAI kind of released GPT-3 and had closed sourced it. We were used to open source GPT-2. We were like, oh man, what are we going to do? How are we going to continue to play with the level of customization and interactivity that we had with GPT-2? Then Eleuther had released GPT-J6B, the cobalt AI community, this community of people who tune models and inference models started to pop up, I think around 2020, 2021, in the face of this. So a lot of us started to have places to centralize and play with these models. We got to contribute and learn how to become better open source AI developers, et cetera.

4:56Eventually, there was a need for more concrete organizations to do this kind of focused work on the creation of these models. We were stuck with like, okay, architectures for a while, like Pythia, but thanks to Meta, you know, we wouldn't be here without Meta. I'll say that first and foremost, like - The great Llama. Yeah. Yeah. Like prior to Llama, right? Like everyone's like, oh, Facebook evil, like my data, et cetera. And here we are, like, they are kind of like the shepherds of this new era of the open source AI movement. So when Llama came out, there was a paper that came out called Alpaca by Stanford lab, right?

5:36And this was about distilling data from bigger models like GPT-3, ChatGPT, GPT-4, and being able to train smaller models on that distilled synthetic data, something they call the instruction data. So that alpaca format really opened up the playing field for everybody to start making these instruct style models, these actual four prod use style models. So there was an idea I had in my head of, well, the Alpaca guys are using only GPT 3.5 outputs. What if I only generated GPT 4 outputs? It'll be a little expensive, but you'll probably get a better model out of it than Alpaca. At the same time that I was looking at this, there was a guy on Twitter named Technium who had just started putting together his own synthetic data set based off Alpaca and the GPT 4 only as well.

6:26So I was working with a group at the time called Open Assistant under Lion. They're a really big nonprofit. And while I was working on that, we had some GPUs. They were cool with us using towards the development of new models. So I reached out to Technium and I said, hey, I have a little bit of compute. You have GPT-4 data in the same format. I have GPT-4 data in the same format. Let's train a model. So we trained a model called GPT-4 x Vicuna. this model was on the vicuna fine-tune we fine-tuned to fine-tune basically the vicuna model was a alpaca style fine-tune and we tried our data set on top of it it was good it was okay then we thought you know we'll probably get a better result if we just train on the base llama model and the resulting model was the very first hermes model gotcha the og the og and and that's kind of how it started to come together was we both had a data thesis on use GPT-4 only and follow Alpaca.

7:28And we trained on Lama and we got Hermes. And we didn't know what benchmarks were. We didn't know anything about any of this stuff. We just made a model and it got a ton of attention. We put it out under this name, Noose Research. Noose comes from the Greek word for intellect. We thought it would be a good name for an AI company. But it was just a place for, you know, fun projects and fine tunes and stuff. It was just a name we were using for our collaboration. And people started swarming and asking, you know, what's news research? Like, what's this sudden, like, mystical, like, open source organization that, like, put out this, like, best model?

8:07And we're like, best model? Like, we just, you know, we just tried something. It was really organic. And it got to the point that people started telling us, you must have trained on the benchmarks. These are doing too well. And we were like, what's benchmarks? We're not really coming from an academic place as much as from an enthusiast that became so committed that it became our life. It became our day-to-day. So from there, people started to ask us, can I join news research? Now, there wasn't a news research to join. It was just two guys, right? What ended up happening was we formed a private Discord server, and we thought there's a lot of people who range from somebody who's like 16, 17 years old, savant on Twitter, hasn't even been to college yet, insane at Transformer stuff, to mid-30s, working a really, really good fang-esque job, and just wants to really create and let loose.

9:05that was another class of volunteer. And then you have, you know, older gentleman who has already exited a company or something who has just been playing with code for a while and wants to jump in and hang out. So we ended up being this really eclectic group. You know, we don't know what your name is. We don't know what your race is. We don't know your gender or anything. It's just discord profile picture, Twitter profile picture. Right. So we came together, grew to about like 40 people all working together on various different projects like Hermes Tunes, Data Synthesis, the Capybara series, Context Length Extension, etc.

9:39And just from this kind of interaction between Twitter and Discord and bringing people in that we thought were cool, we ended up becoming what people would call an open source research org. Yeah, you sort of stumbled into creating this amazing research organization, which is ruling the world, which is awesome. It's what OpenAI might have been. Oh, well, yeah. That's really sweet. Thank you, guys. Yeah, and I love it. It's so cool to hear that story and that background. And I see in my own sort of little snapshots here and there, I'm connecting that in my mind over the past couple of years as I've seen you all post different models and that sort of thing.

10:24This is something we've definitely touched on on the show before, but some of our listeners might not kind of fully grasp when you say this sort of like synthetic data sets that you were focused on in this alpaca format. Could you kind of explain a little bit, like we've talked a lot about fine tuning and, you know, preference tuning and RLHF and different things, but what does it specifically mean that like you would take synthetic data? What does that mean in your case? And like, why does that result in something good in fine tuning an open model? People might think, oh, this is synthetic data.

11:02Why should I expect it to like be any good? So could you kind of help explain that subject a little bit? Yeah, absolutely. So, I mean, out of context, synthetic is like as meaningless as like artificial, right? Data is data. But in this case, it's referring to a particular class of data that's been generated by another language model or another AI, another diffusion model, et cetera, that can actually be used to further train models. Now you might say, why would you want to do something like that? How is it helpful? What was important to us is we were all GPU poor, right? We were all running on laptops or maybe a 3090, maybe a 4090.

11:39As individuals, we don't have data centers. So training or even tuning a large model in the early days, like 70 billion parameters, something like that, was just unfeasible for us. And knowing that GPT-3 is like something like 175 billion parameters and 3.5 and 4 can only go up from there. The question became, how can we make these small 7 billion parameter models even compete with these massive, massive ones? These ones that I want to run offline, these ones that I might want to run on an edge device, on a phone, on a drone, etc. Right? Like, how can I make them even useful? So there's two things to talk about here.

12:18One is synthetic data and the other is distillation, right? So synthetic data is just referring to like any kind of data that's created by a model in this case. And the reason that's useful is in particular distillation. So if I told you to go study comp sci for 10 years, for example, and put in that massive time investment and really focus on general programming, and then I told you, you know, now it's time for you to learn about AI and transformers and stuff and put you through all the math prerequisites, et cetera. Like you're going to come out with like a really strong foundation of how to do the work.

12:56But the problem is you've put in a massive time investment. Now, if I take that guy who spent 10 years doing engineering, then another five years doing AI, and I ask him, Hey, can you teach somebody like just really important, like compressed tidbits that'll help them just get up and running to do the work? That's data distillation, right? That's knowledge distillation. So you look at these big models, like a CLOD or a 70B model or GPT-4, and you can see they're amazing. They're brilliant at everything. They have a bunch of high-quality data they're trained on, and they have a bunch of low-quality data they're trained on that they can interact with and express in a high-quality form.

13:36So instead of me having to read a massive 10 pager for why some chemical reaction or some like tax based process, whatever you want it to be like, instead of reading a massive document on that, and then feeding that to a language model, we can just have that really smart model that already understands it really well, compress that information into an instruction, or into a conversation until like two sentences, three sentences, five sentences, like half a page. And we can just train a much smaller model on that compressed information, and it will learn the compressed information, you know, to the degree that a language model learns something, you know, not perfectly.

14:18But because of that, what the Alpaca guys did was they generated a bunch of seed tasks from GPT 3.5 on various different domains and topics and created these kind of compressed instructions with instruction, an input question from the user, and then an answer. So the instruction could be like, given the following math equation, explain step by step why this is the answer. And then the input is the equation, which is your question. And then the output is the compressed answer. So all of that we can take as one sample in the data set, and we can make hundreds of thousands or millions of samples like that of various different domains and various different tasks.

14:59So the alpaca guys did this, less than 100K examples, I believe. And they trained the llama models on these, and they found massive boosts to performance that this distilled information, like a human, successfully compresses and transfers over. So when I saw that, and then independently when Technium saw that, and then independently when many others saw that, we were like, this is so intuitive. This is exactly how I've learned anything by just going on Discord and Twitter and bothering people to give me the compressed bit of how I do something. We should try doing this with even higher quality models than 3.5.

15:34So we created, I can't remember the exact number at the moment, but at least 50 ,000, maybe 100 ,000 examples originally for Hermes one like this, just using GPT-4. And then we trained on that and ended up getting performance that was extremely, extremely like massive boost compared to the other models that were not trained using this kind of method. So without these giants that have already established themselves in the space, we wouldn't be here. Like without OpenAI, without meta, like we literally wouldn't have the model and the data to do the kind of work that we did to make Herbys. What it allowed for us is like for local models to finally be like comprehensible and for us to finally have like offline capabilities to kind of take the good stuff from something like GPT-4 or something else and make it uncensored.

16:28So it still has all this understanding of all these topics, but it doesn't have all that RLHF inside it necessarily that safetyizes it so that when people utilize the model has all this intelligence, but it has more freedom of thought to kind of converse with you on topics that open AI may reject. Gotcha. One of the things I was curious about as you were going through that was a few episodes back, Daniel and I were kind of talking about the effect of model licensing, you know, on the community and the different kind of licensing concerns that were coming out from whether it be, you know, meta open AI, you name the organization.

17:03Is that ever a challenge for you since you're kind of using those to get started in terms of the inputs? Has that been a concern or do you anticipate it being a concern? I think that, of course, generally, US international regulation on this stuff is evolving. The conversation is evolving very much. So naturally, there's like, you have to keep it top of mind. You have to think about these kind of things. But thankfully, because all of our model releases are like open source and we don't profit from them. If somebody goes off and creates a product using our model, good for them, but we don't necessarily take on the liability or that worry of saying, hey, we're going to sell you this model that was created with GPT-4 outputs.

17:46We actually actively try to stay away from doing that. But because the data distillation paradigm is so effective, if a model comes out that's better than GPT-4 and it's open source and I can use it locally. and in their TOS it says you can use this to make a commercial model, then we can apply the same techniques that we've been preparing and researching and understanding from these closed models and use it there. So right now we don't stand to or try to or have any plans to profit from using any of these outputs. We're not about that because we want to be careful and respectful of these model creators and these companies.

18:25But that being said, we're learning all these techniques and developing all these techniques that will be useful for when that time comes and for when that's available, especially with the advent of something like Mistral. If we do distillation from a Mistral model, like Mistral Medium or something like that, that's completely, from my understanding, barring their TOS saying otherwise, but I believe it doesn't. It's completely okay in that situation for us to create models like this that can be used commercially, et cetera. Regarding the TOS stuff though, as much as we err on the side of caution, I'd find it hard to see a company enforce their TOS when these larger models are likely trained on not all copyright-free stuff.

19:14I'd find it hard-pressed to believe that these closed source companies, their models are totally copyright-free and totally copyright-clean. So if some other company that was feeling a little more rambunctious than ourselves was to say, you know, we are going to commercially release on this, I imagine it'd be difficult for them to be come after without the other group opening their books. And there's actually a pretty interesting interaction that happened regarding this between Google and OpenAI, if you guys are familiar. here. So yeah, I saw this interesting picture the other day. It was like the interesting web of AI and it was like how Microsoft, Google, OpenAI, it's like on one side there's the ones and it shows how they're connected to the other ones.

20:00It's like this visualization and like how many of them overlap in these strange ways between like whether it's Together or Mistral or Meta, Google, Microsoft, OpenAI is sort of very interesting web of connections that probably make some of these things rather difficult. Leave it for the lawyers to sort out. Yeah. Yeah. That's the thing is like, we can look at an example, right? Like you hear that phrase, like good artists copy, great artists steal, right? Like, so the data distillers, we're copying, right? Like we're just distilling this information. Like we're trying to like make our models more like those.

20:38And we don't really plan to commercialize. We're just doing it for free for everyone. But the great artists are, you know, Google, you know, like you look at Bard and it tells you, you know, I was made by OpenAI. Now it's fine for our open source model to say I was made by OpenAI because we're very transparent about this is trained on GPT outputs. But when Bard violates the TOS with a paid product. Bold. Yeah, that sounds like I was trained by OpenAI, right? You think that OpenAI would come after this multi-billion dollar company like immediately, right? Instead, you see a tweet from, first you see Google deny it.

21:12Then you see a tweet from Sam Altman, which was something along the lines of, I'm paraphrasing here, something along the lines of, I'm not mad that they trained on our outputs. I'm mad that they lied about it. And I'm sitting there like, okay, you're mad about this, but aren't you going to pursue the legal action in your terms of services? No, no. Because everyone would have to open their books up too. That being said, I don't condone the commercial use of that kind of stuff. Like making a paid model from GPT-4 outputs, I wouldn't advise anyone sell a model made with them just because we want to respect people's TOS and stuff.

21:51They worked hard and spent billions to make this stuff or hundreds of millions, however much they spent. But there is certainly room for hypocrisy in that realm of the large corps. So that's my thoughts on the licensing stuff. And that's definitely my own individual thoughts. We're a pretty decentralized collective at Noose, so you'll find people with all sorts of opinions all over the place. And as a company, we don't hold any view whatsoever on that. Yeah. I'm wondering, maybe this gets a little bit to the distributed nature of this, but I know that there's sort of various collections of what the Noose Research Group has done over time.

22:33You mentioned Hermes, but then there's these other kind of categories of things too, like the yarn models, capybara, puffin, obsidian, just looking over the hugging face now. I'm wondering if you could just give us, from your perspective, a little bit of a map of these different things and how people might categorize the different collections of what Noose has done. I definitely want to talk about the future things and ongoing things as well. But as it stands now, what are the major categories of what the collective has invested their time in over time? Certainly, certainly. So within the stuff that's viewable on Hugging Face, at least, we've got the Hermes series, of which, like I told you guys, the initial story of how it went down.

23:20But from there, Technium kept going. I haven't personally had any interaction with the Hermes model since the initial. From there, Tech just continued to create more and more synthetic data, collect from more and more sources, use more and more open data sets. And he's just got the, I guess, award-winning data thesis. The guy really knows how to go about curating and synthesizing good data. So Technium, it's his baby, the Hermes project. So everything you've seen since is really his work and anyone who has kind of collaborated with him. But almost like you can't call it anything a solo project because of the open data sets we use to like everything is built on the shoulders of giants and the shoulders of each other as little people.

24:03But tech really has helmed the Hermes initiative so far. I think that's our most popular model series. And he released the open Hermes as well, because we had some data in the original Hermes that we never released publicly. And we wanted to make that kind of an option for everybody. So that's Hermes. Still follows the same kind of philosophy of synthetic data. And it now uses the chat ML format instead of the alpaca format is what we kind of upgraded to. Then you've got a Capybara and Puffin, which are both done by a volunteer and, you know, OG member LDJ, who you may be familiar with, Luigi Daniel Jr.

24:40So the Capybara series was using an amplify instruct method, this novel method that LDJ had worked on alongside another one of our researchers, J. So LDJ and J can get confusing, but the two of them worked on the Capybara series, created the dataset, trained the models. And then Puffin was the idea of using hand-picked smaller samples from some of our larger data sets to make sleek data sets for an easy tune and see how that works kind of in the spirit of the Lima paper, where they just used a few examples to get really good results. Those are really the popular tunes using synthetic data for general use.

25:23Yarn is this novel context length extension method at the time of creation by Emozilla, also known as Jeffrey Cannell, and Bowen Peng, also known as Block 97, alongside Enrico Chipotle and Eleuther AI. So what happened there was these guys were already looking into contests like the extension for a while. And when we kind of came under the noose banner to do the work, it opened up a little bit of resources from compute sponsorships. It opened up a more centralized place for them to be able to do that collaboration. I had no hand in the yarn models whatsoever. And that's the exciting thing is everyone really gets to work in their own spheres and their own kind of autonomous circles.

26:10And then we just check in and see, you know, how's the research going? How's it coming along? Because we really work with people that we heavily believe in and we believe in their idea. So if we don't already have an idea, we're kind of just say, please freely create because we brought you in because what we will freely create will push forth our agenda anyway. So I think those are our big model releases and series that we have available. Outside of that, we have a bunch of stuff on our GitHub as well. Stuff that's being worked on, stuff that hasn't necessarily come out yet. There's a lot of that.

26:44So I got a question for you as a follow-up. It's pretty fascinating, the story that you've been telling us here because of that kind of organic, you know, creation of the organization or collective. And I'm wondering as you've done that and you kind of went through and talked about the different model groups and kind of talked about, you know, the owners or spiritual owners, if you will, of each of those families, how do the different members of the collective interact to kind of share? Like how do you each push each other along or share information or give ideas so that cross-family efforts can kind of benefit from the overall collective.

27:21And as you said, now a C-Corp and you guys are more organized at this point. So what kind of culture has developed around those communications and learnings? Yeah, absolutely. I mean, when it started, it was just like a small Discord, maybe like 10 people. From there, like we kind of created more channels as people wanted to work on more things. and we had initially split up into like three, four different topics or sectors that people could assign themselves to. One being data synthesis, of course, so we can kind of find new novel methods and formats for distillation and the creation of synthetic data.