AI trends: a Latent Space crossover

28:00How do you personally stay on top of models? You know, there's leaderboards, there's Twitter, there's LinkedIn. Yeah, I think it's a little bit spread out for me between the sources that you mentioned. As podcasters, I think that's one of the - Yeah, it's one. Well, it's also a benefit for us. I think like, if I didn't have every week on Wednesday, like I'm going to talk about this topic, whether like I'm planning to think about think about a certain thing or not, it kind of helps you prompt and look, look at what's going on. So I think that is an advantage of like content creators is it is kind of a responsibility, but it's also an advantage that we can have to like have the excuse to have great conversations with people every week.

28:47But yeah, I think Twitter is a little bit weird now, as everybody knows, but it's still a good place to find out that information. And then sometimes too, like, to be honest, I go to Hugging Face and like, I'll search for models, but I also search and I look at the statistics around the downloads of models because generally when people find something useful, then they'll download it and download it over and over. So sometimes when I hear about like a family of models, I'll go there and then I'll look at some of the statistics on Hugging Face and like try a few things. Yeah. And some of these forks, I see the download numbers, but I've never heard of them outside of Hugging Face.

29:26Yeah, it's true. It's true. Yeah. And some of them, like there'll be a fork or like a fine tune or something. And you do have to do a little bit of digging around like licensing and that sort of thing too. But it is a useful, like there's tons of people doing amazing stuff out there that aren't getting recognized at the like, you know, Falcon or MPT level. but there's a lot of people doing cool stuff that are releasing models on Hugging Face maybe that they've just found interesting. Any unusual ones that you recently found? Well there's one that I'll highlight which I thought was cool because I don't know if you all saw the meta released this the six modality model.

30:08Yeah yeah and it was interesting because we did this work with Masakane when I was at SIL. We did this work with Masakane and Koki, which is a speech tech company to create these language models in like six African languages. And I was like, okay, that's cool. Like we did that. We formed like the data sets. It was satisfying. But now I'm like learning that then Meta went and found that data on Hugging Face. And that's kind of incorporated in these these new models that meta has released so it's cool to see like the full cycle thing happen where there was grassroots organizations seeing a need for models gathering data doing baselines and now there's like extended functionality in kind of like a more influential way i guess at like that higher level yeah yeah i think i mean talking about open and and closed models when we started of the podcast, it kind of looked like a cathedral kind of market where we had Cohere, Anthropic, OpenAI, Stability, and those were like the hottest companies.

31:18I think now, you know, as you mentioned, you go on Hugging Face, like I just opened it right now. There's this other news research 13 billion parameters model that just got released, fine tuned on over 300 ,000 instructions. It's like models are just popping up everywhere, which is great. And yeah, we had an episode with, as I mentioned, with Jonathan Frankel and Abhinav from Mosaic ML to introduce MPT7B and some of the work that they've done there. And I think like one of their motivation is like keeping the space as open as possible, like making it easy for anybody to go, obviously, ideally on Mosaic ML's platform and train their own models and whatnot.

31:58So that's one that people really liked. I thought it was really technical. So I was really a little worried at first. I was like, is it going to fly over most people's head? But it was actually super well received. No, we're going more technical. Exactly. Now, that was a good learning. Leaning in. Exactly. And Jonathan is super passionate about open source. He had this rant halfway through the episode about why it's so important to keep models open. And I actually edited in a crowd applause into the podcast, which I kind of love. I love little audio bonuses for people listening along. and I think the changelog guys do that really well especially in their newer episodes yeah we need to, there is a way for us to integrate some of those things, yeah like the soundboard thing and we've never got into it too much, I need to work with Jared from the changelog and see it just spices it up, exactly you can only have so many hour long conversations about ML yeah I keep thinking that but then we keep going sorry I didn't mean It was like a switch it up and makes the audio interesting to add variety.

33:07Cool. I don't know if there are any other highlights that we want to do for. I'll just highlight maybe one more. Kirsten Lum was on. She had an episode about machine learning at small organizations. I think that's a great one. Like if you're a data scientist or a practitioner or an engineer at like either a startup or a midsize company where I think the thing that she emphasized was these different tasks that we think about, like whether it's curating a data set or training a model or fine tuning a model or deploying a model. Sometimes at like a larger organization, those are functions in and of themselves.

33:44But when you're in this sort of mid-range organization, that's like a task you do, right? So to think about like those tasks as tasks of your role and time box them and understand like how to do all of those things, well without getting sucked down into any one of those things. That was an insight that I found quite useful in my day-to-day as well as to sort of start to get a little bit of like spidey sense around, hey, I'm spending a lot of time doing this, but which probably means I'm like stuck in too much. Like I'm making my ML ops too complicated, right? To track versions and like tie all this stuff together.

34:25Maybe I should just like do a simple thing and like paste a number in a Google sheet and move on or something. I think that's a good segue into some of the other work that you do. You run the datadan.io website, which is kind of like a different types of workshop and advising that you do. I think a lot of founders especially are curious about how are companies thinking about using this technology. There's a lot of demos on Twitter, a lot of excitement, but when founders are putting together something that they want to sell, they're like, okay, what are the real problems that enterprises have? What are some of the limitations that they have?

35:01We talked about commercial use cases and something like that. Can you maybe talk a bit about two, three high-level learnings that you had from these workshops on how these models are actually being brought into companies and how they're being adopted? Yeah, I think maybe one higher-level comment on this is even though we see all these demos happening, everybody's using chat GPT. The reality in enterprise is most enterprises still don't have like LLMs integrated across their technology stack, right? So that might be a bummer for some people like, oh, it's not quite as pervasive, but I actually find it as refreshing maybe because some of us feel like stuff happens every week.

35:48It's exhausting to keep up like, oh, if I don't keep up with this stuff, then like I'm getting left behind. But it takes time for these things to trickle down. And not everything like we were talking about the stable diffusion use case and others, like not everything that's hyped at the moment will be a part of your like day to day life forever. Right. So you can kind of take some comfort in that. I think it's really important for people to, if they're interested in these models, to really dig into more than just kind of a single prompt into these models. The practical side of using generative text models or LLMs really comes around either what some people might call prompt engineering, but understanding things like giving examples or demonstrations in your prompt, using things like guardrails or regex statements or prediction guard to structure output, doing fine tuning for your company's data.

36:49But there's kind of a hierarchy of these things. I think you all know Travis Fisher. He was a guest on Practical AI and talked about this hierarchy from prompt engineering through data augmentation to fine tuning to eventually training your own generative model. I've really tried to encourage enterprise users and those that I do workshops with to think something like that hierarchy with these models. Get hands-on, do your prompting, but then if you don't get the answer that you want immediately, I think there's a tendency for people to say, oh, well, it doesn't work for my use case. But there's so much of a rich environment underneath that with things like LinkChain and Llama Index and data augmentation, chaining, customization, fine-tuning, all this stuff that can be combined together.

37:48It's a fun new experience, but I find that enterprise users just haven't explored past that very most shallow level. So I think, yeah, in terms of the trends that I've seen with the workshop, I think people have gone to ChatGPT or one of these models. They've seen like the value that's there, but they have a hard time connecting these models to a workflow that they can use to solve problems. Like before we all had intuition, like I'm going to gather my data. It's going to have these five features. I'm going to train my scikit learn model or whatever. I'm going to deploy it with Flask. And like now I have a cool thing.

38:28Now all of that intuition has sort of been shattered a little bit. So we need to develop a new workflow around these things. And I think that's really the focus of the workshops is kind of rebuilding that intuition into a practical workflow that you can think through and solve problems with practically. You have a live prompt engineering class? Prompt engineering overrated or underrated? Yeah, I think prompt engineering as like a term is probably too hyped. I think engineering and ops around large language models though is a real thing. And it is sort of what we're transitioning to. Now, how much you want to say is like, that term gets used in all sorts of different contexts.

39:16It could mean just like, oh, I wrote a good prompt and I'm going to like sell it on Twitter or something. Prompt base, the marketplace of prompts. I wonder how they're doing, to be honest, because they get quoted in almost every article about prompt engineering. They got really, really good PR. Yeah, yeah. I mean, if people can sell their prompts, I mean, I'm all for that. That's cool. I got prompts right here. But I think it goes like some people might just mean that. And I think that's maybe overhyped in my view. But I do think there's this whole level of engineering and operations around prompts and chaining and data augmentation that is a real workflow that people can use to solve their problems.

39:55And that's more what I mean when I'm referring to like, whatever, however you want to combine the word engineering with prompting and language models. Yeah, I've just been calling it AI engineering. AI engineering. That's good. Wrangle with the AI APIs, know what to do with them. And that is a skill set that is developing that is a subspecialty of software engineering. Yeah. Yeah. It is what it is. And I think part of something I'm really trying to explore is this spillover of AI from the traditional ML space, like where you need a machine learning researcher or machine learning engineer. It's spilling over into the software engineering space.

40:29And there's this rising class of what I'm calling AI engineer that is specialized in and conversing in the research, the tooling, the conversations and themes. What do you think are the unique challenges that like someone coming from that latter group, like engineers that are advancing into this AI engineer position versus like probably more like my background where I was in data science for some time and now I'm kind of like transitioning into this world. What do you think are the unique challenges for both groups of people? Oh, I mean, so I can speak to the software side and you can speak about the data science side.

41:05It's simply that we are, for many of us, dealing with a non-deterministic system for the first time. That, by the way, we don't fully control because there's this conversation about did GPT-4 regress in its quality? And we don't know because model drift is not within our control because it's a black box API from OpenAI. But beyond that, there's this sense that the latent space of capabilities is not fully explored yet. There's 175 billion or 1 trillion parameters in the model. We're maybe using like 200 of them. It's literally that meme where we're using 10 % of our brain. We're probably using 10 % of what is capable in the model.

41:48And it takes some ingeniousness to unlock that. Yeah, I think from the data science perspective, there's probably a desire to quickly to jump to these other things around fine tuning or training your own models, where if you really do take this prompting, chaining, data augmentation seriously, you can do a lot with models sort of off the shelf and don't need to like jump immediately into training. So I think that is like a knee-jerk reaction on our end. And fine-tuning is going to be around for the foreseeable future, as far as I can tell. But data scientists have maybe a different... Because we've been dealing with the uncertainty or non-deterministic output for some time and have developed some intuition around that.

42:38But that's mostly when we've been controlling the data sets, when we've been controlling the model training and that sort of thing. So to throw some of that out, but still deal with that, it's a separate kind of challenge for us. I just remembered another thing that we've been developing on the latent space community, which is this concept of AI UX. Yeah. Right. That the last mile of showing something on the screen and making it consumable, easily usable by people is perhaps as valuable as the actual training of the model itself. Yes. So I don't know if that's an overstatement, to be honest. Like obviously you're spending like hundreds of millions of dollars trading models and like, you know, putting it in some kind of react app.

43:16It's not it's not the biggest innovation in the world. But a lot of people from OpenAI say like ChatGPT was mostly a UX innovation. Yeah, I think like leading up to ChatGPT, like when I saw the output of ChatGPT, it wasn't I don't think I had the same earth shattering experience that other people had in believing like, oh, this output is coming from a model like that. sure it came from a model but the reception to like that interface and like the human element of the dialogue like that was so maybe it's both and right like it's not like you're not gonna get that experience if you don't have the innovation under the hood and the modeling and the data set curation and all of that but it can totally be ruined by like the ux i typically give the example like one day in Gmail, I logged in and like I was typing my email and then like had the gray autocomplete.

44:11Right. I did not get like a pop up that said, like, do you want us to start writing your emails with AI? Like it just like was so smooth and it happened and it made like it created value for me instantly. Right. So I think that there is really a sense to that, especially in this area where people have a lot of like misgivings or fear around the technology itself. And we're going to have Alex Gravely on in a future episode, but GitHub, when they had the initial codex model from OpenAI, they spent six months tuning the UX just to get Copilot to a point where it's not a separate pane, it's not a separate text box, it's kind of in your code as you write the code.

44:50And to me, that's more to domain traditional software engineering rather than ML engineers or research engineers. Yeah, I would say that is probably, yes, to circle back to what we were talking about, like challenges that are unique to like engineers coming into this versus like data scientists coming into this. That's something data scientists, I think, have not thought about very much at all. At the very most, it's data visualization that they've thought about, right? Whereas engineers generally, like there's some human, I mean, unless you're just a very pure backend systems engineer, like thinking about UI, UX is maybe a little bit more natural to that group.

45:30You mentioned one thing, which is about data set curation. We're in the middle of preparing this long overdue episodes on Datasets 101. Any reflections on the evolutions in natural NLP datasets that have been happening? Yeah, great question. I definitely like, I think, are you all familiar with Label Studio? And it's one of the most popular kind of open source frameworks for data labeling. and they've been, I think they've been on, we have them on the show, like we try to have them on the show every year as like data labeling expert. Maybe it's time for that. It's just reminding me. So they just released, so Erin McHale is in the Latent Space Discord.

46:11I think you had her on at the ODSC. She was at ODSC. Yeah, that's right. So they just released new tools for fine tuning generative AI models. Exactly, yeah. It's a good occasion. I think maybe that being an example of this is maybe a trend that we're seeing there is around augmented tooling or tooling that's really geared towards an approachable way to fine-tune these models with human feedback or with customized data. So like I know with Label Studio a lot of the recent releases had somewhat to do with like putting LLMs in the loop with humans during the label process similar to like I think Prodigy has been doing this for some time, which is from Spacey.

46:57So this sort of human in the loop labeling and update of a model, they brought some of that in. But now like this new kind of set of tooling around specifically instruction tuning of models, I think before maybe people, and I've seen actually this misconception, I was in a advising call with a client, and they're really struggling to understand like, okay, our company has been training or fine tuning models. Now we want to create our own like instruction tuned model. Like how is that different from what we've been doing in the past? And kind of what I, I tried to help them see is yes. Like some of the workflow that happened around like reinforcement learning from human feedback is unique, but reinforcement learning is not unique.

47:49There's an element of training in that there's data set curation in that there's pre-training that happen like before that whole process happened. So the elements that you're familiar with are part of that. They're just not packaged in the same way that you saw them before. Now there's this clear pre-training stage and then the human feedback stage and then this reinforcement learning happens. So I think the more that we can bring that concept and that workflow into tooling, like what Label Studio is doing, to make it more approachable for people to where it's not like this weird, like reinforcement learning from human feedback sounds very confusing to people like PPO and helping people understand like how reinforcement learning works.

48:36It's very difficult. So the more the tooling can just have its own good UI UX around that process, I think the better and probably label studio and others are leading the front on leading the way on that front. I was thinking like, so labels are one thing. And by the way, okay, I'll take this side tangent on labels and I'll come back to the main point. I actually presume that scale would win everything. Yeah. And it seems like they haven't. Yeah. And, and I'm sorry, the scale, this snorkel there's this generation of labeling companies like data centric ai right yeah right what happened like how come there's still new new companies coming up there's label box there's label studio i don't have a sense of how to think about these companies like obviously labels are important yeah yeah i think also even before that there was like tool at least um features even from cloud providers or whatever like auto ml yeah like came before that like upload your own data create your own custom model.

49:34So I think that maybe it's that like companies that want to create this sort of custom models, and this is just my own opinion, I'll preface that. Maybe they don't want, like when they're thinking about that problem, they're not thinking about, oh, I need a whole platform to create custom models using our data. They're more thinking about like how do I use these state-of-the-art models with my data and so it's still if those statements are very similar but if you notice like one is more model centric and one is more data centric so I think enterprises are still thinking like model centric and augmenting that with their data whether that be just through augmentation or through fine-tuning or training they're not necessarily thinking about like a data platform for AI.

50:26They're thinking about bringing their AI or their data to the AI system, which is why I think like APIs like Cohere, OpenAI that offer fine tuning as part of their API. It's sort of like people love that. It makes sense. Like, okay, I can just upload some examples and it makes the model better, but it's still like model centric, right? Yeah. I get the sense that OpenAI doesn't want to encourage that anymore because they don't have fine tuning for 3.5 and 4. And then, so the last thing I'll do about data sets and we can go into the lightning round is I was actually thinking about unlapped data sets for unsupervised learning or self-supervised learning, right?

51:04Like that is something that we are trying to wrap our heads around like Common Crawl, Stack Overflow Archive, the books, you know, I don't know if you have any perspectives on that, like the trends that are arising here, the best practices. And And as far as I can tell, nobody has a straight answer as to what the data mix is. And everyone just kind of experiments. Yeah. Well, I think that's partly driven by the fact that the most popular models, you don't really have a clear picture of what the data mix is, right? So the people that are trying to recreate that and they're not achieving that level of performance, right?

51:40Then one of the things that they know is, well, what are all the different data mix options that I can try and try to replicate some of what's going on? So I think it's partly driven by that is like we don't totally know what the data mix is like sitting behind the curtain of OpenAI or others. But I think there's a couple of trends, I guess, which you've already sort of highlighted. One is like, how can I mix up all of these public data sets and filter them in unique ways to make my model better? So some, I listened to a talk, I believe it was at last year's ACL and they did this study of common crawl, right?

52:24And they found that actually a significant portion of common crawl was like mislabeled all over the place, right? Like trash. Yeah. So like, I think it was 100 % of the data that was labeled as Latin character Arabic. So Arabic written in Latin characters was not Arabic, like 100 % of it. And there was like all sorts of other problems and that sort of thing. So I think there's one side, one group of people or set of experiments that you could think about as like, how do I take these existing data sets, which I know have data quality issues? or maybe other data biases or problems that I would like to filter out, like not fit for work data, that sort of thing.

53:11So how do I create my own special filtered mix of these and train a model? So that's one kind of genre. And then there's the other genre, which is like maybe taking those, but augmenting them with this like simulated or augmented data, right? That's out of a model, like a GPT model or something like that. So I think you could combine those in all sorts of unique ways. And I think it is a little bit of like the Wild West because we don't totally have a good grip on what is the winning strategy there. And so I think that's where I would also encourage people to try a variety of models. So this is maybe a problem with benchmarks in general, right?

53:49Like you can see like the open large language model benchmark on Hugging Face and like these models are at the top. And you could come away with that and say, well, I'm like anything below like the top three I'm not even going to use. Right. But the reality is that each of those had a unique sort of flavor of this data under the hood that might actually work quite well for your use case. So one example that I've used recently in some work is the Camel 5 billion model from Ryder. You know, it doesn't work great for a lot of things, but there's certain things around like marketing copy and others that it does a really good job at.

54:30And it's a bit smaller model that I can host and run and I can get good output out of it if I put in some of that workflow and structuring around it. But I wouldn't use it for other cases. but that has a lot to do with the data and you know right i'm guessing writers focus on that copy generation and such so yeah i would encourage people specifically on this topic to maybe think about what's going on under the hood and also give some models a try for different like gain your own intuition about how a model behavior might change based on like how it was trained and the mix of data that went in awesome let's jump into the lightning round we have three questions for you.

55:12It's lightning, but you can take 30 seconds to answer. All right, cool. So the first question is around acceleration. What's something that already happened in AI that you thought would take much longer? Yeah, I think the thing that I was thinking about here was like how general purpose these large language models are beyond traditional NLP tasks. So it doesn't surprise me that maybe they could do like sentiment analysis or even like NLI or something like that. These are things that have been studied for a long time. But the fact that I can like at ODSC, I was in like a workshop on fraud detection and they were using like some, I forget the models they were using, some statistical models to do fraud detection.

55:57I was like, I wonder if I just like do a bit of chaining and like insert some of the examples of these insurance transactions into my prompts if I can get the large language model to detect a fraudulent insurance client. And it seemed to like, like I got pretty far doing that. So that fact of like, you can do something like that with these models or that generalizable beyond traditional NLP techniques, I think is surprising to me. Awesome. Exploration. What are the most interesting unsolved questions in AI. Yeah, I think there is still such a focus on English and Mandarin. It's like that, like you're kind of large language model wise.

56:42If you look at the drop off and performance after you get past like English, Mandarin, German, Spanish to some degree, but German is actually better than Spanish because of how much it's been studied in NLP. And of course, Mandarin has a lot of data. Spanish still does good, but like there's languages, even in the top hundred languages of the world that are spoken by millions and millions and millions of people around the world that don't like perform well in these models. So that's like thing one, but even modality wise, I know there's a lot of work going on in the research community around sign language, but like There's all of these different modalities of language.

57:24Written text does not equal communication. Written text is a synthesis of communication into a written form that some people consume. but the combination of all of these modalities along with all of these languages there's just so much room to explore there and so many challenges left to explore that will eventually I think help us learn a lot about communication in general and the limitations of these models but is an exciting area it's definitely a challenge but an exciting area awesome and so one last takeaway what's something or a message that you want everyone to remember today Yeah, similar to when you were asking about my workshops, I think I would just encourage people to get hands-on with these models and really dig into the new sets of tooling that are out there.

58:18There's so much good tooling out there to go from a simple prompt to inject your own data to form a query index to create a chain of processing, even trying agents and all those things. like get hands on and try it. That's the only way that you're going to build out this intuition. So yeah, that's, that would be my encouragement. Excellent. Well, thanks for coming on. Yeah. Thank you guys so much. This is awesome.

58:52Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. and if you're a long-time listener of the show, help us reach more people by sharing practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Change Talk podcasts. Check out what they're up to at fastly.com and fly.io. And to our Beat Freakin' residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.

59:29trabajapol bots

59:35Game on!