Towards high-quality (maybe synthetic) datasets

0:05Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io.

0:44okay friends i'm here with annie sexton over at fly and you know we use fly here at changelow we love fly it is such an awesome platform and we love building on it but for those who don't know much about fly what's special about building on fly fly gives you a lot of flexibility like a lot lot of flexibility on multiple fronts. And on top of that, you get, so I've talked a lot about the networking and that's obviously one thing, but there's various data stores that we partner with that are really easy to use. Actually, one of my favorite partners is Tigris. I can't say enough good things about them when it comes to object storage.

1:24I've never in my life thought I would have so many opinions about object storage, but I do now. Tigris is a partner of Fly and it's S3 compatible object storage that basically seems like it's a CDN, but is not. It's basically object storage that's globally distributed without needing to actually set up a CDN at all. It's like automatically distributed around the world. And it's also incredibly easy to use and set up, like creating a bucket is literally one command. So it's partners like that, that I think are this sort of extra icing on top of Fly that really makes it sort of the platform that has everything that you need.

2:00So we use Tigris here at Changelog. Are they built on top of Fly? Is this one of those examples of being able to build on Fly? Yeah. So Tigris is built on top of Fly's infrastructure, and that's what allows it to be globally distributed. I do have a video on this, but basically the way it works is whenever, like, let's say a user uploads an asset to a particular bucket. Well, that gets uploaded directly to the region closest to the user. Whereas with a CDN, there's sort of like a centralized place where assets need to get copied to. And then eventually they get sort of trickled out to all of the different global locations.

2:33Whereas with Tigris, the moment you upload something, it's available in that region instantly. And then it's eventually cached in all the other regions as well as it's requested. In fact, with Tigris, you don't even have to select which regions things are stored in. You just get these regions for free. And then on top of that, it is so much easier to work with. I feel like the way they manage permissions, the way they handle bucket creation, making things public or private is just so much simpler than other solutions. And the good news is that you don't actually need to change your code if you're already using S3.

3:07It's S3 compatible. So like whatever SDK you're using is probably just fine. And all you got to do is update the credentials. So it's super easy. Very cool. Thanks, Annie. So Fly has everything you need. Over 3 million applications, including ours here at Changelog, multiple applications have launched on Fly. Boosted by global anycast load balancing, zero configuration private networking, hardware isolation, instant wire guard VPN connections, push button deployments that scale to thousands of instances. It's all there for you right now. Deploy your app in five minutes. Go to fly.io. Again, fly.io.

3:55Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, where we're building a private Securigen AI platform. And I'm joined, as always, by Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How are you doing, Chris? Great today, Daniel. Daniel, how are you? It's a beautiful, beautiful fall day and a good day to take a walk around the block and think about interesting AI things and clear your mind before getting back into some data collaboration, which is what we're going to talk about today. Chris, I don't know if you remember our conversation.

4:37It was just me on that one, but with Bing Soon Chua, who talked about broccoli AI, the type of AI that's healthy for organizations. And in that episode, he made a call out to Argeala, which was a big part of his solution that he was developing in a particular vertical. I'm really happy today that we have with us Ben Bertenshaw, who is a machine learning engineer at Argeala, and also David Berenstein, who is a developer advocate engineer working on building Argeela and Distilled Label at Hugging Face. Welcome, David and Ben. Thank you. Great to be here. Hi. Thanks for having us. Yeah, so like I was saying, I think for some time, maybe if you're coming from a data science perspective, there's been tooling maybe around data that manages training data sets or evaluation sets or maybe MLOps tooling and this sort of thing.

5:40And part of that has to do with preparation and curation of data sets. But I found interesting, I mentioned the previous conversation with Bing Soon, he talked a lot about collaborating with his sort of subject matter experts in his company around the data sets he was creating for text classification. And that's where Arjila came up. So I'm wondering if maybe one of you could talk a little bit at a at a higher level when when you're talking about data collaboration in the context of the current kind of ai environment what what does that mean generally and how would you maybe distinguish that from previous generations of of tooling and maybe similar or different ways so data collaboration at least from from our point of view is kind of the collaboration between both the domain level experts that really have high domain knowledge actually know what they're talking about in terms of the data, the inputs and the outputs that the models are supposed to give within their domain.

6:43And then you have the data scientists or the AI engineers and this side of the coin that are more technical. They know from a technical point of view what the models expect and what the model should output. And then the collaboration between them is now even higher because nowadays you can actually prompt other lamps with natural language and you actually need to ensure that both the models actually perform well and also the prompts and these kind of things. So the collaboration is even more important nowadays. And that's also the case for still the case for text cap models and these kind of things which we also support within our job.

7:17I guess maybe in the context of let's say there's a new team that's exploring the adoption of AI technology maybe for the first time. Maybe they're not coming from that data science background, the sort of heavy ML ops stuff, but maybe they've been excited by this latest wave of AI technologies. How would you go about helping them understand how their own data, the data that they would curate, the data that they would maybe collaborate on is relevant to and where that fits into the certain workflow? So So yeah, imagine someone may be familiar with what you can do with chat GPT or pasting in certain documents or other things.

8:02And now they're kind of wrestling through how to set up their own domain specific AI workflows in their organization. What would you kind of describe about how their own domain data and how collaborating around that fits into common AI workflows? Yeah, so something that I like to think about a lot around this subject is machine learning textbooks. And they often talk about modeling a problem as well as building a model, right? There's a famous mama and matter cycle. And in that, when you model a problem, you're basically trying to explain and define the problem. So I have articles and I need to know whether they are a positive or negative rating.

8:45and I'm describing that problem and then I'm going to need to describe that problem to a domain expert or an annotator through guidelines and when I can describe that problem in such a way that the annotator or the domain expert answers that question clearly enough then I know that that's a modelled and clear problem and it's something I could then take on to build a model around in simple terms it makes sense and so I think when you're going into a new space like generous AI, and you're trying to understand your business context around these tools, you can start off by modeling the problem in simple terms by looking at the data and saying, okay, does this label make sense to this articles?

9:27If I sort all these articles down by these labels or by this ranking, are these the kinds of things I'm expecting? Starting off at quite low numbers, right? Like single articles and kind of building up to tens of hundreds. And as you do that, you begin to understand and also iterate on the problem and kind of change it and adapt it as you go. And once you've got up to a reasonable scale of the problem, you can then say, all right, this is something that a machine learning model could learn. I guess on that front, maybe one of the big confusions that I've seen floating around these days is the kind of data that's relevant to some of these workflows.

10:08So it might be easy for people to think about a labeled data set for a text classification problem, right? Like here's this text coming in. I'm going to label it spam or not spam or in some categories. But I think sometimes a sentiment that I've got very often is, hey, our company has this big file store, right, of documents. and somehow I'm going to fine-tune quote-unquote a generative model with just this blob of documents and then it will perform better for me. And there's two elements of that that are kind of mushy. One is like, well, to what end for what task, right? What are you trying to do?

10:52And then also how you curate that data then really matters. Is this a sentiment that you all are seeing or how for this latest wave of models, like how would you describe if a company has a bunch of documents and they're in this situation, they're like, hey, we know we have data and we know that these models can get better and maybe we could even create our own private model with our own domain of data. What would you walk them through to explain where to start with that process and how to start curating their data maybe in a less general way, but towards some end? I think in these scenarios, it's always good to first establish a baseline or a benchmark, because what we often see is that people come to us or come to the open source space.

11:40They say, OK, we really want to fine tune a model. We really want to do a super extensive RAC pipeline with all of the bells and whistles included, and then kind of start working on these documents. But what we often see is that they don't even have a benchmark. They don't have a benchmark to actually start with. So that's normally what we recommend. Also, whenever you work with a RAC pipeline, ensure that all of the documents that you index are actually properly indexed, properly chunked. Whenever you actually execute a pipeline and you would store these retrieved documents or these based on the question and the queries in RGLR or any other data annotation tool, you can actually have a look at the documents, see if they make sense, see if the retrieval makes sense, but also if the generated output makes sense.

12:24And then whenever you have that baseline set up, from there, actually start iterating and then kind of making additions to your pipeline. Shall I add re-ranking potentially to the retrieval if the retrieval isn't functioning properly? Shall I add a fine-tuned version of the model? Should I switch from the latest Lama model of 3 billion to 7 billion or these kind of things? And then from there on, you can actually consider maybe either fine-tuning a model if that's actually needed or fine-tuning one of the retrievers or these kind of things. As you're saying that, as you're speaking from this kind of profound expertise you have, and I think a lot of folks really have trouble just getting started.

13:01And like you asked some great questions there. But I think some of those are really tough for someone who's just getting into it, like which way to go and some of the selections that you would go with that. could you talk a little bit about the kind of go back over the same thing but um kind of make up a little workflow you know that's kind of hands-on on just like you might see this and this is how i would decide that just for a moment just so people can kind of grasp kind of the thought process you're going because you kind of described a process but uh if you could be a little bit more descriptive about that um i think when i talk to people once they get going they kind of go to the next step and go to the next step and go to the next step.

13:38But the first four or five big question marks at the beginning, they don't know which one to handle. So I can add some practical steps onto that that I've worked with in the past. That'd be fantastic. Yeah. So one thing that you can do that is really straightforward is actually to write down a list of the kinds of questions that you're expecting your system to answer. And you can get that list by speaking to domain experts, or if you are a domain expert, you can write it yourself, right? And it doesn't need to be an extensive, exhaustive list. It can be quite a small starting set. You can then take those questions away and start to look at documents or pools and sections of documents from this lake that you potentially have and associate those documents with those questions and then start to look if a model can answer those questions with those documents.

14:31in fact by not even building anything by starting to use say chat tpt or hugging chat or any of these kind of interfaces and just seeing this very very low simple benchmark see is that feasible whilst at the same time starting to ask yourself can i as a domain expert answer this and that's kind of where argilla comes in at the very first step so you start to put these documents in front of people with those questions and you start to search through those documents and say to people, can you answer this question? Or here's an answer from a model to this question in a very small setting. And you start to get basic early signals of quality.

15:13And from there, you would start to introduce proper retrieval. So you would scale up your doc, you would take all of your documents, Say you had 100 documents associated with your 10 questions. You put all those 100 documents in an index and iterate over your 10 questions and see, okay, are the right documents aligning with the right questions here? Then you start to scale up your documents and make it more and more of a real world situation. You would start to scale up your questions. You could do both of these synthetically. And then if you still started to see positive signals, you could start to scale.

15:49and if you start to see negative signals, I'm no longer getting the right documents associated with the right questions. I personally would always start from the simplest levers in a RAG setup. And what I mean there is that you have a number of different things that you can optimize. So you have retrieval, you can optimize it semantically or you can optimize it in a rule-based retrieval. You can optimize the generative model, you can optimize the prompt. And the simplest movers, the simplest levers, are the rule-based retrieval, the word search, and then the semantic search. So I would first of all add like a hybrid search.

16:30What happens if I make sure that there's an exact match in that document for the word in my query? Does that improve or improve my results? And then I would just move through that process, basically.

16:55what's up friends i'm here with a friend of mine a good friend of mine michael greenwich ceo and founder of work os work os is the all-in-one enterprise sso and a whole lot more solution for everyone from a brand new startup to a enterprise and all the ai apps in between. So Michael, when is too early or too late to begin to think about being enterprise ready? It's not just a single point in time where people make this transition. It occurs at many steps of the business. Enterprise single sign on like SAML, auth, you usually don't need that until you have users. You're not going to need that when you're getting started.

17:32And we call it an enterprise feature. But I think what you'll find is there's companies when you sell to like a 50 person company, they might want this. They actually, especially if they care about security, they might want that capability in it. So it's more of like SMB features even if they're tech forward. At WorkOS, we provide a ton of other stuff that we give away for free for people earlier in their lifecycle. We just don't charge you for it. So that AuthKit stuff I mentioned, that identity service, we give that away for free up to a million users, one million users. And this competes with Auth0 and other platforms that have much, much lower free plans.

18:06I'm talking like 10 ,000, 50 ,000, like we give you a million free because we really want to give developers the best tools and capabilities to build their products faster, you know, and to go to market much, much faster. And where we charge people money for the service is on these enterprise things. If you end up being successful and grow and scale up market, that's where we monetize. And that's also when you're making money as a business. So we really like to align, you know, our incentives across that. So we have people using AuthKit that are brand new apps, just getting started, companies in Y Combinator, side projects, hackathon things, you know, things that are not necessarily commercial focus, but could be someday.

18:42They're kind of future-proofing their tech stack by using WorkOS. On the other side, we have companies much, much later that are really big who typically don't like us talking about them. They're logos, you know, because they're big, big customers. But they say, hey, we tried to build this stuff or we have some existing technology, but we're sort of unhappy with it. The developer that built it maybe has left. I was talking last week with a company that does over a billion in revenue each year and their skim connection, the user provisioning was written last summer by an intern who's no longer obviously at the company and the thing doesn't really work.

19:14And so they're looking for a solution for that. So there's a really wide spectrum. We'll serve companies that are in a, you know, their offices in a coffee shop or their living room all the way through. They have a, you know, their own building in downtown San Francisco or New York or something. And it's the same platform, same technology, same tools on both sides. The volume is obviously different. And sometimes the way we support them from a kind of customer support perspective is a little bit different. Their needs are different, but same technology, same platform, just like AWS, right? You can use AWS and pay them$10 a month.

19:41You can also pay them$10 million a month, same product. Or more, for sure. Or more. Well, no matter where you're at on your enterprise ready journey, WorkOS has a solution for you. They're trusted by Perplexity, Copy.ai, Loom, Vercel, Indeed, and so many more. You can learn more and check them out at workos.com. That's w-o-r-k-o-s.com. Again, workos.com.

20:23I'm guessing that you all, you know, the fact that you're supporting all of these use cases on top of Argyla on the data side makes me think, like you say, there's so many things to optimize in terms of that rag process. But there's also so many AI workflows that are being thought of, whether that be code generation or assistance or content generation, information extraction. But then you kind of go beyond that. David, you mentioned text classification and of course there's image use cases. So I'm wondering from you all, at this point, one of the things Chris and I have talked about on the show a bit is we're still big proponents and believe that in enterprises, a lot of times there is a lot of mixing of rule-based systems and more kind of traditional, I guess if you want to think about it that way, machine learning and smaller models, and then bringing in these larger Gen AI models as kind of orchestrators or query layer things.

21:29And that's a story we've been kind of telling, But I think it's interesting that we have both of you here in the sense that like you really, I'm sure there's certain things that you don't or can't track about what you're doing. But just even anecdotally, out of the users that you're supporting on Argeala, what have you seen in terms of what is the mix between those using Argeala for this sort of maybe what people would consider traditional data science type of models like text classification or image classification type of things and these maybe newer workflows like RAG and other things? How do you see that balance?

22:09And do you see people using both or one or the other? Yeah, any insights there? I think we recently had this company from Germany, Elamind, over at one of our meetups that we host. And they had an interesting use case where they collaborated with this healthcare insurance platform in Germany. And one of the things that you see with large language models is that these large language models can't really produce German language properly. They're mostly trained on English text. And that was also one of their issues. And what they did was actually a huge classification and generation pipeline, combining a lot of these techniques where they would initially get an email in that they would classify to a certain category.

22:57Then based on the category, they would kind of define what kind of email template, what kind of prompt template they would use. Then based on the prompt template, they would kind of start generating and composing one of these response emails that you would expect for like a customer query request coming in for the healthcare insurance companies. And then in order to actually ensure that the formatting and phrasing and the German language was applied properly, they would then, based on that prompt, regenerate the email once more. So prompt an LLM to kind of improve the quality of the initial proposed output.

23:32And then after all of these different steps of classification, of retrieval, augmented generation, of an initial generation and a regeneration, they would then end up with their eventual output. So what we see is that all of these techniques are normally combined. And also a thing that we are strong believers in is that whenever there is a smaller model or an easier approach applicable, why not go for that instead of using one of these super big large language models so if you can just classify is this relevant or is this not relevant and based on that actually decide what to do that makes a lot of sense and also one of the interesting things that i've seen one of these open source platforms haystack out there using is also this query classification pipeline where they would classify incoming queries as either a key terminology search, a question query, or actually a phrase for an LLM to actually start prompting an LLM.

24:36And based on that, actually redirect all of their queries to the correct model. And that's also an interesting approach that we've seen. Quick follow-up on that. And it's just something I wanted to draw out because we've drawn it out across some other episodes a bit. You were just making a recommendation, kind of go for the smaller model versus the larger model. Could you, for people trying to follow and there's the divergent mindsets, could you take just a second and say why you would advocate for that, what the benefit, what the virtue is in the context of everything else? I would say smaller models are generally hostable by yourself, so it's more private.

25:15Smaller models, they are more cost-efficient. Smaller models can also be fine-tuned easier to your specific use case. So even what we see a lot of people coming to us about is actually fine-tuning LLMs. But even the big companies out there with huge amounts of money and resources and dedicated research teams still have difficulties on fine-tuning LLMs. So whenever you, instead of within your retrieval augmented generation pipeline, fine-tune like the NLM for the generation part, you can actually choose to fine-tune one of these retrieval models that you can actually fine-tune on consumer-grade hardware.

25:54You can actually fine-tune it very easily on any arbitrary data scientist developer device. And then instead of having to deploy anything on one of the cloud providers, you can start with that. And in a similar reasoning for a RAC pipeline, whenever you provide an LLM with garbage within such a retrieved augmented generation pipeline, you actually also ensure that there's less relevant content and the output of the LLM is also going to be worse. Yeah, I've seen a lot of cases where I think it was Travis Fisher who was on the show. He advocated for this hierarchy of how you should approach these problems.

26:33And there's like, you know, maybe seven things on his hierarchy that you should try before fine tuning. And I think in a lot of cases I've seen people maybe jump to that. They're like, oh, I forget which one of you said this, but, you know, this naive rag approach didn't get me quite there. So now I need to fine tune when in reality, there's sort of a huge number of things in between those two places. And you might end up just getting a worse performing model, depending on how you go about the fine tune. One of the things, David, you kind of walk through these different the example of the specific company that had these workflows that involve a variety of different operations, which I assume, you know, to Ben, you mentioning earlier, starting with a test set.

27:20and that sort of thing and how to think about the tasks. I'm wondering if you can specifically now talk just a little bit about Arjila specifically, people might, might be familiar generally with like data annotation. They might be familiar, you know, maybe even with how to upload some data to quote, fine tune some of these models in an API sense, or maybe even in a more advanced way with Q Laura or, or something like that. But could you take a minute and just talk through kind of Arjila's approach to data annotation and data collaboration? And like, it's kind of hard on a podcast because we don't have a visual to show for people.