Only as good as the data

0:05Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io.

0:47What's up, friends? I'm here with a new friend of ours over at Assembly AI, founder and CEO Dylan Fox. Assembly AI is where you can turn voice data into insights, chapters, transcripts, summaries, and so much more with their leading speech AI models. So Dylan, give me a glimpse into what you're doing with speech AI models at Assembly AI. So at Assembly, we're building industry leading speech AI models for various tasks like speech to text, streaming speech to text, speech understanding to help developers easily convert voice data, whether it's live or pre-recorded into super accurate text. And then to help developers extract a ton of information and metadata around voice data or even around the text that they just were able to convert from that audio data.

1:34So these are things like picking out entities or PII that was spoken in voice files or summarizing voice and audio data down into custom summaries. It's things like being able to detect how many speakers spoke and who said what and what the names of different speakers were. So we bundle all those things into a super simple API with really great docs that developers can just sign up to for free to start, use the API, build into their apps, and then build these really cool AI apps and products and workflows and automations on top of voice data with. I dig it. Okay. Can you take me a little deeper into the opportunity for developers?

2:13Because it seems like there's a lot of voice data out there and there's a lot of trapped value in that voice data. There's so much voice data being created on the internet now. Podcasts, videos, phone calls, voice messages, audiobooks, virtual meetings. It's crazy. And you can now transform and understand all this voice and audio data in ways that were not even possible a year or 18 months ago. And so what we're seeing with the help of these new AI models that we're creating at Assembly, developers and organizations are just racing to build all these new applications, workflows, automations that leverage the voice data they have either within their organization or within their product to build really cool new products and services and workflows that are just like taking off in the market.

3:00And so at Assembly, we're building the industry leading models for all those different apps and workflows, whether it's speech to text or speaker diarization or speech understanding capabilities to summarize voice data or extract entities from voice data or mask PII from phone calls for various types of automations that might be built. And we're exposing that through a super simple, super scalable API that's just constantly being updated and constantly getting better. And so we're seeing just a crazy amount of developers and companies just build really cool apps and services on top of our API every day.

3:33It's really only just getting started, especially with the model updates that we have planned over the second half of the year that are coming out. They're really excited to launch to the developers on our API. Okay, constantly updated speech AI models at your fingertips. Well, at your API fingertips, that is. A good next step is to go to their playground. You can test out their models for free right there in the browser. Or you can get started with a$50 credit at assemblyai.com. Again, assemblyai.com.

4:07Welcome to another fully connected episode of the Practical AI Podcast. This is Daniel Whitenack. I am the founder and CEO of Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. In these fully connected episodes, Chris and I will keep you fully connected with some of the things happening in the AI news and trends, and we'll also discuss a few topics that will help you level up your machine learning game. How are you doing, Chris? Doing good. How are you today, Daniel? I think you're traveling, aren't you? I am in transit, so hopefully the hotel Wi-Fi slash hotspot holds out and we can keep it going.

4:54But yeah, going over to GopherCon UK, which should be fun to talk to a few Go programmers about AI and integrating it into Go applications. So that should be fun. Fantastic. Sounds good. That's incidentally for people who may have just joined the show. That's actually how Daniel and I originally met way back was through the Go programming community. So this is a throwback there. Yeah, yeah. It should be fun. This is actually my first GopherCon UK. So that'll be good. Well, Chris, one of the things that had occurred to me maybe, I don't know, last week or this week or sometime was seeing a lot of people kind of on general AI threads or on social media talking about how, you know, AI is only as good as the data that's fed into it or, you know, AI, doing AI in the enterprise or in a real world environment is all about data.

5:54It's not about the models or, you know, some type of comments like that on social media. Have you seen these? I have. I have. And I've actually been glad to see that versus all the hype of some of the other topics that we've been dealing with over recent times. So let's get into some data conversation. Yeah, yeah. Basically, my thought was, well, what, I guess, one, what do people even mean when they say something like that? And then second, I think from a practical kind of boots on the ground standpoint, if you're doing data science, AI, machine learning stuff, there's probably a huge number of types and kind of categories of data that you might run across or have a chance to be exposed to.

6:44And so I thought it may also be good to kind of break down and categorize some of those things to give people a little bit of a landscape of types of data or things that they might run across in the AI space or things that they might even have to curate in their own company. So yeah, that's kind of what I was thinking. I guess in that first point, what do you think people mean when they refer to this, you know, AI is only as good as the data you bring to it, or it's all about data? What are people trying to get out there, do you think? Well, I think it is the constraint around and limitation to the models that you're trying to build.

7:24So, you know, when we build AI models, they are self-training. We're not teaching it what to do. And so you're presenting the data that you want to build the model out of. And the model is only as good as what the data that it's going to be able to train on is. And so the quality of the data and the robustness of the data is absolutely crucial. And it's funny, over the last few years, there's been so much hype. We've talked about generative AI and stuff. Folks tend to get caught up in the hype and they tend to think of kind of the AI being on its own. And I think that's today's topic. That's one of the things we've been wanting to bring people around to is there's been a certain amount of disappointment and misunderstanding.

8:09And at the end of the day, your model is only as good as the data that you're bringing so that it can train on. And so it's a moment to get back to basics, maybe leave some hype behind and recognize that if you don't get this part of it right, you're not going to have a very good outcome. Yeah. So you mentioned a few things there I'd love to pick apart, which is this idea of there's some kind of provenance to a model that has to do with the data. So it may be good to remind people that a model, when we're talking about an AI model, is really composed of two things. It's composed of code that executes functions and adds things together and kind of essentially does a data transformation.

8:56So maybe it's an image in and a label out that's a label, whether it has a cat in the image or not. Or maybe it's text in and a generated next token out. And these are data transformations. And that code that executes those data transformations is, you know, written in code, just like normal code. But it includes a bunch of parameters that need to be set. And by a bunch, you know, maybe people are familiar from seeing models now, but that might be, you know, 7 billion parameters, 70 billion parameters, 400 billion parameters. So in order to set those parameters to do that data transformation, there needs to be data that is used to fit those parameters, often called that training process.

9:47Now, one element of this, Chris, is if you imagine like LAMA 3.1, which is a recent addition to our world, has whatever, 400 billion parameters, you could imagine that maybe you're not going to fit that many parameters with a small amount of data. And so there's some relation between the complexity of the model and how much data is needed to fit it. And that may in itself be something that people aren't quite grasping often is that the bigger the model you want to use, the more data you need to have to train it, which is why these data sets have got larger and larger. And I think that's important to call that out.

10:27And that as people are getting into the idea of training their models, there's a certain amount of understanding what's realistic for you and your capabilities and your organization's capabilities to do up front. And I think that's why there's a set of concerns about how you're going to enter into the process to begin with, which I think you're covering here. But I don't think that's very clear for a lot of people. I think they, you know, when they use foundation models, when they're going to go create their own and stuff like that. And I think the data you have available and the quality of the data and the amount of data, to your point about complex models, is really crucial to consider up front.

11:09If someone is interested in taking their organization forward, how do you start thinking about this, Daniel? How do you frame the whole issue of what data you have and what you can do with that data? I think this is something we've highlighted on the show before, but people sort of have this perception that, oh, we've got a bunch of documents in a file store. We've got a big database. We should be able to do AI or do machine learning with that data. Right. And the situation is definitely more complex than that. So I would say that there's really two things maybe that people need to kind of have in their mind.

11:50One is the type of task that you're wanting to do, which maybe is also related to the type of model that you'll use. And also, what is the state of the structure of the data that you have? So let's give an example. So let's say that you want to do object detection, which is the task of taking in an image and detecting what objects are in that image. So that's a computer vision task. You usually require some type of convolutional neural network. And some of this you could kind of search through and find the type of task that you are trying to do and then kind of the typical model that is used to do that.

12:35And you might find, oh, these typical models that are used for object detection usually need thousands and maybe millions of images to train on. So that may trigger in your mind, well, first of all, do I have enough imagery to train that model? If you do have enough data to train the type of model that you're interested in, then maybe you do that. But oftentimes what people need to do is fine tune a model, not train a model from scratch. So that would be taking a model that already exists, maybe is posted on Hugging Face in a repo that is already trained for some type of task related to what you're doing.

13:16So maybe a similar object detection task. And then you kind of continue the training on from that point with your small amount of data. Now, the second piece to that is, like I mentioned, how structured or unstructured your data is. So if I just have a bunch of images in a file store, that really doesn't do me any good for that object detection task because they're not pre-labeled with labels that I would be able to use to further train one of those models. So another relevant thing here is, is your data unstructured or unlabeled or structured or labeled? And in the case of training a supervised type model, which is a model that requires some labels to be trained, like an object detection model or sentiment analysis or machine translation maybe, then you need those labels in order to further train your model.

14:16So just to kind of recap what I just said there, there was the element of determining what task and what kind of model is needed and how that maps to your data. And then whether you have enough data or not enough data for pre-training or just fine tuning. And then finally, if you have labeled or unlabeled data or structured or unstructured data. Let me ask you a question here. To your point about computer vision, in my own experience across a couple of different companies and computer vision projects, I've used YOLO, which is one of the very common convolutional models that's out there. And doing that, we've had to go through labeling process, but you're using YOLO as a foundational model that you're building upon.

15:04Is that in your thinking about that, if you have a set of maybe a few thousand images that you're using on YOLO, is there ever a good reason to potentially say, well, maybe I don't want to use a foundational model, even though it requires more data to train? Would I ever want to go create a new computer vision model to work with? Is there ever, how do you think about that? Because that's come up in topics a number of times as projects got started and stuff. How do you think about when to go use somebody else's foundation model versus maybe trying to do something like that on your own? Yeah, I think that it kind of comes into this element of how big of a model is needed and how complicated of a problem you're trying to solve.

15:52So certainly for something like object detection, especially if you have a bunch of labels that you're trying to detect in your imagery, and generally that task is fairly complicated task, I guess, in terms of even how we would think about doing that data transformation, then it likely needs a more complicated model, which means more data to train that model. And in most of those situations, whether you're talking about object detection or machine translation or speech synthesis or speech transcription, these sorts of tasks, most of the time companies would be much better off doing a fine tuning and not a training from scratch, either because they don't have enough data internally to train a model from scratch, or they don't have it labeled appropriately, or maybe they just don't have a big enough compute cluster to do that training.

16:44And so they could benefit then from a pre-trained model that would do fine tuning on that. But maybe other tasks like a sentiment analysis or a forecasting problem where you're forecasting a time series, it's not that you couldn't do that in the fine tuning approach, but that it may only take five seconds or five minutes to train that sort of model. and on a small amount of data, like thousands of samples, not millions of samples. And that could achieve your goal very well and be a small model that you could run performantly. So in those cases, of course, you would be training a model from scratch.

17:28This is a Changelog Newsbreak. The Phylum Research Team writes, quote, like the island of discarded plastic twice the size of Texas floating in the North Pacific Ocean. NPM has accrued an astonishing amount of spam packages over the past six months. Our 95 % confidence interval for the estimate of T, that's T-E-A, the protocol spam in new packages over the past six months jumped to between 68.66 % and 74.67%, or somewhere between 613 ,000 and 667 ,000 packages. In other words, among all new packages published to NPM in the past six months, about five out of every seven packages are T-spam, end quote.

18:17I first covered the unintended consequence of the T-Protocol's crypto rewards back in February, that was episode 83, it appears the damage is even worse than previously discovered. What a mess. That is one big pile. You just heard one of our five top stories from Monday's Changelog News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com slash news to You also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com slash news.

19:01All right. Well, Chris, we've talked a little bit about training, pre-training. So there's this first category of data, which is training data or pre-training data, sometimes might be called. And this would be the data that you're taking a model where the parameters have not been fit, an untrained model, and you are doing the first fitting or training of those parameters with this training or pre-training data. Now, along with that training or pre-training data, of course, you may have tests or holdout or evaluation data. This is a second category of data that you might have, which may just be a holdout from that training set.

19:45It might be a public kind of benchmark type of test set for a particular task like machine translation or something like that. or maybe it's data that you're going to use and have humans review or something like that. But anyway, it's a test set. It's an evaluation set. It's held out from that training or pre-training. So you have a volume of data and you're maybe taking like arbitrarily 20 % of that data and setting it aside. Is that what we're getting at? Yeah, yeah. And of course, if you look up like how much you should hold out or how you should construct test sets or evaluation sets, that's a very complicated rabbit hole that you could go down.

20:29But I would say at the most simple level, yes, you can take whatever your training set is and hold some out. And oftentimes you would do that randomly so that, you know, there's no kind of, if your data is stratified, meaning it has some structure to it in terms of what comes first and last, then you could randomize that and get a little bit better sample. And that'll then allow you to train your model or fit your model and then make predictions on that test or evaluation set, calculate a metric. Maybe that could be accuracy or F1 score, or in the case of machine translation, blue or comet, or in the case of time series forecasting, some mean squared error, mean absolute error type of thing.

21:16And that then allows you to gauge, well, am I doing better than random? Do I have any predictive power to my model, I guess? And it's relative to that training data set. So specifically, so that's assuming that the model is accurate against the training set that you had there or other data that you may introduce later that is very consistent with what you would see in the training set. Is that accurate? Yeah, exactly. You want to hold out enough to where you have confidence that when your model sees new samples that you would likely see in a kind of production scenario, then you're able to make predictions on those new samples and get ideally some type of predictive power, a result that is useful for the task that you've trained on.

22:04And there's public benchmark data for a lot of different tasks as well, if people are looking for that. People might be familiar if you just search like open LLM benchmarks or leaderboard. There's a bunch of leaderboards for LLMs, but there's also public. You may want to search like shared task data. data. Often this benchmark or evaluation data comes out of peer-reviewed workshop type of scenarios. So if people aren't aware, there's these research conferences in the AI world and research conferences are the primary way that people publish academic AI research. And at these academic AI research conferences, there's sometimes things called workshops.

22:52And this isn't like, I mean, there's learning that goes on, but it's not like a learning workshop like you go to at an industry conference. It's a workshop to work on specific problems related to research problems related to a topic and then share results together around usually a common shared task. So there might be a workshop for computer vision related tasks or machine translation related tasks. There's one called WMT, which always has a shared task around machine translation. And there's many other types of shared tasks that publish peer-reviewed benchmark evaluation data. And how would you, just going for a moment back to the previous thought process around training data and test data, how does benchmark data sets, how do they fit in to your notion of training and test?

23:44Are you using them afterwards? How does that fit into your process? Yeah. So if you are doing a task that is the same or very similar to an existing benchmark that's out there, some or all of the data related to that benchmark may form either test or training data for you when you're doing your fine tuning or evaluation. right so let's say for example you're training a machine translation model from English to Arabic you could go and look at a bunch of different benchmarks related to machine translation and they'll have many many different language pairs including English to Arabic potentially and so you may use a portion of that data for evaluating your model or for adding to your training data set.

24:36However, if you're doing machine translation maybe to a new language pair that isn't represented in any of those academic benchmarks, this would be the scenario where your company's maybe trying to do something that hasn't been represented in the academic research world yet, like a manufacturing company that's wanting to detect defects in certain types of products using a computer vision model. There's likely not a shared task for that, other than the fact that there are many computer vision shared tasks. And so a way to think about it in that scenario is if my company is trying to design this new model to detect defects in chips or in other types of products on a manufacturing line, I could go to a shared task and look at, well, what are the best models that people are using these days for the task of the relevant computer vision tasks, like object detection or something like that?

25:41And so in that case, the benchmark or the shared task data would represent more of a gauge for you or like a starting point to determine maybe what types of models you want to be considering. If I'm doing that, is that fine tuning the model? Is that actually like at this point you're talking about creating a new model rather than using foundation or am I misunderstanding that? Yeah, good question. So really, this is maybe part of the categorization where there's overlap and maybe people tend to get confused. So there's kind of one set of categories which is related to the type of data and the way that you're using that data.

26:23So you're going to be using some data for training your model or pre-training a model, some data for fine-tuning a model or adapting a foundation model, and some data for evaluating or testing or benchmarking a model. So those are the categories of data that you would use in this process. Now, where you source that data could be from public benchmark data. It could be from data that is internal to your company that likely isn't public, or it could be a combination of the two. And so that's kind of, you want to be thinking about both about how you're using the data and for what purpose, and also being creative with where you might get it from, right?

27:13Gotcha. As you're mixing data from your own organization with some of this benchmark data and trying to align that so that you have the benefit of trying to maybe fine tune on a model and use benchmark data to drive that, but you're also trying to introduce your own new capabilities based on data that your company has. how do you get those two sets of data to end up being a you know a high quality data set without a lot of differences between it you know as you're trying to kind of get the best of both worlds getting you know taking advantage of that which is already there but but bringing some new capabilities out that your company uh or your organization can leverage how do you think about merging those two desperate sets of data so that you end up getting a good training set to do some fine tuning with?

28:05Yeah. So I think this really depends on how close the task that you're really trying to accomplish is how close that task is to the public data that's out there. And in certain cases, it may be very close. Like I mentioned, the Arabic translation example, there's language or by text or parallel text data from English to Arabic. There's a lot of that data out there. And if that's a specific task that you're doing, maybe you are using that in your initial training and then fine tuning with your domain specific data on top of that later on. Whereas other cases, you may just treat that public data as a good starting point, or you might even just look at what models are trained on that public benchmark data in order to understand which foundation model you're going to use and fine tune, or which model you could pull off of hugging face to then fine tune because it was high ranking on this benchmark, which was close to the task that you're doing.

29:13and then you can fine tune on top of that. So yeah, remember you're not always, I guess, doing the pre-training step of this. You may only be doing the fine tuning step. And also I would encourage people to think about, I guess, this mix of data within your organization and data outside. So there's a lot of data on repositories like Hugging Face that might be useful to your company if adapted in a very specific way. So let me give a simple example. So there's a benchmark out there called SQUAD, which is a question answering, extractive question answering type of benchmark. And this was used to train models very specifically for question answering.

29:59So not the kind of general purpose large language models that are out there. But you could take the squad data set, which has essentially in the input, it has paragraphs of text and some question that's asked and answered in that paragraph of text, and then paired with the appropriate answer that's extracted out of that. And so it's very possible if you're doing a question answering sort of task, you could test whatever model you're using on that squad output. Even if you're using like an LLM, you could structure that data in a way that you could test the LLM's performance on that benchmark. Or you could take that data and structure it into prompts for fine tuning an LLM.

30:47Even though this data set was made prior to this latest wave of LLMs, it's still relevant and can be used for various purposes related to even these Gen AI models. So there's a lot of stuff out there, and I think it would benefit companies to kind of explore the data sets available in the space that they're thinking about and not just write them off if they're not labeled exactly like they want them labeled because they still may be useful with some strategic post-processing. Just to combine, since you brought up Gen.ai along the way, and obviously that's on people's minds over the last couple of years a lot, we've talked on a number of episodes about how popular RAG is, which is retrieval augmented generation, which lets them take their own data that they have available and use it with a generative model.