Collaboration & evaluation for LLM apps

0:05Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io.

0:42Welcome to another episode of Practical AI. This is Daniel Whitenack. I am CEO and founder at PredictionGuard and really excited today to be joined by Dr. Reza Habib, who is CEO and co-founder at Humanloop. How are you doing, Reza? Hi, Daniel. It's a pleasure to be here. I'm doing very well. Yeah, thanks for having me on. Yeah, yeah. It's super excited to talk with you. I'm mainly excited to talk with you selfishly because I see the amazing things that HumanLoop is doing and the really critical problems that you're thinking about. And every day of my life, it's like, how am I managing prompts?

1:23And how does this next model that I'm upgrading to, how do my prompts do in that model? And how am I constructing workflows around using LLMs? Which it definitely seems to be the main thrust of some of the things that you're thinking about at HumanLoop. Before we get into the specifics of those things at HumanLoop, would you mind setting the context for us in terms of workflows around these LLMs, collaboration on team? How did you start thinking about this problem? And what does that mean in reality for those working in industry right now, maybe more generally than at HumanLoop? Yeah, absolutely.

2:04So I guess on the question of how I came to be working on this problem, it was really something that my co-founders, Peter and Jordan, I had been working on for a very long time, actually. So previously, Peter and I did PhDs together around this area. And then when we started the company, it was a little while after Transfer Learning had started to work in NLP for the first time. And we were mostly helping companies fine-tune smaller models. But then sometime midway through 2022, we became absolutely convinced that the rate of progress for these larger models was so high, it was going to start to eclipse essentially everything else in terms of performance, but more importantly, in terms of usability.

2:40It was the first time that instead of having to hand annotate a new data set for every new problem, there was this new way of customizing AI models, which was that you could write instructions in natural language and have a reasonable expectation that the model would then do that thing. And that was unthinkable at the start of 2022, I would say, or maybe a little bit earlier. And so that's really what made us want to go work on this because we realized that the potential impact of NLP was already there, but the accessibility had been expanded so far and the capabilities of the models had increased so much that there was a particular moment to go do this.

3:15But at the same time, it introduces a whole bunch of new challenges, right? So I guess historically, the people who are building AI systems were machine learning experts. the way that you would do it is you would collect annotated data, you'd fine tune a custom model. It was typically being used for like one specific task at a time. There was a correct answer, so it was easy to evaluate. And with LLMs, the power also brings new challenges. So the way that you customize these models is by writing these natural language instructions, which are prompts. And typically that means that the people involved don't need to be as technical.

3:48And usually we see actually that the best people to do prompt engineering tend to have domain expertise. So often it's a product manager or someone else within a company who's leading the prompt engineering efforts. But you also have this new artifact lying around, which is the prompt, and it has a similar impact to code on your end application. So it needs to be versioned and managed and treated with the same level of respect and rigor that you would treat normal code. But somehow you also need to have the right workflows and collaboration that lets the non-technical people work with the engineers on the product or the less technical people.

4:21And then the extra challenge that comes with it as well is that it's very subjective to measure performance here. So in traditional code, we're used to running unit tests, integration tests, regression tests. We know what good looks like and how to measure it. And even in traditional machine learning, there's a ground truth data set, people calculate metrics. But once you go into generative AI, it tends to be harder to say what is the correct answer. And so when that becomes difficult, then measuring performance becomes hard. If measuring performance is hard, how do you know when you make changes if you're going to cause regressions or, you know, all the different design choices you have in developing an app?

4:58How do you make those design choices if you don't have good metrics of performance? And so those are the problems that motivated what we've built. And really, HumanLoop exists to solve both of these problems. So to help companies with the tasks of finding the best prompts, managing, versioning them, dealing with collaboration, but then also helping you do the evaluation that's needed to have confidence that the models are going to behave as you expect in production. And as related to these things, maybe you can start with one that you would like to start with and go to the others. But in terms of managing versioning prompts, evaluating the performance of these models, dealing with regressions.

5:38As you've kind of seen people try to do this across probably a lot of different clients, a lot of different industries, how are people trying to manage this in maybe some good ways and some bad ways? Yeah, I think we see a lot of companies go on a bit of a journey. So early on, people are excited about generative AI and LLM. There's a lot of hype around it now. So some people in the company just go try things out. And often they'll start off using one of the large publicly available models, so OpenAI or Anthropic, Coher, one of these. They'll prototype in their own kind of playground environment that those providers have.

6:15They'll eyeball a few examples. Maybe they'll grab a couple of libraries that support orchestration and they'll put together a prototype. And the first version is fairly easy to build. It's very quick to get to the first wow moment. And then as people start moving towards production and they start iterating from that, you know, maybe 80 % good enough version to something that they really trust, they start to run into these problems of like, oh, I've got like 20 different versions of this prompt and I'm storing it as a string in code. And actually, I want to be able to collaborate with a colleague on this.

6:46And so now we're sharing things, you know, either via screen sharing or we're like both, you know, we've had some serious companies you would have heard of who are sending their model configs to each other via Microsoft of Teams. And obviously, you wouldn't send someone an important piece of code through Slack or Teams or something like this, but because the collaboration software isn't there to bridge this technical, non-technical divide, those are the kinds of problems we see. And so at this point, typically a year ago, people would start building their own solution. So more often than not, this was when people would start building in-house tools.

7:18Increasingly, because there are companies like Humanloop around, that's usually when someone books a demo with us and they say, hey, we've reached this point where actually managing these artifacts has become cumbersome. We're worried about the quality of what we're producing. Do you have a solution to help? And the way that Humanloop helps, at least on the prompt management side, is we have this interactive environment. It's a little bit like those OpenAI playgrounds or the Anthropic playground, but a lot more fully featured and designed for actual development. So it's collaborative. It has history built in.

7:48You can connect variables and data sets. And so it becomes like a development environment for your sort of LLM application. You can prototype the application, interact with it, try out a few things, and then people progress from that development environment into production through evaluation and monitoring. You mentioned this kind of in passing. I'd love to dig into it a little bit more. You mentioned kind of the types of people that are coming, you know, at the table in designing these systems. And oftentimes domain experts, you know, previously in working as a data scientist, it was always kind of assumed, oh, you need to talk to the domain experts.

8:26But it's sort of like, at least for many years, it was like data scientists talk to the domain experts and then go off and build their thing. the domain experts were not involved in the sort of building of the system. And even then, like the data scientists were maybe building things that were kind of foreign to software engineers. And what I'm hearing you say is you kind of got like these multiple layers. You have like domain experts who might not be that technical. You've got maybe AI and data people who are using this kind of unique set of tools. Maybe even they're hosting their own models.

9:02And then you've got like product software engineering people. It seems like a much more complicated landscape of interactions. How have you seen this kind of play out in reality in terms of non-technical people and technical people both working together on something that is ultimately something implemented in code and run as an application? I actually think one of the most exciting things about LLMs and the progress in AI in general is that product managers and subject matter experts can, for the first time, be very directly involved in implementing these applications. So I think it's always been the case that the PM or someone like that is the person who distills the problem, speaks to the customers, produces the spec, but there's this translation step where they sort of produce that PRD document and then someone else goes off and implements it.

9:53And because we're now able to program at least some of the application in natural language, actually it's accessible to those people very directly. And it's worth maybe having a concrete example. So like I use an AI note taker for a lot of my sales calls and it records the call and then I get a summary afterwards. And the app actually allows you to choose a lot of different types of summaries. So you can say, hey, I'm a salesperson. I want a summary that will extract budget and authority and need and timeline versus you can say, oh, actually I had a product interview and I want a different type of summary.

10:24And if you think about developing that application, the person who has the knowledge that's needed to say what a good summary is and to write the prompt for the model is the person who has that domain expertise. It's not the software engineer. But obviously the prompt is only one piece of the application, right? If you've got a question answering system, there's usually retrieval as part of this. There may be other components. Usually the LLM is a block in a wider application. So you obviously still need the software engineers around because they're implementing the bulk of the application, but the product managers can be much more directly involved.

10:58And then, you know, actually we see increasingly less involvement from machine learning or AI experts and less people are fine-tuning their own models. So for the majority of product teams we're seeing, there is an AI platform team that maybe facilitates setting things up, but the bulk of the work is led by the product managers and then the engineers. And one interesting example of this on the extreme end is one of our customers that's a very large ed tech company, they actually do not let their engineers edit the prompts. So they have a team of linguists who do prompt development. The linguists finalize the prompts.

11:33They're saved in a serialized format and they go to production, but it's a one-way transfer. So the engineers can't edit them because they're not considered able to assess the actual outputs, even though they are responsible for the rest of the application. Just thinking about how teams interact and who's doing what, it seems like the problems that you've laid out are, I think, very clear and worth solving, but it's probably hard to think about, well, am I building a developer tool or am I building something that these non-technical people interact with? Or is it both? How did you think about that as you entered into the stages of bringing human loop into existence?

12:14I think it has to be both. And the honest answer is it evolved kind of organically by going to customers, speaking to them about their problems and trying to figure out what the best version of a solution looked like. So we didn't set out to build a tool that needed to do both of these things. But I think the reality is, given the problems that people face, you do need both. And an analogy to think about might be something like Figma, right? Like Figma is somewhere where multiple different stakeholders come together to iterate on things and to develop them and provide feedback. And I think you need something analogous to that for Gen AI, although it's not an exact analogy because we also need to attach the evaluation to this.

12:51So it's almost by necessity that we've had to do that. But I also think that it's very exciting, right? And the reason I think it's exciting is because it is expanding who can be involved in developing these applications.

13:21If you're listening, you know software is built from thousands of small technical choices, and some of these seemingly inconsequential choices can have a profound impact on the economics of internet services, who gets to participate in them, build them, and profit from them. This is especially true for artificial intelligence, where the decisions we make today can determine who can have access to world-changing technologies and who can decide their future. Read, Write, Own, Building the Next Era of the Internet is a new book from startup investor Chris Dixon that explores the decisions that took us from open networks governed by communities to massive social networks run by Internet giants.

14:00This book, Read, Write, Own, is a call to action for building a new era of the Internet that puts people in charge. From AI projects that compensate creators for their work to protocols that fund open source contributions, this is our chance to build the internet we want, not the one we inherited. Order a copy of Read, Write, Own today or go to readwriteown.com to learn more.

14:39You mentioned how this environment of domain experts coming together and technical teams coming together in a collaborative environment opens up new possibilities for both collaboration and innovation. I'm wondering if at this point you could kind of just lay out, we've talked about the problems, we've talked about those involved and those kind of that would use such a system or a platform to enable these kind of workflows. Could you describe a little bit more what human loop is specifically in terms of both what it can do and kind of how these different personas engage with the system? Yeah, so I guess in terms of what it can do, concretely, it's firstly helping you with prompt iteration, versioning and management, and then with evaluation and monitoring.

15:29And the way it does that is there's a web app and there's a web UI where people are coming in. And in that UI is an interactive playground-like environment where people basically try out different prompts. They can compare them side by side with different models. They can try them with different inputs. When they find versions that they think are good, they save them. And then those can be deployed from that environment to production or even to a development or staging environment. So that's the kind of development stage. And then once you have something that's developed, what's very typical is people then want to put in evaluation steps into place.

16:03So you can define gold standard test sets, and then you can define evaluators within human loop. And evaluators are ways of scoring the outputs of a model or a sequence of models, because oftentimes, you know, the LLM is part of a wider application. And so the way that scoring works is there's very traditional metrics that you would have in code for any machine learning system. So precision, recall, rouge, blue, these kind of scores that anyone from a machine learning background would already be familiar with. But what's new in the kind of LLM space is also things that help when things are more subjective.

16:36So we have the ability to do model as judge, where you might actually prompt another LLM to score the output in some way. And this can be particularly useful when you're trying to measure things like hallucination. Right. So a very common thing to do is to ask the model, you know, is the final answer contained within the retrieved context or is it possible to infer the answer from the retrieved context? And you can calculate those scores. And then the final way is we also support human evaluation. So in some cases, you know, you really do want either feedback from an end user or from an internal annotator involved as well.

17:11And so we allow you to gather that feedback either from your live production application and have it logged against your data, or you can queue internal annotation tasks from a team. And I can maybe tell you a little bit more about sort of in-production feedback because that's actually where we started. Yeah, yeah, go ahead. I would love to hear more. Yeah, so I think that because it's so subjective for a lot of the applications that people are building, whether it be email generation, question answering, a language learning app, there isn't a correct answer, quote unquote. And so people want to measure how things are actually performing with their end users.

17:46And so HumanLoop makes it very easy to capture different sources of end user feedback. And that might be explicit feedback, things like thumbs up, thumbs down votes that you see in chat GPT, but it can also be more implicit signals. So how did the user behave after they were shown some generated content? Did they progress to the next of the application? Did they send the generated email? Did they edit the text? And all of that feedback data becomes useful both for debugging and also for fine-tuning the model later on. So that evaluation data becomes this rich resource that allows you to continuously improve your application over time.

18:23Yeah, that's awesome. And I know that that fits in. So maybe you could talk a little bit about how you're... One of the things that you mentioned earlier is you're seeing fewer people do fine tuning, which I see this very commonly as a, it's not an irrelevant point, but it's maybe a misconception where like a lot of teams come into this space and they just assume they're going to be fine tuning their models. And often what they end up doing is fine tuning their workflows or their language model chains or their retrieval, the data that they're retrieving or their prompt formats or that templates or that sort of thing, they're not really fine tuning.

19:05And I think there's this really blurred line right now for many teams that are adopting AI into their organization where they'll frequently just use the term, oh, I'm training the AI to do this and now it's better, right? But all they've really done is just inject some data into their prompts or something like that. So could you maybe help clarify that distinction and also in reality what you're seeing people do with this capability of evaluation both online and offline and how that's filtering back into upgrades to the system or actual fine tunes of models? Yeah, so I guess you're right. There's a lot of jargon involved and especially for people who are new to the field, the word fine-tuning has a colloquial meaning, and then it has a technical meaning in machine learning, and the two end up being blurred.

19:57So, you know, fine-tuning in a machine learning context usually means doing some extra training on the base model, where you're actually changing the weights of the model, given some sets of example pairs of inputs, outputs that you want. And then obviously there's like prompt engineering and maybe context engineering, where you're changing the instructions to the language model or you're changing the data that's set into the context or how an agent system might be set up. And both are really important. Typically, the advice we give the majority of our customers and what we see play out in practice is that people should first push the limits of prompt engineering because it's very fast, it's easy to do, and it can have very high impact, especially around changing the outputs and also in helping the model have the right data that's needed to answer the question.

20:48So prompt engineering is kind of usually where most people start and sometimes where people finish as well. And fine tuning tends to be useful either if people are trying to improve latency or cost or if they have like a particular tone of voice or output constraint that they want to enforce. So, you know, if people want their model to output valid JSON, then fine tuning might be a great way to achieve that. Or if they want to use a local private model because it needs to run on an edge device or something like this, then fine-tuning, I think, is a great candidate. And it can also let you reduce costs because oftentimes you can fine-tune a smaller model to get similar performance.

21:24The analogy I like to use is fine-tuning is a bit like compilation. You've already built your first version of the language. When you want to optimize it, you might use a compiled language and you've got a compiled binary. I think there was a second part to your question, but just remind me, actually, I've lost the second part. Yeah, basically, you mentioned that maybe fewer people are doing fine tunes. Maybe you could comment on, I don't know if you have a sense of why that is, or how you would see that sort of progressing into this year as more and more people adopt this technology and maybe get better tooling around the, let's not call it fine tuning, so we don't mix all the jargon, but the iterative development of these systems.

22:10Do you see that trend continuing? Or how do you see that kind of going into maybe larger or wider adoption in 2024? Yeah, so I think that we've definitely seen less fine tuning than we thought we would see when we started, you know, when we launched Humanloop back, this version of Humanloop back in 2022. And I think that's been true of others as well. Like I've spoken to friends at OpenAI, and OpenAI is expecting there will be more fine tuning in the future, but they've been surprised that there wasn't more initially. I think some of that is because prompt engineering has turned out to be remarkably powerful.

22:46And also because some of the changes that people want to do to these models are more about getting factual context into the model. So one of the downsides of LLMs today is they're obviously trained on the public internet. So they don't necessarily know private information about your company. They tend not to know information past the training date of the model. And one way you might have thought you could overcome that is I'm going to fine tune the model on my company's data. But I think in practice, what people are finding is a better solution to that is to use a hybrid system of search or information retrieval plus generation.

23:20So what's come to be known as like RAG or retrieval augmented generation has turned out to be a really good solution to this problem. And so the main reasons to fine tune now are more about optimizing cost and latency and maybe a little bit tone of voice, but they're not needed so much to adapt the model to a specific use case. And fine tuning is a heavier duty operation because it takes longer. You can edit a prompt very quickly and then see what the impact is. Fine tuning, you need to have the data set that you want to fine tune on, and then you need to run a training job and then evaluate that job afterwards.

23:54So there are certainly circumstances where it's going to make sense. I think especially anyone who wants to use a private open source model will likely find themselves wanting to do more fine-tuning. But the quality of prompt engineering and the distance you can go with it, took a lot of people by surprise. On that note, you mentioned the closed proprietary model ecosystem versus open models that people might host in their own environment and or fine tune on their own data. I know that Humanloop, you explicitly say that you have all of the models, You're integrating these sort of closed models and integrate with open models.

24:33Why and how is that kind of decided to kind of include all of those? And in terms of the mix of what you're seeing with people's implementations, how do you see this sort of proliferation of open models impacting the workflows that you're supporting in the future? So the reason for supporting them, again, is largely customer pull, right? What we were finding is that many of our customers were using a mixture of models for different use cases, either because the large proprietary ones had slightly different performance tradeoffs or because there were use cases where they cared about privacy or they cared about latency and so they couldn't use a public model for those instances.