Evals, error analysis, and better prompts: A systematic approach to improving your AI products | Hamel Husain (ML engineer)

0:00What are the fundamental concepts folks need to know of getting to higher quality products?

0:04Hamel Husain:The most important thing is looking at data. Looking at data has always been a thing, even before AI. There's just a little bit of a twist on it for AI, but really the same thing applies. When you see a real user input like this, you actually look at what users are prompting your AI with. You realize it's very vague. Absolutely. That's the whole interesting bit. Once you see that people are talking like that, you might actually want to simulate stuff that looks like that. because that's what the real distribution of the data or that's what the real world looks like. I'm sure our listeners expect some like magical system that does this automatically.

0:37And you're like, no, man, just spend three hours of your afternoon, go through, read some of these chats, look at some of them with your human eyes, put one sentence notes on all of them and then run a quick categorization exercise and get to work. And you see this have actual real impact on quality and reducing these errors.

0:55Hamel Husain:Yeah, it has an immense quality. It's so powerful that some of my clients are so happy with just this process that they're like, that's great, Hamill, we're done. And I'm like, no, wait, we can do more.

1:09Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive here on a mission to help you build better with these new tools. Today, I have such an educational episode for people like me that are building AI products. We have Hamill Hussain, who's going to demystify debugging errors in your AI product, writing good evals, and show us how he runs his entire business using Claude and a GitHub repo. Let's get to it. This episode is brought to you by GoFundMe Giving Funds, the zero-fee DAF. I want to tell you about a new product GoFundMe has launched called Giving Funds, a smarter, easier way to give, especially during tax season, which is basically here.

1:52GoFundMe Giving Funds is the DAF, or Donor Advice Fund, from the world's number one giving platform, trusted by 200 million people. It's basically your own mini foundation, without the lawyers or admin costs. You contribute money or appreciated assets, get the tax deduction right away, potentially reduce capital gains, and then decide later where to donate from 1.4 million nonprofits. There are zero admin or asset fees. And while the money sits there, you can invest and grow it tax-free. So you have more to give later. All from one simple hub with one clean tax receipt. Lock in your deduction now and decide where to give later.

2:34Perfect for tax season. Join the GoFundMe community of 200 million and start saving money on your tax bill, all while helping the causes you care about the most. Start your giving fund today in just minutes at gofundme.com slash howiai. We'll even cover the DAF pay fees if you transfer your existing DAF over. That's gofundme.com slash howiai to start your giving fund. Hamill, I'm really excited for this particular episode because I have been building products for a very long time. And this has been one of a few times in my career where the how and what of products that I'm building are so different than what I've built in the past.

3:26They're technically different. They're different from a user experience perspective. And then they have these non-deterministic models on the back end that I'm somehow, as a product leader, responsible for making output high-quality, consistent, reliable, interesting user experiences. And it's such a challenging problem. And what I love about what you're going to show us today is how to approach that systematically, that quality of product building in an AI world systematically, and how you use different techniques to get AI products, which are new to all of us, from good to great.

4:06Hamel Husain:Yeah, I'm happy to be here. Excited to talk about it. So, you know, this is such a new thing for product managers. I'm curious if you could start with the fundamentals. What are the fundamental concepts or things that you think folks building AI products really need to know about the process of getting to higher quality products? And then I know you're going to show us a couple examples of how to do that. So the fundamentals really come down to the most important thing is looking at data. and I believe from working with many product managers in the past is looking at data has always been a thing like even before AI you know like I'm pretty sure that product managers that can like write a little bit of sequel are okay with spreadsheets looking at numbers looking at metrics you know that feels like it's kind of table stakes for being a good product manager nowadays and so there's just a little bit of a twist on it for AI but really the same thing applies and it's just like, okay, how do you do that for AI?

5:07Hamel Husain:And that's what we teach and that's what I'm gonna show you today. Great, and I cannot agree more. I think one of the most transformational skills I learned as a young baby chicken product manager was being able to write SQL and actually do my own data analysis and exploration. But I think the surface area is so broad now with AI and the data is different. So why don't you show us what we should be looking at when we're building these AI products? Yeah. So let me share my screen a bit. Let me give you some background first. So this is one of my clients. The name of the company is called Nurture Boss.

5:44Hamel Husain:And as you can see, it's an AI assistant for apartment managers or property managers. and really like you know you can kind of get an idea from their website which i'm showing right now you know it's a virtual leasing assistant so you know they help with the whole top of funnel of like helping set up appointments helping prospective residents like find their apartments setting up appointments questions about rent so on and so forth kind of like trying to reduce the toil of property managers still having humans in the loop and so when they came to me they had already prototyped something out you know kind of vibe checking it just like everyone does and put everything together but they wanted to know like okay how do we actually make it work well because the ai fails in weird ways and it doesn't always do the right thing but it feels like okay every time you fix a prompt we're not really sure like maybe we're breaking something else or is it really improving things as a whole we don't really know we're just guessing we're just kind of like looking at it and is getting vibes.

6:54Hamel Husain:And that is a very uncomfortable feeling of trying to scale a product. Okay, so the first thing that I'll jump right into is this idea of traces. So traces are this concept from engineering, but it doesn't have to be scary. It's basically like, and it's very topical for AI because with AI, usually have many different events, are, especially like for a chat bot, you have multi-turn conversations where you're going back and forth with an AI. There might be retrieval of information. They might be calling some tools, external tools, internal tools, so on and so forth. And so you want to log these traces.

7:34Hamel Husain:And there's many different ways to go about it, but just to kind of show you exactly what happened at Nurture Boss, let's go into what that looks like. So this is a platform called Braintrust. There's a lot of them. This is one called Phoenix, which is like the same exact data in here. It doesn't really matter. You can see like they're both the same, right? Like, so what we have here, let me just go into a single trace. So this is what I would call a trace. I can make this bigger so you can see in a full screen. And you can see what an AI interaction looks like in this product. So you have, okay, the system prompt, you are an AI assistant working as a leasing team member at some apartment.

8:21Hamel Husain:These are all fictitious because these have all been scrubbed for PII stuff. You know, your primary role is to respond to text messages. So this is receiving text messages. Okay. And you have a whole host of rules like respond, you know, provide accurate information, answer any question, for residents, do the following, provide this website, for example, if you had to ask for a rental application, provide this on and so forth, all these rules, right? And this is a real user saying, hello, there's what's up to four month rent. I don't even know what that means. I got you. I got you. Let me read it.

8:58Hello. Hello there. What's up two, four month I thought I had it I thought I had it

9:08Hamel Husain:It's unclear but okay, I mean like it's fine, this is real, this is the real world these are real traces and then there's a there's a tool call here get communities information it's calling this tool, this internal tool and the tool call result comes back with this information and this is all hidden from the user The user is not seeing this tool call result. He's like, okay, here's information you can use about the community, blah, blah, blah. He's not even sure this is the right tool call. We'll get to that in a moment. And then the assistant goes, hello, we are currently offering up to... So this is like back to the user.

9:48Hamel Husain:This is what the AI responds to the user with. Hello, we are currently offering up to eight weeks rent-free as a special promotion. Please note, the applicable lease specials and concessions can vary, blah, blah, blah. Okay. So like, is this, and I have a cheat sheet for myself about what is actually right and wrong. Okay. So like the comment here is the user is probably asking about lease terms and stuff like that, not about specials. So like, it's not really clear, like this is the right, this is not like what we want. And this is so realistic, right? Like everyone has experienced AI. Like this is like it's kind of, it's being helpful, but it's not really doing what you want to.

10:32Hamel Husain:And it's actually pretty challenging because it's not really clear what the user wants. You could go in a lot of different directions of this. You know, when I'm testing my own AI, this is such an eye-opening example. Because when I'm testing my own AI, I ask it good questions and I spell correctly and I'm very clear. But when you see a real user input like this, you actually look at what users are prompting your AI with. You realize it's very vague. They say stuff like, what's up? the question, there's no clear question. And so I really do think looking at real user data kind of can get a developer or PM out of their own mind on how they think users are going to interact with the system.

11:13Hamel Husain:Absolutely. It's very critical that you do this. And so now you might not have this data. And I just jumped right into a real example just to set things off. And we can go into all these different rabbit holes or like what if you don't have data and stuff i just want to like ground it and like okay so set the stage like this is kind of one foundation is like you have to have data there's different ways to get it one is you can log it from your real system and you have these things to look at another way is like okay you can have synthetic data where you sort of generate with an llm you can generate questions like this you know hello what's you know it might be hard to generate stuff that looks like that because I don't even know we don't know what it means and probably an LLM won't generate stuff like that but that's the whole interesting bit it's like once you see that people are talking like that you might actually want to simulate stuff that looks like that because if that's what if that's the real distribution of the data or that's what the real world looks like you might want to challenge your LLM or your AI system appropriately okay so let's step back here so you have the system it's doing stuff it's like there's stuff like this happening we can look at another trace if you want just to kind of get an idea and this is you know this is not pre-scripted i didn't memorize what's going on these uh traces we're just looking at them naturally so this is something this is another uh apartment complex meadowbrook apartments same idea so we won't read the whole system prompt again okay so we'll scroll down here let's get to what the user is asking walk in tor so this must be another text message situation and the assistant says our team tries their best to accommodate walk-ins me get you now that's hilarious like i don't know what why is the lm that's surprising like why is it saying me get you to someone who can help maybe he's trying to mimic the uh the uh the user somehow um and then it does like uh yes and then okay great so it seems like this one maybe is okay um let's see what we end up annotating uh yeah we said this one is okay there's there's some metadata down here about our labels which we'll talk about next but yeah you can so you can see like this is a real system there's many different things that can happen here so the question becomes like okay so we talked about this like writing sql and data but like how do you take that same mindset to this like what do you even do with this right you have this like crazy like interactions like how do you analyze this without go without getting stuck because like this seems like um intractable right at first no i i was just thinking i was like what is the SQL query I write to get like the first prompt?

14:06And like, how do you query for give me all the first prompts that include typos, like give me all the first prompts that are ambiguous questions, it just feels almost insurmountable. And then, you know, you showed us two examples, and it's two of probably 1000s and 1000s and 1000s. So going through it manually is probably not super scalable. So I'm curious, what is the systematic kind of solution here? Okay.

14:31Hamel Husain:So the systematic solution is something called error analysis. So error analysis just means it's kind of a counterintuitive process that's extremely effective and it's dumb, but it's accessible to everybody and it works. and it's not something that I made up. It's been around in machine learning for a really long time because actually machine learning has the same problem like before like generative AI. Like we had these stochastic systems that can do like a whole number of things and like how do you actually like analyze that and like figure out like what's going wrong and improve it. So error analysis has two steps.

15:12Hamel Husain:The first step is writing notes and it's called open coding and it's basically like journaling what is wrong. So if we go back to like that other trace that we saw, so let me just go back to it, like the first one, we would step into this trace and we would say, okay, like every observability tool has their own, let's say, different ways to take notes. You know, I already have a note in here. Assistant should have asked follow-up questions about, you know about the question what's up with four month rent because it's unclear user intent and this is writing notes about what is going on okay and you do that for like 100 traces randomly sample 100 traces and you do that you and you stop at the most upstream error you find so you read this and you see what's going on and you're like hmm okay the user intent seems like we didn't do a good job of like clarifying what the hell that they're they need yeah and so i think that's the most upstream problem in this sequence of events so i'm going to go ahead and just write that as a note yeah and and you say focus on the most upstream problem because you presume that if you can get early intent early kind of clarity correctness right the rest of the system is more likely to be correct downstream.

16:33Hamel Husain:Yeah, because it's causal in nature. So as we have the sequence of events, whether it's like user prompts, tool calls, retrieval for rag, whatever it may be, any error at any point along the chain, you know, like will cause downstream problems. And so to simplify our lives for this purposes of error analysis, it's a heuristic, you know, eventually you do want to care about the different errors and different downstream, But when you're starting out, just focus on the upstream error because we're trying to make it tractable. And this is like the way that you're going to get results fast. So basically what you do is you go through and you collect a bunch of notes.

17:14Hamel Husain:And then what you do is you can take these notes and you can like download them or whatever. And you can categorize those notes. And you can even put these notes into like chat GBT. It's like, hey, here's all my notes. Like, can you bucket these into categories? And you kind of have to go back and forth with it a little bit. Like, hey, these are my notes. These are the categories. I think like you're missing a category, whatever. Now with Nurture Boss, what we ended up doing is we actually made one of the things that we highly recommend a lot of people think about is to make your own custom annotation tool.

17:53Hamel Husain:Like there's, you see this is here in BrainTrust and it's also here in Arise Phoenix. they're very similar you can see this is a very similar looking ui and you have they even called it error analysis here and you can like add your notes like you know whatever and you can save those notes and same thing if you're going to be looking at a lot of data you don't want to slow yourself down and you want to be able to have like very human readable sort of you know output and sometimes like this markdown stuff is like not that readable and you want to make sure that okay like it makes sense to you and you can fly through it as fast as possible so um you know it's really easy to vibe code this stuff um because ultimately what you're doing is like showing data so when the in the nurture boss situation so as you might have gathered like they have multiple channels that customers can contact them on they have like text message which would be which we saw they have email they have a chat bot on the website so on and so forth so they just wanted something they could like navigate faster just like vibe coded essentially i mean they have the person we were developers but you know we're using ai in our process and do this very fast is okay like what channel is the trace from and then like some other filters about like hey did we already annotate this or not and then just kind of have some statistics at the top you know this is like what the annotation like looks like it's kind of very similar but just like dialed into what we wanted and like you know we just took notes and then what for nurture boss what we did is okay we had an automated process that would summarize like categorize those notes into like what are the biggest issues and then we would just something very simple like counting counting is always powerful as you know as a product manager you can go into a system the c clear you experience like writing sql queries like you know how powerful counting is counting remains powerful and so you can uh count these issues right so so like okay for nurture boss i don't know if you can see my screen if it's too small i'm trying to zoom in more yeah yeah that's great is okay what are the most what are the biggest issues after doing that error analysis exercise which only took you know a few hours yeah it's like okay um we're having a lot of transfer and handoff issues we're trying to transfer the you the customer to a human we're having a lot of tour scheduling issues so like they're trying to schedule a tour but like a rescheduled tours in this case we found that like someone's asking to reschedule there is no rescheduled tour but like the ai doesn't know that it just keeps scheduling more tours which is bad um you know uh follow-up so you know ai not following up when the user has a question you know sometimes incorrect information provided okay so like you see like these are kind of the count and now we have now we're not lost now we know what we should be working on we know okay you know what we should fix this like transfer handoff issue in this tour scheduling issue.

21:08Hamel Husain:We have confidence. Like, you know what? Like, we're not paralyzed anymore. We know, okay, this is what we need to fix it on our AI. This episode is brought to you by Persona, the B2B identity platform helping product, fraud, and trust and safety teams protect what they're building in an AI-first world. In 2024, bot traffic officially surpassed human activity online. And with AI agents projected to drive nearly 90 % of all traffic by the end of the decade, it's clear that most of the internet won't be human for much longer. That's why trust and safety matters more than ever. Whether you're building a next-gen AI product or launching a new digital platform, Persona helps ensure it's real humans, not bots or bad actors, accessing your tools.

21:55With Persona's building blocks, you can verify users, fight fraud, and meet compliance requirements, all through identity flows tailored to your product and risk needs. You may have already seen Persona in action if you verified your LinkedIn profile or signed up for an Etsy account. It powers identity for the Internet's most trusted platforms, and now it can power yours, too. Visit withpersona.com slash howiai to learn more. I love this. Just to recap, so you're taking these traces of these real conversations and, you know, you don't even have to read all of it. you have to read till you hit a snag, right?

22:34To hit an obvious sort of like incorrect or high friction part of the experience. You have Vibe coded an app that makes it really easy for the team generally to go in, annotate these, rate them sort of like good quality, bad quality, automatically categorize them, count them. And then you have a prioritized list and you're like, here are the problems that I need to go solve. And what I love about this is, You know, I'm sure our listeners expect some like magical system that does this automatically. And you're like, no, man, just spend three hours of your afternoon, go through, read some of these chats, look at some of them with your human eyes, put one sentence notes on all of them, and then run a quick categorization exercise and get to work.

23:20And you see this have actual real impact on quality and reducing these errors.

23:26Hamel Husain:Yeah, it has an immense quality. it's so powerful that some of my clients are so happy with just this process that they're like that's great hamel we're done and i'm like no wait like we can do more um you know you've paid for more like you know whatever they know this is so great like i just feel like i i know what to do and so they find so much value in this like process that and it is like very important this is something that no one talks about like people when you talk about evals like well how do you write an eval? What eval do you do? What tool should you use? Before you get into all that stuff, you need to have some grounding in like what eval you should even write because there's infinite eval.

24:11Hamel Husain:So like in this case, we would write, we wrote an eval about tour scheduling issues and we wrote an eval about transfer handoff issues. And we felt really good about that because we knew that like that is a real problem. And we knew how to write the eval because like we saw that error. And we knew how to find data to test that eval because again, we already tagged it and we saw that error, which is exactly the way you want to do it. Yeah. And what I also like about this is it does take the burden off your users. I mean, so many people try to collect this data by like putting a little thumbs up and thumbs down or little comments.

24:43Like I even have that on parts of my product. And yes, it is useful, but it only gives you a sliver of the kind of self-identified errors in the app. And users are highly tolerant of systems. And so sometimes those errors just don't get escalated by user, they'll either abandon or they'll just work through too many steps to get to the outcome that they want, they'll have a quality experience. And so I'm just taking the burden on yourself and saying you're responsible for looking at the data, you can create simple ways to categorize it. And then you have a prioritized list. Now, if your client is willing to go the next step and do something about this and write evals and fix prompts.

25:26What are your kind of next steps here? What's another example of where we're from here?

25:31Hamel Husain:I just want to talk about this for a minute. Like, okay, so this particular technique is so powerful and not that many people know about it. You know, so I actually recently did a training with open ai showing the people at open ai like you know how this works for domain specific evals um if you want to learn more about like this we had jacob the founder of nurture boss like walk through like this whole process in like two minutes so you can find it on this on this page if you like um okay so to get to your question like what do you do now okay so you have uh like you know you've done your error analysis you have like prioritized these things so like now what do you do so now you get into uh writing the evals so now you have to decide like what kind of evals do you want there's different kinds of evals so there's reference based evals which is like you know what the right answer is and maybe you can write some code you don't need like an lm to do the eval for you.

26:35Hamel Husain:Or if it's more subjective in nature, then, you know, maybe like this transfer handoff issue, maybe it's more subjective in nature, then you need an LLM judge. And so what you can do is you can start to write those evals. And so I have this blog post here about evals in general. So there's this diagram. It's really hard to put this whole thing into a diagram, honestly, but because, you know, it can be, it's kind of, it's not, it's a non-linear process. But really what you want to do is, okay, we already covered like logging traces and there's two different kinds of, but there's different kinds of evaluators or evaluations.

27:21Hamel Husain:There's like kind of like unit tests, which is like, well, I would say like code-based evals. And then there's like models. So like LLMs, you know, code-based evals. So like, you know, So for example, what kinds of things that would be good for code-based evals? Like, okay, if you have like user IDs showing up in the response or something like that, okay, you can test for that in code. I have to say you're saving my life here because I was thinking, what is one of these unit tests I need to write? And that is exactly one of them, which is my tool calls need UU IDs and users definitely do not. So that's a great example of one for anybody that's writing a chatbot that does a lot of kind of tool calling.

28:02Hamel Husain:Yeah, because they can show up by accident. Like you might have the UID in the system prompt, and it inadvertently shows up in output for some reason or another, and you don't want that. Okay, you want to write these tests. No matter what kinds of tests you write, you want to create test cases, and sometimes you can gather those from your traces. Sometimes you might want to generate synthetic data. and so um you know this is like a prompt for a different real estate agent assistant called reach out which is for residential real estate um and this is kind of like a simplified version of your prompt right 50 different instructions that a real estate agent can give to their assistant it creates contacts on their crm contact details can include name phone email whatever and basically you know it can generate synthetic inputs to a system that then you can then log traces from.

28:59Hamel Husain:I'm going to jump around a little bit, so we'll kind of come back to that. Okay, we already covered logging traces. This is another custom log annotation thing, yet again, because we really emphasize this, that it's really important to remove all friction doing this, so I won't linger on this too much. And basically, you know one kind of thing you want to do is like okay if you're using LLM as a judge or anything else what you want to do is so one thing that's usually skipped when we talk about LLM as a judge is like people just using LLM as a judge off the shelf like they're like writing a prompt they're saying okay judge it and then reporting that let me actually go to a different blog post that is a little bit better for LM judge, which is this one.

29:54Hamel Husain:Okay, so LM as a judge. So you often see sometimes in LM eval land, like a dashboard that looks like this. Helpfulness, truthfulness, conciseness score, tone, whatever. What the hell does that mean? Does anyone know what that means? Nobody knows. No one understands concretely. Like if the helpfulness score is 4.2 and it goes to 4.7, like, do you really know, like, what's wrong, what changes? No. And so there's a lot of guidance in how to create an LLM as a judge. It's probably too much for this podcast to, like, tell you all of the things. And this blog post is quite long, like, enumerating, like, how to do it correctly.

30:39Hamel Husain:But the main things that you need to keep in mind is, like, one, you need to have binary outputs like is it good or bad for a specific problem so for like you know the handoff problem for nurture boss like okay was there a problem or not and you want specific evaluators for specific problems number two is like you want to you need to hand label some data which you already kind of do an error analysis and you want to compare the judge to the hand label data so that you can trust the judge. The last thing you want to do is like throw up a judge on the dashboard like this and then like people don't know if they can trust it.