Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)

27:42Oftentimes it is the product manager. Cool. So the advisor, pick that person may not feel so super fair that they're the one in charge and they're the dictator, but they're benevolent. It's going to be okay. Yeah, it's going to be okay. You're just trying to, it's not perfection. You're just trying to make progress and get signal quickly. So you have an idea of what to work on because it can become infinitely expensive if you're not careful yeah okay cool let's go back to your examples yeah no problem so this is another example where we have someone saying okay do you have any specials and the assistant or the ai responds hey we have a five percent military discount user responds can you in the switch as a subject can you tell me how many floors there are do you have any one bedrooms available or one bedrooms on the first floor and the ai responds yeah okay we have several one bedroom apartments available and then the user wants to confirm any of those on the first floor and how much are the one bedrooms and then also is a current resident so it's they're also asking i need a maintenance request this is actually pretty like you could see the messiness of the real world in here and the assistant just calls a tool that says transfer call but it doesn't say anything it just abruptly does transfer call so it's pretty jank i would say like it's just not you know another jank another kind of jank a different kind of jank so you don't want to when you write the open note you don't want to say jank because what we want to do is we want to understand what when we look at the notes later on we'll understand like what happened So you just want to say, you know, did not confirm call transfer with user.

29:35It doesn't have to be perfect. You just have to have a general idea of what's going on. Cool. So, okay. So let's say we do, and Treya and I, we recommend doing at least 100 of these. The question is always, like, how many of this do you do? And so there's not a magic number. we say 100 is because we know that as soon as you start doing this once you do 20 of these you will automatically find it so useful that you will continue doing it so we just say 100 to mentally unblock you so it's not intimidating like don't worry you're only going to do 100 And there is a term for that. So the right answer is keep looking at traces until you feel like you're not learning anything new.

30:28Maybe Shreya should talk about that. Yeah, so there's actually a term in data analysis and qualitative analysis called theoretical saturation. So what this means is when you do all of these processes of looking at your data, when do you stop? It's when you are theoretically saturating or you're not uncovering any new types of notes, new types of concepts, or nothing that will like materially change the next part of your process. And this kind of takes a little bit of intuition to develop. So typically people don't really know when they've reached theoretical saturation yet. That's totally fine.

31:06When you do two or three examples or rounds of this, you will develop the intuition. A lot of people realize like, oh, okay, I only need to do 40. I only need to do 60. Actually, I only need to do like 15. I don't know. Depends on the application and develops like how, depends on how savvy you are with error analysis for sure. And your point about you probably want to, you're going to want to do a bunch. I imagine it's because you're just like, oh, I'm discovering all these problems. I got to see what else is going on here. Exactly. And promise at some point you're like not going to discover new types of problems.

31:39Yeah. Awesome. So let's say you did 100 of these. What's the next step? Yeah. Okay. So you did 100 of these. Now you have all these notes. So this is where you can start using AI to help you. Yeah. So the part where you looked at this data is important, like we discussed. You don't want to automate this part too much. Humans will still have jobs. This is a takeaway here. That's great. Yes. Just reviewing traces. At least there's one job left for now. Yeah. Yeah, exactly. And so, okay, you have all these notes. Now to turn this into something useful, you can do basic counting. So basic counting is the most powerful analytical technique in data science because it's so simple and it's kind of undervalued in many cases.

32:30And so it's very approachable for people. And so the first thing you want to do is take these notes and you can categorize them with an LLM. And so there's a lot of different ways to do that. Right before this podcast, I took three different coding agents or AI tools and had it categorize these notes. So one is, okay, I uploaded into a Cloud project. I uploaded a CSV of these notes. And I just exported them directly from this interface. There's a lot of different ways to do this. but I'm showing you the simple, stupid way, the most basic way of doing things. And so I dumped a CSV in here, and I said, please analyze the following CSV file.

33:19And I told it there's a metadata field that has a note in it. But what I said is I used the word open codes. And I said, hey, I have different open codes. And that's a term of art. So LLMs know what open codes are and they know what axial codes are because it is a concept that's been around for a really long time. So those words help me shortcut what I'm trying to do. That's awesome. And the end of the prompt is telling it to create axial codes. Yes, creating axial codes. So what it does is... So maybe it's worth talking about what are axial codes or like what's the point here? You have a mess of open codes.

34:00and you don't have 100 distinct problems. Actually, many of them are repeats, but because you phrase them differently, you shouldn't have tried to create your taxonomy of failures as you're open coding. You just want to get down what's wrong and then organize, okay, what's the most common failure mode? So the purpose, axial code basically is just a failure mode. It's like the label or category. And what our goal is, is to get to this clusters of failure modes and figure out what is the most prevalent. So then you can go and run and attack that problem. That is really helpful. Basically, just synthesizing all these categories and themes.

34:40Super cool. And we'll include this prompt in our show notes for folks so they don't have to like sit there and screenshot it and try to type it out themselves. Yeah, great idea. And so Claude, you know, went ahead and analyzed the CSV file and decided how to parse it, blah, blah, blah. We don't need to worry about all that stuff. But it came up with a bunch of axial codes. Basically, axial codes are categories, like Shreya said. So one is, okay, capability limitations, misrepresentation, process and protocol violations, human handoff issues, communication quality. It created these categories. Now, do I like all the categories?

35:21Not really. I like some of them. It's a good first, like, stab at it. I would probably rename it a little bit because some of them are a bit too generic. Like what is capability limitations? I said a little bit too broad. It's not actionable. I want to get like a little bit more actionable with it so that if I do decide it's a problem, I know what to do with it. But we'll discuss that in a little bit. So you can do this like with anything. And this is the dumbest way to do it. But dumb sometimes is a good way to get started. And this is what LMs are really good at, taking a bunch of information and synthesizing.

35:53Absolutely. Synthesizing for us to make sense of, right? Note that, you know, it's not telling us, it's not automatically proposing fixes or anything. That's our job. But, you know, now we can wade through this mess of open codes a lot easier. Another thing that's interesting here in this prompt to generate the axial codes is you can be very detailed if you want. Right. You can say, I want each axial code to actually be, you know, some actionable failure mode. And maybe the LLM will understand that and propose it. or I want you to group these open codes by what stage of the user story that it's in.

36:30So this is where you can be creative or do what's best for you as a product manager or engineer working on this, and that will help you do the improvement later. So there's no definitive prompt of here's the one way to do it. You can iterate, see what works for you. Absolutely. It's interesting the tools don't do this, or do they try and they just don't do a great job? No, I don't think they do it. We've been screaming from the rooftops, please, please do this. I do think it's a little bit hard, right? Like part of this whole experience with the evals course Hamill and I are teaching are like a lot of people don't actually know this.

37:03So maybe it's that people don't know this and they don't know how to build tools for it. And hopefully we can demystify some of this magic. And just to double click on this point, like this is not a thing everyone does or knows. this is something you too developed based on your experience doing data analysis and data science at other companies? Well, I want to caveat it. We didn't invent error analysis. We don't actually want to invent things. That's a bad signal. If somebody is coming to you with a way to do something that's like entirely new and not grounded and hundreds of years of theory and literature, then you should, I don't know, be a little bit wary of that.

37:42But what we tried to do was distill, okay, what are the new tools and techniques that you need to make sense of the LLM error analysis? And then we created a curriculum or structured way of doing this. So this is all very tailored to LLMs, but the terms open coding, axial coding are grounded in social science. Amazing. Okay. What's funny about you guys doing this is I just want to go do this somewhere. I don't have any product to do this on, but it's just like, oh, this would be so fun. Just sit there and find all the problems I'm running into and categorize them and then try to fix them. Delightful.

38:18I love that. Hamill pulled up a video. What do you got going on here? Yeah. So I pulled up a video just to drive home Shreya's point. Like we are not inventing anything. So what you see on the screen here is Andrew Ng, one of the famous machine learning researchers in the world who have taught a lot of people, frankly, machine learning. And you can see this is an eight-year-old video. And he's talking about error analysis. And so this is a technique that's been used to analyze stochastic systems for ages. And it's something that if you're just using the same machine learning ideas and principles, just bringing them into here.

38:58Because, again, these are stochastic systems. Awesome. One thing, we're working on getting Andrew in the podcast. We're chatting, so that'll be really fun. And two, I love that my podcast episode just came out today is in your feed there, and it's standing out really well in that feed, so I'm really happy about that thumbnail. Very nice. Yeah, the recommendation algorithm is quite good. Yes, here we go. I hope you click on that. Don't screw my algorithm. Okay, cool. So we've done some synthesis. I know we're not going to go through the entire step. This is like, you have a whole course that takes many days to learn this whole process.

39:29What else do you want to share about how to go about this process? Okay, so you can do this through anything, and I've used, the same thing works just fine in chat GPT, the same exact prompt. You can see it made axial codes. I really like using Julius AI. It's one of my favorite tools. Julius is kind of his third-party tool, but it uses notebooks. I personally like Jupyter notebooks a lot. And so it's more of a data science thing, but a lot of product managers that are kind of learning notebooks nowadays, and it's kind of cool. It's like a fun playground where you can write code and look at data.

40:03But we don't have to go deeply into that. Just wanted to mention you can use a lot, you know, AI is really good at this. So let's go into the fun part. Here we go. So now we have all the, we have these axial codes. So the first thing I like to do, I have these open codes, right? And I have the axial codes that, let's say, you know the like that we assigned from the cloud project or the chat gpt and so what i do is i collect them first and i take a look like does these axial codes make sense and i look at the correspondence between the different axial codes and the open codes and i and i go through an exercise and i say hmm do i like these these codes like can i make them better can i refine them can make them more specific um you know instead of like being generic, I make the very specific in actionable.

40:59So you see the ones that I came up with here are tour scheduling, rescheduling issues, human handoff or transfer issue, formatting error with an output, conversational flow. We saw the conversational flow issue with the text messages, making follow-up promises not kept. And so basically what I can do, what you can do now is like you have these axial codes. And so I just collect them into a list. So this is an Excel formula. Just collect these codes into a list. And now we have a comma separated list of these codes. And then what you can simply do is you could take your notes that you have, those open codes, and you can tell an AI, and this is using Gemini and AI, just for simplicity, this is like the, you know, again, we're trying to keep it simple.

41:50categorize the following note into one of the following categories. By the way, for folks watching, I like all these different prompts and formulas you're sharing. This is like the Google Sheets AI prompt. Huge fan. And so basically what you can do is you can then have, you can categorize your traces into one of the buckets. And that's what we have here. We have categorized all those problems that we encountered into one of these things. And this is automatic, which is very exciting. I mean, the AI is doing it. So this also drives home the point that your open codes have to be detailed, right?

42:30You can't just say janky because if the AI is reading janky, it's not going to be able to categorize it. Even a human wouldn't, right? It would have to go and remember why you said janky. So it's important to be, you know, somewhat detailed in your open code. Okay. So avoid the word janky is a good rule of thumb. Or have it with like 10 other words. Yeah, I was being funny. Yeah, okay. What are some of those other words that people often use that you think are not good? I don't think it's specific words. I think it's just people are not detailed enough in the open code, so it's hard to do the categorization.

43:04Great. And by the way, the reason you have to map them back is because say Clotter or JGPT gave you suggestions and you changed them and iterated on them. So it doesn't, you can't just go back and say, cool, whatever, use bucket. Yeah, yeah. Great. That's a really good question, actually. It's good to iterate and think about it a little bit. Like, do I like these open codes? Do these actually make sense to me? Just like anything that AI does, it's really good to kind of put yourself in the middle. Just a little bit. Humans in a loop. Still space. Yes. Great. One of the things that I like to do in this step, if I'm trying to use AI to do this labeling, is also have a new category called none of the above.

43:43so an AI can actually say none of the above in the axial code and that informs me okay my axial codes are not complete like let's go look at those open codes let's figure out what some new categories are or figure out how to reword my other axial codes awesome and what's cool about this is you don't need to do this many many times like no for most products you do this process once and then you build on it I imagine and you just tweak it over time absolutely and it gets so fast People do this once a week and you can do all of this in 30 minutes. And suddenly your product is so much better than if you were never aware of any of these problems.

44:22Yeah, it's absurd to feel like you wouldn't know this was happening. Watching this happening, how could you not do this to your product? A lot of people have no idea. Most people. Yeah. We'll talk about that. There's a whole debate around this stuff that we want to talk about. Okay, cool. So you have this sheet. Welcome to the next. Okay, so here's the big unveil. This is the magic moment right now. So we have all these codes that we applied, the ones that we like on our traces. Now you can do the ta-da, you can count them. So here's a pivot table, and we just can do a pivot table on those, and we can count how many times those different things occurred.

45:04So what do we find? Find on these traces that we categorize, We found 17 conversational flow issues. And I really like pivot tables because you can do cool things. You can like double click on these. You can say, oh, okay, let me take a look at those. But that's going into an aside about pivot tables, how cool they are. But, you know, now we have just a nice rough cut of what are our problems? and now we have gone from chaos to some kind of thinking around oh you know what these are my biggest problems i need to fix conversational issues you know maybe these human handoff issues but not necessarily the count is the most important thing you know that might be something that's just really bad and you want to fix that but okay now you have some way of looking at your problem and now you can think about whether you need evals uh for for some of these so you know with the you know there might be some of these things that might be just dumb engineering errors that you don't need to write an eval for because it's very obvious on how to fix them maybe the formatting error with output maybe you just forgot to tell the llm how you want it to be formatted.

46:29And like, you didn't even say that in the prompt. So like, just go ahead and fix the prompt. Maybe, you know, and we can decide like, okay, do you want to write an eval for that? You might still want to write an eval for that because you might be able to test that with just code. You could just test the string. Does it have the right formatting potentially without running an LLM? So there's a cost benefit trade-off to evals. You don't want to get carried away with it. But you want to start, you want to usually ground yourself in your actual errors. You don't want to skip this step. And so the reason I'm kind of spending so much time on this is like, this is where people get lost.

47:13They go straight into evals. Like, let me just write some tests. And that is where things go off the rails. So let's, okay, so let's say we want to tackle one of these things. So, for example, let's say we want to tackle this human handoff issue. And we're like, hmm, I'm not really sure how to fix this. Like, that's a kind of subjective sort of judgment call on, you know, should we be handing off to a human? And I don't know immediately how to fix it. It's not super obvious, per se. Yeah, I can, like, change my prompt, but I'm not, like, sure. I'm not 100 % sure. Well, that might be sort of an interesting thing for an LLM as a judge, for example.

48:04So there's different kinds of evals. One is code-based, which you should try to do if you can, because they're cheaper. You don't have to, you know, LLM as a judge is something, it's like a meta eval. You have to eval that eval to make sure the LLM that's judging is doing the right thing, which we'll talk about in a second. so okay lm as a judge that's one thing okay how do you build an lm as a judge before we get into that actually just to make sure people know exactly what you're describing there these two types of evals one is you said it's code based one is lm as judge maybe shreya just help us understand what that what code base eval even is it's just like it's like essentially a unit test is that a simple way yeah maybe eval is not the right term here but think like automated evaluator So when we find these failure modes, one of the things we want is like, okay, can we now like go check the prevalence of that failure mode in an automated way without me manually labeling and doing all the coding and the grouping and I want to run it on thousands and thousands of traces, I want to run it every week.

49:06That is, okay, you should probably build an automated evaluator to check for that failure mode. Now, when we're saying code-based versus LLM-based, we're saying, okay, so maybe I could write like a Python function or a piece of code to check whether that failure mode is present in a trace or not. And that's possible to do for certain things like checking the output is JSON or checking that it's markdown or checking that it's short. Like these are all things you can capture in code or you could approximately capture in code. When we're talking about LLM judge here, we're saying that this is a complex failure mode and we don't know how to evaluate in an automated way.

49:46So maybe we will try to use an LLM to evaluate this very, very narrow, specific failure mode of handoffs. So just to try to mirror back where you're describing, you want to test what your, say, agent or AI product is doing. You ask it a question, it gets back with something. One way to test if it's giving you the right answer is if it's consistently doing the same thing, that you could write a code to tell you this is true or false. For example, will it ever say there's a virtual tour? So you could ask it, do you provide virtual tours? It says yes or no, and then you could write code to tell you if it's correct based on that specific answer.

50:27But if you're asking about something more complicated and it's not binary, you almost need, like in one world, you need a human to tell you this is correct. the solution to avoid humans having to review all this every time automatically is LLMs replacing human judgment. And you'd call it LLM as judge. The LLM as being the judge, if this is correct or not. Absolutely. You nailed it. So people always think like, oh, this is at least as hard as my problem of creating the original agent. And it's not because you're asking the judge to do one thing, evaluate one failure mode. So the scope of the problem is very small.

51:05And the output of this LLM judge is like pass or fail. So it is a very, very tightly scoped thing that LLM judges are very capable of doing very reliably. And the goal here is just to have a suite of tests that run before you ship to production that tell you things are going the way you want them to, the way your agent is interacting. The beautiful thing about LLM judge is you can use them in unit tests or CI, sure. But you could also use it online for monitoring, right? Like I can sample like thousand traces every day, run my LLM judge, real production traces and see what the failure rate is there.

51:45This is not a unit test, right? But still now we get like an extremely specific measure of application quality. Cool, that's a really great point because a lot of people dis-evalced for being this It's like not real life thing. It's a thing that you test before it's actually in the real world. And what's actually happening in the real world? You're saying you should actually do exactly that. Test your real thing running in production. And it's like a daily hourly sort of thing you could be running. Totally. Awesome. Okay. Hamill's got an example of an actual LM as a judge eval here. So let's take a look.

52:16I love how Shreya really teed it up for me. So thank you so much. So what we have is a LM as a judge prompt for this one specific failure. Like Shreya said, you would want to do one specific failure and you want to make it binary. Because we want to simplify things. We don't want, hey, like score this on a rating of one to five. Like how good is it? That's just mostly, in most cases, that's a weasel way of like not making a decision. Like, no, you need to make a decision. Is this good enough or not? Yes or no? can be painful to think about what that is but you should absolutely do it otherwise this thing becomes very untractable and then when you report these metrics no one knows what 3.2 versus 3.7 means this is yeah we see this all the time also and even with like expert curated content on the internet where it's like oh here's your llm judge evaluator prompt here's a one to seven scale and I always text Hamill like, oh no, now we have to fight the misinformation again because we know somebody is going to try it out and then come back to us and say, oh, I have 4.2 average and we're going to be like, okay.

53:30It's wild how much drama there is in evals space. We're going to get to that. Oh man. This episode is brought to you by Mercury. I've been banking with Mercury for years and honestly, I can't imagine banking any other way at this point. I switched from Chase and holy moly, what a difference. Sending wires, tracking spend, giving people on my team access to move money around. So freaking easy. Where most traditional banking websites and apps are clunky and hard to use, Mercury is meticulously designed to be an intuitive and simple experience. And Mercury brings all the ways that you use money into a single product, including credit cards, invoicing, bill pay, reimbursements for your teammates and capital.

54:12Whether you're a funded tech startup looking for ways to pay contractors and earn yield on your idle cash, or an agency that needs to invoice customers and keep them current, or an e-commerce brand that needs to stay on top of cash flow and access capital, Mercury can be tailored to help your business perform at its highest level. See what over 200 ,000 entrepreneurs love about Mercury. Visit mercury.com to apply online in 10 minutes. Mercury is a fintech, not a bank. Banking services provided through Mercury's FDIC-insured partner banks. For more details, check out the show notes. Okay, so this is your judge prompt.

54:48There's no one way to do it. It's okay to use an LLM to help you create it, but again, put yourself in a loop. Don't just blindly accept what the LLM does. And in all of these cases, that's what we did. Like with the axial codes, we kind of iterated on this. You can use an LLM to help you create this prompt, but make sure you read it, make sure you edit it whatever this is not necessarily the perfect prompt this is just the stupid like very keeping it very simple just to show you the idea is like okay for this handoff failure um you know i said okay i want you to output true or false is a binary it's a binary judge that's what we recommend and then we then i just go through and say okay like when should you be doing a handoff And I just list them out like, okay, explicit human requests ignored or looped, some policy mandated transfer, sensitive resident issues, tool data unavailability, same day walk-in or tour requests.

55:46You know, you need to talk to a human for that, so on and so forth, right? And so the idea is like now that I know that this is a failure from my data, I'm interested in iterating on it because I know this is actually happening all the time. And like Shreya said, it would be nice to have a way not only to evaluate this on the data I have, but also on production data. Just to get a sense of what scale is happening. Let me find more traces. Let me have a way to iterate on this. And so we can take this prompt. And I'm going to use a spreadsheet again. So the first step is, okay, when I'm doing this judge, I wrote the prompt.

56:28Now, a lot of people stop there and they say, okay, I have my judge prompt. We're done. Good. Like, let's just ship it. And the prompt says if the judge says it's wrong, it's wrong. They just like accept it as the gospel. Be like, okay, the LLM says wrong. It must be wrong. Don't do that. Because that's the fastest way that you can have evals that don't match what's going on. And when people lose trust in your evals, they'll lose trust in you. So it's really important that you don't do that. And so one, before you release your LLM as a judge, you want to make sure it's aligned to the human. So how do you do that?

57:04Is you actually, you have those axial codes and you want to like measure your judge against the axial code and say like, Hey, does it agree with me? Does my own judge disagree with me? Just measure it. And so what we have here is, okay, I say assess this LLM trace. Again, I'm using just spreadsheets here. Assess this LLM trace according to these rules. And the rules are just the prompt that I just showed you. and I ask it, okay, is there a handoff error? true or false. So then this column, let me just zoom in a bit. Column H, I have, okay, did this error occur? And column G is whether I thought the error occurred or not.

57:53You can see... You're going through it manually, you do that. Yeah, yeah, which we already did. We already went through it manually. So it's not like we have to do it again because we kind of have that cheat code from the axial coding, we already did it. You might have to go through it again if you need more data. And there's a lot of details to this on how to do this correctly. You want to split your data and do all these things so that you're not cheating. But I just want to show you the concept. And basically, what you can do is measure the agreement. Now, one thing you should know as a product manager is a lot of people go straight to this agreement.

58:31They say, okay, my judge agrees with the human at some percentage of the time. Now, that sounds appealing, but it's a very dangerous metric to use because a lot of times errors have, you know, they only happen on the long tail and they don't happen as frequently. So like if you only have the error 10 % of the time, then you can easily have 90 % agreement by just having a judge say it passes all the time. Does that make sense? So like 90 % agreement might look good on paper, but it might be misleading. It's a rare area. Yeah. So, you know, as a product manager or someone, even if you're not doing this calculation yourself, if someone ever reports to you agreement, you should immediately ask, OK, tell me more.

59:29Like you need to, you know, you don't need to look into it to give you more intuition. Here is like a matrix. OK, of this specific judge in the Google sheet. And this is, again, a pivot table, just keeping it dumb and simple, is, okay, on the rows I have, what did the human think? What did I think? Did it have an error, true or false? And then did my judge have an error, true or false? The intuition here is exactly what Hamill said, right? You need to look at each type of error. So when the human said false, but the judge said true or vice versa. So those non-green diagonals here. And if they're too large, then go iterate on your prompt.

1:00:13Make it more clear to the LLM judge so that you can reduce that misalignment. You want to get to a point where most you're going to have some misalignment. That's OK. We talk about in our course also how to code correct that misalignment. But in this stage, if you're a product manager and the person who's building the LLM judge eval has not done this, they're saying like, oh, it agrees 75 % of the time. We're good. They don't like have this matrix and they haven't iterated to make sure that these two types of errors have gone down to zero, then it's a bad smell. Go and ask them to go fix that.

1:00:52Awesome. That's a really good tip is what to look for when someone's doing this wrong. Yeah. Yeah. Actually, can you take us back to the LLM as judge prompt? I just want to highlight something really interesting here. I've had some guests on the podcast recently who've been saying evals are the new PRDs. And if you look at this, this is exactly what this is. Like product managers, product teams, right? Here's what the product should be. Here's all the requirements. Here's like how it should work. They built the thing and then they test it manually often. And what's cool about this is this is exactly that same thing.

1:01:23And it's running constantly. It's telling you, here's how this agent should respond in very specific ways. If it's this, this, this, this, do that. If it's this, this, that, do that. And so it's exactly what I've been hearing again and again. You could see right here. This is like the purest sense of what a product requirements document should be is this eval judge that's telling you exactly what it should be. And it's automatic and running constantly. Yeah, absolutely. And it's kind of derived from our own data. So, of course, it's a product manager's expectations. What I find a lot of people miss is they just put in what their expectations are before looking at their data.

1:01:57But as we look at our data, we uncover more expectations that we couldn't have dreamed up in the first place. And that ends up going into this prompt. So that is interesting. So your advice is not skip straight to evals and LLMS judge prompts before you build the product. Still write traditional one-pagers. PRDs to tell your team what we're doing, why we're doing it, what success looks like. But then at the end, you could probably pull from that and even improve that original PRD if you're evolving the product using this process. I would go even further to say you're going to improve. It's going to change.

1:02:31You're never going to know what the failure modes are going to be up front. And you're always going to uncover new vibes that you think that your product should have. You don't really know what you want until you see it with these LLMs. So you've got to be kind of flexible, have to look at your data, have to. PRDs are a great abstraction for thinking about this, but it's not the end all be all. It's going to change. I love that. And Hamill's pulling up some cool research report. What's this about? Oh, this is one of the coolest research reports you can possibly read if you want to know about evals.

1:03:09So it was authored by someone named Shreya Shankar. Oh, my God. Oh. and her collaborators. And so it's called Who Validates the Validators? That is the best name for a researcher. Thank you. Thank you. So I should let Shreya talk about this. I think one of the most important things to pay attention to in this paper are the criteria drift and what she found. So we did this super fun study when we were doing user studies with people who were trying to write LLM judges. or just validate their own LLM outputs. And this was, I think, this was before evals was like extremely popular, I feel like, on the internet.

1:03:55We did this project like late 2023. It was when we started it. But then the thing that really was burning in my mind as a researcher was like, why is this problem so hard? We've been having machine learning and AI for so long. It's not new. But suddenly this time around, everything is really difficult. So we just did this user study with a bunch of developers and we realized, okay, what's new here is that you can't figure out your rubrics up front. People's opinions of good and bad change as they review more outputs. They think of failure modes only after seeing 10 outputs they would never have dreamed of in the first place.

1:04:34And these are experts, right? These are people who have built many LLM pipelines and now agents before. And just you can't ever dream up everything in the first place. And I think that's so key in today's world of AI development. Okay, that is a really good point. That's very much reinforcing what we were just talking about, and that's the way Hamill pulled this up. The research behind it. Yeah, okay, great. You still got to do product the same way, but now you have this really powerful tool that helps you make sure what you've built is correct. It's not going to replace the PRD process. Cool.

1:05:09How many evals of these, how many, say, I don't know, LAnima's judge prompts do you end up with usually? I don't know. I know it obviously depends on complexity of the product, but what's a number in your experience? For me, like between four and seven. Oh, that's it. It's not that many because a lot of the failure modes, as Hamill said earlier, can be fixed by just fixing your prompt. You just didn't think to put it in your prompt. So now you put it in your, you shouldn't do an eval like this for everything. Just the pesky ones that you've described your ideal behavior in your agent prompt, but it's still failing.

1:05:43Got it. So say you found a problem, you fixed it. But in traditional software development, you'd write a unit test to make sure it doesn't happen again. Is your insight here is don't even bother writing an eval around that if it's just gone? I think you can if you want to, but the whole game here is about prioritizing. You have finite resources and finite time. You can't write an eval for everything, so prioritize the ones that are the more pesky areas. And probably the ones that are most risky to your business if they say something like Mecca Hitler, Scroch. Yikes. Cool. Okay, so that's very relieving.

1:06:17Because this was prompt, like, a lot of work to really think through all these details. But it's a lot of one-time cost. Right now, forever, you can run this on your application. Right. And I want to say, okay, data analysis is super powerful. It's going to drive lots of improvements very quickly to your application. We showed the most basic kind of data analysis, which is counting, which is accessible to everyone. you can get more sophisticated with the data analysis. There's lots of different ways to sample, look at data. We kind of made it look easy in a sense, but there's a lot of skills here to do to it well.

1:07:01Building an intuition and a nose for how to sort through this data. For example, let's say I find conversational issues, these conversational flow issues. maybe if i was trying to chase down this problem further i would think about ways to find other conversational flows flow issues that i didn't code you know i would maybe dig through the data in several ways um and there's you know different ways to go about this it kind of it's very similar if not almost exactly similar as kind of traditional analytics techniques that you would do on any product. Give us just a quick sense of what comes next.

1:07:44And then let's talk about the debate around evals and a couple more things. So what comes next after you've built your LLM judge? Well, we find that people just try to use that everywhere they can. So they will put the LLM judge in unit tests. And they will know like, oh, here are some example traces where we saw that failure because we labeled it. Now we're going to make those part of unit tests and make sure that every time we push a change to our code, these tests are going to pass. They also use it for online monitoring. People are making dashboards on this. And I think that's incredible. I think like the products that are doing this, right, they have a very sharp sense of how well their application is performing.

1:08:23And people don't talk about it because this is their moat. Right. So people are not going to go and share all of these things because it makes sense. If you are an email writing assistant and you're doing this and you're doing it well, you don't want somebody else to go and build an email writing assistant and then kind of get you out of business. So I really want to stress the point that it's like try to use these artifacts that you're building wherever possible online, repeatedly, use them to drive improvements to your product. Oftentimes, Hamel and I will kind of, we'll tell people how to do this up to this very point and it clicks for people and then they like never come back again.

1:09:01So either they have, I don't know, quit their jobs, they're not doing AI development anymore, or they know what to do from here on out. I think it's the latter, but I think it's very powerful. Like just watching you do this, really opened my eyes to what this is and how systematic the process is. I always imagine you just sit on a computer. Okay, what are the things I need to make sure work correctly? And what you're showing us here is, it's a very simple step-by-step based on real things that are happening in your product, how to catch them, identify them, prioritize them, and then catch them if they happen again and fix them.

1:09:38Yeah, it's not magic. Like anyone can do this. You're going to have to practice the skill, like any new skill you have to practice, but you can do it. And I think what's very empowering now that product managers are doing this and can do this. They can really build very, very profitable products with this skill set. Okay, great segue to a debate that we kind of got pulled into that was happening on X the other day. I did not realize how much controversy and drama there is around evals. There's a lot of people with very strong opinions. So how about Shreya, give us just a sense of the two sides of the debate around the importance and value of evals and then give us your perspective.

1:10:19Yeah. So, all right. I'll be a little bit placating and I say I think everyone is on the same side. I think the misconception is that people have very rigid definitions of what evals is. For example, they might think that evals is just unit tests or they might think that evals is just the data analysis part and no online monitoring or any no monitoring of product specific metrics like actually number of chats engaged in or whatnot. So I think everyone has a different mindset of evals going in. And the other thing I will say is that people have been burned by evals in the past. So I think people have done evals badly.

1:11:00One concrete example of this is they've tried to do an LLM judge, but it has not aligned with their expectations. They only uncovered this later on and then they didn't trust it anymore. And then they're like, oh, I'm anti-evals. And I 100 % empathize with that because you should be anti-Likert scale LLM judge. I absolutely agree with you. We are anti that as well. So a lot of the misconception stems from two things, right? Like people having a narrow definition of evals and then people not doing it well and then getting burned and then wanting to avoid other people making that mistake. And then unfortunately, X or Twitter is like a medium where people are misinterpreting what everybody is saying all the time.

1:11:44And you just get all these strong opinions of like, don't do evals. It's bad. We tried it. It doesn't work. We're Claude Code or whatever other famous product. And we don't do evals. And there's just so much nuance behind all of it because a lot of these applications are standing on the shoulders of evals. Coding agents is a great example of that. Cloud code. They are standing on the shoulders of cloud. The fine-tuned cloud models have been evaluated on many coding benchmarks. Can't argue against that. And just to make clear exactly what you're talking about there, one of the heads, I think maybe the head engineer of Cloud Code went on a podcast and he's like, oh, we don't do evals, we just vibe.

1:12:32We just look at vibes. And vibes meaning they just use it and feel if it's right or wrong. And I think that kind of works. So there's two things to that. One is they're standing on the shoulders of the evals that their colleagues are doing for coding. Of the Cloud Foundational Model. Absolutely. We know that they report those numbers because we see the benchmarks. We know who's doing well on those. The other thing is they are actually probably very systematic about the error analysis to some extent. I bet you that they are monitoring who is using Claude, how many people are using Claude, how many chats are being created, how long these chats are.

1:13:08They're also probably monitoring in their internal team. They're dogfooding. Anytime something is off, they maybe have a queue or they send it to the person developing cloud code. And this person is implicitly doing some form of error analysis that Hamill talked about. All of this is evals. There's no world in which they're just being like, I made cloud code. I'm never looking at anything. And unfortunately, right, when you don't think about that or talk about that, I think that the community, most of the community is beginners, right? People who don't know about evals and want to learn about it.

1:13:43And it sends the wrong message there. Now, I don't know what CloudCode is doing, obviously, but I would be willing to bet money that they're doing something in the form of evals. We'll also say that coding agents are fundamentally very different than other AI products because the developer is the domain expert so you can short circuit a lot of things and also the developer is using it all day long so there's a type of dogfooding and type of domain expertise that is you know you can collapse the activities you don't need as much data you don't need as much feedback or exploration because you know so your eval process you know should look different.

1:14:31Because you're seeing the code. You see the code is generating. You can tell this is great. This is terrible. Yeah. And so I think a lot of people had generalized coding agents because coding agents are the first AI product released into the wild. And I think it's a mistake to try to generalize that at large. The other thing is, yeah, engineers have a dogfooding personality. There are plenty of applications where people are trying to build AI in certain domains and they don't have dogfooding for like doctors, for example, are not out there trying to get all the most incorrect advice from AI and be tolerant and receptive to that.

1:15:12So it's very important to keep, I think, these nuanced things in mind. So what I'm hearing from you, Shrey, interestingly, is that if humans on the team are doing very close data analysis, error analysis, dogfooding it like crazy, and essentially they are the human evals, and you're describing that as that's within the umbrella of evals. So you could do it that way if you have time and motivation to do that, or you could set these things up to be automatic. Absolutely. It's also about the skills, right? People who work at Anthropic are very, very highly skilled. They've been trained in data analysis or software engineering or AI and whatnot, right?

1:15:54Yeah. You know, you can get there. Anyone can get there, of course, by like learning the concepts. But most people don't have that skill right now. Dogfooding is a dangerous one only because a lot of people will say they're dogfooding. Like, yeah, we dogfood it. But are they really? And a lot of people aren't really dogfooding it at that visceral level that you would need to to have to close that feedback loop. So that's the only caveat I would add. There's also this kind of feels like straw man argument of evals versus A-B tests. Talk about your thoughts there, because that feels like a big part of this debate people are having.

1:16:33Like, do you need evals if you have A-B tests that are testing production level metrics? So A-B tests are, again, another form of evals, I imagine. Right. Like when you're doing an A-B test, you have two different experimental conditions and then you have a metric that quantifies the success of something and you're comparing the metric. And again, right, an eval in our mind is systematic measurement of quality, some metric. You can't really do an A-B test without the eval to compare. So maybe maybe we just have a different weird take on it. Yeah. OK. So what I'm hearing is like you consider A-B tests as part of the suite of evals that you do.

1:17:11I think when people think A-B test, it's like we're changing something in the product. We're going to see if this improves some metric we care about. Is that enough? Why do we need to test every little feature? Like if it's impacting a metric we care about as a business, we have a bunch of A-B tests that are just constantly running. This is now a great point. So I think a lot of people prematurely do A-B tests because they've never done any error analysis in the first place. They just have hypothetically come up with their product requirements and they believe that we should test these things.

1:17:45But it turns out when you get into the data, as Hamill showed, the errors that you're seeing are not what you thought what the errors might be. There were these weird handoff issues or, I don't know, the text message thing was strange. So I would say that if you're going to do A-B tests and they are powered by actual error analysis, as we've shown today, then that's great. go do it. But if you're just going to do them, which we find that people try to do, just trying to do them based on like what you hypothetically think is what is important, then I would encourage people to go and like rethink that and kind of ground your hypotheses.

1:18:23Do you have thoughts on what Statsig is going to do at OpenAI? Is there anything there that's interesting? Just like that was a big deal, a huge acquisition, A-B test company. People are like, oh, AV tests, the future. Thoughts? You know, just to add to the previous question a little bit is why is there this debate, A-B testing versus evals? I think fundamentally, evals is people are trying to wrap their head around what, how to improve their applications. and fundamentally you need to do data science. Data science is useful in products, like looking at data, doing data analytics. There's many different suite of tools and you don't need to invent anything new.

1:19:07Sure, you don't need like necessarily the whole breadth of data science and it looks slightly different, just slightly with LLMs. You know, your tactics might be different. And so really what it is, is like using analytic tools to understand your product. Now, people say the word evals trying to kind of like carve out this new thing and saying, you know, evals and then A-B testing. But if you zoom out, it's the same data science as before. And I think that's what's causing the confusion is, hey, we need data science thinking. And AI products just, you know, it's helpful to have that thinking in AI products like it is in any product is my take on that.

1:19:49Yeah, that's a really good take. Like, I think just the word evals triggers people now. And if you just call it, we're just doing error analysis, doing data science to understand where our product breaks and just setting up tests to make sure we know. That's boring. Sounds boring. No, no, no. We need a mysterious term like evals to really get the momentum going. Your question about Statsig, I think it's very exciting. To be honest, I don't know much about it because, you know, I just imagine that they're this company that there's a tool that many people use. And maybe it just so happened that OpenAI acquired them.

1:20:23I'm sure they've been using them in the past. I'm sure OpenAI's competitors are using Statsig as well. So maybe there is something strategic in that acquisition. I have no idea. I don't know anything there. But I think those are really the bigger questions for me than, you know, is this fundamentally changing AV testing or making evals more of a priority? I think they've always been a priority. I think OpenAI has always been doing some form of them. And OpenAI has gone so far, historically speaking, as to like go and look at all the Twitter sentiment and try to do some sort of retrospective on that and then tie that back to their products.

1:21:01Like they're certainly they're doing some amount of evals before they ship their new foundation models. But they're going so much beyond and being like, OK, let's find all the tweets that are complaining about it, all the Reddit threads that are complaining about it, that go try to like figure out what's going on. So it goes to show that like evals are very, very important. No one has really figured it out yet. People are using all the available sources signal that they can to improve their products. What I will say is I'm really hopeful that it might shift or create a focus within OpenAI, hopefully.

1:21:35Up until now, a lot of the big labs, understandably, have focused on general benchmarks like MMLU score, human eval, things like that, which are very important for foundation models. and you know those not very related to product specific evals like the ones we talked about today but like handoff and stuff like that like those you know they tend not to correlate yeah they don't correlate with math problem solving sorry to say exactly and so um you know if you look at the eval products let's say the ones up until recently that some of the big labs have they don't have error analysis they have a suite of generic tools cosine similarity a hallucination score whatever and that doesn't work it's a good first stab at it it's okay you know at least you're doing something getting people maybe it's like getting people look at data but uh eventually what we hope to see is okay a bit more data science thinking in this eval process, which hopefully the tools will get to.

1:22:44Hamel and I should not be the only two people on the planet that are promoting a structured way of thinking about application-specific evals. It's mind-boggling to me. Why are we the only two people doing this? The whole world. What's wrong? So I hope that we're not the only people and that more people catch on. The fact that your course on Maven is the number one highest grossing course in Maven. Clearly there's demand and interest and there's more people, I think, on your side. Interestingly, just an example you've been sharing on Twitter that I think is informative. Everyone's been saying how Cloud Code doesn't care about evals.

1:23:21They're all about vibes and everyone's like, and they're the best coding agent out there. So clearly this is right. More recently, there's all this talk about Codex, OpenAI Codex being better and everyone's switching and they're so pro evals. I know. Yeah. so gets me every time the internet's so inconsistent my favorite thing was um like yesterday i believe like a couple of lab mates and i were out getting like dessert or something and somebody said like oh um do you like codex or claude better or whatever and the other person said oh i like claude and then someone else said but the new version of codex is better and then the first person said oh but the last i checked was two days ago so maybe my the thoughts maybe i'm not up to date and i was like oh my god this is the world we live in oh my god okay so i want to ask about just top misconceptions people have with evals and top tips and tricks for being successful so maybe just share one or two each of each so let me just start with misconceptions and maybe i'll go to hamel first just what are a couple of the most common misconceptions people have with eval still.

1:24:30The top one is, hey, I can just buy a tool, plug it in, and it'll do the eval for you. Why do I have to worry about this? We live in the age of AI. Can't the AI just eval it? That's the most common misconception. And people want that so much that people do sell it, but it doesn't work. So that's the first one. Shoot. Manitum is still great i think that's great news the second one that you know i see a lot is hey um just not looking at the data you know so in my consulting people come to me with problems all the time and the first thing i'll say is let's go look at your traces and you can see the kind of their eyes pop open and be like, what do you mean?

1:25:26Like, yeah, let's look at it right now. And they're surprised that I am going, I'm going to go look at individual traces. And we always, it always 100 % of the time, learn a lot and figure out what the problem is. And so I think people just don't know how powerful looking at the data is like we showed on this podcast. I would agree with that. Those are the top two? Okay. Is there anything else? or those are the ones to solve those problems? Oh, those are definitely... And then I guess the one I would add is there's no one correct way to do evals. There are many incorrect ways of doing evals, but there are also many correct ways of doing it.

1:26:09And you've got to think about where you are at with your product, how much resources you have, and figure out the plan that works best for you. It'll always involve some form of error analysis as we showed today, but how you operationalize those metrics is going to change based on where you're at. Amazing. Okay. What are a couple just tips and tricks you want to leave people with as they start on their eval journey or just try to get better at something they're already doing? So tip number one is just don't be alarmed or don't, you know, be scared of looking at your data. The process, we try to make it as structured as possible.

1:26:49There are inevitably questions that are going to come up. That's totally fine. You might feel like you're not doing it perfectly. That's also fine. The goal is not to do evals perfectly. It's to actionably improve your product. And we guarantee you, no matter what you do, you're doing parts of these processes, you're going to find ways of actionable improvement. And then you're going to iterate on your own process from there. The other tip that I would say is we're very pro AI. Use LLMs to help you organize any thoughts that you have throughout this entire process. So this could be everything ranging from like initial product requirements, right?

1:27:28Figure out how to organize them for yourself, figure out how to improve on that product requirements doc based on the open codes that you've created. Don't be afraid to use AI in ways that present information better for you. Sweet. Don't be scared. Use LMs as much as you can throughout the process. But not to replace yourself. Okay, great. Still jobs. Great. Hamill. Let me actually share my screen when I show something. To piggyback off what Shreya said is if you heard any phrase in this podcast, you've probably heard look at your data more than anything else. And so it's so important that we teach that you should create your own tools to make it as easy as possible.

1:28:17So I showed you some tools when we're going through the live example of like how to annotate data. Most of the people I work with, they realize how important this is and they vibe code their own tools. We shouldn't say vibe code. They make their own tools. And it's cheaper than ever before because you have AI that can help you. And AI is really good at creating simple web applications that can show you data that can write to a database. It's very simple. And so for the Nurture Boss use case, we wanted to remove all the friction of looking at data. And so what you see here is just some screenshots of what the application that they created looks like.

1:29:05It's just, okay, they have the different channels, voice, email, text. They have the different threads. They hid the system prompt by default. Little quality of life improvements. And then they actually have this axial coding part here where you can see, okay, in red the count of different errors. They automated that part in a nice way. and they just they created this within a few hours um and so it's really hard to have a one size fits all thing for looking at your data you don't have to go here immediately but something to think about is make it as easy as possible because again it's the most powerful activity that you can engage in it's the highest roi activity you can engage in and so um you know with AI, yeah, just remove all the friction.

1:29:55That's amazing. And again, I think the ROI piece is so important. We haven't even touched on this enough. The goal here is to make your product better, which will make your business more successful. Like this isn't just a little exercise to catch bugs and things like that. Like this is the way to make AI products better because the experience is how users interact with your AI. Absolutely. If any, you know, we teach our students, hey, when you're doing these evals, if you see something that's wrong, Just go fix it. Like the whole point is not to have evals, a beautiful eval suite where you can point at it and say, oh, look at my evals.

1:30:30No, just fix your application, make it better. Do, you know, if it's obvious, do it. So totally agree with you. Amazing. How long, a question I didn't ask, but this is, I think, something people are thinking about. How long do you spend on this? Like how long does it usually take to do the first time? I can answer for myself for applications that I work with. Usually I'll spend three to four days really working with whoever to do initial rounds of error analysis, like a lot of labeling, feel like we're in a good place to create the spreadsheet that Hamill had and everyone's kind of on board and convinced and even like a few LLM judge evaluators.

1:31:05But this is a one-time cost. Once I figured out how to integrate that in unit tests or I have like a script that automatically runs it on samples and I will create a cron job to just do this every week, I would say it's like, I don't know. I find myself probably spending more time looking at data because I'm just data hungry like that. I'm so curious. I'm like, I've gained so much from this process. And it's like put me above and beyond in any of my collaborations with folks. So I want to keep doing it, but I don't have to. I would say like maybe 30 minutes a week after that. So it's a week essentially up front and then like 30 minutes to keep improving and adding to your suite.

1:31:46Yeah, it's really not that much time. I think people just get overwhelmed by how much time they spend up front and then thinking that they have to keep doing this all the time. Amazing. Is there anything else that you wanted to share or leave listeners with? Anything else you want to kind of double down on as a point before we get to our very exciting lightning round? So I would say this process is a lot of fun, actually. So it's like, okay, you're looking at data. Oh, it sounds like you're annotating things. Okay. Actually, so I was just looking at a client's data yesterday. The same exact process.

1:32:22It's an application that sends emails, recruiting emails, to try to get candidates to apply for a job. And we decided to start looking at traces. We jumped right into it. hey let's look at your traces the we looked at a trace the first thing i saw was this like email that is worded like given your background blah blah blah blah so i asked the person right away and this is where putting your product hat on and just being critical and this is where the fun part is i said you know what i hate this email like do you like the email given your background when and when i receive a message given your background comma i just delete that so i'm like what is this given your background with machine learning and blah blah i'm like this is a generic thing like so i asked the person like hey you know can we do better than this like this is kind of like a this is like a sounds like generic recruiting and they're like oh yeah maybe yeah like it's the ai because they were like they were proud of it they're like the ai is doing the right thing is sending this email with the right information with the right link with the right name everything And so that's where the fun part is, is like put your product hat on and get into like, is this really good?

1:33:38Something I want to make sure we cover before we get to a very exciting light around is this is just scratching the surface of all the things you need to know to do this well. I think this is the best primer I've ever seen on how to do this well. Nice. I think we did it. But you guys teach a course that goes much, much deeper for people that really want to get good at this and take this really seriously. share what else you teach in the course that we didn't cover and what else you get as a student being part of the course you teach on maven yeah i can talk about the syllabus a little bit and then hamill can talk about all the perks um so we go through a life cycle of error analysis then automated evaluators then how to improve your application like how do you create that flywheel for yourself we also have a few special topics that we find like pretty much no one has ever heard of or taught before, which is exciting.

1:34:28One is how do you build your own interfaces for error analysis? So we kind of go through actual interfaces that we've built, and we also live code them on the spot for new data. And we show kind of how we use cloud code, cursor, whatever we're feeling in the moment that day to build these interfaces. And we also talk about kind of broadly cost optimization as well. So a couple of people that I've worked with, they got to a point where their evals are very good, their product is very good, but it's all very expensive because they're using state-of-the-art models. So how can we replace certain uses of the most expensive GPT-5 models with 5nano, 4mini, whatnot, and save a lot lot of money, but still maintain the same quality.

1:35:17So we also give some tips for that. Hamill, you want? We also have many perks. Yeah, talk about the perks. Okay, the perks. So my favorite perk is there's a 160-page book that's meticulously written that we've created that walks through the entire process in detail of how to do evals that supplement the course. So you don't have to sit there and take all these notes. We've done all the hard work for you and we have documented it in detail, you know, and organized things. So that is really useful. Another really interesting thing and something that I got the idea from you, Lenny, is, okay, this is an AI course.

1:36:02education shouldn't be this thing where you you're only watching lectures and doing homework assignments so students should have access to an ai that also helps them so what we have done is we've uh you know just like there's the lennybot that you have dot com yeah lennybot.com uh we have made the same thing with the same software that you're using and we have put everything we've ever said about evals into that. So every single lesson, every office hours, every discord chat, any blogs, papers, anything that we've ever said publicly and within our course, we've put it in there and we've tested it with a bunch of students and they've said it's helpful.

1:36:49So we're giving all students 10 months free unlimited access to that alongside the course. Amazing. And then you'll charge for that later down the road. I have no idea. I just take one month at a time. I don't know what we're doing after that. Eight months and then we'll have to figure it out. I was thinking this whole interview should have just been our bots talking to each other. That's amazing. I would watch that. Only for like 10 minutes, then I don't know what they're talking about. Yeah, maybe 30 seconds. Did you guys trade it on the voice mode, by the way? That's my favorite feature of Delphi's product.

1:37:21If not, you should do that. I think I'm... I can't remember. I should look at it. Definitely should. Now that we have this podcast episode you could use this uh content to train it it's 11 laps powered it's so good uh okay so that's how do they get to i guess that's okay they get to that once they become a uh then to your course so there's no sign up for the course and then you'll get a bunch of emails everything will be clear hopefully amazing oh we also have a discord of all the students who have ever taken the class and that discord is so active i can't go on vacation without getting notified on the plane.

1:37:55Bittersweet, bittersweet. Incredible. Okay. With that, we've reached our very exciting lightning round. I've got five questions for you. Are you ready? Yes, let's go. Let's do it. Okay. So I'm going to bounce between you two, share something if you want. You can pass if you want. First question, Shreya, what are two or three books that you find yourself recommending most to other people? I like to recommend a fiction book because life is about more than evals so recently i read pachinko by uh benjen lee a really great book and then i also am currently reading apple in china which the name of the author is slipping my mind but this is kind of more of the exposition written by journalists on how apple did a lot of manufacturing processes in Asia over the last couple, several decades.

1:38:48Very eye-opening. Hamill. Yeah, I have them right here. So I'm the nerd. Okay, so I'm not as cool as Shreya is. So I actually have like textbooks, which are like my favorite. So this one is very classic one, Machine Learning by Mitchell. Now it's kind of theoretical, but the thing I like about it is it really drives home the fact that Occam's razor is prevalent, not only in like science, but also in machine learning and AI. So a lot of times the simplest, and also engineering. So like a lot of times the simpler approach generalizes better. And so that's the thing I kind of internalized deeply from that book.

1:39:32And I also really like this one. So another textbook, I told you I'm a nerd. This is like also a very old one. Wow. And this is like, you know, norweg algorithms and it's just like i really like it because it's just human ingenuity and it's very like lots of clever useful things they're down the street i'm in berkeley the people that did that research yeah yeah textbook authors super cool oh man nerds i love it okay next question favorite recent movie or tv show i'll jump to hamel first okay so i'm a dad of two parents i have two parents i don't get to oh sorry uh two kids so yeah i'm a dad of two kids and i don't really get the time to watch any tv or movies so i watch whatever my kids are watching so i've watched frozen like three times in the last week only three okay in the last week okay yeah so that's my great that's my hamel frozen i love it okay sure yeah yeah i don't have kids so I can give all these amazing answers.

1:40:36Actually, so my husband and I have been watching The Wire recently. We never actually saw it growing up. So we started watching it and it's great. I feel like everyone goes through that. Eventually in their life, they decide, I will watch The Wire. I know. So we are in that right now. A year of your life. It's great. It's such a great show. Oh, man. But it's so many episodes and everyone's an hour long. I know. I know. We get through like two or three a week. So we're very slow. Worth it. Okay. Next question. Do you have a favorite product you recently discovered that you really love? And we'll start with Shreya.

1:41:10Yeah, I really like using Cursor. Honestly, now Cloud Code. I'll say why. So I think I'm a researcher more so than anything else. I write papers, I write code, I build systems, everything. and I find that like a tool, I'm so bullish on AI-assisted coding because like I have to wear a lot of hats all the time and now I can be more ambitious with the things that I build and write papers about. So I'm super excited about those. Cursor was my entry point into this but I'm starting to find myself always trying to keep up with all these AI-assisted coding tools. Hamill. Yeah, I really like Cloud Code and I like it because I feel like the UX is outstanding.

1:41:54There's a lot of love that went into that. It's just really impressive as a terminal application that is that nice. How ironic that you two both love clock code when it's just built on vibes. I think it's false. It's not just built on vibes. There we go. Okay, two more questions. Hamill, do you have a favorite life motto that you find yourself using and coming back to you in work or in life. Keep learning and think like a beginner. Beautiful. Shreya. I like that. For me, it's to always try to think about the other side's argument. I find myself sometimes just encountering arguments on the internet, like this recent evals debates, and really think, okay, put myself in their shoes.

1:42:42There's probably a generous take, generous interpretation. And I think we're all much stronger together than if we start picking fights. My vision for evals is not that Hamel and I become billionaires. It is that everyone can build AI products and we're all on the same page. Slash everyone becomes billionaires. Yes. Amazing. Final question. When I have two guests on, I always like to ask this question and I'll start with Hamel. What's something about Shreya that you like most? What do you like most about Shreya and I'm going to ask her the same question in reverse. Yeah, Shreya is one of the wisest people that I know, especially for being so young relative to me.

1:43:25I feel like she's like much wiser than I am, honestly, seriously. She's very grounded and has like a very even perspective on things. And so I'm just really impressed by that all the time. Shreya. Yeah, my favorite thing about Hamill is his energy. I don't know anybody who consistently maintains momentum and energy like Hamill does. I often think that I would start caring much less about evals, if not for Hamill. And everyone needs a Hamill in their life, for sure. well we all have a hamel in our life now uh this was incredible this was everything i'd hoped it'd be i feel like this is the most interesting in-depth uh consumable uh primer on evals that i've ever seen i'm really thankful you two made time for this uh two final questions where can folks find you where can they find the course and how can listeners be useful to you i'll start with Shreya?

1:44:29Yeah, you can reach me via email. It's on my website. If you Google my name, that is the easiest way to get to my website. You can find the course. If you Google AI evals for engineers and product managers or just AI evals course, you'll find it. We'll send some links hopefully after this so it's easy. And how to be helpful? Two things always for me. One is ask me questions when you have them, I will try to get to them, respond as soon as I can. The other one is tell us your successes. One of the things that keeps us going is somebody tells us what they implemented or what they did, a real case study.

1:45:08And Hamill and I get so excited from these. And it really keeps us going. So please share. Yeah, it's pretty easy to find me. My website is hamill.dev. I'll give you the link you can find me on social media LinkedIn, Twitter thing that's most helpful is to echo what Shreya said we would be delighted if we're not the only people teaching evals we would love other people to teach evals and so any kind of blog posts writing especially that as you go through this and learn this that you want to share we would be delighted to help reshare that or amplify that. Amazing. Very generous. Thank you two so much for being here.

1:45:57I really appreciate it. And you guys have a lot going on. So thank you. Thanks, Lenny, for having us and for all the compliments. My pleasure. Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny's podcast.com. See you in the next episode.