Generative Benchmarking with Kelly Hong - #728

0:00If you're familiar with the retrieval space, MTAP is a pretty popular benchmark for embedding models. If you see the release of any new ing button model, you'll probably see something like, oh, our model achieved like this score on MTAP. We outperformed this other model on MTAP. And people would just kind of take that score and assume, okay, this model is better just because it did better on this benchmark. But there are so many problems with these public benchmarks.

0:35All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. Today, I'm joined by Kelly Hong. Kelly is a researcher at Chroma. Kelly, welcome to the podcast. Thank you so much for having me. I'm really looking forward to digging into our conversation. We're going to be talking about a project you recently published called Generative Benchmarking. That one really resonated with me for a few reasons. It touched on themes like synthetic generation data for evals, LLM as a judge, data leakage and overfitting on public benchmarks. So a lot of interesting themes touched on there.

1:14Before we dive in, I'd love to have you share a little bit about your background. Yeah, of course. So yeah, as you mentioned, I'm currently doing research at Chroma. We're a company working on vector databases, but we also work on research around retrieval as well. just to give a little bit more background about myself I started my undergrad at UC Berkeley around three years ago studying CS about two years and I got pretty interested in this space and like search and retrieval so then I spent some time like building projects around it going deeper into the research and I would come across Kermal a lot because like I would use it in my projects but I also saw that they did a lot of good research around like chunking strategies like embedding adapters, just ways to make your retrieval system better.

1:59So I got pretty interested in Chroma specifically, and then I started talking to the team, got this opportunity, and now I'm here. So I'm on a gap year right now from school and working here basically full-time. And Chroma does come up, particularly in the context of vector databases and RAG. And RAG continues to be a topic of interest for folks listening to the podcast and the folks that I talked to in enterprises. So I think this will be an interesting conversation in that light as well. Talk a little bit about the origin of the project. Was it already framed up when you got there? Or did you take some time to kind of scope it out?

2:42Yeah, yeah. Happy to talk about it. It's something that the team has been thinking for a while. And I've personally been thinking about like on my own as well. It just so happened that like we were pretty aligned on the same problems. But when I was like building out my own projects, like one thing that would constantly come up is how difficult it is to like systematically evaluate AI systems. Like if you think about this analogy with like software systems, it's very easy to evaluate them. You have these like very simple code tests, you like run them and then you get like a deterministic output.

3:12But if you have like AI systems, all of your outputs are like probabilistic. You don't really know what it's going to output. So then like how do you exactly like evaluate them? I think initially when I started building out in the space, I would just evaluate based off of Vibes, which is really bad. You know, like an output would either feel good or it didn't look good. So then I might change the prompt a little bit, maybe change out my embedding model. But there was really good, like no good way to systematically evaluate it. So I kind of felt the personal frustration there. And that's something that Chroma aimed to target for a while as well.

3:44Like we wanted people to be able to systematically debug their AI systems. So that's kind of where generative benchmarking came about. One of the main motivations for that was to address this problem of people not being able to evaluate their AI systems in an easy way. If you think about evals as a whole, it's not a very friendly topic to a developer who's just getting started with RAG. They have to go pretty deep into learning about how evals work, what a benchmark is. And it's a lot of work that people don't really spend a lot of time on. but it's very important if you actually want to improve the performance of your AI system.

4:20So yeah, one of the core motivations was to just create a really easy way for people to evaluate their AI systems and understand the importance of it. And I think another motivation for that was to address the problems of a lot of current benchmarking methods. If you're familiar with the retrieval space, MTub is a pretty popular benchmark for embedding models. If you see the release of any new in button model, you'll probably see something like, oh, our model achieved like this score on MTAP. We outperformed this other model on MTAP. And people would just kind of take that score and assume, okay, this model is better just because it did better on this benchmark.

4:59But there are so many problems with these public benchmarks. I can go into these in like a bit more detail as well, but just to give a high level overview, they're very generic. So they don't reflect like the domain-specific use cases of what people see in production. The data sets are also very clean and polished, whereas in the real world, we have very messy data. And a lot of this data has probably been seen by embedding models before. So you don't really know if you're testing for like true retrieval or if they're just, you know, doing this based off of memorization. So just kind of like going off of, you know, all these problems of people not really knowing how to evaluate their systems.

5:38And also like the problems with like a lot of like public benchmarks kind of inspired this idea of generative benchmarking. And you mentioned that vibe checking is the predominant way that developers are kind of assessing the performance of their systems. There is a lot of, I think, kind of a broad call for folks to take a more rigorous approach to evaluation in the space. How do you see folks evolving from Vive coding to a more rigorous or scientific process to evaluating the performance of their AI apps or RAG apps? What are the steps that folks typically take? The first thing that people do is they probably look into, you know, benchmarks like MTEP and they kind of see scores of that.

6:28And then they make the decision on like which embedding model to use. but I think like with these methods of like generative benchmarking what we'll hopefully start to see is like people kind of like taking advantage of these tools that make evaluation like really easy like you don't really need to know like the details of like the entire evaluation pipeline like the generative benchmarking like notebook that we set out is just a very simple notebook you just upload your data you fill in like the context and then you run a few cells in a notebook it makes the process very very easy so I think like hopefully the trend we'll start to see as people start using these kinds of tools that make evals very approachable.

7:05And I hope that from that, people get more familiar with the idea of doing evals and also understand the importance of it as well. And I think from that, people can get more into the details of maybe aligning their LLM judge or adding more human-in-the-loop processes. But yeah, I think initially, we'll probably see people start to favor more of these approachable eval solutions. Yeah, yeah. So you don't see generative benchmarking as kind of an advanced stage that you evolve to. It's meant to be kind of an easy thing that you can adopt pretty early on. Yeah, for sure. It's kind of like helping people with that first step if they don't have a golden data set to start with.

7:49Because for a lot of retrieval systems, people probably don't like log their queries when they're getting started. They just have their like set of documents, right? For the retrieval system. so they don't really have like a set of like query document pairs to test their system on so we kind of help with like uh starting the process of like generating um that golden data set so people can start to evaluate but the eventual goal there is for people to like continue iterating on that data set you know for example if they have like incoming user queries hopefully they use that to align their golden data set even further maybe like improve their reachable system like overall based on that um but yeah i think this sort of more serves as like like a starting step rather than like the end state of evaluation.

8:32Yeah, what I've typically seen in my own projects and in speaking with folks is like you mentioned, you start with kind of vibe checks, you put some prompts in, see what comes out. And then after you've done that for a while, like you collect those and now you have a short list of, you know, evals that you can kind of repeat if you want to tweak it, change a different model or some parameter. And it sounds like generative benchmarking might be the next step. Or historically, I think of the next step is like doing synthetic generation. Synthetic generation is a part of the overall generative benchmarking process.

9:14But the way you incorporate synthetic data generation is specifically tailored as part of this process. So this might be a good time for you to kind of talk through the overall generative benchmarking process. At the high level, generative benchmarking is essentially generating a custom eval set based on your own data. And we do this for Retrieval specifically. So you can kind of think of it as like, okay, user inputs, a set of documents gives us some context on what their application is. And then from that, we generate queries to test your Retrieval system on. And this is broken down into two main steps.

9:51we first do document filtering, and then we do the actual query generation. So I can kind of go through these two steps in a bit more detail. Before you do that, why that two-step process? As opposed to, I've got a bunch of documents, give them to an LLM and say, hey, generate a bunch of queries, and now this becomes my generated test set because I've got relationships between those documents and the queries. Yeah, so we have this two-step process of generative benchmarking. And we do this because we realize the document filtering step, which is the first step in our process, is actually very important because usually a lot of the time there are cases where a few documents aren't very relevant to your use case.

10:33For example, we were working with data from Waste and Biases, their technical support bot. So it's basically like this chatbot where you can talk to Waste and Biases documentation. We noticed there were a few documents in there that were talking about like startup funding news or just content that wasn't relevant to any kind of developer using this technical chatbot. So we first want to filter out those documents because they're not reflective of what users will actually ask about. and kind of the main goal with generative benchmarking is to create a benchmark that's representative of your true use case.

11:05So that means, you know, we want to filter out content that users wouldn't ask about in the first place. So we find that document filtering is a pretty like critical step in this. Like we kind of compared this to like a naive query generation approach where we did no document quality filtering, no like query string anything and then compared it with that, which I can talk about it like in a bit more detail later on. But yeah, that's kind of like the core motivation for breaking this down into two main steps. Like we want to make sure that we're selecting for content that users will realistically ask about and also generate queries in a way that's realistic and reflects like queries that you'll see in production.

11:42And so is that filtering, do you see the filtering step as in and of itself kind of a core differentiation for this approach? Or is it, you know, one of several? I think that's definitely one of the core differentiators. I say that and the way that we are generating queries. Because I think like these two parts combined kind of like make up what a realistic user query is. Like first you have like the content, like you want to be asking about content that would actually be relevant for a user that's using your application. And also the style that you ask the query in. Like for example, a lot of users using a technical chatbot, like the one that Waste and Biasis has, they probably wouldn't be asking like queries in complete like sentences.

12:25is. It's more of like statements, which you typically see in a lot of like chatbot applications. Like people are very vague with their queries. It's not like super comprehensive. So we want to reflect that as well. So I think these two steps like go hand in hand. A big part of the two steps is injecting the context into both the filtering and the query generation. What is that context? Yeah, the context is basically like what your ROG application is for. So for example, in the case of Waste and Biasis, like the context would be something like, this is a technical support bot for Waste and Biasis.

13:00A lot of the users would be people who are, you know, maybe training models or like using these kinds of like platforms. So just giving some context to kind of telemodel like what to focus on. It's kind of like a system prompt. Yeah, yeah, kind of like that. I was thinking about this process and its applicability to my favorite use case, which is I have a bunch of podcast transcripts and I want to create a rag chatbot for those. And I'm evaluating this as like, oh, how would this help me? And one question that I had was, do you think it's applicable to that use case? Because I'm not sure that filtering is, filtering could be interesting at the chunk level.

13:50It's not necessarily interesting at the document level. There's some chunks, probably a lot of chunks that don't add a lot of information and aren't reflective of user queries. But the document corpus as a whole, those are all relevant documents. I guess just to clarify, like we're both doing like chunk filtering and document filtering. So we would filter for chunks as well, because I think in a lot of cases, like even if a document is relevant, like, for example, this case of like podcast transcripts, all documents are relevant. but they're probably chunks that wouldn't be good to generate a query from.

14:23Like, for example, say you have like the end of a podcast where you're saying like goodbye or something. Goodbye, goodbye. Yeah, you probably don't want to retrieve that. And so we would filter out chunks like that because like what kind of question would you ask for a chunk that just says like goodbye, goodbye? So, yes, I think like in this case, it would be more focused on like filtering at the chunk level. You mentioned filtering and that second step is query generation. What do you do different with the query generation part? Yeah, for query generation, it's pretty simple. We first give it like the same context that you would give to like document filtering.

15:01So this would just be like, okay, this is a technical support bot and then et cetera. And we also give it some example queries as well. And that's mainly for the purposes of kind of steering the LLM to generate like queries in a realistic style. So like I mentioned before, a lot of people who use these chop out applications, the queries are very big. They're not like formed as like a complete question with like a question mark and everything. So we want to be able to reflect that as well. So, yeah, we incorporated both like the context and some example queries. Is the key distinction around kind of the sample queries relative to what you see users doing when they're using LLMs to generate test queries?

15:45Or is it more distinct from what you see in kind of the big public benchmarks? Are you asking about like how the example queries differ from like public benchmarks? One of the points you referenced earlier is how with the public benchmarks, the queries aren't representative of actual user queries. Users are, you know, they take shortcuts. They're not always grammatically correct. The public, you know, queries are more structured in some ways. and I'm asking if that is the key thing that you're trying to, if kind of the real user nature of the query is the key thing that you're trying to model or are you trying in this step to correct for the way you see developers or engineers like manually creating sample queries to give to an LLM to generate more queries?

16:50We're, yeah, we're trying to model like what you would actually see in production, which is like pretty different from like the public data sets that I spoke about earlier. Because I think, yeah, like in these public data sets, like a lot of questions are very like well formed and polished. So they would contain a lot of like complete questions and they'd be highly relevant to the documents as well. but in like real production data which we got from like Waste and Bias is like these weren't queries that were you know like written by the Waste and Bias team there were queries that they actually took from like users who used their technical support bot so we know that it's reflective of what they will actually see in production so yeah we wanted to make the distinction between you know like what you actually see in production versus like what I don't know you was just seeing like a typical benchmark data set yeah and I get the distinction between the benchmark data set.

17:45I'm wondering if you mentioned that you pulled this information from the weights and biases, like a query log. And I'm wondering if you have a sense for if someone at weights and biases was like building out these queries manually, would they be different from what they would see in the test log? And the style of queries, I imagine it would be very similar. I think it's pretty easy once you see like a list of like actual queries from like users, you can like replicate them. But I think maybe one distinction there would be like the content that the queries focus on. Because when we're looking at the data, one interesting thing we noticed was that there were some queries that just cannot be answered by any documents in the corpus.

18:30And if you were just like a waste and biases, like engineer, like writing queries, then you wouldn't focus on queries that are not answerable by your document corpus. So I think that was like another interesting point that we saw. Like if you are able to work with like real production data, then you can also kind of use that to, you know, evaluate how well your document corpus reflects what your users are asking about. So I think like the style of queries will probably be very similar, but maybe the content, the content of queries that users ask about will be pretty different. Again, you filter your chunks, then you generate your queries.

19:05What's next? Yeah, after you generate your queries, you have your query document pairs. So this is how evaluation typically works for evaluating an embedding model. You have a query document pair, you embed your document corpus, you embed your queries, and then you see whether you retrieve that document in the top K results. That's how you get recall a K. So then we basically run that evaluation for our generated dataset. And then we compare it against the ground truth queries as well. that kind of serves us like our ground truth for kind of testing whether our generated queries actually reflect like what you would see in production.

19:40And those ground truth queries are the examples that were given or where did those come from? Oh yeah, so these were the actual queries that were logged from like their technical chatbot. And the waste and biases case, got it. Yes, for waste and biases. So it would be like the log queries and then we like manually labeled all the data and then use that to test for representativeness. But this was mainly for like our research report. But if you are maybe like a developer using this tool, you would just end up with that set of like query document pairs and then use that to evaluate your retrieval system.

20:12Because kind of like from our end, we kind of did the work of demonstrating that these are representative of what you'll see in production. So then users can just take that generated data set and use that for evals. And you mentioned that one of the things that having an eval system allows you to do is to swap out different embedding models, to play with chunking strategies, that kind of thing. A question that comes up frequently is how much does that stuff really matter? What's your take on that based on what you've seen? I think it matters a lot. I think we've seen some pretty distinct differences between embedding models.

20:53And I think one of the key things we saw when working with like production data was that, like we saw that Gina AI's model like performed slightly worse than like OpenAI's text embedding large model, which actually conflicted with like the MTEP scores that they released. So in spite of performing better on the public benchmarks, it performed worse on real world data. Yeah, so that kind of goes to show like, oh, like these performance on like public benchmarks don't necessarily translate to your specific use case. And yeah, we saw some pretty like clear differences between like the rankings of embedding models.

21:28We saw that Voyage's three large model performed best. So we'd recommend them to use something like that. And we didn't extensively test out chunking strategies or we have another report on that. And I think there are some other things you can do as well, like contextual rewriting. We noticed that a lot of chunks were missing context, which is a pretty common problem in a lot of these ROG applications. You can just chunk up documents, but some chunks might just not be very valuable if they don't have any context on where they came from. And trying to fix problems like those actually boosts retrieval performance a lot.

22:03Anthropic did a research report on that as well. So yeah, these strategies make a pretty big difference in retrieval performance. Is that typically done in your chunker, just asking an LLM to capture the context and prepend it to whatever the chunk is? I wouldn't say it's super common because it is pretty expensive to do. I mean, you're like passing it into an LLM every time. So it really depends on your use case. If you have like tens of thousands of chunks, probably doesn't make sense. You probably want like a more effective approach to maybe like selectively choose chunks to like rewrite. But yeah, I guess it really depends on the use case.

22:40Do you focus on a particular set of metrics in talking about this work? We mostly looked at recall at K just because there are other scores that are used in a lot of these retrieval research papers. People use NDCG, but one thing that has been brought up is that the ranking of chunks in the context of an LLM doesn't really matter. It doesn't affect the performance of an LLM if you have the relevant chunk in the beginning versus the end. So we don't look at NDCG. We just look at recall, which is like, was the relevant document retrieved or not? Presumably, if it's in there somewhere, the LLM will find the information it needs and create the right response.

23:25Yeah, yeah. So, yeah, I think like rather than looking at the rankings, it's more important to just see like whether it was retrieved or not. Are other metrics like groundedness and things like that, are those out of scope for this work? Yeah, we didn't look at groundedness. We purely just looked at like numerical metrics. I think like for groundedness, a lot of people use like LLM judges for that. But I think for that, you do need a good level of alignment, which I think could be interesting for maybe some future extensions of this. But yeah, we only looked at metrics for these. In the report, you talked about LLM as judge and alignedness.

24:08Can you elaborate on where that comes into play? Of course. So alignment is a very important part of this work. we specifically use this framework called Evalgen, which is basically like this, I guess, like framework for aligning an LLM judge to human preferences. Because if we just blindly use an LLM judge, it's not guaranteed to judge in the way that humans do. For example, when we were first using an LLM judge to filter documents, we compared that to like ground truth human labels. And I think it only got around like 46 % alignment, which is pretty bad. So we basically use this framework or like EvalGen to like iterate on our LLM criteria and get it up to like over 70%.

24:47So I think a mistake that a lot of people make is they just blindly use an LLM judge without any alignment, just assuming that it is aligned. But I think we're too far from that right now. And we do need some level of like human alignment to actually like confidently put the LLM judge to use. And to dig in on that, what you're doing there is you're asking the LLM to determine whether a chunk or document is relevant. And that's the filtering process. So basically, is it good to generate a query from or not based on this use case? Before we dig in on EvalGen, I'm curious how sensitive your results were to the prompting that you use for LLM as judge?

25:31I guess we noticed the prompting strategies through using EvalGen. I guess they both kind of go hand in hand because EvalGen is essentially evaluating how well does the prompting strategy allow your LLM judge to be aligned with your preferences. So yeah, we noticed it was pretty sensitive because we were just iterating on three criteria. And based on how we were wording those criteria or how we were maybe taking out one and adding in another, it would range quite a bit. So we sat on a report, it improved from 46 % to I think like somewhere like in the 70s. So that's a pretty big difference just based on like how you're prompting the LLM judge.

26:11So yes, we did notice that LLMs are very, very sensitive to prompting. Can you talk about EvalGen in a bit more detail, how that process works? Yeah, yeah, of course. So for EvalGen, you basically need a set of criteria. So in our example, we just use criteria like relevance. So like, is this relevant to our context? We use completeness, which is just how is the overall quality of this LLM judge. I think one other criteria as well. But yeah, basically, you would write out these criteria. And then you would also have ground truth labels for your documents. So we had 200 documents. And then we manually labeled them as either good or bad.