The Challenge of AI Model Evaluations with Ankur Goyal

0:00Evaluations are critical for assessing the quality, performance, and effectiveness of software during development. Common evaluation methods include code reviews and automated testing, and automation. identify bugs, ensure compliance with requirements, and measure software reliability. However, evaluating LLMs presents unique challenges due to their complexity, versatility, and potential for unpredictable behavior. Ankur Goyal is the CEO and founder of Braintrust Data, which provides an end-to-end platform for AI application development and has a focus on making LLM development robust and iterative.

0:40Ankur previously founded Empira, which was acquired by Figma, and he later ran the AI team at Figma. Ankur joins the show to talk about brain trust and the unique challenges of developing evaluations in a non-deterministic context. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

1:15Anchor, welcome to the show. Thank you so much for having me. Yeah, absolutely. Thanks for being here. So let's talk about BrainTrust. How did you guys get started? What was the original inspiration? Yeah, so prior to BrainTrust, I used to lead the AI team at Figma. And before that, I started a company called Impira, which Figma acquired. And at Impira, we were in like the stone ages of AI pre-ChatGPT and built a product that helped you extract data from documents. And basically, every time we would change something, like whether it was changing our models or when we started using language models, like the prompts or even the code that fed stuff into the models or process the stuff that came out, we would break something.

1:55Like, for example, we had banking customers and we might improve our invoice processing model by like 2 % or something, but break a banking workflow in the process. And obviously that can't happen. So we had to figure out how to avoid that. And we ended up building internal tooling that helped us do evals. And then Figma acquired us. We basically had the same set of problems with LLMs and built roughly the same tooling. So after doing that a couple times, I was chatting with Elad Gil, who's one of our investors. And he was like, you know, hey, you've built the same thing twice. Maybe other people have this problem too.

2:29So we talked to a bunch of companies, including the folks at Notion, Zapier, Airtable, Instacart, and a bunch of other companies who are now customers. And they were like, yeah, we do have this problem and we need a solution. So we partnered with a bunch of really great companies early in our journey and built a product. And we've just been kind of going since. Why do you think that no one had put a product offering out there for this type of problem? Is it too niche and bespoke? Or is it the classic software engineering thing where people were building and rolling their own auth for a really long time before companies came along and offered that as a service?

3:08Yeah, I think a lot of this stuff is timing. So had we not done this, I think someone else would have. Now I think we've kind of set the standard for how to do evals really well as a software engineering team and kind of built the primary workflow that people are using or in some cases copying into their product. But I think the reason that something like this didn't exist is that chat GPT slash GPT-3 in particular represented a fundamental engineering paradigm shift in AI. So prior to GPT-3 being available over at REST API and accessible with simple natural language, it was really hard as a software engineer to actually use AI models.

3:52I've been trying for a long time. I'm not a stats PhD by background. I am kind of like a traditional old school software engineer. And I struggled through using ML models for a really long time. And it just became dramatically easier when that happened. And so I think that paradigm shift was the first time that software engineers were actually able to use AI models in an effective way. And yet AI engineering and ML engineering is a totally different discipline than traditional software engineering. So it's kind of like an old or well-known workflow for doing evals, but a new group of people that are trying to do it with different preferences and skills that they bring to the table.

4:34And so that vacuum is basically what created the opportunity for brain trust. Yeah, so essentially you took this thing that was probably always a problem, but before it was a little bit more narrow in terms of the number of people that were maybe facing that problem. And now because of, as you mentioned, you can essentially talk to a large language model or some other generative AI model through an API endpoint. Suddenly the scale of that issue becomes much, much larger. So why is the problem of running something like an eval test more challenging when you're talking about interacting with a model versus traditional ways that we might do this for software engineering?

5:10Yeah, I think the biggest difference is that it's non-deterministic. So in traditional testing, you want your test to be 100 % green. And if something's flaky, then it's usually actually a bug. I was debugging something this morning where one of our services would occasionally return the wrong result. That's a bug. On the other hand, in AI, that is something that you deal with literally every day. And no AI model is going to be perfectly deterministic. I think that's kind of the interesting opportunity for engineering with AI. And so being able to visualize, interpret, characterize non-deterministic results is a different paradigm.

5:47Many people, myself included, start by trying to kind of cram this into the traditional, you know, like pie test or jest or VI tests of the world. And it gets very confusing very quickly, you start to have to do stuff like, okay, I'm going to try running this thing like four times. And if it succeeds three out of the four times, then maybe it's good enough. But why three, you know, why not two? Or why not run it 10 times? And then of course, when you're actually looking at the results, you want to, for example, see all four things that you may have tried in one place and try to see what the variance is.

6:19You can't really do that on a terminal screen. I mean, there's no UI, you know, around something like vi test that makes that easy so i think that's the biggest problem the second thing is that you're not just testing based on the code so unit tests are like a pure function of code or sometimes you know infra you also need to test based on data and being able to source good data to build good evals is challenging but once you embrace the challenge i think the art of finding the right data to actually evaluate on becomes the highest leverage tool that you have when you're building AI software. And so that is a completely new activity.

6:56And it's this kind of strange thing that requires reaching into your production logs, finding interesting examples, and then utilizing them in your evals. A lot of the teams that we meet before using Braintrust, they're doing this just in like JSON-L files. So they'll log stuff to a Postgres database or log it in their traditional observability system, and then find stuff and then like click in a UI, download it, and then copy paste it into a JSON-L file, and then try to use that in their evals. And that is obviously not the optimal workflow. And I think the third thing that's interesting is that compared to previous generations of software engineering, the role that non-technical people have is quite different in AI.

7:34So non-technical folks, for example, support people are often incredibly sharp at looking at poor user interactions with an LLM and characterizing why the interaction was bad. And that is an incredibly good way to get good eval data. And so you need to figure out ways to incorporate non-technical folks into the workflow and utilize them effectively. And again, that's different than traditional unit testing. And then in terms of doing this for sort of like, you know, what we consider old school predictive AI versus generated AI, are there differences in terms of how you need to do something like evals for those?

8:08Yeah, I think there's a lot of differences. I was talking to someone at a ride sharing company earlier today, and I think a good example is, let's say you work at Uber and you're trying to figure out what the optimal price is for a ride. That's like a traditional ML problem. Really, I think optimizing an aggregate is the right thing to do. So if one or two writers, things are too cheap, or something is too expensive, you know, Airbnb is another similar example where they use ML to figure out the price or to suggest prices for hosts to list their properties. With those kinds of problems, if the answer isn't quite right, then the user impact is somewhat low.

8:47And I think optimizing over the aggregate of the correct price is much more important than every single price. Whereas a lot of the problems that people are using for AI, for example, if you're working on like a code gen related tool, if you spit out the wrong code, then, you know, it's just not a good user experience. It's not like you need to optimize some aggregate number there. And so I think the cost of being wrong is somewhat different. And accordingly, I think it's very important to look at individual examples, and not just try to optimize an aggregate, which is a really popular sort of old school academic thing to do in AI.

9:20So that's one thing that's quite a bit different. I think the other thing that's quite different is that LLMs are very interactive. So in the old world of ML, like when we were training models at Impira, if we noticed that something was wrong, we would sort of collect a bunch of data and then retrain everything. And you could do that maybe if you're very, very sophisticated once a day. But realistically, many companies that are building traditional models will retrain their models like once a quarter. Whereas with LLMs, because you can do prompt engineering, you can change things really, really quickly.

9:50One of the most popular features of brain trust is that you can go into our logs. And if there's an LLM interaction, you can hit try prompt and open it directly in a modal, and then actually play around with the prompt and even save it right there. And so that kind of very fluid engineering is quite different. I think it feels to me quite a bit like the difference between using Python and using something like C++, or even writing assembly code, you can just move a lot faster. Yeah, there's essentially the sort of feedback loop is much more immediate than you have with sort of the predictive models where you're doing an update maybe once a quarter or something like that.

10:24Like you can immediately make a change and see the change and the impact of that change, which is really powerful. But it's also because of the non-deterministic nature, hard to always know if the change is actually better or you just sort of like the vibes feel better and you're in that direction. Yeah, no, I mean, I think it feels exactly like building software. I mean, I'm old enough to remember when building web servers and writing web apps was pre-PHP. It was just really, really slow and really hard. And at that point, it was inconceivable to me that you'd be able to save a file in your IDE and without even refreshing the page, the browser would reflect the change, which is now something that everyone experiences and honestly, we all take for granted.

11:08So I think the sort of power of software is really how quickly you can iterate things. And LLMs have really unlocked that for AI. Yeah, that's absolutely true. And I think that's true even outside of the technology itself when it comes to what companies succeed and what companies fail, especially in competitive markets. It's like, how fast can the company learn and make adjustments? Because inevitably, the things that you do are probably going to be wrong in some fashion, but can you learn from that, iterate really quickly? That's why it's engineering organizations. You want things like CICD set up and be able to do multiple releases a day because that execution cycle allows you to essentially out-compete people who don't have those types of things in place.

11:49Yeah. I mean, I'll give you an example. Simon, who's one of the founders of Notion and one of our early adopters at Braintrust said that prior to Braintrust, they were able to solve on the order of three issues per day. And now with Braintrust, they're able to solve more than 30 issues per day. And so you're exactly right. I think with the sort of analog to CI, CD, observability, et cetera, in AI, you're just able to move a lot faster. So can you walk me through the process of actually creating an eval? Like I want to use brain trust. I have some AI application that I'm building and now I want to integrate this.

12:23Like how do I go about doing that? Yeah, I think one of the things that we did that really was very popular in the early days, and I think has kind of now become a standard among products and tools in the space is we broke an eval down into just three simple parts. So one part is the data and data is, you know, it's just a list of inputs and then optionally expected ground truth values. You don't always have them, but sometimes you do. So you have to figure out how to get that. Sometimes, you know, if you're just starting, you might just hard code it in a TypeScript file or Python file. Maybe you have it in a SQL database.

12:58Braintrust has data sets you can use, but somehow or another, you provide some data. And then you provide a task function. A task function is very simple. It takes some input, and then it generates some output. And a simple task function could just be a single prompt that you plug the input into the prompt, and then you generate the output, and you save it. It could be agent, it could be, you know, multiple agents. Now it could be something that runs across services. So, you know, that can get increasingly complex, but it's just a pointer into your application. And then the last thing that you provide is scoring functions.

13:32And scoring functions take the generated output, and then the expected value if one exists, the input, maybe some additional metadata. And their job is very simple. They produce a number between zero and one. We have an open source library called auto evals, which has a bunch of scoring functions built into it. Some of them are heuristics like Levenstein distance, which is kind of a good old trick that still works. Some of them are very fancy LLM-based scorers. They're all open source, so you can actually look at the prompts and tweak them yourself. But the scoring functions, you kind of itemize them into these little functions whose job it is just to assess your output on some criteria.

14:11And that's it. You just plug those three components into an eval, and then you can run Braintrust eval on your code in Python, TypeScript, or now a bunch of other languages. And that's it. you've run an eval. How did you come to this design? How did you know that this was going to work? Well, I mean, I've been doing evals as a software engineer struggling through ML for seven years now. And so this is not the first attempt at trying to simplify this. I think at Impira, I remember the first time I tried doing it, I was working with our researchers who are unbelievably smart. And they had like Python notebooks and matplotlib and for loops and matrices.

14:52And I like barely understand NumPy, you know, so it was just really, really complicated. And I think through literally years of iteration, I sort of realized that it's just these three pieces. And I think I spent when we started working on BrainTrust, probably like two or three months thinking about how to really boil down evals into something that was very easy for people to use. Brian, who's the CTO of Zapier, was very helpful because he was also new to AI and a very, very sharp software engineer. So I would sort of send him a draft and say like, hey, is this like sufficiently easy for you to digest?

15:28And he would just say no. And so working with him and a few others, I think we kind of arrived on this design. And I remember when I sort of first wrote an eval this way, it just felt right. Yeah. So it sounds like you had some really good early stage design partners that helped you validate what you were planning before you actually went about implementing it and rolling out. Yeah. Our core bet was that there are some early adopters. And Ilad and I actually wrote down a list of these companies before we really started the company. But there's a list of early adopters that were building software in a way that represented how others would build software with LLMs in the future.

16:07And I think one of the most interesting characteristics of these companies is that most of them did not have ML teams prior to ChatGPT coming out. So they were kind of starting from a fresh slate and thinking from first principles about how to build with AI. And we basically reached out to all of these companies. And I think now almost all or all of the companies on that list are customers. But we sort of bet on what Zapier, Notion, Airtable, Google, Instacart, browser company, Ramp, companies like this, what they would be doing and how they would be building AI software, we sort of assumed that others would build it that way as well.

16:45And I think that's largely turned out to be true. Yeah. I mean, that's a really, I think, fantastic approach and insight that you had early to sort of... You ended up identifying what is probably your ICP, your ideal customer profile. And then it's kind of like, this is your initial account-based selling strategy. Here's our account list. I mean, let's go, now we close those, let's go look for these other ones that are kind of similar. Yeah, I think it's, yeah, you know, what Notion was doing six months ago, I think a lot of companies are trying to do now. And so it's definitely worked really well for us.

17:18That's awesome. And then in terms of the eval itself, like, what am I doing as a company or engineering organization and wants to incorporate this? Is this a library that I'm running, you know, locally? Like where's this kind of run and sit? Yeah, I think it's a lot like unit testing in the sense that it usually starts by you just running it on your local environment. And when you do that, you basically generate a bunch of logs, which get upload to Braintrust automatically. And then you go to our UI to visualize it. And we've tried to make the DX, you know, really, really fast. So the sort of time from when you run an eval to when you see it in the UI is just, you know, a handful of seconds.

17:56And it feels very real time. And that's usually how it starts. And then the next thing that you would do is like send your friend or your colleague a link to the eval and say, hey, look at this. What do you think? Or I discovered this, you know, what should we do? And then maybe your colleague starts running them as well. And soon you kind of realize, hey, it would be really great if we could actually run these on all of our PRs. And we built a GitHub Actions integration that's been really popular that just makes it very easy to kind of translate what you probably have already built locally into running as part of your PR workflow.

18:28And we do a bunch of stuff behind the scenes. Like, if you don't change any of the prompts, then we'll automatically cache everything. And if everything is automatically cached, we don't pollute your PR with a bunch of repetitive information about the thing that's changed. You know this very well, but there's a lot of those like little things that you have to do to make the workflow feel right. But that's usually how it goes. And then once it's in your PR workflow, then you start actually baselining against main. And every time you run an eval, it gets automatically compared to the latest deployed version.

19:01And you can get increasingly more sophisticated from there. And then in terms of any increased inference costs that I might be incurring with this, where I'm going to be running these PRs through evals that might also, one, they're probably interacting with some AI component of my actual product, which is going to have a cost associated with it. And then it also might, in the sort of scoring function, be using some sort of LL-based scoring. How do companies think about those types of things? Well, first of all, it's all cached. So both for speed and cost reasons, I think it's quite useful in evals not to just unnecessarily rerun evals.

19:42And by the way, there's another cost that you didn't even mention yet, which is doing online evals. So we also make it really easy for you to run eval functions on your logs at a sampling rate that you specify. And that also adds cost. But I think the other thing is that I think the cost of doing evals is really low relative to the cost of running a production workload. And yet the value is disproportionately high because evals, for every eval that you run, you're basically ensuring that end users have a really positive experience with your product. And so in the spirit of iteration and getting to the best possible product and product market fit with your AI company or your AI feature as soon as possible.

20:24I think evals end up feeling like a really low cost way to get there compared to users suffering, for example, with your low quality app. So to be honest with you, I think, asides from some competitors or companies trying to create noise about how evals are expensive, it hasn't come up as a practical concern for any of our customers. Yeah, I guess you're really weighing in against like, what is the cost of a horrible user experience? Right. And it's a fraction of the cost of actually running your application, right? And how does this like start to work when things start to get more complicated?

21:00Like if I have some sort of like agentic workflow where there's going to be multiple like sort of planning evaluation cycles, maybe I even have like an agent workflow where multiple agents are communicating and passing information. Am I breaking these down sort of piece by piece and running evals a little bit more task specific, or am I doing some things that's a little bit more aggregate? Yeah, no, it's a great question. And I think at a certain point, how you engineer your evals becomes a core part of how you actually build the agent or the more complex system itself. And I think what I'll say really quickly there is the best systems are often the systems that can be evaluated really well.

21:40And so you sometimes pick the abstraction boundaries in the software that actually allow you to do evals. But yeah, I think the optimal way to build increasingly complex AI systems is to do evals end to end, as well as for the components. And the more you evaluate individual components, like for example, a planner module, the more reusable and modular it is. So you can use it as its own standalone thing and evaluate it as its own standalone thing, and then somewhat reliably and comfortably plug it into larger systems and kind of know that it will do its task really well. And as a business, how do you turn something like essentially evals and testing into an actual business?

22:20Clearly, there's pain and people want to solve it. But how does this become something that you can actually monetize and scale? Yeah, I mean, one of the things I'll share just as a fun anecdote to other entrepreneurs who are potentially starting companies. When we started BrainTrust, we weren't the only company that was thinking about building LLM tooling. But I think we were the only company that really focused on evals. And the reason is that a lot of VCs will give you advice about problem spaces. And multiple VCs told us, you know, CI, CD was not the most lucrative set of venture outcomes for them in the previous generation of software.

22:58And so it's a very bad place to start a company. And instead, you should focus on things like observability. And I think we just knew better than to really listen to VCs that much. and from personal experience knew that the pain was really around evals. I think now the mindset around that has shifted quite a bit. And in retrospect, I think we were right to focus on evals. But with that context in mind, I think monetizing brain trust has not been that big of a challenge for us because evals represent such a critical part of the development workflow and represent so much pain that it's a problem that people feel very motivated to solve.

23:36The other thing about evals, and I mentioned this kind of earlier on, is that the data component, like how you actually source data to do evals, turns out to be really important. And therefore, people actually want to log their production workloads in Braintrust as well. And so at this point, we've built a really, really powerful and seamless integration across logging and evaluation. situation. And as soon as people start logging stuff in Braintrust for the purpose of finding data to do evals, they start asking us for other stuff as well. Like, hey, I'm logging stuff here. Can you tell me how much I'm spending over time?

24:14Or can you tell me how much I'm spending per project? Or can you help me understand when my app is really slow? Or can you help me understand when users are liking or not liking the experience that they have with the product? And so that's kind of naturally expanded what we do to be more than sort of just evals. But yeah, I mean, I think people are willing to pay for tools that help them move quickly. And therefore, I think it hasn't been a really big blocker for us to actually monetize the product. How do you go about like writing evals for really like open ended tasks? Like if really the application is, you know, customer support or something that's like chat based, like, how do I go about like, writing evals there where I don't really know necessarily what the inputs are going to be?