Launching AI products with Braintrust’s CEO Ankur Goyal

0:05So today on No Priors, we have Ankur Goyal, the co-founder and CEO of Braintrust. Ankur was previously vice president of engineering at SingleStore and was the founder and CEO of Empira, an AI company acquired by Figma. Braintrust is an end-to-end enterprise platform for building AI applications. They help companies like Notion, Airtable, Instacart, Zapier, Vercel, and many more with evals, observability, and prompt development for their AI products. And Braintrust just raised$36 million from Andreessen Horowitz and others. Ankur, thank you so much for joining us today on Nobriers. Very excited to be here.

0:38Can you tell us a little bit more about Braintrust, what the product does, and we could talk a little bit about how you got started in this area and AI more generally. Yeah, for sure. So I have been working on AI since what one might now think of as ancient history. Back in 2017, when we started working on Impira, things were totally different. But still, it was really hard to ship products that work. And so we built tooling internally as we developed our AI products to help us evaluate things, collect real user data, use it to do better evals and so on. Fast forward a few years, Figma acquired us and we actually ended up having exactly the same problems and building pretty much the same tooling.

1:24And I thought that was interesting for a few reasons, some of which you pointed out, by the way, when we were hanging out and chatting about stuff. But one, Empyro was kind of pre-LLM. My time at Figma was post-LLM. But these problems are the same. And I think there's some longevity that's implied by that. Problems that existed pre-LLM probably are going to exist in LLM land for a while. And the second thing is that having built the same tooling essentially twice, it was clear that there was a pretty consistent need. And so I have very fond memories of the two of us hanging out and talking to a bunch of folks like Brian and Mike at Zapier and Simon at Notion and many others.

2:06And I've been in a lot of user interviews over time. I've never seen anything resonate like the early ideas around brain trust and really everyone's desire to have a good solution to the eval problem. So we got to work and built, honestly, a pretty crappy initial prototype. But people started using it. And, you know, brain trust just over a year later has now kind of iterated from people's feedback and, you know, complaints and ideas into something I think that's really powerful. And yeah, that's how we kind of got started. Well, yeah, I remember in the early conversations we had around the company or the idea, I should say, it was meant to even potentially be open source.

2:50And as the first time that I was involved with some sort of customer call and people would say, we don't want you to open source it, which I found really surprising. Like people really pushed on, we want this to exist for a long time. We want to be able to pay for it. And so there was that kind of really interesting market pull. Why do you think there was so much interest or need for this or demand for it? What does Braintrust do and how does that really impact your customers? Many of our early customers had built internal versions of Braintrust before we engaged with them. And there's a couple of things that sort of came out of that.

3:23One is it helped them gain an appreciation for how hard the problem is. Evals sound really easy. Oh, it's just a for loop. And then I look at console.log the for loop as I go and I look at the results. But the reality is, the faster you can eval, the faster you can look at eval results, which start to get really complicated as you start doing things with agents and so on, the faster you can actually iterate and build stuff. It is actually a pretty hard problem to do evals well. And many of our early customers, who were kind of like the pioneers in AI engineering, had learned that the hard way.

3:57And I think the other problem is that folks, especially folks like Brian, for example, they saw that AI would be a pervasive technology throughout the whole org, not just a project that Brian might babysit and work on with one team. And having a really consistent and standardized way of doing things was really important. And I remember early on, Brian pointed me to the Vercel docs. And he said, one of the things I love about this is that when new engineers are building UI now, they read these docs and they kind of learn the right way to build web applications. And you have that opportunity with AI.

4:37And I found that actually really motivating and really influenced how we think about things. It makes a lot of sense. I guess like if you're swapping out, you know, GPT for Claude or you're making a change in model or you're changing a prompt, then it just helps you really understand how that propagates and what sets of outcomes for users are better, what sets are worse and kind of troubleshoot them. And then it feels like you've built a whole other series of products around that that really helps support that. One of the biggest things when you're building AI products is this uncertainty about quality.

5:08So you might, for example, get really excited about a feature, build a prototype. It works on a few examples. You ship it to some users and you realize it actually doesn't work very well. And it's just really hard to go from that prototype into something that systematically works in an excellent way. And I think what we have helped companies do is basically like demystify that process. So instead of having a bunch of anxiety about, hey, I shipped something, I don't know if I'm ever going to get it to be able to work well, you can implement some evals in brain trust and then sort of turn the crank and get really, really good outputs.

5:44You know, you work with a lot of the companies that I feel are the earliest adopters of AI into their own products. In other words, they've actually shipped products with AI in them and they're sort of that first wave. It's Notion, Airtable, you know, Zapier, people like that, Vercel. What proportion of your customers do you think are adopting some of the things that people are talking about a lot? And so that would be things like fine tuning or rag or building agents. Like, do you think that's a very common set of things or do you think that's just kind of hype? And because I think you have a very clear picture of at least one segment of the enterprise market in terms of what people are actually doing.

6:17Unambiguously, people are doing rag. So that one is, you know, it's like simple and obvious. probably around 50 % of the use cases that we see in production involve rag of some sort. Fine tuning is interesting. I think a lot of people think of fine tuning as an outcome, but it's actually really a technique. And the outcome that people are looking for is automatic optimization of their workloads. Fine tuning is one way of doing that. And it is a very, very difficult way of automatically optimizing your use case. I think we, with our customers, have re-benchmarked fine-tuning on their workloads, I would say, every two to three months.

7:04And there was a period of time when GPT 3.5 fine-tuning came out before GPT 4 was easy to execute. Now it's extremely cheap actually to run GPT 4.0. But there was this kind of period where it was really hard to have GPT 4.0 access. And GPT 3.5 fine tuning was a way of, it's like the only lever for some use cases to improve quality. But since then, honestly, I think almost if not all of our customers have moved off of fine tune models onto instruction tune models and are seeing really good performance. We even talked about that early on. I remember when we were thinking about brain trust, we thought like, oh boy, you know, everyone's going to need to use this to fine tune models.

7:51And that was one of the first features we were thinking about building. And, you know, no one is, no one's really doing it. Could you explain just for the listeners, like the difference between instruction tuning and fine tuning? Yeah. I mean, I think it's kind of like the difference between writing Python code and creating an FPGA or something. So with instruction tuning, all you do is modify the prompt to include examples of how it should behave. In some ways, it's actually very similar to fine tuning. You're collecting data that guides how the model should behave, and then you're feeding it into a process that kind of nudges the model towards behaving that way.

8:31Fine tuning is a much lower level thing where you're actually like modifying or supplementing the weights in a model so that it learns from those examples. And because it's so much lower level, it tends to be a lot slower, more expensive. There's a lot of ways you can injure the model while you're fine-tuning and actually make it worse on real-world use cases. And so it's just a lot tougher to get right. And then do you see a lot of open source adoption or mainly people using proprietary models and are there other early technologies that you see people adopting right now? We are very close to a watershed moment for open source models.

9:09Like we saw at the watershed moment for Anthropic when Cloud 3 came out, especially, you know, Cloud 3.5, Sonnet has really taken off. We are very close to that, I think, with Llama 3.1, but we're not there yet. So we see very limited practical adoption of open source models, but I think more interest than ever. And I think a lot of what you're seeing is also just things that are in production, right? And so to some extent, there's a lot of discussion in the developer communities around what people are using and adopting and playing with. And then I think you're really focused on the market of enterprises that are shipping AI products.

9:45And, you know, obviously it can be used by hackers and developers as well, but a lot of your usage as well is people who have things in production. And so it kind of reflects the state of the world for live systems at scale. I am a developer and I love open source software. And I have a very difficult time with the fact that every time I use an open AI model, I'm paying a fee per token. But then I actually look at the numbers. And of course, I've looked at them with our customers too. And in some cases, it's just negligibly cheap. And in the cases where it's pretty expensive, the ROI is actually really high.

10:19And so most of our customers are really, really focused on providing the best possible user experience for their customers and the fastest iteration speed for their developers. And everything else is secondary. So I think until open source can really move the needle on one of those two axes, it's going to be tough for it to be adopted broadly. The other place you spend a lot of your career is on sort of databases and data infrastructure and things like that. So you had the BP engineering at Single Store, which I think was renowned for really having an exceptional database-centric team. How do you think about the data infrastructure that exists for the AI world today?

10:56What's needed? What's lacking? What works well? What doesn't work? The shift is that people have hoarded lots and lots of semi-useful data in data warehouses. Prior to LLMs, there was actually this whole industry around AI where companies like Data Robot, for example, would come in and help you train models based on these proprietary structured data that you've collected in your super proprietary data warehouse. And I think the big insight or the crazy non-intuitive thing about LLMs is that something trained on the internet outperforms what an enterprise can produce with their own data trained on data in a data warehouse.

11:43And I think not only is the nature of the data processing problem different, but the value of data is actually, and how we think about the value of data is very, very different. Like just hoarding data about your claims history or transaction history, it might not actually be that useful. The real question is like, how do you construct a model which is really good at reasoning about the problems that you're working on? And I think the way that enterprises will collect data and leverage it into these AI processes does not look like doing ETL on a data warehouse that's running in Amazon or something like that.

12:24I think it's going to totally change. And I've seen a lot of the data that gets stored in Braintrust through people's logs, it actually never makes it to a data warehouse. And people, they just don't really care about that because if they put it in a data warehouse, what are they going to do with it? What do you think is missing from a data infrastructure perspective? So I think, you know, to your point, there's a couple of different steps or some sort of data cleaning step. There's some storage layer there. You know, there's there's different forms of labeling, et cetera. How do you think all these pieces kind of evolve over the next couple of years?

12:58And then I guess related to that, the other topic people have been talking a lot about is synthetic data and how important that will be in the future. I'm sort of curious your views on these different areas. Purely from a data standpoint, it's important to think about what you're going to do with the data and then how the infrastructure enables that. So, you know, Data Warehouse is really designed for ad hoc exploration on structured data, which is it's just neither of those two things is relevant in AI land. You're dealing with lots and lots of text and you're not exploring it ad hoc using SQL queries.

13:30What we see actually as kind of what the most advanced companies are doing is actually using embeddings and models themselves to help them sift through tons and tons of data and find, for example, customer support tickets, which are not well represented in the data that they're using for their evals or not well represented in their fine tuning data sets and, you know, trying to find those examples and use them. So I think the workload is going to shift. And I actually think LLMs and specifically embeddings are going to be core to how people actually query data, not traditional algebraic relational indexes.

14:11That's going to be a huge shift. And there's this huge debate about vector databases and will traditional databases do vector database things. I think that debate's kind of silly. I think relational databases are perfectly capable of adding HNSW indices to them. What will really be disrupted is the OLAP workload. So relational, you can't just slap semantic search and stuff into the architecture of a traditional data warehouse. I think that is actually a much deeper set of things that will need to change than the OLTP workload. This is your, in some sense, third startup experience, right? You joined MemSQL slash SingleStore quite early.

14:50You started in Pyro, which Figma acquired. You're now doing BrainTrust. What are the common things that you've taken with you as you've done this new startup? Like, what are the things that you've implemented early? What are the things that you've avoided? You know, one of the things that I honestly took for granted at MemSQL, but we've kind of re-implemented at BrainTrust is having a really hard technical interview. You know, MemSQL, maybe we pushed it a little bit too far, but it was really known for really strong technical excellence. And I think our interview reflected that. So that was actually one of the first things that we did.

15:24Manu and I spent probably like two or three days working through a bunch of really, really hard interview questions. And I think it's just important that you hold the technical bar really high and try to find people that are attracted to it. Actually, for example, if you do a front-end interview at Braintrust, one of the questions involves writing some C++. And we lose a lot of candidates because of that question. But it's a good signal that maybe Braintrust isn't the right place for you to work because we do like to hire people who are willing to jump around in areas of the stack that they're unfamiliar with.

16:01So I think that's one of the biggest things that we've carried over. Another thing that I think we did really well at both Impira and MemSQL is have an obsessive relationship with our customers and just really, really focus on making them successful. It's sometimes really hard to prioritize customer feedback and think about, you know, 10 customers are asking for 10 different things. What do I do? So what we've done at Braintrust is actually be very deliberate about which customers we prioritize, especially early on. and sort of hypothesized that the Zapiers and Notions and so on of the world would have pretty similar use cases.

16:43And so if you focus on these kinds of customers, then when they ask for stuff, you can pretty readily assume that other similar customers are going to have the same problem. And that's allowed us to be very, very customer-centric while building a product that repeats itself for more customers. And now what we're seeing is that the next wave of companies that are building with AI, both startups and more traditional enterprises, they actually want to be engineering things like the products that they admire, most of which use brain trust. And so a lot of those best practices are now built into the product and kind of the next batch of companies is able to consume them right out of the box.

17:19Yeah, it's kind of interesting. I feel like even early on as companies were first adopting LLMs for actual live products, they would all follow kind of the same startup journey, or I should say technical journey, right? Initially, they'd look into, at least back then, they'd look into fine tuning or some open source model or something else. They'd eventually realize they should just be using GPT-4, which was the primary model at the time. And then they'd go through this big loop of starting to build internal tools and then realize that really their focus should be on product. And it was the exact same journey.

17:51And I remember in their early brain trust customer conversations, you'd talk to them and they'd say, oh, we don't need this. And then three months later, they'd call and say, okay, we really need this. And it was always roughly the same timeframe. Are you seeing any common patterns today in terms of, okay, companies that are now a year or 18 months into their journey using LLMs, like they always have the same thing come up? There's a couple things. So one is companies that are fairly deep into their journey. They have like one or two North Star products that are pretty mature and they're trying to figure out how to get those products to the next stage.

18:22Probably the most consistent thing I've seen is companies kind of walking back from the illusion that totally free form agents will solve all of their problems. So I think maybe like two or three months ago, many of the pioneering companies went way down the agent rabbit hole. And they kind of realized like, wow, this is actually not the right approach. It's so hard to control performance. The error rates are really high and they compound really quickly. And so most of those companies have kind of walked back and tried to build a different architecture where the control flow is actually managed deterministically by their code.

19:05But they make LLM calls kind of like throughout the entire architecture of the product. And so that's probably the biggest thing that we're seeing now is I don't know if there's a good term for it yet. but maybe this kind of pervasive AI engineering throughout a product rather than trying to shove everything into the, you know, while loop of an agent. Yeah. The other thing that I've heard you talk about in the past is the evolving role of what an AI team does at a company. And so I think if you go back a couple of years, people were doing machine learning and they'd hire a big MLOps team. And then the types of things that they'd be doing day to day were very different from what they do today in the context of adopting AI.

19:47And even how you think about the role and who to hire maybe has shifted a bit. Could you talk a little bit about what you've used as the evolution of the role of the data science team, the data team, the ML or AI team, et cetera? Yeah, I think what's really interesting is many of the early adopters of LLMs didn't have any ML staff when ChatGPT came out. What is it now? Almost two years ago. And those companies were able to move really quickly because they kind of started with a fresh slate. many of the smart folks that I know that are classical machine learning people or data scientists have now come around.

20:23But actually, there was this big sort of resistance among them early on that LLMs are, they're not good at the things that we're trying to solve, or maybe it's a scam or something like that. Do you think that was just like a different problem set in terms of traditional ML and the applications of it are different from what JNI can do? Or do you think it was something else? Well, I went through this myself, watching the technology that we built to do document extraction at Impira become, you know, totally irrelevant. And I personally, I think it's an emotional thing. Like you, you try GPT-3 for the first time.

20:56And first of all, you know, back then at least it was kind of snarky. And so that was a little bit irritating. And it, it was also just way better at everything than anything you could possibly train. And I think that is so fundamentally disruptive to a lot of companies, a lot of people's individual identity. It just is not easy to wrap your head around if you've been doing AI and ML for a while. So I think it was largely an emotional thing. You could argue that there is a cost, security, privacy, whatever element of it. But the companies that were sort of on the leading edge, they were able to figure that out pretty quickly.

21:36You know, now I think more companies have come along the journey and I've seen a lot of really smart ML and data science people embrace LLMs and bring a lot of the sort of rigor that is still relevant around evals and measurement and, you know, prototyping and so on and become these like AI platform teams. Usually it's a combination of people with product engineering backgrounds and a few folks with statistics or data science backgrounds. And they start by building kind of like a marquee product for the company. And then they evolve into a platform team that enables like the N plus first project to be really successful.

22:15We see a lot of these teams forming as AI becomes more pervasive. So if you were to enterprise company right now and you were to try and adopt AI or LLMs, like who would you have to hire or what sort of capabilities would you move over into sort of this platform team? I would start with a group of really smart product engineers, because the first thing you need to ask yourself is what parts of my product or whatever I'm offering can be cannibalized or completely changed by modern AI. Product engineers are generally the best people to think about that. You can get really far with a really good UI and very basic AI engineering that sort of proves out a concept.

23:01I think we've seen a number of good examples of that. I know, for example, vZero is a truly incredible piece of engineering at this point, both from an AI standpoint and also from a UI standpoint. But early on, it was pretty simple. And that's the right way to start. And then I think as you find product market fit, it's sort of the right time to think about more rigor, think about fine tuning. Maybe we should use open source models for cost or whatever. Although I think not many people are far along that journey. I think you said something like TypeScript is a language of AI and Python is a language of machine learning.

23:40Yeah. Could you extrapolate more on that? First of all, a vast majority of our customers use TypeScript. And early on, some of our customers were dealing with, should we use TypeScript or Python? And some teams were using TypeScript, some teams were using Python. Now, almost everyone, including people that used to write Python primarily, is using TypeScript. And I think that's going to continue forward. There's a few reasons for that. One is TypeScript is the language of product engineering. And product engineers are the ones who are driving most of the AI innovation, at least in the world that we participate in.

24:15And so they're just literally pulling the AI ecosystem into their world. And that is driving a lot of TypeScript stuff. Another thing is that TypeScript as a language is inherently better suited for AI workloads because of the type system. So the type system basically allows you to launder the crazy stuff that comes out of an AI model into a well-defined structure that the rest of your software system can use. Python has a pretty immature type system. You know, they're improving and I always get trolled on Twitter when I post about this by people who make, you know, somewhat valid arguments. But TypeScript is just a much, much better language for writing software that deals with uncertain shapes of data.

25:04I think that's actually kind of its whole point. So I think it is actually literally a better suited language for working with AI. Have you seen any other shifts in terms of usage of specific languages or tooling or other things that's happened with this wave of AI? Yeah, I think the biggest thing I've seen over the past six months is people dropping the use of frameworks. So early on, I think people thought that AI is this really unique thing. And just like Ruby on Rails or whatever, we're going to need to build new kinds of applications with new kinds of frameworks to be able to build AI software.