Harvey Co-Founder Gabe Pereyra on the Token Pricing Reckoning Coming for AI

0:00Gabe Pereyra:The models are getting more expensive and they're getting better. We're just seeing this huge explosion of usage, cost, tokens, everything. We need to talk about tokens. Winston mentioned you guys got up to$13 trillion. We are consuming a ton of tokens for just serving all this stuff. The big misconception right now is I don't think people realize how expensive this is going to get. And they're going to be like, what did my agent do that cost me$10 billion? We launched Lab, which is our legal agent benchmark. It's a benchmark for measuring the performance of agents on real-world legal tasks. We posted our first sort of initial results of how closed-source frontier models like those from OpenAI and Tropic Deepline perform on the benchmark.

0:49Gabe, welcome to Sorcery.

0:51Gabe Pereyra:Thanks so much for having me. Thanks for having us here. We're in the secret speakeasy back room. Exactly. Yeah, I see. So today we're going to walk through the latest release of your benchmarks, which is such a fun topic. Winston was so excited to talk about it, too. So I guess to start, let's break it down. What are you measuring with your benchmarks and where did we get to today? Yeah, so the thing we open sourced was our legal agent benchmark. and I think the thing that we're super excited about is we have done a bunch of benchmarking in the past so before Legal Agent Bench we had Big Law Bench which was more of a chat-based benchmark which was kind of a QA data set and we shared some of that we worked with kind of labs and the providers but at the time we're a smaller company and we didn't have the bandwidth to like fully open source it and support it and so what we wanted to do with Legal Agent Bench was fully open source this so anyone working on agents could run their agents and kind of work with us in the community to evaluate these things.

1:56Gabe Pereyra:And the way I would think of this data set is we tried to really mimic what the coding agent benchmarks did. And the way to think of the coding agent benchmarks is with things like SWE bench, terminal bench, you have a GitHub repo, which is kind of all the code for a project. You have an issue, which is someone said, hey, this doesn't work. Can you fix it? And then you have a bunch of unit tests that will tell you, you know, if you write a bunch of code and run it, the unit test will tell you, you know, you broke this thing or this thing passed. And the way the agent gets scored is, did it write all this code that passed all the unit tests?

2:34Gabe Pereyra:And I think there's a bit of a misconception that, oh, legal is subjective, so you can't do this. And I think the thing for especially big law is a lot of the work you can actually quantify and this is what senior associates and partners are doing when their junior associates do a bunch of this work and so as maybe an example of a task in the legal agent bench it would be can you do diligence which is you get a data room so you're trying to buy a company and you'll get all of that company's contracts and what the law firm needs to do is go through all the contracts and make sure that there's nothing you know questionable or that might blow up the deal in there and so the way we set up this data set like one of these tasks is you have a data room that's all the contracts you have a partner request which is a very high level instead of instructions you would get from a partner if you're an associate and usually for something like diligence it would be hey we got the data room can you look at it right and you would need to infer I know what I need to do based on that.

3:39And then I think part of the novelty here is we use both of those to

3:47Gabe Pereyra:create essentially legal unit tests. And so what are all the things that a partner would be looking for when they're trying to assess an associate's work product? And so you would take that data room and you would generate a diligence memo, which is what the law firm gives to their client to basically say here's all our findings on this diligence and the things you would look for. So one section is going to analyze all the change of control provisions, which is basically if I'm buying a company, there's going to be a change in control of the company. And many contracts have things that can be triggered.

4:21Gabe Pereyra:And most of them don't matter, but there can be ones where this important vendor contract will change in a material way and you need to flag this. And so that would be like a unit test of like you want to check that. And then the challenge with scaling this benchmark is that's just for diligence. And then as you scale this to every practice area that these large law firms help their clients in, you have to think about how to do this for fund formation and IP litigation and capital markets. And so a lot of what we're building is essentially this taxonomy of here's all the tasks that a law firm would do.

4:56Gabe Pereyra:And here's all the subtasks that associates would do and how partners or senior associates would grade that work. And I think what's nice is that framing is kind of the same thing you need to evaluate these agents to do these tasks. Benchmarking makes things a lot more competitive and people love that. I mean, AI is like the most competitive world ever right now. And so I'm really curious on that standpoint, like how are you then like, where does this lead to next? Yeah. So I think the interesting thing and kind of to your point is, okay, we released this benchmark and now we're going to have all of the labs and everyone else building models competing on it.

5:38Gabe Pereyra:And I think there's obviously some risk of, oh, what if some lab providers models are the best? Then why do they need Harvey to do this is maybe one of the potential risks. But what we've seen is actually every different model is good at something different. And so with the initial results we saw, Anthropics models are quite strong, but there's areas where 5.5 is better. There's some areas where open source is better. And increasingly, it's not just which model is the best, it's which model can solve the task at the lowest price point. Because as these models get better, there's kind of intelligent saturation where it's like, okay, for some simple task, I actually want to know that this open source model is good enough.

6:20Gabe Pereyra:And so a lot of the value that we're trying to provide to law firms is you can't just use one model to solve all these tasks because it's getting too expensive. So how do you think about all the tasks that your law firm does and which model you should use from which provider to solve those tasks? I know you've answered this before, but your research partners are your biggest competitors. OpenAI, Anthropic, NVIDIA, we have Base 10. There's like a handful of across categories, right? Right. So why open source it? Doesn't that pose a risk to you? Yeah, I think we think of all of these providers as there's obviously overlap with the labs and, you know, Anthropic and OpenAI have done kind of like Claude for Legal, Codex for Legal.

7:09Gabe Pereyra:But I think the problem that we're solving is different. I think a lot of the products that they're building are individual productivity and the products we're trying to build are organizational productivity for law firms or large in-house departments. And so when we think about open sourcing, I think there's kind of the initial point that I made of most of these law firms and enterprises can't rely on a single lab provider. So like there's a big risk if you're a law firm that if you just use Anthropic or you just use OpenAI, you run into conflict risk. So imagine you're using only Anthropics models as a law firm and you want to represent OpenAI.

7:52Gabe Pereyra:OpenAI is not going to let you send their sensitive legal data to Anthropics models. And so as a law firm, you need to support all the different models if you want to represent OpenAI and Google and Microsoft and all these providers. So there's the conflict risk, there's like platform risk. Like what happens if you pick one provider and they run out of compute, their models fall behind. And so you need to kind of abstract all these things away. And so we think of the providers similar to cloud. And I think you see the same thing in cloud with like Snowflake and Databricks and Datadog, like all these companies built on top of the cloud, but also compete with the cloud.

8:29Gabe Pereyra:So I think it'll look like that. And then I think the bigger point is we don't see the general legal intelligence as where the value of Harvey comes from because most of the valuable legal data can't go into the general purpose models. And so if you think of these large law firms, the work that they're doing is a very sensitive internal investigation, like mergers and acquisitions that would move the market and they can't go into public models. And so we think of our strategy as we want to open source the general stuff and work with all of the providers to make these models as good as possible at general legal.

9:09Gabe Pereyra:And then we want to build infrastructure for these law firms and enterprises that help them own their own models and build their own systems on their unique data. And that's where we think kind of our unique value comes in. And how are you training the models? You can't pull or pool data from your clients. So how are you learning off of them? Yeah, so the way I would think of the big challenge and where law firms are so different than most enterprises, like at Harvey, all of the data in our company for the most part is ours. At a law firm, a lot of the data is all of their clients and you can't mix this.

9:45Gabe Pereyra:And so a lot of what we're starting to work on with our clients is we have a product called Shared Spaces, which essentially lets a law firm and their client collaborate on a legal project. and so you can imagine a world where you have a large private equity firm and their law firm and they're doing all of their fund formation and all of their kind of M &A together in these shared workspaces. Where you can start training these models is how do you use that relationship to build these unique models and so a lot of what we want to figure out is given all of the like ethical wall constraints, regulatory constraints, client data constraints, how can we help law firms use their data in a way that respects kind of all of their client obligations, but still be able to improve their systems based on that.

10:32So speaking on your background, you were at Google Brain, DeepMind, and Meta. What were the biggest fundamental lessons that you learned from each of them that are driving the decisions you're making?

10:43Gabe Pereyra:Yeah, I think especially Google Brain and DeepMind when I was there, this was kind of in 2016, 2017. And so right when deep learning was starting to take off. And I think it was a really cool experience seeing both of those labs because they had kind of opposite strategies of how they approached research, which I think both were correct, but led to kind of different ways of executing on kind of the vision of where they thought AI was going. And so when I was at Brain, it was very bottoms up. It was, I mean, both had some of the smartest AI researchers in the world, but Brain, the approach was, you know, let's get all these smart people, give them a bunch of compute and then kind of let them do their own projects.

11:25Gabe Pereyra:And I think part of the outcome there is you had someone like Noam Shazir and Ashish Viswani and kind of the rest of the folks that invented transformers. And so that approach worked. And then DeepMind was much more top down where Demis just had this vision of, okay, we're going to create AGI. Here's all the things that I think are required. And so they kind of had this tech tree of how they were going to do it. And then let's have all these teams working on research projects that would solve all these milestones to get us to AGI. And I think our approach is a bit more inspired by the DeepMind one, just because I think because we're in a vertical, the end goal is very clear.

12:06Gabe Pereyra:Like we kind of know here's all the legal work that needs to get done. And so you can think about that North Star as, you know, here's all the practice areas that these law firms do. Here's how we would build systems that help them, and do large parts of those with AI. And so it feels much more defined and it's more of an applied problem than it is kind of a pure research problem. But I think seeing that approach really inspired like how I think about these things. How has your role technically evolved over time? I remember I was like, I was just listening to this interview or I forget what it was, but they were talking about Andre Carpathia now.

12:47Like even back at Tesla, like you'd think like these roles back then were like super glamorous and stuff, but he was data labeling all day. But you know, now he's the almighty God of like AI. So like, well, what is the biggest difference over time?

13:02Gabe Pereyra:Yeah, I think it's definitely been interesting at Harvey where it's like when we started the company, I was more focused on research than we should have been at that stage of the company where I kind of had a strong sense that what is happening now was going to happen. And so I wanted to figure out, you know, how do we train models? How do we scale this? But it became very clear pretty quickly, like, okay, first we need to build an enterprise business because, you know, we were a bit ahead of the curve of like, like when I first, when we first started the company, I was pitching people agents.

13:40Gabe Pereyra:And at that, like four years ago, people were like, what are you talking about? And so we quickly had to kind of like think about what is the right product now for customers that we can make for them that's useful, get revenue and scale up the GTM motion, kind of a traditional enterprise SaaS motion. And Winston and I always had kind of this vision of we're going to need to build two companies in parallel. Like one company is this traditional enterprise seat-based business. And then the second will be kind of the transition you're starting to see now as these things as these models get better things move to consumption and i think you'll need both of these because it takes a while to educate buyers on how to buy this like buying off tokens is insanely complicated um it'll probably at some point go back to something like subscription because it's hard to like forecast these things um but i think my role kind of shifted from when i started the company we're trying i was trying to do research and i was like okay we need to do traditional like build a great product build a great enterprise business and then now it's gone back to research where we've put all those things in place.

14:44Gabe Pereyra:I think a lot of the infrastructure has caught up, like de-agent stuff is happening now, that now I'm full-time. And a lot of it is like data labeling and how do we create good data sets, how do we work with all these partners, but I think it's kind of back to doing the things that I really enjoyed when I was at Brain, DeepMind, and Meta. And so I think now is kind of the time where it seems like all these application layer companies, that you saw Composer 2 with Cursor. Like everyone's starting to make this stuff work. And I think that's like super exciting. The inference layer has gotten really hot right now.

15:17What is your general sentiment about what's going on? How is that going to evolve?

15:21Gabe Pereyra:Yeah, I mean, I think it's tied to kind of what I was just saying is like, as this is starting to work in terms of open source models are catching up, I think a lot of the like performance from the models is coming, not just from, you know, doing this large pre-training, but all of this test time compute of, you know, how do you make these models better at reasoning? How do you build good kind of RL environments for them and good harnesses? And so the more that works, I think the more that kind of shifts the ability to application layer companies to do interesting things. And then for example, for us, we work with Base 10 and Fireworks and Together AI, Applied Compute, Trajectory, kind of a bunch of these companies, Ngram, that are doing either inference, RL as a service.

16:06And I think it just feels like the research is starting again, where there's just so many different avenues.

16:15Gabe Pereyra:All of these NeoLabs that we're working with are trying different ways. We're also working with the traditional model providers because they're also making a bunch of progress. And so I think it's just interesting as we're able to build more of these data sets. And even like we have a bunch of customers that want us to figure out better ways to train on their data. I think kind of all of these things are starting to take off and then I think the inference providers you're seeing them do very well where there's customers like us where we do need to move some of our traffic to open source models because it's just becoming super expensive to serve like the largest closed frontier model.

16:52Gabe Pereyra:And so I think it's like a mix of like performance. We have a lot of customers that want to own their own models, own their own data. And like, I think that's kind of also the infrastructure providers will serve that. And so, yeah, I think they're in a good spot. Sorcery is brought to you by Brex, the financial stack trusted by more than 30 ,000 companies, including one in three venture-backed startups in the U.S. Nearly 40 % of startups fail because they run out of cash. Rex is literally built to help founders avoid that. Unlike traditional banks that let your money sit idle, chipping away at it with fees, Rex is designed to help you spend smarter and move faster.

17:32Their all-in-one solution combines checking, treasury, and FDIC protection into one powerful account. You can send and receive money globally at lightning speeds, get 20 times the standard FDIC coverage through their partner banks, and even high yield from day one, with same day and even same hour liquidity. Access your funds anytime. Companies like Scale AI, DoorDash, Service Titan, HIMSS, Anthropic, Flexport, Robinhood, and Plaid trust and use Brex. Start today at brex.com slash sorcery. That's B-R-E-X dot com slash sorcery. Turing is training the next generation of AI with tasks that require real expertise and real world judgment.

18:16That's why companies like NVIDIA Anthropic, Salesforce, and Gemini partner with Turing. Turing builds realistic reinforcement learning environments and data systems based on real operational traces, the kind of infrastructure frontier labs need to train superintelligence. Visit turing.com slash S-O-U-R-C-E-R-Y. I guess to dig even deeper, architecture is changing really fast. I mean, you might have missed the whole SaaS wave. But starting off in AI, it was all chat based. Now we've completely moved to the agentic era, which is like a thousand times more cost competitive. You need compute, power, memory, all these sorts of things that are reorganizing how companies are being built.

19:03And of course, the stock market too. So how are you thinking through the agentic layer and building that out?

19:09Gabe Pereyra:Yeah, so I think this was like a huge transition we've gone through in like the past six months where exactly like you said, the original product we built was kind of this chat based co-pilot for lawyers. And then we built all this product around it that let it work at a law firm on client matters and all these things. And then maybe six months ago when these coding models started getting really good and you could do kind of things like cloud code and codex and that started working really well in the command line. We started building a bunch of the infrastructure of, you know, how do you get these agents that were running in your command line and could execute all these tools and work really well and how to move that into the cloud.

19:52Gabe Pereyra:And I think now you're seeing this with kind of these managed agent solutions. And to your point, it's like now you need sandboxes. the models use way more tokens. We have queries that like we have simple like assistant type queries where you say, you know, draft me a document that a single query can cost$20. We have like a review product where you can upload a hundred thousand contracts and ask the models to review them. And some of those can cost$20 ,000. And so it is just getting incredibly expensive. And I think in parallel, what we're seeing is, I think even two years ago, the models were so good with coding that most programmers would just use them and be like this is super useful I think we're just starting to see in the past six months that inflection for legal where the models can now generate entire documents and they're starting to work in a way where most lawyers who aren't using this technology just for fun they're like this needs to be so much better than the way that I'm used to doing things for me to change my routine to do it that like we're starting to see that absorption.

20:59And so a combination of like the models are getting more expensive

21:03Gabe Pereyra:and they're getting better. They're using more tokens because they're running for longer and they work so much better that users are using them more. We're just seeing kind of this huge explosion of like usage, cost, tokens, everything. We need to talk about tokens. I mean, there's like so many different ways to take this question because there's obviously the margin sentiment on it. And then there's also the internal usage and efficiency on tokens. And I think Winston mentioned you guys got up to$13 trillion. So how are you thinking through that and the cost optimization? And then we can also talk about efficiency internally.

21:42How do you...

21:43Gabe Pereyra:Yeah, so I think there's just kind of that inflection point I was talking about where up until recently, we've always been capability constrained. And so we always wanted to use the largest model because for the most of our tasks and like if you look on the like legal agent benchmark, even the best models weren't good enough to complete those tasks. And so and then the usage, you know, and we weren't at a scale that the usage, you know, it wasn't something like cursor where it was so expensive to serve these things and we didn't kind of have the latency requirements. but that has changed in the past six months where now we are consuming like a huge number of tokens I think for some of the labs we are like the largest consumer of embeddings we are consuming a ton of tokens for just serving all this stuff and so I think it's a good problem to have in the sense of like the usage is so high we obviously need to think through you know how do we serve these customers because if we kind of look at our usage uh like plots right now it's like this is just the start and it's like there's still so much room where it's like if everyone starts using this long horizon agent stuff the way some of our powers are power users are using it um there is just going to be a lot of like cost optimization you need to do and so i think there's like a couple things we're thinking through i think the most obvious is just doing better and better routing of models.

23:11Gabe Pereyra:So like as the largest models get better, not every task needs to be sent to the largest models. And so that's like a great use case of the legal agent benchmark. Increasingly like serving more traffic from open source models, post training models, like a lot of these like very large frontier models are large because they're good at everything. And so I think a lot of the opportunity for these like specific verticals will be, okay, I probably don't need a trillion parameters if I just need the system to be good at diligence. And so how do you build products and models in a way that you can get that frontier performance without kind of all of the frontier costs?

23:48What do you think the biggest question is people are not asking?