Building Harvey With Gabe Pereyra: Ethical Walls, Agents, and What AI Can Unlock For Law Firms

0:00If you think of what the best associates do for a partner, it's exactly that. I read the whole data room. Here's all the issues I flagged. Here's the things I'm uncertain about. Here's my draft of the memo. And the partner reads that, checks a couple things, and they say, I'm confident enough to now go and give that to the client. Because at the end of the day, this client is paying this partner a lot of money to be able for that partner to say, I'm willing to risk my reputation that this transaction is correct. I don't think there is in the short term like a magic solve for that through AI. I think it's still the same problem.

0:38Tomer Cohen:The holy grail of building a great product is when you can combine two things. A deep understanding of the problem and deep expertise on how to solve it. That's the case for today's episode. On one side, you have a founder who deeply understands the domain and how work actually gets done there. And on the other, a founder who understands AI at the frontier. My guest today is Gabe Priera, a former AI researcher. Together with his co-founder, they've built Harvey, one of the leading AI products in legal today. It's used by top law firms and enterprises around the world to draft, analyze, and reason through complex legal work, contracts, and cases, and filings.

1:22Tomer Cohen:When you think of legal, in many ways, it's the perfect domain for AI. It's language heavy, it's logic heavy, and it's high value. But it's also very hard with AI. Because in legal, every word matters a lot. And every mistake can carry massive consequences. So you need precision, accountability, and trust to get it right. So what does it actually take to build an AI in that kind of environment? I'm Tomer Coyne, and this is Building One. Let's get into it.

1:59Tomer Cohen:Gabe, it's a pleasure to have you on the show. Thank you so much for joining me. Thank you so much for having me. So, well, we start with a personal journey. And yours is always, you know, all builders journeys are quite unique, given where you are right now. But yours is very unique versus our guests. You come from a very academic background. In fact, as far as I know, both your parents have a PhD in computer science. They both worked deeply in AI. In many ways, you're continuing the family legacy. And you actually started as a scientist doing AI research. And I'm curious, was just that that was in the air at the house?

2:35Tomer Cohen:You saw where this was headed and you decided you want to go there. This was a great sense of where you can have impact from your family or something else? I actually think about this a lot. Like when I was younger, I wanted to play soccer professionally. And so I dropped out of high school. I went to Brazil. I played, got injured. And then after that - What's your favorite team? I'm half, my dad's from Argentina. So I'm half Argentinian. So I was like a Barcelona. Yeah, exactly. I was a huge Messi fan growing up and still follow them. But after that, I actually wanted to do finance, like investment banking, start a hedge fund, something like that.

3:13I kind of didn't know what I was doing, but I was like, this seems competitive. I like soccer. that was competitive. And my parents were like, you should try computer science. You would love it. And I was like, I don't want to do what you guys do. I'm going to do something different. I went, I worked at a hedge fund and then I kind of worked at a quant fund and through that found AI when I was an undergrad in college. And I think just through kind of reading about it, this was around 2012. This is when I kind of found deep learning. And this was very early days of a lot of the technology that now I think is working really well.

3:47And my parents had done related stuff. And so I kind of got more interested. And I think like from very, even when I was doing AI research back then, it was always a means to an end of, I wanted to be very technical and I wanted to understand this stuff. But the goal was always to start a company. I just at the time didn't realize that company would be in legal.

4:07Tomer Cohen:So the goal was, let me learn the foundations behind the technology really well. but you always wanted to move towards the company side, the application side of it, the business side. Yeah, exactly. I got, I got drawn to it through like in high school when I was playing soccer, I was really interested in just how do I get better at soccer faster? And I kind of read a lot of like psychology. And I think through all of that, just human learning in general was super interesting. And then through kind of doing that, my parents' background, like I eventually found machine learning and I was like, oh, this is like, and the more you kind of learned about it even then.

4:40I think I just had this strong intuition of like, we were going to be able to build something like AGI. I think at the time, like no one knew the shape of it, but as you saw these models getting better, you kind of had this intuition that this was coming.

4:52Tomer Cohen:And AGI was a dinner, you know, a family dinner conversation on a regular basis when you were young? We definitely, we definitely talked about it. So my mom, she's retired now, but she ran the autocorrect team at Apple. And so she actually had built one of the first like billion user language model products. My dad was a math professor at Stanford and a bunch of other places, and he did numerical analysis, but like a lot of linear algebra stuff, which I think underpins a lot of these algorithms. And so I kind of, as I was doing more research, I would just talk to them about it. They had both published papers and knew a bunch of this stuff.

5:26So I think that was cool, just being able to talk to them about it.

5:29Tomer Cohen:Before we get into actually building Harvey, it sounds like you and your co-founder, it was the perfect duo. On paper, it looks amazing, right? You come with amazing background in Frontier AI, and then your co-founder, Winston Weinberg, comes with amazing domain expertise in legal. You know, it's a dream come true for a company, a dream come true for VCs to invest in you guys. I'm curious about why you chose legal, because I'm sure, again, given your intention, I'm going to learn this really well, And then I'm going to build a company. I'm sure you saw so many opportunities across. Yeah, I think the one thing that I looked for when picking a domain was actually, will this have a big societal impact?

6:13And so when I was doing ad research, the original startup that I thought I wanted to do was personalized education. Because I think for me, it was really important. This technology is going to be very powerful.

6:25Tomer Cohen:Like a personalized tutor kind of thing? Yeah. Yeah. And how do you, yeah. How do you use this very powerful technology to go, not just build a big business, but solve like a really meaningful societal problem. And so I was, I was actually pretty agnostic to the application. Uh, and then I think legal had that property too, where we're starting to do a bunch of work of, you know, how do we help court systems and access to justice? And so I think there's something beyond just, this is working kind of financially. And then I think the bigger thing obviously was just Winston and I were super close and he was kind of the perfect person to start the company with.

6:57And we actually, when we started thinking about the idea, I was brainstorming with some other friends that had worked at Google Brain of kind of more general ideas. And one day Winston said, you know, why don't you build an assistant for legal and showed me his workflows and kind of the work he was doing. I showed him the models and then we pitched that to OpenAI and they said, you know, we'll fund this. And then that was kind of the like the initial spark for it.

7:20Tomer Cohen:as AI labs keep kind of moving up the stack and they're potentially taking parts of the application layer. Like, how do you think about it? Harvey is, as a vertical AI company, is specialized in a very valuable domain. Did you think about that risk at the beginning? Was it just like, I know my path or was it, we're going to like figure this out as we go? I think even when we started the company, like a lot of what we thought about was like, how do we build something that exactly what you said doesn't get absorbed as the models get better? I think that thinking has evolved a lot of what are the things that are defensible.

7:56But I think what I would say and like the big shift that we're seeing now is I think people still, for the most part, think of these products as individual user products. And so they think about the problem that you're solving is if only I could make this model smart enough so this user can do this task that they're trying to do. and I think the real problem that we are trying to solve is how do you help law firms improve their business and how do you help their clients get better legal services at the end of the day like the reason that I pay a law firm to do a large merger is I want them to be accountable for the outcome right and it doesn't matter if the model is smart enough to do that there's like regulatory insurance human accountability there's kind of all these reasons on top of why you need kind of these existing systems.

8:47And I think this will change. Like one of the nicest examples that I like that's legal specific for when we get the question of like, oh, why isn't OpenAI or Anthropic gonna do everything? Like the simple answer is just conflicts, right? Like if you're a law firm and you just use only Anthropic's products and now you need to represent Google, OpenAI, like any company that competes with Anthropic, you can't use their models, right? Because none of those companies are going to let you send their legal data to these models, right? And if you think of as these tech companies get bigger and bigger, they're all going to build their own models.

9:22None of them are going to let you send their sensitive legal data to competitors. And so you need to build this infrastructure that abstracts this all away.

9:29Tomer Cohen:The idea of conflicts or kind of subject matter is unique to legal in many ways, and it's not easy to solve. When you think of these law firms, it's, I have 10 ,000 clients, And now I need to do that in a way where all of their data is separate, but my lawyers are able to like work across that. And so I think there's problems like that. And then there's equivalent problems when you get into enterprise. We work with a bunch of large banks, private equity firms, insurance companies that all these like regulatory challenges that you need to solve to deploy these. And so I think the more that I've done this, the deeper appreciation I've gotten for the problems just any of these enterprise SaaS companies are solving, where it's like once you get to these massive companies and have to solve their problems, you're like, oh, yeah, this is like very difficult.

10:13Tomer Cohen:Did you guys were able to solve the training part as well or mostly the separation of access to information? What we are starting to do is thinking about how do we help a law firm and a client train a model together. the problem you run into is when you start thinking of like all of the data a law firm has a lot of that belongs to their clients and so you can't take all of that data and train one model because now you're breaking all of these ethical walls in these conflicts and even if you redact the data the models are now so good that they can just look at all of the information even if it's redacted and still roughly guess like this is probably this company and so one big thing we're actually working on right now and hopefully open sourcing soon is I think we've made massive progress in how you build synthetic data sets.

11:00And so the ability to basically build these RL environments of a transaction of a litigation that you can train an agent on. And so you can get a lot of the performance gains from either those synthetic data sets, or we use a bunch of contract attorneys and lawyers to improve those. If you're talking to a founder out now, and they're kind of

11:22Tomer Cohen:of battling where you were maybe several years ago, trying to figure out where to go and what to do because you were like, this is incredible time to build. There's so many opportunities, but I kind of want to choose my path correctly. Is there a framing in mind that you have to offer kind of future entrepreneurs? I would say the high level advice is you need to find the things that aren't obvious right now. And so like a lot of how I thought about what companies start when I was in high school and I was thinking about, okay, I'd maybe want to start a hedge fund. I read a bunch of Warren Buffett shareholder letters.

11:54And a lot of what he talked about was like, you need to be very correct, but you also need everyone else to be wrong about the thing you're correct about. Because if you're very correct, but everyone believes the same thing, there's actually not a huge amount of like upside for your company. And so like, I think back in 2014, what I strongly believed was AGI was coming. And I think most people in the world didn't believe that, right? And if you had made, I didn't have, you know, enough money to make investments at the time, but if I'd made the bet of like NVIDIA, right, that would have been a great bet, right?

12:27And I saw like, I worked with Jeffrey Hinton and he was like, we need so many GPUs to train these models and NVIDIA like is starting to make these for us. And so there was things like that, where if you had this really strong belief back then, and then even when we started Harvey, most people like ChachBT didn't exist at the time. And so doing like this vertical AI solution wasn't obvious. And then people had this belief of, oh, if you build on top of the models, that won't work for all these reasons, right? And so there was all these like counterintuitive things. I think the really interesting companies starting now are going to be the analogy to companies like Uber, TikTok, Airbnb, where before the internet, you just couldn't imagine these companies, right?

13:07And so what are the companies that you can't imagine right now that are going to be made possible by this technology? And I think the best way to figure those out is you just need to use the technology 24-7.

13:20Tomer Cohen:When I think about the product development lifecycle itself, you're going to be blocked on ideas soon because actually producing them. But, you know, what was like two years ago was like, oh, that's fine. Now it's like, you know, now it's clearly like the way to build. and now you're going to start moving up the stack into other areas. I would actually say the biggest thing like we're struggling with is not just the bottlenecked on ideas, but bottlenecked on coordination. Like I think the thing that is increasingly getting hard now is like the alignment where now our PMs can generate entire product suites and they're just like, here's how I would build this entire product line fully mopped out in our design system.

14:00And it's like the hard part is like, okay, you have 10 teams that all have these, Like, how do you allocate resources? How do you solve all like the merge conflicts of these?

14:08Tomer Cohen:So when you started, was it, I'm going to pick the kind of high velocity, high repetitive tasks from that kind of lawyers do or legal clerks do. And then I'm going to automate them or, and then is it now like, no, no, I can see how the organization works. So let me walk backwards from what is that kind of throughput of that vector of everybody working together. Let me think about the different atomic units of that. Let me automate those together. Is that, was that the way kind of beginning and end or from the beginning, you guys already had the organizational thing mapped out? No, I think from the beginning, it was very much, it came, the ideas came from the technology.

14:48And so I would think of the challenge we face of building the product is, I think you still need to do this, right? Like if you look at why the coding companies and the model companies build such great products, it's because they basically take the raw model and they put the least amount of things possible on top of it, right? Like now I primarily use the models from the command line, which is essentially no product on top of the models and you just interact with them. And so you have kind of what is the right shape of products given the capability and the shape of the models? That's one constraint.

15:23And I would say that's kind of the bottoms up. And that's what you're seeing the labs and the coding companies do of, you know, how do I add stuff on top of these models that make it useful, especially for programming? And then there's the top down, which is here is how a law firm works today. And here's all the things that you need for this to work in practice. And so when I think of product development, I kind of alternate between these two. Like I use the models for coding all the time. And then we spend a bunch of time talking to customers and especially lawyers and just saying like, how do you work on this client matter?

15:55What do you do? What tools do you use? And then thinking about how to bring these closer and closer together. And they're still really far apart. One way I get a lot of product intuition is I try to do the work that our clients do with the code models. And so a lot of what I spend time doing is if I wanted to automate a diligence or I wanted to do a first pass of a diligence, how would I do that with the raw models? And so, for example, one thing I did a couple months ago was take all the documents you would expect if you were buying a company that would be in a data room, convert them to markdown, and then figure out all the skills and scaffolding you would need for an agent to read all those and generate like a diligence memo.

16:35right and then you conversion control it you can like put this all in github and you can figure out okay from a capabilities perspective this actually works quite well right and so now you have a sense of okay this is what the capabilities are and now you need to do the translation where it's like okay this is usable for me because i'm a software engineer this isn't usable for most lawyers and so you need to figure out like they're not going to use git for version control like they store all their data here this is like the interfaces they need to access those documents in Word and Redline. And so I think, to me, I kind of try to separate, how would you get these things to work really well, independent of how do you make them easy for a user?

17:15And that's kind of, we have like Harvey Labs, which is like a research team that solves these problems, and just figure out purely how do you get the capabilities. So this is fascinating, because in a way, I, you know, one of the shifts I've led was the kind of

17:30Tomer Cohen:transitioned from product and design to more of a full stack builder. But there's a belief that you have like system builders that are building the infrastructure. There's researchers who are building the capabilities of the future. And then you have full stack builders who can just pick it up and figure out like what the customer runs really well, marry it with the capabilities, and then just take it to market. Yeah. And I think this is where it gets to me very complicated because as, as you think of what's going to happen with these agent systems, like a lot, like the product we're building for law firms is increasingly going to be, how do we help your entire law firm or your entire in-house department run on agents?

18:05Tomer Cohen:And it sounds like you see the next big step as like organizational agents working together. So you're no longer representing Gabe, you're representing a team. You're presenting like there's multiple agents working together. And that's a different level of collaboration that is needed. Is this something you're tackling yourself or you're waiting for capabilities to show up to help you do that? The way I would think of this for law firms, increasingly the direction is going to go is if I'm a law firm, I'm just working on, you know, say a thousand different client matters. And what I really want is this infrastructure that for every client matter, it as a partner, it lets me delegate this work either to an associate or to an agent.

18:46Those go off and do that work and then they tag me in when they need my judgment. Right. And so what is all the like process management and infrastructure you need to enable that? at scale. Well, and then we also, we already have a product called Shared Spaces where you can collaborate with your client. And so you actually have this workspace where that client matter is happening between the law firm and the client they're representing. And so to your point, the permissioning and how you ensure the auditability, the quality, the, you know, all of this that you need to do that work. I think this is some of the like very complicated, both product and technical problems that we're solving to enable this.

19:26But I think the capabilities are there.

19:27Tomer Cohen:When you start to work agents to agents, is there a different layer of efficiency you can work because the agents just work with agents, so you can just talk in zeros and ones? I think this is maybe where legal is a bit different than programming. But if you think of what a contract is at the end of the day, this is like an agreement between two parties. And I do need to understand what that agreement is. And so when you think of one of these client matters that these law firms is working on. It is either a litigation where at the end of the day, what I'm producing is motions that I'm going to present to the court.

20:02So those do need to be readable or contracts where I'm buying a company and here's the purchase agreement. And I do need to align with the other party on that. And so I think one of the big problems that we are trying to solve, which I think does exist now in engineering is the bottleneck is human review, right like we see this in our engineering org where now everyone can write so much code that the bottleneck becomes how do our senior engineers ensure that all that code is correct just the same as like when you were running a team and you think of the best people that worked for you it's like they were able to like distill their questions and what they did in a way that you were like i don't need to go look at everything you did because i trust you and it's like we need to solve that.