The future of AI and the legal field

0:00Julian Nyarko:This is Stanford Engineering's The Future of Everything, and I'm your host, Russ Altman. Since we started this show eight years ago, it's become an archive of amazing and impactful work by my Stanford colleagues. Research is not something that just happens in the lab, and as you'll hear on this show, the research at Stanford can impact areas like health, technology, law, and business, and many other topics that can affect everyday life. We hope you'll tune in to learn more about how research has the potential to help your life and to help the lives of people you care about in your family and your community.

0:32So the lab has four pillars. One pillar is evaluation. Evaluation is an interesting one. So it harkens back to this question of, you know, what is actually what actually works. And in law, it's so interesting because I think the evaluation pillar raises very fundamental questions about what good lawyering is, because now that we have AI, you know, and AI can write a contract for us, we got to know what kind of contract we wanted to write for us.

1:05Julian Nyarko:This is Stanford Engineering, the future of everything. And I'm your host, Russ Altman. You know what, you're listening to the podcast right now, why don't you just press the follow button so you're always alerted to new episodes. That'll guarantee that you never miss the future of anything. Today, Julian Neyarko will tell us that AI is pretty good at law. The question is, how do we best use it to decrease costs and increase justice? It's the future of AI in the law. Before we get started, we're continuing our feature at the very end where I'm going to ask Julian some rapid fire questions. He's going to give me some rapid answers and we're going to Recall that the future in a minute.

1:45Julian Nyarko:Also, a reminder before we get started to press that follow button so you're always alerted to the new episodes.

1:58Julian Nyarko:Well, AI is everywhere, and there's a lot of hopes that it's going to improve many parts of our society. One of the great areas of application is AI in the law. Can we use AI to reduce the cost of legal services, make them more accessible, and increase justice in the world? Well, Professor Julian Yarko from Stanford Law School at Stanford University is a lawyer, and he's also a social science researcher who is evaluating the ways in which AI can be used to improve access to legal services and improve legal education. Julian, how did you decide to focus your research on the ways that AI can be used both in teaching law and in delivering legal services?

2:41Yeah, Russ. So that's a pretty big question for me in my case. So I actually started with natural language processing a little bit before, you know, before AI became cool, as I say. So basically, when I was a PhD student, I was interested in contracts. And for my job market paper, I had access to the SEC EDGAR database. So the Securities and Exchange Commission, they require publicly registered companies to submit so-called material contracts. And I was interested in studying these contracts. So I had half a million contracts in front of me back then. And I said, what can I do? Because, you know, even though PhD student time is not always valued that much, it would have taken too long to actually go through everything manually.

3:34And so at that point, I thought, you know, what kind of methodologies are out there? And I came in touch with NLP, National Language Processing. and that was sort of the first time I dipped my toe into AI. I always had sort of an empirical background. I was doing empirical legal work. But yeah, from then on, it just compounded. And as you know, large language models took off a couple of years later. So this was all around 2018. And then five years later, we now have language models. And so I just from an early point, I saw how transformational really these types of methodologies can be for legal research and also then increasingly legal practice.

4:22But what was very interesting was that as sort of the hype picked up around AI, everyone was excited in the legal domain, but most people didn't really know what worked. right we just took whatever worked in the general language domain and we applied it to law but you know do these things really work what kind of tweaks do we have to make where uh questions that came up time and time and again as you were talking to you know people in silicon valley and you know people at law schools and that really sort of piqued my most current interest uh which is you know really looking at how a large language model technology can be adopted for the legal services sector.

5:03Julian Nyarko:Did you come in with a strong computer science background as a youth, or is this something that you've adopted mostly from the perspective of a practitioner, lawyer, law professor, who then is trying to figure out how can we use these tools? Yeah, my path was pretty unusual overall. So I studied law in Germany, and in Germany, which is different from the US, law is a combined undergraduate graduate degree so you never do something other than law basically at a university whereas you know the students i teach now they have backgrounds in economics on political science but in germany if you're a german lawyer you really don't know much about anything you have never seen an equation maybe other than in high school right but after high school you never see an equation again um uh but i i was fortunate enough that my program back in Germany had law and economics as a discipline.

5:58And that got me really interested in sort of using economic and statistical methodology in law. And so my PhD then was at Berkeley Law called jurisprudence and social policy. It's basically a PhD where you do law and, and my focus was law and economics. And I got, I would say good methodological training and empirical economics and statistics and so on and so forth but no computer science background so this was all basically picked up uh and largely self-taught when I sort of approached this project that I mentioned from my job talk paper around 2018 um um yeah and uh but what I find fascinating about computer science is uh which increasingly is true for other fields as well but computer science has always been a very accessible field, I think, as long as you were able to code, which is something you, you know, for a large part, you don't even need to be able to do these days.

6:59But as long as you are able to code, you're, you know, most of the conversation always happened online. And, you know, you were able to plug yourself in and really sort of, you know, copy someone's script and then alter it to your taste and so on and so forth. And so it's really a nice discipline for, you know, self-teaching the material, whereas in many other fields, that's a little bit harder.

7:23Julian Nyarko:So somebody looking at your website might be surprised to know that you lead a lab. I don't think, I don't know that law professors are always thought of as having a lab, but tell me a little bit about the Lyft Lab and what's its reason for existence and what kind of questions do you ask in this lab? Yeah, so first of all, I think your intuition is exactly right. So in law, we don't really have labs. And I think one of the primary reasons is that we typically don't have PhD programs at law schools. We are a professional school, and there are a couple of exceptions out there. But for the most part, PhD students, postdocs are what keeps the lab going.

8:01And so in law, we don't have that. So most professors just don't have labs. But the more you are sort of in the CS world, I think it's just a very good way of increasing your productivity, right? And so, yeah, it requires a lot of creative thinking about how to staff that lab with lab members and sometimes PhD students from other programs and so on and so forth. But the origin of this lab was really, yeah, so the previously mentioned sort of conversations we had, everyone's interested in AI for the legal services sector and how lawyers would use AI. but it's very difficult for people to really know what works.

8:44And so we thought that, so this is together with my executive director, Megan Ma, who's also involved in research. We got together one year ago, one and a half years ago, and sort of thought, okay, can we make a more concentrated research effort? And that gave sort of rise to this idea of the lab. And, you know, I think so the lab has four pillars. I'll keep it short for now. One pillar is evaluation. Evaluation is an interesting one. So it harkens back to this question of, you know, what is actually what actually works. And in law, it's so interesting because I think the evaluation pillar raises very fundamental questions about what good lawyering is.

9:30because now that we have AI, you know, and AI can write a contract for us, we got to know what kind of contract we wanted to write for us, right? And there are different types of contracts out there and presumably wanted to write a good contract. And now we are faced with the question, how do we even define what a good contract is? And it turns out, you know, I'm teaching contracts and it turns out at the end of the full quarter of teaching contracts, we don't have an answer to that question in part because no one has an answer to that question, right? And so when you're thinking about how do you do evaluation in law, you really think about how do I want to measure the quality of legal services?

10:08What good legal services are? Do I want a rubric? Do I want to do what's called pairwise preference ranks where I show two examples and I have lawyers pick out which one they prefer and so on. And so this is an extremely exciting pillar to me. The second one is AI to improve the quality of legal services. So many AI applications are really about helping lawyers do what they always did, but faster, less error prone and so on, which is all well and good. But from a research perspective, I think what's exciting is can we leverage this technology to actually, for instance, produce information for lawyers that otherwise would not be available to help them make better decisions, to help them use their judgment in a more efficient way.

10:56Third pillar is legal education, where we are asking how this technology can be used in order to improve legal education. because, you know, in part, the big question that everyone has in the legal sphere and in many other sort of services sectors is, you know, now that AI can do much of the grunt work, right? How do we train the next generation of lawyers to develop judgment, to develop, you know, all these strategic thinking skills and so on and so forth. And then the fourth pillar is sort of a grab bag, we call it methods, where we are looking at how can we tweak existing methodology? How do we need to tweak it in order to make it work for the legal sector?

11:41Julian Nyarko:Great. And so the LIFT Lab stands for Legal Innovation Through Frontier Technology Lab. We don't need anyone to remember that. Very good. Yes. So one recent study that came out that you that I've heard you speak about, which is just fascinating, and it actually seems to me that it definitely hits legal education, but it also hits evaluation. although you get to tell me if that's right or wrong, is a really interesting study that you did on AI's ability to answer hard questions kind of in the setting of an office hour, where a student might come to a law professor and say, hey, thanks for that lecture.

12:22Julian Nyarko:I had some questions. And can you just tell us about that study and what you found at a high level? Because I think people found it surprising and it's gotten a fair amount of attention. Yeah, sure. So this study, the background of this study was that when I teach contracts, two years ago, I made a contracts tutor, AI tutor available for students. But back then, tutors were not really, AI was not really that good. It was pretty good, but not great. And so we constantly have to monitor what students, how students interact with the AI. And then we have to intervene if the AI would make a mistake.

12:56And that's not scalable, clearly. Right. And so I thought, can we have a more rigorous evaluation of newer models? And so in the summer of 2025, I reached out to 16 law professors, or a greater number, but 16 ultimately agreed to participate in the study. Everyone uses the same casebook, which is important in law. So everyone kind of approaches the contract material the same way. and we did a three-step study with them. In the first step, as you just highlighted, we asked them to come up with 40 questions that students would ask them on office hours or after class. Step two, we asked them to answer each other's questions.

13:43And then in step three, we would do these pairwise preference ranks, meaning we show a law professor two answers to a question and then they have to pick the one they prefer. And one of those two answers that they saw was LLM generated. This was summer 2025. So it was Gemini 2.5 Pro and Notebook LM, which runs on Gemini 2.5 Pro, but has retrieval augmented generation. So it can draw from the actual casebook. Yes. And so we, yeah, we compared basically the win rate and implicitly estimated the strength of the different instructors and the models. And what we found was that the models were on par with the very best instructor who participated in the study out of the 16.

14:32Julian Nyarko:So that certainly gets your attention. So let me, I'm going to interrupt because I just want to make sure I have this right. So these are 16 law professors and their answers to questions that they've made up are being compared to answers that the LLM, the Gemini, and when they're blinded and they're said, this could be a human or it could be... Did they know that it was either human or AI or did they not even know that there was necessarily an AI involved? We did not tell them that there was... They knew AI was in the study, but they didn't know whether one of the two was AI. In fact, some of the pairs included two AI models.

15:11Julian Nyarko:Gotcha, gotcha. And then to repeat what you said, they actually preferred the AI frequently. 75 % of the time. across all instructors, they preferred the AI answer over the professor-generated answer. And a secondary outcome that was quite interesting to us was we allowed them to flag when they considered an answer pedagogically harmful, which we defined as, you would rather have the student not get any answer than this answer. And the instructors flagged each other's answers, I forget the exact number, I think 10 or 12 % of the time as pedagogically harmful, whereas LM generated answers on average were like one or two percent, low single digit percent of the time as harmful.

15:59Julian Nyarko:So I hope everyone can see why this would get a lot of attention, because these are some of the most, these are great legal minds. I'm going to just stipulate that. And they are not impressed with each other. And they're actually a little bit more impressed with the AI. So what does this mean to you? So you did this study, you had the reasons for doing it, which you actually recounted at the beginning. How well does it answer your questions and what kind of conclusions do you draw? And I'm sure it raises a lot of other questions that you then want to proceed. So how have you processed these results in terms of your own research agenda?

16:34Yeah. So I try to be careful and persist. So I try to not extrapolate beyond what we can actually say. And so there are some important caveats here, right? Really, the scenario that I was interested in was, is it responsible to make AI tutors available to students so they can get correct answers? And I think the answer to this question is, if I would be comfortable to hire my, you know, Yale or Chicago law professor colleague as a resource for students to answer questions, then I should also be comfortable having an AI contract tutor. we did not answer the question uh whether uh ai is good for student learning outcomes right um that is something and in fact we see conflicting evidence out there some of it suggests uh you know one of one of one of the my favorite studies is one where they did a randomized controlled trial they had students go through two tasks the first one the treated group had ai assistance in the first task, the control group didn't.

17:40And then in the second test, they took it away from everyone. And what they found was that people who initially had AI assistants gave up sooner during the second task. And if they completed the second task, they did it worse. And we have a similarly designed study in law in particular, and we can't find these negative effects. So probably it depends a lot on implementation details, but there certainly is a risk that if you have this resource that always gives you an answer to every question that you might have instantly that you you get you know you you lose the protect that productive struggle right and so a longer term goal is to really see you know what are good ways of implementing the this type of technology in order to help students learn better what we also didn't answer was you know sort of some people suggested oh this means we don't need lawyers in the future right and this is certainly a discussion i'm willing to have but our study has very little bearing on our beliefs about whether, you know, AI can automate what lawyers do because answering specific contract law questions is very far from, you know, the day to day of what an attorney does in practice.

18:48Julian Nyarko:If I can ask a kind of a detailed question is, you know, I have office hours and it's often not just a question and then my answer. It's an interaction. So to what degree were these questions and answers interactive or was it a one-shot question and one-shot answer? And does that also in some way limit what you can say? Because especially in terms of the student learning, usually there's a misconception. We have to clarify why the question, how to ask the question, why the question is good. And so it's almost a negotiation with the student to understand exactly what they want, where they're falling short in terms of their knowledge base.

19:27Julian Nyarko:So what can we say about that, if anything, from this study? Yeah, no, that's a good point. So we have, I think there's still many questions around how to exactly evaluate longer multi-term conversations. And we set our study up as a single term. So someone asks a question and the AI gives an answer. Now, I will say that some instructors did provide us with answers that are in dialogue form. So they gave us answers in the Socratic style that said, here's what I would do first. You know, student, do you remember this concept from class? And then walk them through. But they were not rewarded, let's say, in the rubric.

20:16Julian Nyarko:Yeah, exactly. Exactly. Well, so they were not rewarded by their other faculty colleagues for providing answers in this scaffolded sort of interactive way. But really, yeah, we were, our study was not designed to exactly test that, which is another good point on why learning outcomes when interacting with a professor might actually be different. And really, you know, this is also, you know, another important aspect is really, I don't think this clearly shows or shows at all that, you know, it replaces professors. The thing we have in mind is, you know, maybe sort of a teaching assistant that's always available as an additional resource to sort of these more productive engagements.

20:58Julian Nyarko:This is The Future of Everything with Russ Altman, and I'm speaking with Julian Nyarko from Stanford University. We've been exploring Julian's work at the intersection of law and AI. He told us about a fascinating study where AI professors are about as good as human professors at asking questions that students ask in their contracts class. We're going to move our discussion to the use of AI for social science research and especially how fairness, bias and equity can be addressed using the tools of AI. So, Julian, what you do is kind of considered social science research. And I believe that you've talked about kind of the special opportunities for social science research that AI offers.

21:41Julian Nyarko:And you just gave us one example in the previous segment. Tell me a little bit more about how you think about this type of research and where AI fits in. Yeah, definitely. So there's sort of AI as the object of study, right, which the contract tutor study falls in. But then AI can really also amplify social scientific research. And to me, this is extremely exciting.

22:09As you know, it all pretty much started December, January of this year, December of last year, when cloud code became really good to work with. And so it really helps in implementing, you know, analyses. It sort of supercharges what we as social scientists can do. And what I find particularly interesting is, you know, there's one aspect of it that is just, oh, we can write the papers as social scientists. We can write the papers that we always wrote, but much faster, right? Because we can send cloud code out to gather data and, you know, match different data sets and then do these statistical analyses and so on and so forth.

22:51What I'm particularly excited about these days is also thinking about how AI allows us to write papers that we couldn't write before. And so just to give you an example, one piece that I'm particularly excited about is going through a legal scholarship. And what legal scholars sometimes do is they, to set up their paper, they say, courts have increasingly considered doctrine XYZ. Or they say, scholarship has discussed topic X, but did not discuss topic Y. So my intervention is entirely new. And so we have AI agents basically go through and automatically pull out these statements and then test them empirically to see whether they were correct at the time.

23:42Julian Nyarko:Oh, because they might have been using them as, what's the word? They're using them as decoys almost, and it actually isn't true. And so therefore, the basis and justification for the work is not quite as strong as they're letting on. Exactly. And so, you know, the project is not just a gotcha, but really sort of, right, the idea is, A, maybe we can sort of increase standards. But B is also maybe this is a useful tool for authors to, you know, check their own preconceived notions of what the world actually looks like. Editors might use this to check and verify claims that are made in papers quite often.

24:20right and so this is really right every claim every individual claim has a research agent that has access to a certain tool scholarship case law the internet and so on and so forth and then examines that right and uh uh those are projects right we're still sort of at the early stages but those are projects that would definitely not have been possible without ai at least not at scale right and um uh they can sort of really you know make investigations uh possible that previously were not possible. More generally, my project is currently to also center the work at the lab, a lot around agentic AI to basically assist us at the lab as much as possible when it comes to thinking about ideas and so on and so forth.

25:10So for instance, every new project I start, I now have certain research skills and I engage in a one to two hour conversation, Socratic dialogue with Cloud Code, which I think is a really helpful process to, you know, sharpen research ideas and make them put more structure around them. Basically, in empirical projects, it surfaces the types of questions early that usually come up later in the project. And so front loading much of that is a really helpful experience.

25:41Julian Nyarko:Yeah, I've been doing the exact same thing in an entirely different area. And I think our colleagues all over are starting to use this. It's like before you have the first conversation with a human colleague, you can deeply practice your arguments and your approach with the AI so that when you do have the conversation with the human colleagues, it's a much more nuanced kind of high quality interaction. That's exactly right. So you've done a lot of work on fairness and bias. You did that separate from computation, and now you're looking at the ways in which computation can either alleviate that or exacerbate it.

26:22Julian Nyarko:So what is the approach of the lab towards these types of projects? Yeah, so to take a step back, I think there's a... So first of all, I think the interesting interactions are both auditing computational tools, right? If we have tools that automate, let's say, detention decisions or automate allocation of healthcare resources or something, there are definitely interesting questions to be raised about whether these tools are fair. But then also there are some interesting applications that can actually use these computational methods to audit human decision making. And we'll talk more about this, but there are sort of two different lenses.

27:04I would say at a high level, much of the work that we're interested in is, so in law, when you think about anti-discrimination law, there are two notions of bias. One is called disparate treatment, which, you know, in a nutshell means you're treating someone differently because of their protected characteristic like race or gender. And we've done some studies. So for instance, we've done auditing studies where we go to large language models and we say, hey, I want to buy a car from Jamal Washington. How much should I offer? And then I want to buy a car from Peter Smith. How much should I offer?

27:43The implication being that Jamal Washington signals racial minority status, whereas Peter Smith does not. And what we found, this was two years ago, I think, was that it's almost ubiquitous among the frontier models that they suggest offering a lower price to Jamal Washington than to Peter Smith. And here, I compare the response of the models across two strings. And I can say that the only difference is the name. So I can make the causal statement that the name is responsible for the disparity that we see. Yeah. So in a follow on project, we then use model editing to see whether we can basically prune out certain neurons in the model to reduce the disparity without hurting the utility.