How Thomson Reuters Built AI Agents That Think Like Lawyers

29:19Joel Hron:And which leads me to believe we'll start seeing that follow in other tools as, you know, research comes about. Is that something you keep an eye on in like, you know, model selection you're working with or is it fully proprietary inside? If you don't mind me asking. Yeah, no, 100 percent. I would say like, you know, we have a pretty agnostic model philosophy internally, and we're constantly benchmarking the latest models as they come out. Smart. And, you know, the rubrics that we have for these different use cases, hallucinations are one part of that rubric. I mean, obviously, they have much more, you know, I think, breadth to them than just hallucination.

30:06But like hallucination is certainly one of the most critical aspects of it. And I think what we find is that like each of these models are, you know, really good at different things. Like GPT-5, really good at a lot of reasoning stuff. Like the anthropic models are exceptional at like tool use and these sorts of activities, code writing. the Gemini Google models are really good at longer context type tasks where you're dealing with huge documents and things like that. And it's hard to break up the context. And so we really find that like, you know, testing these models on a regular basis, having really good internal benchmarks that we lean on to give us good signal as to where these models might be stronger or weaker helps us refine our focus of where we put these models.

30:59We acquired a company about a year ago called SafeSign that was really focused around legal small language model development. And that's certainly an area of research for us as well. As you think about, again, just the volume of raw content we have, but also the number of experts we have in this field who, again, know this legal process, no legal reasoning. I think the ability to train some of these like finer aspects into smaller models, whether those be tools or actual components and planning process has real potential as well. And so that's a big area of research for us at the moment, in addition to the third party models we use.

31:43Like there's a debate, you know, between open source and closed source. Like how do you think about that in your benchmarks, like personally for you and also for the work that you do? Are you worried about open source models and using them in production or how do you feel about it? I'm actually quite excited about it, to be honest with you. I think in my shoes, like as a company with terabytes of proprietary content and thousands of legal experts, I think we have all of the ingredients to really excel with open source models. And so as that tide rises, I think it creates a lot of opportunity for us, creates a lot of opportunity for us to deliver models to our customers in different ways that help them manage the security or privacy concerns they may have in a better way.

32:37So I think we're really optimistic about the future state of open source. I also think sort of the positive momentum of open source drives the commercial models better. They constantly have a higher bar to leap over, which makes them better at the same time. And so I think the future, particularly as you think about agents, there's a very likely ecosystem of models that develops under any agentic system where you perhaps have some larger planning and orchestration models and you have smaller models that are doing more, perhaps like discrete or focused to things along the flow. And I think companies that kind of have some interoperability there are going to be able to build the best systems at the end of the day.

33:28Makes sense. Yeah.

33:29Joel Hron:You know, you're working with, you know, frontier AI models in a highly regulated industry. What do you feel like it takes to make cutting edge AI enterprise ready for legal professionals, like actually in the workplace. I know you've talked about, you know, bringing in experts, but I'm also kind of curious about, you know, what you can share of maybe the technical end a little bit in terms of like, you know, how big a team you're dealing with and how long it took to build this out, stuff like that. Yeah. I mean, I think, you know, So professional grade AI is challenging to build. And I think I talked about Thompson Reuters before.

34:12Like, I think this is one of our biggest challenges. We're 150 plus year old company. And I think if you look over that history, our brand is associated with trustworthiness and truthfulness and like these principles. And I think, you know, as as young startup companies and there's tons of them now and they're doing phenomenal work. but they have a lot of freedom to go out and experiment and explore and like try something if it doesn't work they pivot and it's like from an engineering perspective that's great that feels awesome but as a company with as much history as we have I think anytime we put something in the market it's definitely held to a bar I think above and beyond anybody else just because again of the history behind us and we take that to heart in everything that we do and develop, which is, again, why I think we put so much emphasis and effort on this relationship with our domain experts and our technical teams.

35:14I think, you know, you mentioned like from a technical perspective, getting a little deeper. I think a couple of things that are like really good ingredients like for success in terms of of building one would be um uh developing benchmarks that technical teams can focus on and so that they can spin their flywheel somewhat independently at the early days if everything you do has to go to a human evaluator every turn of the crank it really slows down the development pace. So we try to like kind of phase automated testing or automated evals with human evals so that we can take a lot of swings at bat in an automated way.

36:01And then as we get closer and closer to production, bring in more and more human eval before we ultimately ship. And I think if you can build really good kind of automated frameworks or automated evals at the early stages, is that it helps teams move more quickly in the early days, particularly. I think the other thing that we try to prioritize is perhaps like keeping teams small at the beginning and then growing them sequentially over time. So, you know, we've been working on deep research in earnest for probably the last six months. and I would say that team started quite small, like, you know, less than seven people and, you know, probably every sprint grew by a couple, right?

36:52As we started to sort of like bring in more context and bring in more parts of the application ecosystem and stuff like this. So we try to grow that over time, I think, a little bit. I guess the other thing is from a technical perspective, Like there's a lot of new frameworks that come out to have this like agentic scaffold and this thing. There's a real balance in like diving into one of those too early. Like the space is really evolving quite quickly. And, you know, being too dependent on one of these frameworks in such a fast evolving space can sometimes be a handicap, even if it helps you get started a little faster.

37:35It can.

37:35Joel Hron:They disappear as fast as they show up sometimes. They do. They do. And you get to this point of development where you're like, oh, man, this is actually like more of a handicap than it is an accelerant. And so, you know, we try to be judicious about our choices there. Again, this is where I think really talented, good engineers come in place. Like even though Claude Code can go write 10 ,000 lines of code, like you can't make a decision like that really well and have an intuition for that really well. And so I think this is where really talented engineers come into play and are exceptionally important in the development process.

38:16And then the last thing I'll say is like starting with the whole problem. So like there's this idea in software development of like the MVP, right? And I think oftentimes teams, they try to minimize the problem, like minimum viable product. Like, OK, I want to make the problem as small as I can to be viable. And so this is the hardest part. This is the most valuable part of the focus on it. And I think if you over index on being too minimal, you actually miss the system. And like really the way the AI system and the agentic system operate isn't apparent until you build the whole system. Right.

39:00And so I think I think really focusing on the whole problem as much as we can on day one, rather than trying to like slice it and solve this component too early, has been a shift for us from from like a product engineering perspective. that has allowed us to really think about these things as a system rather than as an individual discrete component.

39:24Joel Hron:I was going to say a thing about, and MVP is such an interesting subject, because it kind of gets contorted a little bit, and I really feel like a lot of people don't spend enough time on the viable. Quite often you see the minimum. What's the minimum we can shove out the door? And then things have a tendency to stall. And I think it's important that if you're going to subscribe to the MVP idea that you have to understand, you know, it's not MVP and move on. It's MVP and finish building it in public. Yeah, exactly. Exactly. Yeah, I do. Yeah, I do. I think we sometimes just human nature index on like the minimum versus the viable word in that acronym or in that pseudonym.

40:14So I think, you know, not to say you don't want to ship fast and like you don't want to try to like build simply and that kind of stuff. I think those are all good core like agile principles that teams should continue to follow. But I do think, you know, at the early days, trying to take the blinders off and really like scope a big, hard problem and and and see, see what like build all the components, see what the agent is capable of. And then when you see, OK, this is where it feels like it's fallen over. Like that's when you maybe dive deeper and spend more time focusing on an individual component.

40:50But like try to put all the pieces together first and then and then dive in. yeah that's awesome that is changing gears slightly because we just mentioned agents again um so deep research was definitely like i would say the first killer app uh for agents like someone asked me like what's the you know what is the the best killer app for agents like i'm still not figuring out like what they're good for and i was like well do you use deep research he was like yeah use it every day i was like well then it's deep research deep research is the killer app but what do you think, just from your own personal opinion, what the next breakthrough agent use case will be?

41:28Do you have any gut feeling about that? Any intuition? We can just wax about it. What's your thoughts? Whether it's possible or not today. Yeah, it's a good question. And I agree for what it's worth. I think deep research is one of the most profound examples of agents in an application today. I think there's a reason for that, though. And that is because that is the highest agency use case you could possibly imagine. Like, you know, a user's entering this situation with no preconception or maybe some small preconception as to like what the right answer is. And you're just scanning the universe to like figure it out.

42:09Right. And so the agent has full agency to to go out and like use its tools and use its content and use like Maybe it's system knowledge to figure that out. And so you're not like constraining the agent in any way. So I think that's why it's such a profound use case. You know, you asked the question about like hallucination risk and how do you build confidence in lawyers and things like that early. I actually think that the agency dial is one of those ways as well. Like there are other parts of our application, for instance, where we do like, you know, more workflow centric stuff like drafting policies or drafting purchase and sale agreements and things like this, where maybe you don't want so much agency.

42:54Like, you know, my company has a way that they do this. They have like standard templates or standard forms. And I don't want to just go draft some document from scratch every time. And so being able to turn this agency dial, I think, is actually not like it's not a cool use case. I think it's a necessity of different types of work that these systems need to do. Bring you some efficiency too. Yeah, absolutely. I think beyond deep research, which is a really cool use case, I'm really excited by some of the stuff we're doing in tax as well. It's like quite different. But I think it also speaks to some of the things like the models are good at.

43:41So a year ago, everybody was talking about LLMs and like they were like, oh, it's not good at math. Like it'll never do math things and stuff like this. And now all of a sudden, like it knows how to use calculators and it knows how to like write its own code and like this kind of stuff. And score well on the International Math Olympiad. Yeah, silver medals, gold medals, whatever it is. Right. So like, guess what? I don't need to train the model how to like do two plus seven. Like I just needed to know how to write a Python script that does X equals two and Y equals seven, like add them together.

44:15So I think that paradigm change is like really profound. And if you look at like what we're doing in tax, it really brings together this concept of tools and code, I think, in a really interesting way where you think about a tax workflow. Somebody starts with a bunch of like W-2s and 1099s and things like that. And I need to use AI and LLMs in particular in this case to analyze these documents, like resolve context between them. Like I've got this invoice, I've got this tax form. Like do these numbers like match with each other? Am I missing information? Then it involves like aspects of being able to take those numbers and look at the tax law in some way and say, OK, like, how do I treat these numbers?

45:01Like, should I deduct this or should I not deduct this? And then how, once I've done that, how do I then map that to a tax engine that like calculates actually the tax and puts it into a filing form? And that's like an end-to-end problem that involves like a ton of different tools and research and data extraction sort of things all in one. And we've got a beta out there on Ready to Review that does this, at least in some of the simpler use cases. But I'm really excited about those forms of agents because they do touch on so many different like capabilities of the model to be able to do math things, call tools, do research.

45:44at all in one system. And so I think if I were to extend deep research, that would be sort of where it's going next is like this broader context of operating. Wow.

45:58Joel Hron:You know, there's something I've heard mentioned by a few people along the way is a tendency to kind of over engineer these products more than they almost have to be sometimes, because more often than not, If there's the cadence with which new models, new features, new tools are dropping, I keep hearing that it's really easy to get caught in, you know, a loop trying to build something out that's actually going to be solved for you in 30 days or 60 days, most likely. Yeah. Yeah, this is kind of what I was getting at when I said, like, solve the whole problem. Take the tax example I just gave you, for instance.

46:39Like, you know, I might decide, let's say a year ago, if I was working on this problem to go spend a bunch of time on data extraction because like, hey, if I don't extract good data from these documents, then like none of the downstream stuff works. So I'm going to spend a bunch of time on that. And in reality, if you look at where we are today, like LLMs are phenomenally good at this now. Yeah. And they've gotten phenomenally better in just in just nine months or so at this particular task. Right. And so, you know, that would have been a case where you like spend a bunch of time on something.

47:14But at the end of the day, maybe that wasn't the most valuable use of your time, which is the point of why I think like built the whole system, like put put the agent system to the test and have it do all the steps. extracting data, looking at the tax law, inputting it into a tax engine, resolving errors, let it do everything, and then figure out where it falls over. I think I've heard some of the, you know, the large research labs and model providers kind of say, hey, look, when we design our applications, we're designing for what we think the model might be able to do in six months rather than what it's capable of today.

47:54And I think there's an element of that in how you approach building as well, where you try to kind of forecast where these models are going and what they might be good at. And if you forecast that today, honestly, I don't think the models are going to get really good at math, for instance. But what they're going to get much better at is writing code and planning and adapting and these kinds of things. And so you really need to think about, okay, well, how do I need to architect my system to make those components as flexible and profound as I want, but then go spend my time perhaps on the tools or the content underneath them, so that as this boat rises, these boats rise too.

48:34Joel Hron:Yeah, because if you think that if we're going to start a project today, we're going to plan it for two weeks, then we're going to go into an early development phase, we're going to do this, and before you know it, we've spent four months on a project that now could have been done in a half hour with a couple simple agentic workflows. And I think that whole idea of building forward a bit is going to be really important to, you know, and you may get there early. You know, that may be a thing sometimes. You may get there early. Yeah, it's not ready for you. You're ready before the agents are or the models are.

49:10Joel Hron:Yeah, but I think it'll keep a lot of people from wasting time and energy. I don't want to minimize the work to get something out the door, particularly in like professional use cases like this. You asked how long sort of deep research took. I told you like six months. You know, our first spike in the first month was like, yeah, this is it. It was very obvious. You're like, this is the path. And we know exactly what we need to do. Not to say there weren't learnings and nuances along the way. But from month one to month six, it was all about refinement of like these edge cases, like these little like feedback loops that need to be incorporated, new tools that need to be incorporated.

49:54Again, like trying to not be so minimum, like solving a bigger problem and making sure that it's robust against a broader array of use cases. That's where like a lot of the legwork comes in, you know. but really when you kind of see what these models are capable of, a lot of times it's pretty apparent just in the first couple sprints that you're onto something. You just got to keep running that. Well, I got to ask, just because I'm curious, beyond Thomson Reuters products, what is your personal AI stack? What tools are you using day-to-day to get work done? Are you using any of these coding AI tools?

50:36What are you using? I like Claude Code. I think Claude Code has been phenomenal for things that aren't even writing code at times. You know, I find it's been like a really like flexible tool for me to be able to use. We also have licenses to copilot as well. You know, internally, we use Microsoft a lot for a lot of things. I think Microsoft has really done a good job of like integrating it into the graph. I think context is in these models and the ability for Copilot to kind of reach into teams and Outlook and SharePoint and all these different systems that like a lot of my context is in already is really a profound shift.

51:27And I think some of the recent stuff they've done with like Copilot Researcher and stuff like that has been really interesting for those kinds of use cases. But just like personal kind of like things like Cloud Code has been been like pretty amazing. Yeah, I just I talked to someone who is not even an engineer the other day and I asked him, oh, what AI tools are you using? And they're like, yeah, I use Cloud Code with like my personal assistant. And I was like, you use COD code? So it inspired this whole rabbit hole of people who are using it as a general purpose agent. And we wrote a whole thing about it.

52:03It's awesome. If anybody, we need to get it on the pod. I'd love to come back and talk to you about that one day. I mean, yeah, if you can get past the fear of looking at a CLI as a non-engineer, then it's really, really cool. That's so awesome. Using it as like a chief of staff. Like I've seen people say that. Yeah. Yeah, that prompt, you know.

52:25Joel Hron:One thing I've been working on is I've been looking at this. I've got this N8N workflow I'm looking at right now that's a template. But it's pretty much a genuine personal assistant. Like it checks your Slack and has the various operations there and in your email and in your calendar and in your project management solution. And I was like, man, it would sure be cool to finally have all of that brought into one simple feed. Yeah. As opposed to a bunch of different places to go all day, every day, you know. You know, it's really hard. Even like I mentioned Microsoft and like we touch a lot of Microsoft systems today just internally.

53:06But even that's not comprehensive. Like just a lot of things that I do day to day that are outside that ecosystem. And so it's not particularly easy. Like I said, context is king for these models, but actually context is really hard. You carry a lot of context around in your brain that you probably don't even realize that you're carrying around. And it's hard to catalog the list of all of those things to give to an AI model because it's a longer list than you might expect if you really think about it.

53:38Joel Hron:And more than you even know, most likely. Yeah. There's so much. And, you know, it's been really interesting. You know, one of the things we see in looking even at just context windows is that, like, depending on the tasks and who you are, like, you know, we know people who are working in gigantic spreadsheets. And a context window boost is absolute gold to them. So is speed. where with what I do a lot of times on the more creative end, I care less about speed and more about getting kind of specifically what I want. And I think we're looking at a future where everybody uses the model that works for them, kind of, or the tool that works for them, you know, and that may be three different ones.

54:25Joel Hron:It may be five at some point. I don't know yet. Joel, for professionals listening who want to level up their research game with AI, whether legal or not, what is one mindset shift or approach you'd maybe recommend? end well i'll tell you like when i think about hiring and talent and building out with my team there's like one skill that or maybe like quality i would say i don't even know if it's a skill that i have indexed on more than anything and that would be adaptability like you talk about the pace of technology changing and you know designing systems that are future-proof i think that's impossible.

55:08I don't think you design something that's future-proof. I think you learn how to adapt really quickly. And I think people that have, whether it's natural or learned, like a mindset or a capability for adaptability, and maybe there's an element of humility there as well, which is to say, like, you know, I always need to be curious about learning the next thing. I think that's one of the most important, I think, skills for people in this day and age is, is like not being too fixated on kind of where you were, where you've been or what you've done before, but being really adaptable and open to kind of try and exploring new things.

55:50So that would probably be the one I'd focus on more than anything myself. That's great. It's like basically making yourself future-proof. Like you are the future-proof tool, not whatever you built. So as long as you're adaptable, you'll be future-proof. So before this role, I used to lead our AI research and development lab. And I used to tell like a lot of scientists have this sort of identity of like being a scientist. I'm like, I do science and like I don't do engineering. And I can appreciate that. Like I think there's a lot of value in that. But I tried particularly in our field, which is more of applied research than it is like academic research.

56:30I've tried to instill into my teams like you're an engineer I don't care front end back end you know QA science not science like you're an engineer and like engineers like engineer new ways of doing things like that's that's your job that's what you're trained to do I don't care if you're a mechanical engineer a software engineer a chemical engineer like you're taught the same foundations of like how to logic and intuit and you should use that skill and it will future proofread to anything, quite honestly. I'm a firm believer in that.

57:03Joel Hron:I agree. I agree. I think we're moving into an age where the smartest thing you can do is learn to get excited about change. Learn to find change exciting and be able to roll with the punches and adapt well. Joel, thank you so much for joining us today. This has been excellent. It's been a really fun deep dive into a very specific industry's use case and to help us better understand where you go, better understand where you all at Thomson Reuters are finding these wins and losses and what you've gone through in developing deep research. I think it's a really fascinating tool that's going to get a lot of use for an awful lot of years moving forward.

57:47Excited about it. Thank you guys for having me today. It's been a great conversation.

57:51Joel Hron:Excellent. Where can readers go to learn more about what you all are working on? Maybe take deep research for a test drive. Yeah, I would say, you know, most of the stuff we do, I try to put out on LinkedIn so you can see the latest. We also have a Medium blog that you can follow that talks a little bit more about hopefully some of the technical stuff that we do. I also try to post links there. So I would say check it out on LinkedIn and you'll be able to see some of the videos and links to how you go out and try it out. Excellent. We'll drop a link down below in the video description as well.

58:26Joel Hron:Thanks again to Joel, to Thompson Reuters, and to all of you for listening. I'm Corey Knowles here with Grant Harvey as always. And today, farewell for now, humans. We'll see you next time.

58:50FR difference!