#290 Joel Hron: How Thomson Reuters is Approaching The Next Era of AI

0:00The ecosystem of tools and capabilities is pretty nascent. And it also from even nine months ago has changed quite substantially to what it is today. Like what you thought you wanted out of an agent, you know, nine months ago is very different than what you want today. We're trying to automate a tax return preparation. Input documents would be like W-2s and 1099s and these kind of things. Output would be like a tax return that's ready to file with the IRS. And, you know, the tools that would need to be used through that process would include tools like, you know, OCR. it would include tools like some document extraction, like entity extraction, APIs.

0:36In our case, what we've been able to do, we have a tax engine as one of our core products in the tax space. And so we've been able to expose that tax engine as a tool for the agent to go use and run tax calculations and validate tax calculations and inspect errors. And so in many cases, those tools that we're building are built on top of our existing products and applications. Build the future of multi-agent software with Agency. That's A-G-N-T-C-Y. Now an open source Linux foundation project, Agency is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework.

1:23All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on agency, including robust identity and access management that ensures every agent is authenticated and trusted before interacting. Agency also provides open, standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next-generation AI infrastructure together. Agency is dropping code, specs and services, no strings attached.

2:18Visit agency.org to contribute. That's A-G-N-T-C-Y dot O-R-G. Nice to meet you. Appreciate you having me on today, Craig. So my name is Joel Ron. I'm the Chief Technology Officer at Thomson Reuters today. I joined Thomson Reuters about three years ago via acquisition. I was the chief technology officer of an AI company that Thomson Reuters acquired. And shortly thereafter, I led our AI research and development lab, which we call TR Labs. It's a group of about 250 research scientists and engineers really focused on more, I would say, applied AI research. um and uh probably close to a year ago now took over as cto and and as thompson reuters uh we are we're really born as a content company more than anything but i would say um you know over the years we've been really invested in not just like delivering content but also selling the experiences of how professionals interact with that content.

3:34And a lot of that comes down to search and retrieval and these sorts of like technical disciplines as well. And we do that across a number of areas, like the legal industry certainly is one of our biggest, but also within tax compliance, risk, maybe most well-known for the news and media side of our business as well. But we really solve that problem across a wide spectrum of industry today. Yeah. And what was the startup that was purchased? Yeah. So we were a document understanding company. We really were focused on the legal sector in terms of understanding and interpreting legal documents. And that's what Thomson Reuters had acquired us for.

4:20In particular, one of the things that we had done, and this sort of predates generative AI, which has sort of like superseded a lot of those supervised AI models that were being built back in that time. but we had built an MLOps platform and an annotation platform that was really tailored to lawyers in particular to, you know, allow them to develop ontologies of certain types of contracts and curate data associated with those contracts that would be required for training those models and evaluating them and deploying them. And we built a really good proprietary operational platform for that type of thing.

5:02and thompson reuters um acquired us to really scale that into our practical law editorial teams to take you know the practical law editors who have experience across a wide spectrum of legal practice areas and have them do that kind of work uh across different document types uh in different practice areas yeah yeah can you can you say the name of the company i talked to a lot of companies in that space and the supervised learning yeah it was called thought trace at the time and uh we changed the name when we were required to document understanding uh as a product name but prior to acquisition it was called thought trace right well congratulations jumping to cto i'm sure uh that's a pretty pretty exciting thing for a company yeah it's it's been a big change over last few years for sure yeah so we're going to talk about what agentic ai really looks like in production and one of the things i'm interested in is what separates true agentic systems from prompt based assistance how ai is planning reasoning and executing multi-step workflows goes autonomously and what it takes to build that responsibly.

6:24That's a mouthful, but if you could just talk about that, I'm sure listeners would be interested. Well, yeah, I'm sure. And a lot of threads that we can pull on throughout that question, it's pretty broad. I mean, I'd say maybe the best place to start would be defining like what an agentic system is. And I think when I think about agency, I think about it on a spectrum. It's not a binary, it is either agentic or not, but it's really a spectrum of agency. And what agency really means in this case is the level of autonomy that you give the system to sort of plan certain actions, the tools that you give that AI system to then take an action, and the ability to coordinate memory across either one sequence of actions or maybe across a full history of a user's interaction with the system.

7:29So those are kind of three, if you can think about it, agency dials that you can move up or down to give an overall AI system more or less agency. So if you start on the very low end of agency, you know, this might be like a traditional, like you said, prompt-based RAG system. Like it's got no agency to write its own prompt. It's got no agency to select what data sources it uses to retrieve information. Like it literally just takes an input and processes a discrete step or sequence of steps even to go execute it. Whereas if you move to the high agency, these systems are more goal-oriented rather than a discrete number of steps.

8:18They have a goal in mind and they have the ability to plan a sequence of steps that they think will help them achieve the goal. They have the ability to go execute tools along that trajectory of steps. And then they have the ability to react through the middle of that. As they see they've uncovered more information, it might re-inform the plan and maybe they re-plan additional steps. And so that would be like a higher level of agency. And I think depending on the capability that you're trying to build into the application, you can move that agency dial up or down in some cases like particularly in certain aspects of law like people appreciate high precision repeatability consistency and so you might want to dial that down to be like less agentic if you will but in other areas like legal research for instance this is a this is a discovery problem it's often very open-ended and you want to follow the breadcrumb trails.

9:23And so you may want to dial up agency in that case to, to give the model the freedom to go out and sort of learn and explore, uh, multiple paths. Yeah. Although that, that dialing up and down, that's done, uh, at the creation of the agent, it's, uh, or are there agents today that you can, uh, you know, there that you can set parameters, uh, after the agent's built yeah so it's a good question i think in general yes it's done at the time of creation or how you design or architect the agent i think there are some good um corollaries of how this can be done in the user experience itself though like if you look at an application like a coding application like cursor for instance uh i can constrain the agency of of cursor by constraining the context over which it works so for instance i can highlight just a line of code that i want it to explain to me or i can highlight a whole function or maybe a file or i can give it access to the whole repo and tell it to just like go to town and so that would be a way to on the fly control the agency dial of context, and thereby you give the AI system more or less freedom to go out and create.

10:50And that's really a function, you know, in the coding example, is a function of the use case and the user, and are they working on a mature code base that has hundreds of files and, you know, tens of thousands of lines of code, or are they building something new and just trying to vibe their way through it like that would determine whether they want to go high or low agency in that in that case and uh and i think i think those systems and those user experiences kind of lend themselves to allow the user to adapt it to what they need at that point in time yeah uh and these are llm based agents we're talking about uh and they're making tool calls and giving instructions to various tools that then go out into the world and execute and then, you know, come back with information that the LLM either returns to the user or does passes on to another agent which at thompson reuters do you guys have a preference for which models you're using to build agents on yeah it's a good question so um they are llm based in general and in particular like the the orchestrator like sort of the the master agent that's sort of planning and reacting along the way is one of these, what I would say, frontier models in most cases.

12:32We evaluate that pretty consistently. You know, we're close partners with OpenAI, Anthropic, Google, in a number of areas, and we use all of those models and more in certain situations. the tools themselves that the agent system is using may or may not be large language model based. In some cases, they could be a web search. In some cases, it could be another LLM, maybe a smaller LLM that's fine-tuned to a particular task. It could be some other type of model entirely. So the tools themselves can be a variety of things. Generally, at the orchestration and planning layer, there's kind of like a master model that's bringing it all together.

13:19And again, depending on the use case that we're working with, we've found that we've had good success working across all three model providers and they each tend to be good at particular things. And we pretty continuously evaluate that as well. And do you build your own agent building platform or do you use uh uh you know commercial uh uh solutions i just i've been getting i've been having call after call with uh with companies that that have agent building platforms and it's very difficult as a journalist they all kind of sound the same to me but yeah yeah uh but but everyone i talk to says they have the answer uh and i'm curious a company like thompson leuters it's been a bit tricky because you know this development pattern or design pattern for ai has really only existed for nine months you know maybe 12 months at best but So I would say that the ecosystem of tools and capabilities is pretty nascent, to be honest with you.

14:41And it also, from even nine months ago, has changed quite substantially to what it is today. Like what you thought you wanted out of an agent, you know, nine months ago is very different than what you want today. So we have used, you know, particularly like MCP, for instance, is something that we leverage. like i think there's a lot of good industry gravity around uh around that um other sort of frameworks uh like lang chain has their own uh crew.ai etc have their own i think they're all good at certain things we found that like building some of those components ourselves has actually made things uh a little bit easier for us that you know 12 months from now when the space is a bit more mature and some of these design patterns are a bit more stable, maybe we would go to a third party.

15:32But right now, we've kind of built a lot of this ourselves for the moment with, like I said, the NCP and other examples where we do leverage industry standard. From a tool building standpoint, one of the interesting things that we've found as we've done this development is that like the, you know, the same way you would design a UI for a human user to use it, or you would design an API for like a computer to computer interaction, like designing tools is very much a thought process of how is an agent going to interact with this? And what is the sort of modalities of information that it is going to provide?

16:21And how does information need to be presented back to it in terms of a tool output so that the reasoning model can sort of act or react on it you know most appropriately so there's a whole another like level of kind of uh design consideration for how you build tools that allow agents to execute and exercise them uh more efficiently and more effectively and again that's a very nascent space as well that we're still learning every day on. I see. So you're building your own tools. You're not sending it out to call. Yeah. In some cases, you know, like if you were going to go do a web search, for instance, there's a lot of good third parties that provide like web search tools that will sort of leverage out of the box.

17:10In some cases, like a tool could be accessing our content systems. And so in that case, we will build a tool that leverages our search APIs in the appropriate way to search and retrieve content from our systems. In other cases, it may be like taking action in the application itself. So for instance, a good tax example would be if we're trying to automate a tax return preparation, like input documents would be like w-2s and 1099s and these kind of things output would be like a tax return that's ready to file with the irs and you know the tools that would need to be used through that process uh would include tools like you know ocr it would include tools like some document extraction like entity extraction apis in our case what we've been able to do we We have a tax engine as one of our core products in the tax space.

18:10And so we've been able to expose that tax engine as a tool for the agent to go use and run tax calculations and validate tax calculations and inspect errors. And so in many cases, those tools that we're building are built on top of our existing products and applications that we've been maintaining for many years now. Yeah. You know, Thomson Reuters is 170 years old or something.

18:44You know, how did companies move from content or software to delivering intelligent systems? I mean, that to me is, it sounds like an enormous task because it's not simply building the systems. It's getting the enterprise to change behavior. And this is something these guys that are building platforms for agentic workflows, I've been talking to them about. If I were young and had some smart friends, I would pick an industry and

19:30use one of these platforms to field, you know, agent after agent. And you could basically challenge the behemoths with a company of agents, couldn't you? Yeah. And I mean, look, I mean, competition in the space today is fierce and it's never been easier to kind of get started, as you say. uh like developer tools and things like that you know make the the time to ship something that's meaningful uh you know orders of magnitude shorter than what it used to be i think i think for us the content and not just content but the the human experts that generate that content we see as a real sustainable and durable advantage for us.

20:26Like it's one thing to just build a superficial agent that goes out and drafts an NDA. But, you know, quite honestly, I could just go use ChatGPT or Claude or Gemini out of the box to do that quite well. And it's seen a bunch of the internet and is pretty good at that. But I think what really separates us in that regard is the fact that we are not just using those models to generate this, but we're using those models with the context of what practical law, for instance, says about drafting an NDA in certain jurisdictions, in certain industries. And we've had lawyers for many, many decades now who literally draft know-how, draft sample documents, draft how-to articles on how to do this well, what pitfalls to avoid, et cetera.

21:19And that content goes into grounding and steering the agents to doing that task better. And then the second thing I mentioned, which is not just the content, but the people who have drafted that content for many years, they are critical in the alignment and evaluation of our AI systems as we build them and as we operate them. and their ability to provide feedback and steer and continuous critique of what's working well and what's not working well, make sure that we get the best performance that we possibly can out of these systems and that we're continuing to keep them up to date with current law and current industry standards and practices.

22:05So I think it's very easy to build a very simple AI system today. I think it's very hard to actually ground it in, you know, a true and verified information and also to provide the, the transparency to end users that you're doing that and they can pick and trust those outputs in that way. Yeah. But on the other question of how do you, how do you get the organization to adopt agentic workflows or is that easier than i imagine i wouldn't say it's um i wouldn't say it's easier than you imagine i mean it's a 170 year old organization as you said uh years of of of legacy uh behind us that certainly we're very proud of and for good reason i think what we've done though is one, demonstrated real success early on, even for those who have been in this world for a long time and may be slow to change.

23:16When they see this happen, it's almost undeniable that this is a better path. So part of it has just been really leaning in and showing success early. And I think that brings a lot of gravity. The other thing we've done is we've been quite acquisitive over the last couple of years, particularly in early stage companies like Case Text, like Materia, who we've acquired, Safe Sign. And I think bringing new talent into the organization who has kind of a fresh perspective on this problem and partnering them with people who've done it for a long time creates a really good outcome at the end of the day.

23:55And I think we've done that quite well and quite aggressively and certainly will continue to do it. And then thirdly, it's just been organically upskilling and integrating talent into the organization who has experience in these kind of skills. I think that's been something that we have been super focused on for the last two years. I mentioned I came from the TR Labs. Like that organization in the last two years has more than doubled in size. We were about 120 people two years ago. We're 250 plus now and continuing to grow. And so I think our talent composition of people who have experience building and architecting AI applications has grown as a percentage of our overall engineering workforce.

24:45And as you get more and more of this talent into the organization, I think it becomes more and more part of your DNA. And that's certainly something that has happened over the last two years and I think will continue to happen over the next several. Yeah. And then there's, I mean, you're talking about building the products, right? The part of the organizations that's building these agentic products for use by other organizations. But within Thomson Reuters, I mean, it's a very large organization. Is there an eager uptake of agents or is that not really your responsibility to build agentic products or workflows for Thomson Reuters more broadly to use?

25:39it is and it isn't so so my team also owns our internal ai platform which we call open arena and and i would say that has been one of the like we we built and shipped that within like two months of chat gpt you know coming on to the to the market you know two and a half years ago or so so we were very early in terms of building our own platform and driving experimentation into the company. And I would say we have huge adoption. So I think our weekly active users on OpenArena today are more than 16 ,000. It's probably more than 60 % of the company who uses that on a weekly basis. And so we've seen really strong adoption of just AI in terms of day-to-day usage.

26:28And Open is not the only tool we're very eager and and leaned in towards um you know using third-party tools in sales or in marketing or uh um in other functions of the business where there's best of breed kind of domain tools but we also have this platform that has just general usage uh that it enables um and i think we've seen a lot of uptick as you think about agents though that's really like what I would say the next level of productivity from an internal optimization standpoint. And I think we are still on the path of figuring out how to do that best. So today, if you think about where AI was a year ago, it was all about AI augmenting or being helpful to a human.

27:19You would go in there and you would like say, hey, help me rewrite this email. And it would like rewrite it for you and it was helpful well with agents maybe now you're saying go write this email and send it you know or maybe you're not even telling it to do it it just knows that it should go write that email and send it and so there's there's this level of sort of autonomy and automation that you're shifting to ai and and that requires i think more technical depth and certainly more attention from a process standpoint about where you want to do that. And again, what level of agency you want to give the system to do that.

27:56And those are areas that we're building around today. I can tell you, like for my teams on the software development side, we're not just sitting there letting AI write all of our code, but there are certain parts of the coding process maybe around test development or accessibility or other like uh aspects of the development process where we can achieve higher levels of automation and we're trying to drive that forward yeah and and in building products or or uh agentic ai for um production uh what skills You were saying that you're acquiring new skills in the workforce. What skills are you looking for?

28:43Because that's another question I get from a lot of young software engineers. Am I obsolete? What skills should I be developing to keep myself relevant? Yeah. I know a lot of people are concerned about that and what it'll have impacts to the workforce. And I'm sure there will be a workforce transition. One of the things I see is that AI amplifies the best engineers orders of magnitude more than it does the rest of the population. So for great engineers, it makes them like 10x greater than they were before, right? And I think, you know, and that doesn't mean that they are a scientist. It doesn't mean that they've trained and deployed AI models for their whole career.

29:33They could be a standard back-end software engineer. So I don't think there's a specific necessarily AI skill that's necessary to be really good at this. Certainly, you need some of those. You need some core engineers. But for all of those people, I think the most important skill is actually adaptability. the pace at which technology is changing right now we talked about like agentic patterns and frameworks like there's a new one every week how do you know whether you should use it or not or whether this is better like you need to be very adaptable in this environment you need to be comfortable like picking up something new and learning it and trying it and failing it and moving on and that level of adaptability to just pick up anything and get a very quick sense for it and be able to apply it if it makes sense uh is a really important skill i think the second skill that's super important in engineers uh is understanding the lack of determinism in ai systems i think traditionally for engineers they've thought about like here are my requirements like when i click this button like this screen better show up and it better look like this whereas ai systems are not deterministic in this way and and so building for non-determinism requires you to think about like, well, how do I eval something?

30:59And how do I like build validation into the application? Or how do I build rubrics that support the validation process that needs to happen that are stochastic and not deterministic? And I think that's also a skill that some people have. And certainly I think any engineer could develop, but is an important thing for engineers to develop, to get comfortable in building AI systems like this. Yeah. So you think coding is still an important thing for kids to study? Because in order to become a software engineer, you need to understand the structure of software and to understand the structure of software, you really need to be able to code.

31:47I think it is exceptionally helpful. And I don't like I said, I think the best engineers are even better than they were before with these tools. And I think that will continue to be true as we go forward. I think the the the population of people who know how to code because of these tools will also grow. So I think there's tons of people out there who didn't go get a computer science degree, but will go learn how to code in a way that they become great engineers themselves. Right. And and I think to do that well, you don't just sit there and like let the AI model run wild on everything. I think anybody that's done that, like written an application knows that like they can go off the rails pretty quickly.

32:36And and so understanding logically, like how do the applications get built? Like what does service architectures look like? Like how do I keep this thing on a leash as I walk it through this process? Like those are how I think really good engineers get a lot of efficiency out of these tools. So I don't think the engineering disciplines going anywhere. Like I said before, I think there are certain aspects of the discipline that lend themselves to really full or substantial automation in a lot of cases. But the core aspects of engineering, I think, will continue for a while. Yeah. And I've also been asked by software engineers that are not AI engineers, you know, what should they study to become an AI engineer?

33:31but like in the syngentic age that we're entering the ai uh is uh accessed through an api call i mean it's the building agentic system is really an engineering problem not an ai problem am i wrong on that no i think you're right it's an engineering problem what makes it an ai problem is this aspect of non-determinism like i mentioned like i think ml engineers historically would have been trained and well versed on like how to build uh good testing and validation sets and how to look at things statistically and say am i hill climbing this thing to get better or worse or on what dimension is it getting better or worse like they think uh statistically in that sense about the system.

34:23And I think that's what every engineer will need to learn how to do to be able to build AI systems well. I think today we call that AI engineering. It's probably the same analogy to 15 years ago when we thought cloud engineering was like a separate thing. Now it's just engineering. And I think probably the same will happen with AI engineering. Today is somewhat of a unique skill, But I think, you know, five, 10 years from now, it'll need to be ubiquitous for all engineers. Yeah. Why is infrastructure and not just models important going forward in AI? Yeah. Well, you know, I think I would separate that maybe for a company like ours or like the hyperscalers.

35:15you know infrastructure for the hyperscalers is like a fuel for them this is like if they're an electricity provider like a utility provider like this is their power plant like that that is what um right or wrong i think is necessary to create more advanced uh reasoning models more advanced versions of these models that keep coming out you know month on month and so i think that's where you see a lot of this infrastructure investment going is really around training infrastructure. But I think what you've seen more recently is this concept of like test time compute, where the inference in infrastructure actually gets quite large very quickly as you start to expand the cycles of inference that happen on any individual call.

36:07And I think this is another thing that's driving a lot of the big infrastructure spend that's going on within the industry. I think what you've seen over the course of the last several years is both the cost to run and the speed at which these models perform has just precipitously dropped, like orders of magnitude from two years ago. Who knows whether that, you know, curve will continue on, on that particular trend or not. But again, what, what, what that really allows you to do is not to necessarily deliver the same output for a lower cost. What it allows you to do is spend more cycles running the model so that you get a better answer.