AI in the Investment Office – Abby Barlow, Laura Hill, Brian Sugrue, Jenny Heller, John Lawrence, Matt Bank, Kristin Kallergis Rowland, Jon Webster (EP.515)

17:22Brian Sugrue:We haven't crossed that hurdle yet.

17:25Jenny Heller:What are some things that you tried that you found AI didn't get to where you were hoping to?

17:34Brian Sugrue:There's a broad category of things where it continues to disappoint. And that's where you venture into areas you maybe don't understand. What it can produce if I ask mediocre questions is something that seems to form fact-checked, footnoted, and probably fictional, but I'm maybe not expert enough to recognize that. I see it from the team as well when they've leaned on their own experience. It produces garbage.

18:05Jenny Heller:In the three areas where you are using it actively. What's your tech stack?

18:10Brian Sugrue:We are primarily using Claude as a secondary tool, ChatGPT. We do have a portfolio management tool called Asora, and they have built a huge amount of capabilities into their UI. They have brought some of their own engineering expertise to help with some customized applications that they can now deliver extremely quickly for private markets.

18:36Jenny Heller:As you look out over the next year, what are some of the projects you're hoping to take on using AI that are not in place today?

18:46Brian Sugrue:The big one, we talk a lot in our organization about the hit by a bus risk. LPs ask managers all the time, what's your key man who's going to run the show? So it's true for us as well. If I disappear in the morning or if my principals were not around or available to share institutional memory, what is our process for how we can transition things smoothly? AI as a tool for us to examine where there are gaps to help us generate some of the content that's required to take my and the team's institutional memory today, commit it to paper so that hopefully it's a positive transition in some number of years.

19:31Brian Sugrue:I'm handing everything off to another experienced CIO who can pick up the ball and continue to run, not go, oh my God, I've been left with a total dog here. I don't know where to start. Try to bring the operational side up to a grade A and above is the one big goal for us. A couple of practical pieces, cash forecasting, I would like to see us primarily leaning on AI for pointers on exposure, risk, inefficiencies on cash holdings, what rates we're exposed to on the borrowing side, what rates we're capturing on the cash deposit side, mirroring that with our expectations on capital calls and distributions, capital commitments made to private markets.

20:18Brian Sugrue:There's a lot that can be done there. The second would be on manager and portfolio monitoring. It's maybe a little easier in public markets. There's more readily available fluid information. I'd like to automate our process a little better on that. We have an organized calendar on manager updates. Still a huge backlog of, well, I'd love to pull all my notes together from the last 10 meetings with a manager when I have a few hours and think about how their narrative has evolved. or why don't I take this firm's fund two, three, and four decks? They tell me their process has not changed. What is their marketing materials telling us about their process and how it's evolving?

21:00Brian Sugrue:These are questions that anybody who's sat in a CIO seat has always wanted to answer. These tools are greased to the wheels, making it easier to go out and find those answers.

21:11Jenny Heller:What does it take for you to get there compared to where you are today?

21:16Brian Sugrue:In theory, one of the great things about AI is that it can draw from unstructured and disparate data sources and put one narrative together. If I could standardize the documentation that I have on a manager, that's what it takes. there's a broader point here, which is how much time does using AI really save? What I'm describing is a time-consuming preparation process to get to an answer. It's not two or three or four times faster than it would be otherwise. It's a little bit faster and it facilitates the final steps, but all of the preparation on getting data quality right and getting the information organized, that's still true today.

21:58Jenny Heller:What do you think it would take for you to start using AI to improve your decision making?

22:06Brian Sugrue:The way I have the framework in my head, if I was just sum up where AI is for us today, is that it's not about producing analysis that we could not do before. It's about creating a little more space for us to use judgment in analysis. There's only so many hours in the day. If I can save 5 % or 10 % of those by letting AI consolidate data, analyze a transcript from a discussion, put all of that into some keynotes so that I can sit back and take a little more time to think about what my next steps following a diligence meeting or a manager update. that's where it's opening our aperture a little bit right now it has not opened our eyes to new sectors we're not exposed to it hasn't helped me make investment decisions yet hasn't helped me size positions better i don't know if we'll get there i'm maybe a little bit of a skeptic on whether it's really ever going to overtake that judgment piece where we're getting benefit today is carving out a little more space for judgment taking up a little less time on producing analysis

23:14Jenny Heller:Brian highlights both the power and limitations of using AI as a thought partner. Jenny Heller, president and CIO of multifamily office Brandywine Group Advisors, takes us from individual experimentation toward building AI across an investment team, with a particular focus on trust, verification, and turning a collection of one-off use cases into a shared operating system. Jenny, how are you using AI in your office now?

23:45Jon Webster:A principle that underlies how we're using AI today and will continue to use it is building layers of trust. One is trusting in the boundaries of AI, having confidentiality in the perimeter of where information goes. Everything we use is enterprise grade, nothing trains on our data. if we ever use agents, which we're not doing yet, they'll be walled off and sandboxed before we let them loose. And this is what earns us permission to use it at all. Another thing we think about is having trust in the output, thinking about accuracy as a primal part of any of the work we do, which means that we have to have a verification discipline, keeping a human involved in any consequential decision as we get there.

24:28Jon Webster:Finally, trusting each other, thinking about what the provenance is of anything that's produced. Over the long term, thinking about what's human and what's AI is important in ensuring everyone has transparency in any piece of work. Today, we've spent a year plus in undirected play inside of our office. We're using AI manager meetings and diligence prep. It could be running investor letters. We've connected ChatGPT and Cloud to our Slack channel. We can pull manager data. We can pull meeting notes together to come up with thoughtful updates before we enter a meeting, generating questions that I think are grounded in a longer history of data than you can consume independently to draw on our accumulated history with the manager.

25:12Jon Webster:Everyone on the team has different use cases now, but we do these manager assessment frameworks where we grade our managers before deciding whether they go in the portfolio or whether we're going to engage in a re-up. I will pull a ton of information together beyond the memo. I can bullet point every part of the framework and ensure that I'm leveling out my own bias by looking at broader history. The team is using it for portfolio and position analysis, actively using Cloud for Excel to recreate manager models and pressure test assumptions. One of the cool use cases has been around HR and mentorship.

25:46Jon Webster:Whoever's responsible for a meeting will pull together meeting notes prior to a meeting. If I read through the notes and feel like an area needs to go deeper, I'll start using Claude as a thought partner to have a back and forth on how can we deepen this line of questioning and then pull from that dialogue to then give that feedback back to the team. Everyone on the team filled out a sheet called How I Operate that focused on how they like to communicate, what some of their pet peeves are. Before I was giving reviews to anyone on the team, I wrote the reviews and then I paired them in Claude with the How I Operate language and said, how could I reframe some of what I've written so that this person can better hear it.

26:24Jenny Heller:What have you tried that haven't met your expectations and the hype of what AI might be able to do?

26:30Jon Webster:We tried to pull together mid-diligence updates, and this would sit between our issue memo, which is early stage diligence sharing with the team, and our investment memo. The thought was that the lead or associate director could pull together this update using AI in an hour. It was a failure. We got some great diligence updates. The team was finding that they were spending hours and hours pulling it together because the context was too big, the questions weren't specific enough, or the outputs that they were getting weren't conducive to the way they wanted to share information. So then we put it more on the team to say, we're still going to do the mid-diligence updates, but it's on you to be informed as opposed to creating another layer of paperwork trying to use AI as a cover.

27:12Jon Webster:I'm hesitant about how AI is used in memo writing. It can be helpful in producing a robust quality of returns analysis, something that's concrete and specific. There's real limitations. If the problem site gets too big and too unstructured, the LLMs start to lose their potency and their accuracy. The more structured and specific the question, the better the answer.

27:33Jenny Heller:How have you managed the trust of all of your data?

27:37Jon Webster:Using enterprise-grade systems was step one, ensuring that the systems have a robust compliance structure built into them. The model isn't training on the data, at least that's what they say. Another component of it has been that we have been deliberately slow in turning on co-work, code, any of the use of agents, because we want to understand how we can ring fence what we're building. We know we want to get there and we have a path. I would love to have Clyde be able to connect with my email. I'm just not at a place yet where we can be sure that that's safe and secure. Until we get there, we're approaching it slowly.

28:16Jenny Heller:Where do you hope to be a year from now?

28:19Jon Webster:The beginning of the arc was having this unstructured play. The next step will be level up the team. We're structuring a number of teach-ins. We're going to have a learn to code through cloud day that one of our wonderful managers, Sam Lesson, is going to teach a number of teams in the New York area. The goal is really to build flexibly so that we have an owned layer that sits above any model that can have all of the skills of our process built into it. where we can take all these one-off use cases that the team has and turn it into something that's more of a shared language and a shared ecosystem that the whole team can use.

28:56Jon Webster:What we'd like to do is start letting the LLMs selectively read data, whether it's tying into data rooms or tying into the underlying documents that will create a powerful knowledge layer. I want AI to be at the core of our operating system as an investment team. Doing things like every time someone on my team writes a memo, the system can prompt them and say, well, instead of writing this for you, here are some questions that I can ask to help you hone your own thinking. It can become more of a dialogue that we have a system that knows us well enough that it can gently flag anchoring bias or help pull in information historically that could be relevant that we may not be thinking of to strengthen a thesis.

29:36Jenny Heller:Jenny describes the challenge of turning experimentation into an investment team operating system. John Lawrence, CIO of Rice University and president of Rice Management Company, lays out the next step, a roadmap from productivity to investment alpha. Why don't we start with how you're thinking about AI in the investment office? We're using it for everything.

30:03Matt Bank:AI is such an amazing technology that can not only help us from a productivity perspective, but from an investment perspective, we're ultimately hoping to get to a point where we can drive alpha out of utilizing AI. We're not there yet, but that's our ultimate goal. First, we have to set up the infrastructure, which is important. We have to empower the team as well and have to have buy-in from the team. The university, there's hurdles we have to get over. We certainly have to do the right thing. There are risks with regards to data integrity and some of the output. We have to be mindful of that.

30:34Matt Bank:I think it can be a meaningful productivity enhancer. It can be meaningful from an investment decision-making perspective and then ultimately drive alpha generation. How are you using AI today in the office? I'll walk you through a few examples. I probably have a dozen for you how we're already utilizing AI. One that's a very easy win for us was we have a daily news summary delivered by AI. We see news from our investment partners or key investments as well. So that's helpful for me as a CIO. One thing more helpful is a weekly summary of all the work that our team has done. We have multiple asset classes, multiple folks out traveling all the time.

31:12Matt Bank:All this comes in one weekly summary, all of our manager meetings, all the manager letters. That's been tremendously helpful for our team as far as staying on top of the portfolio. Another thing that's been helpful is preparing for manager meetings. We have a tremendous amount of historical data with our current partners. Then new partners, we also have some data as well. Preparing pointed questions and due diligence. It's been helpful on the meeting prep side. On the front end, on our board memos, it's been tremendously positive beneficiary. We save about a day of time. That's a very onerous process.

31:47Matt Bank:It consolidates the work that our team has already done and puts it into a template for us. Analyzing financial statements, AI has been great, particularly when we look at co-investments. AI has been a fantastic set of eyes for us. Building valuation models and building financial models, Claude is good at this, and it's been helpful when we start from scratch. From an operations perspective, processing incoming documents, the tool Canoe, that's been helpful. Automating a lot of the integration of those documents into our current systems saves roughly 30 hours of time for our operations team. The culmination of all of this is when you have 10 or 12 data points, It's a tremendous efficiency enhancer.

32:30Matt Bank:It allows our team to spend more time on the value-addive work that hopefully helps us make better investment decisions. We had an AI day for the team, internally for our team. We prepared a demo for our board and we prepared a podcast. One thing that's fascinating with the podcast, we decided to use an AI-generated voice. We put about two hours of previous podcasts that I've done for our board. AI took that. I found it remarkable as far as the likeliness of my voice as well.

32:58Jenny Heller:What are some of the areas that you dove into hoping AI would help but hasn't?

33:03Matt Bank:The real challenge is aggregating our data into one centralized data lake house, which we're working on. It's easier said than done. That has taken more time than I would have expected. I was hoping we would have had this six months ago. We're still working on it. I'd love to hear what your tech stack looks like. On the front end, Canoe has been fantastic for us as far as automating, processing, when we have incoming documents and financial statements. We're building a centralized data lake house. You can use Microsoft or Google. On top of that, we can utilize any platform, ChatGPT, Quad, Google Gemini, Microsoft Copilot.

33:42Matt Bank:We also use a tool called Hebbia, which has been productive for our team. On the back end, assuming we have our data lake house in a good position, then we can utilize any one of the LLMs to access our data. I'm hoping we can utilize our data to identify patterns. At scale, it's hard to connect all the data that we have. Allocators have tremendous amount of access to managers, to letters, to peer conversations, to conferences, to podcasts. Listen to your podcast, Ted. But aggregating all that data together can be a challenge and then making better decisions. That's ultimately where we want to get.

34:19Jenny Heller:What have you found have been the most effective uses of the various LLMs?

34:24Matt Bank:I found ChadGBT helpful from a qualitative perspective. Claude, when we build financial models, helpful from a quantitative perspective. Using Hebbia has been helpful building presentations for the investment discussion. Tying some of the analysis together, putting this into our MC rice management company template, which we have in Hevea. Tremendous efficiency gains across the board for all three of those platforms.

34:53Jenny Heller:As you look out over the next year, what are the steps you're hoping to take to get to the point where you're impacting alpha generation?

35:02Matt Bank:It's going to be proven at some point in five years or not if we're successful. I'm certainly hopeful we get there. a 10 basis point increase in annual returns would be worth it. I'm hopeful to find patterns and information that can allow us to make one or two better decisions a year. Minor increments and improvement can have major compounding effects in the future.

35:23Jenny Heller:In case you're wondering about John's AI-generated presentation for his board, here's a short clip from the introduction. Good luck telling the difference from John's real voice.

35:34Matt Bank:Before we begin, I want to acknowledge that a considerable amount of this content and discussion you are about to hear was generated with the assistance of artificial intelligence. In fact, this is not the real voice of John Lawrence. The narration for this presentation was also generated by AI.

35:52Jenny Heller:John explains why getting proprietary data organized is essential to improving decisions. Matt Bank, CIO of$14 billion OCIO GEM, is rethinking the investment workflow itself. He also describes how dramatically cheaper software development is allowing GEM to build customized tools internally instead of relying on outside vendors. Matt, how have you thought about using AI strategically at GEM?

36:23Kristin Kallergis Rowland:Phase one for us was building out a foundational enterprise tool set, some internal development capabilities. We needed to enable secure, decentralized experimentation across teams because everybody has a slightly different workflow. We're encouraging everybody to think like a shop floor foreperson, deconstruct their workflow, sourcing, diligence, monitoring managers, portfolio management. Every stage has some bottleneck or limiting factor. You start with the lowest order stuff, and that has been time-saving on data aggregation and retrieval. No taking apps. That saves junior team members time in cleaning and parsing, particularly for introductory calls we have with managers every week.

37:09Kristin Kallergis Rowland:For our marketables team, there's a lot of meeting prep that goes on, reviewing old call notes, letters that might have been written over the past several years. For a quick refresh, we want to be able to pull all that together. in a thoughtful way. For our private team, it's sector research. We do a lot in the independent sponsor market. Trying to get a baseline on a new sector is important. We're not so arrogant to think we're going to know more about geofabrics extrusion relative to what the GP or the owners of that business know. Can we get smart enough to ask better questions? In my seat, the bottleneck is consuming research, strategists, papers, podcasts.

37:47Kristin Kallergis Rowland:My commute is only 20 minutes. The feasibility of getting through all of the ones I want to get through in a week. It's not a reasonable exercise. The next level is doing things more quickly and deeply. How do you cut to the chase through data faster? For current manager meetings, that's monitoring theses through time and space. How have the stock positions evolved? Are there any inconsistencies in the way a manager's talking about that? If you're on a biotech call and someone's talking about a royalty stream on a drug. Let's get some comps pulled together within the next five minutes and figure out how we might think about valuing that.

38:23Kristin Kallergis Rowland:Where we've stopped short so far is decision-making an agency. We want to make sure everything we're doing is intentional. That last piece amplifying the pace of decisions we've not executed on in part because we're cautious about what the bleeding edge looks like and making sure we're being extremely thoughtful with client capital.

38:42Jenny Heller:As you look across that order from sourcing all the way to decision making, what tools are you using?

38:51Kristin Kallergis Rowland:Claude, which has been a critical driver of internal development. Granola is the note-taking tool that people have congealed around as most useful. We have an internal search tool called Glean, which is model agnostic. It has basically driven the cost of search across the firm's various data sets to zero. So despite everybody's best efforts to save things in a file structure that's useful that you can retrieve information from, inevitably over 20 years, things get hard to find. We've been able to stitch our SharePoint in with our Microsoft Teams, in with our CRM BipSync, in with Chronograph, which monitors PE-related cash flows.

39:30Kristin Kallergis Rowland:And you can just go on to this thing and ask it to pull various information. We continue to use a number of things that GEM alumni have helped build over time, OWL being one of them. Campbell Wilson's firm is terrific. They pull from a lot of interesting data sources. They are using AI in thoughtful ways. That has plugged in nicely with our map of GPs and LPs. We get a spit out every week from them in terms of position level changes that they're seeing at the managers we have. However, if you rewind the calendar a year, about 90 % of our spend and our tool set was externally developed. We were buying off the shelf things, trying to integrate them.

40:11Kristin Kallergis Rowland:Today, that has flipped. 90 % of the things we're developing are internal. We've got a team of three developers working full time on building tool sets that are specific to our particular use cases and objectives. One example would be they're in process on a sourcing tool that will instantly identify when title changes have happened at various firms, when deals have gotten done by certain people, people have left firms. It will plug that into a relationship map that helps the sourcing team get out in front of prospective manager relationships in the future. That's been helpful because you're not relying on the marketplace to invent this stuff or working with third parties.

40:53Kristin Kallergis Rowland:We're doing it directly. Our team generates about 30 ,000 lines of code per month. I'm told that's a lot. I don't know for a firm of our size, but it's up 10x from where that was last year.

41:03Jenny Heller:What are aspects that you've been hoping you could use AI to enhance that haven't worked?

41:11Kristin Kallergis Rowland:There's agentic chatter about the capacity of the tool to take workflows end-to-end and execute on them. We haven't really seen that in a cost-effective way work. We've had to meter some of our token usage because of things that were way out of bounds with respect to what the benefits were. We're deploying capital to primarily third parties. There's so much unstructured data still that we can capture that will be time-saving and enhance the quality of our insights that we haven't really spent a ton of time on some of this. Well, let's solve this particular workflow. That'll save us three FTEs.

41:51Kristin Kallergis Rowland:We still have all the same people. In fact, we've added folks. We will continue to add folks who will be capable because of this tool, but it is by no means a panacea for operational expense. Part of the challenge is we've got a few folks on the team that feel strongly that the junior team members need to dig through things, learn how to prosecute the process in a way where we haven't been eager to make their lives that much easier. The core function is building out the dashboards, understanding the pattern recognition, making sure you're in the flow of the information in a way where you understand where all the pieces fit together, and then advance the ball.

42:28Jenny Heller:As you look out, what do you think you'll be using AI for a year from now that you're not today?

42:35Kristin Kallergis Rowland:There's a ton of room between where we are and optimal in terms of our ability to capture information. We are effectively in the Mosaic business. We are responsible for taking in an evolution of facts and circumstances and trying to parse out whether we think they're predictive of the future or not. Splitting that up is a critical task of manager selection. What if instead of the numbers, you could capture all the data? You could get the team out in the field, traveling more, meeting with folks, all of the subtle cues. If you could somehow organize those, get them into a system that allowed you to draw conclusions, see patterns.

43:14Kristin Kallergis Rowland:patterns, anything that can continue to push the flywheel faster. That's the thing that if we could do quicker and better, we'd be more effective. We want to see everything that's worth seeing out there. That's a fantastic ambition. The fun thing here, I tend to think of this more in a cosmic sense. Ultimately, you want a firm brain. We don't want to let the team and the thought processes that go into this atrophy, as long as we have sufficient agency, the flywheel is beginning to been a lot faster and we're confident it's going to continue to.

43:47Jenny Heller:Matt takes us to the edge of using AI in actual investment decisions, but deliberately stops short. KK Rowland is embedding AI across the investment lifecycle. KK is global head of alternative investments for JP Morgan Asset and Wealth Management, where$250 billion of the private bank's$500 billion in AUM is under her watch. Her team is currently using agents to create manager profiles, engage in red teaming, and generate signals that influence capital allocation. KK, how are you using AI inside the team?

44:27Jon Webster:Alt in particular runs on documents. Public markets are structured and normalized decades ago. Alt never were. A fund arrives with a private placement memorandum, then side letters, performance comms, as a GP reports, in whatever format they choose. At every stage, we've had this skilled person reading documents, retyping the one that says that work was slow, it didn't scale, doing fund research, how we collect data from fund managers, how we move into a diligence phase. We pre-populate the first draft of an investment review committee memo. We do what's called a rent team before we go to Investor Recovery.

45:06Jon Webster:We use an agent alongside us to play devil's advocate in that process. The ongoing diligence, making sure that we're staying in check with what the initial investment thesis was. AI is typically taking that first path. We get that data in on a monthly or quarterly basis. It's also structuring and onboarding, negotiating terms, doing the compare and contrast is really helpful for us. We have a lot of portfolio construction tools where we've embedded AI in terms of how we generate simulations. The last piece of the process is all the servicing and the operations. It's taken off probably a third of the workload of people just in the last six to nine months.

45:46Jon Webster:AI is embedded into every aspect of our work stream. It's not bolted on the side. It is a part of everything.

45:53Jenny Heller:Let's dive into some of the specifics.

45:55Jon Webster:Within JPMorgan, we have this go slash LLM suite where we keep all of our client data in-house. We're using AI to organize the data around the tracking the themes. So that could be everything from what's going on in healthcare and biotech right now, mixing across public and private information that we have available, all the way to Japanese corporate governance reform. Using AI agents to ingest the news that we're seeing externally, as well as the information that we have internally. The other places in AI is in these profiles. We create internal notes when we onboard a manager to say what the profile of that manager is.

46:33Jon Webster:We have these living profiles to make sure that when we get call notes as words that they use change to describe the same thing as before, or an investment thesis changes, how that's relating to the risk parameters that we put around a manager. We'd use a lot of AI when it comes to coaching our own professionals as we bring on new people or as we try to educate our investment committee, our advisors, and ultimately our clients.

46:58Jenny Heller:I'd love you to jump into that profile of a manager. Walk me through how that works.

47:05Jon Webster:Every manager looks somewhat different. Within an investment review committee, we have a lot of the same initial criteria of what we put out there. What's their philosophy? What is the process by which they've generated that? What you want to make sure is that what a manager does is repeatable. In investment review committee deck, it's setting up the agents to know the thesis that you originally underwrote, whether it was a year ago, three years ago, five years ago, matches the outcome of where that return is. When there's outsized winners or outsized losers, you make sure that it was in line with their original risk expectations.

47:42Jon Webster:AI is able to help us do that because they're matching data to what we originally underwrote versus be swayed by the headlines that are coming in and out.

47:49Jenny Heller:You mentioned using AI for your red teams. We'd love to hear how that works.

47:55Jon Webster:We use AI to help us compare and contrast. We're on a fund six. We ingest the first five funds to help us preempt some of the questions that we're going to have about, is the firm growing faster than the people have grown? There's been a shift between funds three and four versus funds five and fund six. It starts preempting some of the questions that we should be thinking about so that when we do go on site with a manager, we can highlight specifics around some of these attributes. The second thing we'll do is strategy mandate. We'll do private funds as an example. They'll say, had the average hold changed?

48:28Jon Webster:How quick they are to put the money out to work? It goes through that level deeper so that we can get quicker to the key points that we think are driving the wins or the losses. The third thing in a red team that we look at is an investment process. From an AI perspective, a lot of it uses some of the call notes that our operational diligence team has or our investment diligence team has, compares and contrasts over prior calls to see if there's a shift in tone, shift in people, shift in how they talk about value creation. AI is picking up on a lot of those transcripts that we're using to then preempt the question about, is this something we should be worried about?

49:04Jon Webster:It's amazing because we can create these agents and we can share them across partners and people so that we can see if there's something that someone on the growth equity team did that the private credit team wants to think about. We're using our agents across all those things to create synergies across creating red teams and deviled advocates before we go to an investor-hury committee so that we're best prepared to answer any questions that might get thrown our way.

49:25Jenny Heller:What tools are you using to integrate that?

49:27Jon Webster:There's a couple of vendor tools that we use, AlphaSense on the backend. We use all of the models. We have a system within our LLM suite that helps us understand which one's the most efficient. We have what's called Connect Coach, which connects us internally to each other and then our data to advisors and clients. We're using everything in our closed loop ecosystem. There's not a model we haven't used or haven't tried.

49:53Jenny Heller:How have you thought about using the different tools to improve your alpha generation, your decision making?

50:01Jon Webster:A lot of it is still to be seen. What's here and measurable is the LOM signals that we're getting, where we get sentiment from a transcript, from our own transcripts where we're questioning a manager, rethinking forward-looking risk factors. It's allowing us to better manage portfolios in the world of hedge funds right now. There's a lot of features that can be generated from tech sources, as well as the numerical data. There's many more managers that are not buying or analyzing the raw alternatives data out there. In general, the alpha generation from us is more of the signal that we're seeing to recognize that there's an emerging risk or a confidence factor in a manager, something that might make us lighten up exposure or use that next dollar elsewhere.

50:47Jon Webster:A lot of it is in our uncorrelated hedge fund portfolios where we can look at macro managers, plot and relative value signals that we're seeing and incorporate that. We're seeing some alpha generation already because we've built Spectrum IQ and some of these other systems that are able to help us identify some of these things.

51:05Jenny Heller:What were some of the obstacles you had to overcome to deploy these tools into the investment process?

51:11Jon Webster:Three things. One is the walls of JP Morgan, our own cybersecurity. Two is that because we have our own applied AI lab, we have a pretty high standard for the things that we want to use. I have over 50 engineers that work on$250 billion of assets that we oversee. Everyone's an AI engineer. The third thing is the complexity of being a business that had over 40 years of data to ingest, to understand the nuances of the things that we had from the 90s up until today. That took us almost two years. Every time we go to create what our tech agenda is for where we want to deploy AI next, it would take us between two and a half to three months to implement that idea.

51:55Jon Webster:Now that we have AI embedded in every aspect of what we do, it takes us two to three weeks to get something mocked up.

52:03Jenny Heller:What's on the agenda for the next year?

52:05Jon Webster:Taking what we've created from how we've built our own portfolio tools and allowing our clients to see that there's a lot that we're doing in terms of agentic portfolio management. We are going from delivering something within 45 days to four to five days. That's what I'm looking forward to is making sure that it's clean enough to get as quickly as possible.

52:28Jenny Heller:We finish with John Webster, Senior Managing Director and COO of Technology and Operations at CPP Investments, the largest of the Maple Lake Canadian pension funds with$580 billion U.S. under management. John brings all these ideas together at institutional scale, using AI to read everything, remember everything, and challenge everything. John, tell me how you're using AI at CPBIB.

53:01John Lawrence:The test for AI is, does it make us a better investor, better underwriting, better risk-taking, better decisions on behalf of 22 million Canadians? AI now touches every stage of our investment lifecycle. If there was a tagline, it's trying to get AI to help us read everything, remember everything, and challenge everything. The intent around read everything is cross all of our investment teams across total fund manager, the office CIO, bring as many eyeballs to bear on everything that we would have wanted to look at the past but could never reach. Natural language processing with LM is allowing us to read filings and earning calls transcripts across a much broader universe than any team could have historically covered.

53:40John Lawrence:In places like secondaries and private equity, we've scaled our capacity to price under our LP portfolios with tens to hundred plus underlying companies in a single transaction. So we evaluate more deals. Our entire deal history is now querible. So that means for things like comparing governance rights in a new deal against the existing book, GP intelligence, we can now draw on-demand relationship briefings from our own CRM pipeline and performance data. So every GP conversation is backed by our full institutional knowledge. We've put in place an investment committee co-pilot, which encodes senior investors' decision heuristics, plus proprietary deal history into diligence and investment committee prep.

54:15John Lawrence:Even at the individual IC member level, trying to make sure that we bring our entire institutional memory as an asset, as much of the individual investor's institutional memory into everything that we do. We've got a lot of places where we're using AI to challenge everything we do. We have a memo coach, which is a multi-agent reviewer that stress tests, draft investment committee recommendations through a whole range of distinct lenses, logic, risk scenarios, value creation, even how the case is communicated, with the intended bringing forward a prioritized set of questions and actions before the memo reaches the investment committee.

54:48Jenny Heller:I'd love to tackle how you went from zero to where you are today.

54:53John Lawrence:When ChatGPT 3.5 popped out of OpenAI three and a bit years ago, we made a deliberate choice to put the best frontier platforms in front of all colleagues in the organization as quickly as possible. We tried other products before, but we felt the integrated work surface that you got in an OpenAI or an Anthropic suited very well the way our investors think, work, and make decisions. So a lot of those things came out of the real understanding of our investors on the front line, making decisions, working with our partners, and they could then start to see how they could use the technology to scale and change parts of the investment decision making process.

55:32John Lawrence:We're not saying there's the portal where you go to use AI. There are workflows and existing applications that we're building underpinned by AI. So we're trying to meet users where they do their work today and bring the best tools and skills to them in a way that is accessible. Getting people over the mental hump of being brave enough, courageous enough, confident enough to keep moving into the leading edge is part of what we're trying to do as well. People are going to have to be flexible and willing to lean in and use it in a variety of different ways. Wiring it together across 2 ,000 people in a way that reflects the heterogeneity of how people want to look at the world and their specific skills and judgment in a way that does bring the institutional perspective is non-trivial.

56:14John Lawrence:But this technology is giving us the best chance of doing it.

56:17Jenny Heller:If you look at the tools that you're using that are most effective, what are you using inside the organization?

56:22John Lawrence:In order to read everything, you have to get your information into order. But we've spent all the time across the last couple of years making sure our structured data is searchable, traceable, can be leveraged across the organization. And we've took a lot of time into making sure all of our investment recommendations of the past are old-fashionedly indexed and available. And then we've made the unstructured data available via knowledge platform, RAG and Knowledge Graphs. You can now connect the Knowledge Graph into Clawed. You can connect it into OpenAI. You can connect it into Copilot. We do use OpenAI.

56:54John Lawrence:We do use Anthropic. We make those available in web-based chat form. We also make them available in the desktop form so people can use Claude Code and Codex. In the next nine to 12 months, we will start to converge more on a consistent application landscape. Some of that we're waiting for the market to figure out what's going to win. There's a lot of choice out there at the moment, and the choice is good. It also means that as a buyer of enterprise technology, you're having to keep your options too open at this point in time. Most of what we're doing is making sure people are feeling fully conversant, very literate, past the 20 hours of working with AI, where they then start to understand how to get the best out of the machinery.

57:31Jenny Heller:What applications have you found effective when it moves towards the challenging and decision-making?

57:38John Lawrence:If you look at the IC Memo coach, we have a privates platform that we've developed over the last five or six years. We found that the ability to embed lots of different challenge perspectives, being able to quickly understand the skew of upsides and downsides or quickly extract the beliefs on which the thesis is set out, where the evidence for that is strong, where the evidence is sparse, what metrics you might want to further go after, what things you want to monitor, bringing lots of different perspectives, which would be computationally hard in the past to do. No one lens lets you see everything.

58:13John Lawrence:If you can look at it through 10 different lenses relatively quickly, you can start to get a different sense of where the strong parts are and the weak spots are.

58:19Jenny Heller:As you have dove into all of the internal dealmaking and public investing, curious what you found effective for the teams that are tasked with external manager selection. I've probably just got a strategist's view on that.

58:34John Lawrence:Technology never confers a lasting competitive advantage because whatever is available to you is available to everybody else. If you believe that as a base proposition, then what this technology can do is either amplify or undermine your existing competitive advantages. Our competitive advantages are organizational EQ. It's not to dismiss the IQ half of the equation. We are very bright people, think carefully about investment thesis. Our advantages are ultimately in the trust, the relationship, the standing, the patience. Clayton Christensen, law of conservation of attractive profits, when one part of the value chain is attacked, then value accrues to the adjacent parts of the value chain, if IQ is under attack, value is going to accrue to EQ.

59:18John Lawrence:Whilst it's really important we are forward on the use of the technology, one of the things all organisations have to do is work out, are they really competing on the IQ side or are they competing on the EQ side? Lots of our partners compete on the IQ side. They're leaning heavily into the technology. We are leaning appropriately heavily into it, but I hope with an understanding that what we do brilliantly might not be to become the next smartest investor, but make sure we continue to work with the smartest and best investors in the smartest and best way.

59:48Jenny Heller:What are some of the things you have tried using the LLMs that haven't been as effective as you would have thought today?

59:55John Lawrence:If you roll back three years, I would have looked at the machinery as a way of bringing many different challenge perspectives to a problem. The more ways you can challenge a problem, the more ways you can find insights that others haven't thought about. I don't think that's been true. You've got to be far more deliberate about what are the lenses you want to bring to a problem? How do you codify those lenses? How do you make those lenses consistent across the organization? The machinery and the platforms haven't caught up with doing that well at this point in time. What are some of the risks you've encountered?

1:00:25John Lawrence:The big risk is that we put a lot of effort into this and it does not make us a better investor. That's the one we're focused on most. continuing to push ourselves to understand how we can judge whether we are making better investment decisions. It's still early in the technological revolution. We are getting to the point in the next 12 to 18 months where you've got to feel confident that you are getting a return on investment of the technology. In our case, that means we can really feel confident we are making substantially better investment decisions commensurate with the investment we're putting into it.

1:00:59John Lawrence:The areas I outlined earlier, those people feel confident are real. benefits real opportunities. Now, the test case is always in a long term investment opportunity, which might play out over seven years, how do you know that it's helping you make a better decision that turns into a better outcome? We do need to figure it out.

1:01:14Jenny Heller:Thanks for listening all the way to the end. If you made it this far, how about one ask for me? Tell one friend about the show. It's the best way to keep growing this incredible community. Thanks so much. And until the next one, stay curious and keep compounding knowledge, relationships, and capital a little bit at a time.

1:01:38Abby Barlow:All opinions expressed by TED and podcast guests are solely their own opinions and do not reflect the opinion of capital allocators or their firms. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of capital allocators or podcast guests may maintain positions and securities discussed on this podcast.