How Capital One Delivers Multi-Agent Systems with Rashmi Shetty - #765

27:30Rashmi Shetty:How do you approach the speed at which models evolve in this space? Yes, this is not unique to Capital One, right? But there is a nuance there as well in terms of Capital One's risk-first platform strategy coming to help us, benefit us in this specific space, right? As new models come into the fore, there are specific layers that has been built in in terms of provisioning, inference layers, inference optimization layers, and benchmarkings that makes the evaluation of new models quicker and easier, easily provisioned within the platform infrastructure. So whenever there's a need and there's a new kid in the block, we have mechanisms to rapidly experiment on it, deploy and experiment on it.

28:28Rashmi Shetty:And does the presence of inference and inference optimization imply that you tend to favor self-hosted models? This is another core underlying philosophy of Capital One, right? When you are the most successful, when you can offer two things, reasoning and specialization. So reasoning capabilities with our agentic platforms, agentic frameworks in the platform, we are bringing that to the fore. Specialization is something that is very, very crucial, right? So for going back to the chat concierge use case, for us to provide that personalized experience to our customers. And we are privileged to be in the position of catering to many such lines of businesses who have similar needs of distinct personalization needs for their customers.

29:22So this can be achieved primarily using specialized models and fine-tuning. Student distillation gives you that control you need to have on providing personalized experiences as well as having some control over your latency metrics. So, yes, that's been our strategy.

29:49Rashmi Shetty:Which goes back to the point you made earlier about the value being the data, right? That's a unique asset that you have and can build around and doing so requires customizing the models. Right. Exactly. Exactly. I think that's one of the easier questions for us to answer, right? Specialization is not an easy task, but I think for us it's easier because we have all of the pipelines in place. Yeah. When you think about the chat concierge effort and other efforts, like what, and thinking broadly from the perspective of, you know, things that listeners might encounter in their own environments, like what are some of the key architectural decisions or compromises that come into play when you're building a system like this?

30:39Yeah, so I think one of the few things that I did already mention to you about while we are approaching this, we already kind of thought through in terms of a risk first platform approach, although it was our Beachhead customer, it was the first application, the agentic application that we were putting in production, we already had a platform mindset from the word get go. So that is one of the architecture decisions that we kind of made, if you may say. So the other thing that we were thinking of is in terms of a full stack approach, thinking through a full stack approach of, okay, where does our data reside?

31:20Where does our model optimization strategy stay? How does our across layer monitoring look like? How does it need to be optimized for latency? Our strategy around specialization using open source foundation models. Our approach for multiple agents to fulfill this complex goal that needs to be met in terms of understanding which of those different agents are going to be provisions. How does governance look like at this agentic layer? How does execution of tools be provisioned? So essentially, I mean, and how do we improve any part of the stack based on production learnings? So this is famously known as the orchestra problem, right?

32:12How do you orchestrate the entire lifecycle from pilot to production back to experimentation? So these are some of the key decisions that we had to make quickly to understand, you know, how do we optimize for latency for smaller specialized models? How do we optimize for cross model conversations? How do you optimize the context? Whether some specific techniques had to be taken in terms of choosing the right model for the right task. Yeah, and definitely the other decision is human handoff. What is that juncture at which human handoffs need to be done? What are those governed or policy-based actions that determine human handoff?

33:01Rashmi Shetty:To what degree do you rely on agentic standards like MCP or A to A or A to I? Like there are a lot of these emerging approaches to agentic standards. Is that something that you, you know, you're thinking about and worried about or are you, you know, mostly trying to leverage what you already have and not necessarily trying to apply these external focused approaches? No, that's the purpose of the platform. The purpose of the platform is to abstract away any complex underlying technical decisions that have been taken. For example, as you mentioned, tools, whether we go in for MCB or native tools, local tool calling.

33:45These are all things that are abstracted within the platform and the customers are open to, you know, just leveraging what we are exposing to them in terms of accelerating their journey. So these are some things we absolutely keep in mind when we are building our platforms. So we have definitely a standpoint and viewpoint on how we approach MCB, A2A, and all of that decisioning is baked into our platforms.

34:11Rashmi Shetty:You mentioned human in the loop a few times. Talk a little bit about those considerations and how you make it easy for developers to incorporate humans into their agentic systems. Right. These are a part of the agent orchestration, right? And along the multi-agentic architecture, these are junctures where we need to decide the bounded actions that agents need to specifically take. Where is that juncture? Where is that node where human needs to be brought in for validation? So this is a classic multi-turn, multi-pass agentic architecture where one of the decision makers could be a human. And these are things that we bake into via the orchestration mechanism and the frameworks that we expose for these orchestration mechanism in terms of, okay, where is that loop interrupted?

35:05And by a specific human after which the loop can continue.

35:10Rashmi Shetty:To the folks on the Capital One side who are monitoring the systems and those humans that are pulled into the loop, like, how do you think about that from a user experience perspective? Are you rolling out like agent inboxes to folks to kind of centralize all the decisions that they need to make or approval steps around these agents? Or are you integrating into, you know, existing channels, whether it's Teams, Slack, email, that kind of thing? Yeah. So I think one thing to understand is that human loops are typically deployed in high risk use cases, high risk scenarios. Most of the times our lines of businesses already have a strategy for how this user experience is delivered to their end customer.

35:56So meaning that there is a specific strategy in which these human loops, there could be a 24 by 7 support system, a support mechanism where the response times, it's all baked into the end to end SLA to your final customer, right? So the delivery mechanism in which the human is involved in the loop, every customer has a unique nuanced way in which they want to get it done. In some cases, there's a human gatekeeper who's always there. And in some cases, it's an offline notification. So as a platform, we need to be agnostic to how this notification mechanism needs to be. And we have to have an easy way to embed it within whatever is the operating environment within our line of business.

36:39Rashmi Shetty:And then when you think about this from the platform's perspective, like, are there, what are the major components that a user's interacting with? So when you say what are the major components a user is interacting with, technically the user is interacting with every single layer of the component where a service exists. So these services are brought very, very seamlessly to the user where there is a need for configuration. For example, guardrails. There are many types of guardrails. If the user has an input to be provisioned, experience is delivered to the user in a homogenized manner, but they are actually interacting with the actual service, underlying service, which needs to be configured.

37:31This goes for pretty much all of the layers, right? guardrails, memory, any kind of retrieval systems that you have in play. Yeah. So, yeah.

37:41Rashmi Shetty:I think I was imagining the platform there and trying to get a sense for how many boxes are in that architecture and thinking that it was probably a lot and wondering what they were. Yes. Yes. I think we hide the complexity of it. Okay. Okay. So, yes, there are a lot, but there's abstractions and you try to hide the complexity. What does that look like from a developer perspective? Are they, is it, I think that question, you know, what I'm thinking about in that question in particular is, like, are there, you know, libraries that the, and the answer to this is yes, but right. Are there libraries and they have to understand like this, you know, the model that's exposed by these libraries or are there like systems that they're clicking through to configure things?

38:35Rashmi Shetty:Probably a little bit of all of the above. Multiple layers of everything. So it's always a journey, right? What you're talking towards is, you know, a self-service journey for customers. There are multiple layers to it in terms of, you know, how they interact with the system. There are CLIs, there are SDKs, there are UXs, UIs. We have all the ranges. depends on which persona you're trying to serve. So, yeah, we do have all of those. And maybe the thing that I'm trying to move towards here is from the perspective of a person that's providing this platform, how do you manage the complexity for teams that are new on this multi-agentic journey?

39:19So I think customers that are new on the multi-agentic journey are one of our more naive users, right? So we have to offer them pre-built blueprints that they take and run for their use cases. There are savvy customers who know what they want to do and you offer them those developer kits. So we have all of these different ranges that we offer to our customers depending on where they come from, what their use case is, and how can they get to the fastest path to production in the least constrained manner as possible while we make sure that the execution of this is safe and scalable.

40:03Rashmi Shetty:Can you give us examples of the kinds of things that have been kitted so we get a sense for like the level of abstraction that you're operating at? Is it like customer-facing agent versus, you know, workflow agent, that kind of thing, or is it higher or lower? Yeah, well, internal and external agents definitely have a completely different risk profile associated with it. But that is all governance. That's all policy and governance. In terms of the developer experience, there are different kinds of use cases that can come into the fore. And I think it's pretty known in the industry what are the different levels and different bands of use cases that exist.

40:46you have simple summarization scenarios or complex multi-agentic scenarios like you just saw which is the chat concierge so we have this range how does your team interface with like the lower level

41:01Rashmi Shetty:infrastructure i've spoken to teams there that have operated like the kubernetes environments and things like that is that all hidden behind apis that your team has built to manage that lower level infrastructure? So as I mentioned, this is a cross-functional effort to kind of get all of this experience out the door for our customers. So we do have, as I mentioned, an inference optimization layer, which we partner with. They provide all of the hard work with regards to making sure that the right models are hosted in the right environments and optimized for inference are provisioned through specific SDKs and APIs.

41:44And these are SDKs and APIs that get embedded within our own systems along with the other services that we offer. For our customers, it doesn't matter, right? These are all different services, multiple of which you need to kind of go through with your agentic development journey. So the underlying infrastructure, the way we look at it is like we are all teams delivering unified GenEI SDKs for our agentic experience for our customers.

42:12Rashmi Shetty:You talked about on a couple of occasions the closed loop nature of agentic systems. What have you created to facilitate that? Traditionally, you have your golden data set and the way you close the loop is you you know, take runs that are either outliers or, you know, issues and you put them back into your golden data set for the future. In the, you know, Gen AI world, now you can, you know, change your prompts, you can, you know, change, you know, other aspects of your context to facilitate incorporating in like this closed loop, you know, signal. How do you think about that approach? As a platform, the platform's role is to make sure that there are hooks and tools provisioned along the closed loop journey and for the customers to come up with their own mechanisms in which how they want to specifically do it.

43:17So you have observability, which observes specific failure modes. Signals for that failure modes are captured. Those go back into the experimentation environment, which can be used by customers. And they can tune whatever they wish to tune to, you know, kind of realign their agentic systems to the newly changing environment in which it's actually operating in. So multiple hooks, some of which you already mentioned, you know, pure prompt tuning, model tuning, context management, retrieval, grounding, any of these. There are, these are all available in the experimentation environment. Got it.

43:56Rashmi Shetty:So then it sounds like from the platform perspective, you know, the key for you is to ensure that the data pipeline is solid and that they can access the data at various places where it exposes the signal that they're trying to take advantage of. Correct. Correct. Exactly. you. You know, talk a little bit about the, you mentioned the model risk officer or that model risk office that you have there, you know, there, anything you can, you know, share in terms of, you know, how agentic types of, you know, approaches has changed their relationship with that group? Like, you know, certainly they've been on this Gen AI journey with you, so they've gotten comfortable with LLMs versus traditional models.

44:47Rashmi Shetty:Does agentic change that at all? No, no, no. We are partners in this journey, right? I think that's one of our strengths, biggest strengths in terms of banks. I think the world is waking up to agentic governance now and trying to build frameworks around it. We have a robust framework. So these are our strong partners that help us understand how to bake into any new systems modern stacks, tech components that we develop, and how do we bake into what are the current regulatory compliance policies into this framework? So we partner very closely with this organization. So I wouldn't say how this relationship has changed.

45:31We are partners delivering towards the same goal. So I think that's one of our biggest strengths as a tech organization operating to run the bank.

45:41Rashmi Shetty:Taking a step back, you know, if you were talking to folks that have, you know, similar enterprise requirements, but are maybe further behind in the journey, you know, how would you talk to them about, you know, lessons learned, best practices, things that have made your successes possible? I kind of answered a little bit along our discussion topics today. So one of the biggest lessons is that treat - You've dropped a lot of breadcrumbs for sure. True, true. It's time to reassemble that bread now. So I think the core is that you really need to treat agentic AI as a system. It's truly a system.

46:27You have to start with governed data. You have to kind of put in that risk controls baked into multiple layers of your application or your system. You have to look at latency as something that needs to be optimized end-to-end. Take into consideration handoffs as well in this end-to-end experience for your customers in terms of latency expectations. And understanding that your biggest gains do come from post-production telemetry is also critical. I think, yeah, basically in a nutshell that, you know, as you move from the AI ML to the Gen AI world and now to the agentic problems, I think bridging that, bridging that world is important to understand.

47:12You don't throw away, at least for Capital One, we had a very, very good legacy of tech stacks that we had built, which we are leveraging to quickly jump onto the agentic bandwagon in the last year. So for us, especially in our case, reasoning became explicit. Specializing became modular, became very easy. Yeah, so data quality, governance, integration, keeping an eye on end-to-end latency, post-broad improvements. I think these are some of the key things that I would say that key lessons for anybody else wanting to go along this journey.

47:55Rashmi Shetty:For folks that are in similar platform types of roles, is there a different slant or take on that or more nuance or detail? For folks that are thinking about expanding existing platforms or building out new platforms for that matter to accommodate agents? Yeah, I mean, some of the nuances that I mentioned, you need to understand what services you are lacking to build this agent capabilities. What are the policies that you're lacking to build these agentic possibilities? Look at observability from a completely new lens. Instruments sooner. Define escalation thresholds. I think the problem of building agents has been solved for.

48:43We are moving very rapidly into a world where agent execution and governing of agents in execution environments is becoming more critical.

48:53Rashmi Shetty:Talk a little bit about kind of next steps for chat concierge, your team, agents at Capital One broadly. I think basically we have the necessary components. It's now scaling. We are at a lifecycle phase within the agentic lifecycle, development lifecycle, where we are now figuring out a scaling problem. How do we get across to millions of customers at the same levels of speed and governance and scale it out to multiple users, both from a developer standpoint as well as from a consumer standpoint. And when you break that down, like what are the, like how do you get there? Like, you know, certainly part of that is like technical scaling.

49:46Rashmi Shetty:Like, what does it mean to run agents and, you know, many, many agents in production serving many, many customers? But I imagine that there's organizational scale as well, onboarding scale. Like, how do you think about the various components of that? Yes, it's very nuanced, as you rightly mentioned. Like, there's definitely a technical component associated with it in terms of there's also a strategic component associated with it in terms of how do you manage your fleet of agents? across your company, across different lines of businesses. All of these things, you know, it's like kind of keeping in line with what is happening in the governance world around agents, keeping a close eye on it while you keep your scaling standards to these evolving standards out there in the industry and everything that's new and out there.

50:41So there's definitely a strategic nuance to that. But as with all software deliveries, you have to kind of figure out how your tech stack scales. And I mean, we, again, I'll go back to how we said that we are cloud-enabled, cloud-native. And a lot of these elasticities baked into our underlying platforms. It's just about, you know, kind of observing as we scale what we need to adapt or readapt based on changing standards.

51:12Rashmi Shetty:Are there particular friction points that you've observed in the relationship between platform and developer? Are there kind of next steps that you're pursuing in that direction? I don't believe this is, friction is the right word again. We're all partnering in terms of making sure we as a platform provider are providing to our developers and consumers. Developers, on the other hand, we have the benefit of a very fast feedback loop from all our user bases because we are all in-house. So I think it's more than friction. I think there's no time for friction. The world is moving at a rapid pace. So I think we have mechanisms that we have built in in terms of getting this rapid feedback loop with our internal and external users so that we can kind of get that back into the platform.

52:11Contributions from our partners is a huge part of it.

52:15Rashmi Shetty:Yeah, and I didn't mean friction in a negative way. It's just like it's a lot for developers. And, you know, when I'm out talking to folks, I encounter folks from, you know, a perspective of like, you know, I just still don't really believe in this stuff. It's broken. Like it hallucinates. Why are we doing this to, you know, let's go, go, go. And, you know, not to mention if you, you know, once you're in go, go, go, like there's just a lot to know and learn. I think one of the key things to remember here is that we don't take away agency from our developers. they need to have the agency to know to you know be able to configure and play around the system in uh based on their line of visions that is key so what you do is you offer all the toolkits developer kits tools framework services observability is key right to kind of give an agency as they observe the system they know what they need to do so yeah i mean learning curve yes like it's a compounded learning curve that we're all in.

53:15It's massive. Every small evolution in the past is compounding. So for example, we were in a world where data was commodified, like big data the past decade. That became storage, became cheap. And then compute became cheap. And with each of this, we are in a compounded learning curve right now. And our developers are experiencing that as well. So how do we bring the seamless experience to them in a way where it is beneficial for them so that they can jump onto the bandwagon very quickly without having to worry, should I, should I not? I think that's the goal. That's the goal.

53:53Rashmi Shetty:Awesome. Well, Rashmi, thanks so much for jumping on and sharing a bit about how you have been approaching Agentex Systems at Capital One. Super interesting stuff. Thank you. Thank you, Sam. Likewise, it was a pleasure being here.

54:10Thank you.