Evolving MLOps Platforms for Generative AI and Agents with Abhijit Bose - #714

0:00To create that feedback loop, you know, from all the way to the IVR system, to all the way data coming back into our model for refinement. we realize that there is that part of deploying Gen.AI that's not always talked about, but it can get very, very complex, you know, chaining all those events.

0:36All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. Today, I'm joined by Abhijit Bose. Abhijit is head of Enterprise AI and ML Platforms at Capital One. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Abhijit, welcome to the podcast. Thank you, Sam. It's a pleasure to be here today. I'm super excited to have you on the show. We'll be digging into all the things you're working on and thinking about from a platform's perspective in support of Gen.AI at Capital One. But before we dig into that, I'd love to have you share a little bit about your background.

1:15As you said, I lead the enterprise AI ML platform engineering teams at Capital One. What it means is that we build and deploy all of the core AI ML infrastructure for the company, and that includes classic machine learning models like GBMs, neural nets, but more recently, Gen AI applications, you know, that involves LLMs, or agentic workflows. I have been at the company for four years. In that time, you know, my team and I, we have pretty much rebuilt the machine learning stack from the ground up in AWS to the point now where almost all of the companies, data scientists and engineers, use our platforms to build and deploy ML at the company.

2:06It's one thing to experiment with machine learning, but when your most critical, mission critical credit approval and fraud detection models run on your platform, it's a whole different kind of a ballgame. So it comes with a lot of operational responsibility. just building that MLOps muscle over the last four years has been a really amazing experience for me personally and for my team. And with Gen.AI, it's the scope and the possibilities of what we can do is just exponential from here on. And before Capital One, I was the head of research engineering for Facebook AI Research in the East Coast.

3:04So New York, Montreal, and Pittsburgh labs, the engineering. I founded a lot of those engineering teams. That was, again, a great experience. Yeah. That sounds like an entirely different level of scale than what I'd expect at Capital One. How do they compare? It's, you know, in terms of complexity and the kind of the sophistication, I would say what we are doing now at Capital One is probably similar. But FAIR was much more academic style research, you know, so we built, we did a lot of academic style, you know, research in computer vision, NLP, you know, robotics. We build prototype systems. And then for some of those systems, we actually scale them to the entire company.

3:56So kind of similar things we are doing here too in our science team, you know, especially in AI. So we have the research to production path here as well. So yeah, I've been very lucky to be able to do this for a job. That's awesome. That's awesome. Now, I've interviewed many of your Capital One colleagues over the years. Many of those conversations have talked about platforms and MLOps. We had speakers from Capital One at our conference talking about some of the data platforms you were building at that time. Platforms is something that Capital One has been building out and investing in for quite a long time.

4:44You know, talk a little bit about the role of platform at Capital One and kind of how you see, you know, what you're trying to accomplish in investing so much in building out platforms. I will say that, you know, we are a platform company. And having come from Facebook, I have seen the power of centralized platforms like enterprise, you know, scale platforms. Very similar what I see here. There is a very strong... Meaning a platform company as opposed to a bank? That is true, actually. We say Capital One is a tech company that does banking. And it is really true. And part of being a tech company is to have platform mindset and having a very strong data culture.

5:32I see those same kind of ethos in our DNA as well. So, as we advance in our AI journey, we are taking a very similar platform-centric approach. So for Gen AI, for example, we are building all of those AI capabilities in a way that the rest of the company can build on and leverage. And as you know, if you are investing a lot in GPUs and training clusters and inferencing services, those are expensive investments. So being a central platform actually helps to make sure we are using those GPU resources for the highest leverage use cases. We can optimize continuously. The other thing that I see is that when you have central platforms, you can apply a lot of the governance and controls in one place rather than having a federated set of controls everywhere in the company.

6:38And I have seen that personally for machine learning. Like we have all of the kind of the data, cyber, all of the kind of the governance that you have to do as being a bank. We apply that in our machine learning platform in one place for all of our users rather than every organization trying to figure them out. So that I see a big advantage of being a platform company. The other thing I'll tell you in terms of our platforms, ML platform happens to be the platform with the highest NPS score of all platforms in the company. You know, this is a personal kind of pride, you know, for me and our teams is that we are so focused on our users.

7:29We have thousands of users across the company that use our platform. We really listen to them, you know, solved a lot of their pain points. And that's kind of the power of the platform. When you do this right, it solves many, many challenges in one place. I'm curious who your users are for the platform. Sonoma Enterprises offer a variety of levels just like the vendors offer, where a user can interact with the platform at a fairly low level, at a notebook or developer-oriented level. and maybe you have applications running on the platform that they consume directly, forecasting, for example. Are you doing similar things there?

8:20We have different user personas on our platform. There are what we call power users. They want absolute control over their machine learning workflow. So they use our SDKs to actually code the model and the features, but they can deploy it on our platform without our engineers, like platform engineers, helping them. So it's a very self-service environment, even for very sophisticated users. and even over the years, what we have done is to really make it easy for them to distribute their compute over a bunch of CPUs, and we are doing the same thing with LLM on GPUs, for example. So we do a bunch of things to make it easy for our users, but there are also users who like to use more graphically.

9:23They can connect to data and add a few steps. So it's all like low code, no code kind of environment. But behind the scenes, those are automatically compiled and into code and run on our platform. So we address the pain points and needs for a range of users on our platform. And you mentioned that over the past several years, you've been migrating the platform into AWS. Does that mean that you're consuming the AWS services and that is your platform like SageMaker? I have the impression that you're actually doing more kind of building your own on top of whatever infrastructure you're using. So, I mean, this is a great question, right?

10:15A lot of companies are grappling with that, right? How do we build a machine learning platform on the cloud? We spent a lot of time early on, like in my first year, for example, on designing a robust platform control plane. And that's based on Kubernetes. and we enhanced that control plane in a bunch of ways so that it's easy for us to take something from SageMaker, something from open source or wherever we feel like we can add a lot of IP. For example, when it comes to automating a lot of the governance steps, which no other platform or publicly available platform can do it because it's very specific to our processes and model risk framework.

11:13So we have that flexibility of choosing different things from different places, including contributing our own in our platform today. and yeah the other thing i'll tell you that you know actually helped us on the jnai journey uh because now we are in a position to you know kind of extend our original machine learning platform to more like llm hosting or llm training you know fine tuning uh you know agentic workflows So we took the basic kind of the architecture and just extended that to Gen AI use cases, and we were up and running very quickly. I think if we just went for a vendor solution in the beginning, it would probably have blocked us from making so much progress in AI and Gen AI.

12:17Before we dig into Gen.ai and how you're tackling that from a platform perspective, I'm curious how the relationship between traditional ML workloads and Gen.ai rolls out. are you have you kind of shifted or are you seeing your users kind of shift their focus to dive into Gen.AI or are all the same traditional MLAI work streams still kind of continuing when I talk to some folks the sense is that Gen.AI kind of took all the air out of the room of traditional ML, AI, you know, workloads. And then other folks describe it more as it kind of piqued the interest and it kind of raises it to an executive awareness level.

13:15But a lot of the problems that they end up wanting to solve are best solved with traditional ML and AI techniques. And so it actually, Gen AI kind of pulls those along. I'm curious what you see there at Capital One. We are actually investing in both. There are solid examples of where machine learning, as we have practiced traditionally, would still be very, very relevant, especially when it comes to create modeling and a few other things, where it's not easy to just switch to another technique without a lot of research and development. So we continue to invest, you know, in traditional machine learning techniques.

14:03And then in their cases like customer servicing or, you know, other like fraud detection, where we have invested a lot in AI and Gen AI to be able to harness the power of, you know, those kinds of approaches. So, yeah, we are doing both. Taking a step back and thinking about Gen.AI from a platform perspective, how do you think about the extension of an existing MLOps platform and set of workflows into Gen.AI? There are similarities, but at the same time, I would say there are quite a few things that are very different for Gen.AI. I will start with the observability as the area where it really starts to make a difference between traditional machine learning and Gen AI.

15:02In traditional machine learning, you deploy a model in production. We have many hundreds of models every day making all kinds of decisions. you have to have solid monitoring in place in terms of drift of data, input features, the model scores themselves. And so there are alerts, alerting that needs to happen, and then you have to be able to collect all the logs to figure out what's going on with the model. Is it a feature? Is it something changed in the model execution environment? so all of that is you know all of that equally transferred to the JNI side but you also have LLM hallucinations you know that's net new in the observability world you have to have guardrails and then you have to log everything in those guardrails both input and the response in case of agentic workflows you also have to see which tools they are planning to call because tool execution is so much part of the agentic workflow.

16:16You need to make sure that the execution of those tools is being properly logged. They're being properly governed. They're called in a properly governed way. So observability becomes very, very not just important, but it also becomes very complex. in the LLM world. We had to build a lot of infrastructure on top of our regular anomaly detection and model monitoring in case of Gen AI. It just, you know, and we are still building a lot of infrastructure on that. And you said on top of your existing monomonitoring and anomaly detection, how does the observability for Gen.ai interact with or take advantage of the services that you had in place?

17:12So this is an active area of work for us. But what I would say is that, you know, the way a lot of our anomaly detection algorithms work, you know, that's also a platform that we have. So platform is very much part of our DNA, as you can see. So all of our algorithms for detecting deviations, you know, and then alerting, that's a part of our machine learning platform that also operates as a separate platform. So you can actually, you know, basically point it to any data source and it will automatically start generating these alerts and, you know, with some little bit of configuration that you have to do.

17:59So we are using very similar algorithms, but it's just pointing to text and other types of data, not always like structured data. So we are leveraging what we have built, but extending them to new types of monitoring. From a model perspective, how are you thinking about model use and model selection? Are you using open models? Are you using models delivered as a service via Bedrock, for example, or another service? So we generally use open source models like Lama as the base. And then, you know, we take those models and then we fine tune them with our own data and customize them to meet our regulatory and governance thresholds.

19:03You know, and then we also test them very extensively, you know, with respect to hallucinations, accuracy, depending on the task that, you know, you are deploying them for. And the other thing I will say is that we also host these models completely within our premises. like within our own AWS environment. So no data ever leaves our, you know, perimeter. Was there a transition point at which, or what was the transition point at which the LAMA models, you were able to commit to the LAMA models in that way and didn't feel like you were leaving too much on the table by not choosing a best-of-breed, you know, vendor model?

19:50So, I mean, we constantly look at all our options, including third-party hosted models as well. So this is like, LAMA is just one example of the open source models that we have been using. But our strategy is that we will constantly benchmark these models and pick the right one for the right use case. But we wanted to look at models that we can host ourselves and, you know, we can properly safeguard our data. And but at the same time, not, you know, sacrificing the accuracy that we want for some of our use cases. Are you to what degree are you exploring fine tuning as a way to get the most benefit out of those models?

20:43We do a lot of fine tuning in-house. We fine-tune to specific tasks and specific domains. For example, as you know, summarization is one of the harder tasks. Even if you look at MMLU benchmarks, often even the frontier models, you'll see the summarization accuracy is a little bit lower than, let's say, search, for example. so you know you will see much better accuracy and relevant scores if you take one of these open source models and then fine tune and customize it you know with your own data in case like in our case it will be capital one data you know some corpus of text you know for that use case that we have identified yeah so that's pretty much has been like our approach and it has served us really well.

21:46You know, if we want to get into like specific use cases, I'm happy to talk about one use case where, you know, we took Lama and then we fine-tuned it and then we saw much higher accuracy for both search and summarization. Oh yeah, I'd love to dig into that. So this is a system, you know, it's kind of like a rag pattern in terms of like, you know, the workflow pattern. When you call our call center, you know, let's say you have a question like, if I have a declined card transaction, you know, would it count towards my, you know, like my limit, you know, credit limit? or, you know, if you're going to, you know, Canada and you want to put the rental car, you know, on the card, you know, would it come with insurance, for example?

22:49Like common questions, you know, sometimes, you know, it's also about policy. Sometimes it's about a procedure, you know, like early payment on something. So traditionally, our agents would do a very similar to TF-IDF kind of search, keyword search, and try to find the link to the document that has that particular answer. And it would take them quite a bit of exploration of whatever the system returned, going to different links, and then try to find the answer and then answer the question. You would take the agents a lot of manual. Looking around manually. Yeah. Okay. And the human agents we're talking about.

23:39The human agents. AI agents. That's right. Sometimes agent is actually overloaded, especially when you talk about human-in-the-loop systems, like a human agent and an AI agent. And customer service use cases, it comes up all the time. Exactly. Yeah, that's right. So with our new system, basically it runs on our platform. We took all of those knowledge bases that the agents look up, we indexed them and put them in a vector database. And we also fine-tuned our LLM with a lot of the knowledge and the type of question and answer tasks these agents produce. perform. And then we noticed that the accuracy went up a lot.

24:34So now they only have to look at the first or the second link. They don't have to hunt. And the quality of the summaries that they can actually relay back to the customer on the line, those are also getting much better. And it's also like from an AI safeguard and putting together the appropriate guardrails, It's still human in the loop because our human agents are still in charge of the conversation. And we can also collect their feedback. They can indicate on the system what the LLM and the RAC system actually returned as the response. And we get a quick feedback from them. And yeah, the one thing I would say is that it was really interesting working on that project because to create that feedback loop, you know, from all the way to the IVR system to all the way data coming back into our model for refinement.

25:43that, you know, we realize that there is that part of deploying Gen AI. That's not always talked about, but it can get very, very complex, you know, chaining all those events. What's a specific example of where you see complexity there? You know, chaining all the events together, like, you know, exactly all the logs from different systems. if you don't put automation in there, it'll be very difficult to kind of like create that data set, that annotated data set that you want to, you know, retrain the model, for example. Are you trying to understand from the IVR logs whether the user gets to where they're trying to get the first time or whether they have to, you know, spend a lot of time navigating the system just to get to their responses?

26:34Is that the kind of feedback you're pulling into the loop? No, we analyzed those, but that was not the intent of this particular system, you know, because this was really our one of our forays into Gen AI in customer servicing. But we were mainly focused on the question answering part of the process. Got it. Got it. That system is now being used by 20 ,000 agents, human agents at the company. So it's very widely deployed, you know, pretty much a large portion of our, you know, agent community. They have been using this system now. Oh, wow. From a platform perspective, what did you have to put in place to enable the fine tuning?

27:18A lot of work went into this collection of the data, you know, properly annotations of the data. We also set up HPC clusters, you know, like the difference between traditional machine learning, you know, where you are training a GVM or, you know, like decision trees. in almost all of those cases, you can train them on one or two GPUs and they may run for like a few hours to a day. But for LLMs, you need HPC clusters, very tightly interconnected file system, high-speed networking and the GPUs to work together for weeks at a time. And for sure, you know, there'll be a few GPU failures, you know, every day.

28:09And that can bring the whole, you know, training or fine tuning job down. And then you don't want to waste a lot of that expensive hardware, you know, time to go wasted. So we had to do a bunch of things on AWS to create our own HPC environment for training and fine tuning. So that was all in-house. We basically got the GPUs from AWS, but then we built our own fine-tuning stack on top of that. We have checkpoint restart and all of those things that you need to have a production system. And is that also Kubernetes-based? That's also Kubernetes-based, yeah. And do you see, it's been a while since I dug into Kubeflow.

28:58do you see Kubernetes evolving or are there specific projects on the Kubernetes side that are evolving to support these kinds of fine-tuning HPC workflows? Or is that something that you're expecting to continue to have to build yourself? Yeah, so that's a great question, Sam. In terms of Kubeflow, we do use it. We are actually part of the, like some of our engineers are part of the open source Kubeflow project. They commit in the GitHub. And a large part of the company actually use Kubeflow on our platform. We offer that as a service, a hosted service, basically, on our traditional machine learning part of the platform.

29:45In terms of, I think, Kubernetes-based training, whether it's HPC or traditional machine learning, I think that will continue for some time. If you think of some of these large clusters that the labs have set up, some of where the frontier models are built, a lot of them are based on Kubernetes clusters, Kubernetes as the orchestration for containers. But I think the community needs to develop more tooling for scheduling jobs and workload management. I think those would become more common as time goes so that we don't have to build a lot of those things ourselves. Interestingly, I'll tell you, when I first joined four years ago, we have lots of great Kubernetes engineers, but I was like, why do we have to build our own Kubernetes stack?

30:49Why can't we just go and just get that as a service? So over the years, we have worked with AWS to leverage EKS, for example, Elastic Kubernetes Service. So that has made our life a little bit easier. but there are lots of other things that you have to build on top of the Kubernetes stack to really make it easier for our users, scientists, engineers, analysts to be able to use the infrastructure. So that's where we focus these days. Got it. So the Kubernetes community has some work to do to build the HPC semantic layer on top of the underlying container management. Yeah, yeah. You mentioned data annotation was a big part of the work that went into fine-tuning.

31:49Can you talk a little bit about how that differs between the traditional MLAI and Gen AI and what are the things that you had to build to support it? So in traditional machine learning, you take a, like, let's say you want to build a model for fraud detection. So typically you will get a, you know, a set of transactions, you know, you'll divide the data into two parts, like your, you know, training data set, maybe three parts like validation and test, you know, data sets and all that. But then you will have a labeler to label them, you know, as fraud or not, right? Like positive or negative samples.

32:32And then in some cases, you can use some automated knowledge that if nobody complained or the customer did not come back with a fraud or there was no fraud alert, etc., those are maybe not a fraud transaction. So all you need is basically a labeling, like a methodology, and there are systems that can automatically label with a model and with some human levelers validating them. In case of LLM, whether it's, you know, fine tuning or pre-training, it's order of magnitude more complex because you are really creating, for example, an input prompt, you know, like let's say it's a, you know, it's post-training input prompt, you know, you want the answer.

33:29you need to extract that answer, you know, maybe from your corpus of knowledge, you know, so you need much more refined, you know, setup capabilities, not just for humans to review and evaluate, but you also need things like LLM as a judge, you know, to actually, you know, score some of these responses. So it becomes much more complex, this whole annotations and data creation portion of the pipeline. Is the solution to that to deliver platform services or is it like analogous to annotation tools? It's a combination of both. for us. We think there is an opportunity for leveraging platform even in that context.

34:24Plus, there are lots of tools today in the market as well as in open source that we are looking at to be able to do those annotations as well. And what are some of the ways that the platform can support that workflow or pipeline? I would say automated scoring automated evaluation, you know, from production systems all the way to the training system that we can automate, you know, and the tools can play a role, but they will be embedded in our platforms. I guess when thinking about deploying all these fine-tuned models, the next question is like, how are you doing inference? You mentioned the supercomputer for training or supercomputer cluster for training or HPC cluster, I guess is what you said specifically.

35:17I imagine you've got inference clusters as well. That's a different level of scale than in the traditional world. Talk about how you approach that. Yeah, we use GPUs in our inferencing platform. But the interesting thing is that there is so much science happening in the inferencing space. As you know, reasoning during inferencing is actually becoming more and more like a more popular way of being able to answer complex questions, for example. So those things are not going to be possible without GPUs, for example. So we built an inference platform. There are a bunch of GPUs behind the scenes where we do a lot of optimization around caching.