Delivering AI Systems in Highly Regulated Environments with Miriam Friedel - #653

0:09All right, everyone, welcome to another episode of the TwiML AI podcast. I am, of course, your host, Sam Charrington. Today, I'm joined by Miriam Friedel. Miriam is a Senior Director of ML Engineering at Capital One. Miriam, you joined us for a panel at our last Fumulcon conference, and I've been wanting to get you on the show ever since. I really enjoyed hearing a bit about your story and some of the insights you shared into creating organizations that retain some of the desirable characteristics of startups, but in larger, highly regulated environments. So welcome to the podcast. Thank you so much for having me, Sam.

0:47That was a great panel. And I always love speaking on things like that because I learned just as much from my co-panelists, Roseanne Adrian, as I do speaking. So it was awesome to be on that panel. And thanks for having me on the podcast. So you started your career as a theoretical physicist. I did. Yes. So I did start my career as a theoretical physicist. I got my PhD in 2006, which is a little crazy to think about it because in grad school, it feels like it's going to go on forever. And suddenly you blink and it's 17 years in the rear view mirror. But I studied protein folding. So I was using the techniques of computational and statistical physics to understand how proteins fold and aggregate.

1:28So it was a lot of, you know, obviously understanding math and physics, but it was also take this equation, take this concept, put it on a computer, make sure it's correct, run some experiments, write some papers, get a PhD. And as it turns out, that notion of are you calculating what you think you're calculating? Are you really sure about it is actually relevant to my job today. So let me tell you a little bit about the story of how I got from there to here. After grad school, I really didn't want to be a professor. It's an amazing path, but it just wasn't for me. So I went to work at a consulting firm.

2:00And that's where I like to tell people that I learned how to write software properly. I learned about version control and I learned about object-oriented programming. And I learned all of those things that I didn't learn when I was a self-taught theoretical physicist. And I also learned how to ask the right business questions and interact with clients, all that good stuff. After that, I went to work in a neuroimaging lab, which was a little bit of a detour back to academia. But in the neuroimaging lab, I was doing a ton of Python programming, R, statistical analysis, doing a lot of analysis of the scans.

2:34I worked at a place in Toronto called the Mouse Imaging Center. And so it was trying to understand how mouse genotype impacts phenotype and how we can use that to understand the human brain. So it was really fascinating. It taught me a lot about new programming languages, Python and R, which are pretty prolific in machine learning and really got me back in the mindset of thinking about and dealing with data that way. So when I moved to Charlottesville, Virginia, where I now live in 2014, that's where I really got into data science and machine learning. I worked at another consultancy here in Charlottesville, first as an individual contributor, then as a people leader.

3:09Then, as you noted, I was head of data science at a startup before coming to Capital One in July of 2020. I now lead a team of about 70 engineers, machine learning engineers. And what we do is we build out tooling, Python libraries, Kubeflow pipeline components, template workflows to help the data scientists at Capital One build and scale their models more effectively. So that's sort of my story from theoretical physics to Capital One today. Awesome. Can you elaborate on your role at Capital One? And in particular, I've interviewed one of your colleagues, Ali Rodell, who also works on Platform there and Kubeflow.

3:48What your team is focused on differ from what some of the other ML engineering teams are working on, such as Ali's? Yeah, that's a great question. You've actually interviewed a number of my colleagues, Bayan, Disha, Ali. So I should comment that I'm very fortunate to have such an amazing set of colleagues to work with and to learn from. So as you noted, Ali is building a platform based on Kubeflow. And for those of your listeners, you can go back and listen to that podcast. And so he's really making sure that we have the right foundational infrastructure. My job is sort of serving as the interface between his engineering team and our customers.

4:25And there's a lot of collaboration and partnership that goes on there as well. So here's an example of the types of things my team builds. When you think about the model development process, one thing that is really important is experiment tracking, particularly in a highly regulated environment like Capital One. So for example, when you're training a model, you're going to go through a number of different iterations. What hyperparameters did I use? What features did I use? You know, maybe what was my F1 score for that run? Or what was a particular loss function? So being able to track the results of those experiments is something that is really important to us as we train models.

5:00And so when we built this software, it's called Rubicon, we didn't have the plethora of really good tools that exist on the market today. So we had a need and we decided to build something to meet that need. And as the tool itself evolved, it's a Python package, we thought, hey, we want to contribute this to open source and we want to contribute this so it can be one of different tools available in this space. Probably some of your listeners are familiar with things like MLflow. So another thing that we build is Kubeflow pipeline components. So a lot of the data scientists that we work with are not super familiar with Kubeflow, with how it works, right?

5:37There's a little bit of overhead for building out some of the orchestration code. So one of the things that we're trying to do is make that easier and build template components that our data scientists can use, can pull off the shelf and make it easier for them to run on Ali's platform. And I'll provide one more example that's sort of at the intersection, I think, of the libraries and the components and how we pull them together. So as one example of the tooling that we provide is when you think about analyzing the results of machine learning models, you need to look at a lot of plots, right? So you can see what is my data doing, right?

6:13Like what does it look like? And so sometimes when you generate these plots, when you think about things like Shapley values or partial dependence plots, which gives you insight into what your models are doing, there's a little bit of calculation that you have to do on the backend in order to generate the plots and the format that you wanna see. And so this is really important for our data scientists as well. And it's important for us also because of our model risk office, which is something, you know, I'm sure we'll talk about in this conversation, making sure that we're presenting data to them in a way that they need to look at to validate our models.

6:45And so we found there was a ton of duplication around Capital One, sort of this arbitrary, continuing to build out these plots sort of uniquely across all these different lines of business. And we said, you know, there's a real opportunity here to consolidate the plotting and the calculations on the back end, right? If you need to calculate a partial dependence plot or a Shapley value, you don't need to have 14 different copies of your code doing this. And similarly, you know, when I think back to my days when I was doing data science, I felt like I every single time, you know, you think about matplotlib and seaborne, and they're such like powerful libraries, but every single time I would need to like, what is the syntax?

7:23Oh, my x axis is not formatted properly. Oh, the tick marks are the wrong size. Oh, it's the wrong color. And so when we think about the reports we have to generate, sort of pulling all of those things out, creating a standardized plotting library, and then also creating components around it. So when you think about running a pipeline from end to end, being able to sort of automatically generate these things is maybe a last step in a dag. So these are the types of tools that we are building in order to enable our data scientists to leverage the platforms that Ali and team are building. There's a lot in there that I want to follow up on.

7:59But the last thing you mentioned, that last example kind of prompted a thought that's maybe going off in a different direction. But a lot of the work that ML engineering teams like yours has historically done is like creating these tools and libraries so that teams don't need to build boilerplate code as much. But more recently, we've got the rise of things like co-pilot and LLMs apply to code generation also solves a similar problem. Like you don't have to think about how to, you just put in the comment, the kind of chart that you want. You don't have to think about how, you know, how to get the tick marks correctly and all that kind of stuff.

8:39How do you see those tools impacting the way you approach and prioritize projects for your team? Like, are you there yet? So we're obviously thinking about it, right? Like I think anyone in this field right now is thinking about what's happening with Copilot, with GPT-4, with all of the amazing tools and the advances we've had in generative AI. The way I think about it right now is a little bit different. And actually, this is a good opportunity for me to talk about our model risk office, which I've already alluded to. So many companies like Capital One, which are really large and highly regulated, have a model risk office or a similar function And, you know, to make sure that we're in compliance with any applicable laws and regulations.

9:22And so it's really important for our data science population to be able to clearly articulate to the model risk office, you know, through code, through white papers, they have to write white papers, some of which, by the way, are longer than my PhD thesis explaining why. Yeah, it's incredible, like how long and detailed they are. But explaining why this particular model, what is the business problem we're able to solve in really providing like a level of transparency throughout the entire model development lifecycle. So I think that as we start to think about how can we effectively leverage tools like Copilot, right, to take away some of the challenges with building boilerplate code or making our development process easier, I think certainly from our standpoint as a highly regulated organization, we still need to be able to make sure that we provide that transparency to the model risk office to make sure that this is safe, this is secure, and making sure we have the right guardrails and testing in place, which we do right now for all of our libraries.

10:23Getting back to that question of, are you calculating what you think you're calculating? Are you sure that math is right? So I think we have to find a balance there. And it's definitely a really exciting space to be in. So definitely we're thinking about it, but we don't have a specific direction that I would want to dig into at this point. A bit of what I heard there is that beyond eliminating repetitive boilerplate code, a lot of what you're trying to do is to encapsulate aspects of calculations that are easy to get wrong and offer folks kind of a proven way to incorporate those into their models.

11:00And so introducing a degree of randomness through Cogen is maybe the wrong direction relative to what your goal is. Yeah. And I think what you said is the things that are easy to get wrong. So as an example, I don't know, I want to say a year ago, I was looking through some of our GitHub repositories. So Capital One is a really big place. And there's a lot of big companies like Capital One, where you have a ton of really smart people doing a lot of innovation. And it's kind of impossible to be able to know all of the little pockets of innovation that are happening. And in some things, that's like a really good thing, right?

11:36Like we get to build really excellent stuff and you don't know where you're going to find it. The flip side is you don't know where you're going to find it. So about a year ago, I was looking through our GitHub repositories and I found like maybe at least seven different calculations of the population stability index, right? And what's that index? So the population stability index, it's a statistic that measures how much a particular variable has shifted over time. And so you use it to measure the applicability of a certain statistic to your population, right? So it's in and of itself, it's not anything particularly crazy, but every single time you write that code to do that calculation, you introduce the possibility of an error.

12:20And so if you can have like a centralized location or a centralized tool that people are pulling from, then that's going to reduce the possibility of error. So if you think about something like Scikit-learn, Scikit-learn is used very widely in the open source community. We use it. We love it. There's a lot of great open source tools that we rely on. But because Scikit-learn has such a robust community of users, people trust it, right? And people are going to pull from it. And so really, we want to try to build out that level of trust with our users leveraging open source, but then adding in the things that we need to that are maybe specific to Capital One's needs.

12:59This model risk office has come up a couple times already. And I think of it as kind of representative of this broader function of operating in this highly regulated environment. And that's very different from when you were running data science at a startup. Like, you know, talk a little bit about the, I mentioned this in the intro, like one of the things that came up in this panel that we did is like, how do you retain agility and speed and some of the characteristics that like we admire about startups, but still operate in or under the constraints of an enterprise like Capital One? Like, how do you think about approaching that challenge?

13:40Yeah, that's such a great question. Because it's really hard. There's ways to do it for sure. And there's a lot to unpack. So first of all, when we think about like, you know, I did come from a startup. And at one point, The engineering and data science team all together was maybe like 10 people. So we had one stand up and one planning session. And it was really easy to know what was going on. And sometimes we would merge pull requests and you would just yell to the other side of the office, OK, I merged that, you know, try again, right? Which you obviously can't do in a company like Capital One, where we're spread across the whole US, you know, multiple countries.

14:18So I think discoverability, as I've talked about before, is definitely a challenge. I think another thing that happens in startups, which is hard to do in Capital One, and that's a good thing, is in startups, it's easier to just, you know, I'm going to spin up an S3 bucket and I'm going to get access to some example data and I'm just going to go. You know, and at Capital One, we take data privacy very, very seriously. And so access is much more limited, which is a consumer of financial products in this country, right? I think is a good thing, right? I want to know that as a consumer, my data is protected.

14:53So that's another element is just data access. And then I think sort of retaining the agility in such a big environment. So one is we're all aware of the model risk office. And so we know that we need to incorporate that interaction into how we build models. And so I think knowing that from the get-go is really helpful. And I think really for me, what I've found is you want to give your teams a sense of the big picture. What are the broad, sometimes even company level strategic initiatives? What are the strategic initiatives for our division? And then sort of laddering that back down to your team.

15:32So when you're within your team, you have a clear sense of ownership and responsibility, but you sort of know your swim lane so you can iterate in that space. And then since you have the ability to do that, that makes you go a little bit faster because the overhead of communication and communication is incredibly necessary, but it takes time and it takes thoughtfulness. So really finding that right balance between, hey, I know what I own on my team. I can iterate. I can work quickly with my peers, but I also know how that ladders up to the big picture. So I know if I'm stuck or if I have a really good R &D idea, I can maybe reach out to a collaborator, you know, on Brian Bruce's team, right?

16:13He's our VP of applied research. I know you've spoken to him a couple of times, right? Like, how can we make that happen? So I think it's really a mix of transparency, communication, clear ownership, but it's definitely like, it's a challenge for sure. It's a good challenge, but it's a challenge. Yeah. Historically, one of the challenges for MLE organizations is you have these great ideas for tools, you build the tools, you know, maybe in partnership with a customer. And then there's this chasm where you want the broader organization to adopt these tools. And there's often a lot of evangelization that needs to happen.

16:49And it sounds like in your case, and this is likely representative of what's happening in other enterprises as well, that this MRO, this model risk office is a little bit like a carrot and a stick. Like if you use our tool, it's going to be less painful down the line when you need to demonstrate that your models are within spec compliant, whatever the right terminology is. Conversely, if you don't use this tool, it's going to be a lot more painful down the line. Is that the case? Yeah. So the model risk office actually can't be overly prescriptive, right? Because they are supposed to be sort of like an independent auditing function, but they can certainly say like, hey, for this particular use case or for something similar, we saw that this tool was really useful, like, can you explain a different choice?

17:37And it's entirely possible the data scientists will say, yes, I had to use something different and here's the reason why. And so then that's definitely fine. But the model risk office, even though it is a carrot and a stick, but they also need to maintain sort of independence and auditability. So there's that as well. Kind of the flip side of, on the one hand, there's getting the teams, your customers on board with might feel like to an engineer hoops that they need to go through. And all they want to do is deliver their perfect shiny model. And talked about some of the ways you overcome that.

18:12The flip side of that is you're building tooling that enables folks to do these things that kind of aren't necessarily only around delivering functionality, but they're around robustness and compliance and all these other things that are maybe less shiny. From a managing up perspective, How do you justify the investments in robustness and tooling that doesn't necessarily tie to business outcome? Maybe the answer is actually that it does tie to a business outcome, but it's more nuanced. Yeah, I think that's exactly right. It does, but it's more nuanced. So what is managing risk, right? So that's one particular way.

18:55But the other thing is being able to use a standardized set of tools and contribute back to them. We have a data science and machine learning are filled with really creative, innovative people who like to build things. And that's how they understand things. So it's, hey, you know, we're creating this functionality that is going to serve your teams, they have the opportunity to contribute back, which then expands their influence, right? Because it's not just you're making an update that is useful only for you this time, but it can make somebody else potentially faster, right? So there's a time to market aspect where if you don't have to spend a lot of time worrying, like, do I have the right Docker image?

19:34Do I have the right dependencies, right? Like this notebook that was written two years ago, is it still going to work on our current infrastructure, right? So having a centralized team to manage that and to take away these headaches really frees up the lines of business to think more creatively or maybe in a different way about the actual business problem they're solving. So instead of having to deal with, does this functionality exist? Do I need to rewrite this software? This package from several years ago is using an outdated version of Python, and now I have to spend a bunch of time updating it, and now all my dependencies are broken, right?

20:08Like, if we can help take some of that away, I think it actually enables, you know, that standardization really, instead of inhibiting creativity, I think it can kind of unlock it for the data science population. So I think that's how when we talk about managing expectations sort of with senior leadership or with some of our customers, it's really about reducing risk and enabling them to work a little bit faster and in a more streamlined way. It's like when you have like too many choices, it's almost like paralysis. Like I think, you know, when I was thinking about early, earlier on when I was in this field, just wanting to try everything and have the ability to do absolutely everything.

20:50And I can use any Python library and any, you know, or any R package. And then eventually, you know, you have 400 things and it's like, well, actually now it's too many things, right? So striking that right balance between giving people the ability to be creative and innovate, but also guiding them a little bit and taking away some of the pain points. And I guess I'm curious, are there examples that come to mind that really illustrate like that tension and how it gets resolved? Yeah, that is a great question. So you're talking about the tension between like, I wanna be able to do everything, but like help now I have too many things and I need some guardrails.

21:28So I think one example is hyperparameter optimizations. So there are a number of different great packages to do hyperparameter tuning on the market. One example is Hyperopt. Another is Optuna. There was actually an interesting paper that came out last year that was comparing different techniques. And the punchline was, depending on the data and the problem, one or the other might be better. But these things don't necessarily have a common API, right? So if you're a data scientist and you want to experiment with both, you might have to do a bunch of rewriting or sort of data manipulation because they don't have a standard API.

22:09So one of the things we've actually done in one of our libraries is provide this. So if you're doing hyperparameter optimization, creating a common API so you can use one or the other of those things. And so as we think about different hyperparameter optimization strategies, I don't think we need to give people like 100 of them. I think we can give them maybe three or four that are thoughtful, that meet a majority of the use cases that we're seeing, and then wrap them in a common API and a common suite of libraries that work really well together. So we can say, hey, you have the opportunity to choose and experiment with different hyperparameter optimization techniques, but you don't need to worry about everything under the sun, right?

22:49Like, start here. And then we also really try, I think, as I said before, to create a culture of intersourcing so people feel empowered to contribute. back. Like, hey, I tried Hyperopt and I tried Optuna and they didn't really work. But I tried this third thing and I'd like to contribute it back. So now it can be part of your ecosystem and your suite of tools. So I think that's one example we found where there's a lot can do in this space. And so we're starting with two and it might grow. I think we also have some Bayesian optimization techniques as well in there. So really, you know, starting small and then growing.

23:21In the context of hyperparameter optimization, as well as experiment management, you kind of alluded to our things that you built. In the case of experiment management, you built a tool. You might not do the same thing today because the market has matured. I hear that a lot about experiment management in particular, like a lot of ML engineering organizations. That was their first project because there was nothing good on the market and like they wouldn't do it again necessarily. You mentioned the same thing around hyperparameter optimization or something similar. Like, I'm wondering if you can speak broadly to the way you think about build versus buy.

23:59It's been a bit of a theme that's kind of been woven through some of the conversation, but I'm wondering if you can speak to it more directly. Yeah, that's a great question. So I think with build versus buy, and this is something that I've learned, I've had to learn this lesson throughout my career, is that when you choose to build something, there is a cost to maintaining it. And so if you're going to build something, you have to think about, is there really, truly nothing, either as like a proprietary off the shelf tool we could use or something in the open source domain that we could use? Because if you're going to build something, you're then going to have to maintain it.

24:34You're going to have to think about things. Eventually, when something better comes along, you're going to have to think about deprecation, right? We, you know, we all know that there's sort of like a lifecycle to software engineering. And so I think, you know, when I think about build versus buy, really the first thing I encourage my teams to do is look and see, does this already exist out there somewhere? And can we use it? So with the example of Rubicon, right, it was something that it wasn't available when we started building it. So we built something to meet our needs. And so I think, you know, step one in any build versus buy decision, I think, is what is available in the open source domain or in the what are the options there?

25:15And can we use it or can we build some sort of extension of it that is beneficial? I'm not a fan of like, let's wrap something for wrapper sake. Like, right. Like I talked about, you know, you know, this is fun. Let's build it. Right. Like we're technologists. We like to build things. It's fun. But, you know, with like Hyperopt and Optuna, right, like we saw an opportunity there because it was a way to reduce some friction for our data scientists and create a common API that didn't exist. So I think just being really thoughtful about that decision. And then for us, certainly, like, are there things that we need to do at Capital One that need to remain proprietary?