Developer Experience at Capital One with Catherine McGarvey

0:00Modern software development is evolving rapidly. New tools, processes, and AI-powered systems are reshaping how teams collaborate and how engineers find satisfaction in their craft. At the same time, developer experience has become a critical function for helping organizations balance agility, security, and scale, while maintaining the creativity and flow that make top-tier engineering possible. Capital One is continuously transforming its developer culture with a focus on faster development cycles, reducing operational overhead, and boosting productivity across the organization. Catherine McGarvey is the SVP of Developer Experience at Capital One.

0:41She joins the podcast with Sean Falconer to talk about what developer enablement means at enterprise scale. measuring developer productivity, being agile in a regulated environment, AI and enterprise development, the future for developers, and much more. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

1:17Catherine, welcome to the show. Thanks, Sean. Great to be here. Yeah, absolutely. It's good to have you on the show. So, you know, I was looking into your background. You have a pretty fascinating background where you've spent some time in startups. You've done some defense consulting. Now you're leading a developer experience for like 14 ,000 technologists at Capital One. How does that diverse experience across this wide spectrum from fast-moving startup world to a large financial institution shape the way that you think about tackling things like developer enablement at such a massive scale?

1:49Yeah, I've been super, super fortunate in my career to have these kind of pivots or these opportunities to approach different domains and different sizes. And I think what's really nice about that is you learn a couple of things that often there's not one way to do something. and that you've really got to think about what matters in how you're doing software development to the consumer and to the business to really spark down the path of, well, what do you have to standardize on and what can you leave flexibility around? And that's been really great. You know, with the consulting side of startups, I got to be involved in the early days from zero to one and one to a hundred users.

2:27And it was amazing to think through long-term architectural design matters a lot less when you're just trying to confirm that you can survive and you can get those first couple of customers and how much that pivots when you're talking about B2B selling or B2C, where I am sort of now really thinking through how do we make sure what we deliver is resilient to customers and is set up. But at the same point, how do you keep it agile so that you're adapting to their feedback back and able to make changes? So it's been nice to draw on different parts of my career to date to really look to how How do you bring the mindset of where we're focused now with this idea of how do you bring that agility and that adaptability to new information while ensuring you're keeping sort of risk and security and all of those quality concepts really top of mind as well?

3:16How do you think about developer enablement? What in practice does that actually mean? I think it can mean different things within different organizations. Some people might be focusing purely on how do we increase our velocity or become more efficient? What does this mean in terms of your ownership within Capital One and how do you think about it? Yeah, so developer enablement. A colleague of mine actually mentioned this term for me and I just loved it. So I've latched on to it. In the past, I've really thought about developer productivity, which comes down to a lot of throughput, speed, and quality.

3:46Is the team churning out the code at a certain rate? Is it at a high enough quality? And is it getting into the hands of users fast enough? and 15, 20 years ago, as product managers really stood up as like this role that was critical in the software development process, there was a real shift to be, are we building the right thing? Because it's fantastic if your engineering team can move really quickly, but if it's not connected to something of value, who cares? Productivity without connection to value is like a dying product. So the product management side of this really sparked this idea of let's ensure we're building the right thing and we understand our users well.

4:24And so when I think about developer enablement, it's combining speed and throughput and how do we give the tools to accelerate that development, but it has to be balanced by, are we listening to users? Do we understand what they need well enough that we're solving the right thing? Those two things in balance, I think is where enablement's really about. Give them the tools, but the information as well to be able to do their job well. And does that need to be something that you're doing? Like, is that one person or one small team owning horizontally across all the engineering teams? Or do you see this as something more like a product manager that's going to work directly with one sort of cohort of engineers and kind of be trying to answer that question of like, are we building the right thing?

5:07I think for every team, the team size is really interesting that you need a sense of the direction that's connected to the customer and someone being that voice. And sometimes that's design, sometimes that's product, sometimes it's both. And sometimes there's multiple people depending on the complexity of what you're delivering needed there. But you need that connection to the team and you need the swim lane small enough that that team can be effective in solving that outcome or delivering that product. And so that combination of, again, keeping that team nimble enough and then giving them the information they need to do the job well and having someone to talk to about it empowers them because they're not just picking up a product definition and executing on it.

5:49They're understanding it and discussing it and seeing how they can improve upon that experience overall, not just delivering a feature. Okay. With such a large organization, I mean, we're talking about 14 ,000 people, how do you even measure the success of developer enablement at a scale? I love this question because there's so much going on around developer experience. what do you measure you know years gone by there was a really big great push around door metrics and thinking about that's really like your throughput sort of style of a new recovery style of things and then there's been the work around space and really trying to get to the qualitative side of that as well and I don't think there's one great answer but there's a lots of good things you can measure I do think measuring the important things matters but it's In some cases, it's not the only thing that you need to measure.

6:39So a couple of things we do look at is we are on this journey of continuous deployments. What is your time between deployments is a really interesting one to see. Are you getting your code out there fast enough, quick enough, and adaptable to change on that one? And to be able to do that well, you have to have great testing. You have to have great quality measures. So that's kind of part and parcel of that continuous deployment piece. The next one is, is it actually addressing the need of the customer that you're delivering to? So in my space, I spend a lot of time thinking through, am I solving the outcome for the developer?

7:12If my job is to make coding reviews great, and I'm providing tools in that space, is it actually making it great? What's the output of those coding reviews? What's the impact of delivering this tool? Am I getting this NPS satisfaction, other things we can measure? So that's very much tied to what are you delivering? What's the outcome you're expecting to change? And so measuring it from the, are you making the change you're expected there makes a big impact. And so continuous deployment and the tools to enable that. And then is the tool I'm providing or the service or the output I'm providing, is it having the outcome that I'm hoping for it to have?

7:49Yeah. And is some of this always going to be a little bit dependent on the organization too? You're going to have, but there's probably gonna be some overlap, but you might have sort of custom things that you're measuring for success based on maybe where you feel like you have like a pain point today. Yeah, absolutely. And like, I like using specific examples. So we've had teams focused on different parts of the software development lifecycle. And so one of them has been on onboarding. And onboarding is a really interesting thing as you hire or you have people switch teams. Part of it is, am I giving them the right information?

8:21And then am I pointing them to where to go to discover things? And then am I giving them the right training and starting positions so that they can commit code fairly quickly in their journey? And so some interesting metrics to measure there is time to first commit and time to 10th commit are really, really great. And they're kind of standards across the industry for measuring that well. But then there's also like expectations. Does the person joining know there's an expectation on measuring these things? Because if you know you're being measured, that'll change your behavior as well. Yeah, you tend to optimize the things that you measure.

8:54So that's why you got to be careful about what you measure. And, you know, developers, I think of all the customers in this space are fantastic at gaming metrics. Yeah, absolutely. That's definitely one of the factors you consider as you look at this. And then the onboarding thing is interesting. I would think also that potentially impacts retention or job satisfaction as well. because presumably most people starting at a new job want to feel like they can contribute. They're making a difference. If it takes you six months to get to a point where you've done anything, you've submitted your first PR or something like that, this is probably going to lead to a certain amount of satisfaction for a number of people.

9:31Yeah, absolutely. If you know you've been hired to commit code at some level by your job family that you're in, ideally you want to start to show that you can do that because that gives you confidence that you're understanding the space as well. Also, there's that internal anxiousness about I want to demonstrate value as well as that satisfaction of the job is giving me what I expected to. Yeah. Given that Capital One's in a very highly regulated industry, you're going to have certain compliance and security considerations that probably a regular or software company might not have. How do you end up balancing the necessary structure and probably having to slow down and do the right thing while still giving people the freedom to feel like they can work autonomously, they're moving quickly and all these types of things.

10:17Yeah. And this is one of the areas that really prompted me to join Capital One in the first place because I was like, how are they doing this so well? And I wanted to kind of come in and learn. And there's some really great strategies Capital One employees to really help enable agility while still maintaining that strong security posture, that strong compliance. And so the strategy that is employed here, which I really love, is this idea of let's make it easy to do the right thing. So all of the practices, the platforms, the services and the model that we operate is let's give you the best practice and make that your default behavior.

10:52And so instead of having to pick through what database to use, this is the one we recommend. And if that one doesn't meet your need, OK, there's things that you can do to go beyond that. And we can get approvals and go through sort of exception processes where needed. but by making it easy to just pick the thing, it doesn't become a question and that sort of helps with the pattern. So there's lots of best practices built in and then there's standardization. When it is something that should be standardized across the company, we just, the company standardizes and then requires every team to comply to that standard.

11:24That creates migration and other work from time to time but you get great lift because you can put controls, you can put best practices into that standard but they require less work on the development teams. Yeah, and I guess it gives you some efficiency too when people kind of move around in the organization because there's kind of one way of doing certain things. So it's going to be familiar each time, even if you're solving net new problems. Yeah, absolutely. And what does happen, and just like at any company, is when there hasn't been a standard, a few different tools start to exist. And then we sort of bring about a standard or approach.

11:56And that's working fairly well because it doesn't stop teams from starting that innovation or picking something in the absence of a standard. but it does prompt us fairly quickly to establish, well, what should be the standard here? How do you make that decision about sort of what to standardize or centralize versus what giving a team essentially their individual discretion? And then even if there isn't some standard today and an individual team has to make a choice, how do you become aware that the choice has been made and that there might be a time where you have to decide? For example, I'm sure we're going to touch on this a lot with the world of AI.

12:29Like if two years ago, someone started experimenting with an early LLM, then at some point, obviously, you want to standardize, but maybe they were solving some very specific problem. How do you even know, become aware as an organization that that's going on? Yeah, you start to see a proliferation of tools or services that are similar. And that's when you start to realize there is overlap in capability. And then we typically will task a team for what's the right answer to the case for this. And you typically pick this up when we see it as high leverage. Lots of tools are coming in, which potentially prompts more work on security reviews and other things, just because that just increases the level of awareness we have to do on approvals.

13:11But then also we look at things like, is there an open standard? You know, as we look at telemetry, there's a lot that's standardized in the telemetry space. And so picking a standard there simplifies the world for us, then it can simplify what tooling. On the LLM and other front, when you're in an industry like we're in where things are rapidly changing, like the change this year has just been so exciting and incredible. The more you tie yourself to one model or one approach, the more you're going to get boxed in. So from the get-go, we've been standardizing on what is our success criteria and improving upon it each time for every pilot that we do.

13:47And we've been using sort of abstractions to think through, well, if this needs to be a coding assistant, for example, perhaps it uses this API or interface, and we're going to go through this review, and then we're going to use that same sort of template when we introduce another or another. So we're getting some lift there. But that's an area in particular where I don't think anyone's declaring one standard right now. I think we're still going, what's the right tool for the right job? And should we revise that with new information coming in? Yeah, I think it's tough when you're in these like in early innovation cycles, because it's hard to know what would be the right choice.

14:22And the right choice today could change drastically in like a week with the way things are moving so quickly. So you really need to be sort of investing in technology. And if you're building whatever your solution is in a way that's sort of adaptable to this type of change, or you're going to hurt your velocity down the road because you become locked in and tightly coupled to one way of doing things. Absolutely. I think one super exciting thing though with these agentic workflows and other tools that is happening in the industry this idea of lock-in is really interesting because like how locked in will you be if it's easy to migrate now because you have tools that enable that migration yeah i'm excited about just how it opens up doors for us and we're doing a lot of that abstracting and keeping an open mind on areas where there is great innovation happening and we want to be getting the benefit of it rather than just picking one and waiting.

15:13Yeah, makes sense. So I know you have rolled out an LLM-powered coding assistant. Could you talk a little bit about what went into that decision-making process and then what were some of the good things that you saw from this and what were some of the challenges? Yeah, maybe I'll speak about a challenge first, but I think if you go back a year or so ago where we were with an industry and the messaging around coding assistants and AI tools is there was a lot of fear a lot of concern that this will replace my job yeah and so there was a fair bit of messaging we had to do and Capital One does this really well we do this at multiple layers of leaders including our CEO speaking about hey we want to get lift in this space and we want to automate anything that isn't part of the creative problem solving of software development let's all adopt the tools that create that lift and let's try it and see what works So our approach was to really roll this out and encourage everyone to use it and see if it works for them and what it works for them.

16:13And so we went through a full risk review and compliance review and all the reviews that we do to sort of did a POC and a pilot and scaled it out and really started to track, well, who was getting benefit from it? And then as part of that, we created some Slack channels and other things where we could surface who's using it the most. what are the things they're discovering? And it's really fun to watch people share their examples of what worked for them or how they used it. And as we've been rolling out other tools sort of since then, continue to use that same approach of letting peers share what's working well and what isn't, so that we can learn from each other, because it is such an area where there is so much change happening at the moment.

16:52Yeah. And I think, you know, in my experience, I've seen a couple of things that can sort of hinder a company's use of these tools. One is that they kind of underestimate the actual learning cycle of how to use these tools properly. So especially with some of the agentic ones where you're doing sort of more of like a perhaps spec-based development, and it is kind of a new way of thinking about doing your development. I think sometimes people can put in a prompt, they don't get what they want, and then they kind of write off the tool that this doesn't work for me. So I think companies sometimes miss thinking through, like, how do we actually make sure that people know how to use these tools properly and get the value out of them?

17:31So I'm curious to hear your thoughts on that. And then the other thing is that when it comes to developer sort of efficiency, even if the tools for the coding assistants are working 100 % as you would want them to work, the time that developers actually spend coding is maybe 20 % of their job. There's all this other time that they're doing other things, whether it's design reviews or thinking about architecture and stuff like that. So I think sometimes companies are surprised by maybe the ROI ends up being different than what their expectation is because they're thinking about the 100 % time, but it's actually you're kind of optimizing 20%.

18:07So I'm curious to see how Capital One is thinking about that other 80 % as well. Yeah, that's great. And I think really great framing about the day-in and day-out job is not just sitting there coding how everyone works. So I think there's a bit to unpack there. The training and the best practices with some of these tools, we've found the more you can make it use case centric, the easier it is for people to understand the value. You know, I feel like a year ago, there was a lot of focus on how do you prompt well. And there's a lot more focus now on how do you tweak your plan to then get the behavior you want to see happening.

18:44Even the training a year ago is probably not as relevant as some of the training we'd roll out now. on that front because it's just changing so much. But the use case-based training is really great. And we're really thinking about for this type of task, this is the best tool from it and becoming a bit more opinionated. The more we use different approaches, the more we are able to identify this tool is great for this right now. Maybe that'll change, but for right now, this is what we recommend. A spring boot migration or upgrade might use this tool versus a test case generation might be great with this tool.

19:17That's an area we're focused on. And then measuring the impact, there's been some interesting thoughts around, to your point, how much time is each developer spending programming versus other tasks they might do? And so there was some really interesting stuff in the industry around what do you measure? And I'm very much anti, we don't measure lines of code produced. Is that perhaps the worst coding assistants or agents might produce the largest amount of code and that's actually not a measure of value in any way. So we were looking at things like how many suggestions do you accept as being an interesting thing.

19:54But a big part of this is, is the tool providing value to you and over time is it creating lift? And so maybe throughput over a long period of time of a team for a year, you might be able to see some impact there. Quality, you continue to measure to see if that's making the change you want to see. But then the qualitative of are you using this tool and then is this tool providing value and what do you use it for still being the bigger drivers there. If the quality is continuing to either stay the same or get better and if the team is able to do a little bit more then that's really the outcome and the lift we want and hopefully do a bit more of the more interesting programming work.

20:34As much fun as migrating an old version to a new version or resolving an NPM dependency might be that's not where I want to spend my time or where I hope most engineers don't want to spend their time or vulnerability patching or any of these other things that are kind of not where their expertise and skill set really show up. Yeah. I mean, I think a lot of development ends up being these kind of tasks of pushing and pulling data. You're pushing data into something and then you're pulling data into something and then you're transforming it to, you know, render some view. That is not necessarily the most exciting work all the time.

21:08And, or even, you know, I remember when IDEs like Eclipse brought in, it's like, okay, I have a bunch of private variables. Now I can automatically generate my getters and setters. It's like, okay, fantastic. That was very helpful because writing getters and setters is not necessarily the thing that I want to spend 30 minutes on. If I can do that in five seconds, that's fantastic. But are you seeing that in terms of it being something where it does give people more time to either tackle more complex tasks or even sort of shift what the role of a developer is, where it's less about necessarily hands-on keyboard, but more deep problem solving and thinking at maybe a higher level of distraction?

21:47Yeah, I think I was still early, but the things we're learning so far is that for those that are kind of newer in their career journey, it is providing tremendous lift, both in understanding codebases, learning new languages, new frameworks, understanding things that already exist about the code base, the dependency trees, there's huge, huge lift in that part of it. For a lot of the remedial tasks, you know, the basics on style and linting and formatting and the improvements you can make there that are just such low level, but provide great value in that consistency for others. It's great. I think we're starting to see that climb up the stack through factoring and other opportunities, but it is still, I think what we're seeing, it's still a bit hit and miss at times.

22:33And so we very much have the human in the loop in all parts. And then we still have that code review that goes on as well to really ensure that quality remains high. I think the learning and knowledge piece is really high. And then as we're getting into some more of these other tools, we're starting to see mass improvements, migrations, things that are very high value, but low creative, not high value tasks can be automated really well. So we're getting lift on the things that I value for a developer. I think we're still pushing on getting lift for the things where it really pushes them fully into that creative problem solving.

23:12Yeah. I think one of the things you mentioned there was like the code review process. And I think one of the reasons why in many ways, like agents and even some of the things that we're doing around, you know, using these models in the workplace, like engineering has kind of been the tip of the spear for that. is in part it's because we actually have some sort of guardrails around what gets produced and whether there's basically built-in quality checks. Like if it's code that can be compiled, I can at least compile it and make sure that it's syntactically correct. I can run it through unit tests.

23:46Presumably someone, human's probably going to be involved in the review process. It's going to probably go through multiple stages before it ever hits production. And even when I hit it in production, I might be doing some sort of canary-based rollout. So there's a lot of things that go into it before it lands in a customer's experience, essentially, where in a lot of creative work, it's harder to have these kind of like deterministic ways or process in place to actually validate that what's being produced is correct. So I think that's like a huge advantage for this because evaluating the result is hard in a lot of domains, whereas engineering has some kind of built in ways of doing that.

24:22And the other reason I think that engineering is a bit of a tip of the spear is because engineers historically are very good at investing their time into ways of taking work off their plate, essentially like automating tasks and making it so that they can spend time on other things that aren't necessarily like these like rote tasks. And this is the superpower way of doing some of that. Oh, 100 % agree. And I think the qualitative and validation tasks with testing and other things, there's so much we can get lift from the visual testing and behavior of clicking through that are more challenging to write those tests, those sort of feature level tests that are really easy for agents and other things to do, which will again provide lift, but you need some balance in the equation today.

25:06You don't want the agent writing all the tests and all the code, there's no balance in that setup. So figuring that out as we continue to invest in this space is really exciting. You talked a little bit there about these tools being helpful to like a junior developer, a new developer who doesn't necessarily know the code base. And I would imagine having sort of like a not judgmental, psychologically safe assistant to ask questions to is probably really, really helpful. But do you see a difference in terms of people's either use of the tool or where they find value depending on like their seniority, where maybe someone who's more senior really knows the system, they know how they want to implement things, and maybe they see less value in the AI powered assistant?