Automated ≠ Autonomous: Developer Productivity and AI | MongoDB’s Tara Hernandez

0:00And one of my other sayings is automated does not equal autonomous, right? Right. And so, yeah, if you have automation that goes nuts, it probably is doing the reverse of what you would hope. I think a lot of what things boil down to actually in our world is what outcome are you trying to achieve and really start from there and then work backwards? Because a lot of times you think, oh, well, hey, if we had a bot that was, you know, like dependent bot, my dependencies are out of date, go fix. Right. Yeah. But if you multiply that out five times, absolutely. You could get in a situation where it could take three engineers all day long, every day to keep up.

0:35How can you build a metrics program that not only measures, but improves engineering performance? What's the right metrics framework for your team? On May 2nd and 7th, Linear B is hosting their next workshop where you will learn how to build a metrics program that reduces cycle time by 47 % on average, improves developer experience, and increases delivery predictability. At the end, you'll receive a free how-to guide and tools to help you get started. You can register today at the link in the description. Hope to see you there.

1:30Hello, I'm glad to be here. Totally awesome to have you on today. And of course, we have a crucial topic. It's developer productivity. Very, very popular today in engineering teams, engineering leaders. Everybody's talking about it. And you probably can't talk about developer productivity without talking about AI. So we'll dive into that a bit as well. and you know i'm really really excited uh to talk with you i think you have a great approach to improving efficiency and productivity and we're going to get into all of that how do you think about the problem of increasing developer productivity i want to start with this is just like the latest name it's like marketing branding right when when When I first started, it was like, oh, there's the build and release engineers.

2:30And then there was the infrastructure engineers. And then what was this whole platform engineering thing? Yeah. And then Google has the whole engineering productivity. And now there's developer productivity. And it's all the same thing, which is there's just going to be a category of people that help their fellow developers do their jobs better. Right. Whatever we call it. And, you know, DevOps and Agile methodology came out around that. And now we've got the space framework and other forms of new research. And it's really, it's just, you know, the challenge is the industry evolves so fast, right?

3:07It went from, you know, local computers with floppy disks, and then there was optical media, and then there were the beginnings of the hosted SaaS-based solutions, and now everything is cloud. And so it's really just this constant evolution of how do we support the industry needing to go as fast as it does without the developers losing their minds, right? You know, the Dora metrics, which is a very popular thing and valuable, I think. It talks about, like, time to deploy and time to restore. And our appetite for that to be as close to zero as possible as an industry for customers is, you know, it's probably unreasonable now just, you know, how fast we expect things to be.

3:49And so I think that's one of the biggest challenges about, you know, trying to be an organization supporting the facilitation of software development within that realm, right, is the big challenge. Yeah. You made me think of a bunch of things there. All right. Hit me. And one of them that stood out to me is like driving developers a little bit crazy. Maybe I remember that there's all these different movements. And then you said like kind of like the marketing terms behind them, like, okay, we're doing platform engineering. We're doing DevOps. We're doing agile methodologies. We're doing pair programming.

4:25No, we're doing Kanban. We got AI coming. We got bots. We got, it's not exactly the same, but like SRE movements. And then there's like, hey, you're going to be a full stack engineer. We're going to remove QA. Everything's going to be automated. You're going to do the QA and you're going to do it. And, you know, right now, I think probably the popular terms that I'm hearing is like the platform engineering, developer experience, you know, AI. and I'm thinking to myself, do you think it is all the same thing? Like from the beginning of time, it's kind of like, hey, we want to ship code faster. Do you think some of these like marketing names or like some of this terminology we're using now, platform engineering, developer experience, AI, do you think it matters?

5:14Like is there meaning behind it? I mean, yes, mostly, but also no, right? And let me tell you what I mean by that. At the end of the day, as a human being, we've evolved over, depending on who you ask, some 3 million years from Lucy or whatever, right? And there's a certain progression of how our brains have evolved and all that stuff. But you think about technology and you think about where were we 50 years ago, 20 years ago, 10 years ago, last year, right? I think industrial revolution is a nice sort of inflection point that people like to talk about. Like the amount of change around us is like hundreds of orders of magnitude faster than the previous 2.9999 million years or whatever, right?

6:07And so if I were to describe what is the biggest value of anything around developer productivity or the development environment, it's to reduce the noise. Right. It's to allow the developer to be able to focus on what they need to do by removing, hopefully through automation or telemetry or insights, whatever, the things that they don't have to focus on because we've helped them prioritize the things that they do. Right. Yeah. Because we just can't, our brains, that's how we burn out. Like we drive ourselves nuts trying to keep track of everything. Well, let's make it so that we don't actually have to have our developers keep track of everything.

6:45Let's make it so that the system allows them to stay focused on the things that are most important. And then as they finish a thing, they can move on to the next thing because we have a process that supports it. Right. And I think that part is really hard. And I think it's, you know, it's my way of looking at it. Maybe not everybody's way of looking at it. But what I think is adding to the noise, and you touched on this, is, you know, back in the day when I first got into it in the 90s, like the build and release engineers were the not, you weren't the real engineers. You weren't, you know, you weren't working on the product.

7:15You were working on the make files and the shell scripts and the Perl code and whatever. And like, fine, okay, whatever. You're wrong, but sure. But now we've recognized over the, you know, subsequent decades that infrastructure is actually a valid thing. You know, look at GitHub. That's what GitHub is. It's an infrastructure company. Cloud Bs, CircleCI, half of all of the hyperscalers, it's developer tools, right? And so now there's a market and we're trying to sell into that market. And that's, you know, so now there's a lot of more marketing around it. And like, oh, Atlassian, I think has made tons of money on like, hey, we're going to solve all your problems by giving you a wiki and a bug tracking system and a CI, like all of that stuff and just use our system and you're great, right?

7:56And not to bag on Atlassian at all, they recognize it probably sooner than most. And so now I think that has added to the noise, right? Now it's like, oh, if only I have the UI solution or if only I have this tool or that tool or, you know, GitHub stack, PR stacks is one that's coming up for me lately. Everything will be better, right? And so, you know, the job of my team is to try and distill the signal from the noise. Will that actually help us, right? Or not? Yeah, I love the way that you frame it. And there's always pros and cons. I think that's definitely in like a capitalism society, the one that we live in, like we get better through competition.

8:33That's the way we work, at least in the U.S. A lot of people in this. Yeah, on the planet today. And again, there are pros and cons to that. But one of the things that I am seeing is that businesses. So if you think about business leaders, even CEOs, but like heads of engineering. You know, now we have heads of, you know, productivity and all of that. They've recognized that. And I'm not saying it's always like not to make money. I know they want like everyone wants to make money. But at the end of the day, leadership has recognized, hey, if we take care of our developers, if we reduce the noise, if we streamline what they're doing, if we make their day to day better, better things happen to our business.

9:19So I think there's a realization of that, which I think probably overall is a good thing. and you have some of these metrics you called out like the Dora metrics, right? They've been around for a while at Linear B. We offer these for free. It's like, you know, you get your cycle time, you get your deployment frequency, your change failure rate. But when you start looking at those at a business level and you say, hey, our cycle time is eight days or whatever it is. And it would be really cool if it would be like five days. And I think that I could get there by making the lives of the developers less noisy, I think it's cool.

10:00I can buy into that. But I also agree, on the other hand, there's a lot of terminology out there because businesses are trying to make money. Yep. I have seen, hey, we work with a lot of companies that work on reducing cycle time. It helps. And a lot of the ways that they are doing it is through reducing the noise. And a lot of the area, like one of the areas that we see is around, you know, reduce the noise in terms of pull requests and the review process there and streamlining it. It was really cool that you kind of gave like a history of this that you've seen throughout your career. And now that we're at the point of, I think you said noise reduction, what are the top things that you're seeing now, like today, in order to either reduce noise or measure it or like where do you stand with all of that?

10:54One thing that I think will always be true, and it's a tension that we have as an industry, of balancing what are the things that are industry standards that we should conform to and tools that we should adopt versus where are the things that are going to be really relevant to a particular company and the engineers within that company, right? Because it does matter. And I will go on and on about, you know, how there's, to me, there's three key pillars to developer productivity. the first one is the obvious one it's the technology like what are the tools that you use but the next two are the processes that you use you know the mechanics of how you organize yourselves as teams right in service of goals and then the third pillar is the communication which is both how do you talk to each other but then also how do you have transparency around the metrics around the key insights that you need to elevate right and then the three of those things or the three pillows of your stool.

11:52But notice you go back, only one of those is tech, right? So people, I think ultimately, is the most important aspect of good developer productivity. It is always going to fall down to people. How do you engage and understand what is going to enable your developers? It's invariably not going to be a tool or not solely a tool. And I think that's the thing that's really important to recognize, but it's also the hardest thing to quantify, right? Because it's intangible or is often intangible. Right. So two thirds. So if we recap, you have tools and then the other two thirds have to do with people.

12:32You have process. And then is the third one like communication? Well, it's communications. Like how do you talk to each other? Like what are the forms? Like, you know, if you have an idea or you need to give feedback, but also like how do you communicate as senior leadership what the important goals are and how well we're doing against them? That's actually a form of important communication. And then the other way around, how to use a team, communicate how you're doing against your little piece of those goals. Okay. So let's do this. Let's talk about measuring. Maybe we can talk about measuring the three areas, or if you want to dive into the people side, whatever you're comfortable with, like how do you measure success?

13:08And then we can move into, you know, what approaches have you seen or are we taking to actually improve the experience and efficiency? So let's start with the measurement side. Yeah. So one of the things I love when I got to MongoDB, it's almost two years now that I've been here. MongoDB has had an incredibly invested culture around testing, right, from the get-go. It was very much early adopter to the point where we've actually implemented our own CI system purpose-built to build and test a distributed document database, right? It's called Evergreen CI. You can see it in GitHub. that's great, right?

13:44I don't like usually when I go to a company, I have to convince them to write more tests. Well, at MongoDB, we actually have so many tests. Like I want to know what tests are providing the most value, right? So we need telemetry to tell us like these tests are actually providing us the most value. Let's make sure we really make sure those tests are getting the most visibility. These tests maybe are not providing that much value or no value at all. let's stop running them because it's it's taking up thought space right or it's taking up consumption time so that's you know so code coverage statistical analysis like those there's tools that you can use to try and like figure that out right but then there's other parts like okay you have an idea or you've been requested to do something well now you need to understand what are you going to do so there's like review time design time how long does that take right and then we're actually getting into door metrics um there right you know from business insight to implementation and then from implementation to deployment.

14:38But here's an interesting one, which is we don't have one thing to measure here because at MongoDB, we have MongoDB that you could download and run on your local Debian instances, right? You could self-host it. But then we also have Atlas, which is a SaaS, right? So obviously we're not going to say, oh, well, for our distributed solution, we're going to have a two-hour turnaround time because our customers do not want to install a new version on their operating system every two hours, right? But our Atlas folks probably want the latest and greatest, you know, as quickly as possible if there's bug fixes and security remediation.

15:13Absolutely. And so then we have to have different metrics for different types of products and then have that, you know, be a thing. Yeah. And then communication, like if my boss, who's Jim Scharf, the CTO of MongoDB, needs to be able to answer a question, how fast can he get to that answer? Does he have to come find me or can he just go to a dashboard and go, oh, look, I need that. Next time I talk to Tara, I'm going to ask her about that thing. Right. Or the TPMs, you know, are they able to report back to the product team how well we're doing? So there's like all these different tendrils of information and how much and this comes back to tooling again.

15:48How much can we automate that? Right. And make those things be automatically discoverable so that we know they're therefore always up to date. Right. So that's another kind of side challenge around information flow. Let's finish the note because you had me thinking about something. We were talking about tooling, right? Then we were talking about process. And then we were talking about people. Essentially, I think you said collaboration. Communication, collaboration. Communication. Okay. We're working with a customer and we use cycle time. I won't say who the customer is, but we use cycle time to kind of benchmark where they stand.

16:29and in our cycle time, the way that we do it at linear B is we have coding time. So how long am I spending in coding? We have how long, once I put a pull request up, how long does it take for that review to start? So how long am I waiting for feedback? And then once the review starts, how long does it take to complete the review? So that's kind of looking at the reviewer there and like, is it a big PR? And then how long does it take to get deployed, Right. Merged and deployed. Right. So that's our cycle time. And what we saw at this company is they started using a lot of bots. So it's like starting to get into the it's not like pure AI, but they had a lot of bot activity.

17:13And these bots were opening up PRs. Actually, 20 percent of all their PRs were actually coming from a bot. Now, some of these are like the more common bots, like the Penda bot, but some of them have to do with like localization and accessibility. And there's more and more bot generated code. Now, what was so that's on the tooling side, they're using a bunch of bots. But what was happening on the process side is all of this was getting put onto humans for review. for review and not all of it actually needed like a human review because sometimes these bots are doing little version bombs or little like tiny tiny changes so they're creating a ton of prs with a tiny change and then on the communication side it was kind of like okay who's going to do this review i have a lot of work on my plate it's not coming from a human and these prs were just sitting there and now your cycle time is increasing and so you know we worked out a way with them where we say okay we look at what the the bot was actually changing and we make a decision if we need a human review or an automated review and so on we you know we have a solution but i wanted to get like just see what you thought about that because that kind of reminded me of putting all of those pillars together i have many sayings right i'm half irish i can't help it right we just we have the bardic tradition and one of my other sayings is automated does not equal autonomous right and so yeah if you have automation that goes nuts it probably is doing the the reverse of what you would hope right yeah exactly thoughtful about what's the outcome and so uh and i think a lot of what things boil down to actually in our world is what outcome are you trying to achieve right and really start from there and then work backwards because a lot of times you think oh well hey if we had a bot that was you know like dependent bot dependent bot is one of the greatest you know inventions of our modern time like oh my dependencies are out of date go fix right huge but if you multiply that out five times absolutely you could get in a situation where it could take three engineers all day long every day to keep up right so clearly piling up and And distracting me from the work that I wanted to do at this company, which is like, I don't know, maybe build cool features.

19:32Exactly. Yeah. And they're not going to last long, right? Because they're like, I'm tired of just reviewing dependable PRs. So, yeah, that is a great example of where you can kind of get yourself into trouble and why I talk about the three pillars. Like technology is one, but only one of the three. but they all do interact because it is a complete system right that the technology enables processes right but then the communication comes in like is the right information happening and here you can see where those three things are out of balance to follow the analogy your school stool is wobbling right it is not the tooling is taking over there's a bot army doing a bunch of work Yeah, tooling is having a negative impact on your process.

20:17Yeah. And the interesting thing is, you know, and I do want to hit on the three horizons, but maybe we can go into the world of AI. Like, it seems to me more code is being generated faster, if that makes sense. That's what the data says, at least the data that we have in benchmarking, like more stuff is happening, let's say. But that's why I think you said like automation doesn't mean autonomy. It's like more stuff is happening, but it doesn't mean that the experience is better. And it doesn't mean that we're more productive. Right. And it doesn't mean that the outcomes that we're getting are actually what we want.

20:58And let me give you two great examples of the things that we've seen around the use of AI. And by the way, like MongoDB very much is pro AI, right? And as a platform, sticking your data into our databases, we're all about that. And we're trying really hard to make that be awesome. And there's a lot of customers that are using us for that purpose. But you really want to be thoughtful about how you use it and why. So here are two areas of concern from my perspective, from a developer productivity perspective. One is, going back to automated design equal autonomous. If you say, you know what, I'm eliminating 50 % of my engineering resources because this AI thing, I am so confident, right?

21:39It's going to just generate all the code we need. We're good. You have two problems from that. One is, what happens when it fails? Right? The customer is not calling you. There's no one to blame. AI did it. I don't know. Exactly. That is going to fly. Not at all. Right? The customer is going to drop you so fast if you were to say something as ridiculous as that. So there's a joke I saw floating around, like, you know, AI saves you, lets you implement code 10 times faster. But the debugging time is now 100 times slower because somebody is going to have to go in there and figure it out. Right. In a worst case scenario.

22:16I mean, obviously, you can't go to a customer and say, yeah, AI did it. Let me well, at least now let me ask AI what it's like, you know, I'm trying to talk to you as a human. So we don't know who wrote that code. Yeah, so that's not going to fly, right? And so AI as an assistant to work, yes, absolutely. I think that is a great thing. But so many people, I think, are over committing on what AI actually provides as it stands now. now here's the other thing around ai that's kind of interesting so copyright law right okay if you if you have ai by its very definition uh generated code is learned from another source therefore by its very definition is prior art so if you have a whole bunch of generated code in your corporate intellectual property you may not be able to retain your copyright oh that's interesting Right.

23:14Like it's legally untested. So like for the things that our customers are doing in MongoDB and Atlas and, you know, in whatever, you know, they're using vector search and they're they're trying to do very specific things that keep them safe. Right. But as a company, if you put too much AI into your actual product code, you might get in a little bit of trouble. You might lose your copyright. Someone might sue you. And we saw that when Copilot rolled out. There was all kinds of lawsuits that instantly started taking place because of the code that was not licensed to be scraped by a machine language.

23:48So it's a really challenging thing. And as companies, you know, engage with AI, as we should, if the genie is out of the box, it's not going away, we have to be really thoughtful about it. And so part of developer productivity's, I think, task is to help keep our engineers safe, right? To, again, not have to work. Here's where you can use it. Here's where you can't. And we're going to make it so that you're not going to accidentally shoot yourselves in the foot, right? That's another form of noise, to be quite honest. Yeah. Well, it's another thing they have to keep in mind. It's not a good thing, I think, for developers if it's like, hey, go use AI-generated tooling.

24:28But also you have to keep all this other stuff in mind while you do it. Like to me, then that would be like, wait, I thought this was supposed to make my life easier. Like why you're putting more on my plate. So going back to my three pillars, right, there is a tool. But then the process and policy and the communication around that helps create guardrails so that they don't have to think too hard. And that's where the three things work together in a challenging way. Here's where the technology has gone faster than the industry actually was probably ready to absorb it. And so for each company, we have to figure out what is our safe space here?

25:07How do we keep our developers safe? And then how do we keep our customers safe? And that's, I think, a responsibility of every company that's getting into the AI space right now. I like that you mentioned the guardrails. And I'll just add one because I've been thinking about this a lot. We're experimenting with some of that kind of stuff like guardrails and policy. At the end of the day, like I'm an AI proponent, I think, like you are. And I'm trying to say, OK, how can we do this but actually increase efficiency? Because it's not correlating yet. And on the guardrail side, if I'm using the terminology, hopefully correctly, I'd like to see that be automated.

25:47I'll try to explain why. Again, I don't think like the developer should have to keep in their head what all the guardrails are and when they should do this and when they should do that. Otherwise, it's like so think about if we had now like a rule engine, which is the rules are the guardrails. And you can tell the developer, yeah, go use like our approved AI tooling to your heart's desire. Now, once you put that pull request up, we have a set of guardrails and policies that automatically kick in for you. Depending on what the code was changed, sometimes it will need a human reviewer. Sometimes another AI could actually do that review and you'll be OK.

26:32Sometimes it will automatically call in, you know, the security team or legal if something is. It's like, that's the way I think about the guardrails. They need to be automated so the developers don't have to keep all of that in their head. Otherwise, I think it would be crazy. A hundred percent. You cannot expect the developers to keep, you know, a thousand things at the forefront of their mind when they're doing their day-to-day work. Right. It's just not going to work. My staff engineers, they say this to me a lot. I don't know where it came from, but I believe it fully, which is, you know, our job is to make the golden path the easy path.

27:07Right. And to make it so that you will naturally fall, go the direction we want you to go, because that's the easiest way to do it. Yeah, I like that. You have to work harder to go out of bounds. Right. Because people go out of bounds when it's easier. Right. Exactly. Yeah, exactly. This is a challenge that you may recall that, I mean, this is probably still true now, but it was certainly true when cloud hyperscalers started to become more and more of a thing, you know, security controls, right? It was really easy to have super bad security in the cloud because we were used to everybody sitting behind a firewall, right?

27:47And so we would have behind the firewall, our security profile would be Swiss cheese. And that was okay because all of our, all right, the network engineers will make sure the firewall is in place. Well, now we're in the cloud and we have developers that are, you know, in the early days spinning up random EC2 instances and, and, you know, and now they're spending$15 ,000 a month and it's wide open to the internet. Because they could, right? And so then we had to put in controls. So, you know, another aspect of humans is that, you know, we tend to learn our lessons the hard way. Yep. And I think we said like some of the concerns, but we talked about using like guardrails and policies and automating that to extract the efficiency out of AI, but send it in the right direction.

28:32But what excites you on the other side in terms of developer productivity and AI? Anything that improves developer velocity, I'm going to be in a safe way and in a sustainable way I'm going to be in favor of. That's my job is to help my partner teams go fast with high quality because that serves our business ultimately. Right. And so as as much as we can incorporate AI such that we improve our time to market. And we improve our sustainable product quality and we improve our ability to to maintain the stuff that we've deployed, you know, so reduce our support, our call instance rate into support, for example, that's another type of metric that matters.

29:20Right. Then we're being successful. And what are the different ways that we can do that? And then maybe now there's a time to kind of talk about the whole idea of three horizons, because it kind of comes into play around this idea. Right. I would love to talk about the three horizons. I have one point that I'm going to make and then we're going to move to the three. Because you made you made me think of something else. OK. One thing that we're doing right now is at some of these larger companies, we're measuring the adoption of Copilot. So that's something that we do at Linear B. so who's using it how much usage how much code is in the pr is that type of thing except yeah and it hit me because the other thing that we're then doing is saying how is that impacting your cycle time your change failure rate bugs found in production mttr and i think that kind of just rounds out the point that you were making of yeah i want i want all of this to happen, but my success metrics, like the point is they improve like these KPIs, right?