Measuring the actual impact of AI coding (Friends)

31:33A lot of leaders I talk to, I can hear it and I can see it in their eyes. They believe that that is what's going to happen. And when I talk to prospective investors, CIOs, it's a question I get asked a lot about even the dx business yeah how is this relevant in a world where there's way less developers right it's interesting question ax dude you need ax agent yeah ax yeah agent experience so it's an interesting thought exercise i i also talk to leaders who's you know strongly believe this is this is not gonna like result in some any sort of widespread reduction in human headcount. I think that's my personal prediction at this time.

32:26I was talking with a leader. I think I've heard it. It's called Javon's Paradox. Okay. I haven't heard this one. No. Okay. It was just shared with me this week, but it's on Wikipedia. It's the idea that when a resource becomes more efficient, it actually leads to higher utilization. So meaning is, I think there's an example of something with like oil refineries and, you know, the ability to refine oil became much more efficient. So you'd think that the sort of investment in that would be decreased. Like, oh, we can have less oil refineries because we can refine oil faster. But, you know, it only resulted in like more production, more refineries.

33:16And so a similar, you know, applying that to what's happening here, like is if we view these tools as making engineers, maybe not replacing engineers, but making the engineers significantly more productive, you know, that that law would then suggest that, well, we're just going to have more. Yeah, we can get more out of per engineer and we're just going to have more engineers and more software faster. Right. So, yeah, it'll be interesting to see how this all plays out. I kind of, I'm jiving with that because I think that I hadn't heard of this principle or paradox before, but it does make sense that when you make something more efficient, you tend to use more of it.

34:02And that's kind of where I'm leaning towards. Like you're going to have a recalibration of what a developer is because there's going to be more people willing to do what developers do. And so there's going to be a wider spectrum of what to do. So degree of difficulty, easier, harder. And then I think you're going to see the definition change, so to speak, and you're going to see more people come into it. You're just, you're still going to need people to think, you know, there's, I can't extrapolate this big enough, but there's still going to be humans to think about the problem of humanity. You can probably offset a lot of that to AI, but I think you still need like this human intellect, this human, I don't know how to describe it besides feels like what feels right to humanity.

34:49I think you still need that in there because it's not quite in the AI. They're just more bits and bytes more than true intelligence. It's, it's not the same, you know? And this shows up. So we just published this AI measurement framework, sent you guys the link. It'd be great to include in the show notes. One of the big questions as we developed this framework was, how do we measure agents? So do we treat agents as people or do we treat agents as extensions of teams and people? In the framework, we discussed this in the paper, we advocate for treating agents as extensions of people and teams. So another way to think about that is that what that effectively means is that a developer is sort of the manager of these agents, but we're still measuring the developer and the team with the agents being an extension of that team, if that makes sense.

35:59Yeah. So so thinking about how do we measure this? You kind of arrive at similar questions to what we were just talking about in terms of the human to agent ratio and balance and relationship and how that will evolve. Yeah, because the effectiveness of the humans being extended is another factor because I may be better or worse at leveraging an agent than you might be. And so this team plus three agents versus that team plus five agents, there's so many variables there to actually whittle it down to any sort of usable information. Well, that's your job, Bobby, not mine. So I'm sure you guys will figure it out.

36:39And you hear more about, this is another thing when I talk to leaders, they're thinking a lot about number of agents. Like what's the right ratio of human to agents? And I think that's a really interesting question. I also think it's not a practical question right now. I mean, you know, this working with like cloud code, like it's not, we're kind of talking about like single threaded versus multi-threaded. Like it's not, we're not at a point where we're really talking about, you know, I've one QA agent and a designer agent and a front end developer agent. Right. That's not really the paradigm.

37:18I mean, I've seen people kind of trying to do that. There's some people doing that. They're on the edge. Yeah. But that's not really the paradigm right now. So I think right now human extension is the right way to think about it. But that could change. In that paper, one of the things that you poll quoted actually was, companies are no longer limited by the number of engineers they can hire, but rather the degree to which they can augment them with AI to gain leverage. It's kind of like what you're talking about there. It's like you're counting agents, you're counting humans, but you really want to just like augment the human ability, not replace it.

37:55Although some of the leaders have been smirking, thinking, replace, replace. Yeah, they're thinking we'll replace them soon. Gosh. As soon as the data shows that we can, we will. Bye. And thankfully, the data is not showing that, right? And I think the secondary effects of software quality, understanding the bottlenecks in the SDLC, like we talked about things like code review aren't going away, human decision-making and judgment. actually when you think about it, for example, like product management, a lot of seasoned engineering leaders, when I talk to them, know that like product management is actually the big bottleneck, not so much like engineering velocity, right?

38:44It's really like product velocity. It's decision-making. It's that life cycle from idea to code. And I think we're going to see attention on those bottlenecks magnified because as we optimize that the coding we're already seeing this at dx you know companies come those hey we thought we were supposed to get 50 productivity improvement we're not seeing that from the ai tools so now we're asking what really is our problem what really are the bottlenecks and so it is magnifying attention on engineering productivity in general because folks are really focusing on that topic right now because they're expecting these tools to be transformative in their organizations so i think that's an interesting trend we're seeing as well so this could actually result in people caring more and investing more in the other aspects of developer experience that people maybe haven't focused on before because those constraints are being magnified when you solve one bottleneck, then you see the other one for what it is.

39:51And you start trying to solve that one.

40:12well friends it's time to build the future of multi-agent software and you can do so with agency. That's A-G-N-T-C-Y. The agency is an open source collective building the internet of agents. It's a collaboration layer where AI agents can discover, connect, and work across frameworks. For developers, this means standardized agent discovery tools, seamless protocols for inter-agent communication, and modular components to compose and scale multi-agent workflows. You can join Crew, LineChain, Lambda Index, BrowserBase, Cisco, and dozens more. The agency is dropping codes, specs, and services with no strings attached.

40:55Build with other engineers who care about high quality multi-agent software, visit agency.org. That's A-G-N-T-C-Y dot org. And add your support once again, agency.org. A-G-N-T-C-Y dot org.

41:15so budgets i'm curious about budgets because as we look at agents as extensions of engineers let's imagine that i'm worth a hundred thousand dollars a year whatever plus whatever and if you give me one agent maybe i'm worth 120 and so are you spending twenty thousand dollars a year on an agent per engineer are you because that saves you another engineer perhaps i'm probably not going to get to two engineers i'm not a 2xer but maybe i'm 1.1x maybe i'm only worth 110 how much do we spend on this i'm sure these people are trying to figure it out because budgets very much have to be actualized and decided on like how much are we going to spend towards this now when you're just trying every tool there is my guess is the budget is don't worry about the budget we got to figure this out but yeah eventually those things need to be figured out because I don't think we're going to get outright replaced like these CEOs want to soon, but certainly we're going to be augmented in a way where your budgeting starts to change.

42:18You start to think about, you know, engineer plus. What are your thoughts on that? And have you guys done any work with regard to pricing these things out? In our paper, we talk about cost and we talk about ways to think about cost. We, I don't think across the industry are at a point where companies are focused on this problem. They're talking about it because they know it's coming. They know it's the next problem they need to figure out once they get over the kind of experimentation phase. But your example right there, yeah, your cost is$100 ,000 per year. With an agent, your cost is$120 ,000 per year.

42:59What does that mean? Again, this goes back to the well then should there be less developed should there be less people right because like we're offsetting that and so a couple ways that in our paper we talk about like ways to think about this so one is this idea of like human equivalent hours and this came up in our conversation here like being able to kind of measure okay how much of human equivalent work did this agent do so not just looking at number of prs or right but like how much human equivalent work so if you can measure that which is hard then you can take that against ai spend and you essentially have like this idea of like an agent hourly rate right so like like your spend divided by number of hours of work produced, work done, that's your agent hourly rate.

44:02So I think that's one interesting number that we're working with companies on putting some focus on. That's a number you can use to start to rationalize what's the right amount of spend. Another interesting metric is net time gain per developer. So that would be, we talked about the time savings. So well, how much are you spending on AI? And then when you take that, convert it into equivalent of what the developer's hourly rate is, are they actually saving time or do they spend more money than the time they saved? Right. Right. So those are two. So Asian hourly rate and net time gain per developer are two metrics that, again, hard to get at right now.

44:54I mean, a lot of the vendors, it's hard to just get the cost and stay on top of the cost information in general. But I think those are two good frameworks for thinking about the net ROI and right-sizing the investment. That's a complicated task. That's for sure. But I'm glad we got some more people thinking about it. It'll certainly be a huge concern maybe 12 to 18 months from now. Eventually, some of these tools got to shake out, I think. and that's what I've kind of been waiting on is like you know I'll let the edge lords do their edging and then I'll just wait and see what shakes out but um it's been longer it's been a longer grind you know it's probably been two and a half almost a three year since JetGPT changed the world and um they're just now getting to where like for the longest time I'd replaced Google with it but I wasn't going to use it for any software until this last iteration of models and they've all gotten to where it's like okay you know we've we've reached a threshold which is which is significant going back to your javon's paradox that actually tracks with me with just as an n of one like i said earlier i'm not replaceable in the sense but i'm just writing more software like i'm not just doing less although i did confess to going and taking a walk earlier than i would have but i'm also just doing more stuff that I wouldn't have done.

46:19Like it just unlocks me to write more software that I wasn't going to write. And I'm imagining all around the world, imagine every JIRA board or pivotal tracker or whatever tool you're using and the icebox, you know, the backlog. And there's things in that backlog that, you know, they're just never going to get worked on because other stuff just goes in higher and replaces them. And it's just a constant grind. And there's so much unwritten software that we're not going to run out. We're not going to, we're just going to hire, hire, hire we're going to augment we're going to write more software some of that software is going to be really crappy we're going to hire more security engineers and then we're going to hire people to replace the software i mean it's going to be just fine i think that's my just fine i think we're going to be just fine now we do change how we do our work absolutely 100 change how you do your work and i think our teams change slightly i think our enterprises change i think less large enterprises probably smaller teams doing more businesses don't have to grow that head count quite as fast but the large ones stay large that's just an intuition i don't know that you guys well even with your you know confessing to taking a walk early i wonder if maybe during that walk you solved a harder problem that you haven't been able to solve because you were happier you felt more fulfilled and maybe you actually had the brain space to just think you You know, so that walk doesn't actually.

47:42I like to think I did. I'm pretty sure I did. Yeah, you probably did. I mean, it's not indicative of less output. That's the problem, I think, is, and why I'm so thankful DX is here, because you've got the core four and this, you know, kind of four degree of measurement across teams, and you've got this newer one for the AI to measure agents. We need those checks and balances because I'm curious, you know, having said that, if the DX core four needs to change or will change because of AI, Like, does, do we need to add a happiness metric in there? Our morale, I think it's kind of in there, but maybe you can speak to it more so, Abby.

48:14But I'm curious if that DX core four needs to be the core five, because we need to measure the human contentment, I would say, like as an individual. And then that individual is part of a team. That team is part of a culture and a culture of a company. I'm curious if that will change because of AI. Yeah, we talked about this last time I was on the show too. Did we? gosh we have the idea of happiness encapsulated in our developer experience index measurement okay it's very intentionally not called happiness because one of the goals of the core four is to make it palatable for executives and i mean not to sound cynical they don't want to measure they're like not now back in 2021 they did when no one could retain developers right um Right now, they can't.

49:08Funny thing, GitHub recently came out with a white paper on how to think about measurements. And we consulted with them closely. They incorporate a lot of the core four measurements in their article. But one thing specifically I kept telling them was don't call it happiness. They're like, GitHub is all about developer happiness. It's all about developer happiness. I said, don't call it happiness because the irony, if you call it happiness, then executives won't measure it. And so they won't care about developer happiness. If you call it, if you kind of frame it as something else, we frame it as developer experience index, which is how you measure effectiveness.

49:52this because we believe like developer happiness is part of being effective and how you build software uh then they'll measure it and it'll get improved and optimized and so uh yeah it's it's a naming problem adam but it is encapsulated in there in terms of like should the core four change we that's something we've looked at closely and and as of now as a organizational way of thinking about and measuring productivity, we think core four still holds true. What's different about AI is you need more, right? There's a lot of new stuff we're measuring. And so the AI measurement framework actually includes the core four in one aspect of it, which is understanding how is the overall organizational productivity being impacted pre and post AI, or depending on level of adoption.

50:50So that's a lot of what we're helping our customers measure right now is, you know, when we look at the adoption curve in our organization, or the maturity curve, I think it's transitioning into now less about adoption, more about maturity. So not just, you know, are people using Copilot daily, but how are they using it, right? Are they using it just for autocomplete? Are they using the agentic stuff? Are they using the AI to help them create the prompt that they feed back to ai right like so maturity like how is that maturity increases does the organization seem more productive based on the data that's the the big question we're trying to help companies answer right now what has been your adoption strategy at dx for these tools yeah yeah you know we work with netflix and one thing that's been on my mind is how much I have not heard Netflix making a fuss about AI.

51:50Meaning like a lot of these companies are like, we need our developers using them. We're measuring how much they're using. Like I haven't heard that from Netflix, which makes a lot of sense because Netflix is predicated, like their entire culture is, hey, we just hire really senior people. Like developers kind of rule at Netflix, right? because they entrust that their developers are the best in the world and that if there's a really useful tool for them to use, they'll use it. And they're going to use it the right amount. They're not going to use it more because we're measuring them. And so that's the approach I've taken at DX.

52:32Now, I've also encouraged like Cloud Code, for example. So I encouraged a few of our engineers, hey, I'm reading about this workflow, voice to text, to cloud code, to this, to that, right? Like one of those more edge. And so I asked a few of our engineers to go try this workflow for a little bit and see what you think. But we definitely haven't mandated it. It's my understanding that everyone's using one or multiple of the tools and doing their own experimentation. But ultimately, yeah, I trust the engineers to use it to the degree, to the extent that it's useful. And I think that's the, I haven't put any pressure on people.

53:17I haven't been like, well, if you don't do this, you're going to become obsolete. That's not a message I'm bringing. We have some really interesting conversations around just candid conversations. Like where do we all think this is going, right? And that's also relevant to DX because we're a product company. we're thinking about products around AI tooling, AI enablement tooling. And so we're having those discussions as well. Can you say more about that? Well, DX, the business, we've gotten to this interesting point. We've been doing the measurement thing for now almost four years. It's going really well.

53:57But a few months ago... You're getting bored of measuring stuff. Well, a few months ago, I met with Drew Houston, CEO of Dropbox. And he said to me, they've been a customer of ours. He goes, you guys are doing diagnostics really well. Have you thought about what about the interventions? What about solving the things that you're measuring? And so that's actually a question I realized we get asked in different ways, but constantly. Can you actually help us? improve. And we've done a lot of things. You guys saw this partnership we have with ThoughtWorks. So, hey, partnering with consulting companies who can come in and help you with the transformation.

54:46We started partnering more with different vendors. Hey, is there a way we can loop in vendors? We don't really want to play favorites, but hey, at least we can map different tools out there to different problems you know areas of the stlc i think we've gotten a point where it's like hey you know what actually there's like gaps in the market like there is no solution for really some of these problems and you know should dx just just go build them and now with ai we're seeing a whole new generation of problems that are being created uh i mean like you guys my previous company was like a Slack app, right?

55:25So like I got in on that right as Slack was just grown like crazy. And so there was just, it was an entire new paradigm and the entire generation of businesses were built on Slack. And so I think AI and AI engineering specifically. So this new way of working, this new way of doing software is actually a new paradigm in which, you know, potentially the entire tool chain could be rewritten. The folks at GitHub and GitLab are worried that Cursor is going to add code review and source control management to their... Just go for it, right? It completely upends the status quo. And so at DX, I don't think...

56:11We're not going after Cursor. We're not going after GitHub. Who are you going after? I think we're going after the adjacent opportunities. the the the ones that um like on the margins that that you know the the big guys aren't we're not trying to go to war with the cursor and co-pilot we're trying to solve the the adjacent problems we see some things we've talked about today and like how do you actually you know how do you upskill developers how do you um how do you optimize your code for llms how should platform engineering teams think about sort of self-service and enablement in the same way that if you guys have followed things like spotify backstage right like big focus on golden paths self-service developer enable what does that look like in a post ai world it's like well enablement on what golden paths around ai tooling ai development workflows uh shared you know cloud we talked about like claude has workflows literally right they have workflows so like you know curating that like how do you create like a standardized set of like workflows that you hire a developer in your organization boom they have this you know menu of of superpowers uh ai powered superpowers that they can so those are the types of problems uh you know i can't i can't get into specifics but but those types of adjacent problems i think are you know new constraints for enterprises is looking to deploy AI at organizational scale.

57:42So not single player mode, right? Like, but more multiplayer mode. How does an organization become successful with these tools is a different set of problems. It's like AI adoption, best practices as a service. Yeah, that's, I mean, that's one potential opportunity, right? That's just one of your ideas. Yeah, that's not saying that's what we're doing in DX. What else is adjacent to this? Like, you know, don't give us your solutions, but what are the other problems? Spill the beans, Avi. he's going to press you to split those beans. I'm just kidding. He's not pushing you. He's just curious. Yeah, I think what we're seeing is there's really two things.

58:17One is that there's this new set of problems, like some of the things I've hinted at. I think there's this new generation of problems to be solved. The other opportunity is that there's actually the pre-existing problems that are being more magnified, meaning post-AI, it's even clearer that these are real constraints. And in some ways, they're a limiting factor to how much value you get out of AI. Because again, if we go back to the idea of agents as extensions of humans, then the ability of that human and the environment in which that human is working in is actually a more magnified limiting factor.

59:00Because now this developer could be getting 150 % leverage, whereas before there was just, you know, at a hundred to, to use kind of the numbers we were talking about before. So what are the constraints to the human? Well, it's probably the same constraints we've had before some new ones, but some, and, and, you know, those types of things we've been measuring for years and seen go unsolved. And, and we think, you know, maybe we should go try to solve them. Are you raising money? Where do I invest? I'll be, where do I invest? You sound like you're well positioned to actually take this, take these adjacent markets It's over, man.

59:35I think GitLab, GitHub, Atlassian have a platform advantage over like a cursor. Sure. Like that battles on, right? It'll be really interesting to see, like, can GitHub leverage its platform advantage that it has being the system of record for code, right? And like developer communication, can it convert? And, you know, being embedded into much of that CLC, can it convert that? into a really amazing platform that can kind of win out over point solutions that are threatening their business model. I think similarly, DX has some platform advantages, right? Namely, that we can actually measure what's going on with all this.

1:00:22You can prove what's going on. We can prove, yeah. And so, yeah, again, I don't think we're not going to jump into the the battle of the AI code agents, but we see opportunity to bring a lot of value to our customers by, by helping them maximize those investments and continue to just optimize their overall engineering productivity. We're almost out of time, but I got to ask you for predictions since you mentioned this, because when you mentioned the, the popular agents, you did not mention Copilot. Curse was mentioned when service mentioned, Claude was mentioned because Jared sort of interjected it.

1:00:57clock code at least um but copilot was not can you give me a prediction what do you think is going to happen over the next year or so given this tumultuous water just mentioned to github or gitlab the entrenched what what happens if they don't succeed in this transitional moment yeah if i were a cursor i'd go for it right i would go for the full stack like be the platform I think the prediction I'll leave you with is I'm interested in seeing, are we going to end up in kind of like the same way we've sort of ended in like multi cloud, like is the end state of this kind of like multi agent, like, you know, different companies will offer agents and models that are more fine tuned to different types of work.

1:01:41And so for the developer, are we going to be in a world where like based on the task or even subtask, we're kind of delegating to different providers and services and then there's an orchestration problem. So that's another problem we're thinking about at DX is, you know, is that the paradigm? And if so, there's there's a tooling layer needed. man i want to be invited to the one of these think tanks y 'all have i want to be in these i want to be a fly on the wall in the room with all this data you got this uh this moat you've got just looking at the landscape and considering less conjecture because they have actual data you know adam and i conjecture but we don't have any data so we're just talking out of our uh certain body parts all right abby we know you got a hard stop we'll let you go thanks for stopping by again you're welcome anytime man yeah thanks so much always fun chatting with you guys keep up the great work bye friends bye All right, that's all we got for this week.

1:02:33Thanks for changelogging with us. Thanks for frenzying with us. Thanks for making your way to Denver so you can meet up with us and live show with us and maybe even hike with us. You are coming, right? I hope you're coming. July 26th, the Oriental Theater, Denver, Red Rocks. Oh my gosh, it's gonna be so much fun. Learn more about it at changelog.com slash live. Thanks again to BMC for these dope beats. To our partners at Fly.io And to our sponsors, Auth0, Auth0.com slash AI, Depot at Depot.dev, and Agency, that's A-G-N-T-C-Y dot org. Have a great weekend. Tell your friends about the changelog, why don't you?

1:03:13And let's talk again real soon.

1:03:35Game on!