Amazon Q and The Future of Autonomous Development | AWS' Adnan Ijaz

0:07Welcome back to Dev Interrupted, everyone. I'm your host, Andrew Ziegler. And I'm Dan Lines. Awesome to be here. We have an amazing conversation ahead of us today. I sat down with Adnan Ejaz, one of AWS's top product leaders. He's driving Amazon Q. It's practically a household name for developers already. It's only two years old, but it's already made so much impact. I was at reInvent actually when they announced it. And like the reception in that room was really crazy. It was a lot of positive feedback from everybody there. And now it's reshaping how people are building things, their shipping software, how they're scaling.

0:44So it's been, for me, a real big blast to come full circle, bring Adnod onto Dev Interrupted, talk about how teams are actually using and adopting these kinds of tools. And, you know, Dan, I'm sure you've seen and heard Amazon Q out there in the wild. Yeah, for sure. Definitely from the developer perspective. Were you there two years ago? I think it came out early, like it was announced in 2023, right? It was in 2023 at reInvent in Vegas. Yeah. Yeah, that's awesome. I see also they're expanding, I guess, Amazon Q to not only development teams. I know they're probably crushing for development teams, but I saw it's also like, okay, use this across your entire business.

1:24Let's just make everything AI. Yeah. Yeah. And we're going to talk about some of like, what's that secret formula that they're able to use to really scale and have that much success with it. And, you know, every day at Linear B, you help engineering teams to automate stuff and fix things that are slowing them down, whether it's like updates or streamlining pull requests. And I know you got an early listen on our conversation with Adnan. I'm sure you have lots of ideas in your own head about the things he talked about, which were all really fascinating. So let's go ahead and jump in. You know, last time you and I talked, we talked about Dr.

1:56Ashuri's episode from IBM. And you talked about how AI is coming up in like eight or nine out of 10 customer calls you're on every week. I imagine that's probably still the case. I think it's super interesting because, of course, it's still coming up. But the narrative, I would say, is shifting and the questions that are being asked are changing. And I think the last time you and I talked, it wasn't even that long ago. It's coming up, but now most of the questions, because people have been dabbling with AI and they're using point solutions. Maybe it's like coding assistance. Maybe it's something else.

2:35Most of the talk now is not around like, should I use it or not? Or like maybe how to, it's more like, what did it do for me? Did I get any productivity gain? I owe this back to my business. I spent a lot of money and I need to report on that. Where I see the narrative has shifted is more from the early excitement with this untapped potential of what AI could do to transform my development organization a little more towards outcome. What did it do for me? How can I prove it? What should I do next? I think they're really pointing out like a really big thing that Adnan even talks about, about aligning people within your organization with their expectations of like what you're going to get out of it.

3:20And something he points out is like, maybe that workflow that you're trying out, that AI workflow, isn't going to give you that 100 % completely done thing that you were hoping and imagining and kind of even counting on it to do. But if it's giving you that 60 or 70 % is giving you a massive head start and understanding how to use that head start. So it really starts also with resetting those conversations internally about expectations. and I know you've worked with lots of leaders who have definitely been burned by hype, other people who've really nailed it in the moment. What do you think differentiates them?

3:56Let's say a few things here. So one with Adnan, the conversation is just amazing and he's pointing out a few things that I'm also hearing from customers. So I wanted to highlight those first and then we'll talk about maybe what the best DevX teams are doing and that type of thing. So he's speaking to, let's make sure that we can confirm a quick win or confirm a win, meaning, okay, let's say AI could do anything for you. Narrow it down. I think he talks about like documentation or the code review. Pick something very pointed, even though it can do anything, and say, I'm looking to get a win here.

4:42And let's just say that it is documentation for developers because developers don't like to write documentation. Make sure that you say, that's my mission, that's my goal, and I'm going to prove that it makes everyone more productive. That's, I think, something, and he'll dive into it more, but I'm hearing that and I'm working on that with customers as well. Because you have to show impact at the end of the day. That's how the conversation has shifted. It's shifted to productivity now. And then the other part of your question, you were kind of saying, okay, maybe what are some of the most effective teams doing or leaders and that type of thing?

5:19It relates back. They're doing good expectation setting. I think it's all around expectation. I think the leaders that are getting in trouble right now is if you're like saying, OK, I'm going to adopt Copilot. It's going to fix all of our development problems. We're going to move like 100x faster or 10x faster, even 5x faster. That's actually not occurring as opposed to maybe saying more of those pointed solutions to say, you know what, we're going to eventually adopt AI across our SDLC. but let's just first start with the AI code review or a documentation because I want to make sure that I can show incremental gains to you the business right that's who I see is doing the best with this right now that's I think the best way to approach it yeah setting the expectations I think is the biggest key there you couldn't have put it better really and when you set those expectations then you can get that incremental impact as you figure out bit by bit how it's going to impact things.

6:19I also thought it was really smart how Adnan uses the example of taking a task that developers already don't want to do and already maybe aren't even doing and automating that first. That way you're not intruding on their process, but you're still giving them immediate gains. That's like a force multiplier I think that a successful team could use. And when they're doing this, it's obviously really important to track that impact to be able to show it. So that's another big thing I'm sure you're seeing from teams that stand out in those scenarios. us. Yeah, absolutely. So that kind of goes back to the first thing that we talked about.

6:51The conversation has now shifted to what did this do for me? Did we actually get more productive? What was the outcome? How can I prove it? That's where the conversation is, which I know Adnan actually touches on and says some smart things. I can give you my viewpoint of the ways to measure, which is actually very similar to what he said. Here's what we've seen successful in the customer base. There's two different ways to look at it. One way to look at it is you can think of it, I think Adnan says the individual developer, but I'll just adjust it a little bit. Think about an individual task. So right now, for example, our company, Linear B, we have an AI code review, which is an individual task that a human, a developer has to do.

7:44You have to review code. Now, one way to look at productivity gain is, okay, we released an AI code review tool, or it could be an AI documentation tool or whatever you want to like insert there into an individual task. And if you have a metric monitoring tool like a linear B or another solution, you'll be able to say, we used to spend this amount of time on code reviews. It used to be, for example, X amount of hours per week. We have now reduced that by 20%. So we are spending 20 % less time on code reviews, which then returns hours back to the individual that used to be doing that review. That's the first way to measure.

8:34Now, the second way to measure, because what the business will say to you is, okay, that's really cool. You're more efficient. You're giving developers time back. What was the outcome then either for the team or the business or for the SDLC itself? And that's where you need the second measurement that I like to call either like the process time or the flow time. Meaning, okay, in the AI code review that we did, yeah, it's saving, let's say, each developer 20 minutes per review, something like that. But we also see that the end-to-end cycle time, so let's say from when coding starts all the way deployment to release to production, went down by 15%.

9:18Okay, that's really interesting to know. Not only are we giving time back to developers on those individual tasks. AI is doing that now. But we also see business value outcome, meaning the flow of work or the flow of value has also decreased in time. And you got to put two of those together. The only other thing that I would mention is where I think some of the maybe not meeting expectation with some of these AI tools. For example, if I just do your documentation work or if I just do your code review work, It doesn't actually mean that your cycle time will decrease. You also have to have other processes or orchestration in place to take advantage of the AI.

10:03So like what we're doing with customers is saying, okay, you did an AI code review. And actually for your most safe changes, your most safe code changes, since the AI review passed and did a looks good to me, we're going to automatically merge that. Now I'm actually getting time back in my workflow process. And for other times when the AI review goes, and eventually it might even get to a looks good to me, we'll still bring in another human reviewer. So what I'm trying to say is I think that's kind of maybe where some of the expectation gap is. Yes, it might be doing something nice for you in like a pointed area.

10:41But if the rest of the workflow isn't also optimized, I think that's where the expectation mismatches where some of the maybe CTOs are saying, hey, it's going to revolutionize the way that we're doing all of our software development. No, it's like you got to do it in stages and make sure that the whole thing is optimized. Absolutely. You've given us a really clear playbook for how a team could even start experimenting, but then track it back to real gains. It's at a formulaic level, like you said, calculating and understanding for developers how much time is saved. But if you don't tie that back to a more aggregate impact across the organization, tie it back to things that are actually impacting the things that everyone else cares about, and showing what you're doing and gaining with that time back, then it's falling short of actually delivering on those expectations.

11:31And then the further and further those get apart, the less successful anything that you try is going to be. So this is like a really great way of tightly coupling it and doing it in stages. And it kind of blends a bunch of different approaches here too, because you have to figure out how it's impacting the developer and then make sure that they're getting their gains. And then you have to draw that up into like a higher gain for teams in the organization more broadly. Yeah. So before we dive into Adnan's conversation, because I know at this point, you know, people are definitely very interested to hear what he had to say.

12:02I just want to start by saying, you know, thanks for framing kind of how we should think about the productivity and measuring the impact of this technology as we go in and understand what it's transformed. For me, it's really shown how like if you remove that friction, people can really focus on that high impact work, which is what those more aggregate metrics and ideas are even looking at. And so after this break, you know, we're going to bring Adnan Ejaz, the director of product management for Amazon Q on the show. And AWS, they have all the parts of the equation. They infrastructure, tools, and context.

12:35So stick around to learn how they've transformed over 30 ,000 applications, completed 4 ,500 years of developer work, and saved$260 million annually from AI performance improvements with their new technology. If you've ever struggled to explain developer experience to non-technical leadership, this workshop is for you. Join Linear B to learn how to translate DevEx metrics like developer satisfaction and AI performance into clear business outcomes. We'll give you proven strategies to align engineering priorities with what execs care about the most. Faster delivery, reduced cost, and ROI on AI investments.

13:17Plus, you'll get early access to our CTO board deck template to make your next leadership meeting effortless. Head to the show notes to register.

13:28And today we have a really special guest in store. Joining me is Adan Ejaz, Director of Product Management for Next Gen Developer Experience at AWS. And he's at the forefront of AI-driven software development, including projects like Amazon Q. And on Dev Interrupted, if you've been listening, you know we've been discussing the dual dilemma that every engineering leader is facing right now. AI is evolving at a breakneck speed. And if you don't start experimenting now, you're going to fall behind. But if you invest in the wrong AI initiatives, you burn time, money, and worse, trust. So how do you experiment without it blowing your credibility?

14:09That's what we're going to unpack today. Anand, welcome to the show. Thank you. Thanks for having me. I'm really excited to talk to you about all the stuff that you just mentioned. Yes, really excited to dig in. And the topic today is really front of mind for a lot of our listeners. And, you know, everyone's talking about AI agents. And it's early days, but it feels like the term has already started to lose some meaning. And I know you've spent a lot of time at AWS thinking about the evolution of this type of tool and what it means. So maybe let's start with some basics. What makes something an AI agent?

14:45It is definitely the first question we should take on because the word is being overused, particularly over the past four or five months, it seems like everybody's trying to put the word and use it. So the way we describe it, or at least we think about it, is think of AI agents and AI systems that are able to perform a job autonomously for you based on the task that you assign it to. And let me break it down into three parts. At a broader level, I would say the AI agents have these three components. One, they're able to take a goal or a task and break it into some sub-goal and sub-tasks. Then they have the tools, they have the mechanisms to understand the environment that they're in, they gather the information, and then they use that information to go to work and then do the job autonomously.

15:38You know, not just like you prompting every single time for them to do certain things. They take the first two, put them together, and then do the job completely for you. So for software development agents, what does that mean is if you ask it to go implement a website or a feature, it is the agents are able to understand what they need to do, how they would go about it. They are able to come up with a plan, break it down into tasks. Then they have the tools, like maybe they have their own version of exploring the code, understanding how the code is written or what is the context in which they're operating and then use that information and then they go about trying to find the best implementation it can take several minutes for them to come back with the implementation that you need so that is the definition of agent versus it's important to also talk about what is not ai agent and still uh factored as uh agent at least the according to the definition that i'm putting forward is you have an llm which is using a database a rag and you're just going back and forth with that.

16:46It's still very useful. It's still very helpful. It can give you a lot of information, even the programming questions. But the true definition of the agent, that is not the agent. It's not doing the work autonomously for you. The key really falls in the autonomy, that it can do things on its own accord and coupled with, of course, tools. Because when we use a chatbot AI tool, we provide context to it. That's what our conversation is and what we provide it. We can even use tools. Most of those kinds of conversational chatbots, they have ways of implementing tools. But when it becomes an agent is when it's making those actions autonomously.

17:24And that distinction is really important because if they're going to be making actions on their own accord, then it becomes even more important for developers to understand how they work and the types of tasks that they're best suited for. Since they're doing them beyond themselves, it's kind of like they need a little more oversight on the things that they're actually accomplishing? We should decouple or clarify the autonomous nature of the agents from would human be missing altogether or they just like on their own figure out what they need to do. Humans can summon an agent. They can say, hey, go write me, improve the test coverage for my project or for my application.

18:06What happens post that is really where are you going line by line and say, here is my function, go write me a test. Here is my another function, go write me a test. I didn't like this, go change that. Versus all you say is that here's my project, here's my application, go improve the test coverage for me. And then it's able to act autonomously. So that is where it comes in. Because oftentimes people think of this, they hear the word autonomous and say, am I not needed in the process? It's just gonna go on its own and do everything. But it's really the task is coming from you, you're telling what needs to be done in terms of what the job to be done is.

18:41But then how the AI is approaching this is more autonomous, is able to, you know, break it down and figure that out. So that's really where the distinction is. So, yeah, you're spot on in your comment. Yeah, and that's a really big shakeup. It does a change up in kind of how developers should be spending their time and what they need to be focusing on. And when you see teams that are adopting AI agents right now, you know, what are some real world patterns that you're seeing those successful teams do? We are, it's still early days, but we're starting to see really interesting and powerful patterns and use cases and results emerge.

19:17I would first start at home, like at Amazon. The interesting part of my job is because of Amazon Q Developer, which is the AI-powered assistant that helps with all aspects of software development lifecycle, whether you're writing code, writing tests, doing code reviews, managing your application in the cloud, or you're transforming them, going from one version to another, older version to a newer application. And because Amazon is also a large engineering company, because a lot of my peers and colleagues are engineers, so I get to see the whole aspect of how the agentic interactions are evolving through that lens, because developers are using it, they are putting the Q developers' agentic capabilities to use.

20:04So I would say the biggest one that we talked about it extensively with the past several months, we used the code transformation agents to take the Java 8 and 11 applications and modernize to Java 17. and because of these agentic capabilities that are able to do the work autonomously at scale, we were able to save 4 ,500 years of developer work and$260 million annually in cost savings and the performance improvements on all of that. We modernized over 30 ,000 applications. So essentially, putting agents to use to modernize old code bases. In this particular case, we modernize the Java code basis.

20:49Then the other thing that I'm seeing, both internally and externally, teams are using these agents to bootstrap the projects. So it's not like, you know, just say, hey, go write me a website and it's done and you're just done and you deploy it and it's not. It'll go, the AI would go, the agent would go, give you the best possible starting point. And then there are a lot of interaction happens back and forth. We could say, you know, can you please do this? Can you please add that? Like, here is my specification. Can you change that? The unit test generation that we're seeing, a lot of people are not just the, hey, function by function, just take my code and write this.

21:25But here's my entire application. How can you generate unit tests? How can you generate documentation for me? Do the code review? So these are the four or five common patterns that we're starting to see emerge. The stats around how you use Amazon Q and the effects that you can have from it using that kind of agentic workflow at scale is really impressive. And I think there's a lot of lessons there. As you know, like AWS, you have so much context about the organization and the products that are within it, but you also have so much tooling. And like we said at the beginning of our conversation, those are two of the critical elements of what makes something agentic.

22:00And then you, of course, also have the infrastructure to scale the autonomy part. So right there is like a very clear anecdote of how those three factors can really influence the success. So one, I think that's amazing. And another thing I heard that really resonated with me was about the different levels of autonomy that you might have within different projects. And you might have something that's lower stakes that get started. And I think that starting with those incremental projects is key to getting buy-in from folks that are looking at how you're adopting and using it. These agents at scale when they're working, they're requiring probably a large amount of context and tweaking in real time based upon the actions that they're taking.

22:45So how can teams best think about showcasing the best work that they can do for their agents? Like if they're going to mentor their agent to be as effective as possible, how can they work as a smarter engineer to help their agents also be more effective? A lot of this is how the agents are built and what kind of additional context they can gather, what kind of tools they have at their disposal. For instance, I'll give you an example of the software development agent that we have in QDeveloper. The way it works is when you say, go write me a shopping cart API in my web commerce website, right? It is able to take that, break it down, but then it has all these tools at its disposal.

23:29And by tool, for instance, one of the tools that this agent that we have built is the text code, which is essentially an IDE environment, but represented through text. So just like a human, when a human is writing, you know, a developer is writing code, they would go explore the project files, they would open one file, open another, try to build an understanding of what this application is, and where do I go write that application? The agent is doing the same, but we have equipped it with the tool to go do that. So therefore, to answer your question, you know, more broadly, a lot of this is in how the agents are built, you know, and therefore the very first question that you asked me, I think that is why I feel like it was a very important question because these little things make what an agent, a true agent versus it's more like conversation back and forth.

24:19So if you build the agent the right way, then you will think about the memory in the agent so that it's able to understand its interactions like you ask it to do a code review and then it does a code review and you take certain actions then it's able to learn that okay here is what we can do in this environment it is able to collaborate with other agents to do certain tasks it is important for the engineering teams when they as they start they pick the right tool even in the AI it's early days but you can literally find a new tool emerge that tends to do AI development pretty much every other day I mean if I'm not exaggerating.

24:57So it is important to pick the right tool and really explore those authentic capabilities. And then you go about it. And then how would you go about it? I would say the teams should really start off things where there's low-hanging fruit, right? For instance, generating documentation for your project. I have not really run into very many developers who have told me that they enjoy writing documentation. I'm sure there are, and there's nothing against them, But most developers would tell you that, hey, look, if you can do your documentation for me, accurate documentation is great. You know, so when you summon an agent and say, here's my entire application, can you go generate me a readme file and just keep it up to date?

25:39Just go do that. Right. That is the task you can start there. So you're not even really when the engineering teams are coming in and trying to bring in these agents into their workflow. there are things that they can pick which will not necessarily be adding any value and start from there and then build off from there. In the example you gave, like you have an AI agent you've summoned to be in charge of maintaining this readme or this documentation alongside something as it's being developed, that sounds like a lot of free up time. And so now the developers that are building those features can focus more on the critical things that need more attention.

26:13And I think going back to your example with AWS, that seems to be the prevailing takeaway across the board for why teams should be using agents is because of how many developer hours it can save them. And I think I would say we're starting to see the shift in the customer usage behavior. A lot of early days of AI usage across development teams were the inline completions, like the autocomplete. As you're typing the editor, you get the next suggestion, you complete that. It's still being used, but then came in the chat, the conversational aspect, where you can just go and still popular, people use that.

26:49But now we're seeing, particularly I would say over the past quarter or so, a strong emergence of not just the options that are available. We have had, you know, I put a plug-in for QDeveloper. Last year when we launched, they made the Amazon QDeveloper generally available. We're the first to have proper agent, the definition that I laid out, the software development agent. And that is we've been talking to customers about the capabilities and customers who use that, they love it. But now we're also seeing that go mainstream, where now there is the developers who are probably using inline capabilities and chat before now exploring more agentic capabilities, and which is a natural evolution in their journey.