#728: The Duck Talks Back - Using GENAI in Your Work

0:00This is episode 728 of the AWS podcast, released on July 7th, 2025. Hello, everyone. Welcome back to the ADBS Podcast. I'm Les Sheeber. Great to have you back flying solo on this one because it's a little bit of a different episode. We're not talking about any particular service. We're not interviewing anyone. I'm just going to talk for a little bit. And I wanted to share some insights because I've had a lot of questions from folks around, you know, this newfangled technologies and generative AI and people are using it for coding and other things and not quite sure how to necessarily grasp it and how it works and how it all fits together and how you can actually use it.

0:41And I'm not going to stand here and say, I'm the expert of all experts. Everyone's learning as they go, but I'm using it a lot and I have a lot of customers using a lot. So I have some perspectives that might be useful for you. So I'm offering them in that spirit. So firstly, I want to step back before we even talk about LLMs in generative AI and how it all fits together to some fundamentals that I think are really important to consider in this technology revolution, really, that we're facing. And that some of the fundamentals have not changed. In fact, they're more important than ever. And in particular, the ability to do continuous integration and continuous deployment is really, really important.

1:22The reason is because the evolution of generative AI technology, the LLMs that are popular at the time, the capabilities of the LLMs, et cetera, the price performance is changing really, really rapidly. And so it's not necessarily about just building a system that can take advantage of this technology. It's a system that can future-proof itself against using this technology in different iterations. So things like the Converse API in Bedrock is really useful because it lets you change your models as you go, have different models, et cetera. They're having to make kind of wholesale changes to the code.

2:00But you have to think about your end-to-end process. You know, let's say you build a system that has some LLMs involved in the processing. You have to imagine from an architectural standpoint that you're going to change those. In fact, you definitely will change those over time. It's just going to happen. And what does that look like from a development flow, from a testing flow, from a verification flow, from a tooling flow, from a observability flow. So nothing of this is new. These are the good practices we should be doing anyway, right? But I think they kind of get forgotten sometimes. And it's really made me think how important this is because as I've been doing some work myself and been seeking to change models and do different things, I'm like, oh my goodness, all these good practices we're supposed to do actually make a lot of sense and are really important.

2:53So how do we actually get some benefit out of these types of technologies? Well, the first thing I'll tell you is, and particularly this is relevant for coding and software development, yoloing and single shot vibe coding is not the way. You're not going to get a great outcome. And so when I see folks experimenting, I wrote a prompt that said, build me a system that does A, B, and C, and it really sucked. But it looked like it was convincing, but it didn't do anything. Well, I'm not surprised. It doesn't work that way. In fact, what I've found is that the classic case of proper planning prevents poor performance, the multiple P strategy, is even more important when using Gen AI to do software development.

3:38So it is actually a huge payoff to spend an inordinate amount of time designing a system up front, thinking about architectural decisions, clarifying your user requirements, your user stories, doing all that stuff up front is really, really important. Now, the good news is, is you're not doing it on your own. You actually do it in concert with the chatbot, with the LLM. In my case, I'm always using QCLI. It's the thing that works for me and I like using it. And so I won't sit there and go, wow, I've now got to create a set of design documents. I'll sit down with a CLI and I'll say, hey, I'm about to create a set of design documents.

4:18I want you to help me. You can help me by keeping them organized. You can help me by asking me questions, offering me different opinions, approaches, choices, et cetera. Let's start the dialogue. And then I may spend hours having that dialogue with the bot, essentially. And it's prompting me to think about things that I may not think about. It may be asking me questions about testing strategies. It may be offering design choices. I may be coming up with ideas and saying, hey, can you go investigate this capability or what are the best options for libraries to do X these days? What's the best practice?

4:54Or go look in the documentation for product Y about how to do something. These things are really important and help get you to a point where you have a, I guess, a coherent view of what it is you're trying to build. Now, you'll see many different luminaries in this space. We'll talk about these types of process, and there's a bunch of different ones that are out there. um reuven cohen's spack one is a real spark one i should say is really interesting um but there's lots you know it's and it's all evolving all the time so if you find a process you'll go oh well that's great but now it's changed again um that's okay but the key thing is is that the skill of building a system has not gone away but what's really nice is instead of sitting there at your desk um talking to a rubber ducky which is an old debugging technique um the duck talks back And so you actually get to have an interesting informed dialogue with something that has access to the vast amount of information across the internet.

5:52And so once you've gone through this very deep process of asking questions, challenges, design decisions, having it do the documentation for you, so good. Let me tell you, you don't have to write documentation anymore. It does a much better job than I ever did. And Markdown is king. And so it will create all these design documents for you. And then the next step is you can start telling it to work methodically to create the prompts to start to create the designs for these components. And this is where things like separation of concerns and good coding practices also come into play. Because another thing I like to do is, I'm an impatient man, I don't want one LLM working on my problem domain.

6:31I want multiple LLMs to work on my problem domain. So having a relatively clean separation of concerns means that multiple QChat CLI instances can be off running and writing code for their particular functional domain and share the information between them. Or you can go with a sort of Git work tree type approach where you're completely running on separate trees that brings other challenges. And again, you'll see this is a space that's evolving rapidly. But this concept, I think a lot of folks are calling it swarming of having multiple LLMs working on the same problem domain at the same time, but dividing up the work is key.

7:07It's again, fundamentals. What do we do when the web server is overloaded? We auto scale it and have lots of web servers. It's kind of the same thing, except we're working to manage state. And so managing state can be challenging, but having sort of a shared set of project files, et cetera, is a quite useful strategy for that. The other thing that's really useful is when you're having a dialogue with the system, it can really get excited and want to do a lot of stuff. It can be quite enthusiastic. So one technique I find very useful is to tell it to just give me a report on what it finds. Don't actually do anything.

7:44So I'll give you a real example that literally I did a few minutes ago. This is how relevant my examples are. So I've created a lot of code to help me produce this podcast. There's a lot of work that goes on behind the scenes. There's Episodes have to get edited and then they come back and they get approved and all that good stuff happens. And that means sharing of data around there. And I've written some Lambda code to do a lot of that stuff for me. And I've been finding some bugs in a new version of this that I've released. I tried to, you know, classic engineer. I tried to make it better. We all try and make it better.

8:19And I've clearly broken something along the way. And it was related to pre-signed URLs. So these are pre-signed URLs that were related to both uploading to S3 and downloading from S3. And these were links I was going to share with people around the place and get sent automatically and all that good stuff. And they weren't working properly. And I kind of knew that the way I was signing them was not right. There was some sort of problem there, but I just wasn't sure what the result was. So I literally jumped into QCLI and I gave it a simple, what you would think is a simple instruction that actually yields a lot of work that would have taken me a long time to do.

9:01Now I'm just scrolling so I can tell you what it was. So basically I gave it a, I said to the system, familiarize yourself with this code base. I'm going to solve some bugs with you. So I need you to be ready. So I went through, looked at all the files, understands all the files that are in the code base, et cetera. And then I basically said, okay, so the issues are related to pre-signing of S3 URLs. I'm having issues with both the pre-signed upload URL and generated URLs for reviewing processed episodes in S3. I'm getting issues where the expiration time is not at the maximum. I get errors when I paste the URLs, et cetera.

9:39Can you do a full review of the related logic for all URL generation? Review the documentation deeply first to ensure you have maximum reference information at hand to suggest fixes. Do not make any changes yet. Give me a report on what you find. And it did a tremendous job. It chunked away, figured stuff out, etc. And it produced a full and robust report for me explaining that some of the problems that I had related to the credentials that were being used for certain signing in certain cases, because if you use one type of credential, you can get one maximum. If you use a different kind of credential, you can get a different maximum.

10:15So we're getting expirations that were earlier than we thought they were getting. There were silent errors because I wasn't doing correct error checking on the actual things that were being created. So I wasn't sure that was going on. The whole experience was bad, which is what I'd identified, but it broke it down for me specifically. And it gave me some specific fixes that I could do straight away. So things like using the right credentials in the right places, implementing credential artificial aware exploration, standardizing the exploration logic, adding exploration validation, updating the notification system, the whole thing.

10:50Really nice breakdown, pseudocode, Python code, the whole thing. And it gave me a set of five actions to do of different priorities and also a testing plan. And so I was able to then say, hey, go ahead, think step by step and implement all these changes. And it went and did all those things for me. And so now it's waiting for me to go test it. So what's interesting about this process is using the technology for the things it does well. So what this type of technology does well is process text really, really well, understand context really, really well, and connect information sources really, really well.

11:29So rather than me having to go and do a deep dive into various documents to figure out what I've done wrong, because clearly I've done something wrong, it could do it in seconds. And what do you do with that time? Well, it's interesting. You can get some of your emails done. You can be reading something else that's important to you. You can be thinking about a strategy that you're trying to work on, et cetera, et cetera. You've got lots of options. What's also related to this point here is how you instrument your session to have access to the tools you need. And this is where MCP or model context protocol becomes really, really important.

12:07It's nothing amazing per se, and that a lot of people are going, well, this is just sort of like, you know, open API, but just differently implemented, et cetera. It's kind of not about that. What it is, is that it's a uniform system to access information locally and remotely. And doing it in a way that the LLM can natively figure out what to do with it without having to be retrained on it, which is really, really important. And so I'm using a lot of the tools that are provided by AWS to do AWS functionality. So for example, I'm a big user of the SAM CLI to do the serverless application model.

12:43There's a MCP tool for that. So it can have much better error checking, much better validation. It doesn't have to use a command line, et cetera. Another one that's really good is the CloudWatch logs one as well, because let me tell you, spelunking through logs is not a fun thing to do as a human, but it's great to do as a robot. So it does it really well. And essentially thinking about all the tooling that you want makes for a better experience. Now, this can also be around ergonomics and things that make your life easier. So you'll recall earlier on, I was talking about the fact that I kind of had multiple Gen AI agents working on a particular software piece I was working on, not the podcast one, of different, much bigger, more involved one.

13:27And I think I had four separate sessions going at once doing different things. And I'm a big believer in human in the loop. So there were sort of, you know, with each of these tasks, there'd be points where once it was done with something, I would want to know that it was done and I could tell it what to do next and assess the success it had in doing the thing I'd asked it to do. But also because these bots can work 24 seven, I was having them run out of hours, doing work while I was off the clock, essentially. But I like the feeling of having my computer doing stuff while I'm not doing stuff.

13:59So I wanted to keep going, but it would mean that I'd need to keep an eye on it to know when things were done. And so I could implement a really simple MCP called say that would do the say command on my Mac. And I instructed it that whenever it finished a particular task, use the say tool and tell me that you're done and you're ready for input. So I'd be off doing something in the house and this sort of disembodied voice would come and say, hey, you know, project X ready for the next step. Like, oh, awesome. I'll go over and give it what it needs and continue on. So again, you have this real flexibility in how you operate the system to your needs.

14:37And again, best practices are evolving and approaches are evolving. Your way is not necessarily my way. My way is not your way, but there's some really interesting ways. So investigating MCPs is really worth your time. It really unlocks a lot of stuff. I know for a lot of folks, the access to internal systems of record within their organizations has been unlocked by MCPs and lets you do cool things. So let me give you a for instance. You can actually get your QCLI session to essentially become an intelligent bot, a repeatable intelligent bot to do certain tasks. And the way you do this is, firstly, I use a really cool prompt that comes from the prompts.

15:16That's prompts with a Z. I'll have to put a link in the show notes. It's a website that has a really interesting set of rules and prompts for Amazon Q. And one of the prompts is called Project Intelligence. And it tells the system how to organize itself to maintain a full set of project documentation and essentially keep up to date with what it's doing and have a sense of intelligence. and what I found is this means that you can also then tell the system to create a persona of itself to do certain functions. So let me give you a for instance. So again, the production of the podcast is an evolved process.

15:55We have ticketing system, we have the preparation of content, we have guests, We have an approval process, all that good stuff. And I sat down with my LLM one day and said, okay, let me have a conversation with you in a fresh project context and say, this is what I want to build. I want to build an entire production flow. I want you to be able to manage it all for me. I want you to be able to interact with an MCP into the ticketing system. I don't like using the ticketing system myself. I don't have to have an interaction with it. I just want to talk to you. And so I spent probably half an hour to an hour telling it about what I'm doing, how it works, et cetera.

16:30giving it the tools that it needs and now I have a complete natural language system that lets me ask it all kinds of questions and do things you know I can have it update tickets for me I can get it to give me a report on what's coming up next what I need to prepare for I can ask it to give me a brief rundown of my next two episodes I have to record and what the content should be I can generate information and then send it to slack groups as well so for example for my co-host I can say hey here's what's coming up and it'll just put it into the slack group all of these things don't use code that's what's really interesting to me in this particular solution it's just created for itself a set of markdown documents that records how it thinks about things and how it processes information and then it uses the mcps to reach out to information sources and it can also write information so for example quip is one of the tools we use it can create a quip document it can update the quip document it can do you know read write type activities as well.

17:27So there's persistence of information. But this makes for a really interesting use case of what you can do. So another example of something I've been toying with just lately, so it's not fully developed or a complete thought process, but I did the same thing with my time management because I use Outlook as my email client. There's an MCP for Outlook. I like to use a tool called Things for my task management. There's an MCP for things. you're getting the hint here, there's an MCP for everything really. And I also have got a document that I created at the start of the year, which sort of outlines all the things I want to work on within Amazon, the projects I'm working on, my priorities, the things I'm going to pay attention to, et cetera.

18:13And again, I was able to create this instance, this essentially personal assistant bot, and I could give it this document, I could give it a set of my tools, I could talk to it about my priorities and start to say, hey, let's look at my calendar and tell me how I'm tracking in terms of what I'm doing. Am I meeting my goals? Am I devoting enough time to the things I want to be working on? What are some of the things that you're seeing are taking my attention, et cetera, et cetera. So you can see how suddenly you have this intelligent view of your world that you can interact with and query like you would an assistant.

18:48Now, the important thing is asking questions in the right way. And again, not just YOLOing it. And so you'll hear people talk a lot about prompt engineering, et cetera, and you can go down a real rabbit hole of prompt engineering. And I think if I look in my crystal ball, I think the skill of prompt engineering will become less valuable in the future, just like the skill of how to do a good Google search became less important as the search engines got better and could understand context better. But today, what you put in is really, really important. So think about your prompting and your queuing.

19:22And so that's an example of trying to solve a problem. I've been saying for a long time that generative AI will disrupt every knowledge-based value chain. And this is what I mean. It's these kinds of tasks that can be automated, compressed, made easier, made richer. You can get better information. But it's not just about paper shuffling or moving things through ticketing systems. It's trying to actually get insight into the process and what's going on and how it's working. So my current stack, and it changes all the time, but like I said, for these types of jobs that I'm doing, I'm reaching for QCLI or QChat off the rack straight away.

20:03I'm making sure I set up my project rules. I'm making sure I set up my project intelligence prompt. I'm making sure I think about the tools that I want it to have access to and give it access to those tools. Then I create a profile for that particular QChat session. And that saves that information altogether so that I know that whichever session I'm in, what I'm actually doing, because it gets confusing when you have a lot of them going. And that's kind of my approach at the moment. And I'm finding it's helping me a lot. Now, it doesn't do all the work for you. You must pay attention. You're going to have to review.

20:39I'm finding that it's really good at unit testing, not so good at integration testing. So I'm spending a lot more time on my integration testing than I used to spend on my unit testing. Understanding flows and documenting decisions is always really important. So telling it to document architectural decisions along the way and refer back to them is also really important because it forgets. So sometimes you'll tell it to do something, it'll do something, and then it does it the old way. Like, hang on, I told you not to do it this way. Please undo it, et cetera. So yeah, committing to Git is also really important.

21:10Making sure you have regular commit points is also important. The good thing is, is that again, there's a Git MCP. So I've often had situations where the system's made a change and go, look, I really don't like this change. Go look at a previous version and see what you did then and redo it. And it would just do it for me. So again, all this stuff you don't have to do yourself to make your life easier. So yeah, that's what this episode was a bit about. just sharing some of the knowledge work changes that generative AI opens up for a technology person. As a builder, as a technology person, we're really lucky that we have access to these tools.

21:49We either know how to use these tools or can learn how to use these tools. But the real skill is understanding the problem domain and applying the tools. And this hasn't changed. Like I've been doing this for 35 years. The technologies have changed many times. I've done mainframe, I've done client server, I've done web base, I've done mobile, I've done cloud, now in Gen AI, you know, the technologies just change and change and change. But the fundamental skill of being really crisp in understanding what the need is, what the problem domain is, and how best to apply the technology to it is always valuable and will never go away.

22:27I think it was Kent Beck who said recently that, you know, thanks to generative AI, 90 % of his skills now have a dollar value of zero, but 10 % of his skills are now 1 ,000 times more valuable. And I thought it was really interesting. I thought that's exactly right. There's a lot of the drudgery work that just isn't valuable anymore because the LLM will do it. But all the other stuff is really, really important. So it's fascinating to me how it's evolving. And so I wanted to take a minute and just share that with you all. and also to hear your feedback. AWSpodcast.amazon.com is the place to share it.

23:00Tell me about what you're seeing, the experiences you're having. Has anything I've spoken about resonated with you or are you like, Simon, you're way off beam? It's a fascinating space. It's changing a lot. So we're going to keep track of it for you. And of course, until next time, keep on building.