963: Reinforcement Learning for Agents, with Amazon AGI Labs’ Antje Barth

23:36So this is an example from the coding world but I think a similar pattern will arise with those UI automations. Right now, of course, we want to have, you know, for specific things, a human in the loop that maybe approves a checkout workflow or just takes care of some of the important decisions that are part of this task. But ultimately, as we're building up the trust and we have, you know, specific verifications in place, I think this agency will slide a little bit to, you know, to give it more.

24:07Jon Krohn:I think that makes perfect sense. I do think that is where we're headed. You mentioned at the beginning of your most recent response that you had another example other than summarize me. And now would be a good time to go into that. Yeah, that's kind of how I code, right? This is what I want to get in. It's shifting how we're still using natural language, talking to all those tools, but we're giving it a little bit more agency. Right, right. Gotcha, gotcha, gotcha. So we have a quote from you from a blog. We'll have a link to that in the show notes where you point out to where you point to over a thousand generative AI applications already built or in development at Amazon.

24:44Jon Krohn:So from those, do you have some kinds of takeaways for our listeners from those, from a thousand plus internal deployments of Gen AI, where agents can actually deliver value and where they still struggle? Yes. So it's really exciting, you know, being part of Amazon. Also, we're working and collaborating with a lot of the internal teams here and seeing the different use cases coming together. And one of the very exciting ones is from Amazon Leo. You might know Leo. Previously, they were known as Project Kuiper. They're working on those low orbit satellite networks to bring connectivity into the underserved regions.

25:25So very exciting work that they're doing. And they've been looking at NOVA Act as well, how they can integrate it. And they built some amazing automations that help them do testing QA frameworks across different, from the web to Android iOS apps as well. And then also build additional tooling on top of that, where, for example, they can now, product managers in the team can use Nova Act to, from a FICMA design, from a wireframe design, create test cases even before those get implemented. So there's a lot of like exciting work that's happening. And yeah, I'm also curious, you know, out there, like a lot of you, like startups, developers, data scientists that are listening, like how you think UI automation can come together and which processes you can automate.

26:20But yeah, certainly there's, I think, so much opportunity space. And we're really kind of really just at the very beginning of what's possible.

26:27Jon Krohn:For sure. Are there some kinds of general things you've seen that work or don't work? maybe the UI example is a really good place to go because it seems like Nova Act specifically is designed for browser-based tasks. We started with a browser because it's the easiest way to give us kind of the closest to this universal action space. If you look at like how we humans work, right, a lot of our work is actually happening in the browser where we're working across different applications. We have the different tabs open. So this gives us a very great first point to get to this universal space, but we're also looking beyond the browser, right?

27:05So again, we have great research teams here. I really enjoy talking to a lot of our cognitive scientists who are also looking into those problem space. And one of the main use cases really is, to your point, is the QA testing, right? We see a lot of things happening there for a couple of various reasons, I think, but really kind of, it helps you really quickly to automate some of those, you know, as you develop new applications, you're pushing out an update and you want to make sure that the experience delivers. For example, one of our customers, PGA Tour, they're using Nova Act to make sure the website correctly works, especially before, you know, any golf tournaments where there's a lot of excitement and people coming to the website.

27:50So they're making sure that, for example, the weather information, widget, everything works, all the buttons, everything in there. So a lot of this QA testing on websites, but also like, for example, another one, Hertz uses Nova Act. They're doing automated testing for a lot of their core rental workflows, which help them to just increase the shipping velocity 5x. So it's really kind of, and sometimes internally, we use this playful term that we're calling those norm core agents, which really automates some of the most more boring stuff, right? Like, it might not be those flashy cool things that we sometimes come up with at like hackathons or stuff.

28:33And really kind of what are kind of the core workflows that seem a little bit more boring, seem a little bit more tedious, but you can actually automate pretty well. And we see a lot of interest as a first use case in this automated QA testing, but we can imagine like a ton of more use cases out there.

28:51Jon Krohn:Yeah, I was going to eventually ask you about this normcore stuff because we did have some research on it. What does it mean? What does it mean, normcore? Where does that come from? Yeah, so this is kind of, I think it was coined in the fashion industry, kind of normcore looks, which is a little bit more, you know, like kind of a simple white T-shirt and jeans, something. and norm core for us is really kind of you know those agents focus on more kind of the the traditional kind of the boring use cases you would call them maybe sometimes like exactly that form that i need to fill out a thousand times that application that i need to do maybe you need to apply for something you need to navigate you know websites to get a new driver license renewal or whatever it might be.

29:38So you have those tasks that you're dreading, as I mentioned before, or that is just some work that you need to do. And if we can automate those use cases for you, those tasks in a very reliable way, I think that would be amazing to just give you all that time back to focus on the more higher level, a more creative task we're all excited about.

29:58Jon Krohn:All right, norm core. Yeah, I was thinking maybe it's kind of like the opposite of hardcore. And when you were talking about norm core, you were talking about QA engineering and something that's interesting with this shift to agentic is that traditionally QA engineers were focused on finding bugs, really. But now I feel like a lot of it is probably related to auditing reasoning traces that LLMs spit out. I don't know if you have any thoughts on that. This is kind of a tangential question. I think there's a lot of like things that will change how we're doing our jobs, right? Like we learned this in the software engineering world that we're like more talking to AI than we're actually writing the lines of code sometimes.

30:38And yeah, I think similar things for QA engineers that they don't have to write those brittle scripts anymore, right? If you're looking like how you achieve those tasks in the past, you had to write those very kind of rule-based scripts, like Selenium scripts or something. If you wanted to automate those processes, right, RPA, where you said, okay, here's the button, click that, do that here and here. And whenever some element changed, you had to rewrite those. So hopefully we can, again, like simplify this for those QA engineers, and you can use much more QA through natural language, creating those scripts for you.

31:18And then also with those agents, they're much more resilient to adapt to those changes, right? So you don't have to spend that much time in keeping those scripts up to date. And hopefully QA engineers can focus on the real things. And that might include, yes, looking like, you know, a different type of traces, for example, reasoning traces, etc. But hopefully this gives them also like, you know, a little bit more focus on what's exciting in their job, like figuring out what's going on rather than just spending hours and hours and hours in writing those scripts and updating them.

31:49Jon Krohn:That makes a lot of sense. So going back a little bit, going back a few minutes, at least, you're giving us an example, a real world example of how Amazon Nova Act can be useful for the PGA Tour. And we dug up another example here, which I thought of which i thought was kind of interesting which is with one password and so one password highlights nova act handling complex sign-in patterns which is a high-stakes surface area especially for an application that you're trusting with all of your passwords so how has nova act dealt with secure interaction with authentication flows so agents don't become the weakest link in identity and access right and security is really kind of you know one of them that if not the most critical part as well, right?

32:32Reliability, security, to be able to build a trust. So for Nova Act, we're really kind of building on the security foundations of AWS. I briefly mentioned this before. So you can be assured that all of those executions are running in your AWS account. They are protected. You have full control over them. And also like we give you back control, right? So for example, if you're encountering a password that you need to fill in, you can give back to the user. So basically the user would then be in control and you would, for example, with a playwright integration, you can then prompt like locally in the terminal to, hey, get the password from the user.

33:11It's never being sent over the network, but then the agent can fill it in. So a lot of thinking really goes around that. And then one password, obviously given like they are the password manager, one of the leading ones. So they build their framework around that as well to make sure, for example, the password manager can intelligently navigate the site and then they have their part, how they deal in a very safe and secure way with the passwords. But that is definitely an area where you want to make sure you're not just sending any things over network, et cetera, and really want to make sure you have a very trusted and secure way to deal with those.

33:48In a similar way, like with captures, less sensitive maybe, but also this is another case where we would give, you know, user the control back to solve those. And we will probably see a lot of development in that area as well, what's happening, but very critical and very important to make sure, you know, you're not sending any sensitive data.

34:08Jon Krohn:Definitely. Another critical area with agents, we did talk about this earlier when you were talking about the reinforcement learning gyms that teach agents how to behave. But related to that, something that, and it also ties into the QA engineering thing we were talking about more recently, which is that even though QA engineering is easier in many ways than before, thanks to agents and to LLMs, one of the things that's tough and potentially becoming tougher is evaluating agents because they're so stochastic. In a traditional QA role, you could say the result should be this as a result of this test.

34:47Jon Krohn:And when we run an automation, this thing should output as the results. But with agents backed by LLMs, you're getting maybe slightly different responses can make that part of the QA engineering job trickier, I'd think. So how do you choose the right success criteria so that the agent doesn't learn shortcuts that pass the test but aren't really doing what you intended? For that, it's really important, like on a couple of different layers, I think, to tackle this. So for one, you want to make sure that the agents can generalize well, right? So even if things are changing, they're not tripping over.

35:23So as you're building out those web gyms, you want to make sure they're really high fidelity gyms and the agent learns a lot of things. Like for example, it sounds simple, but like, you know, one webpage is light mode, either one is in dark mode that is already kind of a totally, even if it's the same UI environment, right? It's a totally different experience maybe for an agent. So there's a ton of variations that You can even with the same task in UI, just mix it up a little to make sure the agents can handle all of those. And then also to your point, like how do I evaluate the success? You really want to focus on the end result, right?

36:02Because there might be a couple of different ways to achieve it. And you can look into it like, you know, maybe one took a little bit longer than the other one. But ultimately what you're interested in is A, it's a successful task completion. and then there's a couple of ways to verify, right? Let's say you have an agent and you're asking it, hey, I want to schedule a one-on-one with my colleague and the agent needs to figure out which calendar is to open, where to look like for free spots. And then the end state would be, okay, you can confirm that the meeting is actually scheduled in that calendar.

36:36So it's going to be somewhere in a database entry. Like if you're doing shopping, like checkout workflow or something in that sort. So you have a very verifiable end state to do that. It gets a little bit trickier when there's not an easy way to verify that, right? But right now, a lot of those tasks, there is an end state where you can check and then give the agent a little bit of like freedom, right? For the generalization part, like which way they go to figure this out.

37:05Jon Krohn:I like that answer. It makes a lot of sense. Thank you for making something that seems so complex, easy to understand with a relatively simple solution, actually. I appreciate that. So let's look ahead a bit now to what's next for agents as we kind of reach near the end of the episode. Let's talk about where this is going. Obviously, it's a fast-moving space, very difficult to predict what's going to happen. But in a recent podcast appearance, you describe AI evolving from a helper into a kind of a peer in the coding experience. And we talked about that earlier in the episode together already.

37:37Jon Krohn:In presentations, you've expressed the idea that the atomic unit of all digital interactions will be an agent call. And virtually every customer experience we know will be reinvented with AI. So how will this changing environment not only change how software is built, which we've talked a lot about in this episode, but also how AI engineers, data scientists, software developers, how they see themselves and how businesses value their efforts? So to start like where I see the space evolving, right, which I'm really, really bullish about, is we want to build, especially here at the Amazon AGI Lab, we want to build useful AI, useful and practical AI.

38:15So this is really our core focus here. And what we think this might look like and will look like is that the agents become much more of a digital teammate, a digital coworker of yours. And we spend a lot of time here in our research teams to think about how do humans actually learn, right? So the future of agents, as we see it, is definitely like a multi-agent, a very collaborative environment, interacting with other humans, right? Like imagine you have this digital coworker of yours, how do they learn how to do the tasks, everyday tasks? And my colleague, cognitive scientist, Dr. Danielle Persik, she actually talks a lot about that in her podcast, Making a Mind.

39:03And she really looks into those aspects, right? How humans learn things, how humans collaborate on a task together. And then we're working together on how can we translate that to building those agents? And how can we teach those agents to learn like a human, to collaborate, to ask for help, to being humble? So this is a super exciting space. And if we think about what this can bring us, right? It's like we might be able to just, you know, select the agent and say, hey, can you help me with this task? and the agent maybe checks back a couple of times, but then also it learns from those interactions like humans would do and then can do those tasks more independent in the future.

39:45So I think a lot of things will go in this direction, working with agents together to help us multiply our own productivity, which is exciting. So I think for everyone, whether you're a developer out there, whether you're an AI engineer, I'm super excited about the space. I think how we interact with technology will change. And it has, you know, looking in the past over all the decades and the technology innovations, like we always had adapt. We will still have, continue to do that. But I think there's a lot of positive in there. Again, doing it obviously very responsible and safe and reliable, but getting to this stage where we can all use AI to then augment our own capabilities.

40:28So I think this is what I'm excited about, at least for the future.

40:31Jon Krohn:Wow, it's a really cool mission. Is that kind of in the namesake of Amazon AGI Labs? Obviously, artificial general intelligence is this idea of an algorithm or system that can replicate all the aspects of human intelligence. So it kind of sounds like that aligns with this idea that you have someone that feels more like a coworker working alongside you that could be helping you out on any kind of task. Yes. And again, the focus is really kind of what we call the useful AI, right? So we want to make sure everyone gets value out of working with AI, useful and practical for day-to-day tasks. And this will hopefully reinvent how we as knowledge workers work, how our work looks like, and hopefully loading off those really kind of tasks we dread throughout the week and we can focus on kind of more the creative and higher level tasks of our jobs.

41:27Cool.

41:28Jon Krohn:And one last technical question, I think here before I start getting to wrap up my questions, which is, this is tying together a few different things that you've said in different places into one kind of forward-looking question where on where you see the underlying models and the architectures going behind autonomous systems. For the last couple of years, There's two active areas of discussion have been about how much to rely on large language models, like really big ones, versus smaller specialized LLMs. And then another big area of discussion, too, is how much to push the boundaries of model context, or whether it even makes sense to have million token contexts, considering retrieval systems and tools are continually improving to fill in the gaps.

42:16Jon Krohn:So you recently mentioned in a presentation that Alexa Plus relies on hundreds of specialized expert systems orchestrated together, and that an internal agent in AWS manages over 6 ,000 tools using retrieval instead of stuffing everything into a context window. So yeah, tell us a bit about that. How much context is too much? Where do you see that going? And where do you see this idea of lots of specialized models versus, you know, big monolith LLMs with trillions of parameters that can do everything? I think there will be a diverse environment where all of the different kinds will have a place, right?

42:54If we're thinking like AI at the edge, that will most likely be kind of smaller models, right? Just giving the constraints there. I think for a lot of tasks, the models will be the larger models that have the reasoning capabilities and all kinds of different shapes and sizes. And going back to this thousand, and specialists and experts. So my thinking is like, you have your AI within maybe your company, but then your company does business with another company, right? So we saw this last year, like a lot of hype around the protocols, like how can I connect my models to tools? And then how can I connect one agent to the other agent, interagent communication?

43:32So I think it's pretty much still an open area for research. What's the best approach here to build this? And there are a lot of components, I think, still being built. But my understanding and my thinking of how the future looks there is like, you know, we're basically entering a stage where we'll have what I call like every single interaction will be an agentic AI call. And then if you think about this, you know, not just within your team or your company, how you do business with the other companies. So there will be a lot of, I think, level and layers up there where the agents communicate with each other.

44:10and figuring out, you know, the right technology, the right protocols in that space. And that will be super exciting. So I think, yeah, there will be a lot of place for all the different kinds of things we're developing, which is exciting. And I'm really curious to see how the world looks like maybe even like, you know, a couple of years from now.

44:28Jon Krohn:Great answer. It makes me feel silly about the question in the first place because it's kind of like, which way are we going? And you're like, obviously this is a complex ecosystem where it's going to involve lots of different kinds of solutions for different scenarios. Makes perfect sense. Well, Anje, this has been an absolute treat for me. I'm sure it has been for our audience as well. You're an outstanding speaker on all these topics. It's such a joy to listen to you. And for all our video viewers, you'll be able to see, of course, that Anje was smiling this entire interview that probably even comes through in the audio in the way it sounds.

45:02Jon Krohn:You just seem like such a happy person talking through all this stuff. So for people who want to be able to follow you after this episode and get more of your happy, insightful thoughts, where should they do that? Follow me on LinkedIn. There's definitely a lot of content I'm sharing. Follow me on X. And yeah, you will see me hopefully also in person if you're in the Bay Area. I'm planning on going to a lot of the conferences this year, popping into meetups. So yeah, hopefully we have a chance to meet in person. And I'm super interested in learning what you all out there are building. Perfect.

45:33Jon Krohn:Yeah. So we'll have links to all your social media in the show notes. And then my very, very last question for you, which I actually usually ask second last, but for some reason it just felt right to ask you the followers one first. And so, yes. So my final question today is, do you have a book recommendation for us? I do. So it's actually on my desk right now as well. So my former co-author from the O 'Reilly books, Chris Brackley, just released a huge, I think it's a thousand pages book on AI systems performance engineering. And this is definitely an area, you know, as we're developing systems, performance becomes so important and how to optimize performance for the whole stack, like really kind of from the GPU level to the application level.

46:19And that's on my reading list. I haven't managed a thousand pages yet, but hopefully over the next couple of months, I will achieve that.

46:27Jon Krohn:Well, maybe we can have Chris Fregley on the air to discuss this. He's someone that's been on my radar for about a decade, but he's never been a guest on the show. So maybe we'll have to get Chris to talk about that new tome that he's put together. Cool. Thanks, Antje. And hopefully we'll have you on air again soon. We'd love to check in and see how everything's going over there at HEI Labs. We'd love to. Yeah. Thanks so much for having me, John.

46:55Jon Krohn:Well, that was a terrific episode with Antje Bart in it. Antje covered Nova Act, Amazon's new service for building reliable AI agents, available free to prototype now. Check it out. I've got a link for you in the show notes there. She went into detail on how Nova Act achieves over 90 % reliability by training on reinforcement learning web gyms, where agents self-play through thousands of relevant tasks, making them norm core agents that focus on boring but high value tasks like QA testing and form filling rather than flashy demos. She talked about how the future of agents involves multi-agent collaboration, where AI becomes a digital coworker that learns from interactions the way humans do, and how Amazon has over a thousand Gen AI applications built or in development internally, providing real-world validation for all of these approaches.

47:42Jon Krohn:All right, as always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Anje's social media profiles, as well as my own, at superdatascience.com slash slash 963. And yeah, thanks to everyone on the Super Data Science Podcast team, podcast manager Sonja Breivich, media editor Mojir Pombo, partnerships manager Natalie Jaisky, researcher Serge Macisse, writer Dr. Zara Karchet, and our founder Kirill Aromenko. Thanks to all of them for producing another great episode for us today, for enabling them to create this free podcast for you.

48:16Jon Krohn:We're deeply grateful to you and to our sponsors. You can support the show by checking out our sponsors links, which are in the show notes. and if you ever want to sponsor the show yourself, head to johnkrone.com slash podcast to find out how. Otherwise, support us by sharing, reviewing, subscribing, but most importantly, just keep on tuning in. I'm so grateful to have you listening and I hope I can continue to make episodes you love for years and years to come. Till next time, keep on rocking it out there and I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.