Amazon's Panos Panay: The Reality of Building Alexa Plus and AI Assistants

0:00Where is Alexa Plus and how is the battle to bring you a personalized AI assistant shaping up? We'll talk about it with Amazon's head of devices and services right after this. Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge, and it's simplifying car shopping. Using self-reflection and layered reasoning with live API checks, it doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade in value. advanced, intuitive, and deployed. That's how they stack. That's technology at Capital One.

0:41Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. We're joined today by Panos Panay, the head of devices and services at Amazon, who is here in studio with us to talk about the state of Alexa Plus, where the battle of AI assistance is heading, and plenty more. It's great to see you, Panos. Welcome back to the show. Thanks, man. It's great to see you, too. So we spoke last in the spring. February, probably, yeah. March? March, when you just had your March event. And the promise at that event was within a month, we were going to see Alexa Plus.

1:19Yeah. What is the state of the rollout? Because we've had folks who have listened to that episode on this show who said, I'm waiting. All right. Send me a note if you're listening. Send me a note. Where should they send it to? Where do they send it to? I've got to be careful giving out my email. I think it might get me going. How about this? I'll give it to you. You give it to them. You can write to bigtechnologypodcast.gmail.com and I will forward it. Do it. And anyone who wants access, we're giving access. Over 10 million people have access right now. But that's out of what's been reported to be 500 or so million devices.

1:54Large numbers. Very large numbers. Yeah. So, yeah, the first 10 million. But it's a rollout. So we call it early access. It's been pretty extraordinary. Like the people that use it, it's been fun to watch, getting the feedback both ways, like, you know, where it can be better, but also just the energy around the usage, how it's being used. It's been pretty, it's been pretty remarkable. And, you know, you kind of learn, you think about Alexa, we're rolling it out slowly for, I think, for reasons that we just have to make sure our current customer base, you never want to abandon your current customer base, just can't.

2:32And while it will work on 97 % of devices out there right now in people's homes, we still want to do it in a way where when we switch people over because they choose to, that they feel great about it. And so right now early access is, hey, ask for it. We'll give it to you. It's that simple. End of October, though, just so you know where it's at, we're going to be rolling it out to everybody. If you buy a new Echo device, we launched four new Echo devices yesterday. We're pretty pumped about it. Or I'm pretty pumped about it, maybe. But all those folks will get it right out of the box. Okay. So end of October.

3:08Yeah. This show will air right around then. Okay. Everybody will get Alexa Plus if your device is compatible or if you buy a new device. Yeah. You'll opt in if you want it. Like, I'm not going to – we're not going to just give it to anyone because, you know, people love their classic Alexa. And we want to just serve our customers like the way they want to be served. But I think everybody should get it. So I think it's good that everyone's going to get it or have the option to. I want to ask about this sort of gap in time because we watched the live demo on stage in March. Yeah. And then the notion was it was going to come out the next month.

3:47Yeah. And I read that at least as everybody was going to get it in a month, not a small section of people. and was there a difficulty in getting the technology to work that led to that delay or was it you saying we just need to test a little bit more? What happened? Well, the way what we had done and we had asked for if you wanted to get it, you know, email me when ready type of thing, like create a list and we're going to give access to customers as they go. and then as the product continued to develop and finish it was then rolling it out slowly to make sure because you have to learn that all environments are so different and so the whole concept and construct was learn as we release and so we had basically tranches of customers that you release to that were on this list and we got it to everybody who's asked for it at that point and then that list continued to grow and then we continued to release it but it really was about yeah, making sure the product was refined to a point that when we did put it in people's homes, that they were both excited about it, but then can use it for everything that they're already using it for.

4:57There's some nuance on how you would use it. And you're kind of learning to use the new Alexa, not complicated, just nuanced. And so it was just a little bit more methodical. I would say we rolled it out as we intended, but also it did feel a little bit slower than I think what people expected. So maybe it was about how to balance the expectation versus what we delivered. I was listening back to our conversation from March and a topic came up and I was like, wow, I can't believe I didn't ask a follow-up on that. Oh, no. And you've already brought it up now. Oh, no. Which is that you have to think about the current.

5:31And then I came back for more. You're back for more. Then I came back for more. I have to say. No, it's great that you're here. Sometimes some people would probably think, don't do it. Don't do it. No, but we should say it's great that you're here For many years, we didn't hear from Amazon executives in the way that you're having the conversation today. So I'm thrilled that you're here. Oh, cool. And I'm thrilled we're able to talk about it. Cool. This is not a question. And you've been doing this five years. Five years. Doing this five years. We just celebrated our five-year anniversary. Which is rad.

5:59I mean, and I think that's cool. Five years in. I'm sorry I couldn't make it last night, but that's a very cool thing. We had a nice party. Yeah. But I was also, I mean, I was covering Amazon long before that as a reporter. And wrote this book called Always Day One. and this is why I was upset that I didn't follow up. And it's not a gotcha. It's actually, I want to learn a little bit about your product philosophy. Always Day One is something Jeff Bezos popularized within Amazon. To me, it always meant you build as if you're a startup. And the nice thing about being a startup is you don't have a flagship product to maintain.

6:32And I think the message from Bezos, if I read it right, was a lot of big companies spend their entire existence trying to maintain their flagship product, and they don't build for what the market needs at that moment. And that's the nice thing about being a company on day one. You just build what the market needs. Yeah, you can start from zero. You can start from zero. And so that sort of brings us into this conversation of how you approach the Alexa customer. You said in our last conversation and just now, you have to be careful about the way that the current customers are using the product when you're introducing something new.

7:12Yeah. And there's that leadership principle within Amazon, customer obsession. You're obsessed about what the customer wants. And that's where that comes from. And then there's that balance between... And believe me, by the way, if you don't do that, you just lose trust. Right. You lose trust with the customer. You lose it overnight, Alex. Like at a level that they don't come back. they didn't ask for it and even if they did and you didn't give them what they wanted you lose trust and that's such a big deal if you're going to be obsessed but go ahead, there's a dichotomy here I think you see where I'm going because this is like a core question about Amazon in the age of AI is can you hold that customer obsession people who love your current flagship product and that always day one value together and have them work in sync You can.

8:05You absolutely can. But, you know, one of the most incredible parts of Amazon is there's a level of patience to construct, like, where are we headed? And a lot of time, invention takes time. It doesn't mean you're not being relentless. It doesn't mean you're not acting as if it's day one. Like with Alexa Plus, we are being relentless. If you just look at the breadth of what it can do with the amount of APIs it can call, with the amount of experts that are embedded, the fact that you can be full conversational assistant and call them, this takes time. And when you have hundreds of millions of customers, the things that they do, the things that they connect with.

8:40I think we talked about this last time. I don't necessarily want to get into that. But I think you can still be relentless and you can still act as if it's day one and never, never dismiss or not put your customer first. It's very real. And I think that's what we're doing right now. And we don't act. So when we're building, when we're creating, when we're focused on the product that we're just getting after, it is relentless. We are going as if we're not like year three, slow down. Where's the year five or year 10? Where's the customer at? Let's just look at the data. None of that is happening.

9:16Right now it's full speed. What does the customer need? Are we sure we know what the vision is? Are we locked in? And then the team pushes. We're really clear where we need to go. Then there's the commitment to the current customer. But by the way, you're also relentless about that. And this is one where, you know, I see that economy like, well, you have to serve your customer. That doesn't mean it's not day one. It just means you every day you wake up worried, thinking, understanding, learning about the customer, but then acting, acting with, you know, the same passion and relentless focus you need to move the ball.

9:51And I see that happening on Alexa right now. Okay. So let's go one level deeper. Okay. And I could be wrong about this. But I'll give you my read of what Bezos was saying with Always Day One. And since you're in the company, you can help me get to the truth. I don't know. I mean, you know, this will be tricky. I'm not sure I can get into the minds of everybody. But I'll do my best. To me, the message – so when I came into my reporting, I wrote a book title, Always Day One. I came in thinking that day one mentality meant that you just kind of work 24-7 and you have that relentlessness. And after spending time reporting on the culture of the company, my perspective shifted a little bit.

10:30And it was a message. To me, it really meant it was a message to the company. Don't worry about what got you here at this point. Flip when you need to. And one example is Amazon.com, which was this first-party marketplace, flipped effectively to a majority third-party marketplace, which caused real disruption with the first-party vendors who were used to working with a certain system and now found internal competition. Now, for the customer, on the customer side, you never really saw that disruption because it was all taking place effectively on the back end. But with AI, it's different because if you have a system where you're like, all right, it doesn't matter what got us here.

11:12Let's build what the market needs. The customer can feel that disruption. Yeah, I think they can for sure. And that's where the patience comes in. But I still, even as you say it, I go, but the mentality remains this relentless day one. You know, for the first, think about it, activating the first 10 million customers, it's a switch. It's not. We literally build, go, take it, move. And that engagement happens. We learn as fast as we can. And then we're continuing to iterate. So I feel like the spirit of it remains. But I don't think you ever abandon your customer. I don't think that would ever happen.

11:54If you're acting in a kind of day one mindset, you're serving the customer. And so the change is okay. Your example, I wasn't there for the transition, but I will say let's just use Alexa as an example. Right now, we are building on the spirit of like when they're ready, go. And they, meaning the customer, when they want it. Day one is about building, creating, knowing where you're headed, starting from zero, making it great. And then part of that is bring your customer along. It has to be. You can't. You just never leave them behind. But to your point, I think your example might be perfect. I'm sure there's a team that was going full speed.

12:37I don't know the details, so I can't. Like I just got to be careful. I never want to be presumptuous. But now you end up in this place where it serves the customer, then serve them. And I think that's what happened. Again, I wasn't that close to it. But when you think about Alexa, the same thing is happening right now. I'm not even trying to obfuscate. I mean, as you say it, I'm like, I think that's exactly where we are. And I'll say that humbly because I always worry like, oh, I'm overstating it or I'm like, you know, I take so much. Like Alexa should be this much better and it should have been done.

13:07And, you know, you should have known this. And you're like, okay. We have a team that's pretty passionate. They know what they're getting after. But I'll never leave it. We just won't leave the customer behind. I just won't do it. Yeah, I think it's one of these things that's happening with a variety of big tech companies today, which see the promise of generative AI, see the bumps of generative AI, and are deciding how quickly to bring it forward in their products without alienating those who love their flagship products. Like we had the people who were running search at Google in here a couple of weeks ago, a couple of months ago.

13:44And it's that balance of like you have this great traditional search product. A lot of people use it every day. And then when do you decide to flip it and make it AI mode? And I think it's very interesting to see all these companies going through this moment of when is it time to flip that switch to generative AI? Because like you mentioned, you mentioned when we spoke last, a startup can do it quicker because they don't have to worry about their current customers. Yeah. And I think you can, it doesn't mean they build it quicker. It just, they don't, you know, you start from zero and you build up, you know, versus starting from hundreds of millions and handing it over.

14:20I think that's just reality. But it doesn't mean we're not fast or not. That's where I take exception to when I hear it. It's good to hear your perspective on this. What's taking you so long? I'm like, wait, hold on. There's no other agent out there or kind of assistant that's using tens and tens, 70, 100. I won't give you the amount of experts, but those experts also, all of them looking across, calling hundreds and hundreds of APIs and being able to serve. There's no other expert out there that I can send my daughters. She's 15, man. She's amazing, and she's playing on the golf team at school, which is kind of a weird thing.

15:01She's just – and I just wanted to see – like I want to get to a match. And so she sent me a picture of her schedule. It was a picture, you know. I'm just like, can you come? I'm like, yep. So I take the picture. I just share it with Alexa. All right. In that moment, Alexa sends me a notification. I open the app. The app says, do you want me to download everything here or do you want me to add everything to your calendar or do you just want some of these things? And in the future, it's very smart. It's just it's agentic. And it's like, do you want me in the future? Do you want me to add them all or do you want me to ask you again?

15:39I said, just in the future, add them all. Thank you. Add everything. My calendar populates with the location, the name of the school they're playing against, a link to directions, and basically the calendar entry. And then comes back and says, here's a few conflicts just so you're aware. Maybe you want to resolve them. And here's what I suggest. Okay. What just happened? That seems simple, right? Not to me. You added a calendar event. No. I gave it a picture. and now of course it can read the picture but in most cases it just spits out here's the summary. That's not what it does here's the summary and then went and took action and then updated the calendar and then Mary and my wife got the updates on her calendar and so now all of a sudden I think for the first time in like our kids high school career we actually have we actually have the kids like sports calendar on our personal calendars because it's always so hard you know it's like over here over here over there you have to manually enter it Nobody's going to do that.

16:41Yeah, it's like doesn't happen. And then there's just like an argument. It's Monday. Like, are you going to COSIS? Is this on Thursday? I'm like, I didn't even know it existed. Like, why didn't you tell me? Like, well, don't you remember the mail? You're like in this place. But now it's all solved like that. But what happened? There was an expert, the calendar expert. There was a communications expert. There was an understanding the document expert. Then there was APIs called where you're turning on calendar entries, then checking against old calendar entries, and then updating other. And when you start seeing that all happen, yeah, that's pretty – it's incredible work.

17:12But I look at it and go, it's just not happening anywhere else. Not yet. It's not a simple piece, but it's so delightful when it does happen. And back to your original point, like that is a – that's a great feat of engineering at speed and just kind of product making that we're pretty proud of. And then you start seeing it when we say like being day one, like there's nothing that holds that back. Right. There's just nothing. You're starting from zero to create that. And then that's when you're serving a customer, and that's an added value to something they didn't have before. And can't get anywhere else.

17:48Right. You shared some stats about those that do have Alexa Plus that are using it much more often. Yeah. What is it? Can you talk about how engagement? I have it written down somewhere. Yeah, you can get after it. Two times its usage, shopping three and a half times, five times more recipes. You do have it written down, yeah. Yeah, a ton more music, actually. What's happening is they're using the products, and right when you start realizing its agentic capabilities, go far beyond just answering questions. People engage it. Once you engage it once or twice, the usage spikes.

18:25And that's one of the challenges. You know, this is Ambient AI. It's working there. It's in the background. It's right there. How do you address it? When do you address it? How do you get something out of it? How do you learn what it's capable of? Those are all challenges. But once a customer engages it, it's pretty awesome to see. It just simplifies life. And we're starting to see that happen in the product. You have this belief that with a product, you really have to have this central thing, this one thing within a product that you go after. Yeah, that's true. How do you know that? Did we talk about that last time?

18:56You're right. You got your sources. I call it the one thing. One thing. Yeah. Is that AI does everything. So what is the one thing within AI? Well, every expert has their one thing it has to do great. But then you have a product that does – it's a catch-all. Yeah, it's a tricky one, right? Like it – I don't know if it's a catch-all, but it is – it's supposed to be the world's best personal assistant. Like that's the one thing it does at all times. Like is it going to do that? And if the expert that's delivering against it has to do its part, then you can step back and look at it. It's like anything.

19:28I've built plenty of products that were platform-level products in my past. And it's never as, you know, what I try and do with teams philosophically in product making is let's focus on the one thing that your team, these are big teams and big, you know, different levels within a product. But you can break that down and compartmentalize and say, all right, what's the one thing? What's the one thing? As long as the vision is clear on where you're headed and that one vision is clear, then teams can really rally on what they want to go accomplish for the customer. And that does keep you in the day one mentality because it's super clear.

20:01Like here's what we're going to deliver. By the way, that might mean there's tradeoffs in other places. And by the way, it may not be perfect for everybody. That's okay. That's always okay. Like if everybody was – if every single person was happy with what you built, I promise you you didn't push it far enough. I promise you. You just did not. But if everybody's pissed at you, you definitely pushed it too far. And so you have to find a little bit of that balance. But the truth is if you're going to push it to the right level of innovation and invention, and if that one thing, you just have to get that right.

20:33And then majority of your customers, they'll just be delighted. And then if you push it a little bit further, that's okay. Maybe not everybody comes on board, but teams tend to find a balance. I want to go back to this catch-all nature of AI and the difficulty in building it. So I'll just talk to you about the perspective from the outside, and it would be great to hear your perspective from the inside. From the outside, those of us who watch this space see great potential, I think. We've seen visions from Google, from Apple, from Amazon about this catch-all or, I don't know, this always there, contextually aware, helpful personal assistant.

21:14The thing that I've heard is difficult when you want to build something like that is when you're working with a large language model, they can only handle a certain amount of context before they just kind of lose their minds. And when you're trying to integrate calendar and photos and maybe ring camera data. There's so much. I can't tell you. These are just simple high level, but there's so much. Is there a technical wall that you eventually hit when you try to integrate that within an LLM? Well, there's not a – I don't – is there a technical wall? The way to think about it is if you're wrong, it's devastating.

21:53Right. And that's why. So your original question is why isn't it going faster? Well, for sure what I'm not going to do is when you decide to turn off a light, lock your door. That would be bad. Right. And so that's a microcosm of what you just asked. Right. I don't think there's a technical wall, but is there enough training and learning that you have to do and understanding does your architecture have to be set up in a way where it does know and rechecks and triple checks that it's calling the API but still hold the latency that customers come to expect? Because nobody's sitting there going, I think this is a hard problem.

22:31It's going to take a while. You have 30 seconds to get my light on. It's not going to happen. Once it's easier to go touch the light switch, you just go touch the light switch. And so it's inherently true of product making. We always, as our customers, you always have to think they're going to take the easiest path to get to their solution. Like that's it. Because we are efficient in our nature. Like we want to, other people call it lazy, but you could pick. But we want to get to the solution as fast as we possibly can as people on almost anything you do. Of course. Or maybe not fast. How about easiest?

23:04And then you start realizing that time is part of easy. like if you need it quick you want it quick even though it might be easier to say it versus touch it and so we look for that path at all times and so on our products i don't think there's a there's not a limitation at all as a matter of fact and i'm not going to talk about the architecture or the construct of it like that's what i won't do because i think what we're doing is spectacularly unique. And it is forward thinking for sure to be able to do it all. But I will say you have to be right. And that's where the limitations start. Because if you deliver the wrong outcome for the customer, that's when you start losing trust.

23:49And so during early access, rolling it out is a lot about that. Like, okay, if the customer is asking for it, then they're coming on board with us to be like, yeah, okay, maybe it takes a millisecond too long or your deep fryer doesn't work the way you thought it would with Alexa the way it did six months ago. And now you can tell us like my air fryer, it's not turning on when I ask it to. And we're like, ah, you have an air fryer. And you don't think that is something agentic, which it's not. But at the same time, because you have this LLM doing the work, because you have this orchestrator that's trying to call the right APIs and still kind of figure it out, maybe your air fryer wasn't part of the product in the redesign.

24:27And so now we know. And we just make sure we get it right. So there's not a limitation, but it takes time and you have to get it right. And so that's been part of the methodical process. And I don't, yeah. Am I right in thinking that that's the most difficult part in building this, just incorporating that context? Right. I mean, there's a couple of things that are challenging, but I think it's one of the hardest parts is first understanding the customer's context and what they're going for. and then executing on it properly. And they generally, once you understand, you can execute. But if you're wrong about the understanding and the LLM, you know, relative to and our agentic system, if you will, is the way I'll put it, we have a pretty sophisticated way of handling it.

25:11But remember this, you have to do it at speed. You know, customers have that. It's pretty unique. I'll give you an example. When you speak to something, you expect a certain speed. When you type something, you expect a different speed in a response. It's pretty fascinating. As a matter of fact, I call it, you know, our product has to be contextually aware. Like if you use the Alexa app right now and you speak to it, it's going to give you a shorter answer with less information. I think that's a good choice. Right. But if you take that same question and type it into the app, it's going to give you that full familiar AI response that you would want with the bars and the graphs and the context and the link.