Sameer Samat on Android 17 and the Future of Intelligent Computing

0:00I just want to get rid of forms. Like I never want to fill out a form again. I just booked a bunch of flights for vacation over the summer and I now have my 15 year old's passport number memorized, but I don't want that to be memorized. You know, like I don't have my own passport yet. I've entered it so many times, you know, it's like I'm losing my mind here. So Superfill is obviously we all become accustomed to autofill working on your name, your address, maybe your credit card information. Superfill is that, but extended to many other fields. That's awesome.

0:35Hey, everyone. How's it going? Welcome back to Release Notes. My name is Logan Kilpatrick on the Google DeepMind team. Today, I'm joined by Samir, who leads the Android team. We're talking about all of the new stuff coming to the Android ecosystem from the Android show from I.O. and now with Android 17. So Samir, thanks for being here and sitting down. Thanks for having me. We were just chatting off camera on it. this sort of the spark hit me, which was sort of this really difficult job that you and the Android team have of like bringing all of this technology to literally like billions and billions of different people.

1:07And you mentioned to me that as part of the Android show, you didn't say the word AI once, which I think is awesome. As an ecosystem, it feels really like so focused on the technology right now and not enough on like actually what it does for real people who are using the devices. And I feel like you probably have like a really great perspective on this. So I'd love to sort of start there and then we can talk about all the stuff that y 'all launched. Yeah. Well, so for the Android show, it's all know your audience, right? And, you know, our audience is a lot of people who are using devices and they're excited about technology.

1:38They're excited about the product that they're using. They want it to get better. But they're not necessarily the folks who are super, super deep on AI. Some are, but some are not because it's a much wider audience. And I think that there's just something going on with the word AI, you know. I think there's a reaction to it. It's scary for some people. It means different things to different people. I think it's kind of become a little overloaded. Obviously, inside Google and as technologists, we all know that this is a huge moment and it's a platform shift and it's totally changing everything and we're all embracing it.

2:09But then when you talk about it from a consumer standpoint, I think it's really important, we've learned, to not talk about the tech as the lead-in, but rather to talk about the benefit. you know? And, and so I'll just give you one example way we learned this. We launched a feature a couple of years ago on Android called circle to search. And it's a really simple concept, right? You hold down the home button on your Android phone screen lights up. You can then circle any object on the screen. It could be, you know, like the tie that someone's wearing that you're really interested in, or it could be like the, the beautiful lake that's in this photo that you've got on some social media posts.

2:43And then you can ask a question about it. Like, where do I buy that? Or how do I get there? Or what is that thing? You know? And this was cool because like, It was really hard to describe these things to Google before. How do I describe that tie? What would I even say? Same with the lake. It's like if I could describe it, I'd probably know already what it was. You know what I mean? So this is a really simple capability, but that feature just took off. It took off. And there's so much AI behind that. There's so much machine vision behind it. There's so many things that make it go. But we actually didn't mention any of those things.

3:12We just talked about showing what you could do with it. And people loved it. And then we've done other features where we've led with the AI. and people are kind of like, oh, I don't know about your AI. I'm not feeling that as much. So we didn't say the word AI once in the show. A lot of the features are definitely AI powered. And I think the reception was better that way because people are like, okay, well, you're actually telling me why this is good and helpful in my life. Yeah. One of the biggest points of tension, it feels like for this balance of how much AI to lead with versus not is actually to the point that you made, which is like, we're in this platform shift.

3:42And I feel like people are like, who should I be betting on in this? My perspective is it's very clear that Android is like at the absolute frontier of trying to push this rock up the hill from like a new being at the forefront of that platform ship. But I'm actually curious, how has that manifested for Android? Maybe we can use sort of like all the announcements that just landed as a version of the story. I mean, we think it's really important technology to help build better experiences for the user, you know? And I think that's like fundamentally what drives us. And, you know, I mean, it's just wild what these models can do, you know?

4:15It's like really incredible. And there are things that we've wanted to do for a long time. A lot of times I would see mock-ups of some new feature or concept from a product manager who hadn't really maybe been around a long time and be like, okay, that's not going to work. But now I'm like, bring all those back because I actually think the tech is ready for that. I mean, one example that was pretty straightforward, Android is really very popular in a number of countries where the rate of scam and spam is really high. You get a lot of phone calls, you get a lot of text messages that are trying to get you to part with information that you really shouldn't be.

4:52We built this model working with GDM, on-device model that actually, with your permission, can listen to a call from an unknown number. And we worked with a bunch of banks, for example, to make sure the model understood what a bank would never ask you for. A bank will never ask you for certain things. and they actually have like consortiums and associations where they work together to like have these rules you know we won't do this and so it's like if the model hears it like someone's like well give me this you know you know on the phone it pops up and vibrates the phone it's like this is a scam wow yeah and you should hang up right now right and that's like pretty useful you know and so that's something that we think impacts so many people in a positive way and we feel that that's like the promise of this technology is like you can do so much more with it.

5:41So it's a really big part of our strategy. But I think the way we need to describe it to people is like, here's what it does for you. Yeah, I love that. Demis, every time we're in one of these meetings talking about these big moments, he wants to land this point of technology that works for you. That's true at the same time that we also as Google want to tell this, like we're at the lead and at the frontier. And I feel like there's just there's a lot of tension in those things. But something interesting about what you just described is the models actually running on device. We were chatting really quick before we were on camera and the thread of it was like, you know, the open claw moment happened, Gemma 4 happened, and actually the AI Edge gallery app, people were downloading this and playing around with that.

6:18So many people all of a sudden started to grok the on-device models are really going to be powerful and capable and like enable new experiences. And I'm curious, I feel like Android has this like really interesting tension because you have billions of devices. Obviously you need high-end new devices in order to run the models because is it just requires more performant hardware. And then there's like this whole group of people who don't have the new high-end devices who would really benefit from AI. And sort of from a platform perspective, I'm curious like how you're balancing and like what the open model or on-device model sort of like strategy looks like.

6:51Yeah, I think the on-device model strategy is, it's evolving right now because I do agree with you that the on-device models are getting to a point now where they're seriously impressive. I think we are on the cusp of something pretty amazing there. From an Android perspective, our mission is to bring computing, the best of computing to everyone in the world. That's high-end devices. That's the most basic device that you can buy. It's really important to us to bring the best of computing to everybody. That's like really what gets us up in the morning. Obviously, there's like kind of two forms of innovation motion that we go through.

7:21One is at the high end, let's create the future. And you do that at the high end with flagship devices because they have the more system resources. And so before you know how to optimize that, you are just using all the system resources you can and you're trying to push the boundary of what's possible. And so we're definitely doing that. But one of the motions that we've also had over the years is how do you innovate? It's a different kind of innovation where you take those capabilities and then you flow them down. And that flowing down typically requires like a different approach. You know, you are now looking for how can we take the kernel of what we've got here at the flagship level and the features that are most profound and most interesting and then find ways of making those features run on devices which are less capable.

8:07Sometimes that involves using the server, you know, obviously, but sometimes that involves creating different specialized on-device models, you know, and also working with silicon providers to make sure that those can actually run efficiently, thermals, battery constraints, et cetera. But we've done that effectively over the years, and I'm excited about that. I think right now it's a little bit sinusoidal. Like right now we're focused on the flagship side of it because there's so much innovation that happens. And as soon as we get the first set of features there on device, we already have a number of them, like I mentioned, the scam detection one, that we think are really working for people.

8:41Now it's about like, how do you flow that down over time? Yeah. And how much do you think that sort of symbiosis between the new models and sort of the capabilities that are unlocked, like means that like a macro perspective for Android? Is it like, you know, the experience we see today, five years from now, when, you know, the models are really good and work great on device for like pretty much the hardware that everyone has, it'll be a totally different experience. And like, is that sort of the high end model sort of like correlating or the high end devices sort of correlating to like an even more re-imagined experience relative to the later generation devices?

9:17It's a really good question. I mean, if I just take like a half step back from that question, just to put it in context, like the way I think about what we're doing right now, what we've been doing in computing for like the last 40 plus years is we have created systems that put the user in the role of the micromanager of the system. The best we can do right now is like, first, we have a command line interface. It's like, what would you like to do? You know? And then it's like, now you have a GUI where you have menus and pointing devices and what have you, but it's still you as the user conceptualizing what is my goal, taking that goal and breaking it down into like all the discrete actions you want to take.

9:57And then you literally are clicking on sub actions as you go. So that's like the micromanager model that we have for computing. It is still kind of intimidating to do things, you know, like people take classes still to use all this software and what have you, you know, I think that what we have an opportunity to go from is operating systems to intelligence systems, which is kind of how we talk about the evolution over the next and years of Android where an intelligence system is one that you articulate your goals. And the system's job is, of course, managing resources, but also managing context and figuring out how to take your goals, the context that's around those, and produce the actions that you want.

10:39And of course, we want to do that starting with a number of these flagship devices. We'll get it right there. Over time, I do think the interface is going to change. I do think the interface will be more intent to action oriented as opposed to micromanagement. Yeah, I have so many questions to follow up on this. Maybe to start, how do you think this changes maybe in the more short term? We've seen like voice take off as like a new interface. How much do you think that manifests itself in some of these new experiences? Yeah, I think a lot of people want to go directly to agents with when I talk about intent to action.

11:12And I think agents is super important and we should come back and talk about that. But I'm glad you brought up voice because one of the fundamental parts of an operating system is the input method. Like how does the user communicate with the system what they want to have done? And a big part of that is the keyboard. You know, either it's on your phone or a physical keyboard. Hasn't changed that much in a long time, you know. The last thing we really added to the keyboard in a big way, I think that made a huge difference. I mean, we keep making it better, right? But the last big, big thing was like autocomplete.

11:41Yeah. And I'm fighting autocomplete on a daily basis. I mean, autocomplete could be a lot better, honestly, but autocomplete is like 2012. If I'm not, if I'm memory serves, 2013, 2014, we added some, you know, swipe is another thing too, but fundamentally the keyboard hasn't changed a ton. Now the keyboard for me is the perfect example of a micromanagement input system. You literally think about what you want to say. Like what's the goal of what you're trying to say? You break it down into, okay, here are the words I want to use. And then you literally peck those letters of those words onto, I mean, it's what we're just used to, right?

12:16We don't even think about it. We just do it. But so one of the things we announced recently was this feature that we're going to roll out this summer called Rambler. And Rambler is fundamentally different in that the microphone, when you tap the microphone, you just ramble like I've been doing, I guess, here. and and instead of just faithfully transcribing exactly in a dictation form what you were verbally saying it takes everything you've said and then distills it down to like what you really wanted to say lets you look at that and you're like yeah that's right and then and then you go there's a reason why like i think for a long time humans have figured out that like hey just give me a brain dump of your ideas is like a productive thing to do you know because voice and you're kind of verbalizing it and all that, and it's like, another person might be like, oh yeah, that's interesting, you know?

13:07This is kind of like the equivalent of that. Can I show it to you? Yeah, let's do it. Okay, okay, here we go. So here I am in Google Messages, and let's say I wanna text you here. So I've got the microphone, which is in Google Keyboard, which by the way, is used like hundreds of millions of times a day to do like voice dictation, right? So okay, so here's Rambler on the mic. Hey there. um if you're going to the store can you pick up some fruit maybe like some bananas mangoes strawberries and oranges actually actually never mind the bananas we don't need any more of those we have plenty of those i love that ui too okay so what you can see there is like none of the ums and ahs are there right and then you know i went through this whole thing about bananas right and it's just like, okay, I get it.

14:00You don't want the bananas. Can you make it into a list?

14:08Add some emoji.

14:12Wow, that's fast. And is that all running on device? No, so right now this is actually talking to the server. Really? Yeah, so we've made it super fast. And what's also nice about it is we didn't want it to make it feel like it was writing in kind of a style that people think is created by a chat bot, you know? You know, that's why it didn't make it a list in the beginning. You know, we've really tuned it for like more conversational tone. So I had to tell it to make it a list. The emoji obviously is like gratuitous, but I was like, let's go for it. So that's Rambler. And you can, you of course can use this just to send text.

14:49Here, I'm going to send that to you there. You can, you of course can use this to send simple text and stuff, but you could also use it to like dictate a paper or something that you want to write. And it works equally well for those kinds of things. So that's a different kind of keyboard. And this will be a mode that you can use the mic in. Of course, you can still type on the keyboard if you're old school. But we just think this is a big upgrade for the keyboard. And there's a lot of what we mean by an intelligent system. Now the keyboard is actually really intelligent. Input is the first step of the story of the sort of transformation that's taking place.

15:24The second step is actually actions being taken on the device. And so I think one of the announcements that also came was Android is actually like going and clicking things in the phone so that I'm articulating in voice. And then Android is actually doing the micromanaging for me to use your analogy. Yeah, that's right. So we call this app automation. And it's basically exactly what it sounds like, which is you tell Gemini that you want to get something done. Actually, really quick. I think this would have been the most quintessential point to use AI. and to just say app automation and not say AI is like actually like it hits on the point the commitment to like not sort of like sell the AI bit of it but like sell the outcome which is like automating apps that are fine.

16:02That's exactly the idea. A computing system going through and tapping on things and completing actions when we show it to people they're both wowed and also a little bit nervous at the same time and so you know we wanted to make it as approachable as possible. So a few things that we did. One is we built a virtual window system in Android. So what happens is that when you ask Gemini to do something like this, let me start by grounding it in an example. The example I love because it has some complexity to it is I have a list of people who are coming to my barbecue up on the screen. And each person next to it in this list has how many people are in their party and whether they're a vegetarian or not.

16:42And then if I long press on the power button, screen context comes up for Gemini. And so it can see the doc with me now. And I say, hey, can you go to Instacart and put what I'll need for the barbecue for all these people in my cart in Safeway? What it then does is we have a virtual window system in Android. So once Gemini starts automation, it will say, okay, I'm going to start this in a virtual window, which means it's a containerized environment where the app that you've designated and Gemini can now exist in a window, which you can minimize into your status quo. And so you can go back to that window anytime.

17:24The AI agent can't go outside of that container, meaning it can't use other apps. You can watch it. Sometimes it's fun to watch. It just takes the screen. When I was watching it with this example, one of the things is it was navigating through Safeway and figured out the thing that all of us already know which is like for whatever reason the num the hot dog buns package and the hot dogs are never the same number right so it's like hot dog conspiracy it's a big hot dog conspiracy right and so it's like well wait a minute like and it's thinking is at the bottom as it's doing this it's like wait a minute the hot dog buns don't match with the it's like i think i'm gonna need more buns than i need hot dogs well what how should i resolve this and it's like kind of fascinating to watch it do it but it added all the stuff it figured out they need veggie burgers and do the thing out of the cart and then it comes and gives you a notification and pings you and says hey your cart's ready take a look at it we've made it stop before it completes transactions um because right now we feel like it's really important to build trust with the consumer and to have that safety around it so it brings you back you look at the cart you're the one who pushes the the commit button but it saves you some time and i think that's like what it's the proposition is all about yeah that's so cool i did this concept of like putting apps in a container and like actually like running them on your phone did that exist before this because it's such an interesting like concept of of like android having that as like a feature available to it yeah i mean android has um a definitely like an app container and an isolation layer per app but this notion of something else driving the screen also being limited to that container and that virtual window that you can minimize this is different we haven't really had the need for it yet, you know, because these are single user systems, you know, and you're kind of like, okay, I'm on this device and I'm doing my thing.

19:13Why would I need a headless thing going on in the background, which is actually generating UI? You know, like I wouldn't necessarily need that, but the model does need that because it needs to look and see what's going on on your screen. The way I see this evolving over the longterm is, you know, there's MCP integrations, there's Chrome, which can also do some form of auto-browse and manage things and get things done that way. And then there's also this approach. I see all of those kind of coming together under the agent. You sort of tell Gemini, hey, I want to get this done. And it figures out the best.

19:46These are all become tools for that. And I think that's the way it'll evolve. In this example, is it vision based? Is it like actually, so it's like using Gemini and then like choosing the like pixel coordinates, if you will, to like click. It like looks at the screen and it's like using the app just like you. It's like, I think I should open up that menu. It opens up that menu. You see it like tap on the search bar and like start searching for the hot dog buns. And we wanted that to be like handing your phone to a trusted assistant and saying like, hey, could you, I got to get this other thing done.

20:16Can you get this done for me? That's kind of the way I think about the user model around it. No, I love that. And part of that story actually to make this possible as you were describing that use case is, is Gemini intelligence at the beginning as sort of like the way to, I think like kick off actually this workflow. For folks who missed the Gemini intelligence headline, do you want to sort of talk through that as well? Yeah. You know, we wanted a way to describe the best Gemini experience on the most premium Android devices. And we call that Gemini intelligence. We just announced that at the Android show a couple of weeks ago.

20:51And it's a set of capabilities that are really at the bleeding edge that help people with all these kinds of things. You know, we talked about Rambler, We talked about automation. Another one that I'm really excited about is something we're calling Superfill. I just want to get rid of forms. Like I never want to fill out a form again. I just booked a bunch of flights for vacation over the summer. And I now have my 15-year-old's passport number memorized, but I don't want that to be memorized. You know, like I don't have my own passport. Yeah, well, I entered it so many times, you know, it's like I'm losing my mind here.

21:24So Superfill is obviously we all become accustomed to autofill working on your name, your address, maybe your credit card information. Superfill is that but extended to many other fields. That's awesome. Yeah. And the idea is that if you've given Gemini your permission, that's an important step. Then if you've uploaded your passport to Google Photos, if you, for example, have an email with your confirmation code in it, if you have, you know, these are all things that exist in your Google account. if you will, it can pull those pieces of information and suggest them to you in real time when you're filling out that form.

22:01That's great. So that's like another example of one of the capabilities that we've got here. The final one that I think is really, really cool and kind of speaks to how UI may evolve in the future is generative widgets. And so the idea behind a generative widget is, you know, widgets are this like ultimate form of customization. A lot of people have them all over their home screen and in Android, it's been a long-lived way to customize. You know, it's like, but someone has to build that widget for you. Yeah. Our concept here is what if you could just say to Gemini, like, I want to see what the latest lap times are for this, for the F1 race that was yesterday.

22:35And I want you to keep that updated, you know, build me that widget. Right. And no one's maybe built that one. And it'll go off and crank for a while. And then basically give you a widget. That's exactly that pulling from publicly available sources of information. And I think it's the beginning of generative UI for your operating system. You know, I think that evolves over time where more of the UI is built for you on the fly. I feel like we have this, I don't even know who has this saying anywhere, but Android is the platform for builders. All the things you've mentioned so far feel like they're delivering on this.

23:09That Superfill example, what's actually happening is the model sort of saying, hey, there's an open form field. What might go here? and then it tries to like retrieve a bunch of stuff? Or is it like there's a specific set of categories like passport or travel information as one of those and it just pulls that into your personal context or something like that? Initially, for the sake of making sure that we get the precision and recall trade-off right and that we also make sure that users become comfortable with this and feel like it's something they can rely on, is we've limited it to a set of different types of entities, passport information, driver's licenses.

23:45one of the ones I love is your car's license plate like if you have that DMV registration in your email where it gives you search my pictures every time and so like let's you know the good example of this is like let's say you're you're parking somewhere and the parking app that you know this the QR code that you scan in the parking lot is like enter your car's license plate and you're already walking out the parking lot you're like oh man you know and so you're either opening up photos or gmail or whatever you're searching for it now it just like detects that and like, we'll tell you to fill it.