Gemini in Chrome: Your agentic browsing assistant

0:00We serve a lot of users and we know that everyone has their own way of doing work. So I'm not going to tell you you should do anything. But one thing we also launched in Gemini and Chrome is history recall. And we actually know a lot of people keep their tabs around because they're afraid of closing them because they don't know how to get back to it. Try it. But yeah, Logan, you could be liberated of all these times. Yeah, yeah, yeah. Trust the history recall feature. You'll love it. I'm a clean tabs person now. I just trust search, you know, and I use Gemini all the time to find what I was left off with.

0:30So I love that. I feel like I'm a tab hoarder. I'm a tab hoarder.

0:44Hey everyone, welcome back to Release Notes. My name is Logan Kilpatrick. I'm on the Google DeepMind team. Today we're talking about Gemini and Chrome. We've got Rick and Parisa. You want to sort of give context for this moment that we're sitting in now where we just rolled out, I don't know if we describe it as the Gemini sidebar in Chrome or what the right vernacular is, but Gemini is in Chrome now. Yeah, we're super pumped. This week we launched another really big update of Gemini in Chrome. It works across Windows and Mac and Chrome OS and we brought a more integrated experience into it. And so you can open it with a control G or shortcut or a button to get a side panel experience that helps with multitasking.

1:26so you can have multiple conversations at once. We launched more integrations within the Google ecosystem and connected apps. So we're integrated with YouTube and Gmail and flights and other apps and then starting to roll out personal intelligence. And so you can have, you know, continuity of conversations that you have within Gemini app and any settings that you have in Gemini app help give you a more personal experience in Gemini and Chrome. And importantly, we're actually previewing something we're calling auto-browse. And it's an agentic experience where Gemini and Chrome can actually browse for you and do some of the tedious, mundane, digital laundry tasks of the web on your behalf.

2:14And that's something we previewed to our G1 and Ultra and Pro members. So big update. I love that. I'm a proud, we're talking off camera, I'm a proud Ultra customer. So I'm excited to always be trying to do something. You're on the cutting edge. I'm on the cutting edge. I have lots of questions and I'm excited to talk about Browse and sort of all the impacts that that has. I'm actually curious though, before we dive into some of the details, you mentioned end of last year, like the arc to get to the place where we are now, or like, there's like literally a button that says Gemini and hopefully we'll like bring Gemini to so many more folks.

2:46But yeah, I'm curious about that arc and like how we got to the place that we're in today. You know, for a long time, the desktop with AI has evolved as a web app, a direct web app that you'd go to. And then mobile sort of evolved in a slightly different direction where it was, you know, a full mobile experience with lots of applications and with the assistant integrated, usually through activation through a button press. And so they were like pretty different experiences. And we actually felt there was this really important opportunity to be able to bring some of the power that people have in using in-context assistance on a mobile device to the web applications they're using every day.

3:29And specifically, obviously, bringing that to Chrome was like a real key priority for us. And so the opportunity was to bring AI assistance like directly to where you were doing most of your daily workflow and therefore enable multitasking and a lot more powerful experiences. And so that's exactly what we've just built. But it's been a long journey. It took us a while to figure out where it should be the same as mobile and where it should be different. And we really are happy with how it's turned out. Yeah, that's awesome. I think there's something really interesting about um the unique fact that like chrome in a lot of cases actually like on somebody's device versus you look at a lot of other google surfaces and products and like they don't get that flexibility because it's a web app um and i'm curious like as you think about the gemini or more broadly the ai story for for chrome like unique and interesting things that that maybe not yet or in the future, like will unlock because you can actually download something onto a user's phone or laptop?

4:36I mean, we think of Chrome as a platform in and of itself. It really is an operating system on top of your operating system because we have users on Windows and Mac OS and Chrome OS, of course, our favorite desktop platform, Linux, Android, iOS, and a lot of people use Chrome across different devices. And so they're already signing in and are able to get their stuff, like whether it's their autofill data, their credit card passwords, their history. And it really helps you pick up where you left off across devices. And you can imagine with Gemini, we're bringing assistance and intelligence into the browser with your personal context.

5:16And so we can help on a lot of long-lived journeys. For example, shopping. Most people don't sit down and you know, buy a new car. Some people do. But for some, they're doing some research and comparison shopping. And, you know, they're starting on one device and then maybe sitting down and picking things up on their laptop or then leaving and picking things up on their phone. And so I think Gemini provides that assistance layer that can help with Chrome and your personal context and whatever the journey is you're doing for much longer sessions that can span devices. And I see that as the opportunity.

5:54It doesn't have to just be long lived devices. I use Gemini and Chrome for quick questions, whether it's like, okay, here's a really long lecture. I actually just don't want to sit through the full hour of it. Just give me the TLDR summary of the video and specific sections I can jump into. Or summarizing across a bunch of different tabs. People in Chrome struggle, particularly on desktop with like, I've got too many tabs. And whether you're trying to comparison shop or just synthesize across all those research papers and docs, you know, having assistance right in the browser can really help you do that so that you're not having to do as much context switching between different apps.

6:38This too much tabs is like, or framed another way in the like AI moment. It's like you actually have too much context. Like I feel this way in my job. It's like, it's not a, there's not a lack of context. It's like, how do I actually filter the context so that I don't see a million different things? I feel like the user journey for the last two years with a lot of AI products is like, you actually have, back to this context problem, you have context in one of these services, a browser tab, and then you actually have to take a fraction of the context, a part of the context, and leave where you are doing work.

7:12Hopefully, folks will feel the power of not actually having to do that. Totally. Being able to just have the... And a little bit in the manual flow, some of the people on the team who are working on the product, their kids will organize school work in tab groups. So it's like biology tab group. And you can see the power of being able to just ask Gemini, okay, come up with a quiz to help me prepare for my biology tab group. And I think we need to do even more proactive curation so that you don't have to rely on a tab group. But there is so much context in the browser. and if you can have that assistance within it, you're not having to like download and upload and copy and paste into some other app and that just removes some of the friction from doing work on the web.

7:53Yeah, I mean one of the things that we've implemented which I think is great is the integration of Nano Banana. So like you can be on a web page and maybe you're trying to do some home decoration or something and you could ask Gemini to change the color of the carpet or the sofa and then you can see it like as it would appear on the page. And so it's like totally cool these kinds of use cases that you would have to, as Parisa was saying, download the image, upload it to a different web app in a different tab. It's just made super easy with this integration. The file formats aren't compatible. Jim and I want PNG, it's a web image or whatever.

8:28It's great for YouTube, for reading a PDF, for summarizing a super, super long email. It just makes everything a lot more efficient and powerful. Yeah. This tab groups piece, I'm actually curious if there's any suggestion for users. I am, as somebody who chronically has too many tabs open, is that like, as I think about Gemini and Chrome, like, should I be doing that at least right now, like that like manual context engineering via tab groups as like, you know, for my internal project? And that way I can sort of be more precise when I'm talking to Gemini and Chrome about like not making it try to, you know, deduce the context correctly.

9:08Is that like actually a pro tip we should give people? So we serve a lot of users and we know that everyone has their own way of doing work. So I'm not going to tell you you should do anything. That is something you can explore. But one thing we also launched in Gemini and Chrome is history recall. And we actually know a lot of people keep their tabs around because they're afraid of closing them because they don't know how to get back to it. It's loss aversion. In Gemini and Chrome, we made it easier to ask for what you know you had open and actually get it back. So it's like if you knew that you were searching for, I don't know, three Italian restaurants last week and you close those tabs, you can ask Gemini and Chrome over those restaurants that I was searching for last week and get the answer, which hopefully makes people a little bit more comfortable to close some tabs.

9:51You can manually create tab groups as well. Sometimes people do them in Windows. We try to make Chrome, again, it's an operating system, flexible to really work for a lot of different work styles. So try it. But yeah, Logan, you could be liberated of all these tasks. Yeah. Just trust the history recall feature. You'll love it. And so and so it will like in my I'm I'm like a docs, you know, DAU, all that stuff. Is that a good example of like, you know, I close all the docs tabs and then I'm like, tell me the projects my team is working on across, you know, some specific category of stuff. will it be able to get like, is it like caching like the last time I looked at it?

10:34Or like, would it actually know like, oh, and given like docs is maybe a tricky example, because like, it's live updating and like their context is actually changing from a week ago to today. Which version is it? Would it actually be looking at? I think it would just summarize what you've looked at recently and then provide links like directly to the document. Got it. You could probably use that as a great jumping off point to get back to any place where you you left off i like that um do you two use tab groups i do sometimes for specific projects or um like we're doing annual reviews now so like i'll i'll use them yeah not exactly but yeah um but um yeah i'll I'll use them for one-off and I also don't use them.

11:25I'm not as principled. We have people on my team who are very principled about how they use software and tools. I kind of go in and out of it. And I think what I like with Gemini and Chrome, you can add specific tabs in context for a one-off query. It would be too heavy weight to actually create a tab group, at least for me, for that specific use case. But we're trying to really figure out what is this journey? What are the tabs that are the relevant context for you to ask about? And then you can just, you know, get rid of the group. But I don't know. Do you use them? No, never. Yeah. So interesting.

11:58I'm a clean tabs person now. I just trust search, you know, and I use Gemini all the time to find what I was left off with. So I love that. I feel like I feel like a tab hoarder. I'm not a tab hoarder. I feel like there's just there's this goes back to the context problem. I feel like the context velocity, I feel like is another thing. where there's just so much stuff happening all the time. And it's like, I can't stay on top of it. And then I think the thing, actually, the thing that forces me, and there's maybe some interesting Chrome feature in this, which is what forces me to close tabs is when I switch off of an external monitor onto a laptop, it doesn't break, but we stop being able to render all of the tabs because there's not enough space on the screen anymore when I unplug.

12:44And so that forces me to close tabs because I can no longer, like it makes it harder to use Chrome when I can't actually like click and see what all the tabs are. So it is like a manual cleanup process that forces me. See, there's almost no difference between having infinite tabs and having zero. That is. Because you can't tell which is which. That is a good point. On this, I've had a bunch of conversations with Robbie from the search team about sort of this expansionary effect that AI is having for search and like how people are using their product has changed. And I'm curious how people will interact with Chrome differently because of all these AI features and advancements relative than historically, like people's notion of like what a browser was.

13:26I think this totally changes what a browser is. You know, it's the big evolution in desktop computing has been that you spend most of your time in front of a browser, most apps are web apps. And so you're using this for workflow, for communication, for kind of everything day to day. But what bringing Gemini into this means is that now you can start to, especially with all the new Agenda capabilities, like you're starting to be able to automate some workflows. You're starting to be able to take care of things like asynchronously. I, you know, one of the things a lot of people did over the holidays was someone planned the Thanksgiving dinner with their family.

14:01I got a dock with my kids Christmas wish list and I used Gemini to buy everything on it. It was awesome. and the auto-browse capability or agenda capability integrated into there turns the browser a bit into like an automated workflow system. And I think this is like only scratching the surface of it, but I think this is a real beginning of a huge change in how people use computers. Yeah, I totally agree with that. And I think I can feel myself as I'm dogfooding and as we're shipping and improving my own expectations and norms changing of what I can get done. And I think, you know, you can command the browser to do something as, this is going to sound horrible, but like I'd ask my husband to help me with the task.

14:46And now I can ask Gemini and Chrome, like, hey, do this task. Do this task for me. I need a white elephant gift. That's what I used. I used it for that too. In some ways, the browser is the original user agent. It really is moving from, I think, this passive window that just renders content to being an agent on your behalf and being able to operate on the web and handle these complex tasks that typically require lots of manual work and actually can make progress, you know, with you overseeing and making the fun decisions, maybe of picking something and making the sensitive decisions of like, okay, finally taking the action on the purchase, which is really cool.

15:24Yeah, I mean, it's fascinating to see it evolve. Like we've had this assistance capability in phones for a long time, but there's something about the desktop experience where it's just richer and you can iterate more with it. And it really is now becoming more like an assistant, like what you would expect, like an assistant that you might work with to help you with. And it's so powerful for that. And it's really just, we're only starting with it. So I think there's so much potential in it. It would actually be great if we can see some demos of Gemini and Chrome. So maybe we can kick off with a couple of them.

15:57Yeah. Okay, so you said you just bought a new house. Yes. I'm assuming you want to personalize it. And I have horrible taste. You have horrible taste. Well, I have horrible instincts. I want, as when I see great taste. Well, let's just see what those instincts are. So like part of it is we wanted to build Nano Banana into Chrome so that you can really transform images on the fly in the context of things. This is a very simple example. I don't know. pick a color that you want to imagine this room with. Purple wall, like what what color walls? Let's use like dark blue walls. Dark blue walls. Can we get a little more, it feels these chairs don't look super comfy to me.

16:35And oh more comfortable chairs. Because I play board, my main CUJ at the kitchen table is board games with my girlfriend. And so we're sitting there for an extended period, so I need comfy chairs. Okay, that totally makes sense. And so we're using the uh nano banana model obviously and um we got dark blue walls there's some cushions on those chairs that's better it's a super simple example it's a much better room it is much better it is better um very profesh whose house is this this is some random person this is a random picture it is redfin um dunking on this board for someone's watching saying my kitchen was fine we love your It's cool.

17:18It's going to sell. All right. So that's like just one example of Nano Banana. I'll show auto browse and let's say I don't know if this makes sense, but you want to do a housewarming party. That's pirate themed. Let's do it. Why not? Let's do it. I'm intrigued. And one of the cool things about Gemini is obviously multimodal. And so I'm going to do an example of I don't know. Are you a big spender for parties? Oh, we're spending big on this party. Yeah. Okay. Can we come up with some really high quality pirate keepsakes that also just like aren't annoying? I don't know. I don't know if that's too specific, but like I don't want to bring home a pirate sword, but like, is there any pirate themed interesting thing that would actually be useful in my life?

18:04I could give people to sort of remember this party. Okay. And come up with some pirate keepsakes for people to take home. All right. So what you're going to see here is, first of all, we're triggering when auto-browse is enabled. And one of the challenges that we have is when a user, you know, asks Gemini something, do they want just a direct response or do they actually want you to do this browser task? And so the prompt I did was like, go to Etsy, find some supplies to recreate the decorations in this party photo. So it can tell that it's something to do with nautical and pirates. Apparently Y2K parties are really popular right now.

18:47So that was another example that one of our dog fooders had. And now it's going to Etsy. And you can see that in the background, you can see that it's actually navigating on Etsy. And so it sounds like you have opinions about specific things that people want. But right now it's searching for pirate swords, cutlasses. There's a range that you can find. And I've got opinions like no metal swords. It's going to plastic swords. So that's probably both within budget as well as within safety. Reduce my liability. Exactly. And you can see over here, you can take over the task at any point. And on the right, you can see what the model is actually doing.

19:28And we don't expect that people are going to watch this. Like the whole point is for us to take on some of those tasks that you don't want to do. And so we expect that people will do this on the background and then you can go about, you know, other tasks that you want to actually work on and you can run multiple tasks at once. So that's an example of auto browse. In this example, really quick, actually, did you direct it or you just like had Etsy open already and the assumption was or did the model say like, oh, I'm going to go use Etsy to like try to find some of these pirate items? I told it to go to Etsy, but you could also just, you know, to make it go a little bit faster because I wanted it to go to Etsy.

20:04Yeah. But you could, you know, ask it to generically buy something and then it will use the search tool to find a place that it can actually buy something. I bought Stanford basketball tickets the other day and it decided to go to SeatGeek and find seats to what I asked for. And it was great. Totally worked. That's awesome. Nice. All right. I can show just like opening the side panel in general. because I think that really being able to ask questions with a site or a YouTube video is what we see people using it the most for, just this really quick control G open. Demis said this is his favorite use case.

20:43He just asks questions about YouTube videos. Exactly, exactly. I'm like... And you can have, you know, Rick was talking about this, the floaty. Some people still do like this, especially if they have a big monitor and they kind of want to see the agent task off in the side. but we're seeing people really excited about the side panel. And so I think asking a more specific prompt about which hike I might wanna go on in San Francisco is one example. One thing we're trying to do is also give suggestions of things that are relevant to the context. You'd mentioned, this is really new tech. How do you get people to even understand what this can do?

21:20And one of the most common queries we're getting in Gemini and Chrome is what can it do? And so a piece of it is like making sure those chips or pillows have relevant things. And so you can, you know, describe the Dipsy Trail route. It's actually an awesome trail if you're in the Bay Area. And ask other questions about, you know, the context of this page so that you don't have to switch over to Gemini. That's awesome. And it gives you an awesome breakdown of the Dipsy Trail and all links to how to get there and what to look for. It's really, I mean, this is like super useful. You're totally in line with what you're asking.

21:53Yeah. And if you're like, cool, I want to do this hike, this Saturday, trail hike to my calendar. Yeah, that's exactly what I was saying. And like, can I take action then based on the context, which is awesome. Yeah. And we want to make sure that you can see that it creates a hike, can add it. It's super fast. I would say the Gemini 3 was a huge step function in what we've been able to do, the work to really make sure we're integrated with other Google apps. And then the fast model, particularly for auto browse, was a big step function in both like speed, of course, but also quality. And I think that made us feel confident, like now we can bring more of these capabilities to users.

22:41Yeah, that's a great, that's awesome. Well, actually, I have another really quick question that's shown from these demos. Oh, I was going to do one other one. Yeah, please, please, please. Let me do one other one. So you can do a new chat. And one thing you could do, you just moved in, you want to know what top-rated plumbers are locally. Hopefully I won't need plumbers. Hopefully you won't need them, but you want to be proactive. You need your people, you know, proactively. So you can draft a message and ask if they do free estimates when and when they're available. And, you know, this is another example of, okay, it knows we're looking at San Francisco now.

23:18Of course, you can put that in. Nice. And actually, a quick question on this. What is the suite of tools that are available by default in context and then browser actuation? It has connected apps and integrations to workspace, YouTube, shopping, flights, and we're working on building a lot more. I love this. And actually, one of my questions is going to be back through this thread while it's actuating and sort of searching. the sort of agent browser human interaction cycle obviously the internet has captchas to like actually stop in some cases like agentic systems from engaging on websites and using them the way that humans do and I'm actually curious for Chrome specifically and with with auto browse like what is auto browsers like if it hits a captcha is it going to prompt a human in the loop to go and actuate and click the button?

24:11Or how does that sort of cycle into some of these guardrails of the existing internet? I think figuring out that balance of what should the agent do on your behalf or what could you instruct it to do on your behalf that you would want it to do, and when does it make sense to have the human in the loop and actually take action is something we're continuing to think really deeply about. And right now, we're pretty conservative. We want the user, the human to be engaged when it's like posting to social media or actually in that final purchase confirmation step it will click through some pop-ups because it is eager to help you and get the job done but we're more on the conservative side too and often asking the user like hey how do you want me to do this we support if you use google password manager we'll support logging in on your behalf but also we'll ask the user hey do you want me to log in on your behalf and do you only want me to do it this time or do you want me to do it every time I go to this site?

25:08And so figuring out that balance and probably being overly conservative because we want people to trust the technology. I love it. This is a great example of just like so many, like it's really doing the deep research of going and finding all these examples. So hopefully the research will be for nothing because I will never need plumbers. Yes, I hope for that too. You have some recommendations here if you need it. We've also got some Gmail drafts in there if you actually want to send them. So if you went over to Gmail, you can email EJ Plumbing and Mountain View Plumbing. I love it. I think it's not 100 % perfect.

25:47You probably noticed in this example, but it did what she asked, like find plumbers, do draft emails. Pretty amazing. No, this is awesome. I feel like I'm going to spend the weekend sort of playing around with a bunch of these examples. So, yeah, I love it. All these demos were great. Thank you. Yeah. What do you both think the sort of like trajectory? And like I feel like I'm like as we're having this conversation, my head is spinning because it's like very clear that this is going to be the direction that things end up. So I'm glad we're having this conversation. It's what makes these fun. But like, I think there's, and I'm curious about the tension for like, obviously Chrome has like billions of users across all over the world.

26:26And like, they're, you know, one of the unique challenges I think that we have at Google is like, you know, oftentimes, you know, maybe Chrome, the Gemini button in Chrome is like the first time a user experiences something like this. and that education usability gap. There's all this pressure on like frontier AI use cases, but also like the average person has no idea how any of this works or like how to use it. And so I'm curious, like as we're rolling this stuff out, like how y 'all are thinking about meeting people who have like a very different level of understanding and also in some cases, actually even enthusiasm for some of these products, which is also interesting.