Building a frontier AI search experience

0:00Google Search is going through this transition into being a frontier AI product. It's pretty incredible what we're seeing. It can help make plans, it can think. It's the largest scale distribution of Gemini across Google, probably of any AI product. So you can basically get to the frontier model, but deployed at search scale fast for 1.5 billion people. That's crazy. To me, that's pretty mind-blowing that it can do all of this. What's the impact long-term of this search, doing all this stuff, but also just the way that people look for information? I mean, I think that ultimately, it's going to make humans more productive, efficient, and informed.

0:35The mission of Google and search has never felt more current.

1:07Hey, folks. Welcome back to Release Notes. My name is Logan Kilpatrick. I'm on the Google DeepMind team. Today, we're joined by Robbie Stein, who's the VP of Product for Google Search. Robbie, I'm excited for this conversation. Thank you for taking the time to sit down. Thanks for having me. I'm excited, too. So, Robbie, let's dive in. I think you and I had a conversation the other day, and you said something that's actually stuck with me. And the comment that you made, and I'd love your reaction to it again, and then also for us to expand on this conversation is that Google search is going through this transition.

1:35And I think I've seen this as a user, but it's going through this transition to being a frontier AI product. And I love that. I love that vision. I think it's super compelling and exciting, but can you sort of elaborate on what that actually means in practice? And I think people have a lot of preconceived notions about what Google search actually is. Yeah. Well, first the goal is for billions of people around the world to truly ask anything of search and to get great high quality information and access to the world in the web. And so for us, there's a couple of components there. You know, one is a model that truly understands the web and understands all of Google's information.

2:09So it can be the most helpful and highest quality. The second piece is a set of experiences that make it easy to understand that information. And the third is operating really at a unique scale that's fairly unprecedented. And so right now, people can ask a pretty natural question in search. You could say, how do I remove a ketchup stain from my jeans? Not saying that that happened to me, although maybe it did, without ruining my jeans. Put all that into Google search and an AI overview, it's highly likely to show up with helpful content and links to dig deeper. And the DAU of AI overviews, which also has the largest scale distribution of Gemini across Google, probably of any AI product.

2:47So Google search is the largest AI product in the world. That's crazy. And there's about 1.5 billion users everywhere using Google search and our AI experiences now per month. So that includes AI overviews, includes multimodal experiences where you take a picture or use voice and then see an AI response. I could take a picture actually of some books in this room and say, if I'm a fan of these books, what else would you recommend? And Google Lens, with a visual recognition that it has, can parse the books, call upon an AI to help research and get more information and then present back to you information that could be really helpful for recommending other books like what you showed.

3:24Those are completely new ways that people are using search, and they're driving a lot of the growth of search as well. And what we've learned is that there are frontier models increasingly powering those and leading to people to go even deeper with projects like AI mode. You want to give me the quick sort of just TLDR, actually, perhaps for folks who don't know what AI mode is, like what is AI mode? What does it actually do? Why should folks be excited? So AI mode is really a frontier search experience that lets you truly ask anything in Google search. So you don't have to just type in keywords.

3:53You can just type in long sentences. And then what happens in the background is using our frontier state-of-the-art model. It can help make plans. It can think. And it can basically do the Googling for you a little bit and issue queries to do research and find information that can produce this really helpful response with links to click in and go deeper and the opportunity to ask follow-up questions. And it's available in the US and India right now. Can you double quote on the query fan out that happens in some of the different agentic AI experiences? I'm not sure that I have full context of what that actually means.

4:26So how AI mode works, and we're going to also be bringing this new capability into search as well, like through AI overviews for your hardest questions that you ask. But there's models now that are within search, the Power AI mode, that are able to really understand your question and then off of your question, generate a fan out of Google search queries. So it effectively uses Google search as a tool. So if you're asking a question like, you know, things to do in Nashville with a group, it may think of a bunch of questions like great restaurants, great bars with things to do. If you have kids, things to do with kids.

5:02And it'll start Googling, basically. All of that is now possible in this process of fanning out these Google queries and then tapping into both the context of the web, but also all of these real-time information systems. That's all happening behind the scenes every time you type something into AI mode. Yeah. One of the other exciting new things is that 2.5 Pro is now becoming available to folks inside of Search with it, which I think is really, really interesting. I'm curious, like with 2.5 Pro coming in, obviously it's, as someone who loves 2.5 Pro, it definitely takes a long answer to certain queries.

5:34How do you think about that from like a product experience perspective of like where to draw the line or how to find that balance as models get smarter, but also take a little bit longer in some cases? I think ultimately you want to find the balance of the right model size, given the need to fulfill the question at the like highest level. Some questions just don't need a really advanced model. You can take, I mean, there's famous examples I see all the time on social media where it's It's like, you know, think of a seven word, you know, sentence and a model goes thinking, like making a plan, producing an outline, like checking work, like fixing a mistake.

6:07And it's like this sentence is exactly seven words long now. I don't know if that was seven words, but something like that. And it's like, you don't need to do that. So for very simple questions, I think people expect instantaneous responses. If you ask me to compare two universities for a kid that's about to go to college and understand the differences between them, what students say, the teacher satisfaction rating, student satisfaction ratings, teacher ratios, it might take a few seconds. Because you're going to do, maybe it would invoke or you would invoke like a deep research feature that may take even minutes.

6:41And that might be completely okay because a big time decision. And you're willing to wait for that to have that enhanced reasoning and go the extra mile for that kind of a question. We also have deep search. Yeah. So deep search has the ability to think and make a plan. And it might actually do a set of maybe dozens of queries. Maybe it's looking at hundreds of queries and really going deep to find information. So, for example, another example that you'll like, because I have lots of search examples. I was trying to buy a safe because I have some birth certificates I got for my daughters and I want to save them.

7:12I was actually just thinking about getting a safe for my birth certificate, too. So, OK, so what's interesting is it's a really complicated purchase that can be pretty expensive. There's lots of ways to think about them. There's fire safety ratings where there's like codes I don't understand, certain implications on insurance rates. if they have above certain levels of protection for broiler protection, you potentially get a break on certain insurance, like for valuables. There's some fire resistance ratings I didn't understand. It's actually complicated. And so I put all this into deep search in the prototype version I was using like a while back.

7:44And it spent, I don't know, like a few minutes looking up information. And it gave me this incredible response where different sections are like, here's how you have to think about this. And we had a framework for each of these various areas. It's like for fire, you know, if you're really focused on, you know, documents, let's say, and valuables, you really want to preserve, let's say, fairly, you know, family heirlooms, you know, things that like you can't replace, right? Like, so fire matters a lot in that example. And so it's like here are how the ratings would work. And here are specific safes you can consider.

8:14And here's links and reviews to click on to dig deeper. And it would do that for each of the areas. And it kind of blew me away. So that's the kind of use case. It's a complicated thing. You're willing to wait for it. But hopefully it would save you, we think, hours of research potentially with one query. Yeah, I've got a quick follow up on this. Do you notify the user when it's done? Or is there some in AI mode, you'll see that this thing finished and you can click back into it? Exactly. So you select this intentionally. So there's a deep search pill where you can go and put questions in AI mode.

8:47And so it says it's kicking off a deep research thing. It could take some time. and you see it working and it gives you some transparency into what it's doing so you have a sense of it but then if you just leave or you're like close it somehow um it would basically drop it into your history and it would give you a little alert that it finished and you would see a little i think uh like an unread almost icon so you can go back and get the report there how how much is the um model customization something that you all think about as like a a way to hill climb on product experience? Like, or, and the other alternative of this is like, obviously the models just keep getting better in general.

9:23So like, I'm curious how you think about like, we want to go and build custom stuff versus like models just going to keep getting better for us. I think it's always going to be a bit of a mix because I think each, you know, obviously as an application and an experience that, you know, billions of people use, there's always going to be needs that are unique and there's going to be experiences that you want to really dial. And whether it's in the modeling layer or a different layer, I think is an open question. but you probably want some amount of customization. But you also rebase quickly. And so every time there's a new model that comes out and then it gets deployed into search, there's huge gains in quality and helpfulness just because the model is just so much better.

10:01So now a model that can think and can plan and get things done, okay, well, if there's a future version, well, it's more efficient in that process or can think a little bit differently or better. It'll just be better naturally, which is a really nice thing. And so we started to make that process a lot easier. so that search can operate at the frontier. Yeah. And did this happen by default with 2.5? Like I'm assuming there was a 2.0 Gemini experience powering search or a 1.5 experience. Has that felt like materially different for you as one who's like trying to build these experiences? The way that they're being rebased and adopted are now extremely quickly.

10:38It's happening extremely quickly. And so you can basically get to the frontier model, but deployed at search scale fast. And I think that is a really key thing. And that's required a lot of work, obviously, from search, from the infrastructure teams, and from the Gemini team, you know, the deep mind, Google deep mind teams to get to get done. But that's pretty cool. If you think about, you know, your frontier family of models now, going directly to users that, you know, and also the frontier models are particularly useful for these hard questions. So even for AI overviews, for things like math and coding, really advanced, complicated questions that will tap into these really large models.

11:18But that can be done for 1.5 billion people. And that's pretty unique. I think part of this transition for search becoming this Frontier AI product is like people, the mental model of how you use search actually changing in some context. And I'd love to, like we were chatting off camera about like one of these being just like the way that you like bring context into search. And historically, search has been like very keyword driven and like how that will change in the future as far as like even just like the UI of input and stuff. So I'd love to expand on this in here where you're at. I mean, we really want it to just be natural and easy for people to get effortless information.

11:56You shouldn't have to think about where you ask this question. So you just go to Google, wherever that is for you, and you type in something. It could be with any arbitrary length or kind of like complexity. So you could copy and paste in like a massive block of code. Or you might just have a really long, fully expressed question with lots of constraints. Like you want to go to a restaurant, but it's with a group and one of the people in the group has an allergy. And you really just can't do barbecue because you just had barbecue. And you'd actually put all of this into search. And then we would be aware, okay, it's a kind of hard question.

12:26Let's get a model that can think and maybe plan. You may even see thinking steps start to appear. And then boom, you get that information. And so starting now, actually, these kinds of questions are tapping into these state-of-the-art models that have this enhanced knowledge and understanding of the web and Google's information to answer these questions. And that ends up powering some percentage of things that we're bringing into AI overviews now. So that's the vision, is that this just naturally occurs, and we will bring you AI to the forefront wherever we think it's useful. So one example, for instance, is people come to search for a lot is probably is inspiration.

13:04Image search is a really popular destination. So one of the aspects of AI mode that we've been working on and we announced at IO recently was visual AI mode, which means you could ask a question like, you know, what's a good outfit to wear for a rehearsal dinner in Southern California, you know, near a beach? and it could actually give you a visual reply, not textual, with all of the context you can click on and see where you could buy those items. And if you follow it up, actually, I like lighter colors and less patterns, it would understand that and be able to do this multi-turn experience. And so those are examples of how by thinking about it from the lens of informational and search needs.

13:41Obviously, if you search for, you know, just one celebrity name and you get this beautiful visual overview, like maybe actually an AI-related thing is not super helpful there, right? And that's why it typically doesn't show up for those questions. And then the search box and people's expectations of what you can do with it ends up mattering a lot. And so even just adding opportunities to upload photos and use camera, something that we see is enormously popular. All those things are things we're working through and thinking about. Robbie, we were talking off camera about some of the success that visual search is having.

14:14And it's actually the fastest growing thing inside of search right now. I'm wondering, yeah, can you double click on this? Yeah, I mean, it's pretty incredible what we're seeing. but we're seeing people who use their camera to ask questions. They might take a picture through Google Lens, through the apps. They might take a screenshot or use Google Circle to Search on Android to ask questions about what they're looking at. And what's powerful about this is it allows you to tap into the knowledge of Google and your everyday world or to ask questions about what's on your device. And this is one of the fastest growing parts of search.

14:42It's actually up 70 % year over year. It's particularly popular with younger users. Why do you think this is? Is it just like a... it's just so natural and easy to just take your phone out and take a picture of something. And it's a predominant way people communicate, particularly younger users. And I think it also just allows you to ask questions about things that you just couldn't ask before. And so we're seeing a couple of big use cases for it. One is around shopping. So people will take screenshots of what they see on social media or on YouTube or in magazines. And they'll be like, what's this outfit that I see?

15:13And they want to buy it. Look, right. That's like a big example. And then we'll You can be taking pictures of your shoes, shirt, like whatever things like it. Homework is a big one. People can just take a picture of it and get help and like actually understand the visual orientation of your homework and segment the questions. And then there's questions about this world. So people ask, you know, what's this flower? What's that building? What's this plaque? What's this book? And really, it's just your creativity is the only thing that is limiting you. Yeah, I feel like actually another reason that this is probably true to give a shout out to the Gemini multimodal team is like the model has state-of-the-art performance.

15:47So I feel like it's this like great mirroring of like what users want, a product experience that makes it really simple. And then actually like the core model is fundamentally like the best model in the world at multimodal, part of the core Gemini vision and mission. Exactly. It's also another great example of bringing together state-of-the-art models with Google searches, you know, history of quality, because there's been so much work done on visual intelligence and visual understanding with image search and image recognition. And when you bring those together with the Gemini models, it's very powerful.

16:16Let's say you took a picture of the room and you asked about patterns of a rug. Okay, what's a rug? And where does the rug end? And where does the wall start? And like, where's the rug end and the bed where to start or the bed frame, right? Like there's actually logic in even just selection and understanding what a rug is, taking that sample, finding things like it. And then when you ask a follow-up question, like, oh, but I want colorful. Okay, well, what does colorful mean in image space? And so you're able to map that into this representation that can pull that need out of an image corpus that's really unique between Google and Gemini's strengths.

16:52Another parallel question that's actually related to the toolbar is the sort of multimodal experience of search. And you mentioned image as one of those modalities. But I think speech is actually part of that experience as well. I'm curious how you think about like, is there a world where you're actually just like talking to Google search and instead of like reading a bunch of links on a screen or something like that? Absolutely. I think part of being a frontier AI product is having all modalities available to it. At the end of the day, there's just knowledge encoded in this model that's able to produce information and experiences, basically.

17:24And that model should be flexible. And so one of those modalities should be voice and it even could be live voice. And so we recently shipped an experiment actually in labs for Search Live, which we announced at IO as well, which allows you to have a natural conversation with Search. And so in the same way that there's this model that knows how to do these fan out queries, look up information for you, tap into all of these, you know, real time information systems like the Google stock information or sports scores or shopping products or locations. The model can do all the same things, but it's been designed to be pithy, conversational and like have a back and forth with you.

18:00So if I'm in the car now, I can just talk to Google and like work on a project. So if I want to just like planning a trip, I could go, Hey, what are these things going on in this location? I'm going to order cool restaurants. What do people do? I don't like that. I like that. And then when I get back home, I'm on my couch. I can go back into that thread and AI mode and like kind of resurrect that and then keep going. And then I'm at work. I could go back the next day with a full monitor and then go deeper and save some stuff that way. And so that's neat because you can now have it like across all the contexts of your day.

18:30but it's still tapping into this search kind of frontier model that we built. Yeah, I love this. One of the questions, and again, as a DAU of AI mode and of AI reviews, but also as a DAU of the Gemini app, I'm curious how you think about the interplay between these two, where the Gemini app is going to go relative to where search is going to go. I have my own sort of mental model as a user, but I'm curious from your perspective, as you're designing these experiences for people, how you think about the difference. Yeah. So for what we're trying to solve is really what people come to search for.

19:04So we really think of it as AI search and really AI helping you with these informational needs. And the things that we see a lot, whether it's obviously research and learning, but it's also shopping and browsing. It's also information about health, for instance. There's lots of things people come to Google for every day, and we're really focused on these informational tasks. And then obviously with Gemini as a universal assistant, it's a lot wider, I think, of what it's trying to be. And so I think in the future, it can do everything from help vibe code with you to deploy apps to reorganize your life for you in ways that it will be really helpful.

19:38Obviously, some overlap there with general informational questions, but I think that that's how they can work together. And so it's typically if you're the type of person, you're coming to Google search, you have an informational question. This is an AI way to get the most powerful version of search in the same way if you're in Gemini a lot and you're using it for all these interesting productivity use cases or code or whatever it might be. You can also ask questions there, too. And that also obviously has lots of Google search grounding and Google search enhancements as well in that experience.

20:05One of the tangential questions I was just thinking about, and I don't fully know, like, I think you probably have a better worldview of this. Like I know how it works from a Google user perspective, less on the actual like behind the scenes product side, but like Google flights and like some of these like experiences that like show up in search. I think of them from my like developer perspective, AI builder perspective as like tools perhaps that search has access to. Is there any world where like it's like integrating with those things like fully as like we would traditionally see as like tools?

20:34Absolutely where it's going. So first of all, already we've integrated most of the real-time information systems that are within Google. So it can make Google finance calls, for instance, flight data, those kinds of movie information. There's a trillion facts in the knowledge base, in the knowledge panel at Google, 50 billion products in the shopping catalog, in the shopping graph. It's pretty incredible with merchants updating their listings because obviously they want people to see the correct price and correct information for anything that they can buy. It's actually updated, I think, 2 billion times every hour or so.

21:07So all that information is able to be used by these models now. But then going one step further, what are the experiences that you can build to make this really shine? One of the principles that we've been thinking about is something I think Google historically has been great at, which is the right response given the question. And so if you ask about, for instance, the stock question, to explain the trends of a stock in language is a very onerous thing to consume. Let's say you have a question like the top five CPG companies and their stock performance over the last 72 days. Let's say you wanted an arbitrary number like that.

21:46The model would actually have reasoning involved. So it would first figure out what are those top companies. So it would do some analysis and do this fan out query process, look up the top companies. It would then need to do a tool call to Google Finance to get real accurate, real-time information of each ticker. it would then do a tool call for a visualization library to chart those things with a description of the trends. Same thing with products, like you'll see a visual grid today of certain products. If you ask for, let's say, headphones ideas, it'll give you this list and a little tray. And if you click on it, it'll open up the sidebar that has all the pricing information and all that data that Google has, you know, typically available for people in search all within this AI experience.