Google I/O Afterparty: The Future of Human-AI Collaboration, From Veo to Mariner

23:14And that really kind of points to the question you're getting at, which is, I wanna just hand the task off to this agent, but then I wanna be able to just verify what it did at the end of the task, not sit there the entire time and watch it. Yeah, yeah, so interesting. What do you think of as solves and the unsolved technical problem so far with computer use? Because computer use still feels like to me, or maybe in the Will Smith, you know, the spaghetti is still sort of disappearing a little bit, phase, and maybe that's non -fair characterization. But I'm curious where you think we are on the e -vails and the performance so far for computer use and whether the unsolved problems right now I think that's actually a totally valid comparison There's a reason we launched this as a research prototype with the experiment level on it right now I think we've seen really big gains from December to what we launched last week That said there's definitely still model quality improvements to go I think there's also just application level improvements to go there's more seamlessly being able to have the user provide context upfront, which will make the agent more capable of understanding what it is it should be doing.

24:17And then there's just more planning and reasoning that we could do, like at inference time, or at the application layer time, that sort of in addition to the model improvements, you know, improved system instructions, improved checks and calls to different models. And then of course, right now Project Mariner entirely completes a task by actuating or taking action on a browser. You want an agent that has more skills than that. You want an agent that knows when to call the right tools that has memory that's able to you know, take advantage of a lot of the other stuff that we already see out there.

24:51So I think it's just integrating a lot of that in and starting to innovate and climb on that. And then of course, right now Project Mariner, it's in the browser. people use computers. So you know we call this computer use. So there's that entire dimension as well that I think we're going to continue to see innovations in. Really cool. Were there any contrarian opinions you all took in building mariners? So for example I think some people have said screenshots it's going to be too slow. It's not going to be fast enough you should use the website DOM or whether like any contrarian bets you guys made.

25:26So the reason we went with the screenshot is we wanted to make sure that it was a skill that we could develop that could be applied across things that aren't just websites. I think the other aspect of that is like DOM versus accessibility settings or accessibility pieces and other leverage. We're kind of betting on this one right now, but I would say everything's evolving. So we're just willing to take pivots if in one it makes sense. Yeah. Yeah. Makes sense. What is it capable of doing it today and what is the speed? Like if I tell it to go, you know, the canonical go order me a pizza from Domino's.

26:06Can I do that and how long does it take? The speed is definitely an area that we want to keep held climbing on is what I would say. But it's interesting you say that because one of the things that I, so I was recently using Mariner to help me complete a task which was come up with, let me take a step back. I have a three year old at home. She is gonna be four soon. Part of that means organizing a birthday party for her and being able to figure out loop bags for kids at a four year old's birthday party. This task, because you can imagine involves understanding what to put in the loop bag and then actually buying all of those things or like finding links somewhere to go buy them.

26:43And I gave Project Mariner this task and it was basically a personal research that turned into an action taking task, which is finding me the links and save them. And the thing that really resonated the most with me on that one is as it was performing this task first did a search for good ideas to go in a loop bag. And then as it just remembered those five items, that's something any of us could do like that itself was an impressive. But the first one was I think temporary tattoos. So then it started looking for temporary tattoos. It found a great link for it. Instead of having to copy that link and paste it in a doc somewhere else, it could just remember it.

27:16It could remember this massive URL. And then it moved on to the next one. And at the end of these five items, it just gave me all five URLs that had been able to inherently store. So when we talk about speed and efficiency, I think there's two dimensions. One is just the model calls on the, you know, taking action and like, how do we improve it with different tool use? But then the other one is, how can agents just do things in a different way that are inherently faster than the way we would do things? And I think we're going to continue to see improvements on both dimensions. Yeah, I wish I could remember five URLs.

27:47Oh gosh. Okay, good point. Let's see. What do you think is ahead for Mariner? Where do you see the evolving from here? I think there's a couple things. Number one, we had a bunch of announcements last week around project Mariner like capabilities, making their way into different Google products. And I think that this is a kind of core capability that you'll start to see emerge everywhere from the Gemini app to AI mode in search. So I definitely see a lot more coming to Google products with the stuff that we're doing right now in Project Mariner and kind of paving that path forward. And then I think for Project Mariner itself, I actually like to think of things in three categories.

28:28There's the agent itself. I think that's going to get smarter, that's going to get better, that's a better model, that's tool use, that's memory, that's context. Then there's the environment. We talked about how in December it operated on your local desktop and your Chrome browser, so that's in the foreground. Then we moved towards this idea of Project Mariner operating in virtual machines, which meant that is now operating on VMs. I think there's this middle layer, which is an agent that can still operate on your device, but in the background, and there's a bunch of reasons and types of tasks where that becomes a really important kind of way for the agent to operate.

29:00And then of course, there's all the other devices. But really what you want is a capable agent that's able to operate in a way that is omnipresent across all your devices locally on VMs. And then the last one is the ecosystem part, which is where you start to get into the agent to agent interaction and how does your agent interact with all of the things that exist outside of its own world, essentially. Yeah, so cool. I think the canonical examples for computer use are book me a flight or order me a pizza. Is that your sense of what computer use agents will actually be really good for? Or whether you think, I'm sure you've seen a lot of time thinking about what applications will actually be the bulls eye here.

29:42How do you think that shapes out? So I think we default to those because they're just easy to understand. The travel planner, I mean, literally it's a travel agent. It couldn't be more analogous when you think of agents right now. But no, the way I like to think about it is on a spectrum where you have tasks that are sort of in what I would consider do it with me, where you have your agent alongside and you can easily offload certain tasks to it, but it's really working in in unison with you. And then you have these like do it for me tasks, which is hey, I just want to give my agent a bunch of stuff to go do and it will run it in the background.

30:18I think part of the reason we see these tasks being used is twofold one. They're just incredibly easy to understand and everybody kind of gets what that use case is and they're usually starting from scratch. Like there's no context you need upfront. You can just send on an agent out to go do it and the demo as a result is pretty easy. to put together. Yeah. And then the other one is just where the capabilities are at today. And so as agents get more capable and you start to have more of these realizations on what they are actually able to do, you'll see much more advanced use cases or much more complex use cases.

30:55And that also requires the user having more trust that they can give to the agent. So I think that that will evolve over time and we'll see people come up with even more interesting use cases that they're willing to give an agent good to do on their behalf. Yeah, it's all like, It's also going to require, I guess, it's going to inspire, I think, a shift in business model, right? Basically, if you have a bunch of agents going off and browsing, you know, trip planning, for example, they're not necessarily looking at the ads and, you know, the first things that show up. And so it is, I think it's going to create some business model evolution as well.

31:31I agree. I think there's a lot of evolution that's going to happen across business models, across how websites work across how users will always want to use the internet going forward. Like there's a lot of joy. I think we all get in it from content creation to consumption. But there's also a lot of other tasks that it's just right for disruption in a lot of ways. Yeah, yeah. Like, I'm thinking humans are suboptimal in some ways. We see that we get excited, distracted, and I go and buy the dress. And my agent, maybe I can instruct it to ignore the ads. Maybe it actually knows it's going to find the best content regardless of what's showing up on the page.

32:12So it's kind of interesting to think about how that future plays out for agents to do more of our browsing. It's super interesting. I will say that the dress that maybe you got distracted, I always get distracted by things to and end up purchasing stuff that gets done my way. But I'm most happy with it by the time I do end up purchasing it. So I think that there's new opportunities to think about how do you actually involve agents in this new business model ecosystem and hence that third bucket of like, there's gonna be a lot of evolution happening in that space. I think that that's where we need to evolve as an entire ecosystem and it's not just like one player that's gonna say this is how it's done.

32:51So it's been interesting just talking to different companies and different people who are also thinking in that space right now. Yeah, really cool. I mean, I do think also just as a user as well. So, you know, I often don't buy things on the internet because it's such a pain. Oh, I've definitely dropped off of it. I can't navigate this thing. I don't understand it. That happens quite a lot. Or it's just like, I've just not got time. That happens as well. Or I can't be bothered. You know? Maybe it's just me. But I'm not a fan of shopping. Let's put it that way in the real world and online. But I'm a fan in what I get.

33:26You know, I'm a fan in the outcome. and so I don't know, I kind of feel like I might do more, I would probably do more online shopping, I think. If I didn't have that barrier of actually having to do the shopping bit, I don't know, that would be me though. No, I agree with you. What's really interesting is, I don't know about you. There are certain stores that I'll go on to and I'll just like accumulate stuff in my cart and I won't wanna, like pull the trigger until a little bit later on, I don't know how to chance to think about it. Yeah, yeah, yeah. But then I end up with a bunch of half -built carts across a bunch of different websites.

33:59And part of me also wonders, is there a world where my agent is at Universal cart essentially? Where I'm at all this stuff to create this aggregate area of all the items that I might be interested in buying. And it can be across any site at this point, because the agent represents me. And it can remember which sites to go on. And then when I'm ready, it's sort of like, okay, one click, make this entire purchase basically and it can go and check out on all of the different sites or all the different stores. So that'll be an interesting area to think about. Yeah. Okay. When I just heard from you guys, is e -commerce conversions about the skyrocketing?

34:34I mean, on my computer it'll go up. That's what I'm saying. I don't know about anyone else. It's the diversity as well. Like I go to the same old sites. But I would love suggestions. Yeah. Yeah. So yeah, it's like once you kind of democratize computer use, than the laziness of humans to get through, check out, there's no longer the determining factor of which e -commerce companies will do well. It's just like the best product wins. It's, yeah, so interesting. Okay, cool. Thank you for sharing. You're welcome. Okay, Simon, you're last. Hi. Notebook, notebook LM or notebook? We'll go with notebook LM.

35:09We're still in that book. I think it's been so long now that it's definitely notebook LM. There was a period where we were like, okay, it's now the time. Yeah. But I think we've gone through that multiple hockey stick moments, which you can talk about. Yeah, it's going to be hard to remove it. I like it though. I mean, maybe every product that this kind of like has an acronym or some weird letters after it and there are a couple of them in the AI space regrets that. But at the same time, they become part of the team and the identity. Totally. Yeah, it's nice. I like it. I love that. Okay, so notebook LM was one of the biggest, one of Google's biggest viral hits last year.

35:47Last year? Yeah, it went viral last year. Yeah. But you know, the team had been building it for a while before it took off. Totally. Yeah. Tell me about how it's evolved in the last year. Yeah, yeah. Well, so firstly, the viral moment, you know, so my way into notebook at LEM was through audio overviews. So me and the team had, we were also exploring the future of content of my different angle. I think. Well, notebook was the perfect balance of user control, but also the power of the technology. Our hypothesis was that there was an opportunity for personal content. So not content that it is for everybody, actually a content that's from an audience of one.

36:35maybe two, maybe three, small group maximum. And that was kind of how we shaped the product. We didn't think it was gonna, looking at the notebook user base back then, we thought that it was a great place to kind of test PMF, just kind of iterate on the product. We were totally unprepared for the massive success of audio overviews, and then through that notebook, I'll end as well. So it was honestly the first couple of months was really just kind of hanging on for dear life. Firstly, it was making sure that the TPUs don't fully melt. So as a riser, I think, had a gift out back then. But there was also just a lot of iterations and fixing things and improving things.

37:20And that was really the first couple of months. I think since the start of this year, maybe we've managed to take stock. So at the end of last year, we launched the Join mode, the ability to join in a podcast and what you owe you, I should say, and talk with the host and ask questions and all this kind of stuff. But at the start of the year, we kind of took stock. And we've really been thinking about, what is a notebook for the notebook users? How are your notebook users really leaning into notebooks once they've come in the front door through audio reviews? And we've started to think about, and Jack, you kind of touched on this, I think the criticality of context in really enabling these AI systems to be genuinely useful for you.

38:00And we've found that a lot of users, when they're using Notebook, they use them for these kind of more longer running, almost like projects that they have. So either they're hobbies or if they're in the world of work, these, they can be ongoing projects or they can be projects with a goal. You know, like I've got to prepare for a presentation or something like that. And so a lot of what we've been really doing is retooling, you know, how we look at Notebook and also, you know, building a strategy as well that leans more into, And I think there's more sort of longer running opportunities that we see in the internet book user data.

38:37Of course, we've done a whole bunch of improvements too, so we've just launched the mobile applications. Finally, so they came out last Monday. And we also launched international audio reviews as well, which was kind of the end of a long road, honestly, of upgrading the underlying AI infrastructure and models away from the very first almost like research grade model that we used for the initial launch to, you know, native Gemini audio. So what you hear now in the international audio overviews that are very least native Gemini audio. And that was a big push for many teams across labs and also GDM.

39:16Yeah, super cool. It feels like audio overview was almost the viral hook. And you guys have been building out a lot in almost like the rag UI. Yeah. And just imagining what that workspace looks like. Yeah. What do you think the actual audio overview podcast thing becomes? And actually, I'm curious how you even ended up on the shape of two podcast hosts talking to each other. It's just like, it's such an engaging format. I'm curious how you even landed on that. And I feel like it's only in its infancy still in terms of I would love podcasts every morning to type me up for my day and things like that.

39:54And so how much of your time is thinking about notebook, the kind of rag workspace environment for lack of other word versus notebook, the podcast killer, the training data is going to be built on notebook in the future. Yeah, yeah, yeah, well, I hope not. But maybe it can help. So the way that we started increasingly to look at notebooks is that comprised of kind of three, they give you sort of like three superpowers. So one of them is they help you really accumulate information over time. And that's a lot of amazing underlying database technologies that we apply that I think lean on first party Google technologies in a pretty unique way.

40:37The second is they bundle in intelligence and when we launched last year, we used the old Gemini 1 .5 Pro model back at that time, but obviously now we've got thinking models and so on. But the third thing is the ability to, for content and information to be adaptive to your situation. And so, you know, podcasts or audio overviews, a conversation, it's one form that information might take, but you can imagine many other forms that that information or knowledge might take as well. So you might imagine it coming at you in the form of a comic book or maybe a short movie or maybe a mind map which you've also launched.

41:20But you can imagine many other types of media that fit the right circumstance and form and function for the moment, for you to understand information, to be able to analyze it, make decisions with it, kind of do except that I think we have when we're thinking about the different, might hear me, I'll talk about transforming information from one state to another. I think that's a fine word. It's a little bit technical to be honest. It's more like adapting to you and fitting you, I think that's really what we're going for. But in terms of just going back to your actual question around audio overviews and, you know, where it's going, there is a huge amount, I think, of room left in that technology.

42:08So, you know, I enjoy audio reviews and I use them a fair amount, but I also, you know, every now and then I'll be like, that's weird. You know, why do they say that? Well, they've kind of lost the plot there. I didn't quite get the right narrative. It's sometimes it, like the uncanny valley or the illusion is broken, you know, when you're listening to them. And, you know, well, it might seem like there's a small amount of work we might need to do to, to fix that last step. There's actually a ton of work that we've got to do. And so there's a lot of effort being placed into all of the various components that you'll need to make the experience feel like something where you suspend your disbelief more completely.

42:50And alongside that, there are many other different show types. We've kind of had one show type for a bit too long, I think, actually. And we're bringing more out. So we're actually working on some really cool things a lot of them inspired by users honestly So one of the things that we saw users do right back at the start But you can keep on seeing it is Users putting in that LinkedIn You know, they're putting a little in or why well number one It's kind of fun to like hear people talk about you, but a lot of users are using it to get feedback You know like to kind of I to understand from from another person's perspective who they may not have access to you If feedback is truly a gift, like real feedback is hard to find.

43:32So, how would somebody else look at me? How would somebody else talk about my strengths? And how might somebody else talk about areas to improve? This is something that we see users already using audio overviews to kind of access that sort of content or sort of information. We think we can make that easier for people. So a lot of what we're thinking about now are different show types that are in into some of the more viral successes that we've seen are users, you know, explore online. And also think about, you know, brand new formats as well. I think it's going to be fun. Okay, so we're going to have training to the comic strip.

44:10I'm not saying we're definitely going to have it, but I mean, it's, I think, I think not everything has a story, you know, and so applying different adaptations will almost be sort of context dependent, I think. But oftentimes it does help. You know, so one of the things that we were looking at the other day was we were looking at a 150 page PhD dissertation on it was invasive invasive wolves I think in some part of Europe and Yeah, you could have looked at a my map. You could have looked you could have maybe listened to an audio overview If you had like 10 15 minutes of spare, but actually getting a kind of a comic book rendition of of that PhD was really helpful just to kind of understand the overall narrative within it.

45:01So, you know, we're still working on things like that, but I think there's a lot of opportunity there, and of course, you know, comic books are very similar to storyboards and that intersection work. I was just thinking exactly that, yeah. The Thomas is doing too as well. So, yeah, there's a lot of interesting ways that I think labs projects intersect and we'll continue exploring. So you can create the heroes journey comic book of somebody's LinkedIn career arc. I mean, for an audience of one person and one person only, that's probably going to be the most awesome movie ever seen. So maybe, maybe.

45:35That's awesome. Really cool. Where do you see the notebook going from here? Yeah, like I said, where we're really, I think our focus is aside from a whole bunch of different adaptations. We're really thinking about how we can be more useful to our users over their more longer running projects. And so both in the world of the knowledge worker, but also in the world of students these are kind of like core users I think. The project is really the an area where those users both need the most assistance, but it's also the point of highest value I think for them. So if you're in the world of work, the project is where value accumulates.

46:16It's not like you need to work. Yeah, right. It's a real unit of work. It's a, we actually call it units of knowledge, but it's a great way of putting it. And the same for a student as well. The project, if it's a project with a goal, passing a test, that's a big deal. Or if it's an ongoing lifelong learning thing, that's also really important as well. So I think really focusing on use cases in those domains is something we're thinking a lot about. I'll say the other thing is, I think one of the things I'm personally very excited about. I've been in the consumer product space for many, many years.

46:50And I guess one of the things that we did at Google when we went mobile first in the mobile first era is we moved a lot of our desktop products to mobile. And if you look at those mobile products, many of them are the desktop products shrunk down to a small screen. And that's okay. And I think because we were one of the first, because we built Android and a lot of our big products basically got mobilified at that point. We found it hard to change at that point forwards but I've always been really interested in thinking about if you have a desktop experience, what is a companion mobile experience that doesn't have to just be a carbon coffee of the desktop experience that maybe leverages the form factor, the sensors, you know, the actual, the fact that it's on with you all at all times, to deliver an additive experience on top of the desktop experience.

47:42So, you know, we've just launched the mobile experience after a fair amount of time in development, it's fair to say. But what I'm really most excited about there is the opportunity to actually iterate on that kind of novel mobile experience going forward. Like, for example, wouldn't it be cool if I was, you know, maybe I'm in a discussion with some amazing, really smart people and I've popped my notebook down, I've opened a voice, it's native voice recorder and it's just able to record the conversation for me. And then I can transform that to later days and accumulate them and all this kind of stuff.

48:15That's the thing that is probably gonna be weird if I open my laptop and push record on my laptop. But the mobile device is the perfect opportunity. –Cobely. –Yeah. –Really cool. Thank you for sharing. –Yeah, nice. –Okay, we're gonna close it out with some predictions on AI as a whole. Please jump in. Hot Takes, welcome. Let's see. Let's start with what are your favorite Google labs projects that we didn't talk about today like what what is under What is the what are the gems right now that you're most excited about the unreleased all parts that we know last talk about Not then released You guys just announced like 50 things they have to be others beyond the three we talked about Yeah, yeah, I have one which is kind of still in this like video and image space, but I think the virtual try -on on stuff that we presented, like this, a lot of preparation.

49:02And I think that went to me really nice because I think it meets a real direct user need. It's the strength of Google. Obviously, we know we have all the inventory and how to connect this. It's just so fun to just see things on your site. So I'm very excited about that one. I think this says like a good thing. That's my favorite as well. That's so funny. OK. I think that one's a good one. Stitch, I think, is really cool to be able to just talk to the product and describe what design you want and have it actually come out with that front end design. I'd been using it a little in dog food before it was launched and so it's just I want to spend more time Using it now that it's it's actually live really cool without you Well, mine's gonna be stitch I guess what areas do you think will be hottest in the application space for a broadly in 2025 like I think coding was maybe the breakout application in the last 12 months.

49:59What do you think will be the breakout application in the next 12 months? Video. I think there's something around these remixable content. You generate something, I take your thing, I just riff off of it. There's something around this that I think is going to pop up somewhere. I hope it's us. But that part feels really interesting. It's going to feel like if you're whisk is heading a bit that way, view obviously can power a lot of this in video. I think that's going to be something this year. As you look back at you know past predictions of what you thought was gonna be interesting And I what wherever you got been really right and where we've been really wrong Just we know we're let's say where we're really wrong altogether.

50:35All right three two one timing

50:41I think that there's been several examples where we definitely felt like we were on to something and we were on to something We were just too early into the space and so it's been fun to see like projects kind of go on pause or you know, stop for a little bit and then some of them are starting to even come back around. Again, at this point and so sometimes we just were a little too early, but it just gives us a jump start when the models and the capabilities are ready. Good problem to have. Well, you think you've been really right on and like sticking to your convictions on. I think this, at least for me, a nice space like the Show and Tell piece.

51:15This idea that like you shouldn't, you shouldn't ask users to kind of write two pages of text to describe, for example, in image. the idea is that you should be able to show and tell like you would do a friend or an artist that's working with you. I think that has stuck in this, it's kind of moving people away from prompting and towards kind of instructing and relying on the intelligence that lives behind. So I think that's one. But when I'm sticking with my guns and I think you're yours there to stay. Yeah. I mean, this is probably obvious at this point, but when we all started in labs, there was no Google LOM API, Google didn't have a functional instruction tuned language model or anything like that.

51:55I believe it will not back then. In fact, I think the general consensus was that these were not really things that are easy to build a business around because of their cost. I think one of the things that was all done actually is we've kind of stuck with the technology and now it's obvious, right? But in the early days, it wasn't like, it certainly was not obvious. So, yeah, we got that bit of timing right. Yeah, yeah. Yeah, yeah. Inferno's costs, just writing that curve and just capabilities up, cost down, and so what will you build assuming that those curves continue? Yeah, exactly. In fact, when we joined, one of the transitions is right to think inside labs that George started actually.

52:36And a lot of the docs that we'd write were around, well, what happens in two years? And of course, yeah, that curve is something that I think inspired a lot of us. Yeah. Fantastic. Thank you all so much for joining to share what you're doing across the creative sphere, the computer use sphere, and the, what do I call the notebook sphere? The podcast killer slash. Let's not say podcast killer, but yeah, we can say knowledge. Knowledge creation, transformation, space. It's just it's really really cool what you all are building and you guys have such a cool job getting to kind of cook in the little test kitchen of Google And thank you for giving a preview of some of the stuff that's coming down the pipeline Thank you