Google DeepMind's Logan Kilpatrick: Why the Model Eats the Harness

24:26I need to turn on that auto-tweet deleting feature or whatever it is. um last october you tweeted everyone is going to be able to vibe code video games by the end of 2025 yeah did that end up being true it feels close and i think there's i mean it obviously not triple a games like you're not building a you know the next call of duty or gta yet um but i think it's it feels closer than it's ever been um and i think a lot actually a lot of the interesting bit about video games is you actually need to end up building a lot of this like other stuff like models and we were talking off camera before this like 3js is a great example of this like 3js makes a lot of things possible that weren't before but there's still all these like rough edges that like just a coding agent doesn't solve and so you need like you know sprite generation and like the models aren't very good at doing that natively and so you need like some orchestration layer and tooling in order to make that happen there's a bunch of other things like that that like are core to like the gaming video game experience that need to have a high degree of reliability that I think it feels like it's within reach, but actually like requires a lot of like product scaffolding work in order to create experiences that are like reusable and replayable and sort of like have the level of depth and requires a little bit of taste in there.

25:44Do you see people making a lot of video games inside AI Studio and the other developer surfaces that you have? Yeah. And so this was actually based on like us looking at the early data. And there was something like in AI Studio at the time, it was like 20 % of all apps that folks were making were actually games. Like people were trying to build games. A lot of it. Is that the most popular category? It's not the most popular category anymore. Just because I think like the ecosystem has shifted and like the user base has shifted. But it was a lot of a lot of games. What is the most popular category?

26:13I think it was like it's like 20 % like finance related stuff. Wow. People like counting their money that much. People like I think it's it's something around crypto, actually, I think is what people are doing a lot of stuff with with finance. A lot of like personal productivity things and a lot of gen media stuff, actually, because obviously the Google suite of gen media stuff has done a great job. But I also think GDM has sort of like a obviously Demis cares a ton about games and sort of like started his career and doing AI stuff because of games. and so I think we'll have some interesting swings at this and our team actually in Kaggle which is sort of a bunch of the AI benchmarking stuff we do in GDM sort of works with GDM to build this game arena which is sort of our way of sort of like testing progress towards AGI like using games as a proxy which again is like very deeply rooted in GDM's history so.

27:07How close do you think we are to you know rando off the street with a good idea can vibe code a really fun playable game i want to say this year i actually i think it's i think the model capability makes it possible i think this is where like i've gotten excited on the product side and you know we were again we were also talking off camera about sort of like the startups in this ecosystem because um it feels like it's possible it doesn't feel like there's a gap in model quality it feels like there's a gap in like you someone who knows what it takes to build a great game actually like putting the scaffolding together in the right way to make that possible.

27:40I think there are folks who are doing this right now. And so some of it is like a discoverability and awareness thing that like people just don't even know that they can do that. And some of it is just like maybe certain categories of model capabilities are just like slightly off and we're like, you know, weeks or months away from like that chasm being crossed and then it just like working for most people. And so this is a good segue into, I want to ask you about world models next, But do you think vibe-coded video games is more likely going to be game engine plus coding agents based? Or do you think it's more likely to be world model based?

28:17Yeah, I think what will end up happening is the definition of world models will blur, which we should talk about with Omni. And it will still, I think the like coding agent will look like some sort of world model type system. But you actually do need to make world models useful for like real things. You need like scaffolding. And so I think there's, again, there's actually a bunch of interesting startups like doing work, like figuring out what is the scaffolding for world models so that you can take them from these like very open ended. of the inherent design of world models, very open-ended spaces and like do it in a tangible way so that it's like grounded in a use case that like you could use in a reoccurring way.

29:03That could be somebody maybe will figure out the scaffolding for world models to make games possible. But like the inherent nature of world models right now, I think make it so that it's like actually not well-suited for like games in the current form. But the progress has been crazy. So who knows, maybe in like two years, the versions will be able to, but at least in the short term, It's like coding agent plus some sort of game engine, I think, is like where you'll see way more alpha from a game's perspective. That makes sense. OK, so you said the definitions of world models are blurry. Can we unpack that?

29:33Yeah, I mean, I think like Omni is an example of this. You know, we launched this at I.O. You can sort of take in any input, create any output. And I think Demis sort of like framed it to the world, rightfully so, as a world model because of just like the level of understanding that it has of the world. I think that like technically looks different than, and I'm not an architecture expert on like the way that we've done world models before, but it is different from an architectural standpoint than what's happened in the past, which I think is positive because it's getting closer to like some of the ways in which it might actually be more scalable.

30:09And historically, like it's been like super not scalable. It's like very, very expensive to run traditional like online world models. Yeah, like Genie being like. Yeah, exactly. So if you think of traditional world models as being like an action conditioned video model, then like right now, when we say world model, what we actually mean is a model that has some understanding of the world as opposed to being strictly technically a action conditioned video model. Yeah. And so the interesting thing though, is like it has understanding of the world, but then it also has that like really great. And that's where like the line is blurry to me, where it's like it can do a lot of those same use case.

30:49It's not real time right now, but like it can do a lot of those same use cases that you would describe or like visually could create with that same exact world model, which I think is what's most interesting to me. So I do feel like this like world model, video model thing is gonna change and play out in a different way than was obvious before. And how does it work under the hood? Like whatever you're able to share, like is it Gemini plus video models? Is it something different entirely? It is a single model, which I think is the important part. Like this was actually part of the original desire was like you were training like eight different models to do all of those things historically.

31:30It's like you have a text model with the baseline Gemini model. You have audio. You have music models with Lyria. You have Nano Banana. You have VO video models. We have a whole suite of audio models. And like, it would be great for us, our customers, if you just had a single model to do all those things. So it is like a new setup that sort of makes that possible. It's not like routing to a bunch of different models, which like we, you could have imagined, we could have done something like that actually before and done like a Gemini Omni model. But this is like a true Omni model. And it's starting with like the use case that works the best right now, which is why it's the one that's available.

32:08is this video editing capability. Technically, it's functional with the other things. It's just the quality isn't perfect and is not state-of-the-art, so we haven't rolled that out yet. It's also just the first crank of the model turn on Omni. It's the Omni Flash model, the first iteration. And so we'll have much, much more capable, powerful versions, which will be exciting to see. So we could edit this set so it looks like we're... Good. Yes. Yeah. I want this. Again, we were talking off camera. We should do that for the intro because I think it just makes all this stuff more capable. And I've seen these examples of such subtle nuance that make me appreciate that it's like the world understanding playing out.

32:53I was giving a talk and was on stage with my friend Tulsi, who leads the model team, who I don't know if you've ever had on before, but she's amazing. I love Tulsi. um and in i mentioned to someone in the crowd to like edit the video and they literally like took the picture edited with omni in real time and this like dog came on the stage uh and like the other in the edited version the other guests sort of like look down and see the dog they like chuckle a little bit this is while i'm like opining about whatever ai nonsense yeah that it was not my jokes they laugh at the dog coming up it jumps onto my lap i sort of like acknowledge the dog i keep talking i'm like petting it or whatever and just like there's like so much subtle subtlety in getting that right and the model crushed it and it's just it's very interesting and like still trying to like absorb and digest like what that means for you know the way we make content and all these other things that's so interesting yeah i'm i'm the biggest bull on generative media and what it means and i mean one of the things we've thought about for our podcast is the visuals matter as much as the content.

34:00For sure. That's how you catch people's attention in the first place, right? And so, okay, I'm excited to play with Omni. I'm excited too. And I think you probably feel this way as somebody who makes content, but I've historically been very, for myself personally, I don't use AI to make any content that I produce. It's all my words. It's always my voice. It's always my image and picture showing up. I feel like there's just so much alpha and authenticity. And so like, I would much rather it be me than some AI version of me. What I like so much about Omni is that it's like not changing me. It is like changing a bunch of these other bits, which are not me.

34:38Like I didn't choose any of the like set around us or the coffee table. It's like, so our words can stay the same. And like, you can change these bits that are like not personal and do something more interesting with them, which I think is really, really cool and feels like the version of what I want sort of like Gen Media to be, which is like not a bunch of like AI avatars. No Fruit Island videos? Exactly, truly. Like it really is like it's the original content. It's the person. It's like the personhood is there. It's just different and amplified. Super interesting. Okay, I'm excited to play with it.

35:14Yeah, we should send some prompts right after this and try some things. I don't mind the fruit videos though. I'm happy for a world of both. On the coding side, you launched the ability in AI Studio for people to vibe code Android apps. Yeah, yeah. I'd love to hear how that's going so far and where you plan to take that. Yeah, it's super exciting. I think one of the strategic things for AI Studio, and actually this is based on a lot of the feedback from the ecosystem and actually from developers, from others. It's like so many Google products. There's so many different ways in which you touch Google through all these different journeys of building a startup or bringing an idea to your life.

35:51And so we have this like first class principle of like, how do we bring things into AI Studio that make it so that you are exposed to other parts of the Google ecosystem without having to like go through nine different UIs across Google. And so Androids are like a great example, not only of that, but also of enabling people who wouldn't have otherwise built an Android app. And so I literally built my first Android app in AI Studio. Very cool to see. What is it? Yeah, I just did like a plant, not a crypto app, just a plant one. I was planting trees in my backyard. Yeah. And so it was just like playing around with a gardening app as I as I was kicking the tires.

36:32I haven't had my like breakthrough idea yet of what I want for a mobile app, but I'm going to I'm going to come up with something and see go compete on the app store. Have you seen anything vibe coded like really fly in the app store yet? That's a good question. It should be interesting to like see some analysis. I don't know. I'm sure it's like accelerating a lot of things on the app store, but I don't know how much, like, I don't know anyone like personally who's, who's done that. It is interesting. And I was going to make the observation too, that I think the last time I checked the numbers, we were viewing it this morning, it was like 350 ,000 Android apps built in AI studio since last week, which is crazy.

37:06Um, and like, excitingly, it's like 350 ,000 apps that like, probably no one was going to build before. A lot of these are personal too. And so this is where I think this like, maybe GenUI is like farther out there. But I think like the idea of you building software to solve your personal problem is like very real right now. And like people are doing that. It's like one of the most common use cases of a lot of these products. And being able to like unlock a bunch of the native capabilities of the phone, I think is also really interesting because you just have so much context that's like in different places.

37:39So I'm getting very excited about sort of that opportunity. And Android feels like it's becoming the platform for builders. Does it matter that something is an app versus just like the web is so powerful now? Yeah, it's also very interesting to see that play out. Web is definitely powerful. There are certain things that the operating systems have that like you just can't unlock, like lots of like native richness that actually like make experiences feel so much richer. I think about this for like text messaging, actually, that like the text messaging experience and all of the, and all the main operating systems feel way richer to me than like any AI chat app that I've ever used.

38:20Like if I could just talk to AI and whatever texting app I use, like I would be way happier than having to go to some other app because I think we're also just like conditioned on like the operating systems. So yeah, makes sense. Okay. I want to ask about the model eats the harness or the model eats the scaffolding. What are your thoughts? Yeah, I think it's true. And I think part of this is like what we have historically thought of as the model is not the model anymore. Like when I think like two years ago when LLMs were popular, it was like the model was like actually just a set of weights. It was a set of weights and it was like really like, how can you like as simple as possible send tokens in and get tokens out.

39:01And I think we've just like progressively step by step by step. We still call it the model. We still call it, you know, Gemini 3.5. You still call it GPT, whatever, and Claude, whatever. But like, it's actually not just the weights anymore. It's like an entire expanding sprawling system that's built around the weights that sort of like enable a lot of these like next generation experiences from agentic tool calling to tool, you know, like all these hosted tools, search, code execution, et cetera. You know, the models are now being spun up in containers and sort of have an agent harness and all that stuff.

39:33So the scaffolding is like oftentimes a couple of steps ahead of like where the act, what is like baked directly into the model. And then what ends up happening is like the model eats that scaffolding and it becomes part of like the native model system. And there's still value in having sort of the external scaffolding in certain cases and like search maybe is an example of this like there's lots of folks who use different search providers and there's different like use cases that you want and so like sure maybe the model can natively use search but you also want something else code execution another example of that um but it does feel like like maybe the agent harness is like the quintessential example of this right now where like everyone's like ah we got to go build a harness and like the harness is where the alpha is and like i think that perhaps won't be true at least in the way that we think of the harness today in 12 months.

40:23I think the models will have sort of just like digested a bunch of that. It'll be upstreamed into the model and the alpha will be somewhere else now. It won't be in sort of trying to spin your own harness because the model just like does it natively. But I thought that the part of the reason why people are building their own harnesses is because if you use a harness from any given model provider, you're locked in. Right. So a lot of the application companies want flexibility, which is why they're building their own harnesses. Yeah. And I think that's part of the scaffolding story is like that starts out perhaps true.

40:50but then as the model capability improves like it becomes less true over time actually i think the model that like you you don't have a generalized model if it can't use another harness and so it is and is important and i i mentioned this uh in another conversation with someone a few weeks ago but we need something like harness bench which is like actually measuring like how good are all these different models at adapting to all the different harnesses i feel like that seems like a reasonable thing we should we should measure as an ecosystem um and i'd be curious to see like what models are actually best.

41:20But I think over time, you expect they'd be able to use every harness, unless you're completely out of distribution, which in that case, you're still going to be completely out of distribution, even if you're using your own harness. So not sure it matters much. Fair enough. What about the application layer? How do you think about where independent companies can have a hope of surviving when the model eats the harness and eats the stuff around it. Yeah. It feels like there's, yeah, it's an interesting story that like both of these things feel true. Both on one hand, I, everywhere I look, I'm like, there's never been more opportunity to go and build something.

41:58At the same time, obviously the models are doing more than they've ever done before. I think there's like, you know, there's that thread of capability overhang, which I think there's a huge amount of alpha in. There's the thread of the model companies are like going after these like very general problems. And there's just like so much value in these like verticalized domains if you have expertise in that domain you sort of like know the customers you know the ecosystem like it's just you can really like run laps around even the best model labs because like focus is the like superpower of startups like if you can focus you can do anything and if you look at all of the companies that are big or doing lots of stuff like there's just not a lot of focus and for some for some reasons like rightfully so because you know maybe i'm i'm overly justifying, you know, Google strategy, but like, we just have a lot of products.

42:47We have a lot of users. We have a lot of different things going on. And so like, we actually can't focus in one domain. We have an obligation to do a bunch of things as a big company. I think that's not true for startups. And so I think like 24 months ago, we were all asking ourselves like, oh, wow, it seems like the opportunity space is shifting. And maybe it's possible one of the outcomes is there's less opportunity for startups in the future. That feels like so far in a way, not what has ended up playing out, which is really positive. If anything, it feels like there's just even more opportunity than there was.

43:18Like now coding has helped you like close the gap on like larger companies that have like established code bases and all this other stuff, because you can just like run way faster and write software quicker. The agentic like primitive is like a new category that you can sort of build products around that like actually in a lot of cases to the conversation about like the risks involved with building, like there's risk involved. And so like, what's your like the risk appetite of different companies is different. And so if you're willing to take more risk in some domains, like you can win a user cohort who's like interested in also taking risk.

43:51There's so much opportunity. Awesome. I'd love to talk about Google DeepMind's culture. And I'm curious, what does it feel like to be inside GDM right now? You know, we had Demis at AI Sense. He was so inspiring. I've heard Sergei's back. You guys have Noam Shazir back. Walk me through what it's like to be at GDM right now. It's incredible. I do try to take it all in because it's like a moment. I try to reflect as much as possible in the chaos of all the things that are happening just because there's so much cool stuff going on. GDM's culture is interesting and maybe three observations. One, back to this thread of focus, we're doing a lot of things and so i think you see sort of uh i think about this a lot like from a portfolio perspective i think we have like one of the strongest portfolios which is really exciting but you do see these moments where like another lab or another company whatever it is will like pull ahead in a certain area where like we under invested just like hadn't been focused enough in that domain um and it's cool to see like the the way we go about trying to like close that gap i very much appreciate it.

45:03I think I've watched the Demis Thinking Game documentary a few times. And you see a lot of details of that original culture and just the way that strikes work and all this stuff, which is actually really similar today. You just get a bunch of smart people together and go solve the problem. And I love that. And it's very cool to be a part of. Another one is this, I think you see the culture permeate from who the leaders are. And maybe this isn't a perfect characterization of the ecosystem, but Demis is a Nobel Prize scientist and the sort of OG of a lot of this stuff. And you feel that in the DeepMind culture.

45:48I think like Sam is like the, you know, maybe one of the world's best businessmen ever. And like you sort of see that in the open AI culture and the way that they go about the world. I don't have a strong sense of who Dario is, but like I think Anthropic is a very interesting place. And you sort of, at least as an external observer, like he seems like an interesting guy and so somewhat esoteric. And so it seems like they're sort of like that in the DNA and the culture of the company. you know the other labs are interesting um but i like this like very scientific approach to the world um and the way that like demis looks at like the reason he's doing this and the reason they started this mission was like literally to like solve disease and all these things and it's like so easy to get and again i'm always trying to pull myself out of the moment but like it's so easy to get lost in this like competitive race of who's pushing a number higher on sweet bench or whatever it is, it's very easy to lose sight of like the reason we're doing that is so that like we can solve problems that humans actually have.

46:50And there's a my favorite quote from all of Silicon Valley is something like, you know, we can't let other people make the world a better place more than we can, which is like what this moment feels like. And the Gavin Belson quote, and I think about that all the time. And it's like we're all fighting over who can make the world better, more than the other person, which just like when you frame it like that, it seems really goofy to me. And so it's very much not zero sum. And I think that's like a way of looking at the world. I think the last thing about DeepMind's culture is like we're very, it's sort of the engine room of Google, which I think is like literally the Twitter bio now of the DeepMind Twitter account, which I love.

47:30You man the DeepMind Twitter account? I don't want any responsibility manning other people's accounts online too much responsibility to do that. But it does feel like that too. So it's like on one hand, you have sort of like the deep rooted lab culture. On the other hand, you have sort of like all of these partners across the Google ecosystem that we're collaborating with everybody from Android that we talked about earlier to Google Cloud to Gmail to Workspace, et cetera, et cetera. And so it's an interesting blend of like, I think there's lots of research work happening, but like there's tons of applied work that's happening to like actually like work with some of the like the forefront customers like deploying Gemini to billion user products is a problem that like only two companies in the world have and we have 13 of those products and like we the you know Google goes through this all the time now and it's such an interesting place to like see that happen and see the innovation that takes place in order to make that actually possible and I feel like it's You can only do that inside of Google, which is really cool.

48:30Beautifully said. Did they give them a lot of heartburn when you joined and were tweeting a lot? That's a good question. Did you have to get sign off from comms? I'm very, one of the silver linings to my Google experience has been just like how great that group of like folks across marketing comms are to work with. And I think like, you know, their job is protect Google, make sure we tell the right story and make sure a bunch of bad things don't happen. And so I have a ton of appreciation and partnership with them. But it's been an incredible experience to like be able to go try to tell the story that resonates with developers in a way that feels authentic and not have a huge amount of, you know, I don't have to get my tweets approved all the time and all this stuff.

49:18like it's a very very positive culture and i think hopefully i am always trying to walk the line of uh not not burning uh the the trust and goodwill that that i've accumulated with those folks but it's been super positive because ultimately i think it's like it's really hard for google to tell this like authentic story it's just there's just like it's a big company there's a lot of people there's a lot of opinions and so you take the like magic of google and you water it down through like a lot of people and a lot of process. And you actually, you miss the beautiful story, which is like Google's doing the most interesting technology in the world and like helping our users with some of the hardest problems in the world.

49:54And it feels it's a privilege to like get to help tell that story. So it's a lot of fun. I enjoy it. I love what you're doing. I love what Josh is doing. I think you guys have put a really kind of sincere human touch on, as you put it, the most important problem of our time. So thank you. Well, wonderful, Logan. Thank you so much for joining me today. This is a very far-ranging conversation, everything from agents and coding to world models and harnesses and GDM culture and lots of nuggets here. Thank you for joining me today. This is a ton of fun. Thank you for having me. And I'm excited to see what the folks cook up of where we've been sitting this whole time, maybe in front of us.

50:31Maybe there'll be a dog. A dog something. You can make my dog dreams come true. I love it. Awesome. Thanks, Logan. Of course.

50:45Thank you.