Logan Kilpatrick: Building Google Gemini

0:05Hey, everyone, and welcome to Generative Now. I am Michael Magnano. I am a partner at Lightspeed And today we're back with another conversation, this time from the generative NYC stage, where I sat down with Logan Kilpatrick at Google's New York City offices. Logan leads product for the Google AI Studio, where he and his team built the Google Gemini API into one of the best platforms in the world for developers building with AI. Before Google, Logan worked at OpenAI, and he has had a front row seat to the meteoric rise of AI over the past few years. So if you weren't able to attend Generative NYC, you're in luck because we have a full interview here.

0:44Enjoy. Without further ado, please help me in welcoming Logan Kilpatrick, Senior Product Manager and Lead Product for Google AI Studio. Come on up, Logan.

1:01What's up, Logan? Hi. That was honestly the best intro of just an event that I think I've been to in an entire year. Thank you. I'm very humbled. You're doing such a great job. Thanks. Well, thank you for doing this with me. I've been really, really looking forward to this. So like I said, you have had a front row seat to kind of everything AI over the past few years. So we have to ask you about your journey. How did you get here? Maybe talk to us a little bit what came before this. I think you were at NASA at one point. You were at OpenAI. Give us the Logan story. Yeah. Well, one, thank you, everyone, for being here.

1:35Well, I'll tell my story very, very quickly, we can talk about the exciting stuff, which is what y 'all are doing, which is building with AI. I joined Google back in April to work with a bunch of amazing folks who are here in the room to build AI Studio, build the Gemini API. Before that, let developer relations at OpenAI. I joined at the end of 2022, and it was a small startup. Much less conviction at that point that it was going to turn into what it did. I'll give the very quick story, which is I had a job offer at IBM at the time. So this will hopefully, you know, humble the story a little bit.

2:08And I genuinely at the time did not know if I should take the IBM offer or the OpenAI offer. The IBM offer was cool to do something that I was excited about. But it was just like much less clear at that point. I think unless you were in some of the circles that like everything was about to explode. Before that was at a startup doing machine learning and deep learning for digital pathology called PathAI. and before that was a machine learning engineer at Apple. So I started my career as a technical IC and have become a product manager over time. Awesome. What's it been like being at Google after OpenAI?

2:42I'm guessing two very different companies. Yeah, you know, it's the interesting experience for me personally was like I joined OpenAI as a startup and it truly felt like a startup and there was 200 people. And I think it became not a startup over the course of just a year. And I think coming back to Google has actually felt like a startup. Like the Gemini stuff happening inside of Google, despite it being a massive company with huge scale, really does feel like a startup. If you have agency and you are excited to do something, you can actually go and build that thing or ship that feature for developers or for customers.

3:18So it's felt wonderful from that perspective. And I think we have a ton of work that we have to do still. but it's generally we're trending in the positive direction. Maybe before we get into some of that work and what you're trying to accomplish, help us understand what AI looks like inside of Google. Obviously, there's Gemini, there's the AI Studio, there's DeepMind. Help us better decipher what we know from the outside. Yeah, and there's a bunch of folks here from Google DeepMind. So hopefully you'll find the people at Google DeepMind. I don't want to call them out. But so Google D-Mine does all the hard work of sort of building the generative AI models that sort of power all the internal features at Google, as well as a bunch of the developer APIs.

4:02And then there's a whole bunch of different teams at Google who are sort of commercializing, building internal products. And that's everyone from ads to YouTube to search, building AI into the actual products. And then the Google Cloud team takes a lot of the models and puts it in the hands of builders. So if you're sort of an enterprise customer, you might be using something called Vertex AI to get access to Gemini models. If you're a sort of long tail developer just getting started building startup founder who just wants like the fastest possible solution to build the Gemini, you might use Google AI Studio, the Gemini developer API.

4:39And I think that sort of symbiosis between both building first party products with the models and developing models and putting them into the hands of of external developers is a great competitive advantage for us because we feel the problems that builders feel. It's not like Kat, who's here somewhere, and I work on AI Studio. And AI Studio is actually just a first party consumer product sitting on top of the models. And we live and breathe the rate limit problems, the quota problems, the model hallucination problems, the same as anyone does. And I think it's incredibly helpful for us from that perspective.

5:14Yeah, it's got to be really interesting doing this at Google where, like you said, you have this advantage where you can distribute this stuff to millions of developers with the flip of a switch. At the same time, there's probably also disadvantages. I mean, being such a big company like Google, you can't just throw some random model out into the ether without, like you said, really making sure that hallucination is solved and other maybe trust and safety issues are solved. How do you balance some of those challenges with some of the benefits? Yeah, I think the DeepMind team does a good job of sort of having the infrastructure to know what is actually the bar to get this model out the door.

5:53So some of that is abstracted away from us. By the time the model gets to us, it's like, OK, we actually have a reasonable amount of conviction that we want to put this thing out into the world. There is a long tail of other stuff. Cloud has its own set of more customer-specific evals because the model might be safe at the sort of core level and might be useful at the core level. But like on the very specific things that we know our customers care about it might not be the best model So there is that like extra level of check as well at the sort of Google cloud level Yeah, and give us a sense for AI studio Like what is what does that encompass in terms of products which you're leading like what are what are the different?

6:28Products that roll up into that. Yeah, so Google AI studio is the conduit for developers to get into the the sort of Gemini API So success for AI studio looks like you come in you try the model you realize hey the Gemini models are actually pretty good Could they have long contacts, native multimodal, whatever else you're excited about? And ultimately, I want to build something with them. So it's not focused on being a true consumer product. It's really focused on get you to that wow moment building with AI, and then ultimately click a code, get a Gemini API key, and go and build the next company, the next startup that's actually going to provide value for end users.

7:03Got it. Got it. And what does a team building for that look like? I mean product managers, designers, engineers, like how do you even like think about and execute on some of those problems? Yeah, that's another good question. I think this is one of the things that is a benefit of being inside of Google is like there is someone inside of Google and like our amazing venture folks who help put this event together and Jason and Alex and a bunch of other people is a good example of like the leveraging the scale of Google to like go and do a lot of these things. So our team doesn't have, you know, like a model quality specific function.

7:39Like we're fortunate enough that there's other teams in Google that do a lot of that stuff. So it really is like, I don't know if anyone, I'm sure someone inside of Google has said externally that like Gemini really is like this massive cross-functional Google process to make happen. But it's true because of like how much work happens to get models out the door. It's not just our team. And we get to sort of stand in the spotlight in some cases because we have the externalization surface. But there's a ton of teams doing work to make all this happen. I've heard a little bit about like there are these sort of like three different main components of the Gemini ecosystem.

8:13Gemini, Vertex, AI, and is it Gemma or Gemma? Gemma. Like what are those three things and sort of how do we differentiate them? Yeah, the two main sort of model classes, the Gemini models is our sort of set of proprietary frontier models. And then there's also a set of open models called Gemma. And the advantage for Gemma is you can actually own the weights of the model. You can go and put them onto a server somewhere, and then the apocalypse can happen, and you'll still be hanging out with your Gemma model weights doing AI stuff, and you don't need to worry about anyone else hosting the models for you.

8:48There's generally this lag that happens between the frontier capabilities and what ends up in the Gemma models. It is based on the same research, a lot of the same core technology, but it doesn't have long context. for example, the Gemma models aren't natively multimodal. They're really good in their class of text-in, text-out, open-source LLMs, but it's not super competitive yet with the main frontier capabilities. Got it. One thing I was thinking about is it seems like there's a lot of talk about where the value in AI is going to accrue. Obviously, there's a lot of talk about the foundation model layer, and there are several big companies like Google, like OpenAI, which are obviously going to be big winners at that layer.

9:32And then there's also a lot of talk right now about how the apps layer is wide open. And I have to imagine you, your team, where you're working in the developer ecosystem, you must see amazing app layer stuff being built. I'm curious where you all feel like the opportunity is at the app layer. Yeah, that's a great question. So I think just to echo what you said, I do think there's an immense amount of value to be created at the app layer. Like if you just look at like one simple proxy for this is if you look at the cost of LLMs over time, like basically going down to zero. The consumer willingness to pay if you sort of graph it on the same exact, you know, chart is not going down to zero.

10:14Like people are wowed by AI. Obviously, there's like, you know, millions and millions of people who are willing to pay$20 a month for insert whatever, you know, AI subscription you're interested in. People are really excited and it's creating all of this value. And the cool thing is like founders and people building stuff actually are the ones who get to accrue that value Their companies are the ones who get to accrue that value and like the cost continuing to go down to zero is is great for for people who are building stuff That said I think it's there's a lot of you know There's a lot of things that people are building that are you know not long-term Differentiated aren't are not actually different you're talking about like just rappers around models Yeah, and I think like you have in a lot of cases is starting as a wrapper makes perfect sense.

10:56But if you don't like really aggressively figure out a way to get out of that position quickly, if there's not a clear like, you know, taking off point as part of like the baked in strategy, like it's gonna be tough. Because like the model crank is gonna keep turning. You know, there's a bunch of obvious things that all the main like AI consumer applications don't do today that they will do over the course of the next two years. Like those teams are fighting really hard to build great products as well, just like you all are. I think the thing that has gotten me most excited recently at the application layer is all these like differentiated Actual ways of interacting with AI and I think like notebook LM is like the most recent reminder to me and this this came from Google, but there's a lot of I don't think there's actually that much Interesting differentiated stuff that the notebook LM team did other than having conviction that like there's a different way that you could be interacting with AI content and I think there's a whole lot of other things like that that just take time for people to experiment with but I would I would push in every conversation I have with people building out the application layer push on like it's not chat that is going to create all the value it's maybe not even like voice like I think there's a lot of people who are like voices chat it's voice but maybe it's not even voice yeah maybe it's not even voice I think I think you like keep pushing on what that that interaction paradigm might be and there's a lot of value to be accrued and having a differentiated perspective.

12:19Do you have, either through what you're seeing or just a personal intuition around the areas or the categories or sectors within the app layer that you're most excited by, whether it's consumer or enterprise or healthcare, whatever it might be, where in the app layer are you really excited about the future? Yeah, it still feels like consumer is early. I think if you grab a random person off the street and not San Francisco. They don't know or care about AI. I think the New York crowd cares about AI. I'm glad you all are here and building stuff. But most people don't care because it is hard. The use cases today where the most value are being created are the enterprise use cases.

12:58There's just so much. If I'm 50 % better at coding, many hundreds of thousands of potential dollars for some large company to be accrued by me being 50 % better at coding. So I think that will, the sort of changing point for this is creating a bunch of value where it's like not putting the technology first. And this has been my big qualm with people building agent stuff, which is like everyone thinks like, ah, we have to call our product, if it's consumer agent stuff, we have to call it an agent platform because like that's what's getting the people at Lightspeed excited about it or whatever it is.

13:34But I think like really the value is abstract away all the agent stuff. Like the idea of agents is great and like, you know, AI should do stuff for people. I think that's pretty obvious and people can agree to that. But don't sort of force the long tail of consumers to like have to care about the technology because they don't. That's the reality. Like they're not interested in agents. They're interested in their life being better, easier. Yeah. They didn't care about the GPS. They cared about Uber, right? Exactly. They didn't care about the camera. They cared about Instagram. Yep. Yeah. So it's about the product.

14:02It is. Yeah. That's super interesting. thing. Speaking of like where value will accrue, and you know, we talked about some of these sort of big company incumbent advantages. One of the things I've certainly read about in the press a bunch over the past week or so is this notion of sort of reaching the upper limits of scaling. You're around a lot of this stuff. I can tell by your tweets, you're a sort of big thinker when it comes to AI. Like, where's your head at on the scaling limit question? Are we starting to reach the upper limits of what these transformers can do? And then maybe we can get into like, if so, why are we reaching these upper limits?

14:40Yeah, I think it's an interesting question. I think there's definitely people probably in this room who are better, who are closer to the sort of metal on being able to answer this. My perspective is you have to sort of earn the scaling laws. Like the scaling laws is not like a law of nature. It doesn't just happen. You have to literally earn it. And that means innovate and do a bunch of stuff that enables the technology. It's the same thing with Moore's Law. Moore's Law doesn't just happen because someone wrote it down on a piece of paper. It happens because there's thousands of engineers at, insert whatever hardware company, sort of making that the reality.

15:19And I think people are very in the moment of capturing, is scaling continuing to work? Like today, maybe, and I'm not saying this is the case, but today maybe scaling's not working. Tomorrow there's innovation that sort of enables it to happen, and then all of a sudden, you know, the scaling law continues back up again. Right, so like we're going to hit some upper limit, or we are hitting maybe some upper limit of data, or the capability of the current compute clusters. Like something is capping us? What? I don't think so. I think we're still sort of pushing up is my perspective. But I think even if, again, even if whatever the technique of today isn't working, like tomorrow it will.

16:05There's going to be algorithmic scaling. Like we're going to find new ways to - Or the next 100K H100 cluster comes online, whatever it is. You know, meta throws another 100K H100s at it. And then like things start to work again. Yeah. Yeah, I'm glad you brought that up. Like, that is a very, very interesting moat for big companies like Google, Alphabet, Meta. Like, how important do you think size of compute cluster is and how long does that advantage last? I think it's tough. Like, training LLMs is expensive and it's tough. I think there are a lot of companies that have proved to be solving very domain-specific problems, like training their own models.

16:47and I was talking to the founder of a company called Cartwheel, and the founder was actually at OpenAI before, and they're training domain-specific vision or motion models to be able to do animation and stuff like that. And he was showing some of the relative to the foundation model doing the capability and the domain-specific model they trained. Because they're only trying to solve that problem, it is hundreds of thousands of times more compute efficient to actually do inference for those tasks. Being specific. And it doesn't need to have the weights of writing poetry and doing all this other random long tail use cases.

17:24It literally just needs to be good at this motion task. And I think there's a future in that. And you don't need to have crazy amounts of money or crazy amounts of compute. You can tackle those problems as long as you're not trying to build the generalist AI agent. I think that problem will sort of go to whoever's willing to throw the most compute data, money, algorithmic improvements at the problem. So do you think, is Cartwheel, by the way, super, super cool product and model, they actually demoed at one of these events. Oh, no way. Yeah, yeah. Do you think that that is an approach we will see more and more teams go after, like highly specialized, small models?

18:03Is that an opportunity to not sort of get run over by these bigger models? That's my instinct. And I think the proof will be in the pudding, because I'm not sure how many of those companies have actually gotten to legitimate scale to know that this is the case. But I think the same is true for just generally people solving any problem, which is, if you really focus on whatever the vertical is, whatever the niche is, you're not competing against Google and the long tail of these big companies. There's very little competition in those very domain-specific ecosystems, and I think Cartwheel's a good example of that.

18:36What about talent? I mean, one of the things we hear and we talk a lot about when we talk about AI is talent, right? These amazing people, researchers, builders, finding this next algorithmic breakthrough. And we're starting to see some signs that talent really may be that important and that much of a moat. Obviously, there was a pretty public and notable deal between Google and Character AI recently with Noam Shazir and some of the founding team coming back over here. how do you think about sort of people and talent as a moat? And again, like how long is that going to last? Yeah, that's another, I mean, I think it matters a ton.

19:16Like at the end of the day, like if the talent is the difference between you winning and not winning, I think the same, it's true at Google, it's true for every startup that's building. I think more so than I think anything else, I, my personal belief is like you, you can win with having talent, even if you have a worse idea, a worse, you know, et cetera, et cetera. And I'm actually curious for you backing companies like at the early stage, is it a same, a similar? Yeah, I mean, we think talent is absolutely critical. And, you know, as a new-ish VC and former product person, CEO, like when I first came into the job, I was very, very focused on ideas and products and still am, of course.

20:01But I think as I'm getting more experience and learning from some amazing people. It's really about the talent. It's about the people for sure. And it's interesting to see the talent flow. I think the challenge for a lot of things that aren't AI right now, as I'm assuming there's at least one founder in here who's building something that's not an AI, it's hard because the center of gravity continues to shift between all these different things. And if you don't have the ability to pull in the best people, it's tough. Yeah. Yeah. Let's get into everyone's favorite topic. What is AGI? Yeah, I think I still align with the version of AGI where it's, you know, the models are able to do most of the things that are economically productive that humans are able to do.

20:50And I don't know if anyone listened to the full five hours of the Dario Lex Friedman podcast. You did, you have too much time on your hands and you should be, you should. It's really good though. It is good. It's good entertainment. But yeah, I think the 2026, 2027 timeline to make that happen seems, seems interesting. Do you buy that? Do you think we're headed there? I think it's tough. I think the models will continue to go up and do so much more. I think the real question is it's like a, to do most of the things that humans are able to do that are economically productive. It's not like a moving bytes around problem.

21:29It's a moving atoms around problem, which is actually just a lot harder to scale. Like, I'm fully bought into the idea that there could be digital versions of like, maybe it's AGI in the sense that it can do all the things that are digitally productive that humans are able to do. But the long tail of making robots that actually do most of the things humans are able to do, like in the physical world, I think is much longer off than 2026 or 2027. Just to clarify what you're saying, so it sounds like what you're saying is like, yes, we need some breakthrough on the model side, but then we need to have a physical embodiment of it and get into robots and, like you said, humanoid robots or whatever, full self-driving.

22:13Is that what you mean? Like it has to make its way out into the physical world? It has to make its way out into the physical world, and you could have AGI today. assuming you had AGI today and the current state of robotics was the same, we're still end of five years away from any large-scale manufacturing run of humanoid robots that are actually going to be able to scale out and have any meaningful impact on the world. So it's like, we can definitely scale up the digital intelligence, but I think actually manifesting intelligence out into the world in a physical form is going to be much, much harder to make happen.

22:48How does Google think about this? Obviously, there are startups out there that they're very publicly stating that their mission is AGI or superintelligence. Is that something that Google talks about and thinks about internally? It's like, hey, we're doing Gemini because we want to achieve artificial superhuman intelligence? Or do you not even really speak in those terms inside of Google? I've heard Demis not to speak on Demis' behalf. I've heard Demis say a bunch of times he wants to make AGI. I think for Google, like Google's mission as a company is, you know, organize the world's information and make it universally accessible.

23:24So I think it's like less, it is much less at the Google level, like, you know, AGI focus. I think DeepMind and I'm sure Demis like probably want to make AGI as sort of the focal point. He's been pushing on that for, you know, 10 years now or something like that. Yeah. Are you looking forward to AGI? Are you an accelerationist or a doomer or somewhere in between? I think I'm looking forward to it in the sense that I think there's a lot of tough problems in life. And I'm excited for AI to help solve a bunch of those tough problems. I think the real practical downside is, and again, back to this narrative of like you can grab random people on the streets of any city besides San Francisco and ask them about AI.

24:04And, you know, if the tool actually does deliver on this promise of everything that everyone says it's going to. The time horizon to educate the world about whatever this new tool is or technology is just very long. And you can even look at ChatGPT as like the most successful version of this. There are billions of people on earth who have never heard of ChatGPT, have no idea how it works, all that stuff. And if the technology is actually that powerful, I think there'll be a lot of downside for the people who aren't sort of in the know that this thing is. think exists or have access to it. And I think that has the chance for accelerate the discrepancy and the delta between people who have access to things and who don't, which I think is a very tangible downside.

24:53And I don't think there's been enough push, in my personal opinion, at a government or world level of actually helping get those types of resources and education into the hands of people who are going to need it. So maybe tying this back to app layer, developer ecosystem, like where it's where it feels like a lot of value is being created and it could be an area that experiences disruption sooner than in other areas as a result of AGI is coding. Right. It's like these things are really good at coding and they're getting better very, very quickly. Like, how do you think about the long term impact of AI on software?

25:33Like, what does software become? Is it all dynamic? Is the God model just like spinning up products for us in real time? And how do you sort of intercept that future with maybe what you're doing like at the developer ecosystem level? Yeah, that's a good question. And it's something that I think about a lot as a weekly active cursor user who I think are closest to this. I think the cursor example is relevant in that I think the path to get to that point is software engineers increasingly having this really powerful tool in their hands. Which I think is like a different version of the world than like AI is going to replace software engineers.