Building Gemini's Coding Capabilities

0:00Let's talk about all the stuff that got us to the point of having really great coding models. Everyone on the team is always so curious and genuinely just wants to make the model better. You can also just, you know, see so many possibilities as the next frontier of what's going to become possible. The external reaction has been, we have one of the best coding models, if not the best coding model in the world right now. People should have access to this incredible intelligence and be able to do more. Team Gemini, great note to end on.

0:53Hey everyone, welcome back to Release Notes. Today we're talking with Connie Fan, who's the product lead for Gemini's coding capabilities, and Danny Tarlow, who's the research lead for Gemini's coding capabilities. Thanks for being here and hanging out, guys. Thanks for having us. Yeah, I love that talk. I'm excited. So the sort of context for this conversation is we have great coding models. And sort of the thread of the conversation is let's talk about all the stuff that got us to the point of having really great coding models. We released the sort of Gemini 2.5 Pro variant late March, which folks were really excited about.

1:26Maybe you can both sort of just walk us through the sort of year leading up to the code sort of breakthrough moment for Gemini, if you will, and what it took to get there and sort of, yeah, what was happening behind the scenes to get us to the place where I think the external reaction has been, we have one of the best coding models, if not the best coding model in the world right now. What was the bit that changed so that we could make all the progress that we've made? Yeah, I think if you were to say there were three goals, it would have been competitive programming number one and folks like open ai did an amazing job of setting early evals like human eval around um trying to measure these models coding capabilities at all but but um someone who's really good at leak code isn't necessarily someone who's like the strongest teammate right so so that was a bit divergent from what um folks ultimately needed not to anthropomorphize the models too much, but competitive programming was one.

2:30LMSys really set the tone a lot as well. And that's not really what you're actually doing day to day. And then if there was a third that was more productive, it was still sort of limited in a code completion space, not everything that we know the models can now do and everything we still hope the models can do in the near future. So the first two being, to your point, not really reflective of what developers are actually doing. And the third one being not quite ambitious enough for them would be how to characterize the goals. Danny? Yeah, I think that's a good answer. I think maybe you could expand that out and just say, I think one of the things we looked at when we first started a year or so ago on this was just like, do we have the fundamentals right in what we're doing?

3:22From a code perspective or just generically? Just from a model building perspective. Okay. From a, there's many, many parts, both within people who are looking at code and people who are sort of tangentially working on code or people who are just building general model capabilities across Gemini. And really, any individual capability is about the coming together of all of these things. And so when I say fundamentals, I mean what Connie is saying in terms of, do we have the right focus and when we say we want to make the model better at coding does that mean the same thing to everybody working across all of these different things I think that's really important that you know in order to make sure we all are pushing in the same direction and pushing things in the right way but I also just mean in terms of you know the model itself and in terms of you know if we have an idea about hey you know we observe that the model has a shortcoming in this space and then we think we have good ideas about how to address that or which part of the process needs to be fixed in order to do that i think i would say if the fundamentals are not right then you go and you try to make this change but then you don't get the outcome that you want in terms of the model's behaviors and the reasons for that there's many many reasons for that and i think those are kind of like you know the very first things to be looking at uh understanding you know Why aren't we getting the performance that we want in this space and sort of tracing that back as a first step?

4:52Can we opine on the point for a second of like why the competitive programming bit doesn't generalize? Because my intuition and to use like the human analogy, like I feel like the folks who are like, you know, world class competitive programmers like tend to be like sought after as software engineers and folks in that position from like working at great companies and like going and doing engineering work there. I've never done competitive programming. I actually don't really have a good sense of what the tasks look like, but why does that not generalize well from a thing to hill climb from a model perspective?

5:23I don't want to be too negative about it. I think it is a very interesting, challenging problem that really stresses the model's coding capabilities. It's something important. It's something we do need our models to be able to do. I think it's just when you think about what it means for a model to be good at coding, one example is working inside of the context of a large repository. That's something that doesn't show up at all in a competitive programming environment. You're just working in a very self-contained thing. Here's your problem description, start from scratch, build me a relatively short solution.

6:00Compare that to what a software developer is doing on a a day-to-day basis is like, there's this bug report coming in about some crash that I don't know. It could be spread across, you know, a hundred different locations in the code base, and I need to go and figure out. So it's just the sort of the set of capabilities that the model needs is much larger than just what's encapsulated in competitive programming. All right. So we talked a little bit about sort of the history of the Gemini coding effort, but would love to know, like from today's perspective, like what are the things we're focus on to like, you know, the, the main ingredients of what makes a great coding model.

6:37And we don't have to go into all the technical detail, but like from a North star, is it like we're focused on like completion tasks, but then also like entire code-based tasks or like how does all of that break down? Yeah. I mean, I think it breaks down always to data methodology. Um, And so they're shifting to meet each other accordingly. But on the data side, I think to Danny's point that repo context is so important. That's probably the biggest shift that you'll see. Is it just completion anymore? No. I think we specifically care about making multi-file edits, bigger changes than just here are umpteen lines for you.

7:31Here is what you really wanted to do if you had an hour at a time to sit down and do it yourself in the context of the code base that you're working out of. Yeah, that makes sense. Yeah, I like to take inspiration from just where people are using our models and where they're really getting joy and value out of that. And then try to think carefully about it and say, you know, where is this going to be in three months? Where is this going to be in six months? And sort of like try to be ahead of the curve in that. And so, you know, I think there's a lot of really interesting stuff happening in the vibe coding space of, you know, people are doing really cool stuff already with the current generation of models, but you can also just, you know, see so many possibilities as the next frontier of what's going to become possible there and thinking about that.

8:23And then also thinking about the, you know, the more professional developer and like Connie is saying about the repo context and all of the considerations that come up, not just in the code editing part of the developer lifecycle, but, you know, where do people really spend their time and struggle and where are the difficult challenges that come up in the entire software development process? I think that's the hardest part of our job. Maybe it's not the hardest part of Danny's job, because he does all the smart research stuff too. But like getting the timing right and like making the right bets is constantly a question for us and um you know we were to your point that 2.5 pro is being really well received we were looking back at the like hopes and dreams we'd written down a year ago six months ago and like I was patting ourselves on the back anyway and you're like you know what yeah yeah that that was good but I think we're going through a cycle right now where we're like, oh man, we really hope we're making the right bets and we'll find out in six months.

9:27I feel like the intuition of building things to resemble the developer workflows, just to hit on this point again, it is really interesting. I have two questions related to this. One, why did we not do that to begin with? And was it just like, if you look at the previous era of coding models, very completion heavy, Was it just that the models were like not good enough and like the next token prediction paradigm just like stopped working as you tried to do like larger edits? Or was it like an architectural thing that the models were like, all the previous code models were like very diff slash completion focused?

10:05I think my take is there just wasn't enough mindshare on it. I think it's something that, you know, I think there had been a lot of work in various pockets of the research world and whatnot on these kinds of problems. But I think they weren't getting the same level of mainstream attention that they are now, maybe because the model capabilities weren't strong enough. And so, you know, crossing that threshold of I really trust this to be more involved in more parts of the software development process might not have been there. But I think if we had a time machine and we'd go back in time for years, then yeah, I think we could accelerate the attention that was placed on these.

10:49yeah i i i mean dan even knows in which part of an eyelid but it feels like the models were capable if you put the effort in but you need to suspend disbelief that someone would even like want to just come with a couple natural language sentences and walk away with a web app like yeah that wasn't clear i think to folks that that was going to be such a dominant use case and i think Like for us now, it's like this idea of vibe coding, like it doesn't even have to stop at web apps. Like there's so many things as long as it's possible for the person to validate the final artifact and feel confidence in the output that can be vibed.

11:34So, yeah, I think we just want to keep keep pushing along that dimension. Yeah. One other random comment, which is it's also just interesting to see the disconnect between like how like people who are learning to code and this like this thread of the like whole repo task and like being in a large code base. Like if you think about like what you're taught when you learn to code often, it's like not like you're not in university and they're not dropping you into like a mono repo and being like, okay, now go solve this bugs. It's like all these like very contrived discrete tasks, which actually, interestingly, I feel like map to some of the early model capabilities.

12:12And then as soon as you graduate school, if you go through a traditional computer science degree program, your most first jobs, you're going to be in some large code base that you're not building something zero to one. And it's very interesting that the model development process actually kind of tracks that human learning process of early code stuff. But I'm also to go back to the point that you were both making about vibe coding stuff um how has that become something that we're thinking about from a code perspective is it like materially different than you know the north star from six months ago pre-andre carpathy tweeting about vibe coding um was that something that we were already thinking about yeah i mean i think at least in In my mind, I've always had this division of the kinds of users and audiences that we care about and that we want to make sure that we're building the right capabilities for.

13:11One of them is the more sophisticated professional developer working in the enterprise kind of environment like you're talking about. And then the other is this idea of what we're seeing with VIVE coding, but it's been around for longer than that, of the tools that we're building are going to help people who are not professional programmers or who have a little bit of programming experience, you know, expanding out what they're going to be able to do with programming. I think we've seen that as an important slice of what we're trying to do for a while. I think, you know, the recent expansion and excitement is, you know, maybe sort of teaching us some new things or giving us some new inspiration of what you might be able to do in that space, though.

13:58nice totally yeah i mean like maybe we didn't have a pithy thing to call it so thank you andre for that um but i think like the greatest hopes and dreams are always to uh take this skill set that is like very concentrated in this geography we're currently sitting in um bay area and like just empower people who don't have it to do basic things for themselves like we'll always need professional developers and that's not by any means a substitute but um people should have access to this incredible intelligence and be able to do more and and i think we've always cared about that yeah as if uh andre needed yet another hit uh but yeah he he got the vibe coding it.

14:49I'm interested to talk about sort of the connection between how we think about the coding pillar within Gemini's role in bringing new capabilities to other areas inside of the Gemini ecosystem, if you will. And maybe, I don't have a good sense of this. I'm curious from both of you, instruction following as an example, does the, I don't even know off the top of my head if we have like an IF pillar or something like that. But like, are they looking to like, you all from a code perspective to like help upstream and like hill climb on the general capability? Or like, from our perspective, is it like, you know, the thing we care about is code and these like very discrete specific tasks.

15:32And like, that will generally hopefully translate to these like other capabilities as well. But like, it's not, and you know, maybe insert multimodal as another example, like, is it how connected are all these different capabilities from like a model perspective? Yeah, I mean, I think it's extremely interconnected is the short answer in terms of, you know, I think like, you know, we're talking about the model capabilities getting better at code. This is undeniable of what's happened. But, you know, like how did that happen and which pieces of work across all of Gemini helped to make that happen is definitely a sort of a team effort spread across the entire set of sort of both capability focused and more kind of like horizontal focused efforts within the project.

16:28And so to the extent though of like how much does improvement in coding contribute back to other parts of capabilities, I think there's also like many ways and many degrees to which that happens. I think in some ways, you know, there are some capabilities where, you know, maybe you have a problem and you weren't thinking about representing it as a coding problem, or like, you know, you want to solve a word problem to help somebody with homework. And maybe it's useful to convert that first into code that says, you know, here I understand what, you know, who has how many apples and what gets multiplied together.

17:07And then you can actually, you know, maybe execute some code in order to do that, or reasoning in code space is a natural way to solve problems that are outside of the coding sphere. I think that's a very interesting way. How much does that happen? Because just to opine on this point for a second, it feels like there's a, and I don't know what, I don't know if we have analysis of like what percentage of queries, like could you, user questions or prompts, could you actually just like write a bunch of code under the hood to like solve whatever the underlying problem they have is? But have we thought about like I feel like today it's like somewhat reserved where like if you're like oh do this thing the model will it's probably not going to generate code unless you like really ask it to but like do we from like a north star perspective is there a world where we just like are generating code for every query to like build you some bespoke software to like solve whatever the thing is that you're interested in it's hard for me to imagine that for you know write me a poem about something something Like a polar generator app?

18:05I don't know. That's interesting. Yeah. Yeah. I mean, there's obvious exceptions, but I do feel like code is everything in the context of LLMs. And a couple examples that I like to hark on with Danny, so sorry to hark on them again. Like, I think if you look at our query stream today, people recently in the US would have asked a lot of like give me tips on how to do taxes like help me with this general concept like make sense of this like purely natural language based concept for me but at the core of it if you were to take their holistic problem um and you were to write them a bare bones tax calculation um it'd be hard to even call it software but just solve their problem for them Like that is, I think, totally doable via generating a lot of code to your point of maybe the prompt didn't specifically ask for like, write me a mini QuickBooks or something like that, you know?

19:09But that's the spirit of what they're really asking for when you look across all their prompts. I think that's a future we can probably orient around. and we should build that i feel like this would be a fun experiment to like what happens if you just like force enable the model to like always build you software to answer your query and like what is how do users feel about that i feel it could be like really really like and i feel like that seems somewhat outside the state space of like what developers would be doing with this like i feel like it's not a developer workflow it's like an end user workflow but i feel like that'd still be super interesting totally um totally and and like i think even beyond that of moving beyond like an individual person's needs and an individual developer's needs.

19:55My favorite pseudo eval recently that I liked to hark on with Danny was, I saw this post online of like someone who's not really into AI being like, if AI is so smart, why can't it bring the cost of eggs down? And I was like, why can't it bring the cost of eggs down? You know, like, theoretically, this is a lot of publicly accessible, but hard to reach data that you can write, just write some code under the hood for, to your point, and solve a problem that could meaningfully help a lot of people. That's not a world we're in yet. And that's not the capabilities we have. But it sounds like a really tough email.

20:36It's like, it's like, how much new Gemini model, how much did you lower their price of eggs how much did you help people writ large how much did you just help society writ large that's the eval that is interesting to think about um yeah i feel like that's that's like a great we could have a whole conversation about how hard the eval story is and i also think like i'm curious if there's any threads like maybe this is the thread on this around just like how um like future north stars for coding from an eval perspective like does today's set of evals like capture how we think models are going to develop or like i've seen some of the other ones in the ecosystem where i don't fully grok what they're doing but it's like how much economic value do you create as a developer doing these tasks like how much money could you make in these contexts or like an agentic system like how do we think about that from a eval perspective yeah i mean the way i like to think about it is we want to understand and predict and make bets about where the real world value is going to be.

21:38We want to make sure we're not kind of like over optimizing for that. I think one of the things that we want to do with how we steer the model capabilities is really make sure we're tackling you know the hardest most crucial bits of the capabilities around this thing so we don't want to just like you know spin off some quick wins because if we got really good at x then we could make a quick buck i think it's we want to stay focused on you know what are the core fundamental challenges in this space that are useful for sort of the real world value and then let's you know try to to push the model's capabilities as directly as possible in those directions and so then like what is the eval that you need to go alongside that um i think you know there's always a range of of put something out there into an A-B test, see what happens in the real world.

22:33That's always going to be the most reflective representative thing. But then we always try to make different trade-offs in our evaluations of things that are easier to measure. You don't have to ship your model to a new startup and have them run for a year building on top of it and see how successful they are. That's not really practical. And so, you know, we try to be pragmatic about what proxies we can find. Yeah. How difficult of a problem is that? And I ask this through the lens of I feel like every coding surface now as sort of like the agentic harnesses and like what they're doing specifically changes or like maybe they don't have any agentic harnesses or whatever it is, is sort of using code models in a different way.

23:17And I think even if you look at the spectrum of like Google products of like how folks use some of our internal coding stuff, how folks use like Colab as an example or Project IDX or Project Firebase Studio, like there's a huge spectrum of different of those different coding use cases. How like I feel like there's a lot of capabilities where like maybe they're not being used that like chat, chat is chat. And like, you know, there's not all these like different interesting form factors of how people are doing that, at least right now. How much is that like a challenge from making the models better at coding so that it generalizes across all the different ways that people are using code models when I feel like there's actually seems like a pretty reasonable distribution?

23:59I think that's the main challenge of what we do is, you know, we are trying to build capabilities that work for all of the use cases simultaneously. And so we need to figure out, you know, what are the things that we can do that will generally help the model's capabilities across all of these different sort of like real world use cases. And that I would say is really the difficult part in all of this. So tying to like this broader thread about like all the breadth of use cases and the breadth of feedback that we're getting from a coding perspective, I think one of the coolest things about Google is we have like literally 100 ,000 plus engineers across like every, you know, language and skill set and different stack, et cetera, et cetera.

24:50How does that help from a code quality standpoint? Yeah. Historically, we haven't leaned on this enough, but we are starting to, and it's amazing. um a hundred thousand of the smartest most brilliant sometimes most opinionated engineers in the world and and to danny's point that we are still lacking evals and we don't want to pigeonhole ourselves into uh over rotating around any one thing to have this broad exposure for sort of like a known gap in coding evals right now of nuanced tastes of professional developers um getting that live feedback is incredible and there's gradients of it with the baby tests but even just going back to the idea of vibe evals like um a vibe eval that i want to set i we have a couple of vibe evals but like um jeff dean truly one of the best engineers to have ever engineered um and he's so nice about everything but what's the day we get a gemini output in front of him And he's not just like, oh, good.

25:58But like, wow, that was incredible. And that made me more effective today. Or you had Emma on the podcast a little while back. And when he's like opinionated, Emma's opinionated. But like, let's like, like, let's pretend he's on call and something's on fire and it's going terribly. And he like needs to like turn it around. When is he going to like actually trust that Gemini isn't going to mess it up? And like in this critical moment where he's feeling the stress, like he would actually prefer to have Gemini in the loop than not to have it in the loop. Like these individuals who really represent like, OK, this is a new tier of capability for the model.

26:38It's incredible to be able to interact with them directly. And of course, so many others as well. And then the whole broad base of Google engineers is it's just so lucky that we can we can talk to them. And we're all one company. Yeah. And Denny, how do we balance that with external developer reception or external developer feedback? Do we see a big divide between what sort of internal Googlers tell us about coding models versus the external developer ecosystem? Or is it pretty similar? I think it's pretty similar. I think there's obviously different segments of users. So you know, like the developers that Connie is talking about here, we probably don't have high bandwidth communication channels to external specific use cases there.

27:26And so you can get sort of different kinds of richer bandwidth signals just directly by talking to somebody and saying hey try this or try that yeah um but in terms of what are the things that matter in terms of the model's capabilities for these use cases i don't think there's anything pulling in different directions here i think it's um you know kind of clear the the set of capabilities that you need you know and there's different for different segments of the population but for the you know analogous external versus internal developer, I think they're pretty similar in terms of what they're looking for.

28:04Yeah. To pull on this thread around the people who are using coding models, how do you think about the AI skeptics of people who aren't that interested in coding models or love writing code by hand? And I feel like, again, through this lens that developers are opinionated for you know in the most positive sense um is it like a you know to win the next you know and number of developers is it like we need to make the models just like better in general or is it like there's a different product experience you think we need to build in order to like showcase to developers that this stuff really works well and um is going to be valuable for them i mean we're very model oriented people here right so at least for me it's um I think it's really motivating to sort of like look at this, to take exactly what Connie said before of, you know, what is it that they're looking for and what is it if they are a skeptic that's holding them back?

29:01And then I think we've seen a few cases of converts recently as well. And then, you know, to understand, okay, this is what changed about the model and sort of, you know, it got to this level of nuance and depth in terms of its understanding at that point it won over the trust for you know this particular slice of use cases I think it's a great sort of like hill climbing metric in some sense on its own is just how do we take the more and more challenging the more and more skeptical people and what is it that we need the model to be able to do in order to win them over I think we enjoy the existence of skeptics they give us something to orient around and um yeah we need to make the model better but not just broadly better like they're likely skeptics because the model is amazing at the things that they genuinely need and care about and so we it's it's our job to figure out what that is and how to do it and and for most of those folks does it tend to be or just like generally for classes of areas where the model doesn't perform well does it like is it um pocketed into like different use cases or is it like different languages like the models really there's tons of python code in the world the model's really good at python c plus plus is harder because it's there's less code or something like that like how does like do from a programming language and like framework perspective is there yeah what what how do you all think about that i think there will be languages like cobalt that's it's just really hard for us to get data we need cobalt data because i I feel like that's the whole, the like running bit of like the cobalt engineer, because there's, you know, a hundred thousand banks or whatever that are all still running cobalt or something.

30:44Yeah, the white whale. But for the most part, like the languages that you mentioned, I don't think we have a dearth of data, but I think we have not always historically optimized our mixture properly. and back to the idea of figure out what's actually needed right now, what people actually care about, what will they actually use the model for, work backwards from there. That's sort of the name of the game. Yeah, I think my experience with some of these people is everybody has a different kind of challenge in their mind of like, okay, if the model can do this, then I will be impressed. But it seems like everybody has their own.

31:24some are you know very basic things of like I want you to do this very routine thing in my workflow but I want you to do it perfectly other people are like I think Connie's example of everything is on fire am I gonna turn to you in desperation like you know that's another another might be you know just the complexity and sort of how accurate and coherent the sort of the reasoning and the solution is in a fairly complex problem. And so I think, I don't know, my experience has been with when you really talk to these people and you say, okay, what is it that like test this out? We think the model has gotten much better.

32:03Do you believe it? It's interesting to see the different ways that different people go. And it's also hard to predict. And that's another reason for us to sort of keep pushing in generality, right? I don't think we can optimize for these seven people and then Apple also convince, you know, the next seven people as well. I think we really just need to be pushing the sort of the overall model capabilities in a way such that we just get these like bigger and bigger jumps. Yeah. One thing that like I just want to give a shout out to our entire code team about is this like shared mentality that I just felt as Danny was speaking of like if there's negative feedback, I think no one on the team is ever defensive.

32:41Everyone on the team is always so curious and genuinely just wants to make the model better. and sometimes I feel defensive on the team's behalf because I see how hard everyone works and how brilliant these people are. I'm like, what do you mean you're still not happy? But then the researchers themselves get the feedback and it's never anything but like, okay, how do I make this better? And it's just like the most amazing atmosphere to be in and the most incredible team. I love that. I feel like the focus on helping make developers happy is I need to come hang out with you all more and spend time with the team.