Google DeepMind Developers: How Nano Banana Was Made

24:06And I think as it gets better, it'll be useful for even more things too. I think one of the really interesting things we're seeing across a bunch of modalities, of which image, edit, and generation obviously is one, is like, I think the arenas and benchmarks and everything are awesome. But especially when you have like multi-dimensional things like image and video, it's very hard as all of the models get better and better to condense every quality of a model into like one judgment. So it's like, you know, you're judging, okay, you swap a character into an image and you change the style of the image.

24:43Maybe one did the character swap and consistency much better and the other did the style much better? Like, how do you say which output is better? And it probably comes down to like what the person cares most about and what they want to use it for. Are there like certain, you know, characteristics of the model that you guys value more than other things in like making those trade-offs when deciding which version of the model to deploy or like what to really focus on during training? Yes, there are. One of the things I like about this space is that there is no right answer. So actually there's quite a lot of, I don't know if it's taste, but it's like preference that goes into the models.

25:21And I think you can kind of see the difference in preferences of the different research labs in the models that they release. So like when we're balancing two things, a lot of it comes down to like, oh, well, I don't know, I just like this look better or this feature is more important to us. I'd imagine it's hard for you guys too because you have so many users, right? Like Google, like being in the Gemini app, like everyone in the world can use that versus like many other AI companies just think about like, we're only going for the professional creatives or we're only going for the consumer meat makers.

25:53And like you guys have the unique and exciting but challenging task of like, literally anyone in the world can do this. How do we decide what everyone would want? Yeah. And it is, sometimes we do make these trade-offs. We do have a set of things that are sort of like super high priority that we don't want to regress on, right? So now because character consistency was so awesome and so many people are using it, we don't want our next models to get worse on that dimension, right? So we pay a lot of attention to it. We care a lot about images looking photorealistic when you want photos. And this is important.

26:26One, I think we all prefer that style. Two, you know, for advertising use cases, for example, like a lot of it is kind of photorealistic images of products and people. And so we want to make sure that we can kind of do that. And then And sometimes there are just things that will kind of fall down the wayside. So for this first release, the model's not as good as text rendering as we would like it to be. And that's something that we want to fix in the future. But it was kind of one of those things where we looked at, okay, the model's good at X, Y, Z, not as good at this, but we still think it's okay to release, and it will still be an exciting thing for people to play with.

27:01If you look at the past, right, we had for previous model generations, a lot of things we did with like sidecar models, like ControlNet or something like that, where we basically figured out a way to provide structured data to the model to achieve a particular result. It seems like these newer models that has taken a step back just because they're so incredibly good at just prompting or giving a reference image and picking things up from there. Where will this go long-term? Do you think this will come back to some degree? You know, like, I mean, from the creator's perspective, having, I don't know, open post information so I can get a post exactly right, right?

27:32For multiple characters, this seems very, very tempting, right? Or to rephrase it a little bit, it's like, does the bitter lesson hold here that at the end of the day everything's just one big model and you throw things in? Or is there a little bit of structure we can offer to make this better? I mean, I think that there will be, there will always be users that want control that the model doesn't give you out of the box. But I think we tried to make it so that, because really what an artist wants when they want to do something is they want the intent to be understood. And I think that these AI models are getting better at understanding the intent of users.

28:06So often when you ask text queries now, the model gets what you're going for. So in that sense, I think we can get pretty far with understanding the intent of our users. And maybe some of that is personalization, like we need to know information about what you're trying to do or what you've done in the past. But I think once you can understand the intent, then you can generally do the type of edit. Is this a very structure-preserving edit, or is this a freeform? We can learn these kinds of effects, I think. But still, of course, there's one person who's going to really care about every pixel, and this thing needs to be slightly to the left and a little bit more blue and those people will use existing tools to do that.

28:43I think it's like, you know, I want an image with 26 people spelling out every letter of the alphabet or something. That's the thing where I think we're still quite a bit away from getting that right in the first try. On the other hand, with pose information, potentially. But then the question, I guess, is like, do you really want to be the one who's extracting the pose and providing that as information? That's a very good question. Or do you just want to provide some reference image and say, like, this is actually what I want. Like, muzzle, muzzle, muzzle, muzzle, you'll figure this out, right?

29:12There are 26 people. Yes, yes. The alphabet. No, they're in a different style. Fair enough. Yeah, I think in that case, I wouldn't spend a ton of time building a custom interface for making this picture of 26 people. It seems like the kind of thing that we can solve. Just transfer. Do you think the representation of what the AI images are will change? So the reason why I ask the questions as artists, there's different formats we play with. There's the SVGs. We have anchor points and bezier curves. And on the other side, there's, you know, procreate or like fresco, what have you. There's layers that we can also play with.

29:48There's the other parameter, which is what's a brush you use, like the brush, the texture of it. So every one parameter, you can write script and actually do something very personal about it. Do you think like pixel is the right representation, the end game? for image generation model, or do you think there's a net new representation that we haven't invented yet? That's an easy question. Wow. I'll say that everything is a subset of pixels. That's true, yeah. So text is a subset of pixels, because I could just render all the text as an image. So how far can we get with just pixels is an interesting question.

30:27I think if the model is really responsive and handles multi-turn interactions well, then I think you can probably get pretty far because the primary reason I think you would want to leave the pixel domain is for editability. And so in cases where you need to have your font or you want to change the text or you want to move things around just like with control points, it could be useful to have kind of mixed generation which consists of pixels and SVGs and other forms. But if we can do it all, if the multi-generation is enough, then I think you can get pretty far with pixels. I will say that one of the things that's exciting about these models that have native capabilities is that you now have a model that can generate code and it can generate images.

31:11So there's a lot of interesting things that come in that intersection, right? Like maybe I wanted to write some code and then make some things be rasterized, some things be parametric, stick it all together, train it together. It would be very cool. That's such a good point because I did see a tweet of someone asking CloudSona to replicate an image on an Excel sheet where every cell is a pixel. Which is like a very fun exercise. It was like a coding model and it doesn't really know anything about, you know, images. Yeah, it worked. Yeah, there's the classic pelican riding a bicycle test. Right, yeah.

31:45Yeah, totally. I have one on model, like on interfaces, if that's okay. Sorry if I'm bringing up too much product stuff, guys. I'm just very curious on the product front. Like, I guess I'm curious how you think about, like, owning the interface where people are editing or generating images with Nano Banana versus really just wanting a ton of people to use the model for different things in the API. Like we've talked about so many different use cases like ads, you know, education, design, like architecture. Each of those things could be, there could be a standalone product built on top of Nano Banana that prompts the model in the right way or allows certain types of inputs or whatever.

32:30Is your guys' vision like that the kind of the product in the Gemini app is like a playground for people to explore and then developers will build the individual products that are used for certain use cases? or is that something you're also kind of interested in owning? I think it's a little bit of everything. So I definitely think that the Gemini app is an entry point for people to explore. And the nice thing about Nano Banana is I think it shows that fun is kind of a gateway to utility where, you know, people come to make a figurine image of themselves, but then they stay because it helps them with their math homework or it helps them write something, right?

33:07And so I think that's a really powerful kind of transition point. There's definitely interfaces that we're interested in building and exploring as a company. And so, you know, you may have seen Flow from Josh's team in labs that's really trying to rethink, like, what's the tool for AI filmmakers, right? And for AI filmmakers, image is actually a big part of the iteration journey, right? Because video creation is expensive. A lot of people kind of think in frames when they initially start creating. And a lot of them even start in the LLM space for, like, brainstorming and thinking about what they want to create in the first place.

33:40And so there's definitely kind of plays that we have in that space of just us trying to think about, like, what does this look like? We have the advantage of it kind of sitting close to the models and the interfaces so we can kind of build that in a tight coupling. And then there's definitely the, you know, we're probably not going to go build a software for an architecture firm. My dad is an architect and he would probably love that. But I don't think that's something that we will do, but somebody should go and do that. And that's why it's exciting because we do have the developer business and we have the enterprise business.

34:12And so people can go use these models and then figure out like, what's the next generation workflow for like this specific audience so that I can help them solve a problem. So I think the answer is kind of like, yes, all three. Yeah, I brought that up. I don't know if you guys have been following the reception of Nano Banana in Japan, but I'm sure you've had it. It's been insane. And it's so funny, like I now half of my X feed is these really heavy nano banana users in Japan who have created like Chrome extensions called there's one called like Easy Banana that's specifically for using nano banana for like manga generation and specific types of anime and things like that.

34:51And like they go super deep into basically prompting the model for you and storing the outputs in various places, using obviously your underlying model to generate these like amazing anime that you would never guess were AI generated because like the level of precision and consistency and that sort of thing is just beyond what I've seen any single model be able to do today. I guess, what are some, like to Justine's point, what are some force multipliers that you guys have seen in the model? So what I mean by this is, for example, if you unlock character consistency, you can generate different frames and then you can make a video and then you can make a movie, right?

35:31So these are the things that if you get it right and get it really well, there's so much more downstream tasks that can derive from it. Just curious, how do you think about what are the force multipliers that you want to unlock? What's the next big wave of people who can just use Nano of Nano as a base model for all the downstream tasks? So I think one current one, actually, is also the latency point, right? Because I think it's also just like, it makes it really fun to iterate with these models when it just takes 10 seconds to generate the next frame, right? If you had to sit there and wait for two minutes, you would probably just give up and leave.

36:08A very different experience. So I think that's one. Just like there has to be some quality bar because if it's just fast and the quality isn't there, then it also doesn't matter, right? Like you have to hit a quality bar and then speed becomes, of course, a multiplier. I think this general idea of just like visualizing information to your education point from earlier is sort of another one, right? And that needs good text. It needs factuality, right? Because if you're going to start making kind of visual explainers about something, It looks nice, but it also needs to be accurate. And so I think that's probably kind of the next level where at some point then you could also just have a personalized textbook to you, right?

36:47Where it's not just the text that's different, but it's also all the visuals. The Diamond Age, that was basically, yeah. Yeah, basically. And then it should also internationalize really well, right? Because a lot of the times today, you might actually be able to find a diagram that explains the thing that you're trying to learn about on the internet. but it's maybe not in the language that you actually speak, right? And so I think that becomes just like another way to improve and open up accessibility of information to just a lot more people. And again, visually, because a lot of people are visual learners.

37:19Interesting. How do you think about like images generated? So the reason why I ask is that there's another very cool example I've seen someone making a work with Nano OpenElla, which is he wrote a script and then he kept prompt the model to say, generate the frame one second after this. And then it became a video. And then when I saw it, I'm like, well, is every image just one frame in a continuum? Like you always know about the continuum in a parallel universe. You could have generated any one of them. It's one big directed graph. Right, exactly. And then maybe it's video at the end of the day.

37:54So how do you see that? Where does it intersect or not intersect? I think it's very, yeah, video and images are very closely related. And also I think what we're seeing in these kind of what's coming next or sequence predicting use cases is the generalization and world knowledge of the model as well. And so where do I think it's going? I think that we will have, yeah, I think video is an obvious next kind of domain. I think that when you have editing, a lot of times what you're asking is like, what happens if I do this? And that's what video has. It has the time sequence of actions. So it's like we have a slow frames per second video that you can interact with, but obviously making something that's fully interactive and real time is the direction this field is headed.

38:51So you're probably in the 0.001 % of most experienced people in the world using image models. What are your personal favorite use cases? How do you use it day-to-day if you're not just testing the existing model? Well, I, so I'm not sure I am in the very, but I'll tell you what, I mean, it's like we were saying earlier, the personalization aspect is the thing that totally drives it home for me. I have two young kids and like the best things that I do in the model are the things I do with my kids. and we can make their stuffed animals come to life in these types of applications. It's just so personal and gratifying to see.

39:34We all saw a lot of people taking old pictures of their family, for example, and restoring them. I think that's the real beauty of the edit models is that you can make it about the one thing that matters most to you. That's what I use it for is my kids, basically. Very nice. You're basically making content that you probably would have never made before and it's for the consumption of one person, right? or one family. And so you're kind of telling these stories that you would have never told before. So kind of similar, like I do a lot of family holiday cards and birthday cards and whatnot. Now, anytime I make a slide deck, I like force myself to generate some images that are like contextually relevant and then try to get the text right and all of those things.

40:14And then we try to push the boundaries around like, can you make a chart in the pixel space? Do you want to? That's another question, right? Because you also want the bars in the bar chart to be accurately positioned relative to one another. So I think we do a lot of these things. I'm actually really impressed with the people we work with on the team who are just like very creative. We have a team who just works really closely with us on models that we're developing. And then they just like push the boundary. They'll do like crazy things with the models. What's the most surprising thing you've seen here?

40:42Like, I didn't know our model can do this. Yeah. This is even just kind of like simple things where people have been doing like texture transfer. Like they will take. Yeah. Like you take a portrait of a person and then you're like, what would it look like? But if it had the texture of this piece of wood and I'm like, I would have never, I would have never thought of this being a use game because my brain just doesn't work that way. But people that kind of just push the boundaries of what you're, what you can do with these things. That is an interesting example of the world knowledge because texture technically is 3D because there's like a whole 3D aspect of it.

41:16There's a light and shadow of it, but this is a 2D transfer. Yeah. So that's very cool. I think for me, the thing I'm most excited by and maybe most impressed by is are the use cases, the tests, the reasoning abilities of the models. So some people on our team figured out you could give geometry problems to the model and ask it to solve for X here or fill in this missing thing or present this from a slightly different view. And these types of things that really require world knowledge and the reasoning ability of a state-of-the-art language model are the things that make me really go, wow, that's amazing.

41:53I didn't think we would be able to do that. Can it generate compiled code on a blackboard yet? Like if I take a picture of my, I don't know, like code on the laptop, would it know if it compiles on the image model? I've seen examples where people give it like an image of HTML code and have the model render the webpage. Wow, that's very cool. The coolest example I saw, so I came from academia, so I spent a lot of time writing papers and making figures. and one of our colleagues took a picture of one of the result figures from one of their papers with a method that could do a bunch of different things, a bunch of different type of applications in the paper and asked the model to erase the results so you have the inputs and asked the model to solve all of these in picture form in a figure of a paper and it was able to do that so it could actually figure out what is the problem that this one figure is asking for, find the answer and put it in the image and then do that for a bunch of different applications at the same time, which was really amazing.

42:53That's very cool. Has anyone built an application on top of that capability yet? What's the application that will come out of that? I think that there are a lot of very interesting I would say zero-shot transfer capability, problem-solving type things that we don't even know the boundary of yet. And some of these are probably quite useful. If you want to have a method that does solves some problem X, I don't know, like finds the normals of the scene or something, the surface orientations or something, you probably can prompt the model to give you kind of a reasonable estimate. So I think there's lots of problems, like sort of understanding problems and other types of things that we could maybe solve with zero or a few-shot prompting that we don't know yet.

43:37There's one thing you mentioned I found super interesting, which is the world knowledge transfer. But in a lot of world models, like, or video models, there always is something that keeps the state. Like just because you look at a way doesn't mean that the chair should disappear or change color because that's not what the state of the world is. How do you see that? Do you think there's relevance there in image model? Is that something you even consider optimizing for? Yeah, I mean, if you think about an image model that has a long context where you can put other things in that context, like text, images, audio, video, then I think it's definitely like you're reasoning over the context of things you have to produce a final output image or video.

44:17So yeah, I think there's definitely some model capability to do this type of stuff already. Got it. I haven't tested it out yet. For this big use case, I'll let you know. That's one of my favorite things about these models is just finding, and I'm sure it's really fun for you guys, and you guys probably have much more of a hint than we do about what they can do, but sometimes you'll just see some crazy X or Reddit or whatever post about some incredible thing that someone has figured out how to do that you would never expect that the model might be able to do necessarily. And then other people kind of build on that and say like, oh, and then I tried the next iteration of this thing.

44:55And suddenly you have this like most entirely new space that's been discovered in terms of what the models are capable of. It must be fun as people much more deeply involved in kind of building these models and building the interfaces to kind of watch that happen. Yeah. So if you talk to visual artists today, I personally love this stuff I post about it on the internet. You get some very skeptical answers. People are like, oh, this is terrible. Do you have any idea what triggers this reaction? I'm convinced that this ultimately really empowers the artist. It gives you new tools. It's like, hey, we now have, I don't know, watercolors for Michelangelo.

45:34Let's see what he does with it. And amazing things come out. It's a similar thing. But what triggers this strong reaction against it? so I think it's something to do with the amount of control over the output so you know in the beginning when we had these kinds of text image models it would be very much like a one shot you put in some text, you get an output and people would be like oh this is art, this is this thing I made and I think that maybe rubs people a little bit the wrong way who come from the creative community because most of the decisions that were made were made by the model you can't express yourself anymore physically.

46:11Yeah, exactly. So as a creative person, you want to be able to express yourself. So I think as we make the models more controllable, then a lot of these concerns of like, oh, that's just the computer is doing everything, kind of may go away. And the other thing is, I think that there was a period of time where we were all so amazed by the images that these models could create that we were pretty happy to see just like, oh, this stuff comes out of these models. But I think humans get really bored fast of this type of thing. So there was a big rush. and now if you see an image that you know is just like oh that's just like a single prompt, a person didn't think about it much, you can kind of tell that's an AI-generated image, not that interesting.

46:47So I think there's still this boundary of now you need to be able to make interesting things with the AI tools, which is hard, but this will always be a requirement. We need someone to be able to do this. We still need artists. We still need artists and I think artists will be able to also recognize when people have actually put a lot of control and intent into it. Maybe an artist. Maybe you'll get it. But it is, there's a lot of craft and there's a lot of taste, right, that you accumulate sometimes over decades, right? And I don't think these models really have taste, right? And so I think a lot of, like, a lot of the reactions that you mentioned maybe also come from that.

47:24And so we do work with a lot of artists across all the modalities that we work with. So image, video, music, because we really care about, like, building the technology step-by-step with them and trying to figure out, They really help us kind of like push the boundary of what's possible. A lot of people are really excited, but they really do bring a lot of their knowledge and expertise and kind of like 30 years of design knowledge. We just worked with Russ Lovegrove on fine-tuning a model on his sketches so that he can then create something new out of that. And then we design an actual physical chair that we like have a prototype of.

47:57And so there's a lot of people who want to kind of bring the expertise that they've built and kind of like the rich language that they use to describe their work. and have that dialogue with the model so that they can push their work kind of to the frontier. And it is, you know, it doesn't happen in like one prompt and two minutes. It does require a lot of that kind of taste and human creation and craft that goes into building something that actually then, you know, becomes art. At the end, it's still a tool that requires the human behind it to express the feelings and the emotions and the story.

48:28Yeah, absolutely. And that's what resonates with you when you probably look at it, right? You will have a different reaction when you know that there's a human behind it who has spent 30 years thinking about something and then poured that into a piece of art. Yeah, I think there's also a bit of this phenomenon that like most people who consume creative content and maybe even ones that care a lot about it, like they don't know what they're going to like next. You need someone who has a vision and can do something that's interesting and different. That's right. And then you show it to people and like, oh, wow, that's amazing.

48:58But like they wouldn't necessarily like think of that on their own. Right. So when we're optimizing these models, one thing we could do is we could optimize for the average preference of everybody. But I don't think you end up with interesting things by doing that. You end up with something that everyone kind of likes. But you don't end up with things that people are like, oh, wow, that's amazing. I'm going to change my whole perspective of art because I saw that. It's the avant-garde edition of the model. If I use it at the term, I don't know, what's the other end of the spectrum? The marketing edition or so, where it's very predictable.

49:31Yeah. Well, since we're coming up on time, last couple questions. One is, what's one feature that you know the model is capable of that you wish people ask you more? Interleave? Yeah, interleave. I think we've always been amazed that nobody ever posts anything about, so interleave generation is what we call the model's ability to generate more than one image for a specific prompt. So you can ask for like, I want a story, like a bedtime story or something, like generate the same character over these series of images. And I think that, yeah, people haven't really found it useful yet or haven't discovered it.

50:06I don't know. Oh, interesting. Well, if you're listening to the podcast, go try this out. Try it. And what's the most exciting technical challenge that you look forward to tackle in the next, I don't know, months, years? So I think that there's really a high ceiling in terms of quality for where we're going. I think people look at these images and say, oh, it's almost perfect. We must be done. And for a while, we were in this cherry pick phase where everyone would pick their best images. So you look at those and they're great. But actually, what's more important now is the worst image. We're in a lemon picking stage because every model can cherry pick images that look perfect.

50:43So now I think the real question is, how expressible is this model? And what's the worst image you would get given what you're trying to do? So I think by raising the quality of the worst image, we really open up the amount of use cases for things we can do. Like there's all kinds of productivity use cases, like beyond this kind of like immediate creative tasks that we know the model can do. And I think that's a direction we're headed. We're headed to where if these models can do more things reasonably, then they're just, the use cases will be far greater. So that's the moral equivalent of the monkeys on typewriters, basically.

51:13Any model given enough tries will eventually be an amazing adventure. But the other way around, it's hard. Yeah, the other way around is hard. One monkey writing a book would be very hard. You'd be a good monkey for that one. What are the applications you think that would come out when we raise the lower bound? So the one I'm most interested in, we mentioned this before, is education factuality. I have, you know, I have every, I don't know how many times I want to use these models for creative purposes a month, but like I have way more use cases for information-seeking factuality, kind of like learning, education-type use cases.

51:48So I think like once that starts working, then it'll be opening up all these new areas. Amazing. There's also something about, I think, taking more advantage of the models context window. So you can input a really large amount of content, right, into these LLMs. And some companies, you mentioned a few before, they will have like 150-page brand guidelines on like what you can and cannot do, right? And they're like very precise, right? Like colors, fonts, right? And like the size of like a Lego brick maybe. and so being able to actually like take that in and follow that to a T when you're doing generation, that's like a whole new level of control that we just can't, we don't have today, right?

52:30To make sure that you're actually kind of like following that to a T. I think that will build a lot of trust with, you know, very established brands. So we have a second creative compliance review model that then checks between the producing systems. The model should do it on its own, right? Like it should kind of have this, yes, it should have this loop as like, okay, I generated this, but then page 52 says that I shouldn't have. I'm going to go back and try again. And then two hours later we'll come back to you with a respect. So we saw with the text models how this inference time scaling, how much it can help, right?

53:00Being able to critique your own work. So this feels really important. Boy, incredibly amazingly exciting future. Yeah, and congrats on all the amazing work. Thank you. Thank you. Thank you so much for coming on the pod. Thanks for listening to this episode of the A16Z podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode.

53:42As a reminder, the content here is for informational purposes only, should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com forward slash disclosures. Thank you.