Genie 3: An Infinite World Model with Shlomi Fruchter and Jack Parker-Holder

26:15So like in that example, maybe have a reindeer coming through. Right. We can have a reindeer or we can have like another person is walking into the scene and then we have to, if it's an agent, basically they can respond to it. And if it's just for entertainment purposes, then it's much more interesting than just walking in the world that nothing happens. And this is all a key advantage of the auto-aggressive part, I guess, right? That you do have control over the future. Exactly, yeah. So you can just inject things sort of on the fly as you're generating things in real time. The thing that I find extraordinary about this is the consistency, right?

26:48Like the memory of the system. You turn away and then you turn back and it's exactly how you left it. But presumably if you're turning in a direction you haven't yet turned in, then it's sort of a stochastic process, right? So how do you balance it that it's sometimes a memory and sometimes generated statistically? Well, it's kind of a mixture of things. So you can, in the text prompt, specify things that are out of sight, which I think is quite powerful compared to image prompting, right? Because you can say on the right is X, Y, Z. And then when you actually look around to the right, it's there when you actually play or interact in the world.

27:23But then there is also an element of the model using its world knowledge to generate things. So in the Hopper example with the artwork, it does generate the street that we haven't seen before. And you can't be exactly certain what it's going to generate. The model uses its own sort of, I guess, intuition of what should be there. And that's intuition based on having watched, I mean, incredible amounts of video footage in advance. So basically the model is trying to just generate a sequence of frames that is representative of the world, right? So if it already saw some part or generated some part of the world, then the right thing for the model to do would be to recall this memory and use it.

28:01So if I look back to where I've already been, then the right thing for the model to do would be just to use the same thing. But when you look to a new area, from the model perspective, again, it can allow itself to generate something new because it hasn't been seen. So the model doesn't really treat it in a fundamentally different way. So the model learns to basically balance the two aspects. And again, all comes back to anchoring all of the generation to the prompt or what the user provided, right? That's where the information to the generation comes from. But then I guess going back to the language analogy that we were talking about earlier, it's not that surprising that once a statement has been established in a conversation that it remains consistent when you refer back to it.

28:40Exactly. Yeah. And we've seen with language models, this sort of ability to have memory and consistency has been something that's improved a lot recently, especially with the latest Gemini models now versus two years ago. Yeah, I think what's interesting here is the size of the memory or the level of detail. So if we think about I'm talking to Gemini and after a few sentences, it might refer to something I've said before. That's great. But the number of details that we have in those visual worlds is just like it's staggering. The quality of the memory, considering how much detail and how much information it has to actually remember.

29:16Well, how do you do that? Does this require you having a sort of 3D representation of the world that you're in? We don't use anything like that for this version of the model. It's largely just learned as an emergent property, just from this autoregressive prediction. You guys with your emergent properties, my goodness. We're very good students of the bitter list. Yeah, I think it's similar to if you're predicting the next frame, you just have to learn to remember these critical things in the past. And obviously the model has some representations that prioritize the important details, but it really is just predicting the next frame.

29:47And is this analogous then to the fact that language models have this conceptual understanding and can describe the same things in different ways? Is it the transformer architecture here that's allowing you to do that? Yes, so the architecture is, you know, I think almost everything today is a transformer. So yes, this is also a transformer. Looking back, basically the model will see what was already generated, including in the user-provided inputs, and based on that makes the prediction. So I think the interesting question about explicit 3D is that the model probably had to learn some representation, but it's just not an explicit representation.

30:22So we see that the ability of the model to understand 3D environments is very strong. And I think the most, to me, like an emergent capability is that it actually works where it was not trained for like, for example, in taking an oil painting in 1942 and actually making it into some kind of a 3D environment. And that's pretty much out of its distribution. Absolutely. I want to go back to this idea of it understanding physics. Jack, if you've got this world that you can put an agent in, does that mean that you can test how well it knows physics? I mean, could you get a hammer and a feather, for instance, and allow it to drop it at the same time?

31:01You definitely could do that. I think that would probably be close to the frontier of the model's capabilities at this point. We've seen it's quite good at visual things and things that are more general concepts, right? So you can imagine water occurs in quite a lot of different scenarios it's seen before. Gravity has probably occurred in many of those, but probably not these exact objects. But I think if you were to specify in the text prompt, like in the world there is a feather that has less gravity and a hammer that is heavier, maybe then it would work. Yeah, I think there are a lot of, like, those models are inherently visual, right?

31:36They don't know anything about the world except for what we can see. And that's a limitation, I would say, right? Even for video models, some things sometimes don't make sense. And the model has to guess if something's heavy, for example, right? There is no real way for a model to look at an image and say, okay, that's how much this thing weighs. So it kind of makes up the weight and then try to kind of simulate what would have happened. And sometimes it breaks. And we know that from video models, even the best video models. and I think what we have is basically we're solving an even harder problem because we have to, as we said, we can't fix the past, right?

32:11We have to go, if once the model generates something, it has to roll with it and I think we have really good progress in terms of simulation like fluid dynamics but maybe some other aspects of the world physically would not be accurate because of those limitations. Well, tell me some of the work that you've been doing with agents so far because actually on a previous episode we got to talk about SEMA, the scalable, instructable, multi-world agents, which they were putting into existing computer game environments. But, I mean, you can now take CIMA and put it into these generated environments too.

32:45Exactly. So the cool thing with this is that we've got, at Google DeepMind, we have these agents that are trained to be general. The M is multi-world, as you just said. And so what we're able to do is take the worlds we generate and then test whether these agents and the latest versions of them are able to already use them for agent training or collecting experiences or evaluations as they are you can say to it something like navigate to the robot over there and then you can give it an image from the world and it can take the first action and then from that point onwards it's interacting in the world through three actions but the genie 3 model does not know what the goal is right so that makes it like a genuine simulation rather than if you were to tell genie 3 the goal that agent is trying to achieve this goal it might make the experience not authentic right because it might make it happen in an incorrect way yeah exactly i guess if you've got a robot in the real world the world isn't helping the robot exactly like you can't say okay the robot has to go to fetch the red cube and then it looks to the left and there is right there is a bit making yeah and this is a problem that we've encountered in other scenarios but you don't really get this problem if you have this separation between the agent and the environment and that's the really nice thing of working with agent teams that are focused on building really capable agents, but then having them access our environment as if it's any other environment, right?

34:02So the new worlds that are created by Genie3 look the same as the existing worlds that the same agent was trained on. But then what if there isn't a red cube? What if it searches around forever and there is no red cube? So yeah, I think that's part of the things that we're looking into is how we can add more details as we go, right? Like, for example, if you put the agent in some room, and then it has to open maybe a drawer and find something in there, right? So we want to be able to inject events into the world and control it. So I think this is kind of like the interesting frontier here is how you can make the world look very realistic, and also control what's happening in the world in a way that still makes sense.

34:43So I think what we've seen is that we have our promptable world events, we can add things that happen in the world. But if you just want something to pop into the world, then it's not necessarily plausible. I think like if you look at the, if you're in the desert, and then you're going to ask it, okay, now I want to see an elephant, then where does this elephant going to come from? So maybe it's going to come from the side when you look to the left. So I think there is something really interesting in how what does it mean to change a world because the model has some assumptions. And when it comes to agents, this is definitely an important capability to be able to inject this new event into the world.

35:17So you've done this already? Yeah, we've got signs of life for this. I think we haven't really, say, we've got a full agent training loop where we're already doing some large-scale training in these environments. But what we can already do is test the agents in them and see how they do, right? And I think it's quite remarkable that given these things weren't developed together, we just dropped the agent in and it can already do things. and you can imagine all the different things you could now use this for. Colour that in for me. Give me an example. If you have like a factory where you have some robots and you want to introduce maybe a very boring example, but a new machine, right?

35:52And that wasn't there. Or you change somehow the structure of the building and you want to test the robot before you actually put them in the new building, right? So again, this is like you can simulate the world. That's basically a variant of what maybe the agent has seen in the past and see if it breaks. And everything can happen in a simulated environment and not necessarily break your new machine. So that's one example, I would say. Find the unintended consequences. Yeah, and evaluation of the model. So that's even not training the agent, just testing how well it adapts maybe to a new variation of an environment.

36:26All of these examples you've given so far are where the agent has a specified objective, which I know is sort of the point of SEMA thus far. But what about if you had agents that didn't have an objective? There was a really nice quote that I came across, which is that almost no prerequisite to any major invention was made with that invention in mind. Can you imagine a point in the future where you are letting agents loose in these environments without specifying an objective for them? Yeah, exactly. So the quote comes from Why Greatness Cannot Be Planned, which is from Ken Stanley and Joel Lehman.

36:59It's a great book. And the general idea there is that searching for interestingness might actually lead to things that are more useful for practical goals than if you just directly optimize for the practical goals themselves. And clearly, the bigger the domain and the space for discovery, the more interesting things could happen. So they had this really nice example quite a while ago with this paper called Pick Breeder, where essentially they allowed people to select images and sort of combine them to create new images that were mutations of those two. People weren't directly optimizing for specific end goals, but by just choosing what they found interesting, they ended up discovering really cool structured pictures like a skull or a butterfly that weren't obvious how you would reach that from the starting points.

37:45And some of the stepping stones along the way didn't really look much like the final goal. And they wouldn't have been things you obviously would have chosen to go for if you had those goals in mind. And there are lots of examples of this in the real world. if you are for example trying to you know reach the moon you wouldn't build a bigger ladder right so optimizing along one dimension with with maybe like a greedy myopic approach doesn't always lead you to make these big leaps well i mean evolution itself right is like the classic example of of iteration without objective yeah and we see this a lot in research yeah my perspective on that is that i think we as humans we kind of decide what's interesting i think there is even an example where the entire evolution of mathematics was guided by people deciding what's next, what's interesting, what's not interesting.

38:30And just the problem being hard doesn't mean that it's interesting at all, right? And I think in a way, when we think about generating new things in science, there is some aspects of maybe beauty or interest that is coming from us, and models might maybe learn to simulate that. But I think it's really important to remember that we ultimately decide what's interesting as our preferences as people. I mean, you haven't had the move 37, as it were, in this. Even in this case, right, the goal of Go or the game was designed in a certain way that people find it interesting to play, right? Otherwise, like, and it's a very ancient game, right?

39:07So I'm just saying that the setup, even when there is an, like, getting to the moon, this goal was made by people. So I'm just saying, like, I think there is still a broader constraints set by us in a way. And, like, if a machine comes up with a problem to solve, we still have to say, is this an interesting problem? Because otherwise we'll just, okay, I don't care about that. So I think there is still an aesthetic part to anything that's open-ended. Yeah, I think there was this quote from Demis a while ago about the levels of creativity. And it was, like, interpolation is one. Like, you see a new cat and you can identify it as a cat.

39:43Extrapolation was one where it's like, given the rules of Go, can you discover a new move like Move 37? And then the third level is generating completely new things. Like, could you actually invent Go is what he said. And we actually had this as like a motivation for the Genie project at the beginning. It was like, can you create completely new things? And I think we're starting to see that happen. Since it's a completely new kind of model, someone on the team will do something like create a certain kind of world. And then immediately other members of the team are like, that's really interesting.

40:10and then they start sort of evolving that idea themselves. And then we post it on social media and we see the reaction to some things and then we know that's interesting. So then we create new things that way. And that's just with a very limited access to the model. So clearly you can see if we open this up a bit more in the future that it could lead to some sort of open-ended creativity that way. It's like an evolution with the criteria that people find interesting. Yeah, exactly. There's people in the loop guiding the interestingness Well then, okay, could you take, I'm just thinking back to the conversation that we had with Dave Silver earlier in this series, where he was sort of saying, you know, actually there's one way you remove humans from the equation and actually you get even more surprising results potentially.

40:51Could you, okay, just play along with me here for a moment, but could you get to a point where you just simulate the first, I don't know, single-celled organism and allow it to evolve inside of Genie, and actually watch the process of evolution happen in a virtual environment? That's a great question. And that's kind of the dream of like an Alife open-ended evolution community. I think maybe the worlds that we create are not fully rich enough, but I definitely think that we're getting along that path, right? So open-ended evolution and Alife, they've been designing worlds that could facilitate this kind of thing, typically in code.

41:27And so this could be an alternative approach to getting maybe richer real-world simulations. And so in theory, I mean, we've made quite a lot of progress pretty fast. But if you got the simulation to be fully like the real world and it had the kind of objectives and constraints that lead to these kind of evolutionary steps, then it's definitely plausible. But I can't say it's definitely there yet. It's not a direct answer, but I actually tried. I think there is maybe a very basic example of Game of Life, right? So it has four rules, right? And then I actually tried using Veo to simulate it, right?

42:07You give it an image and it doesn't work. It does look like it's evolving. And if you don't know the rules, you would look like, yeah, it looks reasonable, like different pixels light up. But it doesn't follow the four rules of the game. I think this is a good example for what our current models are able to do and what they are still limited in their ability to follow specific rules. but to actually evolve maybe life forms I think you need much more ability to do both to also simulate the physical world but also follow some basic rules of physics in a very kind of like accurate way constrained yes and constrained and I think we're not like we see some glimpse of it but not it's definitely very far from being able to have evolution in on a GPU well thank you for going philosophical with me for a moment there I enjoyed that let's come back down to earth with the thump though because i mean there are safety implications with this what what are your main concerns i think there's different levels of concern right there's sort of the known things and they're quite obvious i mean things like violence maybe we wouldn't want to occur in the world in new ways and that's something that we can we can already start addressing but there's also maybe some more gray areas where we're not actually really sure how we feel about these things like historical settings for instance some of them may be unsavory for subtle reasons and those are just things that we as a team can see quite clearly but i think there's probably also some things that we haven't considered and we'd rather get those things right by limiting our early access and getting feedback which we're already doing and we've already learned a lot from the folks that we brought in a few weeks ago and the folks that we're still interacting with like what we've found a lot of new use cases that i didn't think of vocational training could actually be quite impactful right lots of people can't go into the role of things like like firefighting for instance what does it feel like to actually be there without having a visceral sort of experience it's probably something that you would benefit a lot from to be able to simulate in advance even if it's not perfectly correct from a simulation perspective there may be elements to it that just getting a sense of what it's like to be situated in that specific circumstance that it's quite nice to be able to simulate in advance minus the heat yeah minus the heat and the smoke And the genuine jeopardy.

44:27Yes. But actually, you raise another interesting point there, though, because you said even if it's not perfectly realistic, is there also another danger about this gap between what's simulated and what's real? How do you make that as small as possible? This is something, the sim to real gap is something we've spoken about in this podcast lots of times before. Let's say you've got your example where you've got a robot in a factory. Oh. And it's moving around. It's sort of. In terms of free labor. Interacting. You can't directly take that and map it onto the real world, right? Yeah. So I think over time we'll see more control, basically being able to take maybe a real environment, map it into the model so the model can base its generation on the real environment.

45:07And we see it to an extent we're starting from an image or starting from a video. And now the question is, would it be perfectly the same as the real world? Probably not. I don't think it's even well defined. What does it mean? But I think the gap is definitely narrowing. So we can take environments, you know, if the past, we know a lot of the RL environments look very far from anything that's photorealistic or real world. But now we can go even closer. So, but definitely there still remains a gap and we will have to kind of see what are the implications. And like, we're still not using it for any real world deployment, of course.

45:40Yeah. Yeah, I think it's kind of an iterative approach. I don't think we're saying right now we've got GD3, we've solved simulation for any possible embodied task. but I think what we can do is combine it with other techniques so we still would train our agents in the same way that we already did without this and we use it to kind of augment the training process and then another element to it is it's really important that it has some diversity right so if it's always wrong in the same way then agents might learn to exploit that inaccuracy whereas if the model can generate quite diverse sort of different worlds then what we can do is really test the breadth of the agent's capabilities and make sure that there's no scenario where it does something really wrong.

46:22That might actually be a strength, right? So same in Sim2Real, we want to do domain randomization. Maybe by having a model that is a generative model, being able to sort of search the space of possibilities and check that all of the agents do something sensible might be a good thing. But that doesn't mean you want to completely train it that that is the real world. Maybe you want to use it to make it more adversarily robust rather than learn specifics. Let me make sure I understand that then. So if it's wrong, but wrong in unpredictable ways, then actually that might end up making the agent more robust in the long run.

46:53Yeah. So what you want to do, it's similar to domain randomization, is you want to make sure there's no plausible scenario where the agent could do something really unsafe. It's quite a different objective than if you had one specifically incorrect scenario and then you told the agent exactly how to behave from that one. Instead, it's sort of make it so that in any possible future world, the agent should be able to do something sensible. That's so interesting then, because I was sort of imagining that you were trying to kind of nudge this towards it being more realistic, towards getting like more reliable outcomes towards sort of like closing that gap of sim to real.

47:28But I mean, the way you're describing this is that not necessarily. I think the question is, what do you mean by reliable? Because reliable, to me, is probably a lot about following the instructions that we provide, right the model so if we want the model to simulate a specific environments and we describe it with a lot of detail we want the model to follow that if there is something not so plausible in this description the models should still follow that and i think the challenge is that sometimes we as people or like for various reasons we're interested in the less plausible scenarios right for example in the some of the examples we saw today you wanted some vodka and the calf liver All right.

48:08So that's not a very problem. It's not like if you just sample from all of the possible tables in Siberia, probably that's not like in the middle of the distribution. So I think that the reliability to me comes mostly from following the description that we provide the model with and simulating the world in a way that would be close to that. I think that's a really good point, actually. I think it's in underspecified environments. You want diversity because you want to be able to adapt anything within the plausible distribution. but if you have a very well specified environment then you want it to be accurate and I think we're kind of seeing improvement on both those dimensions but we're probably not fully there yet let me go back to that AGI question if I may the end question that everyone always wants to ask do you think that this is a step towards it Jack I think AGI is something itself which is relatively subjective and people have different interpretations of what you mean by AGI so I think it would be quite maybe grandiose to say our model is the key thing in the whole field that will enable AGI.

49:10But I think for me, an AGI needs to be embodied and be able to act in the physical world. That's what really excites me. I think that could really improve people's quality of life in any demographic anywhere in the world. And so with that framing, I definitely think this is an important tool. I can't see how an embodied AGI or an AGI that is embodied would be able to operate any scenario in the world without being able to simulate it to gather experience and learn from its own experience because that's the paradigm that we've used in other settings to get superhuman capabilities or even just robust capabilities and so i believe very strongly that we need simulation and i also believe very strongly that we won't be able to build a simulator of the real world any other way so when you combine those two things i think yes it is a big step for my version of AGI?

49:59Yeah, and I think it's a really good answer. And on top of that, I would just say that our current generation of AI is limited to the digital world. For AI to be useful for us, definitely it has to have some kind of real world interaction. So I think, again, it's a small step towards embodied AI. And there are definitely a lot of gaps to get there. So I think we need much more better signals that, for example, robots get while they walk through the world, right? They need to get some physical response. It's not enough just to have a visual input and output. So I just think that this is definitely a step towards that vision.

50:34Because there is still a lot that this can't do. I mean, the additional sensors being one of them, but also doesn't handle people that well at the moment, does it, Jack? Exactly. And I think that's really the key thing, is I both think this is one of the most promising technologies to achieve, sort of sociable and socially aware robots and embodied agents. but also i think that's the biggest limitation probably of the current iteration of the model is that it doesn't do this perfectly because our standards have raised and that's the thing that we now think is something that isn't good enough but i think it's really critical that we do have that right because even if our robots and embodied agents fully understand physics i think physics are fairly consistent around the world but people are not right and we want these agents robots whatever form factor they come in to be able to really augment humans and work with humans to make our quality of life better.

51:23And so they need to understand how humans think, work, interact, and be able to work with us on things. And so that's what I think we're really excited about as one of the things that might be enabled by our model. I think there are definitely a lot of limitations in terms of the quality of the generation, but I'm very excited about the pace. If you think about it, we had G2, V2 in December. We definitely feel the pace, the impact on it in our personal lives, but the field is just moving fast. And if you remember just less than two years ago, we had images generated with six fingers. And that was a big thing.

51:59And nobody's speaking about that anymore. So I don't see why we won't be able to generate people in much higher fidelity and with everything that follows that. Is the goal here then to have a foundational model to essentially do for simulated worlds what LLMs have done for language? Yeah, exactly. I think you've put it better before you've got it. it i think this is really a step change as a foundation model in terms of the breadth and generality and capabilities and i think this is probably similar to sort of what shomi alluded to as we've seen with images recently where there were things obviously like the fingers to them now being i mean pretty incredible we saw the same thing with video maybe in the past year where once we had something like vo2 it's looking pretty amazing at this point and we saw this with language models maybe three or four years ago where they started to get really capable And we wanted to get to that point for this new kind of foundation model, sort of an autoregressive world model.

52:50Now we're there. There's a whole host of different potential things that could be used for and have impact on. And we're still fairly early in that right now. But there are, you know, other elements of simulation too, right? Like, do you think that you will ever be able to use this kind of idea to recreate a lived experience rather than just a visual one? So I think that we have many, many senses that we're not even aware of, right? Like, so for example, proprioceptive reception that we are basically, we feel where we are. And we have this kind of notion of where we are in the world. And I think when we think about actually putting people in a simulation to really feel kind of immersed in that, I think this is a huge part of it.

53:34Basically, the constraint to visual and maybe audio is still too much of a constraint. I think that there is definitely potential for that, but it goes through multiple technologies that we have to build to actually get there. So before that, I expect people to be able to interact with photorealistic environments, but still through some kind of an interface that will be like a hybrid interface that maybe they can feel some sensation through like maybe gloves or something. I think I would also say that there is definitely something about being interactive in real time. that does make a big difference for the experience.

54:11And we've had members of the team say that they visited childhood locations, for example, and did actually get like a sense for it that you couldn't really get from an image or a video. So there is already some sort of a degree of experience that you can gain from this kind of model already. And obviously, we're working hard to make it an even more capable model in the future. So maybe that will extend. Yeah, there is actually, it reminds me of a project that we had earlier in the year, that the team at Google used Veo actually to help people with early onset of dementia to go back to their childhood memories and reconstruct them.

54:48So I can imagine that diet might be, for example, potentially a therapeutic tool as well, that they can not only look at the video, but maybe actually relieve or remember some things that's from their childhood. So I think that even before, we don't need to go very far for things to have positive impact on the world. Amazing. That was absolutely fascinating. Thank you so much. Thanks for having us. Thanks for having us, Hannah. I think the most impressive part of this is not what you're looking at on the screen. It's how that is generated. It's this change that this model represents from creating realistic images or videos of the real world as though they were frozen moments in time into something that can actually handle time in the way that we experience it with an arrow that's pointing in only one direction, where effect follows cause to build this consistent, forward-moving world where the present is a direct result of the past.

55:50And that is why I think that this is an early hint of something that's much bigger. This is not just a new way to design games or beautiful environments. this here is the bedrock for machines that can genuinely plan and reason about our world you have been listening to google deep minds the podcast with me professor hannah fry now we're going to take a little bit of a pause over the summer but we're going to be back with more episodes in the autumn from google hq in california and in the meantime do take a look at our extensive back catalog, which covers everything from tools for creators to AI for drug discovery.

56:29See you soon.