Genie 3: A New Frontier for World Models with Jack Parker-Holder and Shlomi Fruchter - #743

0:00I think in Genie3, we really tried to push it to the limit across all of the dimensions. So we see that we have models that are more capable in terms of the quality of their generation. We see that if you look at the resolution, the duration of the interaction, how fast the next frame can be generated, you get a very significant, if you multiply all of those dimensions, you get quite like a 100x improvement.

0:40All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. Today, I'm joined by Shlumi Fruchter and Jack Parker Holder, researchers at Google DeepMind to discuss the recent release of the Genie 3 model, which is an impressive world model that we first introduced to you here on the podcast in our conversation with Ashley Edwards almost exactly a year ago. I'm super excited to dig into Genie3. Jack and Shlomi, welcome to the podcast. Thank you. Nice having us. Excited to be here. This is an interesting interview to try to dig into because we covered Genie relatively recently, but I don't want to assume that people have listened to that interview or even know about Genie at all.

1:31So we're going to start a little bit from the beginning and dig into the project and where it comes from and why it's exciting. I'd love to have each of you just introduce yourselves before we dig in, though, and share kind of the highlights of your path to ML research and what you're most excited about, what you research, that kind of thing. Jack, why don't you get us started? Awesome. Thanks. So without, you know, being, you know, too much of a suck up, my path was that I was working in finance just around 10 years ago. And I did a master's part time in the evenings after work and actually resisting to your podcast quite a lot around 2017 time.

2:20I got into ML at the time research and doing actually evolutionary methods for reinforcement learning. I'm working with some folks from Google Brain as well, because I was in New York and there was an office there. And then I decided to do a PhD where I focused on open-ended learning, reinforcement learning still, and then got a bit into world models. And then by the time I finished my PhD, I was increasingly convinced that the combination of these ideas would be the really powerful big thing to do um so after my phd i joined google deep mind and then worked a little bit on a project called adaptive agents um which was in the xland environment and then after that pretty much started genie so i've been doing that for a few years um i'm in the open endedness team which encompasses some other areas too but we've been focusing on this idea of using world models as a path to open-endedness as part of the Gini project.

3:15Awesome. Awesome. How about you, Salami? So I started actually, my first programming kind of experience was in my teen years developing game engines, actually. So 3D engines and coming more from this kind of like a world of trying to simulate effects and like lighting effects, liquid effects, et cetera. So I've been working in this space for some time. And I really like this visual domain. Then I joined Google and I've been on the Google Duplex team. So what we've done in the Duplex project is basically, it was pretty much very different from visual stuff because it was mostly getting stuff done over the phone.

3:59So the Duplex project was, if people remember from Google I.O. 2018, actually, It was like people thought, okay, wow, we have AGI. Yeah, it was one of the easy AGI moments, but I don't think, you know, probably not. But I think it was definitely the nice thing. And specifically, this was the project where Google was going to call restaurants and hairdressers to make appointments on your behalf, right? So when we started Duplex, the goal was actually a question of can we build already today a bot that talk over the phone with people without them feeling this is actually a machine, right? And hitting this kind of like maybe over the phone Turing test, if you want.

4:43and what we found is that although having a completely general conversation over the phone was not achievable until very recently with LLMs, already then, and that was an era of RNNs, LSTMs and definitely not transformers, then basically we were able to develop something that was, at least for this particular task, accomplished a lot of tasks. So that was kind of like my first touch with machine learning. And it was very interesting because it was also like it was very research oriented, but also had the deployment in the real world. We ended up scaling that to call hundreds of millions of calls over the, for overall like 15 countries.

5:35So basically, that's quite, quite a lot of it. Yeah, it's less, less people are aware of it, but because it's not like it was mostly calling businesses to update Google Maps. And I think when you saw, I also, of course, followed things that happened on the LLM from GPT and internally at Google. We had the MENA and Lambda and other models very early on. And then we kind of integrated this technology into Google Duplex. But at some point, I just felt that maybe my visual roots kind of like came to play again. and I felt like the revolution in image diffusion models was really appealing to me. I just felt like this is a huge opportunity.

6:17It's really something like it's hitting some kind of a point of maturity. Yeah, since then I've been working on video models and one incarnation of that was Game Engine, which was a bit of a side project. We were very excited, me and a few friends. Actually, one of them was on Duplex with me, He was actually the founder of Duflex, Yannick Levyathan. And together, we basically asked the question of, is it possible to simulate an existing game, but in real time, completely by a neural network, right? And then I've been also working on Veo, Veo 2 and Veo 3. But around the time of Game Engine, I started talking to Jack.

7:04I was very impressed with the Genio line of work. And we can talk more about that, of course. One of the things that people are most excited about in looking at Genie is this concept of a world model. Maybe, Jack, you can kind of dig into what that means for you and how you see a world model fitting into kind of the broader trajectory of AI models, transformer-based models. like however you think about that um what all what all is captured in this idea of a world model for you sure yeah it's a it's a great question which is i guess on one level quite easy to answer but also could be a whole old book probably of different philosophies um so for myself a world model the definition has actually slightly changed recently i think but what i would have said until about a year ago is a world model is essentially a model from the reinforcement learning paradigm that models an MDP.

8:10So it models taking the state, um, and action, and it predicts the next state. And this idea has been around for a while. A world model essentially is the model in model-based reinforcement learning. Uh, it's modeling the environment, right? And so in the early nineties, I think. Jage Spithuber had a paper about what like recurrent models. And then also the Diner paper from Rich Dutton came out around that time as well. And that was kind of the starting point for this like direction of research and model-based reinforcement learning. And then for me, it was really the Harren Spithuber work in 2018 that was around the time I mentioned where I was getting interested in this field in general that really resonated.

8:53And I was looking at sort of reinforcement learning tasks, right? And they were at the time these Mojoco tasks, like the little half cheetah and these kind of ones. And then looking at the world models paper, and it was basically saying that with a few offline examples, we could actually simulate an environment. So predicting the next state sufficiently well that we could then train policies in that model and then transfer them back to a real environment. And the policy would never train in the real environment. and obviously that's super cool but the setting that they had um was basically that you had data from the real environment so although it was really cool that you could do this you could also just train in the real environment and get the same result in theory right um since you needed to collect that data exactly yes but you could but you had the ability to do it and that was kind of the paradigm that world models were in for a few years is that um it was showing that we could do this at all for increasingly complex environments where we actually had the environment so if we didn't want to do model-based reinforcement and didn't want to use world models we could have just used like a distributed rl algorithm and probably got really good performance a different way and there were still benefits to doing the world model approach maybe it was more sample efficient but it wasn't solving something you couldn't otherwise solve right um but then um kind of going a little back to to like sholomi's comment about this text to image models i mean uh i think we are both excited about that at the same time from different angles right and for me that was sort of saying look if we can do text to image this well uh and i think it was what like four years ago maybe that like imagine came out um then okay video will happen at some point and then after video maybe we'll be able to get to world models right and we'll be able to simulate anything with large data sets right and and so the idea was basically can we actually take the same same concept and apply it to what i would view as a world model which is simulating an environment and therefore we can simulate any environments right and so that's why we came up with this idea of foundation world model so it's like a world model but rather than just being for one single environment it was any possible environment or new environments like a foundation model um and so this was kind of a hard line we stuck to in that a foundation model is this one thing foundation world models is one thing it predicts the next state given actions uh and that's the kind of definition that we took but then increasingly other kinds of models have been considered to be world models recently uh so for example video models text to video um and at first i i didn't really fit in with how i height how i saw world models but actually i think it kind of does at a different level of abstraction right so i think now i have a slightly broader view of world model which is models that simulate the future given the past and some form of actions um and simulate dynamics of a world right so it's not explicitly modeling every transition of an mdp uh which is why i would have said before but maybe just simulating the dynamics of the world in some way that you can act and intervene in it and then get counterfactual information.

12:01And that enables things like planning or like learning policies and simulation. And Salome, did you have the same kind of shift in the way you thought about word models? So, yeah, I'm coming more from the, I think maybe simulation or I think about more the visual. I think what we call word models today are very visual specific, which I think is also one of the limitations of current, what we call word models at the moment. But if we go back to the, you know, Ward Moll's paper and Jack mentioned, again, the definition was pretty clear. And I think it's a good definition to start from and to kind of like anchor to because, again, the term is definitely used in many, many ways.

12:47But I think there is also a bit of an intuition to it and which, of course, can be formalized. but at least from my perspective is like when I started seeing, I think video models, but also image models. But when you write text, right, you provide a text prompt and then you get, you see some kind of like, it feels like there is a world behind those pixels, right? I think that's the, if you want, the intuitive part. Like for things to really look like a realistic image or video, then the model probably has to have some internal representation of what's going on, how the world behaves, our physics to some extent, right?

13:28And I think that's kind of like maybe in layman's terms, the intuition behind the world model. The model probably has some understanding of the world and it's visual. And I think that's key. But the world model doesn't have to be visual. It doesn't have to be something that generates pixels. And I think that's the broader... Yeah, so I think in some of the, again, the literature, the world model can be in some latent space. It just should be something that we can use to make decisions and predict what's going to happen next and maybe perform planning in the RL and learn basically how to operate in a more optimal way in an environment.

14:17But again, going back to the visual domain, which is kind of like what I think happened is basically the visual domain really worked well because diffusion models just happened to work really well for images and videos and audio as well. And then this intersection between those fields kind of like became very obvious. And I think that's where we are right now, that we just have models that are capable of generating very realistic environments. And we're trying to push along this statement. Jack, you worked on Genie 1 and 2 as well. I'd love to have you talk a little bit about kind of the trajectory of the project.

15:00Maybe for a little bit of context. What's most amazing for me about Genie 3 is I remember the conversation with Ashley and like in my mind, I made the assumption that this world was like real-time playable and it was like, oh no, it takes like, I don't remember what the specifics was, like 20 minutes between frames in order to do this. It's not real-time playable. And now I'm looking at this thing that in just a year is real-time playable. But talk a little bit about what the big, you know both functional and research milestones have been between these iterations of the project sure yeah um i definitely i think we did try to emphasize it wasn't real time in in both the challenge was like if you're announcing a new piece piece of research you're not going to say welcome to our like non-real-time breakthrough because why would i guess not like the best way to like you want to make it sound exciting you don't want to just open with a limitation but But then if you put the imitation halfway down and you've got a bunch of cool videos, people don't really get that far, right?

16:10So we definitely had this challenge. But I will say it was slightly less than 20 minutes. I think it was only a few seconds. And if you think about like text-to-image models... I totally over-rotated on the conservative side there. I like that because it's better to set the bar a bit lower and under-promise and over-deliver. But I think we did the other way maybe. but essentially like text image models I mean they they would need to operate at like a 20th of a second essentially to be at the same kind of speed as Genie 3 so I think it is quite remarkable what we what we did achieve with that but going back in time a little bit so I thoroughly recommend all your listeners to listen to the Ashley Edwards version as well essentially Genie 1 was quite a different beast to Genie 3 it was really the first kind of proof of concept of this foundation world model idea, right?

17:04A model that could generate new worlds, right? So there'd obviously be an amazing progress in world models for single domains, starting with the higher-inching people work and then things like Dreamer, Dreamer v2, Dreamer v3 that could model increasingly complex individual environments and show that you could use them for agents to learn amazing behaviors that could solve really complex tasks. And that's like really important work in one axis, right? It's like single domain complexity whereas Genie1 was basically saying we don't care about the complexity of the environments per se can we train a model that can generate new environments and new worlds at all and the challenge with that was on the data side because unlike training from existing environments we don't have action labeled data from your target environment if you are going to be trying to generate new things so we basically collected a data set of unlabeled videos that had no action labels.

18:01And we had this kind of neat approach where we learned latent actions. And the really nice thing about that was that it was an example of people, and I've been very fortunate with this a few times in my career, but bumping in someone who had almost a perfect skillset for the thing I was excited about doing and the things I didn't know about. So Ashley had worked on latent action learning for maybe like more than five years, right? She is basically one of the ones who really pioneered that direction, but in a different context. So she was working on it to learn behaviors from videos, right? So you take videos and you want to extract the actions so you can behavior clone or do imitation learning from those videos.

18:42You need to learn like unsupervised latent actions so you can do that. Whereas we had the opposite kind of setup. We were instead of trying to learn behaviors from the videos, we tried to learn a world model from the videos. So then we could use the world model to learn policies, right? So it was kind of a different framing. And there had been a paper at CBPR called Playable Environments, which did a similar thing. It was Menopass et al. And that came out as I was writing my PhD thesis. And I was thinking, like, how am I going to do, how are we going to do these world models from internet videos?

19:15Or like, how are we going to make this possible? And then I saw that paper and I thought, well, that's at least one path. and then basically got chatting to Ashley and she was thinking, well, actually that does make sense. We could definitely go for this approach and it'd be a really exciting way of getting the later actions to work at larger scale than she'd previously tried before. So for Gini1, it was kind of this quite new idea and could we do it at all? But it was limited in scope. So we trained actually two models, which people don't really notice as well. So we trained one model on this 2D platformer game dataset.

19:51And we trained another model on the robotics data set from I think the RT1 paper, which was this like everyday robotics arm. And in both cases, you could unsupervised learn an action space that meant that you could give the model a new image that wasn't in the training data. Maybe it was generated by a text image model. Maybe it was something to photograph you took yourself. And then you could from that point, play it like a world using the latent actions. Now, that sounds amazing, but there were some caveats. So it was 90p, so it was very small. And it lasted maybe a couple of seconds before it degraded.

20:29And then also, it didn't really generate anything new when you went off screen, right? So we used this mask git approach, and it tended to have like mode collapse when you moved away. So if you look in the platformer examples, when you move to the right, it just kind of continued a flat platform rather than generating like exciting new content. So people did actually notice that. Uh, but we were definitely aware of it. Uh, uh, it was more just like, this is kind of cool that this remotely works. Um, and it was very early stage projects, right? Like essentially when we started this, no one really thought it was worth doing.

21:01And lucky we were in a place like, like Google deep mind where people encourage these more exploratory projects, but it wasn't like a heavily resourced effort, but it was kind of a few of us, um, kind of scrapping together. so that was genie one um that one we actually wrote a paper on it and we submitted it to icml which came out last summer um and so uh that's why i guess it feels very recent because there's a cycle between finishing the work and the paper being published right so it feels much like short time housing um for genie two the setting was um basically genie one paper came out around february march 2024 um we had the results for a while and we'd already kind of started thinking about the future plans and actually got to work on it.

21:45But then around that time, there was a lot of progress on video models in general. And sort of they had their moment like text to image had had a few years before. And so it became very clear to us that for the next phase of Genie, we could go to something a bit larger scale and it would probably work, right? Because we'd seen that video models had scaled pretty effectively. So we decided to go from 2D games to a data set to any 3D world data set. And we scaled it up to 360p. And it had the ability to generate environments, again, from image prompts that lasted maybe 10, 20 seconds before they degraded.

22:27So it, again, wasn't real time. But so you could play with it for a couple of minutes and wait a few seconds for each frame. and it kind of worked for 3D worlds. And so that was also, at the time, not completely obviously going to work, I think, because it was doing this like auto-aggressive generation. And so it still wasn't clear that this would last like even 10, 20 seconds. So I think it was like quite an exciting result. And did Genie 2 overcome the mode collapse issue as you... Yeah, to an extent. And so Genie 2 was a diffusion model. So it had slightly different properties, but it still was only supporting image prompting.

23:13So it wasn't anywhere near as expressive as video models, right? It still required you to select an image for it that worked, which didn't always work. And it couldn't generate its own world, right? It required you to generate an image that was in a certain kind of format where there's like a clear agent in the right location, for example. And then it would simulate from there. Whereas I think what I'm hoping we spend a lot of time talking about with Genie 3 is that it can definitely do that. So yeah, I mean, to kind of bookend this part, Genie 2 at the time, it was sort of a good sign of life for this general approach, but it wasn't real time.

23:59and we'd seen with Game Engine that that was really impactful if you get that right. And the visual quality was good, but it still wasn't using text as input, wasn't as expressive and it was nowhere near the visual quality of state-of-the-art video models like VO2, which came out at the same time, which was absolutely mind-blowing for all of us. So I think that's where my co-lead Genie 3, Shlomi came in and really had expertise. I like to borrow people's expertise And at that point, it was pretty clear we had an expert in those areas. Yeah, talk a little bit about how you incorporated these other approaches, you know, Game Engine, FIO.

24:41How is that incorporated into the Genie line of research? And let me think about this, because I think, Jack, you make a good point, which is we've been talking about this trajectory. We've been talking about Genie 1 and 2. we haven't really talked about, you know, Genie 3 and like, you know, when you pull up the demo, like what's super impressive about it? You know, maybe we'll let you do that as well, Shlomi, and then talk about how, you know, these other projects you've worked on kind of, you know, influenced the overall research. Sure. So basically, I think in Genie 3, we really tried to push it to the limits across all of the dimensions, right?

25:23So if you see it as like, if we see the progression that Jack just pretty much just talked about, right? So we see that we have models that are more capable in terms of the quality of their generation. We see that GNE1 and GNE2 are able to basically generate more and more like worlds that are more consistent but still to a limit, right? So we wanted to push on how long this trajectory can remain consistent. So that was one of the dimensions we wanted to improve. And of course, the resolution, right? So I think all of those, like if you look at the resolution, the duration of the interaction, how fast the next frame can be generated, you get a very significant, if you multiply all of those dimensions, you get quite like a 100x improvement.

26:23If you just think about it as raw compute, right? And I think it was very clear that this is not something that's obvious that's going to work. So there is some risk to this project. But we also felt that this is the time to go for it. So I think what happened basically is that after we launched Vero 2 and basically it was very well received, we felt like the quality really improved, but it was definitely not real time and not interactive. And then Genie 2 came out and definitely also pushed the envelope in this different direction. We just said, okay, let's try and combine those vectors of improvement and go to the next level.

27:05and that's pretty much what Genie 3 is about, trying to bring the best of all worlds, pun intended maybe. And when you talk about this, like bringing the best of these worlds together, is it bringing together ideas? Is it bringing together architectures? Is it bringing together data sets or like some combination of all of the above? First, I think it's all, you know, this may be a cliche, but it's definitely about the people as well. So we had people from different teams kind of bringing their experience and their motivation and energy into this project. So I think that was a big thing. And in terms of the technique, there's definitely shared technical challenges, right?

27:54Basically, the output is eventually pixels. And we had text as input and we want to be able to generate something that is, that feels consistent. So even if it's a video of eight seconds like Feoda generates, you still want to feel that it's consistent, right? If the camera moves around, things should look consistent. You need to want to get a feel that this is a really, like, was taken maybe in the real world, right? And the same goes once it's interactive. We won't be able to generate the next frame based on the user's input, but still, this consistency is really important because if, you know, if we just, there were some early models that were able to maybe generate the next frame.

28:38Like actually, you know, Game Engine in a way was that. Like it didn't have very long context. And the reason it worked is because it kind of learned specific properties of this game of Doom, basically. So it kind of like remembered how the level looks like. So it wasn't really generating it as we would want, right? So I think that was actually being able to generate things from text. this is the core capability that you see from image models to video models. I think that was kind of the main innovation and breakthrough for all of this line of research, that text became like you could start with text.

29:17And text is such a compressed representation and it's such a strong way to learn concepts. So I think this is really, it was obvious that we start from text this time. We want to describe the world and then we just want to drop you into that and then to let the user or agent to just go around and explore it. So I think those are the similarities of the projects. And this is how we, of course, the infrastructure hardware, we have a lot of, in general, the approach we take at Google DeepMind is to try and understand the core mechanics of how those models scale. So those concepts are pretty much, we could leverage them across different modalities and definitely different trade-offs of latency and memory, for example.

30:07Do you have a favorite example from Genie 3? I like the lizard, although it's not photorealistic. I really like the lizard that jumps, you know, the origami lizard. I really like that it splashes a little bit of water once it hits the origami river. And of course, the puddles is, if you know, I think some of the examples were basically posted on X by team members. And I think I actually really like those that people kind of played with the model. And there is one that you walk around and the user is looking down to their shoes and basically sees them in a puddle. And it's pretty, it's very realistic.

31:00I think that's it. Jack, your favorite example? There's loads of really cool examples, I think, that show different capabilities, right? But the one that I think was the most surprising was the one which is the Inception sample, where essentially it takes a minute to probably explain this, but essentially we can prompt our model with videos. And this is a really exciting capability because obviously with an amazing video model like VO3, you can generate really cool videos and then actually prompt Genie 3 with the video and then continue from there. And that's something that's really exciting that we were playing around with.

31:41But then one person in the team, Jakob, by mistake, actually didn't put the right caption. And so essentially what you realized was if you don't align the caption or text prompt with the video latence, then actually the model kind of makes it work. um so basically what happens is if you you're facing you're facing the world with the video prompt and then you'll look away and this other kind of magical world will be there and so what it what he tried was actually prompting the model with a video of people playing the model demo uh so we have this video that was also posted on on social media um of a couple of folks uh in google deep mind playing with the live demo and then in an office room and then the prompt was like a jungle with a t-rex or whatever it was and then during the the genie three generation um the screen that um is showing what they were actually playing switches to this jungle uh this jungle world and so does the laptop so genie 3 knows that it updates both which I think is pretty incredible.

32:55But then it also is the case that when you turn away, you see outside is actually the jungle just as in the prompt. And when you go into the jungle and turn back around, you see the office that they're in and you see them playing. And you kind of have to see it to believe it. But I think it's quite incredibly, incredibly represents how the model kind of actually does have some understanding of things because it understands that like when it updates the screens, it should do both, right? And then it also understands that if you're in an office and you go outside and you look back, then you should see some kind of building that they should be in.