World Models, Robotics, and the Future of 3D AI

0:00Language models are these general horizontal engines for processing streams of discrete text, discrete tokens. Those have had tons of applications from, you know, everything that we know and love today. And our thesis is that there exists another category of model called world models that should be based in visual understanding, should be based in physical understanding, that can be used to generate, simulate, reconstruct worlds. And if we can build these with the right generality, they should be applicable to tons of different industries, right? From entertainment to VR to construction to robotics.

0:25We live our lives in this physical built space all around us, and we need models to help us with these things as well. Language models gave AI a way to work with words. What happens when models can understand and simulate the physical world? World Labs co-founder Justin Johnson joins Theo Jaffe and Sofia Puccini on MTS to discuss Atlas and the broader idea behind world models. They explore how models could generate and reconstruct environments, what separates them from video generation, and applications across gaming, VFX, and robotics, including turning a few photos of a real space into a simulation for training a robot.

1:05We're live with Justin Johnson, who is a co-founder of World Labs, which is a startup that builds spatial intelligence products. And just today they announced Atlas, the world's first multimodal world model that generates image and video frames with pixel-perfect camera control and reconstructs them in 3D. Model the world, move the camera, and simulate space and time. So congrats on the launch. Like the launch video looks really cool. Welcome to MTS. Yeah, thanks for having me. Absolutely. So tell us a little bit more about like what this product is specifically, what you can do with it, what people will use it for.

1:37Yeah, so this is not a product launch. This is a model announcement. So the products are coming later. This is the base model. It's going to be used to power our future products from World Labs. But at its heart, it's a world model. It does basically three different kinds of things. Generation, reconstruction, and simulation, right? generation is you want to generate new worlds that don't exist, right? Like I can start from a text prompt, I can start from an image prompt, then I can direct the camera and sort of fly through that world and whatever, have the model generate a world that never existed before.

2:05So that obviously has tons of applications across many things in creativity, right? From VFX to gaming to all the kinds of use cases that people are using there. The second major capability of Atlas is reconstruction, right? There, sometimes I don't want to generate a new world. Sometimes I I have an existing space in the real world that I like to model and bring into virtual reality or the virtual world in some way. Right. So there what Atlas can do is sparse reconstruction. I can take just a few photos of a space, as few as one, up to 100 or more. And the more photos you give it, the more you can reconstruct that existing real world space accurately.

2:39And the third thing we can do is simulation, right? And this is going after more robotics. Yeah. Well, both VFX and robotics. But there we want to be able to take an existing space, recreate that space in 3D, then simulate how a robot might behave in that space. And we show some examples of all these different use cases on the blog. Yeah. Is the ultimate vision for WorldLab something more like environment for teaching robotics how to work? Or is it more like a metaverse that people will like spend time in? All of the above, probably. Or is it just kind of all of the above? Yeah, I think it's all of the above.

3:10The central thesis here is that there's a new, there's a different kind of horizontal model that we can build called a world model, right? Like language models are these general horizontal engines for processing streams of discrete tax, discrete tokens. Those have had tons of applications from everything that we know and love today. And our thesis is that there exists another category of model called world models that should be based in visual understanding, should be based in physical understanding that can be used to generate, simulate, reconstruct worlds. And if we can build these with the right generality, they should be applicable to tons of different industries, right, from entertainment to VR, to construction to robotics.

3:41You know, we live our lives in this physical built space all around us, and we need models to help us with these things as well. Right. So the model is the input that would go to all of those industries, like the input intelligence. I'm curious like how this has evolved and why Atlas is a big jump here. So I would imagine like one of the difficulties with procedurally generating worlds is that like after a certain point, like I feel like, you know, with video models, like 10 seconds in, I feel like the output could start to get jumbled, especially if you have like a reference image. And then it starts to take its own direction.

4:11It starts to like hallucinate in its own way. So how How do you fix this thing of like at a certain horizon, things would start to get jumped or distorted? Yeah, that's a great question. So here we built Atlas specifically around spatial control and the notion of a spatial context. So when you give Atlas reference images, they aren't just images. They're actually grounded in 3D space. So you can stage those images in 3D and say this reference image should be here. This reference image should be here. This reference image should be here. And when you generate a video from Atlas, you're not just giving it a text prompt.

4:41You also go in and steer the camera and say, I want the camera to fly this way, this way, this way, this way with exact precise pixel control. So then we show some examples where even up to a minute of video, we give it more reference images and sort of leave breadcrumbs of those images along the trail in 3D. And then you get that direct tutorial control over the video that you're generating. So the idea here is that even as you go to really long generations, it's not a slot machine. You're generating and directing and really controlling what the model outputs. Yeah. So yeah, we were just talking right before we went on about our friend Tice from OpenAI did this project that went kind of viral that was like GTA, but it's San Francisco.

5:22So it like, I think it was like a giant Gaussian splat of the whole city or something. And you can like walk around in it. You can steal cars and drive around. You can fly around. It's not very high resolution, definitely not as much as Atlas. You think ultimately this is something you'll be able to do with Atlas? Yeah, I think so. And I think there's actually two different futures here of like the tech path that gets us there. One is sort of the explicit 3D route that you mentioned, right? Like the idea of generating a Gaussian splat reconstruction, then importing that explicit asset into a game engine or something else.

5:52This is something Atlas can do, right? Like it can be used to generate, you know, 3D Gaussian splat scenes, and then you could import them and generate and build games around them. Another direction where this could go is, you know, you skip the engine entirely and go directly for frame generation, right? And I have the model directly outputting the frames without any explicit 3D in the mix anywhere. And Atlas is built to do this as well. So it's one model that both handles, you know, 2D when you need it, 3D when you need it, and can handle both modalities in a native way. If you're somebody using this world model and you have all these inputs, like how would you use, how would you steer Atlas to do what you intended?

6:27Like if you're like a film producer or video game developer, how would you work with it? Yeah, we have some internal tooling for this, but it's going to take a little bit more, a little bit more iteration on the product side to figure out the right way to expose that. But the internal tooling is actually pretty awesome, right? You can basically take images, drag them into this interface, have this sort of spatial 3D canvas for all your input images. Then they end up with this like proxy geometry in 3D space and you can steer your camera kind of with an FPS type control to like decide exactly where you want this thing to go.

6:56And that's a pretty complicated user experience or user interface. So I think we'll probably iterate on different ways to expose these capabilities in nice ways. How does it differ from Marble, which is WorldLab's first product? Yeah, so it's a totally rebuilt stack of models, right? I think the central thesis is the same. We want to build multimodal models that can input many different modalities, output many modalities. This is something that Marble did as well. But one way in which they differ is that Marble really put Gaussian splats really at the center of many things that it did. So Marble could output videos, Marble could output 3D scenes, Marble could output meshes, but everything was kind of bottlenecking through a 3D Gaussian splat representation.

7:31So whenever we output videos or images from Marvel, you were always generating a Gaussian scene, then rendering those Gaussians back to images or videos. And this is really nice, but we wanted to break that dependency. So Atlas, you know, was built fully from the ground up to both rethink the architecture, to make it more unified, more easy to scale, and to have that sort of that branch between 2D and 3D happen earlier. Right? So that when we're generating 2D outputs or video outputs from Atlas, you're not bottlenecking through a Gaussian splat scene anymore. The model is directly giving you those beautiful 2D pixels.

8:03And then only you lift them up to 3D only for the applications that you actually know that you need 3D. So I guess at this stage, like, are we outgrowing Gaussians as a technology? Or like, what is their role within all of this? I think Gaussians are a tool just like anything else. And you got to use the right tool for the job, right? Like, if you're working on something like VFX, then the ultimate output is a 2D image. And or a 2D image or 2D video. And maybe that's all you need. But for other applications, especially real-time applications on embedded devices, like I think Gaussians and other explicit 3D representations still have their place, right?

8:37I think there might be a world in the future where we're all streaming pixels directly off these models in real time from powerful server farms, but we're not quite there yet. And even if we are, it's gonna be pretty expensive. So something like a Gaussian or a mesh is really useful for client-side rendering, especially on mobile or VR devices, and also really useful for integrating with existing workflows and pipelines, right? I think we all believe in the future of technology and we're excited about building these solutions that give us whole new experiences. But the reality is that it takes a lot longer for these things to permeate than we expect.

9:10So people in gaming and VFX and design and architecture, they have their existing workflows that work on 3D representations. And instead of trying to switch them over all at once to a fully AI-native thing, if you can get with AI into their workflows and meet them where they are, then that's another great use of splats and other explicit 3D representations. Got it. Yeah, one of my favorite video games ever is Microsoft Flight Simulator, which is you get in a plane and you fly around the world and it uses like a lot of, not quite real time, but it uses satellite data in order to like reconstruct a 3D model of the world, basically in reasonably high fidelity, not incredibly high, reasonably high.

9:51Do you think Atlas will eventually be able to, like, what will it take to get Atlas from where it is now to able to dynamically generate content quickly enough to do what Flight Simulator does? And like, is this even like the right tech tree for that? Imagine like Google Maps in this world, like you could just traverse like any space basically. Is it kind of overkill though? I mean, I think it'll happen sooner than you expect, right? And that'll happen in one of two ways. Like there's one sort of, And actually, you know, Atlas was built to handle both of those tech trees. One is the explicit 3D route, right?

10:24Like maybe we're generating a Gaussian scene or a 3D point cloud scene and then interpolating that on the client. That's something Atlas can do because of its native 3D outputs. The other version, like I said, is just directly streaming the frames, right? And that's something that I think we'll be able to do pretty soon. Okay. I'm also really curious about reel to sim for robotics. So, you know, taking like a reel video and then building a simulation. So how is this useful? and how do you see this possibly being, like helping evolve the field? Yeah, so I think there's a couple different ways that this is useful.

10:57One is for evaluation, right? Like you definitely want to evaluate robotics in the final situation, in the final environment where they're actually going to end up, right? If you're going to have a robot working in a particular factory environment with a particular type of machine in a particular environment, like you want to make sure the robot works exactly in that intended environment. So there's a really useful role in simulation at that like last mile, that last mile part of robotics training and evaluation where we build a simulation that matches the final use case as accurately as possible.

11:26Then even if you've got your big pre-trained robotics foundation model that knows about a lot of robotics actions in general, then you could either fine tune that thing directly in that final simulation that matches the end application or even do evaluations to make sure that it's gonna work. Yeah. Right, so that gives you like a different avenue than you could just collect a ton of demonstrations of a robot either working in that space or a person teleoperating the robot in that space. But simulation gives you a different angle to simulate those environments. Yeah, interesting. So is there a world where like, you know, you have your like general pre-trained robot, but then you always need like a realtism of a specific environment.

12:02And could that get us basically to like a working like general robot faster by scaling this? Maybe. I think there's a couple of different tech trees that people are working on. One is this notion of fully in context learning, right? Like maybe I've got a robotics foundation model, then I can kind of demonstrate the robot once how a task should be performed. And that's enough for the robot to figure out, like stick that demonstration in the context. And now the robot can figure out that task. Another version is more this, you know, what we're calling reel to sim to reel. There, maybe I've got my pre-trained robotics foundation model, but I want to adapt it to this like particular environment, right?

12:35Like this studio, like this table, like moving this microphone around. Yes. Then there's a version of that where you could come in here and take just a couple of casual videos with your phone or a couple images with your phone, use Atlas to reconstruct the space and now stage all kinds of robotics interactions in like this studio space in particular. And if you could lower the barrier to doing that, like literally you could do it in five minutes to like take five photos, upload it to Atlas, have a general simulation, maybe describe in natural language what you think, what kind of task you want the robot to do, have an agent go build that simulation for you, fine tune your thing, like RL fine tune your general purpose robotics foundation model.

13:12And now you've got a robot, maybe in the span of a couple minutes even that could come in and is perfectly adapted to this space. Yeah, like a very fast onboarding. Exactly. Yeah, that'd be great. I'm kind of curious about all the way back in 2023, 2024, when the original version of Sora came out, opening eyes paper on it was called like video generation models as world simulators, which is seems like very similar to the vision of world labs. And then, you know, they did Sora as that. They did Sora 2 basically as like a TikTok clone that didn't seem that interested in the like fundamental research direction of video models as world simulators.

13:52It seemed like a consumer product instead. And then they dropped Sora and now they're not working on world models at all. So why do you think they decided to drop it? Well, I don't want to comment on OpenAI in particular. I don't really know much about their internal strategy or discussions. But our thesis is that world models are this general horizontal platform that can be used to go after a lot of different applications. And there's going to be a ton of value unlocked there. I think there is maybe a sense of a bit of innovators dilemma, right? Like you're sitting on LLMs and you've got this as a really powerful thing and you're leading the pack there.

14:22It doesn't necessarily make sense to try to invest in that next thing. But, you know, for us, we think that there is another category of models out here. There's a ton of opportunities out here and we want to be the ones to go and grab them. Yeah. Do you think that there's like, I'm curious about the mass like consumer applications of world models are the first ones. I feel like VR or AR feels natural and then video games feels very natural. I feel like there could be like we could start shifting in a direction where everyone's like vibe coding video games and world models could be a part of that.

14:53Whereas right now we're just like vibe coding websites. We're already kind of doing that. Yeah, I think we're moving towards that. Do you see like world models being like that rapidly deployed so that everyone can kind of play around with them? Yeah, definitely. And that's what we want to do. So we want to get this, that's kind of like fits into our thesis about these models as horizontal general things, right? So if we want to expose these models to a lot of people, let them be consumable by humans and agents alike and let them interoperate with other tools, right? So if we can provide a model that has all these general capabilities in simulation and reconstruction and generation in explicit 3D and 2D, and then we can have an agent, a coding agent go and stitch all these capabilities together, then we can let people really quickly iterate on really cool experiences.

15:34So I'm really excited to see once we get this into people's hands, what kinds of cool experiences people will build with it. Right. Yeah, I'm also curious how you think they'll naturally work with LLMs. So are we all going to basically be like adding another subscription to our AI stack? And like, do you think like agents are very easily able to like work with world models in theory? Yeah, definitely. I don't see why not, right? Like, once you offer something on an API, like, you know, you get certain things in, certain things out. Like, they interoperate with other pieces of software. Like, I think we've always wanted to build these things for both humans and agents to consume.

16:09Mm-hmm. Yeah, I do wonder if, like, GTA 6 is just the last great piece of, like, human-written software before, like, all future video games are kind of generated on the fly with stuff like Alice. I don't know. I disagree with that. I think there will always be like human in the loop with, because video games are an art. I'm not saying there won't be humans in the loop, but I'm saying like - But like fully hand stitched to like video game? Yeah, like, you know, you won't need to like, you know, physically place like millions of different assets in a game engine or anything. Yeah, but this is just the evolution of technology in general, right?

16:43As we see like, you know, from the early, like some of the early Disney animation films where they go and they hand paint every cell, like that was already a new type of technology that enabled new, a new way to make movies and creative experiences at that time, right? Or some of the early Disney films figured out how to sync audio and video for the first time. And that was a really transformative experience. But, you know, new technologies come out and creative people always find really interesting and innovative ways to enable new experiences. So moving from, you know, live like live action, silent films to like animated animated early Disney films to, you know, the kind of computer generated films like, you know, the Pixar era of CG.

17:18Yes. you know, into AI. I view these as just new creative tools and there'll always be people, like creative people behind the wheel that will be interested in creating new experiences. Yeah, I can imagine like a Pixar, but in 3D, like in like a 3D navigable space sort of thing emerging. Like I think this is going to be really good for creatives specifically. Yeah, I think so. Right. And what we've wanted to build on all these things is this aspect of creative control, right? We never wanted to be building these generative slot machines where you just like pull the thing and hope you get out a good generation.

17:49or you don't feel like that generation was yours or that you controlled it or that you knew what you were getting out. Like we want to build these tools that have really deep control, right? Like whether it's 3D control, whether it's text control, that you feel like you're a director in the thing guiding this model and deciding exactly what it outputs rather than just like the model spits something out and maybe I can stitch it together after the fact. Yeah, totally. How does Atlas perform with like difficult physics stuff? Like for example, if I wanted to do a simulation of a black hole or something, like would it be able to say like, Like, oh, this is like a scenario where like normal object collisions don't really apply.

18:21You'll need to do like spaghettification or whatever. Yeah, black hole is a good one. I haven't tried that. Maybe I'll try this afternoon and tweet it. You should do that. You should. You should. I'm actually really curious to hear. Yeah, my guess is it probably won't do something super exciting for black holes. We'll see. We'll see. Yeah, is the value... Or what about just like hair or something? Hair is pretty good. Like, I think one of the most amazing applications we got out of this thing that was actually a bit of a surprise to us how well it worked were these bullet time shots. So we can do this thing or you can take like a couple of regular iPhones and stick them on tripods, just like three iPhones, take synchronized capture of some event happening.

18:56And then you can use Atlas to do these bullet time reframing shots, like freeze frame fly around. And this worked better than any of us were expecting. It's completely insane. So you have one of our headliner examples there is a strawberry dropping into a bowl of oat milk. And then we can like, as soon as the strawberry hits the milk, the milk splashes up, We freeze time and like go in there and fly all around it. Like it's just incredible. This works from just three input views. Yeah. I'm also curious about that, like added capability of being able to like adjust where the camera is. Does that help?

19:29Like what part does that help with? I can imagine like that helps if you're trying to make like a movie or something. But yeah, how is that a useful capability? Yeah, I think it's very useful as a, so I think there's a couple of notions, places where 3D is useful. One, like we already talked about the output side. Like sometimes you want to have that explicit 3D output to plug into a workflow or a game engine or something like that. But I think there's an underappreciated value of 3D on the input side as well, right? Like even in a situation like a VFX shot or something like that, where the ultimate output is 2D, it's useful for the model to offer these 3D control surfaces to let the human director figure out like what kind of model, what kind of scene the model should output.

20:09Right. So in this case, you know, we can do this, you know, there you can control like precisely where does the camera move? Because I don't just want to write in text like pan in, pan out, truck left. I want to be able to grab the camera and make it like zoom exactly as I like. And there's really no way to describe that explicitly in text. You really want to be able to just like grab the camera and make it do exactly what you wanted. So Atlas has camera control as a native input type, right? It's multimodal. And one of those modalities is cameras. Yeah. So if you think it wouldn't do a Blackpool correctly now, like what more would be required in order to get it to handle all kinds of edge cases in physics?

20:46Oh, I mean, so I think there's a couple different regimes of physics. So like anything that's like roughly Newtonian, like that was in the pre-training data. So anything that's like roughly Newtonian that doesn't work at is probably just a quick SFT away. If you want something that's sort of non-Newtonian, right, like black hole physics or like, you know, microscopic stuff, once you're down in the quantum mechanics realm, I think then you'd have to rethink a bit the architecture and the data side. But the black hole example is really interesting. I'll have to give this a try. Yeah, yeah. I would definitely be very curious to see, or like at a nanoscale.

21:19Yeah, that's probably not going to work. Not yet. But one of our dreams we joked when starting the company is you want to build these 3D world models. And how cool would it be to use a 3D world model to design a chip that we then use to train the next generation world model? I was literally just thinking that. Yeah, yeah, yeah. Because we're talking about quantum stuff. you have your like semiconductor also to me what feels very natural is that like the natural modality for navigating these and even prompting the world models is just going to be VR basically because you can like I guess do WASD and then like move your mouse around to traverse a world but I think eventually you will have to basically like well not just eventually we've had we've had marble in VR for like almost a year at this point it is that is a really transformative experience if you guys haven't tried it right I haven't tried it then you were like really like stick on the headset and like I'm in this complete AI generated world that I created.

22:10Like it's quite amazing. Yes. Yeah, yeah. Yeah, absolutely. Yeah, I can't wait to like see all of the amazing stuff that people are going to do with Atlas once it's fully out into the world. Yeah. Yeah. Justin, thanks so much for coming on MTS. Yeah, thanks for having me. Absolutely. This was awesome. This was really awesome. Yeah. Thanks for listening to this episode of the A16Z podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts and Spotify. Follow us on X at A16Z and subscribe to our sub stack at A16Z.substack.com.