Neuralangelo Unveiled: Nvidia's Groundbreaking AI for Transforming Videos into 3D Worlds

0:00AI researchers over at NVIDIA have figured out a way to turn a video into a 3D object. and they do this using AI. So today on the podcast, we're going to talk about this project, why this is important, what the implications are, where this is going, and also some other interesting projects we're seeing in this space right now. So the first thing to say is that this project is called Neuralangelo. It's a funny name, obviously. But essentially what they can do is they can take a video, and this includes video just shot on a smartphone. So a video on a smartphone, for example, you would go in your backyard, walk around with your smartphone, you know, looking at your backyard, and they would be able to take that video and turn it into a 3D rendering, where they could essentially get a 3D map of your entire backyard, the shapes of the trees, the grass, the buildings, and anything else that you have.

0:48And they could do this for any other space. There's a lot of really impressive implications, and the way that they describe this in their blog post is they say it's like when there's a sculptor and they have a piece of stone and they're, you know, chopping in little bits into it to kind of get like the, when I guess they're trying to go off the whole sculptor thing, because it's called Nelangelo, off of Michelangelo. But anyways, they're saying, you know, as a sculptor kind of breaks down the stone from different angles and gets the different depths and hues the stone, blah, blah, blah. That's essentially what their AI is doing, where it's taking this footage, and as the footage switches angles, so as you like kind of pan a video around, it's looking at all of the objects from these different angles.

1:29Now, this is really interesting because this reminds me a lot of what Tesla did with their self-driving technology where a lot of car companies are saying, no, you need radar, you need LiDAR to be able to really detect the distances and to have these things maximum safety. and Tesla said no we think we can just use a camera similar to a human's visual eye to be able to you know determine the distance between things etc etc and would appear that the AI that NVIDIA is releasing right now validates this theory or this thesis and that they're they're actually able to get the 3D shape of objects just by looking at them from different angles of a camera and being able to put that into AI.

2:12So they have a bunch of examples that are impressive in their blog posts. One of them is them kind of going around and looking at a sculpture and it's able to show all the different angles of the sculpture and then create it into a 3D object. They're also, they go and do buildings, which I think is really impressive. And I think one really interesting area that this is gonna be used in is with drones. So you could fly a drone around a building and or around a venue, etc. And based off of the footage of that different venue or that different object, you know, they flew a drone around a truck and it was able to reproduce a perfect 3D model of that truck where it has all of the different specifications.

2:57It has all of the different dimensions of that thing. So a lot of people are saying for like use cases for this. Number one, a lot of people are talking about video games, right? They're like essentially if you wanted the map of you know, you wanted Palo Alto, California as the map of like one of the levels of your video game You just walk around or you fly a drone around a bunch of different areas and it could create a 3d map of there or any You know other different location when it comes to reproducing, you know models There's a lot of different implications for this. I mean you could think of a version I mean, I've seen a lot of people where they like will have a sculptor go and create like a you know, a sculpture of like an object, and then they got to go convert that into like a 3d model, scan and do different things.

3:40But like with this technology, you could just take a quick video, like, you know, someone goes and makes like a some sort of product demo, and then you go take a video of this product from a few different angles. And all of a sudden, it actually goes and generates the 3d file. And it's very easy to quickly go in and start, you know, putting that into an AutoCAD software technology and go use that. So I think it's a really powerful technology. and someone recently on twitter was sharing uh an image just to say how far this this uh you know 3d um modeling ai has gone they show they showed one picture which is said was from three years ago of a tractor and it was like i don't know if you've ever seen um someone print something with you know a 3d filament like print doing 3d printing but like oftentimes if it's not a very high quality machine you can tell there's like lines where all the plastic is it kind of gobs up There's it's just like very rough texture.

4:32And that's kind of what the 3D model image they said from three years ago was. And then they showed what it is today. And they have this like 3D shape of Michelangelo's carving in stone. And it's just so much more advanced. The texture is really impressive. And I think one of the most impressive things they say that this can do based off of NVIDIA's blog is they said that it is very good at detecting textures. So like the texture of a shingle or the texture of the asphalt on the ground or the textures of blades of grass. This video, it can literally get down to the very finite textures based off of what it's seen, which is really impressive because from a video's perspective, that doesn't seem like something that would be very easy to do.

5:20And it's going to make things ultra, ultra realistic, I think, for video games. I also think for whatever Facebook and or Meta and Apple decide to do with their VR headsets and VR and AR, it's going to make things ultra realistic. So I think that it's going to be really, really impressive. NVIDIA on their on their blog, they recently said the 3D reconstruction capabilities and Erlangelo offers will be a huge benefit to creators, helping them to recreate the real world in the digital world. This tool will eventually enable developers to import detailed objects, whether small statues or massive buildings, into virtual environments or video games or industrial digital twins.

6:02So this is really, really powerful technology from a corporate side, from, you know, video games. In their demo, they, you know, showed just a video of going around a flatbed truck, a video going around like a wedding venue, a video going around a sculpture of and of a of the uh nvidia bay area campus and all of those things just from a video going around were instantly created into 3d um 3d shapes i mean you can think of a lot of different implications including like in the military being able to fly a drone around perhaps your enemy's base bringing that back and instantly having that turned into a 3d model um for you know reconnaissance or whatever um a lot of implications i think even with the war in ukraine right now we're seeing a lot of, you know, I've seen a lot of footage of Ukrainian soldiers using drones for all sorts of things.

6:53And I think that this 100 % will be something that is integrated into that entire industry, that sort of technology, this will be integrated into, you know, architecture and into video games in a lot of different areas. So I think it's really powerful. This is really cool for me in particular, about a year ago, about I was working on developing a VR app. It was a meditation VR app for one of my software companies, which is called self pause. It's like a, it's like a positive affirmation meditation. And now it's more of an AI life coach application on iOS and Android. And we were developing a, we're developing a virtual reality application for it, where you could go, you know, put on a VR headset, you could select some sort of location, right?

7:38Like a beach. And you could listen to your meditation where you're like completely surrounded by nature. And the idea was, you know, regardless of where you are, you could be, you know, in your apartment and you could go to anywhere in the world and kind of have this virtual reality meditation experience. So that was the idea. But the reason I bring that up is because it's really interesting while working on development for that application, coming up with those virtual worlds and those virtual environments and looking at, you know, how those are created was really eye-opening for me. So essentially what you have to do when you're creating these 3D VR world is that you, a lot of people have done it where they essentially just take a camera and the camera takes like a thousand pictures in a circle, right?

8:22So like you'd stick it in the middle of the beach, it takes a thousand pictures in a circle and it would take a picture above it and below it. So it gets like a 360 degree view. And so if you plug that into a VR headset where you put it on, when you turn in the VR headset, you're seeing everywhere around you. You turn around, you see what was behind the camera, in front of the camera, above the camera, below the camera. So that's how you do, or that's how you like, that's kind of the most basic version of how in VR, you create these 360 environments. Now, the problem with that is it's just an image.

8:53So it's the equivalent of, you know, you're just seeing a picture from a lot of different angles. And that's not a very immersive feeling. So what the better AI VR technology does is it creates like 3D shapes and objects all around you that make it feel a lot more realistic. Now, the problem is creating ultra realistic, you know, blades of grass, for example, is incredibly time consuming. It's incredibly, it uses a lot of like GPU usage and all that kind of stuff, a lot of usage of like your hardware. And so it's not very realistic to have like individual blades of grass that are like have every single detail of the blade of grass on there for example um and so what people did was they would create shapes so for example if i was to create a palm tree i would create the 3d shape of a palm tree in one of these virtual reality environments and then i would put the the skin which would be the image so like i would essentially take the a picture of the bark of a palm tree and i would superimpose that over the shape of the bark uh i'd take a picture of like the leaf the texture of a palm tree leaf and superimpose that over the 3d like palm fronds on the palm tree so you get the idea it was like images superimposed onto 3d objects that's how currently you make the most realistic uh vr and of course it's going to get better where it's just more and more detailed um but a lot of that was very very difficult to come up with and for us when we were doing our vr application we just went with the most basic one which was just doing um you know 360 images and some people do 360 video as well.

10:26So I think this is really powerful technology as you look at what Meta and Apple are moving towards with their VR AR headsets. And now developers and other people will be able to create really realistic 3D environments of virtually anything because they're gonna have this ability to take that picture and turn it instantly into a 3D model. And then I think very quickly they're going to add on the ability. it wouldn't surprise me if this was nvidia they're going to add on the ability to take that same video and take the um the textures and the look of the objects from that video and be able to superimpose that on the 3d shapes that they created because currently when it does the 3d shapes um of the of what you're looking at it's just like a it's like a white 3d object so it's just like you know if you were to if you were to um you know do a truck and then you were to 3d print it with a 3D printer.

11:19It's just like with white filament, it'd just be like the white 3D truck, right? But I think in the future, they'll be able to superimpose the actual image and look and texture and colors of the truck. So when they put this into a 3D environment, it looks completely realistic. So really interesting technology, really impressive. I do want to give one other shout out to a company I recently saw this week called Blockade Labs. And I believe their product it called skybox and essentially it is a ai chat gpt text interface that allows you to create 360 3d world environments from a prompt so i'm looking at one right now on their website that is like a desert it looks very similar to arizona where i live and there's all of these incredible red mountains and 3d all around you you're able to kind of like spin around and look at them there's a sunset but um what's amazing is it says you know dream up your world there's a text input box where you can type in right something like you know tropical island with a river and a stream during sunset waves all around me you click generate and it will generate that 3d 360 world um which you could instantly go and like i was talking about earlier that would be perfect for integrating into vr apps vr games um and making really really powerful technology that makes it feel very very immersive so i think that's an incredible technology that came out this paired with some of the 3d environments that were seen out of nvidia we could literally bring you know the real world into ar vr in a way that's not just images it's complete 360 experiences you could go for a walk around your entire neighborhood essentially in vr where everything is completely 3d all around you just really really interesting implications that i'm sure we'll be able to continue looking at into the future.