Shap-E Chronicles: OpenAI's Text-to-3D Revolution

0:00What can 160 years of experience teach you about the future? When it comes to protecting what matters, Pacific Life provides life insurance, retirement income, and employee benefits for people and businesses building a more confident tomorrow. Strategies rooted in strength and backed by experience. Ask a financial professional how Pacific Life can help you today. Pacific Life Insurance Company, Omaha, Nebraska, and in New York. Pacific Life and Annuity, Phoenix, Arizona. Welcome to the Chat GPT Podcast. I'm your host, Jaden Schaefer, and today on the podcast, we are going to be going into some breaking news and a new product that OpenAI has recently launched.

0:41So the new product is called Shap-E, and essentially it is a text to 3D model creator or generator. So you put text in and it can create a 3D object out. So today on the podcast, I know that with the whole 3D and AI space, there's a lot of technical jargon I came across as I was trying to research this topic. And so I'm going to try to break it down to you in a way that is easy to understand no matter what level you are at in your AI understanding and 3D imaging understanding. Hopefully I'll be able to draw a little bit from my background helping develop and work on VR applications and kind of some of the 3D assets we had to build there.

1:22And it's going to be really interesting to see how this differs from a previous tool that OpenAI created that was also a 3D generator. and apparently this is the latest and greatest updated version of that. So without further ado, let's dive into it. But really quick, I did want to let you know that if you want to watch the video, because we do have some video clips of some of these 3D images moving, you're going to want to watch this on Spotify to see some of the visual breakdown. But otherwise, I'll explain everything, and Apple Podcasts, which I think is 70 % of you, will be just fine. So what I did want to say right off the bat is OpenAI has been working on a lot of different tools that they are currently milking us for.

2:05And this tool here, our gracious overlords, open sourced and gave it to us for free. So there is a GitHub with the repo to the code for this that anyone can go and access. And they have examples of what they're kind of what you're able to generate in here. And I will say though with this github repo and with this whole open source thing of course It's a super nice token for open AI to create this because number one. It's free number two You don't have to get an API you don't have to pay them for anything But that being said the documentation is definitely lacking as another Blogger Tom's hardware noted he was talking about how it's really really hard to actually get this thing up and running Not a lot of documentation open AI is just like hey you run it on Python So it is difficult to do don't you know don't think that this thing is super easy, but it is pretty impressive so how this thing actually works I guess a really layman's Descript a really simple layman's perspective and breakdown on how these AI 3d Assets are generated is a couple steps.

3:09So first essentially what happens is you input a text, right? So you could say something like a chair that looks like an avocado that would be your text what what this is actually doing is converting that text into an image with something like dolly 2 and that could be why some of these icons look so I don't know silly or basic is because dolly 2 as you know is like way inferior in my opinion and to like something like mid-journey so it's a little bit more basic but um essentially it's going text to image and then it's going from the image and it's running it through an AI that will turn that image into a 3d object so if I was saying something just like a gray rock let's do something basic it turns text to an image of a gray rock it turns that a that image into a 3d object but now it's just a 3d blob then it has to essentially put a skin on top of that 3d blob or a texture so it actually looks like you know the texture of a rock where its crevices and cracks would go So this is really similar to what is currently being, so the reason why I think this is important is because it's really similar to the strategy that's used in VR.

4:18If you've ever used a VR headset, you know, you put it on, you're in a virtual world all around you, there's 3D objects, and they all have different textures. how those 3D worlds are created is essentially that 3D objects are created inside of those VR environments and then a skin is placed on top of that that object like a texture so that it looks realistic and if you ever take the time in a lot of VR games a majority of them because budget constraints and other things and like for example if you're walking through a meadow and you look down at the ground at the meadow you're going to notice that it looks like you're in a grassy field but when you look down a lot of times it's a flat field with like the image of grass just imprinted on top it's just the texture so um and the more advanced ai games or simulations or vrs will have like the the 3d objects broken out better so like you have blades of grass and different things like that but even then the individual blades of grass would have like the imprint of the picture of a blade of grass on top of it.

5:21So all this to say, I think this is a really big advancement in how these 3D images are being generated because it's going to be really useful as people are trying to pull these into different tools and come up with real world applications. So in their examples, they have a chair that looks like an avocado, an airplane that looks like a banana, a spaceship, a birthday cupcake, a chair that looks like a tree, a green boot, a penguin, an ice cream cone and a bowl of vegetables and all of them look like very fundamentally like their object not a not a very high quality but I can only look forward and kind of project forward to imagine if this wasn't open AI throwing this thing into something like dolly 2 to create this imagine if imagine when not if this is only a matter of time when these images and objects are going to be something generated out of let's say closer to mid journey where you say you know like imagine a forest with a boat and an elf warrior and the sun streaming down and it's going to create that image and then it creates a 3d and then all the objects in that whole situation are all 3d that will be able to be ported into video games and movies and all sorts of really cool i think online graphics and things that we'll be able to generate so really crazy environment and technology that we're going into.

6:40And I think what is important to know with all of this is that now that they have kind of released and unlocked this new technology, it's important because it's able to actually be opened on different things like Microsoft Paint 3D, and you can even convert them into an STL file that you can output on some of the best 3D printers. So you can create these objects. In the now, in the current moment, you can get it to create something, and you can 3D print it, which is cool. And eventually, I believe that these will start getting built and created and used for a lot of different purposes, video games and graphics online and different things.

7:28It is interesting. I want to show you a couple examples of what Tom's Hardware.com was actually able to generate or talk about them. So he said a shark and it generated an image for him of a shark that looked pretty decent but when he opened the PLY in Paint 3D, so that's a file type, it pretty much just seemed lacking as far as any of its like texture on it. So it's just a very basic, I mean, it's the shape of a shark, but it doesn't have a lot of the, um, it doesn't have the texture we were talking about earlier, right? These textures that get imprinted on top of it. So that was an interesting experiment that he ran.

8:10He also did one asking it for an airplane that looked like a banana. Um, and he asked it for, I think he did one that was, uh, a Minecraft creeper and got something that I got something that was a gif correctly colored green and black um that was in the basic shape of a creeper however the real Minecraft fans he said would not be satisfied with it as it was too messy of a shape to 3d print in any case so this thing isn't perfect it does uh create some pretty good uh 3d renderings of dogs although a lot of times once you move it over to something like Microsoft Print, the resolution would appear to be getting a little lower, and it's also pretty messed up.

9:03Once he opened it as a PLY3D file in paint, you probably can't see my screen if you're listening on Apple, but pretty much it comes out as a pretty good looking corgi, and then when he opens it in an actual tool that would make it useful it just kind of looks mutilated like something you'd expect to see out of dolly 2's images but now for 3d objects so i think that there's a long ways for a lot of this to go there's a lot of people that are going through different blogs and talking about ways to improve this the thing is this is open source so at this point it's kind of like open ai has released this to the public to the fans and now it's kind of up to us and everyone else to take it perfect it integrate it with things and work on it.

9:45There's going to be a lot of people that fork it. And it'll be interesting. And so at the end of the day, of course, it's not as polished of a tool as we'd expect OpenAI to come out with if this was something that they had an API access to and they were charging for. And maybe they hit some roadblocks and decided they had other priorities, so they decided to release it for free as an open source thing. But I would believe that there's going to be a lot of developers that will help to develop this, get some progress on it, And perhaps OpenAI will continue to release another version of something similar since this isn't very far off from something that they just released back in November when they released PointE, which was kind of the original text -to-image product that they created.

10:28So I think though the objects might look kind of pixelated and rough at the moment, there's also the other limitation that is only really capable of producing objects with single input prompts and really simple attributes. And so pretty much it struggles to find multiple attributes. And even their paper that they came out with pointed this out. So you could say like an airplane that looks like a banana, and it's merging banana and airplane together, but anything beyond a couple attributes, it's going to struggle to really properly create that. So it's going to be really interesting to see how this goes, how this unfolds and the advancements.

11:07You know, a lot of people looking at this now might say, oh man, look, you know, it's not very useful when you transport it to another software or it's not perfect or it's got glitches or it's super grainy, whatever. And like what I've been saying since the beginning is every time a tool like this comes forward, It might seem like a small insignificant step, but it is a massive step forward because just in order to get it there, now all we're working on is tweaking and improving. And before you know it, these things are going to be, you know, photorealistic. You'll just be able to generate a full-on, you know, realistic-looking 3D dog that can move and walk and be imported into different software.

11:45So it's not very far off, and it's going to be very crazy to see how this advances. So thanks so much for listening to the podcast today. I really want to make sure that if you're listening to the podcast and you like it, if you could do me a huge favor and go and rate it wherever you listen to your podcast, I'd really appreciate it. Mostly I get positive reviews and it's really encouraging. If you have feedback, I love it. Also, feel free to join our Facebook group and send me any feedback you have there. It really helps me to improve the podcast as this is something that I'm passionate about and I'm just doing for free at the moment to kind of help people out.

12:20So really appreciate you being here. Hope you have a wonderful rest of your day, and I will see you next time