Runway Aleph Changes VFX Forever, Adobe Harmonize, Wan 2.2 & More! | This Week in AI for Filmmakers

0:00All right. Welcome back to Denoist. Good to see you again, Addy. Welcome back, Joey. All right.

0:18So The one that's been overtaking my feed, Runway's Olive. Olive. Yeah, this one's pretty wild. So basically now, instead of fully generative scenes, you upload an existing video and you just tell it what you want to change and it changes it. This is exactly the direction AI tools should go. It's not about generating once in, you know, what is that saying? You set it and you forget it. It's not that. That's not how a production works. Production is all about iterations, changes, refinement. And this is giving you access into that world. Have you seen some of the demos and some of the examples?

0:52Yeah. So first of all, any sort of traditional green screen workflow, I think this jumps right into that territory. Yeah. Relighting a lot of demos with relighting, a lot of traditional VFX stuff, adding objects, changing clothing, removing cars, removing people, changing weather. I mean, this is so groundbreaking and so powerful. It's just we're not covering it to its full extent here. and I don't think we can because we're not like running it and showing it to you. I'll show some clips. I got some clips up here. Yeah, I'll show some clips. Yeah, we got one of the woman walking and she's just changing her hair, changing her eyeglasses, changing her clothing.

1:26This covers so many aspects of traditional VFX, right? So you just said relighting. I think it'll do green screen, chroma key, background replacement. I don't think you need green screen. I think the joke was I saw them, one of the posts with a shot of a boat, like a real life, you know, stock footage shot of a boat. and then the joke, they made it a joke where it's like they put the boat on a green screen and that made it look like it was like a fake behind the scenes shot. She was shooting the boat, but it was a real life shot of a boat. Yeah, right. It doesn't even need the green screen. Joey, what are your thoughts on, I mean, the inevitable question, job replacement and where people are in VFX?

2:04I mean, yeah, that's where this, well, let me go to some of the technical specs first because let me pin that. That's gonna like, I feel like, you know, if you're a professional VFX artist, like there are a lot of you'll see this and be like that's definitely not usable for us right now yet sure but right now currently it only supports five second output and i think the output's 1080 and all of the other limitations with color space and color science with ai generated outputs so that's still a limitation for now obviously like this will like what we're talking months maybe yeah i mean i yeah i have not seen i can't think i've seen any ai generative outputs that are like EXR, like have the quality that you would need for a professional.

2:48I think, yeah, it goes back to the total addressable market, TAM, as it's called in the business circle. If you look at TAM of, you know, consumer goods, hard goods, I don't know, something like energy drink, right? Like the TAMs of things that are on store shelves are so much bigger than like the film industry. uh yeah you're talking about like you know a few billion dollars versus hundreds of billions of dollars and so the ai companies are building things that are addressing tams right so like our film our endearing market is so small and it takes so much work to build like an exr output or you know 16 bit in and out yeah and have to train on that yeah and then where's the return on the other end of it to go to level that up yeah yeah so that said about the original question of you know vfx artists well i mean you need to get on this like yes i feel like it will i mean obviously it's going to displace a lot of stuff um and like more people can or someone could do less but i still think there's going to be a lot of gaps i mean the stuff that's coming to mind now is the demos we're seeing well first off you know the five seconds that's short and we're just sort of seeing one shot demos i haven't really had a chance like i think i just got access to it i made a shot i'm trying to just find what clips i randomly had that were uncut and uh i had some shots walking around the airport and so i added an elephant in the airport it looks kind of good yeah it's a little blurry yeah i mean any any vfx artist in their right weight in mind can be like they can dismiss it quite easily yeah but i'm saying don't do that because you cannot look that way that's yeah that's where i mean yeah i interrupted my own thought but that's where i was going with this where like if you're a vfx artist you can see this can get a good first step but like there will still be gaps and there will still be anyway the thing i was saying with the testing was we've seen one shot tests like how does this do with the continuity how does this do with like you're in a scene you need to shot reverse shot and or replace a shirt but you want that shirt to be the same you know in every single shot yes i'm sure there are still limitations there and i think what you need to figure out is like how does this tool work and become like a prompting expert at that so that you can be the one that either gets it to the continuity space or get something from Runway to be the good first output.

4:54And then you use traditional compositing tools and methodologies to get it to level it up. But this can like save a ton of time in the start. Yeah. And keep in mind, we're also in a transitionary period. Like these tools are not final, right? Like Aleph is just the beginning of what Runway is going to release, I'm sure a year or two from now. Or a week or two from now. Right. Like they're not done. They're just getting started. So like if you are in on this early enough, you can sort of be in a really good position a year or two from now and be ready for the next wave of tools and innovation rather than sitting it out.

5:29I don't think sitting it out is a good option. No, you just got to start messing with this stuff. Yeah. Yeah. Cause I mean this, yeah. It's like when, what would have taken days or weeks of compositing work, even Cristobal, your, your favorite posted out a joke, you know, it was like listing out every single thing you'd have to do to erase all the cars and like a street shot of new york yeah and now you just type in a prompt and be like erase remove the cars yeah so like rethinking you know changing the way you think of like oh you know this was a whole process now i can just type to do what i want you need to start messing with this stuff so my advice is like start doing the research now start watching denoised and get in get in runway out of yourself like you're doing yeah no you gotta to mess with it yeah i mean you know also in the in the the vfx world like i know there's gonna be pushback or of the like you know this still doesn't give you the finite control or this still you know like in a big budget you know hollywood movie where they're going through vfx and they're going through every single frame and giving notes and it's like oh you know the shirt needs to change here this needs to change here and it's like you don't get that yet with the outputs but on the flip side of that is something like netflix's eternat right that the way they use that ai inserted shot we're going to see just more and more of that yeah and it's just like is it just going to change where it's like okay we do more stuff and this may be the pickiness level yeah goes down or the control gets better or you figure out a way to blend olive with nuke or traditional compositing to like get the best best of both worlds i i think productions around the world are not gonna wait until ai is ready they're already gonna start using it and push ai to be ready, if that makes sense.

7:08Yeah, and figuring how to put this in the pipeline. Nobody's going to wait for Aleph 2 and then be like, you know what, Aleph 1 sucks because it's 1080p, blah, blah, blah, blah, blah. I'm going to wait for Aleph 2. No, they're probably already going to use Aleph 1 to some extent and then figure out how to plus it and make it work until Aleph 2 comes up. Yeah, and so much need of these tools anyways. It's like, okay, well, where do you need to get and then what tool can help you get there? Which, you know, is usually a variety of tools. Like there's really no one shot tool that's like going to do everything for you.

7:37Right. It's knowing all of these tools. Again, we're going to small project now. And it's like, I went through like every single model to like figure out for this type of work, which models work best for the specific thing. Yeah. And these models wouldn't work best for a different type of project, but they work well for this project. But it's just a lot of like, you got to know all these models and tools to figure out. Yeah. What is the right tool for the right job? And it's going to continue to be that way for a long time, because I think it's like the tools are very specific. You know, you have your screwdriver, your Phillips head, your flathead, you have your drill.

8:06And like, it's not going to become this Swiss army knife anytime soon. Are you finding yourself using different models at a shot level? Sometimes I try to avoid that because then, you know, like we've drawn the film stock analogy, it gets, you get different looks. And so like, as much as I can stay the process now, it's been like, let me get the blocking right that I want. And I'm trying to stick with one model for that. And then there's a certain style we're trying to go for. So then I'm style transferring it with a different model, but trying to keep that all consistent. And then animating it with a different model.

8:39But then that hits hiccups because sometimes, depending on the model, they only support first frame. But out of other shots, I'm like, well, I know I want this first frame and this end frame, but this model doesn't support it. So I got to use a different model. So I've been trying to stick with models because then when I try a different model, you get a completely different look. And then it just kind of takes you out of it. So I've been trying to figure out like what model produces the best results on average for this specific project. Yeah. Which is such a interesting challenge when you're making like a 90 minute thing, you know, 60 minute thing, whatever, like shot consistency and look and feel and texture, maintaining that on top of like all the creative changes.

9:14Yeah. And for longer projects, I mean, we're messing with some stuff now, but I haven't gotten there yet. But I've been thinking of breaking that down of thinking of at least for the scene to stick with the similar workflow for the scene, maybe different scenes. We use different models or workflows, but like at least get the scene continuity and keep things kind of consistent there. Yes. Yeah. And those of you who have not caught up on our AI workflow video, we released it. What, a couple of weeks back? About, yeah, two weeks ago on. Joey covers it in detail. We even have a very specific diagram in there.

9:44Yeah. And now there is a link to the diagram because people are asking for it. So you can check out the link to the diagram. And I opened comments, which I was worried about online. No one's left anything crazy yet. But if you have any comments or any, um, yeah. To like of your own tools that I did not include on the diagram, you can publicly comment on the, uh, on that workflow to add your own tools that I missed. Yeah. One other thing I want to say about Olive that we didn't really talk about too, is you can also change a camera angle. So you can give it a shot of a video. And I saw someone pose a demo with like a car driving down a street and then being like, give me the low angle by the wheel.

10:13Give me the like aerial shot. How cool. And so there's not even just changing elements in the scene, but just changing the perspective from having a single shot. Opens a lot of possibilities and things that we're not even thinking of yet, but just kind of makes things crazier. My mind instantly goes to lazy cinematographer. It's like, yeah, just shoot it like this. We'll get it in post. Fix it in post. It has a way new meaning now. It's like, aren't you going to rent a techno crane for that shot? Why? We'll just shoot it head level and we'll just move the camera. yeah this is gonna be yeah i'll just run it at olive after yeah don't worry man we got it it's like why are you well dude this this dp is like an oscar winning dp he's on his iphone we don't need an re for this he's just running around in circles just captured everything what are you gonna do with this he's like a glorified photogrammetry artist at that point yeah fix it and post it's gonna have a new meaning uh that's crazy all right now not to you know, let Olive have all the fun, Luma AI released their own update.

11:14I'm loving the competition here. I'm also not surprised because it seemed like with Modify Video that they released a couple weeks ago, they had the capability. Modify Video was you would give it a driving video and then you could update the first frame or kind of give it, tell it what you wanted and it would sort of change it. But now you don't have to have a first frame or have like a driving modification. Similar to Olive, you just upload the video and it's now called Modify with instructions. And so you can just chat it out and say what you want to change. And so the demos are very similar to Olive where you give it a video and just say, change this.

11:46And it changes it. You know what my dream scenario would be? Like a roundtable with like Amit and Cristobal and all the big. I wake up dreaming of that. That's what you dream about. Yeah, yeah, yeah. So no, shout out to Luma. So you and I send each other videos all the time. I sent you a John Finger video where he's sitting on a fire hydrant and then Luma Modified removes that fire hydrant. And it just goes to Cristobal's point is like, what is the existing cost and effort of something like that, you know, with traditional VFX and how instantaneous is it now? It fundamentally changes the game.

12:24And if you're concerned about quality, frame rate, maybe John's face not looking right, things like that. I think you're missing the point because there's a big shift, a big seismic shift happening with all of these tools. And I'm just so fortunate that we're living through such a revolutionary time. John without an H. Sorry, John, I forgot it was J-O-N. Yep. Where he posted a lot of Luma stuff. Yes. He just posted another fun one at Muscle Beach because he's near Venice. He's in Venice, yeah. Shooting in an empty, the empty courtyard area. It's all that. stadium area and then running a bunch of different modifications to fill it in with a crowd, to like add, turn it into a sci-fi stage.

13:03I've done crowd replacement and crowd addition work. How easy is that? Oh my God. I mean, so one of the challenges was I was using Unreal. And as you know, putting 10 ,000 humans in Unreal is a challenge in itself, right? Just real-time processing slows down. And then on top of that, you now have a camera match move. of variation in crowd animation, variation in crowd ethnicity, body type, costume. Yeah. Like, this solves that. Yeah, I will say in the commentary I've been seeing from demos online between Luma's modified video and Olive's, Luma does seem to modify the faces a bit more and Olive does a better job right now of doing what you want to do and sort of leaving the rest of it unchanged.

13:43So like in some of these demos, you can see John's face kind of warps a little bit. Yeah, yeah, I saw it. It doesn't look exactly like him, but... I think John knows it, but even with that it's so powerful the the yeah just modifying the models and improving upon it but yeah i mean still look just overall complaining about the face and be like oh well just removed yeah the hydrant below him and made him float in space or this other demo removed all the graffiti on a building and just painted it normally yeah i mean on the competition side and we're we're we're comparing apples to apples right like they're so close but um it would you say that Aleph distorts less than Luma?

14:18That's what I've been seeing. I mean, I haven't messed around with them as much, but that's what I've been seeing in some other commentary online from people that have run both and compare. I just got access to Aleph this morning. And then how much longer before VO3 has all this functionality? Something else, right. Also, I didn't want to leave this out too because it sort of went under the radar because they're so new, but Moon Valley also has, they have some features where you can shift camera angle. They have features where you can kind of do a style transfer on the video. So they also have some similar features to this they have like pose transfer motion transfer newer to the game and the social media game so like there's less awareness of it but yeah paul trello posted some demos too of like hey you can also do similar stuff in moon valley it is possible yeah uh you know you just might not might not know about it yet because runway and loom have been around for so much longer yeah but yeah wild all right next update in the uh open source video space big update one 2.2 huge released.

15:13Juan's a sleeper. Yeah, Juan is from Alibaba, and Juan is probably the best open source video model. Like, VO3 level, realistic quality, but you could run it. If you have a powerful enough computer, you could run it on your computer. It's free. Or, yeah, you can download it for free, run it for free on your computer, or a lot of the API websites like Fall and Replicate offer pretty cheap access to just calling it up and using they're much more faster and powerful computers because I ran this, and it takes a while. Dude, video generation on a local pc is no joke yeah they have two versions or they have three models one is a text video model with 14 billion billion parameters another one is an image video models also 14 billion parameters and a unified video generation model that has 5 billion parameters yeah 5b that's the one you want to roll with for local yeah i did roll with the 14b oh shit and it took 44 minutes to generate a five second video clip yeah that's not too bad honestly that's not too bad and i was running on a beefy card and it's a 720p right it was 720p yeah yeah so yeah i mean if i knocked it down to 480 it would obviously been faster yeah but it was um yeah five seven but yeah it's crazy you know to even 44 minutes to have something that just generated on the your local computer for quote free i mean obviously the energy and the and the um hardware cost but yeah i i so one of the things that i love to watch on youtube i know i i'm such a sucker for this stuff is um And some of the folks take historic photographs.

16:41Like, they'll take a photo of Cleopatra. And then they'll, like, slightly, I'm like, oh, that's cool. That's what she looked like. Yeah. So, I did some of that with, like, my family. Like, I have childhood pictures of myself, my cousins, whatever. So, I did some of that with 1.2.1. And I sent it to them as a gag. It was fun. Yeah. So, with 1.2.2, there is a slight architecture change that I want to cover. Yeah, yeah, yeah. And I find really interesting. Yeah. so the way they broke up the diffusers and the safe tensors is they broke it up as a high noise and a low noise model so you have to download both or several versions of both i think it's in pieces because each piece is like five gigabytes and it adds up to like 30 or 40 gigs sounds right and um the way it denoises is first uh when the noise level is really high in the earlier steps it'll use the high model and then at some point it'll switch over to the low model and supposedly like that's what is giving you the more superior result i just find it really fascinating because is this the trend of what we're going to see moving forward as like more segmented more broken out models of like how the model works entirely or yeah yeah like architecturally yeah yeah i just find it really benefits of you like i i've seen people comment on the controller with high noise low noise but i don't know exactly what difference that makes yeah so no idea myself so So just from like a high level AI perspective, the higher noise is where a lot of the creativity happens.

18:10Like the first formation of the image happens in the high noise steps. So like if it's, maybe if you're trying to get more of a creative output, then just let the high noise thing churn for a little bit longer and then let the low noise stuff refine and give you that fine detail, fine texture and all that. Yeah. Maybe that's how it works. Yeah. I'm also seeing in their post too, because they call it a mixture of experts architecture. Yes. And specialized experts handle diffusion, denoising time steps cooperatively. So it also seems like an efficiency gain, too, to run both to get faster outputs.

18:43Yeah. The other improvement, too, is more understanding of cinematic language. So lighting control, camera movement, composition. And it has done, and even in the demos, has a really good job with demoing of just very cinematic looking and realistic looking shots yeah and you can kind of tell what they're marketing to is that they're really aiming it towards uh you know your luma your runway your vo with this one because even like you know as generic as the alibaba website is for this it still has like the word cinematic in there and what where one 2.1 didn't have that uh okay interesting yeah so that's an update so they're like shooting for our industry yeah and i mean the quality is up there i mean look it doesn't generate audio like uh vo3 does right but uh Realistic quality wise, really good.

19:30Yeah, yeah. Especially for open source. I haven't played around with it. I don't have a nice GPU like you. It's technically not mine. Barring it from a friend. Yeah. So I just have a 3090, which is what, two, three years old now. And I'm going to try to run this at home in the next week or so. Yeah. Anything you're going to try to do offhand? Yeah. Maybe all the Roman emperors. I don't know. Yeah. so yeah that one that one's been a fun one yeah all right ideogram which uh sort of you know kind of forgotten about a little bit hasn't been in the news as much i've always been a big fan because always they they did text they did text really way better than anyone else for a long time yes that's their claim to fame yeah and i've liked a lot of their output sometimes i like their outputs more than mid-journey usually as my it's usually one of my go-to image generators oh sure but they released a new feature called ideogram character and it is basically one shot just upload a single image of you or whoever you are trying to add into your generation and it puts them in the scene uh and it's one of the best looking yeah character generators that i've seen from a single shot i'm looking at their website now i mean obviously it's going to be polished because it's on their website but i mean those are pretty impressive results from just one scene yeah i mean i've tested it too from a single image and got like really good looking impressive shots i haven't tested it as much for because it was a lot of the a lot of the demos that they show and like templates to kind of give you some template ideas are kind of portrait-y or modeling where it's like very posed, facing towards camera.

21:00I haven't tried it as much where it's like if I'm trying to stage a scene and, you know, maybe get more of like obscured or profile looking up the side of the faces. I haven't tested it with that to see if it handles that well because all the demos are like smiling at a camera, smiling at a camera. Yeah, it's really changing of the angles that test the model. Yeah, like if the face is obscured or you just see part of the side of the face or you want to put a hat on them or something else. Yeah. But look, overall, from a single image, the results are really, really good looking. Yeah, it's right after Flex Context, which also has similar capabilities, I believe.

21:35So on the video generation side, things are leveling up, but also on the image generation side, things are leveling up as well. Yeah, yeah, Flex Context I've really been enjoying. That's been a good one. Yeah. Yeah, I recently learned that like a lot of the, I was looking for kind of just controls, like doing style transfer of Flex Context and then it doesn't have specific controls to give it. But then I realized like basically you just prompt it. Yeah. Of like, don't modify the framing of the original image, but change this and then just does it. Oh, wow. Where I was trying to like look for like a denoise slider or something to be like how the weight of how much it should look for the original image.

22:08And it's like, it doesn't even need that. You just tell it, don't mess with it. You're trying to be super technical about it. Yeah, I'm trying to be like, where's the slider? And I was like, just tell it. I'm a pro, right? I'm going to use pro words. It's like, dude, just tell us. Did you see on their website, they use the Coldplay meme, the Coldplay concert meme, to do a magic fill. So we're looking at the picture of the astronomer CEO, and I guess that works. Yeah. Look, I'm for that, for pulling from pop culture references to memify yourself. And they also have a selfie with Einstein. Some Kodak camera stuff in there.

22:46Yeah, that's great. Ideagram understands social media and pop culture. Yeah. what is the picture with the guy holding a gun and a puppy oh no are we gonna get banned from youtube for showing to blur it out oh yeah you could draw matt so yeah if you have an existing image too you can mask out the image to swap out the face yeah it's doing regional in painting regional correction which is awesome uh in painting is one of the hardest problems uh for models to solve you know it's like uh don't change anything else just this little bit and then blend it in yeah so another good update for my diagram i'm glad they're back in the game because I always enjoyed them as an image generator.

23:22And they have good API support as well, which is also sometimes I forget about mid-journey because they're talking about rolling out an API, but they don't have an API. So like, and you're in the multi-tool space, they never show up because you have to use them on the platform. Yeah, I'm not a big API consumer, but I think you are, right? I mean, I've been messing with them more and comfy, but also using like FreePick more. These tools only work if the API exists. That's true. If the company offers an API. It plugs into other things. Yeah, so you need that for these like super tool all-in-one things to work or to build out a comfy workflow like it's only going to work if there's not a local model and you need to use an api version it only works with the api all right uh photoshop also super impressive feature back in the game is a adobe's back don't forget about it about us and we're offering you very useful features um big one in photoshop and beta yeah a new feature called harmonize so this one's wild and looks extremely useful and every talk i've seen online is like oh this is actually like extremely powerful and useful yeah this is what people are actually going to use yeah so basically you take whatever image you got you just take cut out of what you want to place in the scene so let's say we had i don't know a coffee cup for lack of a better example and we put it on the table and then you say harmonize and then it just automatically blends it in the scene matches the lighting the shadows uh the color perspective well you're already at your perspective like assuming you have a wide shot of something and you drop but the the object could be off perspective i see what you're saying i see yeah i think so yeah yeah that's awesome yeah and it just blends into the scene one click.

24:46This is so useful even for our industry. We're doing conceptualization, previs, and you're adding a bunch of things into your set. Now you can just do it in Photoshop and then hit harmonize when you're done. Yeah. Yeah. For art design, planning. And yeah, that's a great idea. You really have no excuses now with ideogram and this and all the other tools to have the perfect little previs. Yeah. I think back to you with um uh i remember a friend worked in one of a company that like they would just do the like 3d previs stuff for like commercials sure yeah but the previs was like the worst looking like blocky 3d characters like all chopped at the scene yeah to move it you just literally like the characters literally like floated across the scene and and it was like they would get paid a lot of money to do these like previs things i mean that's what that's terrible floor like a big company here in LA that's their whole business was that right delivering that level of previous because before that there was none there was no computer generated previous yeah right there was like hand sketches and things storyboards if you want yeah yeah yeah so like we go from that to this like yeah now there's no reason why your previous should not look good and that's a big use case too of and bigger movies where it's like oh the the generative AI stuff like sure maybe it didn't some it doesn't make sense for a final pixel but uh to like be able to like roughly make got a pretty decent version of your film to like see if it works and tested and see if you need to do script rewrites or anything like that to have an audience test with the rough version especially with the way our industry is now there's so much scrutiny in the beginning to green light a project yeah so it's like you need this more than right for right 50 100 grand you can make a decent looking version of a film to test it before you commit to 20 30 million yeah go into that netflix office with this stuff uh you'll be way better yeah yeah no reason why this stuff should look bad yeah the other demo on the harmonize is a picture of a flamingo floaty on a pool and then it just like adds the reflection and like gets the shadow the reflection everything right yeah i mean sorry adobe i gotta say that they're still they're they're limited by firefly itself right i think because i i believe harmonize and well i mean it's interesting firefly is not mentioned in this at all yeah but i i think end of the day like firefly is doing some of the generative filling and generative like harmonize which is powered by adobe firefly image model yeah so it's like they swapped out firefly for like flux or something like the results will be infinitely better because i think the harmon the harmonized technology rides on top of their in-house model probably but i mean that's probably also how because they know their model is commercially safe so like yeah i think they would open up i mean maybe they'll give you the option if you want to use other models in the future like the firefly web platform now is like being more of an all-in-one spot and it's like oh you want to bring another model sure cool do whatever you want is now it's trying to be more like a free pick kind of tool set yeah yeah inside photoshop itself they probably and also probably going assuming bulk of photoshop users maybe you're not deep in the ai space you know about the model so it's just like oh this is a feature right you don't want to know what's going on under the hood but we also don't want to commercially jeopardize you if like your company's policies are still kind of up in the air with that.

28:01So we know that the model that's using is safe. Yeah. It's like the training wheel for AI, right? Like Photoshop, you know, anything accessible and clickable in Photoshop is like a lot of artists first foray into using AI for an actual project. Yeah. But look, I think this is probably a good indicator too of the future of like what AI looks like for like mass consumers where it's like Firefly is not like a big showcase thing or the AI model. It doesn't say anything about AI. It's just like, oh, hey, this thing does something useful for me it's harmonizing that i know and i click a button and it just works yeah and i don't have to think about like how do i prompt this or anything like that yeah i think this is a good indicator of like what this like ai use in the future will look like or just does the thing you want to do and under the hood and you don't think about it for sure i mean i think this is equally as groundbreaking as when liquify and photoshop come out uh came out like what 10 years ago now do you know what liquify is there's a great and kind of like you can like nudge uh endpoints and splines without distorting the image it's like automatically like if if you're modifying a face but not distorting like yeah so like if you want to make a fat face skinny or something like that right like you'll push the cheeks in and it won't mess up the nose okay yeah so it's like hyper aware of the structure of the image and this is pre-ai right and so when they dropped liquify instantly it changed like instagram and social media overnight because everybody was using it to like have these absurdly crazy body proportions or things like that so i i think with harmonize you're going to see an equal revolution with product photography and advertisement and things like that yeah yeah you just drop it in and be like your product yeah you take a picture of your product on an iphone and drop it into a scene and then be like okay looks good there you go and then some other updates they have another beta of generative upscale so this is interesting too because it's kind of going in Topaz's turf.

29:48Yeah, for sure. But basically using generative models to higher upscale an image using, yeah. I forgot, I don't know the science of traditional upscaling. Yeah, traditional upscaling is just using algorithms and math. So it'll interpolate between the pixels. So if there's a red pixel here and a green pixel here, it'll go red, yellow, yellow, yellow, yellow, green. Just like make that up. Versus here, if there's a red pixel here and a green pixel here is letting ai do that work for yeah and it's using the base image as the like this is the original source and then just as the noise yeah and you could clearly see like it gets rid of that low uh low resolution grain and the noise from the camera which i i don't know if that's a good thing or not because then the image comes out super plasticky and yeah when i've upscaled like archival stuff i've like in my brain it's like does this look way better like it's just like look really good yeah or and i think it's weird because it's so clear or does this look too weird because it's not it's upscaling weird um i have a weird analogy for you on generative upscale and i'm going to compare it to tall towers tall tower yeah like uh like the space needle in seattle okay so as you know i just came back from both tokyo and seoul okay so tokyo has the tokyo tower which is equivalent to like eiffel tower it's not that tall because it was built a long time ago uh but because it was a old school build uh the building sways a little bit and it's kind of weird like at the top you feel like you're on a ship so the observation deck is like 250 meters it's not that tall you know and you go up there but you feel like you are way the hell up there like you're really swaying around and then you're like oh my god i'm on top of the world here okay Okay, so then we were at the Seoul Sky in Seoul, which is at 550 meters, which is like more than twice the height of the Tokyo Tower.

31:46But because it's a newer building, it was like built 10 years ago. It's rock solid. Yeah. And so you're like, oh, I guess I'm really high up. But there's no sensation. Right. You feel like you might teeter over. Yeah. So I think that's the same thing with generative upscale. I know it's a crazy analogy. It's like that little bit of grain, that error, that noise is what's giving you a sense of realism in your brain and to remove that. It's the same reason why, you know, we add film grain back into digital images to give it some sort of texture, some feeling, which, I mean, you know, ultimately is your creative choice.

32:21Yeah, I think pristine, sterile, you know, noise free is not the way to go here. yeah there's also even uh there's paper always keep coming back to with netflix because they wrote about on their technical blog how for encoding they'll like strip out the grain from the films to encode it but then they'll add the grain back in during the streaming process something i think i'm very paraphrasing what they do but it was like basically like they the technical encoding process they wanted to get rid of it but they don't want to play it super clean because it'll look too weird yes they need to add the grain back well that's uh that's the whole reason why ari cameras are preferred like one of the main reasons why people send dps love ari cameras is uh you have infinite control over the grain you insert into the image like you can even pick the grain on the camera oh i didn't realize that yeah so like you start off my camera usage i'm not actually oh and speaking of ari alexis uh they just dropped the high frame rate camera recently i think a couple weeks ago so now aria lexus can do like 240 fps is this a new camera or a firmware update i think a new new body okay yeah so 240 that's pretty good yeah that's at uh 4k it looks insane yeah yeah that's pretty good yeah man i remember you had to buy those fanta or rent those it's crazy those extremely expensive phantom cameras yeah and they would sound like this no wait they didn't sound like anything did they yeah they were they're quite allowed the phantom that i used oh yeah they had massive fans on them i remember you had to have like a whole computer and stuff attached because like they were not built for film they were built for scientific studies yeah um when you were at uh nab i remember you covered a product that like has 90 of a phantom it was like yeah they do they or they say they do a thousand frames per second at 4k something like that for like cheap yeah i mean they had it was a beta and the other cameras in the video because i think i touched one he was like oh no it's just a model it's a mock-up yeah it was uh it was like a 3d print of the camera but they had one working camera there i mean they said they're a spaceship later this year i'm yeah curious i mean his background was in scientific imaging so he yeah he had the yeah he had the the goods so like i completely could do it okay uh yeah i'd be curious to see when he comes out because yeah also that would be crazy because the pricing was i don't know yeah it was like black magic pricing yeah crazy pricing and for something like that to compete with the phantoms which those things for like 100 grand five figure six figures easily that'd be awesome yeah also i think a good copy a good a good buffer against ai generated stuff where it's like oh okay well let's just go shoot really crazy stuff in real life that people are probably gonna say is that ai generated but no we shot for real well i think imaging is now more important than ever because it it is the input into all this stuff right like so the better your images are going into an ai system the better results you can expect i guess it's the same with vfx too but you know in the past um we've had a lot of hands on the frame to get it to a place where you need it to be the whole premise of ai is you don't need a lot of hands to get it to the place yeah so then it comes down to like what is your source image how good is that yeah all right next up hanyan 3d world model 1.0 so they released a new a world model yeah product well well i think world models sort of a it's not a world it's a 3D generation model yeah so you could generate a 3D space text or image to 3D it looks similar to kind of World Labs demo that I saw a while ago where it creates like a 3D sphere and you can sort of move around it in the demo it seemed like you could only move around a little bit but it also generates the meshes and textures so you could bring it into Unreal or Blender or something else yeah yeah I'm kind of skeptical right from the beginning because it's not just about generating the geometry or the mesh but how what quality the mesh is yeah how segmented they are or how grouped they are and as you know modeling and world generation is such an art uh for sure for sure yeah yeah above my head but uh yeah before to do it well i'm interested well first off this can run locally so that's cool cool i'm curious about this stuff not necessarily from a photo real or highly accurate version but more like generating a 3d space for those use cases i've talked about before of like maybe filming a scene like in a virtual production kind of space where you want to have reverse angles and have like a consistent background and it's like okay if you have something that could quickly generate a 3d environment that looks kind of consistent it gives you some parallax that could be a good lift yeah for virtual production i absolutely see the use case here however for games i think it's far more complex yeah to move around yeah you need yeah like uh game maps are designed to a t right like yeah that's yes because then that's also like Like you're not the director where you can control what angles we see.

37:02We only need to worry about what we see. Users need to go anywhere. Yeah, and also like compute resources, memory size, all that stuff is so like precise, right? Because you know, in a given game, you're given, I don't know, 100 gigabytes for the whole game. And then each level comes down to like two gigabytes. Like you have to fit the geometry, the texture, the players, the props, everything in there. And this is not going to give you that level of control, I'm sure. Oh, here's some more details. 360 panoramic generation completes complete complete immersive world scenes explorable 3dc generation interactive editable exportable meshes so i've been yeah i can run run locally so hey uh off topic what are your thoughts on where virtual production is colliding with alif and tools like like do i guess my question is do we need led volumes in the near future in the near future yeah i mean look i'll have this five seconds a shot and like we talked about before with you know where the consistency is at because all the demos we're seeing is like shot shot shot shot shot so but it's like if you're in a scene and you know let's say if the example you're trying to do is like oh let's shoot our actors on like a white empty stage and we'll add this background in through olive if we're going to be able to consistently get the angle and background shot to shot setup to setup i don't know if it's there yet i just uh you know i spent the last four years in led volume world and i you know all of us as an industry we fought tooth and nail for every inch of taking green screen business and putting it into a volume business.

38:33I feel like with all this new stuff, it just kind of goes back to green screen. Or white screen. Yeah, or no screen. Yeah, I mean, what are the bigger arguments for VP? I mean, it's still the benefit of like, everyone can see what you're trying to film, what's happening, the actors can see what's happening. Sure. I was going to say like the lighting benefits. Relighting. On materials and people and stuff. but you know if olive does its thing like it seems like it can kind of light the people too yeah um i mean yeah it's still version one of olive i i don't know if like i'm trying to think is there a way is there a blend where it's like maybe you don't have to have as great of an led screen but you can put and you can put something sort of projectors projectors but also like oh what if maybe you can maybe you can kind of gray box or kit bash the environment you don't have to That's such an awesome, amazing environment.

39:24It can save on the pre-pro work that normally has to go into VP. And you can sort of have something up that people can see, but then maybe you run it through something like Olive where like that changes the background, but like we have the geography and stuff that's consistent. So maybe it's some hybrid of the two. Yeah, yeah, no, great point there. All right, yeah. As of right now, no, I don't think this is at the point. I mean, also like you're going to be getting, have to do it in five second chunks with possible questionable consistency and a 1080p output in like 8-bit color. All right, other tool that I saw that I just want to dive into more, it's called Morphic.

39:58And so they said introducing 3D motion. And so similar to what we've been seeing in the world, you give it an image and it builds out a 3D model of the space and you can move around the camera in this 3D space and create different actions, different movements. Yeah, I mean, just another sort of crazy example. Well, it's kind of doing what Han Yan's doing, right? It's making a 3D world, but not for exporting, but just to navigate around it. Or just extrapolating some depth. I think also this is something similar that Moon Valley does too. Like in this image here, it's like it took the image and kind of extrapolated the woman and then you could add some sort of camera movement to the woman.