Stability AI Gets Another Oscar Winner on the Team, Sony's New Camera Tracker, Panavision Layoffs

0:00In this episode of the DeNoise podcast, we're going to cover Stability AI's new virtual camera and Rob Legato joins the board. Sony's new camera tracking system called Ocellus. And more sad news in Hollywood with PandaVision and Village Roadshow. Let's get into it.

0:17Hey, welcome back, Joey. Welcome back, Addy. I've seen you too many times this week. We just covered the special GTC episode. That was fun. Yeah, yeah. But it feels like this whole week's been denoist, denoist, denoist. We got a lot of love from the audience. I just want to shout out people that have reached out to us individually and said how much they love the show. We've got some comments on YouTube. Yeah, some good YouTube comments. So thank you. And yeah, we love the support and comments are great. Subscribes are great. And I know we talked about this, but like the biggest way to grow a podcast on the actual podcasting app is leave a five star review and Spotify or Apple podcast.

0:52Yes. Favorite podcast app of choices. So yeah, but YouTube comments are also great. And we appreciate the love and also a great place to leave your own thoughts and comments and we can engage in the conversation since this is a two way between us. But like kind of one way out to you. But if you want to share your feedback, comments are the best spot to do that. So thank you and keep going. Yeah. So real quick update before we jump into it. In the last episode, I had talked about O Cinema from Miami Beach, and they're facing the battle from playing No Other Land, the documentary. And good news.

1:19So the Wednesday was the city commission meeting. And so there was tons of supporters for O Cinema there. And there's a they have an open mic kind of public forum before they get into it. And so there's lots of great discussion about First Amendment rights and art and kind of definition of art. So really great comments, a lot of support for a cinema. The commissioner spoke. And then fortunately, after everyone had their piece, the mayor just dropped the proposal and pulled it. So he kind of like, I don't know if you just want to have his moment of voicing everyone's opinion and letting everyone have their say.

1:49What was the proposal? Also, the proposal was that they would evict O Cinema. And so they're proceeding with the eviction? No, no, they pulled it. He dropped it. The mayor had a that was the issue with the mayor issued a city proposal to evict o cinema and cut their funding that they received from the city of miami beach that caused the whole issue then everyone came to voice their support against his proposal yeah and then after everyone said their piece he withdrew his proposal which means the bill amazing whatever not bill it's not congress but so happy ending happy ending yes so basically everything stays the same with o cinema they stay in their spot they keep receiving their funding yeah so such a victory for m e for me and entertainment because we don't see our civil liberties eroded away, our freedom of speech eroded away.

2:33Yeah, and everyone rightly pointed out too, it's like, this is a slippery slope. You're going to have government or someone come in and dictate what art can and can't play and display. And what we echoed before, it's like, you disagree with the film, you don't want to see it, your choice, fine. Don't go see it. Yeah, don't censor a movie theater. So yeah, it's a great news for that and glad to see that everything worked out with Ocinema and a lot of support for them. Yeah. It's great to see the community. Shout out to your friends that are running it. I mean, they're doing such an amazing job.

3:01Yeah, great work. You know, keeping the community engaged and all that. Yeah. So glad to see it. Stability AI. Got a couple updates. Yeah. Stability is, I think, the most interesting company to watch in the world of media and entertainment AI technologies. Well, for one, they're one of the OGs, right, with stable diffusion. And two, they have heavy hitters on the team. They have James Cameron, Sean Parker. And now, none other than Rob Legato. Friend of the show, unknowingly friend of the show, Rob Legato, who we talked about a handful of times. Yeah, he was in that, judging that AI hackathon that I talked about a few weeks ago.

3:37And has been definitely vocal about exploring AI and taking advantage of it and seeing it as a tool, another creative tool in the kit. He also did a great talk at VU's Virtually Everything Summit at NAB last year. I don't know if they ever, I don't think they ever put the full video online. I hope one day they do. But it was a really good talk from Rob about AI in general. And just like, you know, he's been around since Avatar, Titanic, developed the pipeline for the virtual camera workflow in Avatar and just sees this as another step in the tool. And now is, I mean, getting on the other side of it and being involved in helping develop these new AI tools for professional visual effects pipeline.

4:11I can't think of a better person. Like he is an Oscar winning as tier one as a Gets level VFX supervisor. he's super familiar with ai limitations and strengths of today and i think he's a quote unquote chief pipeline architect that's the name of the title yes i believe which which is sort of a parallel to what companies like promise ai and asteria and staircase are doing is uh it's not necessarily building foundational models like stable diffusion it's stringing together a bunch of tools the same way vfx pipelines string together a bunch of tools and to get the most amount of control and quality and how do you make it build the tool in a flexible enough way because ai is developing so rapidly where when something new does come along an improvement in a model you can just swap it out and not have to rebuild your entire workflow i'd be really curious to see if stability has any sort of internal short film that they're championing and then rob legato comes in and then drops the quality bar, right?

5:15Or rather improves it significantly. Like producing like a proof of concept kind of film. Yeah, I would love to see how good that looks with Rob Legato at the helm. Yeah, and I feel like the comparison and the advantage of something like stability is unlike these other AI production companies that we've covered in the past where they're using existing models and plugging and playing stuff. This is stability, yeah. And they're training their own models and you have a team of developers building new models and tools. And speaking of new tool that also just came out from stability is stable virtual camera.

5:45And this perfectly ties into a lot of things Rob has been talking about. So cool. The trickiness of AI where he's like, you don't have the control, you know, you need that camera control. Specifically cinematography. It's a matter. Yeah. And even, you know, he would give great examples of something like the live action Lion King that was all CG and another virtual production pipeline that they developed with operating a physical unit that was a virtual camera. Yeah. And he also gave great examples in his talks where he's like just the art of the camera operator comes through in the virtual camera like the the weight transfer and the footsteps and all those things that that language we're used to right and stuff that just doesn't come across if you're operating in a fully virtual space and correct doing keyframes and it's like camera goes from here to here with the key frame it's like you lose the actual finesse of a seasoned camera operator it's such full circle because when Rob Legato was working on Avatar 1, this was all so brand new for real-time render engines, right?

6:41I think this predates Unreal, but having simulcam and virtual camera for Avatar 1 was so pioneering at the time. And now they're introducing that all over again with AI, which will obviously have far more creative control and quality than what they had back then. Yeah, exactly. And so just highlights of what this new, it's a, I believe it's in research preview mode, but you can download the model and test it out. But you can feed it a 2D image, either a single 2D image or up to 32 input images. And it will turn that into a 3D space. And you can, I believe you can either program, it wasn't entirely clear how you can program manually the camera movement.

7:26I'm guessing you can kind of draw the arc or the curve. Yeah, I don't know if it's a curve. I don't know how that translates. how you map that out in a 3D space. I don't know if it's something similar, like Runway has a camera movement tool where you can kind of click and drag and animate a camera movement. I'll have to test it out. Yeah, we'll test it out. But they also have a handful, 14 preset paths. So like an orbit or push in. This is something similar that Luma has as well, or if you want to have a preset camera move. And tilt, push in, pull out. I mean, it's not, I don't think it's real time yet, But wouldn't it be cool if it could take XYZ input or perhaps 3D data?

8:073D is like the most common camera tracking protocol. And so you could tie like a Ocellus or an NCAM or a MOSIS system in real world, feed that into stable virtual camera and get even like a five frames per second update on what you're seeing. Yeah, I'm waiting for something, even not a real camera, even our phone. iPhone, yeah. Yeah, which is like when I even have dabbled and connected like the Unreal virtual camera to the iPhone. And I'm in an Unreal space and I'm looking at it on my phone. It just brings this whole extra sense of like realness to the virtual space where I'm like, oh, I can like tilt up here and I see like my virtual space.

8:43If we could distract that with AI. I mean, also, first off, it's really wild that this can work with just a single input image. Yeah. It's doing a lot of generative filling. Yeah. But yeah, there is some intermediary step where you can give it the image. it can create, even if it's a rough, gauze and splat looking kind of 3D environment, you operate your camera in this, record your movement, and then generate the high quality output. Yeah, it's almost the same challenges that a real-time game engine like Unreal or Unity would have to push through, which is, okay, fine, you are now doing location scouting in a virtual environment, good.

9:19Okay, so now can you make stuff move? Can the palm tree swing and can the water, you know bump up against the rocks okay that's a challenge let's get past that now the ultimate challenge is uh characters okay can you put a human in there and can that human perform and move at the same time you're doing a camera move because these are things that are happening on set in real life and you have to recreate all of that in the ai neural engine and are you saying this would be like a human actor that we're filming and we're putting them in a virtual space or a generated i would imagine a generated human would probably be easier in the first step and then the second step would be like taking input camera input of a real human performer and putting it into the neural engine yeah yeah and i think that's yeah i mean we've talked about that's got some work to do being able to film a real person their performance yeah and make the environment virtual and also a consistent environment which we've talked about before as well of like a consistent 3D space.

10:21So I can, when we have the shot here, we have the shot here. The background is not regenerating and is something completely different every single time. Yeah. And then the arm's race right now in the AI M &E space, I think is the image to video models, right? So like Google VO, Juan, obviously stable diffusion runway. We're going to move very quickly into the world of video to video generation. I think that's where most of the content creation for Hollywood will happen. Nobody's expecting you to have tier one level quality just from generation alone. You're going to guide it. And what better guidance than a camera input or a screen grab or something that's constantly feeding the system at 24 frames per second or whatnot.

11:08Yeah, like the next step of the virtual production pipeline that we've established, like cameras are tracked in a hybrid virtual environment. It'll just be generated in the future. Pure speculation here, but I think we're going to kind of see almost a fallback to green screen because you don't need the complexity of a volume and everything that comes with LED panels and processors and media servers. If you can just put two actors in front of a high-end cinematographer in a green screen stage, well, you can fill everything out and look at it in real time and make your decisions that way. I think, yeah.

11:47And I think there could also be a hybrid approach as well. And I got a bit of a tech preview and we're going to film it at a NAB pre-show of Phantom Track. Yeah. Using, which we've seen that with an LED wall, LED wall, big, bulky, running that with a projection instead and being able to do a Phantom Track with a projection. You have two outputs, one, your virtual environment or maybe a Gaussian splat environment or a AI generated environment. And then the other one is green screen. Oh, amazing. And I think some projector manufacturers are working on this, actually. Yeah. I'm not going to name who because I think I was under NDA at the time.

12:20Yeah. Yeah. I'll wait for the video. I don't think it's a secret of who I saw, but I'll wait until the video comes out. Yeah. Yeah. We know that guy really well. It was a collaboration between a handful of people we know pretty well. Yeah. Exactly. Yeah. So I got a little preview and it's really interesting what they're doing and I think is a good middle ground for applications like this. Yes. That you can do stuff that looks really well and don't need the huge heavy lift of having a huge LED wall. Also, the ICV effects is not final. Like, it's just for reference at that point. And so your final pixel in the background will be AI generated.

12:56And also I've been seeing, and we're also going to dive into this more at NAB, with something even like Lower Lift Jet Set, Lightcraft's app, which goes on, runs on the phone, but you can connect it to your camera. and they've been doing a lot with Gaussian splats on the phone and a lot of really interesting workflows. And that's like so low lift because I know it's literally an app on your phone. Yeah. And you could just buy a phone and you could strap it onto any cinema camera. Yeah. And I think it connects to cinema encoders like camera lens encoders and then obviously Unreal or Unity, I believe.

13:28Yeah, I think. Yeah. I think you're going on Unity. A lot of really interesting workflows that are filling the middle to lower ground of you don't need a huge massive volume to do really creative workflows yeah the indie filmmakers of today all they really need is a couple of really nimble agile creative technologists to just harness all this power all these tools that are available to them and shout out to voya who is i believe a competitor to light craft in that same space of prosumer simulcam technology okay okay yeah we're watching the space out here on denoised yes and then speaking of uh virtual production and camera tracking yeah new high-end high-level camera tracker is out from sony i love the direction that sony is going so this new product is called ocellus i think i'm pronouncing it right ocellus l l u s i would go with that looks right looks sounds correct okay so what this is uh and what it seems like to me is under the hood it's probably a intel real sense or some kind of depth sensor that's doing depth perception of the world and then in real time slam tracking meaning simultaneous location and map mapping is that what slam means creating a digital twin of the world in real time as it's sort of figuring out the point cloud around it simultaneous locate localization and mapping.

14:51That's it. And using SLAM, it is able to then deduce where it is in three space and send that data both to the Sony camera as well as any external devices that need it. And to clarify too, it is from Sony. It works on any camera. Yeah. Obviously they're picturing it with Sony cameras and that's the ecosystem. It's such a good pairing. It is a tracker that will work on any camera. It's a sensor. The sensor has five faces, but I believe only four are being used and they're pointing in each direction. And I believe it could operate with just one. So if you have four running, but if you're all the water, yeah, but if three of them are getting blocked or obscured by something, it still can accurately track with one.

15:33I mean, they must've been working on this for a long time because this is not something you could put together quickly. And it's a very competitive space. You have NCAM, which is obviously another depth sensor base. Now that's a Zeisson craft. Sorry about that. Yeah. That's a new name. Zeiss acquired NCAM. Yeah. And now you have Stipe Red Spy, which is in the same vicinity. The MOSIS Star Tracker. You have, there's several others. Obviously, the Vive Mars for the very consumer-oriented people. Vicon and OptiTrack traditional low-cap solutions. Yeah, and also anti-latency. Anti-latency, yeah. Now, as far as how this works, I mean, you don't need any star tracking.

16:11It just looks out, tracks objects. I don't... Correct. I couldn't find info on the calibration process or how much is needed, because that's always a part with the N-Cam system. This is so close to what N-Cam also uses a stereoscopic camera, perceives depth. The way N-Cams calibrate themselves is through QR codes. You just hang a bunch of QR codes on the wall that you're in. The nice thing about this system is it's ready to go to the Super Bowl. most systems are for indoor spaces but because this is relying not just on visible data but also depth data at different lighting conditions you can take it outside yeah and there was a feature as well where if you're in a environment with a lot of strobing light so like a live event environment you can just operate off like infrared i believe it'll just turn off rgb yeah and so it can just still track and ignore the the lights flashing the sensor which can throw off other sensors.

17:08Yeah. And the other nice thing here is it's got lens encoder, like Fizz stuff, Zoom or a Focus Iris built in. That tells me that this is very much oriented at tier one professional market, you know, something like what the RE Trinity systems do, you know, where it's like a very high end Fizz or a Preston device. This is probably along the same line. Yeah. And just the layout of what it entails, it's the sensor unit, which has all the sensors. You can strap that they have a picture on top of the cameras uh it's small there is a separate processing box which is on the back of the camera okay i didn't know the sensor is small but there is a separate processing box you can strap on the back and then it does include three lens encoders but i believe if you are using a sony camera and with lenses where it can get the data then you can just run off that i don't think you need i think lens coders are there only if you're running a lens that okay okay that doesn't send the this is just like the zeiss intercraft system uh the in-cam system also has a auxiliary computer that you attach to the back of the camera so the fact that sony built it it says so what's the yeah and they have the biggest pipeline of like end-to-end pipeline of using their entire hardware yeah for a virtual production setup so like what what are their key points or their key products that they've got yeah so obviously they have the Sony C-Leds, which are arguably the best LED tiles in the world.

18:34If you look, if you're looking at it from a sheer color performance in camera quality, I would say the Sony product is better than any of the Chinese products. Now, if you're looking at it from like a touring, ruggedized, portable, travel-ready version of a tile, perhaps the Rho, Absinthe Auto tiles are better in that sense because they just have more years under their belt. So they have the tile. They definitely have the processor to go with those tiles but those tiles also work with as far as i know like a megapixel processor obviously they have the cameras venice 2 the burano cine altas all that and now they're adding camera tracking ocellus the only piece that i think is missing in the entire chain is a media server and software to run the stage it's a stage management software they do have some software i believe that's more for the previs they have some virtual production software to simulate uh prevent more array more warnings oh yes so that's color calibration it's an unreal plugin so if you're running uh in display stage then the plugin complements a sony venice 2 nicely and so on but it's not a standalone uh pixel generation tool which they would need a media server yeah product exactly to make sense to have that the thing that makes a media server special is the incredibly high throughput of data read write speeds the fact that it has to have multiple outputs like four 4ks or whatever because you're just serving a lot of pixels at the same time the media server also needs to have really strong gpu sometimes multiple gpus and sony i i just heard rumors this was like a year or two ago that they were working on something like that.

20:21I even heard that they were working on their own game engine, which is probably just speculation and rumor, but they could certainly do it, right? They're a big enough company where they have the ability to do stuff like this. I mean, they have a whole separate arm for AR VR. I don't know how, I mean, I feel like a lot of, you know, massive companies have like verticals and I don't, I never quite clear how like collaborative they are if they're just operating in their separate lanes. It is pronounced Zen, but it's spelled X, Y, N. And it was their full kind of AR VR design studio. Oh, interesting.

20:54For their headsets, for their mocap solutions, and much more geared towards a, you know, for working in a VR space, or developing things for a virtual environment. But I mean, if they're like, this is a virtual environment. And so, you know, honestly, it doesn't sound that far fetched. If you're mentioning like, oh, they might be building their own game engine or something. Yeah, I would say game engine is maybe a wrong term for it. And more like a 3d interpreter, 3D environment where you can have an accurate digital twin of the stage placement of where the camera is. It doesn't have to be super fancy, you know, just taking in 3D objects, having USDZ support, having basic shader support.

21:35I think that's enough for. And yeah, that doesn't sound like Farfetch'd with the existing apps and solutions. Whatever they have already. Yeah. And I mean, they also, they literally make gaming devices too. PlayStation. That sounds like it's a whole different department altogether. My sort of caveat here is Sony products tend to be a little bit hard to use. User experience is probably not optimized. So if you're going to release software, is it going to be easy to use? Yeah. Got that like engineering interface vibe. Yeah, exactly. What do you think? There is no word on pricing about this, but what do you think it's going to be?

22:12Competitive, similar, probably do. Pricing wise, I'm guessing it's going to land in the Red Spy range, about$20 ,000 to$30 ,000. Yeah, that's similar to Syncraft. The other thing with the trackers is most of them also charge ongoing license fees to operate yearly, which kudos to Zeiss and our NAB video, they were very open with what the numbers are. But yeah, you feel like something like this too is also going to have absolutely. I think every tech company realizes now that recurring revenue is the best quality of revenue so they attach that to everything everything they ship has some kind of subscription i mean you and i have tons of subscriptions on stuff yeah you know and i consumer and you think don't think as much about like a subscription for hardware where it's like i bought the thing it should work i bought it but uh yeah for a lot of the higher end i mean now it's coming to a point where they're attaching subscriptions to automobiles yeah that's crazy it's crazy like BMW, I think they have something where if you want to unlock the extra 100 horsepower of the engine, it's a subscription.

23:19It's not good. You piss off a lot. I can't remember if I saw something. I don't remember if it was a joke or not, but that it was like they display ads on your entertainment system. Unless you turn it off, maybe that was a joke. I don't remember which company that was. That was a meme. But it's realistic enough where I'm like, I can believe that if you told me that was a real story. Right. Because it's kind of gotten a bit out of hand. Yeah, yeah, yeah. Yeah, until someone comes in, like a Vive Mars. I know they're not at the tracking caliber of these, but like you buy the Vive Mars, you got it.

23:46You're good. Yeah. Yeah. Yeah. I mean, I don't support... Learn for just tracking disruption. I think the subscription should have real value attached to it. So, for example, if Sony has a subscription model for this system, then it should come with some crazy level of support or free upgrade whenever their new unit. I'm just making stuff up. But you should have some value where you're like, OK, I don't mind paying that much per year because it is worth it in the long run. Yeah. And when you're at these high levels, too, and you're on upstage and production is costing. You need support. Tens of thousands, hundreds of thousands of dollars a day.

24:21You can't have any downtime. So like you got to pay for that peace of mind. Like that's what it's got to come with. As far as I can tell, Stipe and Moses and Zeiss, they charge in addition to the subscription to provide support. So if you have to have somebody on site, it's like an extra few thousand dollars that you have to write a check for on top of the... I can understand having someone on site. Yeah, that's expensive. What if you just need like... Remote support. Yeah. I'm not sure. Maybe that's free with the subscription. Yeah, I'd hope so. NAB, so we'll dive into that more. Yeah, I found out, unfortunately, Pixo Akira, which we talked about after CES.

24:57Thought it was going to be demoed at NAB, but don't believe it's going to be demoed at NAB. But that was another. Talking about like wild full-fledged tech setup from Sony. Yeah, the Sony ecosystem of VP hardware. We never found out how much it costs or when it's going to be available. No, and it seemed like it was if that or if it's just going to be like a service. Right, a rental thing. Like a J.O. Fisher dolly, like you just rented forever. Yeah. Do they sell? They never sell. They never sell? They sell to rental houses? No, I think every unit is leased. I never realized that. Yeah, yeah, yeah.

25:30We tried to buy one. They're like, no, you could lease it for like five years. They don't sell them. they don't sell. I did not realize that. Yeah, that's interesting. Yeah. Maybe that's why my school never had one. They had like Panther dollies. Exactly. Because you buy Panther dollies. Yeah, you can buy pretty much all of their competitors. Huh? Yeah. Interesting. But they are the best in that game. Yeah. Well, you know, that's actually a good segue into the next story. Oh, yeah. Which is, yeah, more kind of crappy news for the Hollywood and studios and companies. Two other stories popped up after we kind of did our Technicolor breakdown.

26:01But the first one is Panavision Hollywood is closing down. So there are two in LA, there's two Panavision offices, there's Hollywood and there's Woodland Hills. So Hollywood locations closing down. And there I know there were layoffs and they're moving or condensing into the Woodland Hills facility. So there's not a Panavision is going understory, luckily, but they are shrinking. But I think a big kind of part of that is like with Panavision, they never sold cameras. Yeah, only even a rental company kind of like jail fisher yeah that's why yeah that's what uh triggered it so yeah i mean compare that to ari and you know especially with the digital transition cinema not pay vision does have digital like they did i don't fully know this story and i mean it is their own sensor i believe yes right they developed their own they they were one of the first to have a de cinema sensor obviously their lenses are legendary yeah and you can't buy their lenses either no everything was rental and that's you know the only production that can these are tier one high level productions that yeah can afford panavision and how many of them are happening in and around la anymore yeah so i go like yeah it's a limited business model that uh makes it tougher in today's times i believe panavision has offices around the world probably one in london probably one in asia that's uh more or less a rental house.

27:24So maybe those will kind of carry the company. But this unfortunately does track and does make sense given the current climate of Los Angeles and the state of California in general. Do you feel like this is another kind of transition or end of like Hollywood 1.0, 2.0? Yeah, I feel really bad for Hollywood 1.0 companies like, you know, Technicolor, Panavision, Warner Brothers and all those companies that have been around for about 100 years. They're kind of seeing the end of the road a little bit here. I don't think this is the end. I think you'll see more Hollywood 1.0 companies reach this conclusion.

28:03What I hope for is with the churn, you get a new generation of companies that are able, like Stability AI, and that can thrive in M &E, in Hollywood, in Los Angeles. Yeah, I mean, I think back to, you know, you see behind-the-scenes photos of, like, from movie sets from, like, the 80s, and it's usually, like, you know, a person on a crane with a Panavision camera. Yeah. And, you know, seeing the Panavision logo on the magazines was, like, pretty synonymous with, like, big-budget Hollywood films. I mean, it's such a legendary brand. You think they'll ever, like, release a camera that they actually sell?

28:37Like, someone could... Would it matter? Buy it? Like, it's a legendary brand to us, but to new up-and-coming filmmakers, does it matter if it's a penavision like even up and coming of like are aware of it or have heard of it exactly yeah i mean do they release something like an ari that you can purchase or that you can that i don't know just something to put it in more people say on something that's not just i don't think they have the camera yeah i don't think they have the engineering chops anymore they certainly don't have the product chops anymore to develop a camera productize it sell it and have or acquire or something and license the name or that would be my move it's what Kodak did, right?