In short
Podcast Episode Summary: Denoised - Netflix Adds Vertical Video, Runway AI Does Animation, and Gamepad ComfyUI
Episode Overview In this episode of *Denoised*, hosts Addy Ghani and Joey Daoud discuss significant recent developments in the media and entertainment sector, focusing on Netflix's new features, Runway AI's animation project, and innovations related to ComfyUI. The podcast aims to provide insights and practical takeaways for filmmakers and media professionals, while addressing the evolving landscape shaped by AI technologies.
---
Key Topics Discussed
- Netflix's New Features
- ChatGPT-Powered Search:
- Netflix has introduced a beta feature using ChatGPT for a more conversational search experience.
- Users can input casual queries (e.g., "I want something funny") to receive tailored content recommendations.
- This aims to enhance user engagement and improve content discovery from Netflix's extensive library.
- Vertical Short Clips Feed:
- Netflix is testing a vertical feed for short clips, similar to platforms like TikTok and Instagram.
- The goal is to create a new discovery engine that utilizes clips from their shows to attract viewers.
- The idea is to emulate the success of UGC (User Generated Content) seen on social media platforms.
- Insights on Netflix's Strategy
- The introduction of these features signals Netflix's intent to adapt to changing viewer habits and compete with other platforms that use short-form content effectively.
- The hosts discuss the challenges of curating content from Netflix's extensive library and the importance of good recommendation algorithms in driving viewer engagement.
- Runway AI's Animation Project
- Runway AI has launched a hand-drawn animated pilot titled "Mars and Civ."
- The animation stands out due to its unique stylization, contrasting typical AI-generated content (often in a 3D Pixar style).
- The process integrated hand-drawn elements with AI to create fluid animations, showcasing a blend of traditional artistry and modern technology.
- Creative Collaboration:
- The hosts emphasize the importance of human creativity in the animation process, asserting that AI should augment rather than replace human storytelling.
- They discuss the potential of AI in increasing efficiency in animation without sacrificing artistic integrity.
- ComfyUI and Gamepad Integration
- The episode introduces innovations in ComfyUI, a tool that allows for greater control over AI outputs.
- A developer has created a gamepad controller interface for ComfyUI, enabling users to manipulate character poses in real-time.
- This development simplifies interactions with AI, making it more intuitive compared to traditional text prompts.
- Future Perspectives on AI in Media
- The hosts speculate about the future of AI in entertainment, considering the potential for real-time 3D environments and interactions.
- There is a discussion on the necessity for spatial awareness in AI to maintain consistency in generated content as users interact with it.
---
Key Takeaways
- Innovation in Content Discovery:
- Netflix’s ChatGPT integration and vertical video feed exemplify a strategic pivot towards enhancing user experience and content discovery.
- AI-Powered Animation:
- Runway AI’s focus on merging traditional animation techniques with AI showcases the potential for innovation in storytelling and creative processes.
- User-Friendly AI Tools:
- The advancements in ComfyUI reflect a broader trend towards making AI tools more accessible to creatives by integrating intuitive controls.
- Human-AI Collaboration:
- The discussion underscores the importance of human creativity in harnessing AI's capabilities, ensuring that technology serves as a tool to enhance artistic expression.
---
Conclusion This podcast episode from *Denoised* captures pivotal moments in the intersection of AI and the media landscape, highlighting the ongoing evolution of content discovery, animation, and user engagement. As technology continues to advance, the nuances of how these tools are adopted will play a critical role in shaping the future of storytelling. The hosts encourage listeners to explore these developments and engage with the tools discussed.
*For additional resources and links referenced in this episode, check the show notes.*
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Transcript
Automatic transcript. May contain errors.0:00Like a vertical short video clip is what TikTok is, is what Instagram stories are. YouTube is now that. Yeah, I mean, it is the discovery engine for finding new stuff. So yeah, I'm curious how this plays out. In this episode of Denoise, Netflix tests out a Shorts Feed and AI Search. Runway gets into animation with a cool short film. And controlling comfy UI with a gamepad. Let's get into it.
0:25Welcome back, Addy. Good to see you. Hey, it's heating up here in LA. Yeah, the clouds have broken and it is now getting toasty. Yeah, and we've got AI LA coming up. It's literally heating up figuratively and literally yes so what do you got all right so first thing on my radar was a couple updates from netflix and so one of these we talked about previously where they were debating or experimenting with rolling out chat gbt in their search that's right we covered that yes and so now it is officially in the beta you have to opt into it but it is in their beta and so it is a new search feature i believe it's just on the app currently not on like apple tv or anything but it is basically like a more conversational way to try to search for something to find something so one of their examples is you know i want something funny and upbeat and then getting movie recommendations based on just right conversational movies yeah although netflix has been really good at associations with search in the past so like let's say you are searching for a show like star wars you type in star wars it'll show up like star trek will show up it's like we know we don't have this in our library but we know what you're searching for similar similar things that is an original production right so they have been really good about that and i know that there is a large ml team in los cantos where headquarters is so this makes no surprise like this yeah it feels like the movies already have they already have this data around the movies yes also they have that famous whatever 30 000 different like genres genres if you remember there used to be like you could there was like a hack where you could figure out the code to like find all the different genres that was like you know action movies from the 80s like they take place in China or something.
2:01Yeah, and there was also a hack to use a VPN to get around their geographic sort of categorizations as well. Yeah. But did they crack down on that VPN hack? Not sure. Having Gen.AI search, it's a good way to help find stuff. And this has also been a big issue for Netflix throughout the years of just improving, finding things that you want to watch at the right time that you want to watch them. I remember too, wasn't it a number of years ago, didn't they have like a million dollar reward for someone who could develop like a better algorithm. Really? To recommend. I mean, it was like a while ago, like when they were first getting off the ground.
2:34That makes sense, right? Because so much of their business and the sort of watch time comes from a good recommendation, right? Yeah, of course. And finding the thing you want to find. And we mentioned that stat before that Ted Sarandos had mentioned on the earnings call where he was like top 1 % of their content or the top 1 % of like their most famous titles is still like 1 % of the viewing. Yeah, it's nothing. Yeah, and there's just like such a massive library and an international library of content that there's just so many different things that people are interested in watching that there's no one size fits all for.
3:05Yeah, a massive library that they have is a double-edged sword. On one hand, you have every single conceivable genre and show kind of covered. On the other hand, good luck finding it. Yeah, how do you surface that? How do you find it at the right time? I mean, I feel like the next thing from a Gen.AI search, because you still have to like, it's a bit active, you have to get on and you got to like, kind of know what you're in the mood for. you know i'm thinking the future is maybe like a gen ai channel or not a gen ai channel a generative playlist channel where oh yeah like you know like disney's been adding this spotify has that they have they have a dj that is just uh right right you're probably automatically welcome to i'm your spot i'm x your spotify hey joey welcome back man today we've got hip-hop from the 90s i'm mixing things up for you you know what i'm talking about yeah yeah maybe not as annoying but certainly right so right that is with songs something that but like something that sort of feels like it's already on so like disney plus has been doing this where they've had i mean we're basically like reverting back to cable tv with channels but just like channels that are just oh you know movies playing where i mean this was a better experience not a better experience but it's a different experience in the 90s and early 2000s when you're on cable you're like i don't really know what i want to watch and just channel surfing and then your mid play of a movie and then you just leave it on because it's already playing yeah i'm thinking you know the next phase of like netflix is a generative channels based on your interest but it's like already playing for sure you know it's customized to you and your interest they do do that with the kids accounts a little bit the kids accounts are obviously you know because they're kids you want to be careful about what they watch and also fine-tune, no pun intended, what they watch.
4:46They have a more specific algorithmic output. And then the cool thing about the kids' channels is the parents will get a report every other week on what their kids watch and what the big categories are. So one of my girls, she loves to watch stuff that's about bravery and about courage. And you're like, oh, that's an interesting underlying theme on all the stuff she watches. That's cool. Yeah. That's cool to know. Yeah, yeah, yeah. Do they tell you if they tried to search for anything? Like what they searched for? So the kids, when you go into a kid's account, it's very much restricted. Kind of like YouTube kids.
5:21Like you can't really get terrible content out at that point. Yeah. But if you tried to search for it, like your kids tried to search for Chucky.
5:30Red flag. Red flag. They're looking for Psycho. They heard about that. That's cool. That's interesting. Yeah. With the channel thing, I'm thinking back to the like 90s and stuff. I'm thinking like, I've probably seen the actual beginning of Terminator 2, maybe like once or twice. But I've seen the like, I've seen so much of the second half of that movie, like a million times. But I've like maybe seen the beginning a few times. Where he walks into a bar, grabs the clothes from the guy and walks in naked. Yeah, I've maybe seen that a few times. But it's like, you know, the river scene and the bus chase and like, you know, falling and sinking into the pit of lava.
6:03Like I've seen that a million times. Yeah. But because, you know, I'd always catch it on like TNT or something. And so like, I've seen the second half of that movie a million times, but maybe the beginning once or twice. Yeah, that's an awesome example. Yeah, that's the nature of when you were just catching stuff on cable. And so the other interesting thing with the Netflix update is they are pushing into a vertical feed, short clips. Very useful, very, very relevant to today's media landscape. Yeah, and so this is not UGC type stuff. This is short clips from their shows more as a discovery engine.
6:36But I think this is really smart because there are a number of times that I've been on TikTok or wherever and I see it's not the studios that are putting them out or maybe they're, you know, secretly putting them out. But it's like a scene from a movie is playing and you're like, you kind of get sucked into it. 10 second bite. Yeah. It's like, oh, one of the good dramatic scenes. And it's like, oh, interesting. And then it cuts to like a montage of like shots from the movie. And they don't even tell you what the movie is. You're like going to comments and people are like, what movie is it? What is this?
7:00And then someone comments like, oh, it's this movie on this streaming platform. So it's almost like an automatically generated trailer that's vertical. Yeah, it's like a trailer, but it's more like a clip. Sort of like if you're on Netflix too and you scroll past the titles and then you let it hover and then it starts playing a clip from the film. Right. Like that, but in a vertical feed. Yeah, I got to say, at this scale, with millions and millions of titles that they have, incredible feat in deploying this. having said that i hate to bring out spotify again but they have done this already and it's i think a year or two into it where if you go to like kendrick lamar's page right you'll see a bunch of shorts and then you could swipe through each of the songs and each of the songs is is in a video format it's a video maybe it's the music video itself or some kind of visuals and then when you like the song you click on and it goes and then plays the song right and yeah i can see how that works i feel like it makes more sense on netflix because you're like it's visual yeah and you're like looking at visual stuff and so you see a video clip and you're like well let me add it to my list or watch it now yeah spotify it's like i'm going to spotify because i want to listen to something not like scroll but that's the thing like because you know we're on our phone so much yeah and our screen time is just so high like having that ability to scroll through visuals for a music product.
8:27Or a podcast product. We put up shorts. We have shorts up now. Yeah, we do now. But I mean, that was another reason why Spotify did such a big push into video podcast and hosting the video podcast for this exact reason. Yeah, but I will say like, you know, like a song is what? Three, four minutes versus an episode is like 30, 40, 50 minutes. Like there is such so much more data to churn through to create these shorts on the Netflix side. Yeah, yeah. And a lot more discovery. and yeah it it makes sense and uh but you know i've seen the show the vertical clips come to netflix reminds me of the you know ongoing conversation of like you know real real shorts not even that but sort of that i guess but like the vertical the vertical soap operas yeah but i was thinking more of just like you know is netflix going to become youtube first or youtube going to become netflix first and this like continuing ongoing yeah battle between the two or like they're the two dominant players for like most for screen time and owning screen time especially on a TV.
9:22I will say, I think that the line has already been drawn and Netflix will never fully become YouTube and vice versa. Like they know their market, they know where their revenue is. It's always going to be, uh, you know, an acquisition or a content wall. It's interesting that Netflix is sort of, you know, tipping the hat to the format that is working for all of the other social media formats. Like a vertical short video clip is what TikTok is, is what Instagram stories are youtube is now that yeah i mean it is the discovery engine for finding new stuff so yeah i'm curious how this plays out i haven't updated my app i don't know if i have it yet but uh i'm curious too if they do special edits for these clips or if it's just going to be like we took a clip yeah and we just framed it for vertical also their framing process are they running this manually maybe they bought the quibi thing yeah the quibi framing algorithm the quibi tech ended up yeah you never know yeah i mean i'm curious like are they running uh using AI models to like track the faces to like auto reframe?
10:19Yeah. Or are they doing this by hand? I mean, it would probably be cheaper to just have a farm of, you know, artists in India somewhere and just pay them to get this done, at least for the initial rollout until they figure out what the automated process is. I don't know. I think the automated stuff already exists. That's been something that's like been doable for a while, you know, just track the face and just make sure the face just, you know, stay center framed. Yeah. And this is not new news, but But Netflix's app, at least on iOS, on my phone, has games and stuff built into it already. Yeah.
10:48The games are... I've used the games. Yeah. They're fun. Yes. Exactly. Included. I think I played the Squid Game game. I haven't played that one yet. Yeah. I like the Infinite Run ones. They had a bowling one. Yeah. That was just like an infinite bowling ball. You have to jump the... It was like a speed run thing. It's like Temple Run? Yeah. It was like Temple Run with bowling balls. Awesome. Yeah. Netflix has been doing that push into gaming. Yeah. That's always been a bit like, where's this going? And now they have live events. And now, like, what are they not doing, you know? A bit of sports.
11:20But yeah, like, sports is probably the next. Sports and news. Yeah. So, on the flip side, on the competitor side, I think Amazon is, like, two steps ahead on the sports front. Like, now they have, I think, NBA. They've already had NFL. And now they have NHL and NBA. And so, they're really pushing into that end. Yeah. Yeah, they've also built themselves into more of like a platform as well, where it's like you can be on Amazon, Amazon Prime, and Prime Video, but then also subscribe to other streaming services through the Amazon hub, like subscribe to Starz or Paramount Plus through Amazon. So it's kind of interesting, more of like a hub for like engaging with other streaming services.
12:03For sure. And then finally, I think we covered this as well. Netflix's Tudum, which is their UGC platform. where a lot of like show fandom exists. Yeah. Like maybe there's potential there to have a YouTube like creator output. Yeah, I mean, they do have a bunch of YouTube channels around. Without YouTube, without utilizing YouTube. Well, what do you mean? Like a. So like if somebody is a huge fan of Squid Game. Yeah. Right. And they just want to make a review for each episode as it comes out, they have to now put it on YouTube. Right. What if they were able to put it on to do? Like more of like an in-house kind of like to doom creator.
12:41Yeah, so it goes back to like when, do you remember when Chernobyl came out on HBO? Yeah, they had the official podcast. Yes, and that was so successful. They do that for pretty much all of the shows now. And that was the first show that was piloted with. Yeah. And it did so well. Yeah, and I'm thinking, that was what, when you mentioned this, that's what that reminded me of. Because pretty much all the HBO shows now have a dedicated podcast. And yeah, it's like from, usually it's someone that was like, had a separate fan channel or something. Yeah. Paired with, I don't know, some other like podcast producer.
13:08And then they like bring on. there's so much stuff there's there's just so much you're leaving on the table if you're not extracting out of that ip like a podcast or other things you know because that's all engagement engagement numbers yeah the same reason why there's like the behind the scenes stuff afterwards it's like yeah you have your you know about a lot of people just watch the show and that's it but then you have like you know your your bigger fans where it's like you watch the show and it's like i want to process that i want to like learn more about that or buy the toys yeah and this is just It's another way to kind of further that engagement.
13:37And a podcast is pretty low lift. And if it was a YouTube video, yeah, like that'd be pretty low lift. And yeah, I don't know. I mean, I'd be curious if that made sense to build out some sort of extra streaming thing onto Doom, or I feel like all roads lead to like, you'd throw it on YouTube just for the pure discovery aspect. Yeah, it's like, why wouldn't you? Because you just, you want that top of funnel discovery aspect to get more people finding out about the thing and get them on your platform and watching the actual show. The only reason for putting it on 2DOOM, a very good reason, is you could insert ads into those UGC stuff and then just make money that you would otherwise not make from YouTube.
14:16Yeah, yeah, for sure. All right, next up, interesting stuff with Runway. Yeah, runway killing it lately after the Gen 4 release and Crystal Ball's latest sort of speculative audio tool where you just kind of tell the AI engine what to do. You know, bring me a building, bring me a sky, and it just kind of does it for you. So they're really thinking in terms of how M &E would use it. And the latest from them is an animated series. It's called Mars and Civ, and we'll play it back for you here. The first thing you'll notice is that this is very stylized. This is obviously like a film noir in the way that the story unfolds and in the mood.
15:00It has a very like kind of the hand-drawn animation vibe. Yeah, so visually it has a hand-drawn animation vibe, very much like a 2D look. And these are things that you generally don't associate with AI, right? AI outputs typically Pixar style, 3D. If you prompt it, Pixar style. Right, but like that's what most people do. Yeah, or yeah, you think it's a, you associate it with an existing style already. It's style that you know for sure is going to be output correctly. So in this instance here, I'm sure there was a big post process and an additional process on top of the generation, right? Yeah.
15:33So they have a nice interview in-depth blog post. And so it seems like it's from Runway AI, their studio arm, which has been tied in. But it's sort of a separate thing where it's like to produce content and use Runway for like producing professional content. That seems like a fun department. so yeah with the creators it seems like i mean and they have images the everything was hand drawn storyboards and then it seems like the bulk of the process was hand drawn locations frames of characters hand drawing the elements and then combining the elements in runway yeah creating the first frames and sending it to runway and then using runway for that in between animation perfect stuff we've talked about yeah well you know me i'm an animation fan and when i saw this like other than a few technical things i thought this was really effective and it really worked where i think a traditional animated thing would really differ from this is there would be a lot more set pieces a lot more complexity in each of the shots where here it's just you know background character background character um and i think that's this is a good step they were probably focusing more on lip sync character animation for some of the runway work and then just compositing everything in post.
16:48That's my guess. Well, yeah, I mean, because Runway did have the whole Act 1, which was putting a character and taking a human performance and being able to match the emotions and stuff. It doesn't seem like they use that at all for this. Yeah, there is a tool called Adobe Character Animator. I think they renamed it to Adobe Animate now. That does really good lip sync off of an iPhone camera. Yeah, you rig up your animated character and then it'll like sync with your iphone as a yeah like face capture so that could be like this is exactly how we envision all just tools and that like it's like there's no one quick fix tool for this it's just tools to speed up the process yeah that's how movies get made there is no you know magic silver bullet you know it's just what do you need to do let's do it with this thing move on do it with that thing and go on yeah and this is a great example too because i mean they're like hand-drying elements they're compositing it and probably just traditional compositing software and then they're you know using runway to speed up the in-between process and like bring you know animate the actual shots yeah the last time we saw a short film out of one of the big ai companies was uh do you remember when open ai released that balloon head thing with zora yeah airhead airhead yeah yeah so i i think there was a big article on it after it came out that said something like uh one out of 300 or one out of 3 000 shots was used like it was a a large number of generations It was also heavily composited.
18:13Yeah. It would be like, you know, they'd get one element out of one shot, another element out of another shot. And yeah, and that was sort of, especially last year in 2024, that seemed to be the common way to make something that looked realistic. Like just toss the dice many times. Yeah. And then get the elements you want and composited. And it was like After Effects seemed to be like glue for so many AI generated shorts that were of a high enough quality that looked professional caliber. And most of these were created from media professionals who like work in VFX or animation. And so they know what they need to get.
18:48And it just turned into using generative AI to just get the elements out of it that you need and composite it later. Yeah, really. I'm really surprised at the level of output and also the stylization and the creativity that went into it. it gives me a lot of courage that you know ai is not gonna ruin everything if you know like use it you know correctly it just help you do more and exactly you know for budgets where it just might be cross-prohibitive for something like this to exist yeah i mean we talked about before recording like this reminds me of the cartoon network stuff like dexter's lab and stuff before in the 90s like classic adult swim cartoon yeah and i'd be curious like i don't really know much about the making of process from those like how big the teams were and like sure what the time frame was to make those and how that would compare with something like this yeah i think i remember reading into that a little bit this is like a decade ago those were big teams yeah so dexter's laboratory power puff girls and i remember um family guy i mean basically similar process with how they're using runway they would do the key frames here in the states and like send it to korea right i think it was india okay because also if you looked at the names.
19:59It was like very Indian names in the credits and they would do the in-betweens. And so pretty AI. Well, that's the thing. AI is so good at in-betweens, right? You give it first frame, last frame. And middle frame, which yeah, Gen 3 can do. So yeah, I mean, this is a great example of combining human creativity and having originality by using AI to speed things up and make a film that potentially would not be possible otherwise. 100%. it still doesn't take away from the fact that you need a human storyteller with genuine creative idea yeah all right next up so new news on comfy ui yeah what do we got so look comfy ui is this ever pervasive tool you know we've seen people at the ai hackathon use it heck i run it on my computer here on a very slow gpu yeah when you want the most control out of your ai outputs and what you're doing with it yeah all roads lead to comfy ui exactly and you're not just limited to your own crappy GPU, you can connect ComfyUI to AWS or wherever and access a bunch of different high-powered GPUs.
21:04Or run it off a cloud. Yeah, or just run it off a virtual machine in the cloud. But it is the most common language and the most common tool for AI builders today. I could definitely confirm that even within the walls of AI manufacturers, they're using ComfyUI to prototype things and sometimes to deploy custom solutions and so on. So ComfyUI is a node-based tool, but it's still not very user-friendly. So it looks like you've found something that's really interesting here. Yeah, so this ties into another thing we talked about of the lack of input or control in controlling AI outputs and using more traditional methods than text prompts.
21:47So it seems like someone built a controller where you could use a game controller to control poses of characters generated in Comfy UI. That's interesting. Yeah. So in the quick video demo, they're running Comfy UI and they're using a gamepad controller and hitting some buttons. And it's like they're creating a character. And it seems relatively in real time, you know, they're like hitting the button to the right and it's changing the pose in the face to the left to the right. so it's a good indicator i mean i my personal what i would love to see is doing this with a camera in a 3d space right but this is like in that right direction of just like a relatively real-time way to control an ai output yeah with something that's more intuitive than a keyboard a game controller yeah my 3d brain instantly goes to well why not like that woman's face why couldn't be a 3D character and then the controller just controls the rotation of the head and then you take that or if you can orbit around her orbit around yeah right yeah and then take that feed and do a neural rendering pass in comfy yeah i would love to see that yeah you know more about comfy than me too you should download this i'll definitely download it if it's available oh that's another thing uh for those of you who have not messed around with comfy yet comfy runs on what it calls workflows which are just a giant text file uh typically json files so if you see a workflow you like you can download it load it and that loads up all the nodes and all the connections to the nodes all you have to do next is then load each specific nodes with the right models or the right where i always hit issues so comfy now has a feature called install missing nodes yeah that but also it's his miss.
23:35But it's like, it's almost there for like the average Joe. It's getting there. Yeah. I'm like, I see a McMuppets tutorial and I'm like, I want that. And I download it and I install the JSON file. And then it's like load missing models. And then it's like, can't find this one. And I've noticed that a lot of creators hide that under the Patreon page. So you have to pay and to get access to it. Yeah. Makes sense. Yeah. I paid them. I still want them. I subscribed to McMuppets. I wanted the missing, I wanted the advanced nodes, but yeah, I was talking to JD and Skyler who run a playbook and they've been doing a lot of stuff with Comfy and they have a cloud comfy instance where you can load their models and run on the clouds.
24:12You don't have to have a PC or a beefy machine. And we've talked about this and it's like, they're like, yeah, that's like when people bring in workflows, that's like the number one thing that breaks is like just finding the models that the person who built the original workflow has and then reconnecting them properly is like, yeah, just the kind of basic, but like the easiest part where just stuff falls apart. Yeah, and also like the models are so specific. Like you can't just drop - And the version of the model. Yeah, yeah. Because some of them are going to output float 0.8. Some of them are going to output float 0.16.
24:40And if you mix up the two, then everything down chain is messed up. Oh, it breaks it all. Yeah, yeah, yeah. So the, I don't know the name of developers behind this, but the website company that developed it is virch.ai, V-R-C-H.ai. So I just want to give them a shout out. But yeah, this was on a Reddit post. And so we'll link to it in the show notes. But yeah, I mean, this reminds me too of the hackathon I covered with Rob Legato, who is now at Stability as a creative... Chief pipeline architect. Right. And this was his number one request too, which was like, give me the controls that I'm used to to operate in this AI space.
25:12Right. And yeah, I mean, so what do you think, in order to get to the point where we're talking about, let's generate a world and move through it in real time or relative real time, what else has to happen? It's happened already. It just hasn't happened at a consumer, comfy UI level. so if you look at startups like in world ai that's exactly what they're doing they're building a game engine that is rendering nearly so you can take traditional you know uh controls like a d-pad and just walk through this world i guess that ties into like that uh the doom the generative doom game exactly world that they built where you just keep infinitely walking through doom and it just generates it in real time yep yep look having the capability aside i think just the the notion of having a human interface devices, like a keyboard or a, you know, a D pad or whatever into something as complex as comfy UI, that in itself is encouraging.
26:08That means that this is a barrier that is very soon going to be crossed. And somebody like Rob Legato or myself, or you can easily control all of these complex nodes easily. And then also being able to save. I'm thinking too, that's the other issue of like you're moving and you're navigating, maybe you're changing stuff, But they make sure you're able to save it and it's persistent. So then the model needs to be spatially aware and have location tracking or some kind of coordinate system in it, which doesn't really lend itself to how a neural network actually computes it. It's not like a traditional...
26:42Is the neural network remembering anything or is it just generating and then... So memory is tricky in a neural network. You know, when it's trained on like a trillion images, it has quote unquote memory of like a cat, a dog, a human, a tree and individual things. Right. And you wouldn't even really call it memory. It has like a part of the neural network attached to that association of that idea. So when you prompt it, what the prompt does is it will, and this is where attention comes in. So you say, you know, I want a cat on a tree. Yeah. And it'll figure out, I need more attention on the tree.
27:22That is the thing that the cat needs to be on. So secondary attention on cat. It'll go find in its memory what a tree looks like, what a cat looks like. And then it'll put those two things together based on what it was trained to be a realistic image. So if you're really thinking about how a neural network computes it, it's nothing like what we're actually used to. so how do you actually build in a system with coordinates and 3d spatial awareness yeah or i'm thinking if i'm generating a space with like the cat in the tree in a garden and then i'm like navigating my camera and i pan to the left away from the cat the tree when i pan back i want to make sure i see the same cat and tree that i just saw before and it's not like regenerated yeah so that would have to be built into like a reference add-on like a fine tune of a reference of that tree but if you think about it then you need it for every single object that's in the scene to be consistent when you put the camera back right yeah so it's a it's a tough problem to solve but having said that uh we're just in such an early stage of all of this that i think it will be solved yeah you know it's like the early days of compute when we couldn't figure out that um our computers will one day have 24 gigabytes of ram yeah we're running 16 kilobytes of ram we're Like there's no way silicon could be packed that densely into have that much memory.
28:44Yeah. I was in San Diego over the weekend and I took the tour of the USS Midway, which is like a decommissioned aircraft carrier. Been there. And yeah, fun tour. I go to walk around. But they had their like computer room or their like radar room. And it was, what was he written with the original computers? Yeah. So I think the USS Midway is like World War II era. It was, but I mean, they didn't update it because it operated in the first Gulf War. Oh, okay. So they did. So like floppies. But it was, I don't, I mean, I don't know what era they were gathering the, what, I think they gathered stuff over its operation over the years.
29:20Yeah. But it was a like, yeah, Univac. It was an old, it was a Univac computer that was like a massive, huge cabinet. And it had like 32 kilobytes of data, of storage. That's amazing. Yeah. Well, off topic, but I heard that this is one of the main reasons why our ICBMs, all the nuclear missiles and silos are more or less like useless or close to being useless is because they're running on ancient machines and they don't connect to the modern day Internet and none of that stuff. Right. Yeah. Or like our FAA. Yeah. Is that running on like Fortran or these older languages that haven't been updated in 50 years?
30:02Exactly. Exactly. Well, all right. random note, but it reminded me of just, yeah, this was 50 years ago. It wasn't that long ago. It wasn't that long ago. So we're in that moment now. We're in that 32 kilobyte moment now. We can't imagine what this is going to be. Yeah. All right. Good place to wrap it up. Links for everything we talked about in the show notes, and we'll link to the Comfy UI workflows if you want to mess around with it and let us know how that works out for you. And if you found any cool controls of how to use AI with alternate controls, always interested in that. And if you're just new to the show and tuning in, guess what we've been doing this for a while there's over 30 episodes uh go back and watch the catalog we cover all things ai media and entertainment tech so yeah see you soon thanks for watching catch you in the next episode
From the publisher
Netflix unveils two major platform changes that could reshape how you discover content. Hosts Addy Ghani and Joey Daoud dissect Netflix's new ChatGPT-powered search and vertical short clips feed. Plus, they examine Runway's impressive hand-drawn animated pilot and a practical innovation that lets creators control ComfyUI with a gamepad instead of text prompts.
#############
The views and opinions expressed in this podcast are the personal views of the hosts and do not necessarily reflect the views or positions of their respective employers or organizations. This show is independently produced by VP Land without the use of any outside company resources, confidential information, or affiliations.




