The Oscars Say AI is OK, Test-Time Training, and Descript Goes Vibe Editing

0:00In this episode of Denoised, the Academy sets new rules on how films can use AI. A new AI model for long-form video called Test Time Training. And Descriptease is a new feature for Vibe video editing. Let's get into it.

0:15All right, welcome back to Denoist. Addy, good to see you again. Good to see you again. I found out today is the day we are recording this is the 20th anniversary of YouTube. It is the 20th anniversary that the first video was ever uploaded. Wow, I feel old. It's been 20 years. Although I have been using YouTube very early on, as I'm sure you have. Yeah, I remember, I think we talked about this a few episodes ago, too, where I like, I remember back in the day, it was like, you know, especially in film school, and they make your film and it's like, do I put it on YouTube? Or do I put it on Vimeo?

0:43You know, where like, film cinema is? Yeah. And YouTube was sort of the kind of weird, you know, cat video place. yeah well the the moment the aha moment for me was uh when a buddy of mine wanted to share like something that happened on hollywood boulevard it was like you know uh late night after hours and i was like oh uh send it to me i want to see the video and he's like oh i uploaded to youtube just sent me the link i was like yeah whoa yeah i don't think people under our age realize how complicated it was to play video on the web because you would have to upload the video and if someone wanted to play it they'd basically be downloading the video file it was common practice to upload like three different versions of your video like small medium and large yes depending on their bandwidth they could pick which option they wanted yeah avi mp3 mp4 divx divx yeah i forgot i forgot about divx and yeah the original youtube was built uh with flash right was it a flash based so yeah yeah and then they eventually switched to html5 and i have no idea what it is now.

1:45Yeah, it's probably its own unique ecosystem of tools. But do you remember our very first mobile phone playback of YouTube? I remember the iPhone, one of the big features was that was YouTube was a built in app. Crazy. And I remember the first time seeing someone who had an iPhone, and then they played a video back. Yeah. And it was like, man, that's a crazy moment where you just like kind of like a glass brick, you're holding in your hand, and it's just playing a video. There's no buttons on it. There's no borders, no buttons, just like a smooth piece of glass with the video plane. Yeah. No, I feel like we've aged ourselves a lot.

2:17No, I'd like the, the, those moments of amazement that we had, I compare it to like the first time I saw Sora's like that preview video. Yeah. This is like what, two years ago, not a year ago, a year ago. Time flies. And I was, it's the same level of amazement. Whoa. AI can generate an entire video. Yeah. I think I had dismissed it. Cause I think I saw the clips And then I was like, I think I had to keep scrolling. Like, oh, that's just like, that's a reference video or something. That's not the AI output. And then I realized, I'm like, oh, what? No, that's the real deal. Yeah. All right. Speaking of, actually, before I get rolling, real quick, we appreciate all the love support.

2:54But we have a favor to ask you. If you are enjoying this podcast, we've been getting a lot of love, usually on LinkedIn, sometimes in the YouTube comments. And that is amazing. If you want to copy and paste those comments onto Apple Podcasts or Spotify Podcasts, besides YouTube, that is the best way to help people discover this podcast and for it to grow. podcasts are notoriously tricky and hard. And it's also crazy because podcasts are about 20 years old, a little bit more than 20 years old. Really? I believe, you know, it's funny. I was doing a short film in college on life hacking when it was an early concept and it was like lifehacker.com and stuff.

3:27And the guy who created the term life hack, it was nominated for word of the year, I think in 2003. Okay. And it lost out to podcasts, which was the word of the year. amazing i think the pod comes from ipod i yeah i believe so right because it was an apple right sort of developed thing it was on itunes yeah right with the rss feeds and all yeah you had to host it yourself anyways 20 years later they still have not really improved how people discover podcasts so the ratings are one of the biggest ways that the podcasts go up in the ranks and people discover it so if you would not mind please leave a review on apple podcast or spotify podcasts We appreciate it.

4:02We really appreciate it. And we want to bring you Denoise News and have this have not just value on your career and on your professional life, but also a little bit of fun. You know, we like to laugh and riff on things. And we hope you enjoy the conversation. All right. And speaking of interesting things and AI, we have talked about this before, but now there's an official ruling with new rules from the Academy Awards. And they have officially clarified their stance on generative AI. No AI, right? No, they said they're looking at it as it is a tool. And you are free to use this tool. Basically, there's literally just one paragraph in the new rules.

4:41I'll just read it because it's pretty short. Sure. With regard to generative artificial intelligence and other digital tools used in the making of the film, the tools neither help nor harm the chances of achieving a nomination. The Academy and each branch will judge the achievement, taking into account the degree to which a human was at the heart of the creative authorship when choosing which movie to award. That's fair and square. Yeah, it's a tool, as we've been talking about, and as they see it. And I felt, you know, where do you draw the line? We talked about this before. Where do you draw the line if you're like in a track, like what generative AI went into a project and at what stage?

5:12And, you know, so yeah, I think this is a smart decision. Yeah, the Academy really unlocked a lot of filmmakers in today's world, I think, with this simple paragraph. Because you're up against crazy budgets and crazy resource limitations. And there are these tools that are incredibly helpful, but you're not sure if you use them, then your project gets disqualified from the Oscar. So now I think this alleviates a lot of concern. Of course, you can't just generate something fully and have little to no human involvement and expect to win an Oscar. That's just never going to happen. No, and I think that's where the voting comes in.

5:50You would not get nominated or voted. I mean, I am a little like wondering because it's, you know, the second part, each branch will judge the achievement, taking into account the degree to which a human was at the heart of the creative authorship. How do you know that? Like, is it based on interviews? Is it based on director statements after the fact? Like, how do you? One good thing about the Academy, I will say, is they're incredibly technically capable because the people that are on the SciTech Committee and stuff, they're very well aware of all the AI tools and the practices. So when the short list of movies gets to the top of the chain, and my guess is there'll be a vetting process, there'll be a formal Q &A process, and they'll quickly determine if the usage of AI was in line with their expectations or not.

6:35yeah and then the voting members can make that decision which as it should be i mean the peers are voting on peers and yeah you know like it's their call if they feel like it was authored enough by human or went too far with ai use depending you know we'll see how that plays out in the next few years yeah one of the things that i always wonder about is like okay visual medium like a frames of an image that's pretty easy at least to me to see okay what part of it was traditional CG or AI, like based on knowing about the pipeline or talking to the VFX supervisor, you can quickly determine, okay, is the use of AI here fair use and good, you know, it enabled a filmmaker in a certain way, or is it just unethical, if you will.

7:17Sound and composing music, like that stuff gets way murkier. Like, have you used Suno? Like some of the stuff on Suno sounds really good. Yeah. And it's hard to tell. It's like a background beat or something. Yeah. And it's hard to tell how much human intervention went there, right? Like if you mix a suno track with the human tracks and a human instrument, human plate instrument. Or if you use it to generate some sort of, yeah, some background beat or something. Yeah. Yeah. A track instrument that gets mixed in with the real instruments. Where does that? I don't know enough about audio production at the highest level to kind of weigh in, but my guess is like other use of ai outside of frames will come into play as well all aspects of filming yeah and i would assume you know if you use tracks on your soundtrack that were completely generated with suno that would disqualify you from like best uh for like a best score nomination or something yeah especially because uh i think suno as a platform is still a question mark on how they train their model uh-huh and what the outputs are and if they're copyright clear or protectable Yeah.

8:21Yeah. Like, I mean, also, that's I don't think that's the Academy's job to make those calls. But obviously, that's up to each production and studio to figure out like as well, like what is there since the copyright stuff is still a bit up in the air, as far as you know, are the outputs protectable? That's more of a project to project decision. Does this feel like it's a step in the right direction? Yeah, I thought some of the early things were some of the early ones that was that early ideas was possibly like disclosing or tracking all the use that just seemed a bit kind of cumbersome. And like, where do you draw the line of?

8:49You're going to need a whole like overhead department to monitor the use of AI. Where's that budget coming from? The pitch deck that I made like three years ago, you know, that I used some AI generated images, but yeah, that maybe never came into play again after we got funding. Like, where do you draw the line of like tracking what tools and stuff get you? So I think this makes sense. You know, it's like, yes, it's a tool and it's up to the voting body to kind of make the call on their peers. goes back to the James Cameron little title card on Avatar, the next one. Oh, no generative AI was used.

9:21I guess you don't really need that. Yeah, I'll be curious if they kind of just remove that when the film does come out. Yeah. If people had forgotten about that, or maybe there's more acceptance of AI or less bias. I don't know about the bias thing, as we talked about the last episode, where people were biased against AI generated stuff, but didn't have a preference. If it was AI generated. The public bias won't go away overnight. That'll take a long time, possibly years, because AI is not just impacting media and entertainment. It's impacting automotive and industry and, quote unquote, taking away jobs.

9:53So there is a negative connotation across the board. Although I see it as an enabler in all those industries. The same person can now do their work better. The company can output twice as much and so on. Having said that, the financial reasons why AI is getting adopted in M &E is rock solid. right like if they see like ted's randos if they see a 10 efficiency they're going to go for it why would you not why would you not yeah if the tools are there and if it keeps the quality the same and for the most part keeps the same uh amount of people employed yeah i went to a dinner party a couple weeks ago and somebody sort of reminded me what we already knew was like people getting in the way of technical progress never really fared well like throughout history Yeah, it's the Indiana Jones boulder.

10:40It's coming. Yes. So like, I'd rather get ahead of it than get crushed on it or have a spear impale me. The other interesting thing that came out of the new rules for the Academy Awards that I think also got more attention than our little AI corner is now you're required to actually watch the films to vote on them. You mean the whole time there was voting done without watching the film? There was not a requirement or a rule that said you had to watch the films before voting on them. Oh, boy. Yeah, you need to watch all the nominated films to vote in your category, all the films in your category to vote in your category.

11:17I am guessing this is probably more of a tech thing, where it's just like too hard to enforce and track that in the past. But lately, there is basically like a Netflix-ish streaming platform for Academy members where they can watch all of the films. And so that's obviously trackable. So they can track if you use that system, if you played it. And then I think I saw some reporting somewhere else where like, if you see it at one of the Academy screenings or something, you have to just like, I don't know, send a ticket stub or provide proof that you like saw the movie. But they are tracking it. I'm guessing it's more of just the tech is easier to implement this.

11:47Yeah. I mean, the Academy members at that high level, they're very busy people. Right. So maybe they're just they just don't have 10 hours. Like if it's five nominations at two. Yeah, I get that. But then don't vote. you know i mean yeah a lot of times it turns into uh you know people voting for their friend or more just i mean that is the all a lot of the art of oscar campaigning and stuff of like you know is it really for like the the the merit worthiness of the film versus there's a story behind the actress or the film or something and people are kind of voting for the story or voting because they're yeah i mean end of the day you should vote for this person yeah you should vote for the end product yeah yeah or at the very least watch everything that is nominated and then make a decision even your decision is biased which i mean sure of course it's gonna be biased yeah so you know glad to uh glad technology is helping there dude the academy is uh is doing some good stuff over there yeah but yeah overall a good one and then also they've got a new award casting director there's a new award and then in a few years they're having stunt coordinator oh they added a new category they added a new category coordinator yeah okay I think that's rolling out in 28.

12:55So John Wick's going to win that one. If they just bust out a John Wick's five or six by then. Yeah, I heard the John Wick franchise is going to keep going and possibly have a couple more spinoffs. Yeah, well, they got that other, the Anadarmus one coming out, Ballerina this summer. So yeah, maybe Ballerina 2 or 3 will have the stunt coordinator, Oscar win. Excited to see that. What do you think about, because there's always talk, it's like if they add more awards, then the telecast is going to get longer. and with the more technical awards, you know, general population gets bored. I'm for if they want to move the shorts off broadcast.

13:29I feel like the shorts are always a kind of a weird thing because you've like... I like the shorts. Never seen them, never heard of them. I'm not seeing it really category. I'm just saying, does it need to be on the telecast? Yes, because that is the quote unquote DEI version of including the small, low budget filmmakers who can't afford to make a feature. This is their chance at the highest level. I mean, they can still get the award. They can still get honored. Yeah, just don't include it in the broadcast, you're saying? Yeah, because no one... I mean, it's really hard to see these films. And unless, you know, sometimes some of them were like...

14:03I think the one that won a few years ago was like an LA Times documentary produced on one of the music schools or something. Okay. But I mean, for the most part, it's like no one's ever heard of these films or seen them. That's their... I mean, if you go to the... You can't even watch. I mean, it's very hard to even watch them unless they're already playing somewhere online. It's about the recognition. So nobody wants to be recognized off camera. I don't think. No, but these are shorts. I'm saying as someone who the closest I've ever come to like Academy Awards was I had a film that was shortlisted, didn't actually make the qualifying process, but it qualified for the Academy Awards for a short doc.

14:36Oh, that's cool. Yeah. I won the best doc at a festival that qualified it to like be nominated. Okay. It was not actually nominated, but that was the closest it ever came. Yeah. I would totally understand if it was like, oh, hey, cool. You know, your short doc was nominated. But like, it's not going to be on the telecast because, you know, but you can still go to the governor's ball, whatever the award's saying, where we will present the award and announce it. It's just not going to be on TV. I would understand that. If it also cleared up spots for like other roles that worked on future films.

15:00I guess to your point, like the Academy Award, even just a nomination can make careers, right? Yeah, for sure. And you don't need TV coverage for that. No. Yeah. I mean, even if you, right. I mean, you could say I have Academy Award nominated director. Yeah. Even if the nomination was for best animated short. Sure. Like still worthy enough. Yeah. The Academy is probably under pressure to change the format of the show as we're moving through time, right? Like it's a very classical format. It's very long. And now we have shorter attention spans. There's YouTube. There's all these other things. So my guess is like they're just going to keep doing what they're doing.

15:39and hope that it's just so etched into pop culture that they don't really need to change. It's kind of like Saturday Night Live. Like that format is from the 70s, right? And now they're like still producing the show as is, but then they're chopping up bits and putting it on YouTube as shorts. And they're sort of modernizing with the times. But something like the Oscars, something like the Super Bowl, it just has to be the way it is. Yeah, I mean, I'm not for cutting out any of the technical awards. Like, remember a few years ago, they kind of mixed it up and they gave out, I think it was like best editor and sound mixer, either during the commercial break or it was like some weird thing where they just like walked down the aisle and like handed them the award and they didn't really get to do a speech.

16:21It was bizarre and weird. And that's not good. And there was a lot of backlash, I think rightly so. So like, I am not for that. I think. I think just keep it focused on the features. Yeah. Of all technical levels. Look, I mean, there's a sound design and sound mixing award. 99.9 % of people have no idea what the difference is. I have a hard time explaining the difference, like why there's two separate awards for that. But don't cut that. I mean, that's still part of the art of filmmaking. Yeah. Now you got me thinking about sound design and sound. All right. But yeah, if you got opinions about, you know, modifying the Academy Award broadcast, let us know.

16:53Give us a shout. Should the shorts stay on or should they go to the streaming service? All right. Next story. Tom and Jerry. Yeah. one of uh my favorite cartoons as a kid i'm dating myself but this is before computer generated pixar style animation tom and jerry is what a lot of uh older millennials grew up on there was a period i don't know if it's the same with you but like cartoon network yeah i feel like there's a period where they didn't they seem to stop making new cartoons yes and so a lot of the stuff was the 1960s 70s hannah barbara reruns like uh flamestones and jetsons scooby-doo johnny quest the original johnny quest not the remake johnny quest yeah that that and tom and jerry looney tunes that whole uh bugs bunny so yeah i love tom and jerry yeah so nvidia and stanford yeah a bunch of universities so we're talking about stanford dc san diego uc berkeley uh with nvidia at the helm they created a new video generation model called test time training so right now with video generation one of the big issues is length right and I think you and I were talking about this the longer of a video you have the longer the temporal consistency has to be from frame to frame to frame so naturally the heavier weight the memory if you will of the AI model has to be it just has to remember what it what the attention was on each frame as you progress through the frame and the more frames it's got like go look back yes to make sure that the whole thing makes sense but every time it looks back i believe it's now like logarithmically expanding it's not like oh if it's a one second or two seconds it's logarithmic yeah three seconds that it just adds one it's like no now it's like x logarithmically the amount of tokens and memory it needs is it gets astronomical absolutely and that's one of the main reasons why video generation is so much more compute intense than image generation so this research paper goes through a new training model called test time training, where they reduce that compute overhead by a big factor.

18:57And they are also able to get super temporal consistency to the point where they could generate, in the case of Tom and Jerry, a one minute video, which is unheard of from a single generation. Yeah. And so do you know, like, what is test time training doing that is different? So what it's doing is essentially on the attention layer side. Attention in an AI model, I know it sounds like a very vague term. So when an AI model generates something, it has to focus, it has to put attention to the thing that it's generating and also cross attention to the thing, other things in the frame. and then it also has to have self-attention which is uh that in the last frame it generated tom so now this frame has to generate tom so that's self-attention and then jerry would be like a cross-attention because it's uh it's you know another thing in the same frame so there are all these attention mechanisms built into the video generation model this has i think a different fundamental attention hierarchy.

20:02Is it basically instead of reviewing the entire video kind of learning within itself while it's making the video so it doesn't have to review the entire thing? Yeah so they released the paper as well as code so you can actually build this into your video generation model which is super nice. Sounds like where a typical video generation happens is you go back in the number of frames and the attention has to be through frames so you're talking about in a 24 frame per second video, 30 milliseconds of attention span at a time. TTT is doing it with three second segments of attention units. And so because the time span is much longer, it's able to actually store a lot more information about each of those three second segments, which results in more consistency in the next three second segment and the next one and so on.

20:55so instead of just having a frame reference it's having a sequence a shot reference if you will so it's just a higher level of abstraction i'm not sure how they achieved it but it seems like it's computationally as efficient as a regular video generation model yeah and that yeah it is a way where it can kind of just figure it out as it's going along which is it versus looking at the entire thing again and so the there was a sample video of like a minute long yeah can we take a look yeah yeah so yeah we got the scene and so the interesting thing was it uh the inputs were basically they did have to give it a text input text text is a little bit jumbled yeah um that walk cycle wasn't that great there yeah and also no people are moving but uh tom looks great i mean this looks like tom tom yeah even the style of the world is very consistent yeah and there's a computer which also it's like that would not have been in the original tom and jerry training so right uh i mean the keyboard is jumbled but it still works his mannerisms are like still so good and so is uh jerry yeah like he's hitting the poses that yeah i mean jerry just disappeared there but look the feet the spinning motion of the feet is there the like behaviorisms this and even when he hits the wall of that that like comic book yeah explosion thing dust uh explosion oh i love this dog here yeah he's he's the bully right but so to make this they had to give it a text input basically a detailed shot list right but it's still a single text input and this entire minute long video was the output i wonder if the fade out was even part of it too yeah because the original show had that same fade out yeah this is so impressive because you feel like that could never be a single generation right like you look at that and you're like oh i'd have to like generate shots and rate shots and then edit it together yeah and every time you do that you're going to have discrepancy between the shots, right?

22:42Because you're going to, oh, that shade of color is different than this shade of color, whatever. But yeah, the thing that this unlocks, if this is able to kind of roll out in a, I mean, they trained it on, I think, billions of parameters of like Tom and Jerry material. So it has a lot of training data. But if this could roll out into, you know, give it a single character image or something, that could be huge because it unlocks the issue, you know, we have like consistent characters. Where right now, a lot of the process is create a character sheet and then try to create the frames you want with a consistent character.

23:14And then you give that to Runway or Pika or Luma or whatever to generate your videos. Correct. But if you could give it your one character and then be like, oh, make a sequence with the same character, that unlocks having much more consistent characters, environment, scenes, if it's all being able to be done in a single generation. That's so cool. And then on the topic of character consistency, it's more difficult now to have multiple character consistency. So like having Tom and Jerry in the same frame and then have them in the next frame and next frame, like without distortion, that's difficult.

23:47And they seem to be doing it without the step that you mentioned of generating the frames and then feeding that into a video generator. Yeah. Yeah. Very cool. Yeah. Awesome. Awesome paper. And yeah, this came out a few weeks ago and it came out like right before NAB. So I was preoccupied. We were preoccupied. Yeah, so I wanted to come back and revisit this because I think it was really impressive what this unlocked and what this could lead to in the future. Yeah, I think we're going to see longer length video generation from the commercial manufacturers like Runway and so on very soon because why not, right?

24:21And now you're at an upper limit of maybe, I think, 20 seconds with those guys. And now you're going to a minute and then who knows, maybe in a few months we're going to push to five minutes. generation and shots switching shots too because it's like one thing because a lot of times it was like if you make a minute long pov drone flying video it's like cool but what am i gonna do with that but yeah if you can give it like they did with tom and jerry where it's like oh here is our shot list that we need yeah and then you generate that i wonder if at this sort of a mid quality level can an artist go in manually and fix all those things we noticed and have it usable like even at this level oh like just come in and clean it up yeah like a paint fix probably yeah yeah i mean it's already so usable yeah yeah for sure i mean this could probably unlock a lot for even just youtube kid cartoons yep oh big market over there yeah yeah all right and then our last story descript video editor app we covered descript before we've covered descript before i'm a big fan i also think there's a lot of things they can improve from a professional video editing standpoint But they have teased basically vibe video editing.

25:28So they are in private beta right now rolling out, but they rolled out a launch video for basically a chat interface inside Descript where you just have a conversation and kind of tell it what you want. And it edits the video based on your feedback. I think that's a intermediate step. Like we're not going to be prompting our way through video editing. Ultimately, video editing will be done through a timeline. it's just that it'll be a lot easier of a timeline task than what how it is today yeah i mean i don't think any editing tool can get around like i think this is interesting because you know i've seen other text-based video editing tools i've messed around with like eddie ai i didn't really think it was quite there and it was just the text prompt interface it gives you back the thing you can take it or leave it and then you could send it to your video editor but you can't like manipulate the timeline there descript because it's all built in you have this chat interface It uses the already existing AI tools that Descript has started adding, but then it's making the changes to your composition, which is either the text-based interface that Descript is known for.

26:29I still think they have the best text-based editor, or you can pop up the timeline on the bottom. And then if you're familiar with the timeline editing, same exact interface, it's there. You can manipulate and fine tune this stuff. Yeah, some of the examples they're giving were, I mean, a lot of their use cases are more on the social media, marketing video end. I've probably pushed it to the limit of like what it can do, giving it hours and hours of interview footage. But their main use case that they focus on is like podcast editing, you know, angle switching, layouts, trimming out bits and low hanging fruit stuff.

27:02Yeah. Marketing video, tutorial videos, very much geared towards like content creators and marketing teams. And so the use cases are targeted for that where you can give it, you know, a text instruction of like clean up this interview or like edit this interview and, you know, switch ad B-roll shots. And then it kind of understands the context of what should be shown there. I think in the demo, it also included some examples of, you know, the person is talking about this thing and this B-roll shot. So we should like edit that here. So interesting, promising. Yeah. And I think for the market they're targeting, like the more consumer end of things, maybe this is enough.

27:39I do think the winning formula is in the hands of Resolve and Premiere. The way you can bring in generative elements into Premiere now, like a B-stock footage or whatever, or extend the timeline past what it's capable of, generatively fill it. In Resolve, you can do the rotoscoping, the matting, the power windows, podcast, camera switching, all of the AI. Like adding AI super tools into an existing timeline based tool. Yeah. I think that's the way it's eventually going to go. For sure. And especially for professional video editors that are already working in these tools. Like these content creators, when they get big enough, the video is handled by a professional video editor.

28:19Yeah. Right. So there's like a small intermediate step where they don't have enough technical know-how to really produce a polished edited video. And for that, the script is perfect. or even just if you're looking, I mean, even if you have a professional video editor and you're just looking to put out more content, I mean, this is a bottleneck we have right now, like with all the NAB stuff where it was like, you know, I brought on some additional editors, but even with that, it was like between editing the videos, but then also like we want to edit short clips and social media clips. And so, you know, we're using Opus for that, but Opus, you still have to clean up.