How AI is changing filmmaking in Hollywood, with Tom Graham

0:00Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential. a non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up. Every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no preset spending limit, they were able to spend more and earn more rewards while bringing their vision to life.

0:36Today, Raz's success is proof that with passion and the right support, it's possible to make your dreams a reality. Learn more at CapitalOne.com slash business cards. Pioneers of AI is made possible with support from Inflection AI. It's not just enterprise AI. It's your enterprise AI.

1:04so recently at the vmas we won a vma award for the music video we did with eminem where we brought slim shady back tom graham is the co-founder of the ai company metaphysic and they did bring back Slim Shady at the MTV Video Music Awards. It's the song Houdini, and Eminem's in the music video. He is kind of fighting a battle against himself, and himself is Slim Shady. And then his today self and Slim Shady merge into this kind of like hybrid. It's a lot of fun. The song's a lot of fun. Slim Shady is the alter ego of rapper Eminem, but more specifically, the younger version of the artist. Metaphysics technology allowed for a realistic and younger version of Eminem to appear alongside the actual rapper.

1:55But that's not all. During the VMAs, where his performance opened the VMA Awards this year, we had live real-time a Slim Shady, where it's a young performer who's fantastic, kind of rap battling with Eminem on stage. But what's happening there is that the camera is taking an image of Slim Shady and Eminem next to each other and putting it through our computer and we're adding the young Slim Shady face, head and shoulders onto the impersonator. And then it was going out live broadcast television. So people in the theater just saw the impersonator, but anyone watching the live feed from home saw an AI version of young Slim Shady generated in camera in real time.

2:40And I think that's the first example of generative AI feeding into broadcast television. It's certainly the first time that's ever happened. Using AI visual effects during a live televised performance is just the beginning for Tom. His company Metaphysic is changing the future of Hollywood. They're making AI-powered visual effects for movies and TV. On this episode, we talk about AI in the entertainment world, how metaphysics technology works, and its most recent debut in the new Robert Zemeckis film, Here, with Tom Hanks and Robin Wright. I'm Rana El-Khalyubi, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

3:39Welcome, Tom. Thanks for having me. It's good to be here. So I just want to start kind of with a broad lens on what's happening with Hollywood as it relates to AI. We've seen strikes from the actors and animators unions and AI is a major sticking point with all of that. Even back in February, which seems like forever ago in the world of AI, Tyler Perry put a pause on expanding production studio that he was working on because he could see that AI was going to change everything, especially the traditional ways of making film and TV. So how would you describe Hollywood's relationship to AI right now?

4:15Yeah, I think that the most important thing for me is to focus on the human part of that relationship. Love that, by the way. We're all about human-centered AI here, so that's awesome. Yeah. If you see an AI-generated version of yourself, and it's very realistic, and that AI-generated version of you is doing something that you didn't do, it is very concerning. you have a strange feeling, not just of uncanniness in terms of your relationship with yourself and reality, but like if you feel kind of sick and unwell. At the core of it is it's really disenfranchising you from controlling your body and the outward expression of your body.

4:53So when you come to Hollywood, people's performance is really their body and who they are. And they get to do that and they get to choose to do that in front of a camera or in front of an audience. And so anything that disintermediates that person's control over their performance is not a good thing. And so we've always focused on the idea of consent when it comes to creating an AI version of somebody. The reality today, like kind of up until today, if we just suggest that today is like a bit of an inflection point, all of the technologies to make realistic AI generated human performance have mostly been outputs on top of human performance.

5:31So like a deep fake, you take one human performance and you put something on top of it to change the appearance of it, which is manifestly different than using AI to fully create the performance from scratch with no initial human intervention yet. That kind of, up until today, it moves a lot of the use of AI in Hollywood into the realm of it's a tool that is used creatively by people with their consent, just like CGI or VFX. So there is an initial period of generative AI exploding into the world and people feeling that deep emotion of concern, which is the right thing to feel. Because that technology is very powerful, we should harness that concern to drive institutional policy regulation responses.

6:13But I mentioned this inflection point. We are certainly moving towards a world where human performance in its entirety can be created by these algorithms in a way that regular audiences might not be able to tell the difference. And so that's where consent becomes even more important. So give me the elevator pitch for metaphysics. Yeah, so back in early 2021, we were experimenting with kind of AI-generated content in the context of deepfakes and autoencoder architectures. Autoencoder architectures. An autoencoder is a type of artificial neural network. You know, the machine learning algorithm that is the basis of a lot of the AI we see today.

6:56An autoencoder's superpower is that it's really good at learning data representations in an efficient way. It's useful for tasks like facial recognition. And we built the company to kind of build the software and infrastructure to scale photorealistic AI-generated content to kind of everyone on Earth. Tom is talking about democratizing access. Metaphysic wants to enable everyone to use and create realistic AI-generated media. That's the mission. When you understand that it's kind of a data science pipeline to create this content with algorithms, it's a software problem. And so you see quickly that premium quality content that looks like reality, that we can immerse ourselves in and enjoy the benefits of emotional responses, which are elicited by reality, harness those, become scalable because that software can scale.

7:49The hardware is there. It's not a hardware problem going forward. It is a software problem. So the thing that I'm really excited about is can we create immersive personalized experiences, which are like a Star Trek holodeck, but content-wise, maybe from our memories, maybe from our loved ones' memories, could you relive your kid's first birthday party or your first birthday party? And can you interact with that? Inside that idea is the data that we capture in the real world becomes kind of a repository of human knowledge and understanding. It's our library of Alexandria. And so if we can harness that to create communication between future people, then you can build more empathy.

8:30fear you can build, lots of positive human emotions. There are many bad things that can happen also. And we need to really diligently, from the top down, from regulators through the people designing products, think very carefully about how we harness the technology and make it safe. But the benefits, I think, are incredible.

8:53Today, metaphysic is not directly consumer-facing, Right. And so it's not a tool where I can, you know, download metaphysic and create a digital twin of myself. But it sounds like this is where you would like to go and this is where you see kind of the roadmap for the future. Yeah, we chose not to kind of open source or make both set of tools available for retail applications because of the bad things that people can do, really. So from political misinformation to non-consensual image-based abuse, which is AI generated, all of those things are immediately harmful to individual people or have a broader social context.

9:28If we can create content that looks exactly like reality and people can't tell the difference, this is not something that we should open source or allow to be used in a general context without the right type of content moderation. Increasingly, platforms are being able to deploy content moderation in a way that may meet those needs. But I think that that's a couple of steps beyond where we are today. I do hope we get there in a way that's safe for everyone. But for now, Metaphysic is using their technology for some pretty awesome applications in the entertainment industry, like the Slim Shady clone.

10:04And it all started with a viral deepfake of Tom Cruise playing golf. So, you know, going back to when we started the company nearly four years ago in early 2021, there is no one who has any idea what we're talking about, period. My co-founder created Deep Tom Cruise, which was the first AI-generated content where hundreds of millions of people thought, oh, that must be Tom Cruise. But it wasn't actually. It was an AI-generated version of him on top of a fantastic performer. Hey, listen up, sports and TikTok fans. If you like what you're seeing, just wait till what's coming next. When I saw Deep Tom Cruise and I rang that guy up, Chris Ume, my co-founder, and asked him, what are you doing that's different?

10:48He's like, oh, it's kind of data set on top of data set algorithm. Okay, that's a data science pipeline. Great. That's software. How many people know how to do that? And he's like, seven. So four years ago, nobody knew anything about how to harness this technology to create content. So much so that we're looking to hire people. And there's only a couple of people working on master's theses that have some kind of relevance to what we're doing because they hadn't even got to PhD level yet. About two years ago, GPT and generative AI blew up. And so increasingly today, everyone is focused on how to create content with AI, but the data science nature of this pipeline, going from real world data to final outcome that looks real, is a very difficult process.

11:30And there are no software components, libraries, primitives that you can just plug in to do this. So you have to build this from scratch. So that's what we've done. But you can build the software, but then any data scientist will tell you, well, I've got all the tools, but one data scientist is better than a different data scientist, right? Like, how do you featurize the data set? How do you parameterize the model? How do you bring these things together? How do you solve problems? That is a difficult skill.

11:56Tom says that because this technology is so new, we haven't seen tons of commercial applications of it yet. But there is one pretty big project using metaphysic that you may have already seen on the silver screen. After a short break, we talk about how metaphysic made Tom Hanks and Robin Wright look like they're back in their 20s. Stay with us.

12:33If you've spent any time building AI products or leading technical teams, you know this. Transformation doesn't fail because of ideas. It fails because teams can't move together. Enter Atlassian's teamwork collection. It has planning in JIRA, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at Atlassian.com slash TeamChanger. That's A-T-L-A-S-S-I-A-N dot com slash TeamChanger.

13:22Hey, Dad. I'd like you to meet Margaret. Nice to meet you, Margaret. Nice to meet you, Mr. Young. So the movie here came out on November 1st. It's a Robert Zemeckis film with Tom Hanks and Robin Wright. And I am so excited to see it. Your company plays a major role in the filmmaking process. So first of all, congratulations. So in the movie, you see the actors across a wide range of ages, right? And Tom Hanks, I believe, is 68 and Robin is 58 years old today. But we see them at a much, much younger age, right? So I'm so curious, how did your company help with that filmmaking process? But to kind of contextualize this, I want you to take us before metaphysics existed.

14:04What would they have done as a team if your technology was not around? Traditionally, if you're trying to create digital humans, digi doubles in a VFX, CGI, computer graphics sense, the prevailing technology is 3D modeling. So you might do a photogrammetry session where you capture images of different angles of somebody's face. That's the one where you've got the little dots? Yes. So you might start by capturing images of a face without the dots and then kind of mold those, shape those onto a 3D model that you've designed with software. And then you might animate that 3D model of someone's head with the motion capture, where now I'm doing the performance with the dots on my face, and I take the dots and I map them onto the 3D model and I get the 3D model to move just like that.

14:50So at the core of that is this 3D model, which is kind of like heuristically generated human programming. And imagine that, you know, if we're programming something, we can create tens of thousands of different connections and ways things move. Right. You can lift the eyebrow. You can pull the lip corner. Exactly like a rig on a face. There might be 200 different things, levers that you can pull, cheek up, nose to the side. But in reality, when faces move, there's billions and billions of combinations of things. And the interplay between a smile and how your ears wiggle or how the light on your forehead changes is so complex.

15:27This new genre of technology, which is essentially neural nets, which are trained on imagery and video from somebody's face over maybe half an hour of video or something like that. you can train the neural net to understand the interplay between those different parts of the face as they move, such that if you go into that neural net and conceptualize it as a 3D model of a brain, like a human brain, and you grab the neuron that adds cheeky smile number five, 30 % more cheeky smile. Because of the entanglement between all of the different expressions and facial movements around cheeky smile, when you grab cheeky smile, you drag everything in the direction of what would happen if you were smiling.

16:08Yeah, you see a few wrinkles around the eyes. Yeah, exactly. And that's profound in the context of comparing it to 3D models. Because when you do that in the context of this neural net, and you go through the process of inference, which is like creating the image from what it's trained on, the process of inference is incredibly cheap, like fractions of pennies. Today, we can run live 1K by 1K inference for a face swap or a head and shoulder swap from a gaming laptop at 100 frames a second. That's how cheap it is to create one image. But if you're trying to create that same image in a CGI 3D modeling sense, you need to compute all the different movements of all the different parts of a face.

16:49And then on top of that, you have to compute the skin texture and then the lighting and ray trace all the lighting. Some models are like down to the single hair, right? How does a hair move? You get all of that for free from the neural net that understands implicitly all of those things. It's a million times cheaper. That's the core. That's the real driver that means that this technology is going to be creating content for every single one of us. But I want you to take us to this movie set, right? Were you there? Was the technology doing this modeling in real time? Because my understanding is that Tom Hanks and Robin Wright were there.

17:26They're acting as if they're like their 18-year-old selves when they met. And you're capturing this video and then in real time, basically generating their 18-year-old selves? Yeah. Fundamentally, audiences as they're watching the film, they're falling in love with Tom and Robin and some of the other characters' performance. And that's them on stage. And there is something magical about one person's performance versus what you can generate with AI today, or even other people trying to imitate that person's performance. There's a reason that Tom Hanks and Robin Wright are amazing actors. And so their performance that we see is them on stage.

18:03And that's amazing. Then there is this layer on top that can kind of work as a tool to take their performance and just make it look like it's the 20-year-old version of them. But a large part of that was really them having to act like they were 20-year-olds also, so that their visuals lined up with... Their tone of voice and movement. Yeah, and like how you kind of hold your face and things like that. We could do that live on set so they could see themselves in kind of like the youth mirror and adjust their performance live in real time. And then the director, Bob Zemeckis, he could see what it would look like in real time on the little screen as they're going.

18:42You know, it comes back to this kind of like live real time technology, which is amazing. You could never do that with CGI VFX because the compute cost of generating, rendering each image is so huge. Right. It doesn't happen in real time. So it has to happen after the fact. It probably takes days or weeks, right? Yeah. So from a production point of view, this is a really amazing development. And that's really part of that promise of this bundle of technology for creating content. It's just really fast and cheap. So you can scale it. You can do amazing things. Yeah. So this is actually really cool because you must have competed with a whole bunch of special effects studios to win this project.

19:16It sounds like you didn't just win on cost. It's almost like you're empowering the production team to be in control of this content as it's being generated. What do you need to unlock about the human aging to make it visually compelling and real? So ears are a problem. The ears? Yeah, ears and nose. I'm very curious about that. As we get older, our ears grow and our noses grow. And so if you're trying to de-age somebody, you have these things sticking out the side of your head, which are now on screen bigger than they were 20, 30 years ago. So that's an interesting problem. How do you deal with that?

19:52The other thing which is counterintuitive is generally when we think about impersonating somebody, we think about what is inside the face, but that's being kind of replaced by the AI as a tool on top of the expressions rendered by the human underneath. So it's actually the face shape which becomes the most important thing. So when we're looking for someone to act as a stand-in to then put a person on top of, face shape is really, really important. And you can imagine that in the context of prompting that people are very familiar with, the face shape is like the prompt. If you have a good prompt, it is easier to get a good outcome.

20:28If the chin's really, really too big, you might need to use other AI tools to shrink the chin a little bit to adjust the prompt before you put it through the final algorithm. You know, I spent almost my entire career building emotion recognition technology and face shape is important, but it's also these wrinkles on our face that make it that kind of, you know, if you don't have this kind of wrinkling and texture, that's where the uncanny valley happens. How did you solve for that? And I'll push you on this a little bit. Did you even solve for it? Are we done with the uncanny valley or not yet?

21:00I think it's fair to say that you get all of that for free from the architecture of these neural nets, such that what you're training into them is tens of thousands of images, maybe on top of a pre-trained that is trained on millions of images. So it understands a human face and how the different parts of it work. And then you train in a specific person and it's just looking at pixels, pixels over pixels over pixels. And so what we see is kind of like a micro expression. It is just a fundamental structure in its understanding of what goes with what. And so you can't avoid it. We have technology that can kind of like unpack these neural nets and make them navigable so we can pinpoint specific expressions.

21:44But if I add cheeky smile number five and it's entangled with all of the micro expressions and everything else down to a very, very large resolution, a fine layer of detail, it would be harder to take that detail out than to just let it come along with what the neural net imagines is the outcome from that prompt, you know, more smile. What did Tom and Robert think? Did they embrace the technology? Were they skeptical? Yeah, I think that, you know, Bob Semecki is such a frontier running director in his use of technology for his entire career. And so, yeah, I mean, he did Back to the Future and Forrest Gump.

22:20Yeah. And all of motion capture is really like being kind of driven by Bob Zemeckis and what he's been working on over a long time. He and the team really embraced this. But it's hard to explain how much of a step function change this technology is in the context of making humans that look real. If you went from a very, very expensive tens of millions of dollar process to now you can do it live real time on set and you can see it and it looks perfect. And over here, it looks a little bit weird. Wow, you can do many, many creative, amazing things with that. So across the board, I think the experience for directors and actors was a really, really positive one.

22:55I love that. Is this a major signal to Hollywood? Do you think it's going to change a lot of things? I think that if we go back to that economic statement of fact about how cheap it is to create content that looks like reality with this set of technologies, I think that it is very likely that 99.9 % of all of the content on the internet 10 years from now is AI generated, even in the context of live sports. We'll have the feed from live sports. So you've got the basketball and you take that feed, but then the feed will run through a set of algorithms which pump out the content that you look at.

23:30But the thin layer that the algorithm is putting on top is changing all of the logos and the sponsorships to serve ads directly to you. But it will look like it's perfectly embroidered on the jersey as they're running around bouncing the ball. So it will look perfectly like reality, but really that'll be an AI generated layer. And when you think about it in that context, like, yeah, obviously people are going to do that, right? It's just serving ads, like everyone's going to do that. Serving ads, personalized ads as you watch. Yeah. You won't notice that it's not real in a sense, but the computational efficiency will mean that you could probably do that rendering at the edge.

24:03So like on your mobile phone or very, very close to the edge. And in real time as you're watching the event live. That's exactly right, yeah. We're going to take a short break. When we come back, we peek behind the curtain and talk to Tom about the data that powers metaphysic. Back in a minute.

24:32Meet Nicole Nicholas, Capital One business customer and co-owner of Ansett Uncles, a plant-based restaurant and community space in Brooklyn, New York, that got its start from a need for unity. The inspiration, it was born from the desire to create a space that felt like home, where we can connect community culture, good food, and come together with family and friends. That's how we birthed aunts and uncles. Nicole and her husband, Mike, were fulfilling their dream of bringing people together out of their home kitchen. But they soon learned that the demand for community was greater than they knew.

25:03It became overwhelming and we were like, we need home, but not in our actual home. We realized that there was also a need in our community for something bigger in our neighborhood. So we had to find a place. Moving from a home operation into a storefront was a huge next step. But Nicole and Mike were able to take it on with the help of Capital One Business. It's not for the weak as a small business. Finding resources is super important because that's the way you'll be able to manage and scale. We would have never done that without having Capital One to be able to help us along the way. The cashback rewards are very helpful.