120 | Create impactful product videos with AI - Step by Step guide w/ Rory Flynn

23:36And if we're working with clients in this regard, like a marketing agency would, showing them that might give them a better perspective of what the end result can look like so we can trim time off this decision-making process. That's where it's been really helpful for us. Now, the other thing that we're using it for is basically... Or why we're using it that way is because it's smaller format. What we're doing really is right now we're using it for things like asset expansion, meaning if we have a product image, we'll take that and turn it into a video. So now one image of a product becomes two assets.

24:11Same thing we're doing, we're using it in like stories, reels, things for TikTok, YouTube. So it's like a storytelling enhancement. So if we're creating a reel for something, like typically there's a lot of shortcuts in a reel, maybe a voiceover, we can add these images or videos in to enhance the story to make things better. Same thing with product advertising, right? Like when we have these product ads, sure, you've seen a lot of static ads on your Instagram feed, whatnot, where you're scrolling, oh, here's a product, here's a picture of lipstick, right? It's a buy now. Great. We can also make that lipstick move.

24:45Now, that's a really important thing in my world, because we have a static asset of lipstick. And then we also have a moving asset of lipstick and we're running multiple tests. So whichever one's going to work better, that's where we're going to get more results. So we're looking at how do we expand our asset base to then basically test, to then optimize, to then build more assets that look just like that. So that's the story behind that piece of it. Now, why we're doing that and why we're using it externally in those mediums, mostly on social or mobile, is because they're They're small format.

25:18They don't have to be on a IMAX screen. It doesn't have to worry about resolution. They're shorter format. Again, social, you don't have to have 30-minute videos. They can be 20 seconds. Same thing with a lot of the different social platforms, whether that be Instagram, TikTok, whatever. They're all going to be 9 by 16 aspect ratio, meaning the size of your phone. So again, these are really important to us. And oftentimes, there's less scrutiny. I'm in a smaller format, it's really hard to find some of the more particular details. So sometimes we can get away with that stuff. And being quite candid, marketing is a little bit of deception.

25:56I hate to break the news, but sometimes it can be, right? So that's why we're utilizing it in that fashion. Now, how can you do this? That's what we're going to talk about now, really. So just before you dive into this, one of the questions Curtis is asking, which tool are you using for storyboards? I assume it's a mix of all of the above, depending on exactly what you need? It depends. Most of the time I'm using Runway, to be quite honest. So mid-journey to create the images and then Runway to create the videos? Okay, that's correct. Good answer. Yeah. Runway to me has given me the most consistent results and I don't have to end up spending a ton of time on there.

26:35Other ones I do, Luma especially is probably my secondary where because of the keyframes, right? I want it to start here and end here. That can be super helpful, especially if you can't get something on runway. So that's where majority of this is coming from. But everything you're seeing here is done on runway right now. So everything we're going to go into right now. But basically, to sum up what runway has done in this Gen 3 model is they have made a huge improvement from the last model, like a significant jump, not just a random update. This is pretty big. They have a text-to-video model, so you can text and generate a video.

27:13They have an image-to-video model where you can put a reference image in and generate that. The natural motion is really impressive, as you can see in some of these things here, like the roller skater up in the top right. He's moving. That doesn't really seem like AI. Very smooth motion. Body movement is correct. The prompt coherence is strong. So when you're text prompting in there, it's going to listen to you, meaning you have to be better at prompting it or else it's just going to go off and do its own thing. The output videos, you have an option here, which I love, which is five seconds or 10 seconds.

27:44The 10-second ones are more expensive, of course, from a processing standpoint, but they're really good, especially if you're doing something in slow motion. Just a little tip there because the slow motion ones, they end up with... Basically, you get more frames per second, so it's just going to look more smooth and natural. The Gentime is surprisingly fast. Their new turbo model, this can be done in less than 30 seconds after a generation. So it's pretty impressive that this can render that quickly. But the output... As our CEO said, it renders faster than you can write your prompt. It does. That's not a joke.

28:17It's amazing. It's really incredible. Yeah. And the output resolution is 720p. So it's not the best yet. But that being said, the other reason why we don't use things in large format is because to me, the upscalers in the AI video space are not great. Images seem to have that figured out where you can upscale an image get really high definition quality. The video is way harder. Hoping that one of these tools really does makes a concerted effort to put a video upscaler in there because I would love that feature. But right now you have tools like Topaz Labs. They can do some upscaling. Their new model is pretty good.

28:54It's just it runs locally on your own processing. So it can take a really long time. A 10 second, 30 second video for me. I don't have exceptional hardware to be quite candid. But maybe a 30-second video, if I was trying to upscale, it could take two or three hours. So that's something I'm going to wait for. But regardless, that's why we're using this tool. Now, for prompting, there's always some frameworks that you want to have in going into this. I try to keep my prompts very short and punchy first. If they don't work, then I expand them. Because really, everything is built on tokens the same way chat GPT or any of the large language models are.

29:33So the more tokens you add in there, typically, the more confused it can get, especially if your language isn't punchy or pretty compact. Anytime you add a lot of fluff words, it's just pulling randomly from its memory, and it's going to affect the output. So I try to keep things very short, concise, visceral. Oftentimes, if something is not working, I'm overly communicative. meaning if you want someone like you want a shot of feet walking away from you like a like maybe where someone's walking on a sidewalk but you just want the feet walking right sometimes if you just prompt that it'll have the feet facing forward and walking backwards towards you so it's a very weird sort of visualization it's because you didn't tell it walking away from us right like instead of just walking they're walking away from us so you're giving it the direction of here we go.

30:27The character is walking forward and we're shooting it from behind. It just takes one extra step, like trying to explain it to someone who cannot see. I think it's the best way to think about prompting is you're explaining something to... It's like you're writing a novel. You want to tell people exactly what's happening so they can build a picture in their mind. Now, again, with this text prompting, I'm going to focus on this first. It's really important in the output with runway runway you can just put an image in and then press generate and it'll do whatever it wants but really you control the experience in the text prompting portion of it now it's really important because it's controllable and that's what i mean is that you have significant control over every little aspect of the actual image or video output but there's certain things that you need to look at whenever i take something into account whether this be doing something on mid journey or doing something on runway, really I'm breaking things down into its DNA.

31:26So an image, right? If you have an image, like what is the shot type, the subject and the character, the environment, the colors, the textures, the composition, that's all important. Same thing with video. When we break this down, there's just more variables, right? So we try to break this down. I like to use an example of the difference between And pizza and a calzone, right? Essentially the same thing. But when you take pizza and you break it down into individual components, it's dough, it's sauce, it's cheese, you can then rebuild those three ingredients into anything else you want. So a calzone can come from the same three ingredients.

32:01That's how I'm thinking about breaking these sort of prompts down, right? Now, the variables here that we look at, there's the shot type, which is important. That's going to be in both image and video. But the camera movement is something specifically because that's something you can control. Now, you might not think that's a big deal, but the difference between a static camera and something that's orbiting in a 360 degree around an object or a person looks totally different, and it can be way cooler. So camera motion is a big one to me. Again, we always look at the subject and action. So who or what is in the image and what are they doing?

32:33The environment where it's set, the lighting plays a big role in storytelling. So whether that be something that's happening in the morning or at night, or you have a silhouette lighting versus something that's really bright. It tells a different story, each type of shot. Color grading can also be utilized within here. So if you want to keep a consistent look and aesthetic across a movie or an ad or anything like that, having that color grading in your prompt can be really helpful. Cuts and transitions, this actually works in a runway. I wouldn't say it works all the time, but if you want to have specific cuts, like a quick cut to or a wipe cut to meaning like, oh, we want to show somebody in one area and then quick cut to them in another area that can be done.

33:16And then also the style, which I find to be the easiest way to guide runway here. So think about there's a lot of different styles in there from a 35 millimeter might be your standard site photorealistic look. Cinematic is going to be more big and exaggerated animation that'll give you more of an animated illustrative style or something along 3D render or Pixar, things like that. That'll give you more of that look that you're looking for. It's basically like a general holding sort of idea for runway to then go and that's what I need to do for the whole thing. So those can be really helpful. Now, so I want to pause it just for one second because I'm sure some people, especially people listening and not watching the screen, lost us with all these parameters and people like, how do I, what do you want me to do?

34:02Write all of this down every single time. I'm like, it's just like best practices for any of these AI models. you need a prompt library that has the structure with all the different components in it or the components that are relevant for that shot or have all of them and you can delete the ones that are not relevant for that shot. But then that makes it very easy. You have your camera motion, you have your subject, you have the action, you have the lighting, you have it in a prompt structure. And I'm sure you have examples, so we're going to dive into them in a minute. But you have the prompt structure and then just filling the parameters.

34:35And what parameters you have, This can also be a library that you have a cheat sheet on the side in the beginning. And then once you've used them 50 ,000 times, like Rory, you know what they are and you know the look that you're looking for. But if you have the vision in your head of the outcome you're trying to create, you just need to be able to translate it into professional terms that, again, you don't need to invent. They exist. Like these camera motions exist. The action exists. The color tones exist. So in the beginning, literally have a cheat sheet on the side of your desk or in a document that you can open from Notion or Google Docs, it doesn't matter, and use the cheat sheet in the beginning until you get a feel of what works, how, and so on.

35:13There is no magic. There is literally a prompt structure and then what can go into every second, every segment of the prompt structure, and then just repeat this 50 times and you'll know how to do this pretty consistently. Great point. That's exactly, we will go through some prompt structures here so that it's easier to start with a formula and then just plug in some stuff, right? Because it's what you said. And it doesn't even have to be off the top of your brain. You can go and utilize a prompt formula in something like ChatGPT. Can you fill this in for me? Thank you. It can be very useful there.

35:48But I think the biggest hurdle for myself personally when I started doing the image generators is you're working in a different dimension, if that makes sense. Images are very flat, right? You can create some depth in an image, but you can't control motion within an image. Now, that's where this comes into a little bit bigger of a meta theory here. It's where we're working in four dimensional space, if I'm conceptualizing that correctly, because you can move forward, backward, up, down, diagonal, towards the camera, away from the camera, right? So there's a lot of different ways you can move. And it can really alter your prompting.

36:22We have to take that into consideration when we're prompting because there's a lot of depth within the video now. It's not as flat as an image. So really what's important to me when I'm looking at this is the camera motion and the subject motion direction. This is very granular, but those two to me are the ones that have the most effect on the output of a video. And it really just gives you more control if you know what to do with them. So I'll show you a couple of examples here. This is a very simple prompt structure. This is someone everyone can go and take it, use right now, I guarantee you, you will at least trim your reprompting down about 80%.

36:58So typically, I'm using this simple structure. It's shot type or camera movement, then the subject in the action, and then a style, right? So if I'm using an example of that's going to be a 360 degree orbiting shot, meaning there's just a camera, think about it like a drone shot, someone standing in the center of the frame, and the camera's just circling around them, right? That's an orbiting shot. Now, of a man on a mountaintop, right? And then cinematic is the style. So it's going to look and feel like it's cinematic. And the camera is going to orbit around the guy, right? So when we do that, you see the prompt, very simple.

37:37This is a couple different ways to change the shot type. So if it's not 360 degree orbiting, all I'm changing here is this camera motion. It's the same prompt every single time. So really I have, if we're looking here, we have a sweeping camera motion. We have a descending camera motion. We have a zoom out where the camera's zooming way out. We have a panning shot where the camera's moving from left to right. We have a zoom in where it's zooming in. So really that camera motion is the one variable here that can change the entire look and feel, right? Because they all tell a different story. Now those of you who are listening and not watching, we have six different examples of video clips on the screen that are showing really spectacular images of somebody at the top of the mountain, but each and every one of them are very different.

38:24And you use them either in different sections or you sequence them to like the zoom in and then the close up with the face. And then you can actually use these different camera shots to tell the continuation of the story as the camera comes in from the sky, turns around the person, zooms in on his face and all of that while he's standing on the top of the mountain with the scenery around him and the clouds and the sunrise and so on, you can use them either as one-offs or as I mentioned, combine them together. Yeah, and that's why I look when you have a structure, what you're trying to fill in the blank for is camera motion and you have subject and the action is man standing on top of a mountain.

39:06Then it's just, okay, this is now how we control this little piece. So very simple, but you can see it, the varying effects by one, essentially one or two words in a prompt. So that's why it's important to work off of structure. And then honestly, to have what you said before, Isara, is just the end result of what you're trying to look for in the little bit of vision. Try and get there. Not to say that you have to have one, but sometimes it's just easier to fill in that blank. Now, if we go a little bit further, right, like I want to control more of the variables here, right? So going back to our little variables that we broke down before at the end of a video, we have our shot type or camera movement.

39:43then we'll have our subject in action again who's in the scene what are they doing we'll have our lighting description which i think is can be very important in terms of telling a story the difference between really bright sunlight and a really moody dark light it just gives you a different feeling so lighting really shapes emotion the prompt here the example is we have close-up shot of a man wearing sunglasses walking in the tokyo streets strobe lighting and then And rapid flashes creating a staccato effect. So the reason why we put this little lighting description here is because sometimes strobe lighting might be too general and you need to explain it.

40:22This is what I'm talking about being direct. Rapid flashes create a staccato effect. That's going to tell it, oh, do this quick flashing motion. So again, I put a little compilation together showing you like how these different lighting effects look. So again, if we're looking at this, like the staccato effect, or sorry, like we're just looking at for different lighting effects, they all affect this differently. So even the difference between warm light and cool light, warm is going to be more orangey, cool is going to be more blue, volumetric is going to be more hazy. There's a lot of different ways to just change this one little factor and give it way more life.

41:01So again, you can look at with lighting, you can go into different categories of lighting, It could be time of day. Each one tells a different story, whether that's twilight, midday, midnight. They all give you a different sort of look and feel. But again, there's just a way to, again, isolate these little variables and then show where the output is going to go with them. But you can go really far into the descriptive side of things. Now, this is when you want real control. If you have a real vision, just to show you that something like this is possible, right? Like I'm going to use a really grand prompt structure to what I call that.

41:38I'm asking a lot of it. So basically the prompt here is a cinematic behind the back shot of a character and then we're going to have their movement away from us to an environment plus or environment in details where they are. It'll do the lighting and the color scheme right now. This is a larger prompt structure. There's a lot of things going on here. So I'm asking a lot of this. But as you can see, when we rip through this, we have a cinematic behind the back shot. So we're getting it from behind the back. It's following him. So it's as he's going somewhere, right? Following him, meaning he's walking away from the camera, right?

42:15And it's of the Pope. We have our subject heading towards the balcony. So we have him heading towards a balcony. He's walking away from us again, directing exactly what he's doing through a doorway. So we have him going through a doorway. You'll see here overlooking a massive crowd. So he's overlooking this giant epic crowd, if you can't see this. And there's this high contrast lighting. There's a big difference between light elements and dark elements, as you can see as he's walking through the doorway. And then we have this contrasting red and white color. So I asked it to do all these things, and it did this.

42:46That's not a simple prompt by any means. But you can see as you ask it more, if you're directing it correctly, it can do it. So it's not... Oftentimes, I think a lot of people blame the tools. The tools are capable. It's just how you're instructing them. Same way with ChatGPT. Same thing with any of the LLMs, any of the image generators. So again, it's a little bit different. But the other thing now that we can do with this is use image references. So that was just straight text promptly. That's me putting a text into the image generator, pressing enter, and then the video generates. Now I'm going to guide it with an image.

43:20So we have a starting point. The image references to me are great for consistency. there might like you can't the one thing with things like runway luma cling is if i wrote that text prompt of the pope that we just saw and then i did it again he's going to look different in every single video he's not going to look exactly the same with an image reference now i can use consistent characters consistent settings consistent styles and it'll be very much so similar so you're thinking branding you're thinking that look and feel so they have the The images play a huge influence in guiding the tools.

43:55And the one thing I will say about this is it is less controllable with the majority of them. Because you're giving the runway or Luma 50 % of the work. It already generated the image. Now it has to take that image and then go animate it. So the base image, as Isar said before, plays a huge role in all of this. So if it's a really crappy image, to be honest, it's just going to be a crappy output. But what I mean by that is if it's bad resolution, if it's grainy or fuzzy, don't expect to have really great output from runway then. So the base image plays a big role in all of this. And one tip, I don't know if I put this in here or not.

44:34We'll see. If you want to get better motion from runway, if you're generating these images ahead of time, like in Mid Journey or Dali or Leonardo, whatever. Sometimes you bake motion into the images themselves beforehand. You're going to see a lot more good video generations because you're giving it basically a head start. You're giving runway a head start into generating. So this car, for example, when it comes back around, you'll see that there's already some motion blur attached into the image. So it looks like the car is already driving. Now that's going to tell it basically what direction to drive.

45:11I don't have to prompt it as much. So you can see here the motion blur. The wheels are spinning, which means it gives it also a good starting point of, okay, the wheels are spinning on the car, meaning it's driving. Sometimes when you prompt something like a car, this is just a really random example here. This is a big problem that I see for a lot of people. They have an image of a car and they want to animate it driving. If the image of the rims are static, meaning you can see all the spokes, you can see the tires super clear, it's just going to push the car across the frame. It doesn't look like it's driving.

45:40It looks like it's hovering with no tire motion. So sometimes prompting spinning wheels onto your car in the image generation process can then help the video generation process because it's already doing what you want it to do. So again, it's like problem solving in multiple ways. Really long sidetrack there, but just wanted to make sure that was a hit on that for a little bit. Now, when you're prompting with images, again, I would say keep the prompts even shorter than they were when you're just text prompting. because there's already the characters, the setting that's already in there. So really, you want to focus on the motion.

46:21I think that to me is the most important is to focus on the motion, what you want it to do, how you want it to move, because then that'll help the outcome. Now, with this example, I'm going to show you this long prompt here. Again, we have tracking shot, tracking aerial shot of a white pickup truck. As you can see, the image is already that, but I want it to look like it's tracking the car and I want to show the car driving, but then I also want to show something that doesn't exist, which is the environment, right? So we have this tracking aerial shot of a white pickup truck. It's driving away from us.

46:52So it knows to push the car forward on a muddy road. So we'll get that little like detail in. So we get some mud splashes because it would look pretty weird if this car was driving through this muddy, wet road without any mud splashes or anything happening there. Dynamic motion is just something I add every once in a while to maybe trigger. It's like a little hack. It works with a lot of prompts. Muted color palette just to keep the consistency because I know I'm going to essentially show a part of the environment that is not available in the picture. So I'm trying to tell runway what to do there.

47:23And then basically, we have this second point, which is revealing a vast Wyoming landscape. So when you look at this, we have our tracking aerial shot, right? We have the white pickup truck. It's driving away from us. The mud splashes over here. We have our dynamic motion. The muted color palette makes sure that when we pan up right here, this stays muted and not turns into this green sort of lush environment, which it's not, obviously. And then we have revealing a vast Wyoming landscape. So again, we're directing every piece of this from this one image. Might be an extreme example. I'm just trying to show everyone what is possible when you direct it correctly.

48:00Now, the other thing that we can do here is what's very interesting. All of the images, all the videos you just saw were first frame generation, meaning you put the image in and then it'll animate everything that happens after that frame. What Runway can do also is you can basically have it start with the last or have the last frame as a generation point, meaning you put that image in everything before that image will generate. So you can see what's going on here in these videos. If you're not watching, I'm sorry. it's basically we have these images of food and the food is super close up in the camera and when you run this again for those who's not watching it's like a very detailed sandwich or burger that now makes me hungry because it's lunchtime and I didn't eat yet but very amazing like view of a burger and somebody's holding it between his hands like showing it to the camera on an extreme close-up that's the original image and then the video outcome is the person like bringing it closer to the camera or the camera zooming in into the person that's holding it and so on.

49:06Yeah. So it's for this, right? The reason I like to use last frame for something that's maybe a close up because it'll keep all that detail from the image. If you have a really good detailed, crisp, sharp image, using it as the last frame sometimes makes that way more prevalent than it doesn't give runway the opportunity to really go and morph it or take it in a different direction. So it's a great process. Now, it becomes a little bit more of like a mind game when you're using the last frame because you have to think about, okay, I want it to end here. How can I make that shot look a little bit more?

49:45What do I want to happen before it so I can prompt to the end? So it's a little bit crazy to think about it in that term. It's reverse of what you normally would think. So it's just I like to use this for certain things like brands. If I'm using a product that has text or logos or things like that or specific color schemes, I like to use the last frame because I know the details are going to be held in place because they have to be because that's the last frame. It's essentially where it's going to end. So that to me is the one thing for branding or products or marketing purposes like that, which I would say is super useful.

50:21So you look at things like this, you can utilize ENGs, you can utilize static images, you can, like I said, it works best with the last frame. This top image right here is definitely just a PNG. I didn't create these. I just pulled these as examples. So I wanted something with a PNG. So if you're not familiar with what a PNG file is, basically, if you have any sort of product image with a transparent background, there's nothing else. It's just like the soda can, or it's just like the lipstick or whatnot. You can plug that in and generate the environment around it. So you can see last frame here.

50:57They're super helpful because it keeps the details of Prime and you see the logo that says Prime for anyone who's listening. First of all, I think this is absolutely brilliant and it's amazing. And I think that's where it becomes really cool when it comes to specific products is the fact that you can use the actual product. So it's impossible right now to recreate an actual exact product in MidJourney or in Flux or in any of these tools. You cannot create the exact image of something. But in this particular case, you can use the actual image of the product that you have in the video. So that's the way to do this, is to use it as the last frame and then build whatever you want around it and build whatever graphics and motion and so on.

51:43And it's perfect for, again, what Rory said in the beginning, social media stuff. Short, punchy, to the point, very cool. You can create whatever environment you want around it and a short video that will be, again, perfect for social media. The other thing that I want to add is how do you go the next step? And the next step is old school video editing, right? You can create four, five, six, 20 shots with these tools and then use any editing tool of your choice. But if you're a beginner, you can use Canva and just stitch them together into a longer video. So if you have the story in your head, and if you don't, use Chachupiti or Claude or Gemini to help you craft the story of the idea that you have.

52:26I want to promote this to this audience. I want to end up with a close-up of that. This is the feel that I want to get. This is the audience that I'm trying to target and then have it and then create the images in mid-journey or any of the other tools, bring them here, create the short videos, pick the ones you like that connect to a logical story, and then do the cuts between them in any basic traditional editing tool So you can go beyond the three, five, 10 seconds videos to a 20, 30 second video that makes sense because there's a story and you go from one frame to the other. Over there, going back to what Rory said, starting with images as references that use consistent reference of the background, the image, the colors in the original images makes a very big difference.

53:10Otherwise, it will look weird because the transition is not going to be normal between the images. but that's like the next evolution that you can go after all the incredible, amazing stuff that Rory showed us. Rory, this was magic. Like people haven't seen this before or maybe have seen it, but never knew how to do this. You really shared with us like the whole process from your thought process to the tools, to the prompts, to like everything people need to get started. If people want to learn more from you, follow you, work with you, et cetera, what are the best ways to do that? LinkedIn's a great spot.

53:43I'm normally there just sharing what I'm doing. I think it's the best way for everyone to pick up on this stuff and go. So yeah, there's that. If you have any questions, please feel free to hit me in the LinkedIn DMs. I'm a little bit slow on there, just candidly, because there's just a lot of messages. But I will definitely try to get back as much as I possibly can. But I'm glad this was helpful. Like I said, first time going through all of it in sequence. So I know there's a couple of rough spots I need to clean up. But it was good to lay this all out. No, this was fantastic. And people in the chat saying this was amazing.

54:17And I think it was amazing as well. Yeah, follow Rory on LinkedIn. Like he really shares everything. Like what he's done today, every time he figures something out, he puts out these amazing guides together and he just shares them with the world. So I really appreciate you. I appreciate everything you're doing for AI, the community and just helping people out. And like I said, follow Rory. You can learn a lot if this is what you're trying to learn, how to use imagery and videos from AI. And I really thank everybody that joined us. We had a pretty big crowd on both LinkedIn and definitely on the Zoom call today.

54:49So I appreciate all of you taking an hour out of your day to come and join us and geek out with us on AI and how to do things. And again, thank you, Rory, so much. Thank you for having me. Have a great day, everyone.