Using Veo 3 to create AI-generated music videos, like a Tiny Desk Concert with Notorious B.I.G. and Kurt Cobain

0:00Anish Acharya:It's like the most creative satisfaction I've had in my whole life. So I generated all these clips in a pretty straightforward way. I used GPT-40 to help me with the prompts. I said, hey, help me capture grunge 1990s Seattle inspired by some of these music videos. And then as you can see, it gets progressively more like camcorder, grimy. So I generated all this stuff and then I threw it together into a music video. All right, let's watch it.

0:28you get the patented clairvaux raised hands reaction on this one i cannot believe this is ai generated it's so high quality it's so specific in an aesthetic in a wardrobe an emotion you have inspired me after this podcast what music video am i going to make it's so much fun

0:50Welcome back to How I AI. I'm Clara Vo, product leader and AI obsessive here on a mission to help you build better with these new tools. Today, we have a fun and inspiring episode with Anish Atraya, general partner at Andreessen Horowitz and AI consumer investor. But we're not going to talk about portfolio companies or the future of AI. No, we're going to use AI to build music videos, analyze our bookshelf, and help us plan our personal finances. Let's get to it. To celebrate 25 ,000 YouTube followers on How I AI, we're doing a giveaway. You can win a free year to my favorite AI products, including VZero, Replit, Lovable, Bolt, Cursor, and of course, ChatPRD by leaving a rating and review on your favorite podcast app and subscribing to YouTube.

1:42To enter, simply go to howiaipod.com slash giveaway, read the rules and leave us a review and subscribe. Enter by the end of August and we will announce our winners in September. Thanks for listening. This episode is brought to you by Notion. Notion is now your do everything AI tool for work. With new AI meeting notes, enterprise search, and research mode, everyone on your team gets a note taker, researcher, doc drafter, brainstormer. Your new AI team is here, right where your team already works. I've been a longtime Notion user and have been using the new Notion AI features for the last few weeks.

2:27I can't imagine working without them. AI meeting notes are a game changer. The summaries are accurate and extracting action items is super useful. for standups, team meetings, one-on-ones, customer interviews, and yes, podcast prep. Notion's AI meeting notes are now an essential part of my team's workflow. The fastest growing companies like OpenAI, Ramp, Vercel, and Cursor all use Notion to get more done. Try all of Notion's new AI features for free by signing up with your work email at notion.com slash howiai. Anish, I am so excited to have you here. And let me tell you why. It is because I have spent the majority of this podcast talking about enterprise B2B product management, how to manage your manager or manage yourself as a manager, or how to vibe code.

3:22That has been the topic of how I AI. And today we are just going to have a little bit more fun. So why did you start to come to these AI projects that are a little less like work related or technical and actually just a little bit more fun. How did you get here?

3:41Anish Acharya:Great. Well, I'm excited to have some fun today. I mean, I've been passionate about music forever. I think most of us are. I've been DJing and making music for 30 years, but music is very constrained. There's only so many ways you can work with it. An example of that is if you look at a track that has all the instruments mixed down into a final mp3 or wave file there's no way to just extract the vocal or just extract the drums so you're really limited by a set of choices that were made in the studio and with ai you can do all this crazy stuff like disentangle a track into just the vocals and just the instrumentation so what really got me excited at first was everything you could do with ai and audio and then that of course fed into all of the new video models and video gen and lip sync and all the new technologies we're seeing.

4:26Anish Acharya:So it's just, it's like the most creative satisfaction I've had in maybe my whole life. Yeah, I agree with you. One of the things that I have so much fun with AI on is people are really worried that it takes away the most fun, most human, most creative parts of not just building things, but creating music, creating art, creating writing. And I, in fact, feel like it just gives me so much more tools, so much more breadth, so many more things I can play with and build. And And so it really opens up this like creative artist side of me in a way that has been really hard to access as an adult, also with limited time.

5:02Anish Acharya:Yeah, no. And it's actually a fun conversation we'll have over a glass of wine sometime. But if you look at music culture, music culture has kind of been defined by remix culture for the last 40 years. You know, like the mixtape was the first time that you could take the music and do something, you know, the cassette tape and do something of your own with it. And then that, of course, evolved into, you know, hip hop, which also sampled and which also had a lot of suspicion on it. But sampling was the foundation of hip hop. And I think AI is just the next manifestation of sampling and it'll be as important for music as hip hop was.

5:31Well, and we'll stop opining about AI and the arts. But the other thing that this remix culture makes me think about is kind of the next step that we've seen in the past couple of years, which is kind of audio and video remixing this like TikTok memes, these dances, these things where you're taking a snippet of creativity, turning it into your own thing and then releasing it to the world in a new version. So I definitely think we're seeing this not just the audio side, but also at the video side, which brings us to your use case. So tell me what you built or what you created, maybe. And I'm excited to walk through how you got it done.

6:09Anish Acharya:Amazing. Amazing. Great. Tiny Desk is the best. So if you haven't gotten into Tiny Desk, most people have seen it. It's just it's so cool. It's so fun. And of course, you know, like creativity loves constraints and the constraints of Tiny Desk are incredible. There's a really good one from Clips that just dropped last week. And I mean, anyway, there's an infinite number of them. It's a fun format. It's sort of like the unplugged format of the 90s. So I love Tiny Desk and I got to thinking about all the artists I'd want to see on Tiny Desk. And, you know, of course, some of them are no longer able to be on Tiny Desk because they're not alive anymore.

6:45Anish Acharya:So that got me thinking about how I could do a Notorious B.I.G. Christopher Wallace Tiny Desk. And do we have the tools and technologies? And of course, can we do it in a way that's respectful and not derivative? And I did it and it seemed like it kind of worked. Maybe we can cut to it so your audience can check it out. And the workflow is pretty simple. We'll do a little clip of it, I think, and then we can work through how it got there.

7:22Back to the club, sipping my wet is where your mom was. Back to the club, making holes, my crew's behind. A mad question asking, lung passing, music blasting. But I just can't quit because one of these... Okay, we love it. It's great. And you made that.

7:38Anish Acharya:I did make it, yes. And it took surprisingly little time. Yeah, so let me show you exactly how I made it. So I started with 4.0. 4.0 is the best general purpose multimodal model, in my opinion. I use it for everything. and I just ask it to generate an image. We're going to do Kurt Cobain. That'll be fun today from Nirvana, of course. That's from when I was in high school playing a Tiny Desk concert. So let's see what it comes up with. While this is loading, you mentioned that 4.0 is the best kind of multimodal, all-purpose model. I generally agree. You know, 4.0 ImageGen had this super viral moment a couple months ago when they released it.

8:18What do you feel like 4.0 ImageGen is particularly good at compared to some of the other image gen models.

8:26Anish Acharya:It's very good at prompt adherence. So you can do things, and I think that's because of the infrastructure underneath it. It's a different infrastructure from the diffusion-based models that preceded it. And BFL, Flux, a bunch of others do this now as well, and it's great. But I think it was just the most productive image model because you could manipulate it in such a fine-grained way. Yep. And I remember the biggest improvement when the 4.0 Image Gen came out is that it could actually spell things and write letters out. That was a magical moment. So I have to call out that NPR in the top corner of this image is actually done correctly.

9:00Look, there he is with his cardigan.

9:04Anish Acharya:Okay, I'm going to remove the guitar, actually, so that it is acapella, because I think that might work a little bit better. But look, this is the vibe of Tiny Desk. You know, it's as if you're seeing a photo from the 90s in the Tiny Desk studio. So I just I love this. And I think that we become so attuned to what's possible. We forget that this would be, you know, witchcraft three years ago. Witchcraft. Right. What is the purpose of this? Are you storyboarding? Are you creating an asset that's going to go into another tool? Why start with it on this flow? So so I'll talk through essentially what I'm going to do.

9:37Anish Acharya:So there's this product called Hydra, which is the best way to, I think the best way to take a still frame and add custom audio to it. So create a video that has sort of animated from the still frame and includes the audio with the right lip sync. So, and there's a bunch of amazing tools to do this. Sync Labs is one of my absolute favorites as well. But Hydra is nice because it actually generates the video. So it does the frame to video, and then it also adds the audio. So what we're going to essentially do is take this frame. We're going to get the audio from YouTube. We're going to stem separate the audio so we get the audio track we want.

10:17Anish Acharya:And then we're going to put them together in Hydra. And that's it. This really is remix culture. It's amazing, isn't it? It is amazing. Okay, so the asset that you really need to go into this VideoGen lip sync tool are two things. You need a still image that can be used to generate the video. And then you need some sort of audio to sync this to. So I know we're looking at this music example, but what other examples have you seen people use this kind of workflow for? I think we underestimated how useful it would be to add custom audio to video. And there's been a bunch of great, you know, one of the early examples was taking a speech that somebody was giving.

10:57Anish Acharya:I know Javier Millet did a really famous one. And essentially lip syncing, changing the language to English and lip syncing it. That went really viral a couple of years ago. So we've seen. And then, of course, you can imagine a character, a photo of a character that you generate. And then you want to animate them doing something and speaking at the same time. So, you know, stories are told this way. And these technologies make it really, really easy to do so. Oh, we got him. Great. Okay. So now he's got bad posture, but we'll allow it. It's very grudges about it. I think he always did. Yeah. Exactly.

11:30Anish Acharya:Okay. So now we've got Kurt. Now, what I would do if I didn't actually have a... So Tiny Desk has got a really specific acoustic aesthetic, which is it sounds like live instrumentation. So for the Biggie example, I actually found a Biggie cover band playing live in Brooklyn. And I pulled that down from YouTube. And then I extracted the actual vocals from the Notorious B.I.G. and laid them over. But in this case, Nirvana did a really famous New York City Unplugged concert in 93. So there's video of them playing in the way that they would and audio in the way that they would on Tiny Desk. So that is right here.

12:10Even in the same cardigan.

12:12Anish Acharya:Even in the same cardigan. Isn't that amazing? Yep. Okay. So I use this nifty little tool called 4K Video Downloader, which is slightly sketchy, but that's okay. I love these little utilities that you just, you know, you Google like, how do I get audio out of YouTube? And then you look at the scariest website possible. And you just cross your fingers that your computer won't go up in flames and you download 4K video. Yes, my yes, my data is definitely going somewhere sketchy as a result of this. So for the vibe coders that are listening, I have a request for startup, which is go find all these slightly scary little utils and build me ones that are less sketchy looking.

12:56Anish Acharya:100%. 100%. It's a great idea. Okay. So now we actually have this. So we've got the video. Yep. Now we're going to open Adobe Edition. Okay. So this is a tool that people who have been working in computer audio have been using for 30 years plus. It used to be called CoolEdit Pro. It's completely beloved. and it's very, very easy to use, which is why so many of us use it. It was, of course, acquired by Adobe many years ago. It's now called Audition. So I go to Audition, and I take this video, and I just drop it in. So here we actually have the audio from the video, which is really, really cool.

13:33Anish Acharya:I'm going to zoom in, and I'm going to see the first few seconds of it are blank. So let's just cut that out because we don't want to hear that. then we're going to zoom out and we're going to take, I don't know, let's take 15 seconds. And you can kind of see the audio, the video in the bottom left corner there. Oh, got it. So it's combining the audio and video just so you know exactly what you're syncing up to. Exactly. And I'm going to pretend like you're doing 15 seconds because we're doing a very efficient podcast here. But one of the limitations I know, having used some of these audio and video gen tools, is you're getting small clips right now with what we're working with.

14:13And so, you know, what I'm looking forward to is the day where I can have the, you know, hour-long Nirvana unplugged tiny desk. Totally. But, you know, do you ever feel constrained by the kind of length of assets being generated or the quality?

14:30Anish Acharya:I mean, sort of. But again, I think creativity breeds constraints. So not to over-rotate on hip-hop, but if you look at the reason that so many samples were used in hip-hop in creative ways in the 80s and 90s was the actual drum machines and samplers had very limited sampling time. So you could only sample a second of anything, so you couldn't really sample four bars. And that's why so many producers put tracks together that use these many one-second samples in surprising ways. And once we actually got the technology to sample for more time, we actually got less creativity, I would argue. So I sort of love the constraints that the technology gives us today.

15:08Well, I also love my complaints. I'm like, isn't it annoying that you can't provide Nirvana and overlay their audio and generate a completely fictional concert for longer than 15 seconds in probably under a 30 minute podcast? My complaints are so ridiculous because the idea of creating something like this even a year ago sounds so, as you said, impossible that we get so spoiled once we get used to these tools.

15:36Anish Acharya:100 % right. No, exactly. I mean, this stuff, we would have called it witchcraft three years ago. It would have been. Okay, now there's two things you can do with this. If we wanted to do an acapella-only version, for example, we can use a technology called DMUX. So DMUX is this amazing technology that allows you to extract the vocals from any song. So here, I've forgotten what the actual command line is. So I just do this. I looked it up in perplexity. What's the actual way to extract two tracks with DMUX? We do this, DMUX two stems vocals, and then let's go find the path. Okay, so this command is going to take that audio file we saved of the first 15 seconds of this concert, and it's going to extract the vocals from the instrumentation.

16:28Anish Acharya:So this will be Kurt Cobain singing come as you are acapella which as far as I know has never happened which is pretty cool and then we simply come back here and we say start frame upload an image let's use this okay that's our Kurt Cobain audio script upload audio and let's use actually the full audio with all the instruments add a video and then we just say man singing on tiny desk what i love about your prompting compared to other how i ai guests is every prompt has been sub six words six words you're very simple in terms of describing what you want and uh get high quality outputs there so i don't know what that says about the prompt engineering industrial real complex.

17:20But prove here that you can use simple prompts to get pretty cool stuff if the tool behind the scenes does the work for you.

17:27Anish Acharya:I think you've got to give the AI the space as well. You know, if you overly constrain it, it just really struggles to satisfy you. Whereas if you give it less constraints, you know, sometimes it has unexpected results, but often they're unexpected, you know, delightful. Well, that's what I've heard a lot from folks that come from the more creative backgrounds. Designers in particular tend to be less precise in their prompting because they want that exploration space that then they can narrow in on. And so I really think it also comes into play, your prompting technique can come into play based on kind of what profession or what background you're coming from.

18:04Engineers want like the most precise. They not only want the code to work, but they want the code to be written exactly how they would write it. And so they're very precise in their prompting. Where I found designers and more creative folks building different kinds of assets really like that wide open space.

18:20Anish Acharya:Totally. Yes, exactly. And while we're waiting for this to load, it might be interesting. I'm just looking at some of the options at the bottom here. So you have different kind of models that you can use, including one that looks like that they specifically fine-tuned for this, different aspect ratio, orientation length probably based on the script and then you know the prompt says prompt your character with emotion and gesture so i am very curious if you put like angsty man singing versus cheerful man singing if you'd get a different a different version here even if the audio video were were the same it works really well absolutely yeah no this this is such a useful storytelling product it's it's amazing and when you combine it with other video gen models like vo3 you can start to tell real stories, you know?

19:11Anish Acharya:Yeah. Okay, let's check it out. All right.

19:24All right.

19:25Anish Acharya:Pretty cool. It's very good. It's very good. Very satisfying. He even manages his mic well, you know, pulls back on some of those notes. Totally. That's incredible. And so, you know, could you take this and take different clips of the video and sort of generate a string of these these videos and maybe put them together in a longer form version? A hundred percent. Yeah, I actually was inspired by this. So I put together a music video, a little mini music video for a different Nirvana track. Can I show it to you right now? Yes, we would love to see it. Okay. I used VO3 to generate the clips and And it turned out great, I think.

20:09Anish Acharya:Hold on one moment. Yeah, and I think if you haven't tried VO3, it is pretty incredible. I mean, I can only generate like two and a half videos every day of three, you know, seven second length or whatever. I'm still capped on usage, but the quality is really good. The physics are really good. It's one of my favorite video models to play with right now. Just as a consumer, it's kind of, to me, my experience with that model was very similar to my first experience with Mid Journey, where just the breadth of things coming out of the model were so incredible to me. So highly recommend folks give that model a little spin.

20:54Anish Acharya:It's amazing. Yeah. You've got to get on Gemini Ultra, Claire. We have a household Gemini Ultra. account. But my husband is the video gen guy. So he's up there. And by the time I get to it, we burn through some tokens. But I spent all the money on Cursor. Fair. I know. My wife, for the first time this month, was like, babe, what is Cursor? I'm like, ugh, don't worry about it. I know all these little secret AI tools popping up on the credit card. How I AI is now on Lenny's list with my personal selection of the best AI engineering courses on Maven. You can spend months thinking and playing with AI before really integrating it into your workflow or shipping an actual AI feature.