“Nobody wanted to do this work”: How Emmy Award–winning filmmakers use AI to automate the tedious parts of documentaries

19:52Like generative metadata that is a, we know it's accurate because it's kind of been put on these guardrails by our metadata extraction steps. And then also it provides this like nice visual for us. We can see what this thing is at a glance. but the next step of this now that you have this like api running in the background is you can generate something that maybe i can't read but the ai can read pretty well which is vector embeddings so i'll jump back to stills for this because i think it's a maybe an easier illustration of it every asset in our database gets put through two modes of embedding so we'll send the thumbnail through and run it against an open source model.

20:34I use clip for this. And I'll generate an embedding off of that. And then we'll send the description through. I use, again, an open AI text model for this and get an embedding for that. And then we'll fuse them. And the purposes of that is that so now we have like the ability to discover things semantically, like prior to this, and I think in a lot of film production today, you're working with exact text search, you know, like if that description says dog, but you know, somebody wrote in puppy, you're not finding that image. And so this has been like, kind of the most exciting part of it, not necessarily where I knew it was going when it started.

21:11Like I was just excited to generate a description, right. But now the ability to discover semantically is I think, you know, the most the most robust part of the system.

21:21Tim McAleer:So what I love about this, I mean, a couple things is one, you've really pushed every step of the way you know you could have stopped at like we got good descriptions or we got like the structure metadata out and now i have a script that runs it you could have stopped at images only but you took it to video and video and audio you could have stopped at structured data only but you went to embeddings to get semantic search so i love just the breadth of applicability of the ai in this process but what i probably love more is i doubt this was anybody's favorite part of their job. Like, I doubt it was anybody's favorite part of their job to be like, I'm going to go read some Library of Congress metadata.

22:01It used to be my job. So I can tell you firsthand, not my favorite part. And it's also like, I think the best argument I have for all the work I've done creating this system is that like the same people who used to write this data were the ones who are responsible for doing the research. So you've now freed them up to just look more, right? Like maybe now we could gather 25 ,000 still images for the Muhammad Ali project because you have that much more time. You're not just like copy and pasting stuff off a website to put it in this form, you know?

22:30Tim McAleer:Well, and you probably get to select from this big archive of data, better assets to use in your content because they're more discoverable because you have more confidence in the source and the content of that data. So I bet it up levels at the end of the day, the quality at at the end because you have just much more data to work off of. 100%. I mean, like a real quick example of that too is like, I'm going to use Abe Lincoln here, which is maybe not the best use of this image. But embeddings enable us to find things in ways we never would have thought to find them before. So like I have a button down here where when I click it, what it basically is going to do is reverse image search within our own collection.

23:10So if I'm an editor and I like an image, and this is going to take a while because I'm not on site. But if I like an image, I can click the find similar button and it's just going to go and find every image that kind of has that vibe. You can see here we have a duplicate of this one. But then there you go. It recognized the man and it started pulling in other portraits.

23:30Tim McAleer:This episode is brought to you by Brex. If you're listening to the show, you already know AI is changing how we work in real, practical ways. Brex is bringing that same power to finance. brex is the intelligent finance platform built for founders with autonomous agents running in the background your finance stack basically runs itself cards are issues expenses are filed and fraud is stopped in real time without you having to think about it add brex's banking solution with a high yield treasury account and you've got a system that helps you spend smarter move faster and scale with confidence. One in three startups in the US already runs on Brex.

24:15Tim McAleer:You can too at brex.com slash how I AI. I love this. Okay, so this is more of your archival and footage data, but you capture a lot of stuff in the field where people are not sitting in front of cursor or their desktop. I'm looking through these assets and I know that you use some vibe coding and a creative approach to get more information about those assets. Could you walk us through that? Yeah. So the next use case is an app that I developed for archival research in the field. So I think that we really pride ourselves on like turning over every rock on not just relying on what's digitized and available online and going and visiting physical archives.

25:01And so the process of visiting a physical archive is basically you have a bunch of folders that you pull ahead of time you arrive there and your goal is just to snap like low res resolution iphone snaps of everything you can possibly get and so you're snapping the front of the image and you're snapping the back of the image because the back is typically where there's going to be like a scrolled description or maybe like an accession number an id number that the archive is out of themselves and so this process used to look like you show up at the archive you take iphone snaps for two days you get back to the office you have the messiest camera roll you've ever had you cannot actually pair your fronts to your backs because it just got out somehow it got out of order along the way and so the goal was basically to make that process like a little better so i i vibe coded this ios app to deal with this problem.

25:56And I, I tend to just like speak in screens, like the way I maybe it's because I'm a visual person, like the way I deal with it is I just think like, okay, I see a screen that does this and a screen that does this, I imagine a button that does this. And the purpose of this was basically like, I want people to be able to create collections for each folder they're capturing, I want them to be able to snap a front and a back, like the flip side of the image, so that they can easily associate those, so the file names associate them. And I want to immediately transcribe any information on the back and embed it into the original image.

26:29So now I have this app called Flip Flop. I ask ChatGPT at the end of my dog walk to generate some kind of specs doc or requirement doc. It pretty much does it in one go. If you chat with it for 30 minutes, you know, you can get a lot done. And then I fed this PRD to Claude Code. and it this one it like it it didn't build it in one shot but it certainly built the ui in one shot and so i guess maybe we should just jump into like the actual app yeah let's do it so flip flop which is my cute little name for it is uh basically designed to capture those fronts and backs that i was talking about so you have three screens here you've got a collection screen where you're going to create your folders you've got a capture screen where you're going to take your images.

27:14And I'll just quickly highlight this part, which is where you kind of have your AI processing options. So I allow people to define a separate prompt for what I call the flip side of the image, the front and the flop side of the image, the back. And so in this example, I'm going to show you some photos of my dog now. And the flop side of the image is going to have some text on it. So our prompts here are really just designed to get a decent caption from the image and to transcribe any text that we see on the back end.

27:43Tim McAleer:So let's create a new collection. We're going to call it how I, AI, that's good enough. There's also an option here to add more context. You know, the AI loves context. And so maybe if you're, you know, you could imagine if you're digitizing an entire collection of, you know, someone's personal letters or someone's portrait photographs, you would add that kind of thing here. But for now, we're just going to create a collection, tap into that collection and capture. So here we go. It's a screen share within a screen share. We're going to not care about the glare too much. I'm going to capture the front side of this image of my dog Tony's third birthday.

28:24I now have the option to add notes if that's what I want to do, or I could just add a flop side of the image right here. And when I complete that, it will have, because it's lightning fast already, sent it up to OpenAI for a description and embedded it. And this is the really crucial thing because you just saw the first system I had, embedded it in the image metadata itself. So the flop details have the transcription, Tony's third birthday, and all of that will show up in the, what we call EXIF metadata, which is just the image metadata standard.

28:59Tim McAleer:Got it. And just for people that may be passed by, instead of simply generating kind of the text description and storing that in a database, relative to the original image you took, you actually now have this structured metadata on the image file itself, which again, like what a pain to do. Oh, a giant pain, yeah. A pain to do manually. And so now anytime anybody uses one of these images, even if they don't have access to this app, even now that image is embedded with that metadata. 100%. So you could pull this onto any computer or any app, anything that can read underlying metadata, and it's going to be able to see that this was Tony's third birthday.

29:41And so that's structured metadata in the sense that we've now structured the actual information about the image. But the other thing that's really crucial, honestly, is that we've structured the files themselves, right? So you can see they're getting named in a particular way. And so we've moved from like camera roll mess to like files that are going to sort in your computer that you're going to be able to import cleanly. You're going to be able to distinguish easily what's the front of the image, what's the back of the image. And that has, I think, been the other unlock. Like I had two colleagues out in the field a couple weeks ago, and they came back with 1400 images.

30:16And I don't think that's only because they were able to use Flipflop to capture it. But I think Flipflop is certainly making the process easier since they've gotten back.

30:25Tim McAleer:The thing that I want to call out for folks, maybe a general takeaway here is these AI models are so good with files and code can do a lot of stuff with files. And a lot of the people we talk to, you know, Markdown is the file type du jour these days, which is, you know, just like a specially formatted text document. But if you start to look at other file types and really understand what can be put in a particular file type, you can actually discover some pretty interesting things you can do with a combination of AI and coding to make those files much more useful for your use case. So this is one of these takeaways where I'm like, I haven't thought about like what can be embedded in an image file or what can be embedded in a video file.

31:17Tim McAleer:And even just having, you know, ChatGPT or one of your general models say, hey, I'm working with an image. How can I load it up with as much context and specificity as possible? What's available to me. And then using that as a jumping off point for what you do is a pretty interesting use case of AI. I didn't even know, like, I'm very familiar with stills underlying metadata fields, but I didn't really know what was available in audio or what was available in video files. And I just sort of, I go into cursor and I ask, right? Like, now we have a music workflow, which we're not going to look at, but like where we embed artist album kind of like licensing data into any music we consider for a film.

31:56And I didn't know that there was in the metadata field we could just store that in. But of course there is, you know, somebody thought of this a long time ago.

32:03Tim McAleer:Yep. Amazing. Okay, we have one last use case, which, mom, if you're listening, I think you're gonna like this one. My mom's a genealogist. So I think she's gonna like this, this use case. But let's show it first. And then I'll call out mama where I think you can use it. Okay. All right. So you can imagine in our films, we work with a lot of documents. And we're not always interested in the entire document. Sometimes, like, we just want to transcribe maybe part of it. Maybe we want to translate and transcribe part of it. Like, take this newspaper document, for instance. Like, maybe the Arkansas State News is the article we're interested in.

32:43That's the transcript we want to be searchable. That's what our editor might want to consider for the film. We can't just, like, put this in Adobe Acrobat and OCR the whole thing. It's, like, it's not going to work. and even more than that like the quality of the image would not work with most OCR engines you know so AI is really good at OCR of old documents it's really good at handwriting it's pretty good at translation too so I built and we're not going to get into the building necessarily but this is this is one of the few like xcode builds I had to do so this is a swift build a little mac menu bar app.

33:18It's called OCR Party, which stems from the fact that we're just OCRing part of the image. You got to have fun with these things. And let's see, we're going to open up that newspaper in OCR Party. We're going to get like a little preview window. So let's say actually what we want is Coolidge seeks peace in the world. So let's zoom in a little bit. Let's open up our cropping tool. this little thing down here is basically a choice between mac os vision and uh an ai api call and the purpose of that is because sometimes people don't sometimes people don't trust ai you might have heard and so i i built that in as an option essentially i would i would think the ai option gets used more but nevertheless now you're going to select just the part of this article you care about or this paper that you care about and you can see there's like a crease in the paper there's a weird black mark here.

34:13But you can imagine we submit this for OCR. Now we have just that text that we pulled. We're also calling out for our editors, like where on the page, they're going to be able to find it if they want to sort of zoom in on it, crop to that particular article. And I can't exactly remember what text we were looking at, but it certainly completed those sentences where there was a black marker. Yeah. Right. So AI was able to kind of infer to the best of our ability what that sentence might have said. And, you know, if this ends up in a film, I could guarantee it would get fact checked later. But for the purposes of gathering documents, thousands of documents, this ability to kind of like precisely OCR is, is, has been a nice little unlock for us.

34:56Tim McAleer:One thing I also want to make sure people take away from this episode is we've seen basically three form factors of apps. So yes, they've all used AI, but you've been able to swap between sort of like a Python API service that gets called by another software application or database, a iOS app that, you know, you can run on your phone, and then like a little desktop toolbar widget. And what I like, what I love about this moment in AI with, with regards to software engineering, is like, if you have basic software engineering practices, and then you know, enough to be dangerous like yeah you can you can vibe code uh and you know a swift swift app to run on on your local desktop hyper specific app yeah no one was gonna make me this app and so the ability to make like an extremely specific app that makes a workflow you know on my team and my company easier it's been it's been an unbelievable moment yeah i would say the tam for this app is like you.

35:58Yeah. Yeah. I mean, I think I could sell it to like two colleagues.

36:01Tim McAleer:Well, and then my mom. So what I was going to tell you is my mom is a genealogist for the Daughters of the American Revolution, of which I am one. Fun fact on Claire. Oh, no way. And she does the lineage tracing. And do you know how many times she screenshot something and is like, can you read this cursive? Like, what in the world is this name? And it's like, you know, one name and a big, a big image. And so I do think AI is and I'm like, yeah, I'm gonna drop this in a chat GPT. And I'll tell you what I think it says. And I think its ability to read handwriting, old typefaces, kind of understand the nuances of spelling and things like that are just really, really interesting for these sort of research use cases.

36:44Yeah, we didn't look at a handwritten doc here. But that is definitely something happening at our company, like the ability to read letters that we could not read before and also just other languages. Right. And then we immediately have that text to you have letters written in some kind of cursive scrawl from the 17th century that is now translated to English and made legible for you.

37:05Tim McAleer:Amazing. Well, we've seen three great use cases. I am sure you are the hero on the team for this kind of stuff, because I can imagine again, people might be tired of hearing me talk about AI, but thank you. Yeah, but I mean, this is hard stuff. It's tedious work to do. It requires a lot of time, a lot of detail orientation. And I'm sure people love using this information to produce amazing things, but probably is not their favorite thing, like zooming in and squinting at the text to try to get it the most accurate as possible. Trying to automate away painful processes, right? Not the things people liked.

37:46Tim McAleer:Automate away toil. That's what we want to do. Okay, well, we're going to do a couple of lightning round questions. I'm going to get you out of here to go digitize a thousand more or more images. So the first thing I want to ask you about is just your approach to learning. It seems like from what I'm seeing, you're pretty fearless about new technologies, new things. I think this moment is such a critical moment for upskilling and learning. How do you think about learning in this moment? I think that one of the reasons that I find like tools like cursor or cloud code kind of intuitive is to me, there's a parallel with creative software.

38:24So like at various moments in my career, I have been deep in Photoshop or deep in Adobe Premiere, Avid Media Composer, whatever it is. And those softwares are so complex. They are like a maze of tool menus. And you end up on Reddit and on YouTube, doing your research, trying to just like figure out how to accomplish the thing. And I think that that's essentially what a lot of these tools are today too. Like I've been on cursor YouTube and cursor Reddit and learned tips and tricks from the vibe coding people of the internet. And I think it sort of starts from knowing what could be done or what's possible.

39:01And the like path to get there is swifter than ever before.

39:06Tim McAleer:What I like about this, I started sort of my fascination with technology in these creative tools. I will like this is like pre Photoshop where I would go and how can I make my text look like liquid gold and I would follow these like five step you know graphics tools tutorials and what I love about this moment in vibe coding or AI assisted engineering is coding feels such so much more creative than technical where these tools feel really like creation engines to me more than functional tools to write code. And so I love that parallel because it's what's made me so excited about technology my entire career.

39:50Tim McAleer:And I think it's why I'm so leaned in this moment, like activates that same feeling of like, oh, now I can do, can make this thing that I didn't think I could make before. I think that there are a lot of people too in my industry who have a kind of creative brain and creative approach to these things that would, you know, maybe like looking at a cursor window right now when you have no idea what it is, is a little scary. But I actually think that they are more well suited for the work than they might know. Well, let's talk a little bit about your industry, because I know that the film and creative world is deeply skeptical of AI.

40:23Tim McAleer:Sometimes we wade into the waters of AI video generation on this podcast and get a little feedback. And I totally understand. I have family that's in the creative industry. I'm curious, you know, what's your point of view of AI, particularly in the film world? What are you excited about? And where do you think these kind of concerns are really warranted? And then where do you think the most practical applications are? I think today it's like sort of where we started at the top. The practical applications are more in like tooling than they are in creation. But I do you think that like the creation is going to get there?

40:57Like today I play with, I play with all the generative video models. Like how can I not? They're, they're super fun. Um, they are not like at professional grade quality yet. Like the amount of time you spend throwing tokens at even the highest end video models, you're not going to be able to match your shots that well. You're not going to be able to match the footage you shot yourself that well. And so I don't think they're there yet, but like, I'll be honest, they're going to get there. I think that like, they are still exciting to me but i would separate a couple things like in the non-fiction world i think i think people should be careful like i think we should not be generating archival footage we should not be trying to fool our viewers into thinking that there was video in 1750 you know and i think that that's the part that's like a little scary and then of course there's the display like job displacement aspect of things i think people are scared if you film stuff for a living, you're definitely scared that you're going to be able to just use text to generate that same video you used to shoot.

41:56So I don't know how to like, I don't think anybody has good answers to that part of it. But my approach has certainly just been like jump in and learn the tools. Like they are, they are gonna be here, whether we want them to be or not. And I think that they have a lot of practical benefits today that are less scary.

42:16Tim McAleer:Yeah, the best advice I can give to people, and I have, of all the spaces, and I'll say this honestly, of all the spaces I have the most job displacement concern, it's in video generation for non-archival, non-documentary. Commercial. commercial use cases. You just see how it could be very applicable. And the best advice that I can give to people in this moment is the more you learn the tools, the better off you will be. Whether or not you love where the tools are taking us as an industry or as a culture, knowledge is power. And so the more you learn and understand, one, you can identify opportunities where it does add value even in your creative process.

43:02Tim McAleer:And two, you're going to be differentiated in the market from a job perspective because you're going to have a more robust sense of what's available in your industry. And I think that stands for people in your industry. I think it stands for people in my industry and technology. So I just say there is no harm in learning this stuff. Yeah, absolutely. I also think that like there's a place in the process for it which allows you like a place to learn without thinking it needs to end up in the final product, right? You can use video models for storyboarding all day. You can maybe prove whether or not that shoot is worth spending that money on.

43:33Now you've learned how to use the video models a little bit. And you haven't necessarily displaced anyone. But you've made your production a little bit more efficient, a little smarter. Maybe you've shot better footage as a result of it. Yes.

43:45Tim McAleer:But we're not generating fake archival footage of Genghis Khan. We are not doing that. Definitely not doing that. And I'm like, PBS, which is where most of our films end up, have a lot of guidelines around that. And I think that's a good thing, but it's the other stuff. It's commercial, it's visual effects. Like a lot of stuff's going to get easier. And so it's coming one way or another. Great. Well, last question have to ask you when, you know, you're on your dog walk with chat GPT doing voice mode and it's not listening to you or not giving you what you want. What is your personal prompting technique, especially because you use voice?

44:20Tim McAleer:Like I'm willing to type things to AI. I don't know if I'd be willing to say them. So what's your technique here? It definitely is different when you have to say it out loud. I am super nice to the AI. I can vividly remember the one time I was mean to it. I'm nice to the AI. I don't know where this is going. I'm going to be nice to all the models. What I do is like, for lack of a better way of describing it, I just start over. Like I will, I know that a lot of these things have ways of like consolidating the context window now and sort of summarizing, but I will ask for what I call like a resume work prompt.

44:53I'll be like, this isn't working. I want to resume work later with another AI dev. Can you give me a prompt with everything they'll need to know? And typically what you'll find is that that prompt shows you where it was off, you know, like in its summarization of what it was doing. I'll be like, oh, see, like I wasn't asking for that. That's that's why we were not communicating. And then I'll take that resume work prompt. I'll prune it a little bit, pop it into another chat. And then, you know, you'll find that you wish you hadn't beat your head against the wall with the previous chat for 20 minutes.

45:24Tim McAleer:You know, I am also team be polite to your AI. But then again, like you hurt the one you love the most. And I've found myself occasionally getting testy. And you know, when I stopped being mean to AI is when reasoning really started to show and I could see it reasoning how upset I was. It'll be like the user is mad at me right now. The user is really frustrated with me right now. I need to totally rethink. Go sweet, sweet baby AI. I'm sorry. Apologize. I'm not that mad at you. okay so create a you know return to progress prompt really get the summary take that to understand if there was some misunderstanding improve that and then just start fresh that's great well tim this has been super fun so much for me to learn i have tons of ideas even just for my day-to-day life about how i could use i have kids so i probably have 30 year thousand let me know if your mom wants the ocr party i will she'll love it okay mom i have gotten you your first vibe coded app direct from the podcast source.

46:22Tim McAleer:Tim, where can we find you and how can we be helpful? Yeah, I'm not that active on social, to be honest, but I am on LinkedIn, you can find me on there. I have a website that is itself a fun vibe code project. So you find me timmacle.com. I have a little chat bot there that GP Tim, you can go chat with him, learn a little bit more more about me and my work. And then other than that, I would say tune in to Florentine Films upcoming production. We have a series about the American Revolution coming out in November. So on your local PBS station. My kids are obsessed with the American Revolution. So everybody sounds like it's in the family.

46:58Tim McAleer:Yeah, we will we will be big fans. Tim, this has been great. Thank you so much. And thanks for joining How I AI. Thank you for having me. Thanks so much for watching. If you enjoyed the show, please like and subscribe here on YouTube or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiaipod.com. See you next time.