Causal Inference with Video Features as Treatments

0:00Imagine you are watching, I don't know, a 30 second political ad on your phone. Right. Just scrolling away. Exactly. And at exactly second 14, you suddenly feel this massive spike of anger. Oh, yeah. We've all been there. Right. But was it the specific word the candidate just said? Was it the, you know, dramatic shadowy shift in the lighting? Or maybe it was the subliminal slow motion footage waving in the background? So hard to tell in the moment. It is. For decades, marketing executives and campaign managers just called this intuition. They treated video persuasion as this ineffable art form.

0:36Yeah. But today, artificial intelligence has turned that magic into brutal, exact math. Yeah, because when you step into the realm of high-dimensional data analysis, that creative vibe completely evaporates. It just disappears. Totally. You realize that every single frame of a video is a highly structured, measurable input. But the challenge has always been that isolating exactly what caused your emotional reaction in a constantly moving visual landscape is just a statistical nightmare. Oh, I can imagine. If you've ever tangled with time series data, you know the absolute nightmare of confounding variables.

1:11It's a mess. And for you listening, whether you're, I mean, building predictive models for a tech firm, prepping for a marketing strategy meeting, or you just want to understand the modern world without drowning in information overload, you've definitely felt this. Oh, absolutely. You scroll through TikTok, you watch the news, campaign ads, whatever, and you realize your emotions are being played like a piano. But what exactly is pressing the keys? Okay, let's unpack this. Let's do it. Because our mission today for this deep dive is to explore a groundbreaking methodology, one that uses generative AI not to conjure up fake videos, which is what we usually hear about.

1:49Right. Not deep fakes. Exactly. But to mathematically decode the frame by frame cause and effect of the visual media we already consume. And the primary hurdle we have to clear first is understanding why traditional statistics completely shatter when applied to video data. They just break down. Completely. Analyzing a video is fundamentally different from parsing a spreadsheet or a text document. Video is aggressively dynamic and, well, staggeringly high-dimensional. Because there's just so much going on at once. Exactly. A single second of footage contains dozens of individual frames, and each of those is packed with millions of pixels, fluctuating audio frequencies, semantic speech, and, you know, complex spatial relationships between objects.

2:32Which brings us to the core trap, right? Confounding features. Yes, the confounders. Let's say the specific variable we want to measure the treatment in statistical terms is a politician physically appearing on screen. We want to know the pure, isolated effect of seeing that specific face. Right. The insurmountable problem is that a human face never appears in a vacuum. No, it doesn't. The moment the politician appears on screen, the lighting likely shifts to a warmer color temperature. Sweeping, upbeat orchestral music swells in the audio track. Oh, like a crowd cheering in the background. Exactly.

3:07The tone of the voiceover changes. All of those elements are confounding features. They are statistically tangled up with the appearance in the face. It's like trying to isolate the exact chemical that causes a complex reduction sauce to change flavor. But you poured four different liquids into the pan at the exact same millisecond. And the heat is rapidly rising. Right. And the ingredients are constantly reacting with one another. It's an entangled mess, which, you know, makes me wonder why overcomplicate this. What do you mean? Why not just rely on the gold standard of testing? Take a group of people who watched the video, compare their reactions to a group of people who didn't watch the video, and just measure the delta.

3:47Well, what's fascinating here is that we aren't trying to measure the aggregate impact of the entire video. We already know the video as a whole moves the needle. Okay, sure. The objective is to extract the causal effect of one specific fleeting feature as the video unfolds over time. Furthermore, human emotion is not static. Right, it fluctuates. Exactly. If we measure a viewer's reaction at second 15, that data point isn't just a response to the pixels on screen at second 15. It is a compounding result of what they observed at second 5, second 10, and second 12. Oh, wow. We refer to this as temporal inertia or carryover effects.

4:27So you're basically dragging the emotional baggage of the first 10 seconds into the back half of the ad. That is a perfect way to put it. Traditional aggregate methods completely fail to capture that reality. They look at a baseline feeling at the start and an average feeling at the end, completely missing the temporal journey. They miss the whole story. Right. They cannot map the sequence of how persuasion is constructed, framed by agonizing frame. So traditional math can't handle millions of simultaneous pixels, and it can't account for the temporal inertia of human emotion. It just doesn't scale.

5:00We desperately need a tool capable of compressing an oceanic amount of visual data into a manageable format, but without losing the vital context of the lighting, the background flags, or the facial expressions. Enter generative AI. Yes. This methodology relies on a brilliant workaround called Gen-I-powered inference, or GPI. Historically, if researchers wanted to track visual variables, they had to rely on human coders with stopwatches. Oh my gosh. Manually logging events. Exactly. Sitting there going, flag appears at.02, candidate smiles at.04. It is painfully coarse. You'd lose all the subtle stuff.

5:40All of it. You lose all the high-dimensional data, like the RGB values of the lighting or the spatial proximity of objects. Because nobody is pausing a video 30 times a second to write down the hex codes of the background pixels. That's insane. Nobody has the time for that. So instead of human coders, this approach takes an existing real-world video and feeds it into a deep generative AI model. Specifically, highly advanced video tokenizers like NVIDIA Cosmos. Okay, but the critical distinction here is that we are not prompting the AI to hallucinate new content. No, not at all. We are demanding that it perfectly reconstructs the original video from scratch.

6:15And here's where it gets really interesting. Oh, this is the best part. The reconstructed video itself is basically useless to us, right? It's just a clone. The actual goldmine is the AI's internal representation created during that reconstruction process. The latent space. Yes. For a generative AI to rebuild a high-definition video frame by frame, it first has to profoundly understand the structural essence of what it is observing. The encoder maps all those millions of chaotic pixels into a low-dimensional mathematical vector. It compresses it. Exactly. It creates an incredibly dense, highly structured numerical blueprint of the video's contents.

6:55We are effectively stealing the AI's private study notes. We're ripping out its brainwaves. Pretty much. Because that mathematical blueprint automatically contains all the confounding features we were just talking about. The internal representation inherently maps the lighting radiance, the background geometry, the atmospheric mood. even if we never explicitly programmed the AI to look for a waving flag. Right. And by taking this dense AI blueprint and feeding it into a longitudinal neural network, we can finally mathematically isolate the exact variable we want to study. The neural network acts as a dynamic deconfounder.

7:26A dynamic deconfounder. I like that. As it ingests the sequential latent representations across time, it traces the trajectory of the video. It learns the temporal rhythms. So it maps the timing. Exactly. By mapping out the precise timing of all those hidden variables embedded in the AI's representation, the network can strip away the correlated visual noise and isolate the true causal trigger. Okay, I understand the theory. You use the AI's internal blueprint to filter out the noise. But, you know, theory is cheap. It is. If you want to prove this mathematical extraction actually works, you can't test it on messy, unpredictable human emotions first.

8:06You need a testing environment where you already possess the absolute, undeniable, ground truth answer. You need a completely closed ecosystem. You need a video game. Yes. The validation environment chosen for this methodology is arguably one of the most elegant stress tests for causal inference ever designed. I love this part. They built a massive benchmark using 10 ,000 custom-generated levels of Super Mario Bros. That is just incredible. A classic 8-bit platformer used as a high-dimensional data laboratory. It's brilliant. Every single one of those 10 ,000 levels is played by a fixed AI agent, a bot, programmed to play the game with perfect consistency.

8:43Okay. The gameplay footage is recorded into exactly 26-second auto-scrolling videos, so the data environment is completely controlled. So we have 10 ,000 videos of Mario just running to the right. To test our dynamic deconfounder, we need variables, obviously. Right. We define three specific variables. The treatment, the thing we are trying to measure the effect of, is Princess Peach appearing at the top of the screen. Okay. Peach is the treatment. The confounding feature, the distraction, is the classic green pipes appearing on the ground. Finally, the outcome we're measuring is the number of times Mario jumps.

9:16Got it. The treatment is Peach. The confounder is the pipes. The outcome is jumping. And the entire system is intentionally rigged to create a statistical trap. The appearance of Princess Peach in the sky is heavily mathematically correlated with the generation of green pipes on the ground. If there are a lot of pipes, Peach is highly likely to appear above them. Oh, I see. However, Princess Peach is a purely cosmetic overlay. She has absolutely no hitbox. She is a ghost. Exactly. The Mario AI agent does not even register her existence. her pixels have zero interaction with the gameplay engine.

9:51Wow. The only things that actually trigger the Mario agent's jump mechanics are the green pipes. Therefore, the absolute, objective, ground truth, causal effect of Princess Peach appearing on screen is exactly zero. She causes zero jumps. This is a brilliant trap for bad math, because Peach almost always shows up when the pipes show up, right? Right. And the pipes constantly make Mario jumps. So a naive statistical model is going to look at the aggregate data and confidently declare, look at this tight correlation. Every time Peach appears, Mario jumps. Peach is clearly causing the jump. That's precisely what happens when standard models process this data set.

10:27They fall face first into the confounding trap. They get fooled. Completely. They assign a highly positive causal effect to Princess Peach. Even if researchers manually intervene, if they count the pipes on the screen and instruct the model to adjust for their presence, the traditional methods still fail. Wait, why do they still fail if you explicitly tell the model about the pipes? Because a raw pipe count is too coarse. It completely misses the spatial layout and the temporal evolution. What do you mean by temporal evolution? Well, knowing there are three pipes on screen doesn't tell the model if they are clustered together or spread far apart.

11:05And crucially, it doesn't map the exact millisecond a pipe enters the agent's interactive range relative to the jump. Ah, I see. Enter the Gen AI cheat sheet. Exactly. When the dynamic GPI framework is deployed, the system takes the Gen.AI reconstructed internal representations of those 10 ,000 Mario videos and processes them through the longitudinal neural network. The deconfounder. Right. The neural network analyzes the sequential flow of the latent space. It correctly maps the temporal sequence of the game, separating the correlated MOIs from the true physical triggers. And it perfectly calculates that the causal effect of Princess Peach is exactly zero.

11:43Okay, I have to push back here for a second. You're telling me the AI figured out the actual physical cause of the jumping just by watching the reconstructed pixels over time. That's right. We didn't feed it the game's source code or explicitly teach it the rules of Mario's collision detection. The system had absolutely zero access to the underlying game engine or source code. That is wild. The neural network learned the true causal structure purely by capturing the high-dimensional visual context from the tokenizer's internal representation. Just from the visuals. Just from the visuals. By tracking the sequential timing of the latent variables, it learned that the pipes consistently preceded the jump within a specific temporal window.

12:24While Peach's appearance was temporally and functionally disconnected from the jumping mechanics, it organically learned to see the physics of the game. That is wild. It saw through a rigged correlation purely by reading the temporal flow of the latent space. But, you know, as impressive as that is, Mario is just a closed loop of code. Very true. Moving from an 8-bit plumber to the most emotionally volatile, unpredictable data set on the planet, American voters watching political television, is a massive leap. It is a massive leap. But applying this exact latent space analysis to human emotion is where the methodology truly proves its value.

13:01Okay, let's hear it. The real-world application utilizes 849 actual television ads broadcast during the 2020 U.S. presidential campaign, specifically tracking the appearances of Joe Biden and Donald Trump. Okay, let's set the parameters for this. How are we mathematically measuring human emotion in real time? That sounds impossible. The data set relies on over 1 ,100 unique survey respondents. As they watch these 30-second and 60-second campaign ads on their screens, they continuously manipulated a digital feeling thermometer dial. Like a literal slider on the screen. Exactly. They could slide the dial anywhere from zero, representing very cold or hostile feelings, up to 100, representing very warm or favorable feelings.

13:43So we have a continuous second-by-second graph of their emotional state. We do. The videos were then sliced into five-second segments to map the temporal flow. The audio transcripts were extracted and processed using OpenAI's Whisper model combined with an instruction-fine-tuned LAMA 3.1 model to extract the exact semantic meaning of the spoken words. To understand what was actually being said. Right. Simultaneously, the visual frames were rebuilt using the NVIDIA Cosmos model to extract the dense visual blueprints. Wow. So we have isolated the AI's understanding of the semantic speech, the AI's internal representation of the visual context, the lighting, crowds, camera, angles, and we have mapped all of that against the human's real-time emotional dial.

14:26All synced up. What exactly are we hunting for in this data? The objective is to isolate the pure causal effect of the political candidate physically appearing on screen. If the candidate's face enters the frame during a five-second window, does that specific visual event physically drive the viewer's feeling thermometer up or down? Entirely independent of what the candidate is saying or what the background looks like. Yes. Independent of all those confounders. We are bringing the Mario Deacon founder to the American electorate. What did the data reveal? Just the neutral facts. The extracted data is incredibly stark.

15:04When isolating the effect of a Democratic candidate appearing on screen, the framework found that for Democratic respondents, increasing the probability of that visual appearance over time caused a cumulative increase in their feeling ratings. So the more the candidate's face was visually present, the warmer the emotional dial became. Exactly. Well, that aligns with basic logic, right? You see your preferred candidate, you feel validated, and that positive feeling compounds sequentially as the ad continues. What happened when the other side watched? When Republican respondents watched those same Democratic candidate appearances, isolating the face on screen revealed a severe, measurable backlash effect.

15:39A backlash effect. Yes. The visual presence of the opposing candidate directly and cumulatively drove the Republican respondents' emotional ratings downward as the video progressed. Again, completely logical. But here is the statistical twist that stood out in the data. What happened when the roles were reversed? You mean Dem respondents watching rep ads? Yes. When Democratic respondents watched Republican candidates appear on screen, did they experience a massive cascading backlash effect? They did not. There was no equivalent statistically significant backlash effect observed among Democratic respondents when Republican candidates appeared visually.

16:18Wait, really? Why the asymmetry? Were they just immune to the visual trigger? No, it is a fascinating mathematical quirk regarding the floor of the data set. The Democratic respondents feeling ratings for the Republican advertisements were already sitting at the absolute bottom of the thermometer from the very first frame of the video. Oh, regardless of who was on screen. Exactly. The overarching context of the ad immediately tanked their emotional state to zero. You mathematically cannot drive a score down if it's already sitting at the floor. Oh, wow. The candidate's visual appearance couldn't cause a cumulative backlash because the baseline hostility was completely maxed out from second one.

16:57The dial physically couldn't go any further left. That makes so much sense. Now, I do have to challenge the actual metrics being used here. OK, go ahead. When we talk about these interventions, the methodology focuses on increasing the probability of a candidate appearing. Why are we dealing with probabilities? If we have this omnipotent mathematical framework, why not just ask the AI model what happens to the viewer's emotional dial if the candidate's face is plastered on the screen for an uninterrupted 30 seconds? That's a fair question. Right. If we can't test a basic baseline like that, how can we trust the model's math?

17:31Well, this raises an important point about the limitations of causal inference. When dealing with real-world high-dimensional data, you have to operate within the bounds of dynamic stochastic interventions. Okay, translating from data science, that means what exactly? If we connect this to the bigger picture, it means we have to respect the underlying distribution of reality. If you force a completely alien, unnatural scenario into the mathematical framework, like a political candidate staring silently and unblinking into the camera for 30 straight seconds with no cuts. That sounds terrifying.

18:03It is. And the statistical model will completely break down. That scenario violently violates real-world patterns. The model has no baseline data for how a human would react to that because it doesn't exist in the training distribution. It would register as a psychological horror film, not a political ad. The viewer's reaction would be panic, not persuasion. Exactly why forcing impossible binary interventions corrupts the math. Instead, the framework subtly shifts the mathematical odds. How subtle. It analyzes the temporal sequence and says, based on the internal representation, the candidate naturally appears 40 % of the time during this specific sequence of lighting and audio.

18:42What happens to the causal effect if we marginally bump that probability up to 60 %? Ah, keeping it realistic. Right. This ensures that the counterfactual scenarios we are testing remain mathematically sound and anchored to plausible reality. So what does this all mean? We have navigated a massive conceptual shift today. We really have. We started with the sheer chaos of a video file, millions of pixels intersecting sounds and unspoken spatial context that historically trapped data scientists in this inescapable maze of confounding variables. A literal maze. And we've seen how, instead of desperately trying to untangle that mess manually, we can essentially outsource the heavy lifting to generative AI.

19:23We can utilize a tokenizer's dense internal representation, its low-dimensional mathematical cheat sheet to filter out the noise. Whether we are analyzing a digital plumber dodging green pipes or an American voter reacting to a television ad, we now possess the mathematical architecture to prove exact frame-by-frame cause and effect. And if we connect this to the bigger picture, we have officially moved from treating media engagement as an unpredictable art form to treating it as a measurable, highly sequential science. The science of persuasion. Yes. Every subtle shift in lighting, every background object, every split-second visual appearance of a face carries a specific, calculated weight.

20:07And for the first time, we can mathematically isolate what that weight is, completely free from the noise of the surrounding pixels. And for you listening, I think the most vital takeaway is how this applies to your own daily media diet. Oh, absolutely. The next time you are scrolling on your phone and you find yourself suddenly furious or deeply emotional or completely persuaded halfway through a video, realize that your reaction is not a spontaneous accident. It's by design. Every single frame you just consumed is part of a heavily structured sequence. Data scientists and algorithms now possess the exact tools required to decode and measure precisely which millisecond of visual data manipulated your thumb to stop scrolling.

20:46It really forces a profound re-evaluation of your own agency online. You begin to see the structural skeleton, the calculated mathematical triggers, lying just beneath the surface of the content. It's a lot to process. And I want to leave you with a lingering, highly provocative thought to mull over. Right now, this generative AI methodology is being utilized strictly as an analytical tool. Right, looking at the past. It is looking backward at videos that already exist to decode why we reacted the way we did. But if we now possess the definitive mathematical framework to pinpoint exactly which split-second visual features manipulate human emotion, what happens when the pipeline is reversed?

21:27That's the scary part. Imagine a world, and honestly not in the distant future, but rapidly approaching, where the video content you watch is no longer pre-recorded. Imagine an AI dynamically monitoring your live reactions through your device's front-facing camera. Reading your microexpressions. Yes. Reading your microexpressions, your eye tracking, your engagement time, and then using this exact causal math, it generates every single frame of the video on the fly. Real-time generation. It custom tailors the light ingredients, the background geometry, and the exact timing of the faces to perfectly and relentlessly trigger your specific subconscious biases.

22:03that magical vibe that makes you want to buy a product or vote for a candidate. It's just a mathematical formula. It's just a formula. And the algorithm might already know your formula better than you do.