Causal Abstraction with Lossy Representations

0:00Welcome to the deep dive. you know it's pretty incredible when you think about it humans are naturally brilliant at two things mm-hmm first we're masters at figuring out cause and effect why did this happen because of that right the why and second we can take the most complex information and somehow distill it into simple abstract ideas like you see millions of tiny colored dots on a screen and your brain instantly says that's a cat just like that and that ability to simplify to abstract is, well, it's key. When you combine that with causal reasoning, that's when you unlock true understanding.

0:36It's how we plan for the future, assign responsibility, and apply what we've learned in one situation to a completely new one. Absolutely. So imagine if we could give AI systems this exact same powerful combination. How do we get AI to reason effectively about cause and effect, especially when the information it's working with is by its very nature simplified, or as the technical folks say lossy. Lossy. Because real world data is never perfect, right? When you try to simplify it for AI, you almost always lose some crucial details. It's the central dilemma, really, in making AI truly smart and reliable.

1:11How do you make intelligent decisions when you don't have all the pieces of the puzzle? And that's our mission today. We're going to dive deep into a groundbreaking approach called projected abstractions. Projected abstractions. This method helps AI navigate exactly this kind of complexity, allowing it to extract vital insights even when some information inevitably gets lost in translation. Think of it as a shortcut to getting highly accurate answers from fuzzy data. Which ultimately makes AI much more useful and hopefully more trustworthy in the real world. Exactly. Okay, so let's zoom in on those two human superpowers we just mentioned.

1:49First, causal reasoning. This is our innate ability to figure out cause and effect. It's what lets us plan our day, understand why a car won't start, or even learn from our mistakes. It's like the engine of intelligent action. And the second abstraction is just as fundamental. When you see a photograph, you don't process every individual pixel's color value, right? No way. Your brain takes all that raw data and interprets it as something meaningful. A tree, a building, or that cat you mentioned. That's abstraction. distilling complex patterns into simpler, more manageable concepts. So the grand challenge for AI is to truly combine these two.

2:27To tackle causal reasoning, AI often uses something called structural causal models or SCMs. SCMs, right. Think of SCMs as a formal blueprint for how different things in the world affect each other. They map out the variables and the mechanisms that connect them. And these SCMs are the basis for what's known as Pearl's causal hierarchy, sometimes called the ladder of causation. Judea Pearl's work. That's the one. It lays out three distinct levels of causal understanding. At the lowest level, layer one, or L1, you're simply observing what happens. Just watching. Yeah. You see the people who eat a lot of fast food tend to gain weight.

3:03You're just reporting facts. Correlation, basically. Then you climb to layer two, L2, which is the interventional level. Here you're asking what would happen if I actively change something. The what if I do this question? Exactly. What if I stopped eating fast food and started a healthy diet? What would happen to my weight? You're intervening in the system and observing the outcome. And finally, layer 3 L3, the highest rung, is the counterfactual. This is where you deal with what if scenarios that go against actual events. Like looking back and wondering. Right. Like what would have happened if I hadn't taken that medication given that I did take it?

3:38It's about imagining alternative pasts or presents. really tricky stuff. Now, a big goal in AI is what's called cross-layer inference, trying to figure out higher-level information like interventions or counterfactuals from lower-level data like observations. Yeah, can we get from L1 to L2 or L3? But here's the catch. The causal hierarchy theorem, or CHT, tells us that you generally can't just magically jump up this ladder without making some additional, often pretty strong assumptions. That's a key limitation. Observational data alone usually isn't enough to tell you what would happen if you intervened.

4:14You need more. So previous work on causal abstractions has tried to bridge this gap. They typically compare a high-level model, let's call it MH, MH, high-level, MH, with a low-level model, ML, using an abstraction function. Let's call it tau. Tau, like the Greek letter. Yep. The goal is often to ensure that the simplified model, MH, which behaves consistently with the detailed one, ML, especially when you're trying to figure out those what-if scenarios. Making sure the simple version doesn't lie about the complex one. Basically, yes. And this concept is also very much tied to representation learning.

4:48Ah, representation learning. We hear that a lot. Right, which is exactly about mapping detailed high-dimensional data, like images, into simpler lower-dimensional spaces, making sense of complexity. Okay, this sounds great in theory, mapping things down, keeping consistency. But you mentioned a catch earlier. Ah, yes, the catch. Yeah. This leads us straight to a really stubborn problem. Most existing definitions of causal abstractions run into a major limitation, what's called the abstract invariance condition or AIC. Abstract invariance condition, AIC. Informally, the AIC is a rule that says you can't group two detailed low-level situations together under the same abstraction.

5:30So map them to the same simple concept. If they have fundamentally different impacts downstream. Right. If simplifying them causes you to lose information about their differing effects, then you violated the AIC. Okay, hang on. Let me try to picture this. Imagine you have two different ways to get to a meeting, Route A and Route B. They both end up at the same building entrance. That's the abstraction, at the building entrance. Right. But maybe Route A involved fighting through terrible traffic that made you stressed while Route B was a calm walk through the park. So even though the abstract state at the building entrance is the same.

6:06The downstream impact how you perform in the meeting might be very different because of the history that got you there, the history that was lost in the abstraction. That's the AIC violation, losing relevant details in the simplification. Exactly. And the paper we're looking at gives a perfect, relatable example, cholesterol. A nutritionist might want to simplify a person's detailed cholesterol profile, which includes high-density lipoprotein, HDL, and low-density lipoprotein, LDL. The good and bad cholesterol, right. That's the common way to think of them, yes. They want to simplify this into a single number, total cholesterol.

6:40Yeah. Seems reasonable, just add them up. Yeah, simple sum. But here's the rub. We know HDL actually decreases heart disease risk while LDL increases it. They work in opposite directions. So if you just sum them into total cholesterol, you've created a single abstract value, TC, that lumps together two components with opposite effects on heart disease. So a high TC could be good, high HDL, or bad, high LDL, but the single number doesn't tell you which? Precisely. Total cholesterol becomes an ambiguous or lossy representation regarding heart disease risk. The downstream effects are different, but the abstracted value is the same.

7:17That's a classic AIC violation. Okay, I see. So why is this AIC such a huge hurdle in practice? Why does it matter so much for AI? Well, first, we rarely have the true underlying detailed model of reality, the low-level SCM available to us in the real world. So it's incredibly hard to even check if the AIC holds or not. You don't have the ground truth to compare against. Right. And crucially, the AIC is frequently violated when we use common AI techniques like representation learning or dimensionality reduction. The very tools we use to simplify. Exactly. These methods are designed to simplify data.

7:53They're inherently lossy transformations. If any information that gets lost in that simplification process turns out to be relevant for predicting downstream effects, then boom, the AIC is violated. But wait, there's a paradox here, isn't there? Go on. We need these lossy transformations, right? They're essential for making sense of complex, high-dimensional data. Makes it tractable, interpretable. Absolutely essential. You can't work with raw pixels for everything. So what do you do? How can you simplify data, which you need to do, without breaking your ability to reason causally about it, which you also need to do?

8:27Seems like a catch-22. It was a major catch-22. And this brings us to the ingenious breakthrough proposed in this work, projected abstractions. Okay, the solution. This is a brand new way to think about abstraction that actually embraces these lossy representations and AIC violations rather than being crippled by them. Embraces them. The core idea is brilliant, really. Even when the AIC is violated, you don't have to throw up your hands. Instead of thinking of a high-level quantity as a single fixed value. Like total cholesterol is 200. Right. You reinterpret it as a distribution over its corresponding low-level quantities.

9:03A distribution. So like total cholesterol is 200, which probably means this mix of HDL and LDL, but possibly that mix. Exactly. So a hard intervention on the high level, like saying set total cholesterol to 200, translates into a soft intervention at the low level. It's like saying adjust HDL and LDL in a way that's consistent with TC being 200 based on what we know about how TC usually arises. Ah, okay. It carries forward the uncertainty or the ambiguity from the low level. Precisely. It acknowledges the lost information. This approach builds on simpler ideas called SCM projections, where you either completely include or completely exclude variables from your model.

9:44All or nothing. But the innovation here is partial SCM projections, where variables are only partially projected away. Think of it like, instead of a perfect photo or no photo, you get a slightly blurry one. Still useful, just not perfectly sharp. Exactly. It retains some information, even if details are lost. Okay, let's make this concrete. You mentioned an insurance plan example from the paper. Yes. Example one and two. Let's walk through it. Imagine a low-level model, ML. You have an insurance company, variable Z, that offers different plans, variable X. Okay. Z causes X. And these plans, X, determine whether a claim gets approved, variable Y.

10:18So X causes Y. Z, X, Y. Got it. Now, suppose plans by one and by three are highly effective, say 90 % chance of claim approval, Y1. But plan by two is pretty ineffective, only 10 % approval. Okay, F1 by 3 good by 2 bad. And let's say company Z1 tends to offer the good plan by 1, while company Z2 tends to offer the bad plan by 2. So Z influences which specific X happens. Right, company choice affects plan choice. Now a data scientist comes along and wants to simplify things. They abstract plans by 1 and by 2 into a single category. Cheap plans, let's call this abstract value XC, and plan by 3 becomes expensive plans.

10:57Okay, my 1 and by 2 are now just cheap. And right away, we have an AIC violation, don't we? Ah, yes. Because by one leads to 90 % approval, by two leads to 10%, but they both map to the same abstract value, cheap? XC. Exactly. So you just say, interview to make the plan cheap, setting XH as XC. Yeah. It's completely ambiguous what will happen to claim approval. Why? Does it become 90 % or 10 %? The abstraction lost that crucial difference. This is where projected abstractions come in. This is where they write in to save the day. The ingenious solution is to introduce a new kind of variable at the high level.

11:30M-H. Let's call it X-U-H. Think of it as U for unobserved, H for high level. Okay, an extra variable in the simple model. This new variable represents the lost information. It's like a hidden switch that tells the high-level model whether cheap plan, X-C, originally came from the good low-level plan or the bad one by two. It tracks the ambiguity. Precisely. The high-level function for claim approval, FYH, then uses both the simplified plan category, X-H, X-H, C, and this XUH variable to perfectly mimic the low-level behavior. It essentially recovers the critical loss context needed to predict Y correctly.

12:07Wow. Okay, so it builds the lost information back into the structure. It does. And what's truly fascinating, and maybe a bit unexpected at first, is a side effect of this process. You can actually introduce new dependencies in the simplified high-level model, MH, that weren't obvious the original detailed model, ML. New connections appear. How? In our insurance example, remember company Z influenced which plan by one or by two was chosen? Yeah, Z1 liked by one, Z2 liked by two. Because of that, the probability that an abstract cheap plan actually corresponds to the good plan by one depends on which company Z we're talking about.

12:41Ah, so Z affects the hidden variable XUH, which represents the lost by one by two distinction. Exactly. And since XUH now influences Y, claim approval, in the high-level model, we get an effective new dependency directly from Z company to Y, claim approval, it is ZY. Even though in the original low-level model, Z only influenced Y through XZXY. Correct. This new Z-to-Y link appears in the abstracted model because Z helps resolve the ambiguity introduced by the abstraction of X. This is absolutely critical for doing practical causal inference later on. That's subtle but powerful, and the paper provides an algorithm for this.

13:19Yes, algorithm 1 systematically constructs this high-level model, NH, defining that hidden variable, XUH, in the new functions, ensuring MH stays consistent with ML for all causal queries, even with AIC violations. It uses those soft interventions based on probabilities. Okay, so we have this powerful theoretical breakthrough. Projected abstractions let us handle lossy data, but in the real world, we rarely get to see that perfect low-level model, ML. That's the rub. Usually we just have data, often observational data, that layer one stuff. We need to infer the high-level causal effects from that.

13:53Right. So how do we do that, especially if the AIC is violated? Do the old tools work? And this is where existing tools, like the cluster causal diagram or CDAG, fall short. CDAGs are great graphical tools for reasoning about abstractions, but they generally assume the AIC holds. When the AIC is violated, they're simply not sufficient. Why? Because, as we just saw with the insurance example, the partial loss of information, the projection, can introduce those new critical dependencies. I think there's a Z to Y connection. Exactly. That the old CDAG just doesn't capture. It assumes certain independencies that might no longer hold after the lossy abstraction.

14:29So the map is wrong, basically. The map needs an update. To solve this, the paper introduces a new modified graphical model, the partially projected CDAG, which they denote as GAC. G-DAG or C. He updated that. Think of your original causal diagram, the CDAC. When an AIC violation occurs for a specific variable, like X in our example, this new GAC tool intelligently adds new edges, new arrows, between other variables to account for those emerging dependencies. So in the insurance example, it would explicitly add an arrow from Z to Y in the high-level graph. Precisely. It adds edges like Z to Y if the original path was Z, S, X, Y, and X is an AIC violator.

15:11It systematically accounts for these induced dependencies caused by the lossy abstraction. Okay, so we have a better map now. How powerful is it? The power is enormous. It's been mathematically proven, theorem two in the paper, that this projected CDAG, GIC, completely describes all the necessary conditional independence constraints for doing causal inference over the high-level variables, even when the AIC is violated. Wow. So we finally have the right map to navigate these complex lossy causal landscapes. That's the idea. And it dramatically simplifies something called abstract identification.

15:44Abstract identification. That sounds like figuring out if you can answer your causal question with the abstract data you have. Exactly. Can this high-level causal query, like PY do XAJC, be computed from the available data, usually observational, and the causal graph? And the projected C DAG helps here. Massively. There's a really game-changing result, theorem three, called dual abstract identification. It essentially says that this abstract identification problem, figuring out causal effects across different levels of abstraction, even with AIC violations, is exactly equivalent to doing standard causal identification on the high-level space using this new projected CDAC.

16:24Oh, wait. So figuring out this complicated abstract thing is the same as figuring out a standard causal thing just using the new corrected map. You got it. What this means and practice is huge. All the existing well-established methods and algorithms for causal identification tools researchers have been refining for decades, like the backdoor criterion, frontdoor criterion that you calculate. Wow, that standard stuff. Can now be directly applied to these more complex abstract scenarios involving projected abstractions. You just apply them to the GAC graph. That's incredible. It takes this seemingly new hard problem and makes it solvable with the existing toolkit.

16:58It really bridges the gap. So going back to our insurance example, answering what's the probability of claim approval given we force a cheap plan, POA2 XCC, using the GAC graph, which now has that ZY edge. You can use standard techniques, maybe like adjusting for Z using the backdoor criterion. Precisely. You can identify the query using standard methods on the GAC graph. The complexity of the AIC violation is handled by the structure of GAC itself. Okay, this theory is clicking. It's elegant. But does it actually work in practice, especially with really messy high-dimensional stuff like images?

17:34Great question. The researchers didn't just stop at theory. They bridged theory and practice by testing these advancements using neural causal models, or NCMs. NCMs, using neural networks for causal reasoning. Essentially, yes. Specifically, they used GN-NCMs, which combine generative adversarial networks with causal models. These are particularly good for generating and understanding high-dimensional data-like images. Okay, so what did they test? In their first experiment, they really wanted to nail down the proof. Are these new projected seed eggs actually necessary when AIC is violated? The necessity test.

18:07They set up a system using colored MNIST-DITs. Imagine ground truth where the digit itself, say a 7, influences a specific colored image of that digit. So 7 might tend to be, I don't know, green. Could be. And then that colored image influences a prediction about the image's overall color. The crucial step was the abstraction. They abstracted the detailed colored image into just a simple binary shade, either light or dark. Okay, very lossy. Lots of AIC violation potential there. Definitely. Then they tried to estimate a specific causal query using different models. And the results. Truly eye-opening.

18:43When they tried to estimate the query, two models succeeded. First, a complex abstractionless model that just operated on all the raw, high-dimensional image data. The brute force approach. Right. And second, their new projected CDAG model using the GIC graph. Both of these show their error decreasing nicely as they got more data samples. They learned the right answer. Okay, good. But what about the old way? Here's the kicker. The regular CDAG model, the one that didn't use GAC and thus didn't account for AIC violations, completely failed. Its error stayed high. It couldn't learn the correct answer.

19:17Proof positive that the old map fails here. Absolutely. It simply couldn't make sense of the lost information captured by the AIC violation. And interestingly, the projected C DAG model actually performed better than the complex abstractionless one, likely because it worked in a much more manageable, lower-dimensional abstract space. Wow. So simpler and more accurate if you use the right map. That's the takeaway. Yeah. Their second experiment really hammered home the power of the inference side using something they call projected sampling. Projected sampling. This was a more intricate colored M &S experiment.

19:48Here, both the digit itself, D, and its color, C, independently caused the final image, I. So D, I, and C, I. Okay. But there's a twist. The digit and color were correlated or confounded. For example, maybe G's tended to be red and 5's tended to be cyan due to some unobserved common cause. A common setup in real data. Things are correlated for underlying reasons. Exactly. They then tested the model's ability to answer questions across the ladder of causation. L1, L2, L3 again. Yep. First, L1 observational. What does an image of a zero look like normally, given the confounding? You'd expect to see mostly red zeros.

20:29Okay. Second, L2 interventional. What would an image look like if we forced it to be a zero, breaking the usual color correlation? Here, you should see zeros of all possible colors, not just red. Right, cutting the confounding link. And third, L3, counterfactual, a trickier one. What would an image of a zero look like if it had the color characteristics of images that were originally fives? PID 0, D5, since fives tended to be cyan. You'd expect to see cyan zeros a color zeros don't normally have. Precisely. It's asking what a zero looks like under the typical conditions of a five. Okay, so how did the models do?

21:02The results were incredibly compelling. First, a basic non-causal approach, like a standard GN, could generate the L1 images, red zeros, but completely failed at L2 and L3. No surprise there. Doesn't understand causality. Right. Then they tried a standard representational NCM, RNCM. It could reproduce all the queries, L1, L2, and L3, but only if it used a relatively high-dimensional latent representation for the images, say, 16 dimensions. Okay, so it needed a fair bit of detail in its abstraction. Yes. But then they forced it to use a very, very low dimensional representation. Literally just a single binary variable, 0, 1, to represent the entire image.

21:42Whoa, that's extreme compression. Basically just light or dark again or something similar. Pretty much. And under that extreme compression, the standard RNCM simply couldn't generate meaningful images for the L2 and L3 causal queries. The abstraction was too lossy for it to handle. It broke down. Makes sense. But here's where everything clicked into place. The RNCM Plus projected sampling approach using their new framework. The one that embraces the lossiness. Even with that impossibly tiny binary representation, remarkably reproduce high-quality, visually correct images for all L1, L2, and L3 queries.

22:16Get out! With just a binary abstraction, it could generate cyan zeros. Yes. It successfully performed the intervention and the counterfactual generation, even though its internal representation of the image was incredibly simplified. It's like magic, but it's math. That is genuinely stunning. What's the takeaway there? This is a monumental takeaway. Projected sampling, built on projected abstractions, allows for high-quality causal understanding, inference, and even data generation, like images, even when you've reduced the data's complexity to an extreme degree. Which is exactly what you need for real-world AI.

22:53Absolutely. That's vital for making AI models tractable, able to be processed efficiently, and interpretable, easier for us to understand when dealing with massive, complex data sets. It means AI can potentially make sense of incredibly simplified representations without losing its crucial causal reasoning capabilities. Wow. What an incredible deep dive. Okay, let's try and wrap our heads around this. We started by appreciating those fundamental human intelligence superpowers, causal reasoning, and our amazing ability to abstract complexity. Yeah, the bedrock of how we think. Then we tackled the colossal challenge AI faces when trying to do the same thing, reason with cause and effect, using inherently simplified or lossy real-world data.

23:35A challenge largely created by that tricky abstract invariance condition, the AIC, which most simple abstractions violate. Right, the rule that says you can't simplify if you lose important downstream differences. And we found a powerful, really elegant solution in projected abstractions and their accompanying graphical map, the partially projected CDAG or GC. These innovations allow AI systems to work with abstractions that do violate the AIC. Effectively recovering, or at least properly accounting for the lost causal information and making sense of the ambiguity that abstraction introduces. So what does all this mean for you listening right now?