Are LLMs Good at Causal Reasoning? with Robert Osazuwa Ness - #638

0:07All right, what's up, everyone? Welcome to another episode of the TwiML AI podcast. I am your host Sam Charrington. Today, I am excited to be here with good friend and longtime friend of the show, Robert Ness, a senior researcher at Microsoft Research, professor at Northeastern University and founder of altdeep.ai. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Robert, welcome back. It's great to see you. Thanks for having me back, Sam. Yeah, it's good to see you as well. Thanks for having me back. In a sense, this interview was maybe foreshadowed a little bit.

0:45Just recently, our conversation from the beginning of the year in which you reviewed research progress in causality and causal modeling for our AI Trends series was published. that was published back in February. And kind of at the end of that, or maybe throughout, we were talking about LLMs and the implications of LLMs and causality. And it was a little bit of an open question at the time, but you and some of your colleagues just published a paper, Causal Reasoning and Large Language Models, Opening a New Frontier for Causality, which really dug into the topic. So quick work on that, apparently.

1:27And I'm looking forward to digging into that. I think folks on the show may know a little bit about you, but briefly share your background and what you do at Microsoft. Sure. Okay. So I'm a senior researcher at Microsoft Research. I work on the special projects team that's out of Redmond. And my background is in probabilistic machine learning, probabilistic programming, and causal inference and how these things come together of causal AI. I've worked a lot of language models too, and it's really exciting now working with large language models at MSR, I can think a few places where that are competitive of MSR in terms of a place to work on these problems.

2:04And then my team at Special Projects, we look for these research applications and how we can transfer them into impact both inside of MSR and externally. And so we were very keen to explore this problem of what exactly can large language models do in the area of causality. And this was in collaboration with my colleagues at MSR, Amit Sharma and Amri Kisman, as well as Tung Hau Tan at the University of Chicago. And we had essentially explored some benchmarks in the space, as well as different areas of causal analysis and brought that together in this paper. So it was a really exciting work. I was happy to be a part of it.

2:44Awesome. Just to ground the discussion for folks that are new to the idea of causality and causal modeling, causal analysis. Give us a quick overview so that we can have that context for the specific conversation about LLMs. Oh, so summarize all of... Summarize your life's work in 30 seconds or less, please, sir. Sure. Yeah. So causal analysis is something that we're interested in across different fields. So ranging from econometrics to epidemiology to statistics, obviously, and natural sciences. and the issue is we have data and oftentimes it's from just observational, passive observational data.

3:26There's sometimes it's experimental data and we want to make causal conclusions from the data and knowing the aphorism correlation does not imply causality. We make certain modeling assumptions and then use the data to try and make causal conclusions and recently it's become, I don't know how recently, but semi-recently it's become an interesting topic of focus in machine learning, for one, because oftentimes we have very large data sets. We want to leverage the ability of machine learning to scale up to large data sets. But also because causal reasoning is an essential task. If you say that our goal of AI or the goal of our field is to achieve generalized artificial intelligence, we believe that the ability to reason causally is going to be a part of that generalized AI.

4:12So one aspect of this field is figuring out, okay, how can we get learning agents to reason causally and make correct causal conclusions or make causal conclusions in a way that is maybe comparable to a human baseline or is aligned with some standard for reasoning? And it's definitely a growing field. And it's particularly interesting now that we have these foundation models to work with. What's an example of causal reasoning in the context of LLMs? And you can use one from the paper. Okay. A simple example might be, let's say, so pair-wise causal discovery. So this is to say given two variables, assuming that one causes the other, let's just say that we know that one causes the other, trying to find out which causes which.

5:00And so this could be done with what in the paper we call covariance-based analysis, which is to say analyze the data and using certain modeling assumptions, try to conclude whether A causes B or B causes A. And with the large language models, we can also just ask it, given the variable names, for example, does temperature cause altitude or does altitude cause temperature? I call that pairwise causal discovery. There's also a full graph causal discovery, which is to learn an entire causal graph of relationships between variables. There's also kind of, if you're particularly in econometrics epidemiology statistics, looking at whether or not something causes an earth things that were the gold standard would be like a randomized clinical trial.

5:45Then we got some sales from some months in November and December and January, and we saw a spike in sales in December. And in early December, we placed an ad, did the ad cause the spike in sales, right? And posing that in that natural language form to the large language model and asking it to give you a response. That would be a kind of causal query for what we would call an average treatment effect or a conditional average treatment effect. Okay. And I think we may come back to that example and kind of talk about what you saw, but maybe frame the paper a little bit in terms of your broad goals for it.

6:30Yeah. So I think when we started looking at the question, we were seeing like, okay, well, GPT 3, 3.5 and 4, these are very impressive models in terms of the various benchmarks that we would use to evaluate a large language model. And how well does it do on causal inference, causal or causal queries in general. And so the question in the room, we were thinking, can large language models reason causally? And of course, when you say the word reason, it's a bit anthropomorphizing the model. And so to be a bit more specific, at least the way I would frame it would be to say, are there conditions under which we can pose a natural language causal query and reliably or predictably get the right answer or not get the right answer, according to some definition of right, in some cases, that being some objective truth, or in other cases, maybe reasoning along the lines of how a human might reason about a causal question.

7:24Then we discovered this is a very broad question. It's very interesting, given the types of questions that we can ask it. We found these benchmarks, we posed it to the large language model, and we got impressive results. Upon looking at it more closely, we also found that actually the benchmarks aren't very good at answering this specific question for reasons we can talk about. And that's in order to answer this kind of question, we worked with the OpenAI's series of models. And for our analysis, we didn't have access to weights, the training data, the architecture. So maybe this is a kind of question that you need that kind of insight into the weights, into the training data to be able to answer provably.

8:07And so we think that that's a very important question that I think We think it needs to be perhaps better posed, but there are some interesting experiments that could be set up there with various levels of access to data or perhaps training a new model that has certain disinterrata for causal reasoning. But we did find that the answer to a different question was an enthusiastic yes. And that question was, can large language models essentially supercharge causal analyses? Our work as modelers and building workflows for answering causal queries. And that discovery and how it can open up what we call in the paper a new frontier for causal analysis and causal research.

8:49So I think what you're saying there is you're kind of curious about how LLMs would do at tackling these causality, these causal problems. You found existing benchmarks in the causal reasoning community and you kind of threw LLMs at them. and they did well, comma, but dot, dot, dot. And so you've got some open questions there and you found some areas where they didn't do very well, if I'm hearing that correctly and reading, inferring that from the paper. If not as standalone causal reasoners, they're good tools for complementing human causality, you know, researchers, modelers, analysts, whatever.

9:32So dig into that, you know, but dot, dot, dot part. They apparently do well on all these causality benchmarks, but you were still left with questions like, what did they do well at? And what were the questions that you were left with? What is it about those questions that you feel like you would need access to the model weights or training data sets or other things to really suss out? Well, okay. So in terms of what didn't they do well? Well, for one thing, we saw that we didn't really see performative results with respect to these benchmarks with respect to state-of-the-art baselines until we got to a certain level of model size, right?

10:17So essentially, text DaVinci 3, GPT 3.5 Turbo, GPT 4, when we started seeing comparative or better than baseline results. And so this kind of indicates, so if you saw the sparks of AGI paper from Microsoft, where we, you know, essentially one of the stories there was that things that, you know, with this movement from GPT-3 to GPT-4, we saw entirely new capabilities that were very weak or non-existent in GPT-3. It has that same flavor. Meaning GPT-2 did not have the capability to do any causal reasoning, but you're starting to see it with 3 and 4. Yeah, in other words. Or is it 3.5 and 4? 3 and 4.

11:003 and 4, okay. It was a text of NCO3, so that's kind of what instructs. That's the instruction training, fine-tuning, that's reinforcing learning of human feedback. To me, that's mysterious, right? What that means is that before these models, they were doing worse than random, right? So that's, you know, if on these benchmarks, you had a multiple choice question or you had a binary question, then there's an accuracy number that is the number that you get if you're just guessing at random. And it wasn't until post-GPT3 where we saw baseline comparative or better performance. Why is that mysterious?

11:30I think we take it for granted in a lot of contexts that the models are a lot better and there are a lot of things that they can do better and perform better on as they get bigger. My personal take on this type of thing is there's a lot of talk in our community about emergent behavior. And I agree with that. And I understand how the intention mechanism, the more data you give it, the more it can move up a hierarchy of abstractions in terms of what it's attending to. But it's mysterious. And I don't... Mysterious in the sense of we have no idea why this works. And it's kind of interesting that it does.

12:07Yeah, and that makes me a little bit queasy, particularly if we start talking about, I mean, this is causality, right? And so like one - Queasy in the, if we start asking important questions and we don't know why or how this works, it's kind of like - Right. I mean, so the machine learning community, oftentimes our metrics are kind of focused on prediction. And maybe if you're nuanced, you might define some kind of cause function on making errors. But traditionally speaking, the causal inference community is conservative in terms of what it's going to conclude. We're saying that this vaccine prevents this illness or prevents the worst cases of this illness, right?

12:42So the idea that it's... It's like, is it good performance, a correlation or causally valid? Yeah, it's just like if we're going to start deploying this thing to actually make decisions based that are involving kind of causal logic, it feels like a mysterious capability of doing well on accuracy benchmark, on an accuracy metrics, on benchmarks, which, as we talk about in the paper, are often memorized by the large language models, is not a high enough standard for that sort of thing. So let me give you an example. Let's suppose that I were to deploy a model that would ingest somebody's resume and then tell me whether or not I should hire them.

13:24By the way, nobody should do this. There's issues with this. But I suppose I was going to deploy that system and just have it automatically give an accept or reject for somebody who's applying for a job. And I could go in there and in my prompt, I could say, I can kind of give it constraints that would force it to omit certain, in a causal sense, certain factors that would drive its logic for why to hire or not hire somebody. Say, for example, I said, you may not consider gender, you may not consider ethnicity, you only may consider factors in their educational or professional background, for example.

14:00And it could generate a do train of thought reasoning, show me your logic step by step. It gives you a step by step argument for why it says you should not hire this person, right? From a causal perspective, you would want to make sure that despite the fact that it says that that's the reasoning that is deploying to arrive at this decision, how do you know that it's not just BSing you with politically correct answer when maybe after maybe given all the, say, toxic content it was trained on is still somehow biasing the answer? So in maybe a lot of scenarios, this isn't a big deal, but from a causality standpoint, we care a lot about identification.

14:37Identification, I mean, kind of provably you're able to use the data to answer a causal query. And you care a lot that the model is using your causal assumptions to arrive at the answer. Yeah. And so I think that... And so if you could look at activations and things like that, when these questions are being evaluated, you might get some hint as to how they're grounded in the model or something. Yeah, exactly. Using kind of probing procedures, if the latent representations in the model align with some kind of causal model or causal abstractions or causal assumptions. this is very fresh research i know you know we're working on this other people are working on this and so that's not in this paper but i think my personal belief is that that's the type of analysis you want the kind of experiments you want to run to understand exactly how it's reasoning if causally if it's if that's what it's doing and so to be clear in several of the types of questions you know from these benchmarks that were posed to the models like they did really well is that fair to say that they did very well compared to the state of the art and these kinds of causal reasoning tasks am i interpreting that correctly like there are some numbers even in abstract 97 13 point gain for pairwise causal discovery 92 for counterfactual reasoning 20 point gain i'm assuming that's 20 points over the prior state of the art actual causality 86 there's no gain discussed here but like that seems to be good performance i guess one of the questions that you implied earlier is it like hey these benchmarks are out on the internet did the model just read these benchmarks and it somehow knows the answer yeah so some of you mentioned that 97 so with the tubing in tubing in pairwise discovery benchmark so this is a benchmark so i mentioned altitude and temperature so what this benchmark has several examples like that from across different domains.

16:34So zoology, chemistry, geology. So it gives you a pair like that. So temperature, altitude, and then it gives you, say, a bunch of measurements of temperature and altitude from different geographical locations on Earth. And the intention in the construction of this benchmark was to say like, okay, well, let me use a statistical method that's going to look at these pair of variables and figure out which is the more likely direction that temperature causes altitude or the altitude causes temperature. So what we did in the experiment is that we threw away all the data, just took the variable names, and then posed some questions.

17:12Specifically, the prompts was something along the lines of, you know, which cause and effect relationship is more likely A causes a change in template variable B, or changing template variable B causes a change in template variable A. Let's work this out step by step to make sure that we have the right answer, provide your final answer within these tags so that we can parse it. We did not use any context learning examples, but we did give it a system prompt that said, you are a helpful assistant for causal reasoning. And that's what we did. And so one of the things we noticed right there was that chain of thought mattered.

17:46The system prompts mattered. So we saw like a five point increase for GPT-3 when we use the system, but we didn't use the system prompt versus using the system prompt. And then in terms of some of the kind of errors it made, so yeah, it got very high percentage when we looked at some of the errors that it made, it would often, in some cases, things that you read and they were coherent, but we're kind of making kind of maybe small errors in judgment. But oftentimes, we saw cases where we actually kind of agreed with the large language model in the sense that the example I'm thinking of is age is still altitude and temperature.

18:20Sure. And it gave an example where, now I'm forgetting, trying to think of an example where it made an error. Ah, okay. Ozone and radiation. So the answer was that radiation causes ozone. So the presence of radiation causes ozone. And it concluded that ozone caused radiation. But when we looked at the chain of thought, you can see that it was talking about ozone in the stratosphere, so that when the ozone layer is depleted, there's more solar radiation. But we were talking about, no, at surface level, there's radiation from some source that causes there to be ozone in the air. So oftentimes when it got it wrong, we realized that it could get it right with just a simple, a bit more clarification.

19:07And those were rare examples, but when they did happen, it kind of implied that, okay, well, again, sensitive to the prompt. And also that so far these models aren't particularly good at asking clarifying questions, which is another concern there. And so, yeah, there were some failure modes, but often it was because they were rare when they were just kind of outright wrong, at least for the causal discovery task. And when it came to some of the actual causality, causal judgment tasks, it was more often making kind of errors in logic. But still, the logic with GPT 3.5 and especially GPT-4, especially GPT-4 in a causal judgment and actual causality tasks were pretty coherent, even when they were wrong.

19:50And so you can imagine that with better prompting, you might be able to get it into the right place. Just out of curiosity, when you said you did some testing around the system prompt, and you mentioned presence versus non-presence of the system prompt, did you vary the system prompt? I'm wondering if you're able to identify whether you just said, hey, you're a helpful assistant. that got you a boost? Or was it like contextualizing that causality is what you're looking for that conditioned the results positively? We didn't do much exploration in terms of prompt optimization, right? So we just basically gave it a kind of obvious system prompt that stood out to us as for chain of thought reasoning.

20:38And particularly with some of the tasks that had to do with counterfactual reasoning, actual causality, causal judgments, there was a clear sign that better prompting would have an impact, but that was not the dimension that we explored deeply. So going back to this kind of core question, it did very well in pairwise causal discovery. There's some concern about the validity of that as a result. Like if you can separate the, hey, I don't really understand it and I want to understand it before I use it. You mentioned the possibility of memorization as one concern. Part of me says, is it really in the sense of like, that's kind of what you wanted to do.

21:23You wanted to go out and if you're talking about these broad relationships like altitude and temperature, you want it to have kind of memorized the answer to that or seen a bunch of examples to that and pull that into its reasoning. Right. You pick up on a really interesting and nuanced point, which is that we know that most large language models tasks, we have this problem of, is the large language model ingesting the benchmark and just kind of spinning back out the memorized version of the benchmark? And we want to avoid this problem of memorization. But to your point, we want some memorization, right?

22:00We want it to know what altitude is and what temperature is and have memorized those. It's not an agent acting in the world. It hasn't learned it through, can only learn it through the training data, right? So we wanted to know what those things are and that there is a causal fact that altitude affects temperature. And so we wanted to memorize that. We just don't want it to memorize specifically what's in the benchmark itself. And so - What does that even mean? And how do you do that for broad facts like altitude and temperature? Well, so first thing we did is to test whether or not the benchmark was in the training data.

22:37And to do that, we did a kind of text completion task that basically tried to complete elements of the benchmark. So we put in the readme and the kind of all the background from the benchmark and then put in a specific row, like the first two elements of a row in the tabular data in the benchmark and then asked it and basically try to see if it could auto-complete the rest of the row. And for the tubing in data, it did. It did that for about 20 % of the examples. And in terms of just predicting the next cell in the row, it got about 60 % accuracy in predicting the next cell. So clearly the tubing in data was in the benchmark.

23:16And so we had to think about, okay, what does that mean now? Now, what are we supposed to do with this in terms of deploying it? And so, well, the actual problem that we care about is to what extent does it generalize beyond the benchmark? And so we kind of partitioned the task of answering the question into two parts. or say two kind of conditional probabilities, or two probabilities. So one being the probability that the causal fact is encoded in some sense in the large language model, or in other words, it's a large language model has learned this causal fact. And then the second being that conditional and having learned this causal fact, to what extent can it answer your specific question given the context, given your prompt in a way that transforms that causal fact into part of that answer.

23:58And the benchmarks are good at evaluating the second, but we can't know generally if it's doing well, to what extent it's happening on the first. We can't know, we can't evaluate how well a large language model has learned some causal facts about the world, just how well it can answer a question based on the fact that it has learned it. I mean, that's always going to be kind of confounded with its ability to kind of BS you and hallucinate. So kind of the conclusion there is like, okay, these benchmarks themselves don't tell us the full picture. And then so also we want to know how well we can generalize.

24:32I mean, the whole point of the benchmark is like, can it generalize to tasks that are like the examples in this benchmark? And so we explored some ways around this problem. Can you give some concrete examples of that generalization in the case of related to the Tübingen examples you've already given? Like, are you asking it to answer or kind of reason about different word problems that relate the facts around altitude and temperature, for example? Or is it some other kind of generalization? So one example we talk about in the paper, again, was causal relationship between the length of a seed mollusk called the abalone and the age of an abalone.

25:17And that's the fact that the truth there was that the older the abalone gets, the longer it is. We also saw an interesting error mode there where if we changed length to diameter, it actually got that wrong. So it's an interesting example for how it's not quite doing whatever it's doing. It's kind of off in the sense that it's... It's kind of more like memory than understanding. Yeah, right? So changing the word to what's effectively in this context a synonym, it gets the answer wrong. And what you're hoping there in terms of generalization is that it could generalize to other causal relationships, causal facts in the area of zoology.

25:54If something similar would be like the number of rings in the stump of a tree and the age of a tree. for example, which is not in the data, right? And so you want to make sure that, and so that's what I mean by generalization. Okay. We started exploring ways, are there ways that we can get around this? Let's say, for example, could we prompt it with some sort of inductive bias that would help it generalize? So in machine learning, we often talk about inductive biases in terms of the syntax of the architecture, right? So that's, say, for example, convolutions and max pooling give you translation and variance, right?

26:26Coswell inference, we're often getting our inductive biases from explicit model assumptions that we formalize into a model. So for example, a causal graph, functional assumptions like in a structural causal model. The interesting things with the large language model is it allowed you to kind of use inductive biases that were much easier to use in the form of a natural language. So like, for example, Occam's razor. So we ran an experiment where we said, you know, we asked the large language model with the pairwise discovery to say like, okay, give us the best argument you can for the best argument for why A causes B.

26:58Now give us the best argument that you can for why B causes A. And just try to make this argument as kind of coherent, well-reasons as possible. And then in a third prompt, we said, okay, look at these two arguments that it just generated. We didn't tell it to the just, but these two arguments that were just generated and tell us which argument is preferred based on Occam's razor. And this is based on kind of research and psychology that suggests that the causal explanation that introduces the fewest external factors to explain something is preferred, at least for humans. And we saw that's the wrong direction to come up with a good argument, how to kind of add stuff.

27:37So say, for example, with the Abalones age and length, it said that, well, longer Abalones could be much more competitive in a resource restricted environment and getting food and they could kind of crowd out all the other Abalones and therefore live longer. right and so like that's the very reason well reasoned and plausible arguments but it inquired introducing kind of resource constraints as an additional factor while the simple explanation that's that as things get older they get longer did not require that and so in that case we got lower up we got obviously lower accuracy than gpt4 we got about 85 accuracy there but considering the fact that we know that's the benchmark was memorized the thing that's pretty good.