Is ChatGPT Getting Worse? with James Zou - #645

0:09All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington, and today I'm joined by James So. James is an assistant professor in biomedical data science and by courtesy of computer science and electrical engineering at Stanford University. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. James, welcome to the podcast. Thanks for having me. Excited to be here. I'm super excited for our conversation. I'm looking forward to digging in with you on ChetGPT and how its behavior has changed.

0:45This is one of your recent research projects, as well as some work you've been doing around vision language modeling for medical applications. But before we dive into those topics, I'd love to have you share a little bit about your background and how you came to work in machine learning and AI. Great, yeah. So I've been working in machine learning and AI for the past 15 years. I did my PhD at Harvard, which is starting to do some machine learning. That was right around the time when deep learning was becoming more powerful, more popular. And then I came to Stanford about seven years ago to start my group.

1:18And here at Stanford, we really focus on developing machine learning AI tools. And oftentimes we develop tools with biomedical and healthcare applications. Awesome. And give us some examples of some of the tools that you've worked on. Yeah. So for example, we developed systems that can look at videos of people's beating heart and can use that to assess whether someone's at risk for heart failure or for stroke or heart diseases. So that's an algorithm that we published a couple of years ago. We actually did a big clinical trial testing how good that algorithm is when it's deployed in clinical settings.

1:53It's in the process of getting submitted and getting FDA approvals. We also develop algorithms that can help people to design clinical trials or to discover new drugs. All right. Interesting. So one of the interesting things that you stumbled upon as you were working on one of your research projects was this change in the way ChatGPT behaves. Give us a little bit about the background of that effort, what you were seeing and what led to that research. Yeah. So we were very much motivated by the fact that first, ChatGPT is becoming hugely popular. So a lot of people are using it for all sorts of applications, right, from helping them to write emails, to writing codes, or to write, do homework even.

2:40And there's also a lot of people who are reporting, you know, on social media, online, that, oh, they're seeing all sorts of changes in ChatGPT's behavior over time, like over the last few months. And we see some of that too in our studies, in our applications. For the actual study itself, for this paper, which we put online in July this year, we wanted to do this assessment systematically. And for that, we ended up picking eight different types of tasks to cover, I think, a quite diverse set of applications of people, how people are using chat GPT. So some of these tasks involve solving, doing coding.

3:15Others involve answering knowledge-based questions or knowledge retrieval questions. Others would involve answering opinion questions or solving math puzzles. So there's a bunch of these different tasks. And for each task, we would have sometimes a few hundred or sometimes over a thousand different questions for that task. And then for each of the questions, then we'll ask it to both the March version of ChatGPT and then the Tune version, and then systematically compare the results to see how is there a shift and what is the shift. And one of the challenges that users of language models like ChatGPT are having today is the difficulty of creating concrete baselines and systematic comparisons when your output is text, right?

4:04If you're baselining predictions, classifications, those are easier to evaluate, but text gets a little bit trickier. How did you go about creating an evaluation methodology for the text results? I'm imagining it had some substantial differences, but was similar if you were just to read it cursorily. Yeah, that's a really good question. And that is one of the big open challenges in how do we evaluate output of these generative AI systems like ChatGPT, where the output is very complex and very rich. So the way that we tackle this problem in our paper is that for some of the tasks, we have eight tasks.

4:43So some of the tasks, the outputs are actually simpler, yes or no, or maybe just a specific number. So I could give you an example of this. One of the questions we had was around whether we'll give ChatGPT a number, like a five-digit number, and we'll say, is this a prime number or not? So we know the ground truth, And chat GPT, maybe we'll do some sort of reasoning, but in the end, it would tell us, okay, so this number is a prime number or it's not a prime number. And then we'll just look at that final answer to provide and say, okay, is this actually correct or incorrect? And were there other question types where you took on this challenge of trying to assess the relative performance of textual output?

5:26Yeah, so for other tasks like coding, so the output of the coding is not just binary yes or no, but it's actually a piece of code. So in that case, then we would actually just copy and paste over the code produced by ChatGPT and then run that code and see is it executable, right? Does it parse, right? And if it's executable, then does it actually produce the right answer? So for coding outputs, then we can actually just try to evaluate the code. And the third kind of metric that we use to evaluate is just be in terms of the verbosity, which is a simple metric, but just looks at like how actually, how verbose is the output of ChatGPT, right?

6:03Does it actually give you an expanded response? And we actually see a huge difference. Sometimes the verbosity of the response has changed a lot over the last few months. Got it. So I don't know if this is exactly right to characterize the things that you were looking to compare as quantitative relative to qualitative, But if those are more of a spectrum and qualitative, like very qualitative analysis of blocks of produced text is at one end, you didn't really go that far. It was more looking at things that you can get more concrete comparisons around numerical outputs, code blocks that you can run, things like that.

6:43That's correct. Yes. We wanted to focus on a bit more quantitative outputs, metrics. for the reasons that we want to do this at a large scale, to do this across thousands and sometimes many thousands of questions. And the second is that the quantitative outputs are a bit more objective to measure. So that's why for this initial assessment, we focused on these eight different tasks where we have more quantitative metrics. Okay. And so given that we're talking about two different versions of ChatGPT, it's not really surprising that there would be some behavior change. And I think the hope would be that the later version is kind of unilaterally better than the former version.

7:26Tell us about what you actually found comparing these two versions. Yeah, so this is where it's actually quite surprising to us. We were quite surprised to find that for some of the tasks, the later version in June did do better than the version in March. But there were actually quite a lot of tasks where the later version was actually substantially worse than compared to the earlier version. So these are even some seemingly simpler, relatively simple tasks. Like example I mentioned, you give it a number and you say, is it a prime number or not? So we'll give the same numbers to JAT-GPT in March and June.

8:00In March, we actually get most of these numbers correctly. So most of the time, we'll be able to tell whether it's the prime number and this is not a prime number, actually. But in June, if you give the same numbers, actually, this performance has dropped quite a bit, often by 30 or 40%. Oh, wow. And I think there's actually a really interesting reason for this in that if you think about, I guess, how do I figure out if the number is a prime or not, right? So if you ask ChatGPT to do this chain of thought reasoning, it would have a very reasonable chain of thoughts of, okay, so here's a number.

8:33I'm going to find all the smaller numbers and try to divide each of the small numbers into my original query number, right? If it divides into it, then it's not a prime by definition. So that's a very reasonable way of solving that problem. That's also how humans would solve it. So when we asked Chachipi to do this chain of thought reasoning in March, that was actually very effective. We will find the right strategy for solving this problem, and it gets a very good accuracy. But in June, when we asked it to use chain of thought reasoning, then it seems like chain of thought no longer works very well for this task in June.

9:05So it would either not be willing to use chain of thought reasoning for specific numbers, or it would just attempt to do chain of thought reasoning, but not be able to execute the different steps correctly. And why do you think that is? I mean, chain of thought has become a very popular way to get GPT to do more complex tasks. You would hope that over time it gets better at doing chain of thought. So that's where it becomes really interesting and also quite mysterious, right? So I agree with you that it's one of the most common strategies to improve the performance of these models, especially on logical tasks, as to ask you to do chain of thoughts.

9:40It's a very popular prompt strategy. But what we found is that actually this popular problem strategy actually does not work as well in June compared to before. Did you compare performance for other tasks using chain of thought beyond the primes? Or was the prime kind of your proxy for the way it did chain of thought? Yeah, so we also give it other puzzles. So other math problems. Instead of looking for primes, you can look for other kinds of numbers with other mathematical properties. And we saw a very similar pattern of that was chat GPT. So obviously GPT-4, the performance was much better in March compared to June for these other math problems as well.

10:21And similarly, the effectiveness of chain of thought reasoning was actually much higher in March compared to in June. So it got much worse in chain of thoughts for these other math problems in June as well. And was the behavior consistent between chat GPT-3, 5, and 4? When you saw a change, was it the same direction for both models or did it even vary from model to model, whether it got better or worse on a given question type? Yeah, so that's actually where it becomes even more interesting and mysterious is that it's often the trend of behavior change is divergent for GPT 3.5 versus for GPT 4.

11:02So everything I was describing was for the latest version of GPT, the GPT-4. That's where we see the performance and chain of thought got worse in June compared to March. We also asked GPT-3.5 the exact same math problems and actually GPT-3.5 did better in June compared to March. So there it was actually doing better. And then chain of thought reasoning was also working better, was also working well for GPT-3.5. So this actually highlights this additional complexity is that if you look at the same model, a GPT-4, right, it's quote unquote the same model, but its behavior can change over time. But if you look at its sibling, right, GPT-3.5, that change of behavior of 3.5 can be very different from the change of behavior for the version 4, right, even though these are from the same family.

11:49Yeah. And you mentioned, you kind of alluded to the mystery of it all and this underlying lack of transparency that we have because it's a proprietary model. But that said, and we can return to that. Did you develop any intuition about what might be happening and what might be causing performance decreases for these models? Yeah, so one conjecture that we have based on some of the other experiments that we've done, this idea of what I like to call neuropleiotropy, right? So pleiotropy is the idea that if I change the behavior for one type of task for a system, it could be AI or for human, that can have unintended side effects that can change its behavior for other tasks, even though those might seem to be orthogonal to what I'm doing.

12:43So we've done some experiments where we would take these large language models and then do additional instruction fine-tuning to improve the safety of these models. by safety here I mean make it less make the models less likely to respond to questions like how do I steal someone's credit card or how do I hurt some people and we do the instruction fine tuning by giving it the safety demonstrations so that improves the safety of this language models that's great but we also see some really unexpected side effects of that model in that for example maybe some of these models now would if you ask it okay how do I kill weed so the model will stop would say, oh, you shouldn't kill weed because killing is bad and weed is intelligent systems.

13:28You shouldn't try to kill it. So somehow it gets better in some aspects, but then it also has some strange behavior as a consequence. Those examples don't seem quite as orthogonal as safety, instruction, tuning, and general chain of thought performance though, right? Yeah. So I think for ChatGPT, right? So instruction, chain of thought reasoning is an example of the model trying to follow instructions, right? Like the users say, okay, so do the step-by-step reasoning. And if the model, if the large language model follows that instruction well, then it would actually do the step-by-step reasoning.

14:07And oftentimes we do see this sort of a trade-off, this tension between how well the model follow instructions and how safe the model is, right? I think that makes sense, right? Because if the model really follows the instructions perfectly, then if I ask it, okay, I'll make a recipe for poison, then it's going to follow that instruction and make the recipe, right? But when I do the safety training, I'm asking the model to not follow my instructions perfectly, right? Or at least not always. So then that can have side effects. Got it. So what you're really describing is like this very fundamental trade-off between kind of control from a safety perspective and control from a instruction following perspective.

14:50And in lots of ways, they're orthogonal, but like they're opposing perspectives or opposing forces on the model's behavior. Yeah, they could have competing objectives. So we don't know if that's the causal mechanism, but that seems like that could be one thing that happens. And the other evidence where we see of this is when we asked ChatGPT these opinion questions. And what we did there is that we actually just got public opinion surveys that was done in the US. So these are questions like, so what do you think will happen in the US in 50 years? Or do you think US will still be a dominant country in 20 years?

15:26So not very sensitive questions. These are things that people like you and I would take. And what we found is that if you ask GPT for that question in March, most of the time we'll just give you some answer or tell you what it thinks. but if you ask it in June it actually stops a lot of equivocating right yeah it just stops stop engaging it doesn't even respond to the questions it'll say oh these are subjective questions it's an AI system I don't have opinions maybe you should go find an economist or something like that I've seen behavior like that yeah yeah ask your friend I think that's also a potential symptom right of the model less willing to follow instructions but also less willing to engage And so do you have, you know, has your research suggested any approaches that, you know, an interested party like OpenAI might be taking to try to kind of decouple this goal of safety from the performance from a control perspective?

16:28There's still an open question about how do we really decouple safety and control and decouple all these different competing objectives. Because one of the challenges for a system like large language model is that they are extremely powerful, and they can do so many different tasks. So it's not a simple classifier like a few years ago, but it's actually it can do many different kind of tasks. And because of this player-tropy effect, so if I fine-tune the model to improve its performance on one task, that could actually change and hurt its performance on these other tasks that might seem maybe a little bit unrelated.

17:04it. So I think a really interesting and important direction of research that we and also our colleagues are exploring is on how do we do more precise and surgical edits to these large language models. And by surgical edit, I mean, how do I maybe debug the model on some problems without actually changing its behavior in all these other dimensions, right? Because the current way of doing these instruction fine-tuning or updating the gradients to update the weights of the model, It's actually changing all these billions of parameters at the same time. That could have all these potential side effects.

17:39And when you describe these methods, we're talking about instruction fine-tuning or fine-tuning more broadly. The mechanism for asserting control over the model is through giving it additional data that's curated to produce the direct desired outcomes. when I hear you describe something as more surgical, what comes to mind is, you know, going and fiddling with weights or layers or connections or things like that. Is that the kind of thing that you're describing or is it more akin to the way we manipulate the model today? Yeah, I think it's somewhere in between. The current way that we manipulate and train these models is, as you said, we give it some demonstrations, like instructions or human feedback, right?

18:24And then we use those demonstrations to fine-tune the parameters of the model. And when I do this fine-tuning, maybe with updating it based on gradient descent, then I'm updating potentially all of the parameters, billions and billions of parameters. And when I say the more surgical updates, it would be like, okay, so can we identify specific circuits? And by circuit, I mean maybe a subset of the layers or neurons, artificial neurons in this model that control specific behaviors. Maybe there are some circuits that affects the chain of thought reasoning or some circuits that are more specifically responsible for safety or for specific knowledge.

19:05And if I want to modify or debug that particular circuit, then I just go in and try to modify that subset of the model without having to necessarily change the entire model. We throw around this idea that these LLMs are amazing. It's incredible that they actually work the way that they work. And we don't really have any idea why they work. Is it your sense that we're starting to gain enough understanding around the mechanisms of how they work that we can exert this kind of surgical control that you're suggesting? I think that they are quite amazing systems, incredibly complex systems, and they have amazing capabilities.

19:49And we're just still scratching the tip of the iceberg, right, of all the different things they can do. I think these kind of surgical edits would greatly improve the transparency and the control that we have over the models. Because right now, we essentially have very coarse control over if I want to tune the model, then I change all the parameters, which is very coarse. We're not quite there yet, But I think our vision would be something akin to, analogous to CRISPR, right, for the human genome, right? Interesting analogy, yeah. Yeah, with gene editing, you can actually say, okay, if I have a particular disease I want to fix, now it's find that mutation and go into the human genome and fix that mutation without having side effects of introducing other mutations in the genome.

20:32So that's sort of the holy grail of medicine, precision medicine, and that's where we'd like to get to with AI as well, right? So right now we're sort of have these very blunt hammers. We're making all sorts of mutations in the genome of the AI system. But hopefully with these more surgical edits that the folks are working on, then we can make much more precise modifications and improvements. Yeah, I'm curious just to push that analogy further. Clearly the human biology and the genome in particular is incredibly complex, but we've also spent a lot of time studying it and understand quite a bit of it, have come to understand quite a bit of it over time.

21:11I'm wondering how you would compare our understanding of LLMs with our understanding of the genome or the mechanism that CRISPR operates on. I think that's an interesting question, especially since I spend a lot of time both studying the biology, studying the biological systems and genomics, and also we study a lot of the AI systems. I think the biological systems will have much better causal and mechanistic understandings. We don't understand how everything works, but for many of the genes, we know that if you have this mutation in this gene, this increases your risk for breast cancer. If you have this other mutation in this other gene, it increases your risk for lung cancer.

21:50So that we have these causal understandings of. I think we would very much like to get to that level of understanding the mechanistic understandings with large language models as well. We're not quite there yet, but we're starting to get there by saying, okay, what if I take this transformer and then delete this module in the network, right? How does that change the behavior of the transformer and change the behavior of the language model, right? Some of my students are starting to do this kind of experiments. And then I think that will help us to get more insights and more control. It sounds not unlike some of the early experiments in the human genome project.

22:25Yeah, I think it is quite analogous. With human genome, we also have to do a lot of these perturbations, right? You do a large CRISPR screen to figure out what's going on. Similarly, with the artificial genome of the language models, we also want to introduce a bunch of artificial perturbations and silico perturbations and see that can give us insights into how does the model behave. I also just mentioned that we've also studied behavioral changes in other AI systems prior to language models. So before language models, we actually spent three years monitoring the performance of computer vision systems.

22:58If you look at the computer vision algorithms provided by Google or Microsoft, then we can track how those algorithms change over time. And it's actually interesting that those algorithms also change over time. Sometimes they get better, but on other kinds of data, they can get worse. But I will just say that what we're seeing for large language models like ChatGPT is that first, the size of the change and also the speed of the change is orders of magnitude larger and faster compared to the kinds of drifts and changes that we saw on the previous generation of AI systems. Like the kinds of changes we saw previously over three years, we're seeing now like over less than three months.

23:35And so this continuing with that theme for a moment, what do you see as the implication for developers, engineers, researchers who are trying to use these models? You know, certainly there are maybe things that we need to kind of continue to push open AI for from a transparency perspective. I'm wondering if there are specific things you have in mind there or things that you call out in the paper. But it also calls to mind for me maybe additional tooling requirements to track how these models that we're depending on are using as we build on them. How do you think about the things that are needed in light of the confirmation that your study provides around how these models can shift under our feet?

24:22Yeah, I agree with you. I think this really highlights the need to have more tools first to monitor and to continuously monitor over time, right? The behavior and performance of large language models, right? And I think we need to have these more monitoring tools that can be run like every week, right? That can really assess the behavior change of these models. And in addition to monitoring, I think we also need to build the rest of our stacks, right? If you have a software stack that uses ChatGPT as a component, then we need to sort of robustify the rest of the stacks to be more robust to these formatting and behavioral changes in GPT.

24:59So one day the model is doing a good job returning properly formatted JSON around some requests that you give it. You can't expect that to continue. Yeah, so that could change tomorrow, right? And you need to have a sort of a safety mechanism in place in your software stack. Yeah, it calls to mind some of the conversations around security, like defensive programming, that kind of thing. Yeah, sort of like a new generation of robust programming, program design, right? When we're integrating language models. Mm-hmm. And with regards to monitoring, do you think there's a role for both kind of global monitoring, like making your research an online system so that we can all go see the GPT weather report and see how it's performing today, or very specific application-oriented problem?

25:54I'm assuming if you're going to depend on this, you're going to want the latter. You're going to want to know how it's performing with regards to the very specific things that you're depending on. But do you see a role or a need for kind of broader kind of global monitoring of the way the systems are performing? Yeah, it's a great question. And I think both kinds of monitorings would be critical. We are planning to continue to provide evaluations and updates, right? We're going to continue to monitor TPT and also other models like BART and from other companies. And We're going to provide that kind of, as you mentioned, like weather reports, right?

26:30So make that public resource so you can see how does these global behaviors of these large language models change over time. And I would also recommend that if you are a specific company and using these models for your own applications, it could be a finance company or a service company, then you want to design some task-specific monitorings based on your application profiles. And then you can run internally to a continuous to assess the performance and behavior of that model for your task. Awesome. I also did want to talk about some other work that you and your team have done recently focused on visual language models for image analysis.