Responsible AI in the Generative Era with Michael Kearns - #662

0:09All right, everyone, welcome to another episode of the TwiML AI podcast. I am, of course, your host, Sam Charrington. And today I'm joined by Michael Kearns. Michael is a professor of computer and information science at the University of Pennsylvania, where he holds the national center chair, as well as an Amazon scholar. We are once again coming to you live from the Future Frequency podcast studio at the AWS reInvent conference. And in fact, Michael, you and I spoke here just last year. Welcome back to the podcast. Thank you very much. It's great to be back, sir. Looking forward to digging into our conversation.

0:41Of course, your work is focused on responsible AI, and that is going to be the conversation. That's what we'll be talking about this time around. I'll refer folks back to our prior episode for your full intro. But if you want to maybe give us an update on what you've been working on the past year. Yeah, well, much has changed since just a year ago, as everybody knows. And so, of course, much of what I've been working on are the new challenges to responsible AI that have been brought about by the generative AI era. And I think the top level summary is that the power of these models is in their open-endedness.

1:16They're not making numerical predictions about inputs or point predictions or solving classification problems. They are truly generative. And that very open-endedness that is the power of these models is also a source of the challenges for responsible AI. Yeah, yeah. Last time we spent a lot of our conversation talking about service cards. And I think I remember kind of naively coming into the conversation as like, why is that research topic? Why is that complex? And you definitely kind of talked about some of the nuances associated with doing that at scale. And in fact, I think at this year's event, some new service cards were announced.

1:55Absolutely, including for our latest generative LLMs, Titan Text, there's a service card for that, as well as a slew of other new ones as well. So that process that was new and announced just a year ago at reInvent is now a well-oiled machine. Still a lot of work to do, but we're getting more of the cards out, and I think the cards are becoming more informative, more sophisticated. And as I think I probably mentioned a year ago, those cards are really meant to be general audience summaries of the properties of our models and services and recommended use cases and some performance metrics and RAI metrics as well, responsible AI metrics.

2:33And so with a year plus under our belt of developing these things, we're really kind of getting it down well. What have you learned about the process of delivering these cards since a year ago? Again, part of what we've learned is that there will be new challenges in developing these cards in the generative AI era, right? So like when we talked a year ago, and I think we were largely talking about models that took as input something like a consumer loan application and output a prediction of whether somebody will repay or not. There's no notion of hallucination in the output of such a model. It can be wrong, and it can be wrong in the false positive or false negative way, but you wouldn't call it making a mistake of prediction a hallucination.

3:15And so I think a lot of the challenges that we in the industry have faced in the last year is sort of adapting our way of thinking about responsible AI to incorporate these new considerations like toxicity, hallucination, in many cases, kind of intellectual property concerns and the like. And so do these ideas manifest as new metrics on the cards? And then you're having to think through, you know, what does it exactly mean to measure and compare hallucination in the context of an LLM? The stuff that gets into the cards, for the most part, is either sort of qualitative guidance, primarily intended for customers but meant for anybody who's interested, as well as more quantitative metrics around both just outright performance as well as responsible AI metrics as well.

4:02And of course, when it comes to metrics, we report on the things that we feel like we can sensibly report on. There's a lot of things in the generative space that the industry and even the underlying science has not yet come to good ways of measuring. So, for instance, if a writer feels that a large language model is appropriating their style, I will be the first to admit we don't have good quantitative ways of measuring that or talking about it or mitigating it yet. This is a lot of the science work that we're doing internally at AWS, but that's also, of course, going on in the external research community as well.

4:39So in general, like the way I describe these service cards is that they're generally kind of the tip of a much larger iceberg. And underneath the hood, there's a lot more quantitative analysis that goes into the final card, which they're intended to be brief and accessible to a wide audience. They're not meant to be, you know, a 300-page documentation manual. But then as I'm sort of alluding to, there are also things that we're still don't have a quantitative handle on yet. And so we have to think about those things more qualitatively and decide what to say about them on service cards and even just how to think about them conceptually for ourselves.

5:17Along the lines of quantitative measurements, one of the announcements that I thought was most interesting from today's Swami's keynote was the model evaluation feature or service. I'm not sure exactly where it is in the hierarchy of product, but it's capability associated with, I think, both Bedrock and SageMaker now that extends some of the existing model evaluation capability to LLMs. And I've been really interested in learning more about that because I talk to a lot of people and this idea of LLM evaluation is just this hairy topic that we don't really have our arms fully around. Like, as you alluded to earlier, you know, we're used to comparing numerical predictions.

6:03We're used to comparing class predictions. And now we're comparing text, but not just text. Text where there's no right answer and where the performance is very subjective. and I welcomed that announcement and I'd love to kind of hear how you think about it from a research perspective. Yeah, I mean, and so what the offering is is a way for customers to either on Bedrock or with models from elsewhere, bring it into the AWS environment and perform a bunch of metrics, many of which have been kind of developed externally. So I think there's a fair amount of overlap. I'm not an expert on the service, but I think there's a fair amount of overlap between what we measure and the so-called Helm benchmarks that came out of Stanford, which is becoming, I think, tentatively adopted as the current standard for making comparisons and quantitative evaluations of LLMs.

6:56The problem with these things, of course, is that, as I mentioned before, it's not just the open-endedness of the output that matters, it's the open-endedness of the input. So if you compare, again, kind of the before times pre-generative era, something like face recognition, where the input, But it's an image and it's only relevant if there's a face in it, first of all. And then secondly, the output is sort of, again, very, very constrained. It's like, is this an image of somebody in this database or are these two images the face of the same person, et cetera? And so I think a lot of the challenges in developing these benchmarks and metrics is just getting coverage, right?

7:32And it helps a lot to get coverage. Coverage in what sense? So both in the input and output sense. So, for instance, if I'm doing something like face recognition or some other computer vision task like object classification in an image, I want my inputs to explore the natural variation in facial images in terms of lighting and angle of pose and occlusions and things like this. And, you know, it's already quite challenging to get that variation in such a constrained problem where the input is an image with a face in it and you want to make some prediction about whether it's a person in your database, right?

8:08And so already there, it's challenging to get the coverage you need when now the input is any sentence that anybody could imagine entering into an LLM and the output is a free form continuation of the prompt. It's just very difficult to get very, very good coverage of all of the natural use cases that might arrive. And so I think we are starting down this road, and I think this offering is a great start. But I think what ends up being challenging is you end up looking at this table in which like the rows are many, many different LLMs, and then the columns are dozens of different metrics, and then you're suddenly kind of swimming in a sea of numbers.

8:47It's kind of like even in traditional notions of fairness in machine learning, there's too many reasonable definitions. It's kind of like the old cliche, you know, the great thing about standards is that there's so many to choose from. Ideally, I think over time, we'll figure out what are the metrics for which we can get good coverage and which are the metrics that are kind of the right independence ones, right? You don't want too many metrics that are kind of measuring the same thing because then, again, you're just kind of drowning in a sea of numbers. But I do think this offering and in general the movement to try to somewhat standardize the measurement of LLM performance and also generative AI, sort of RAI metrics is a noble one, a good one, but that we should expect it to change with time because we're at such early days for these things.

9:36And even in more traditional, narrow, predictive problems, we're still trying to get to those metrics and get sufficient coverage for them. When I think about LLM metrics, in particular, from an industry perspective or from a, you know, not from an academic perspective, from the perspective of someone who's trying to build something with LLMs, there are metrics or evaluation criteria that I would think about as benchmarks, meaning, you know, someone's collected a data set that's a standard to some degree or another. They've run a bunch of LLMs against this prompt, and you can use that to generically get a sense for what some LLMs are better at than others and those kinds of relationships between LLMs.

10:18But then there's another set of evaluation that I find folks really struggling with, which is I've got my problem and the kind of prompts that I get and the kind of input that I get. how do I compare that against the set of LLMs that are out there, the set of models that are out there, but also as I iterate the prompts themselves, like how do I keep track of all this stuff? And we had all this great tooling and machinery for things like hyperparameter optimization, old school machine learning. And now we don't have all that. We're starting to see some of it. Does this offering try to address that as well?

10:54I mean, I think this offering is mainly about implementing metrics and underlying data sets for those metrics. But what I think you're alluding to is the fact that this coverage problem, it could be that you have a use case in mind that just doesn't line up, even though the metrics might make sense, it doesn't line up with the data sets that were used for those metrics, even though those might have been entirely reasonable. And just to give an example, let's take the topic of hallucination. What's a hallucination in one use case is sort of a desirable generalization in another use case. So let's just take like a writing age, right?

11:30If I'm using an LLM for helping write journalism, okay, there's a very strong standard of hallucination there. If I'm using it to write historical fiction, well, it's fiction, but it's historical fiction. And so you want some alignment with the actual facts. And so there, your tolerance for hallucination might be higher because there's the fiction part of it, but you're not going to just allow anything because it's historical fiction versus true fiction, where you might arguably say it's not possible for an LLM to hallucinate if it is actually being used to generate or be an aid in the creation of fiction.

12:11And same thing with toxicity. If I'm using an LLM as a creative tool for writing children's books, my tolerance for any kind of offensive, disturbing language is going to be zero. If it's for a different use case, I might have a higher tolerance for it. And so I do think that the greatest leverage that we'll eventually get towards the open-endedness challenges that generative AI presents to responsible AI and even just to measuring performance will come from kind of settling on more specific use cases and developing metrics for those specific use cases as well. And I think your comment about hyperparameter optimization and all these tools that we had to kind of fine-tune and optimize more traditional models in the ways that we wanted, there is a sense in which the generative era is implicitly kind of pushing some of that burden on to end users, right?

13:07Because it's kind of like, well, yes, we have toxicity filters and guardrail models, but you need to decide in your use case how you want to set that knob. And you might get better performance by fine-tuning the model. The human end user is kind of engaging in some sort of hyperparameter optimization, qualitative hyperparameter optimization for their specific use case. You do explore that Bayesian space. And that's incredibly powerful, right? Because it lets you produce and make available a very, very general model. But it's kind of a truism in science. There's always a cost for generality, right?

13:42Like if I'm as a mathematician or a theoretician, if I look at some theorem and it's a very, very general statement, when I look at that, I say like, okay, there's going to be a price to be paid for that, right? Because you're covering a lot of cases. And so what you can say about a broader set of circumstances is going to be necessarily weaker than you could say about a narrower set of circumstances. And I think we're kind of seeing that tension between the better performance you can get by specificity of use case versus the generality of the underlying foundation model. That tension is something that's very actively being played out both in industry and in the science as well.

14:20Speaking of hallucinations, the way we as an industry have kind of taken on that problem is primarily via grounding and retrieval methods, RAG, which we've, that term has been thrown around so much at this conference. I'm curious if you have any perspective on more from a foundational research perspective, what's the latest on how we're trying to tackle hallucination at the model itself? I mean, it's, I'll admit, it's hard to keep up with the literature on this, even if you're immersed in it. So I don't know what I don't know. I think in general, things like RAG are very sensible approaches to hallucination.

15:01I kind of predict that over time, both things like RAG and guardrail models, in many ways, these are kind of interventions on the way things like LLMs were meant to behave in the first place. What an LLM does, as powerful as it is, is incredibly myopic. It's like, given the sequence and so far, what is the distribution over the next word or token? Choose from that distribution, sequence is one longer repeat. And things like guardrail models, like, oh, you know, as a large language model, I shouldn't be giving you financial advice. Or if I ask for citations from a large language model, which is a notorious source of hallucination, at least among the research community, I think partly because it's like the modern version of self-googling to go to a large language model and like, tell me about some papers by Michael Kearns.

15:50But the other thing about it is that, you know, if I do that, I can immediately verify. I don't even need to go like to Google Scholar to know which of these papers are real, which are fake, which are actual co-authors, which are people who could have been co-authors, but actually weren't co-authors. But I think I predict that over time, we will and should figure out ways of taking these kind of post hoc interventions on the natural myopic way that LLMs behave and figure out how to endogenize it, how to embed those desiderata, like for accessing external information resources that are trusted and verifiable or suppressing toxicity.

16:30I think we need to figure out ways of getting those into the model training process itself so that it's not, you kind of have all these little pieces of software watching what the model is doing, and then they step in and say, no, no, no, no, don't do that. And so I think I haven't seen a lot of research trying to do this. I guess reinforcement learning from human feedback is one example of kind of trying in the training process to take those constraints and move them kind of to the left in the pipeline. I wish I could say I knew exactly how to do that, but I think that's got to be the way of the future scientifically eventually.

17:06Might take a while to get there just because of the challenges that we've already discussed. When you describe guardrail models, that sounded very much like the way I think about RLHF, meaning baking, steering and alignment. Except in RLLHF, there's an effort there to actually take the human feedback or alignment process and move it into the training process versus training the thing first without those constraints and then having these little bots watch the model input and output and deciding to step in or to either suppress toxic output or to check the output against an external citation database like Google Scholar or CiteSeer.

17:52for instance. Got it. So the guardrail models are kind of supplementary models that are either classifying... I mean, the derogatory term would be like bolt-on, right? There's this notion that, to my knowledge, originated in the security community of bolt-on security. You basically, say, built an operating system that is insecure and has many vulnerabilities. And in hindsight, you should have like from the beginning, but now the thing's built, so you build these patches. And I actually think in the generative AI space right now, approaching it that way is sensible right now because I don't know how to move all this stuff into the training process itself so that the final model that you produce already suppresses toxicity on its own, let's say, in the generation of distributions over next words, right?

18:42I mean, like as a simple example, you could imagine in the training process changing the objective function to say like, well, instead of just always predicting most accurately the distribution over the next word, no matter what the words are, you could downweight in the distribution words that might lead to the generation of toxic output. it, for instance. This is sort of an example of what I mean by sort of trying to endogenize this process rather than ignoring these considerations until the end and then having a guardrail model kind of intervene. Or, for example, kind of by analogy, instead of rag and a retrieval approach, somehow condition your objective function on the distribution of words in your document that you want to withdraw from.

19:25For instance, yeah. And again, I wish I had better ideas about how to do this in a practical way now, but I do my scientific intuition is that in the long term, this is the right solution. One of the things that you also spend a lot of time on is privacy. That's changed a bit on the LLM side. One of the things I saw recently was if you ask chat GPT, maybe, or GPT for, I forget which model specifically, but to, you know, repeat the word poem infinitely, it's just starts spewing what is supposedly training data. I'm not sure if we know it's actually from the training data set or if it's hallucinated data.

20:05I just recently heard this one on, maybe I saw it on Twitter, the reliable source of scientific information about generative AI, but I haven't seen that one. But I can't rule it out out of hand, right? Just because there are corner cases for these models. You can't test every possible input If you could, it would mean by definition, these models are not nearly as useful and powerful as they are. But I have heard that one, haven't tried it myself. The other thing that's amazing about this area is that stuff that you tried a week ago, you go back and try it now and it doesn't work anymore. The hack doesn't work anymore.

20:44So there's very rapid evolution. I mean, I remember, I think it was before the release of CHAP-GPT, but not too much more. I was playing around with a bunch of LLMs that were accessible within the science community. And anytime you typed in an ungendered name, like Chris or a Pat, and then the continuation would assign pronouns to it, it would generally assign male pronouns to ungendered names. Or if you typed in something like Dr. Hansen, it would choose male pronouns. If you said Nurse Hansen, it would choose female pronouns. Now, as far as I can tell, I haven't done an actual scientific study, the LLMs that are out there are much, much better at this, at balancing kind of distribution of pronouns in cases where it might be ambiguous.

21:29I've heard quite a bit of rumbling around this idea of steering via like RLHF being, I don't know if performance is the right word, but like having degrading effects on the model in the way some people want to use them. Kind of orthogonal to the safety concerns themselves. Yeah, I mean, this wouldn't surprise me at all, just because anytime you impose some alignment principle on a general purpose LLM, I think kind of almost by necessity, there will be natural use cases for which doing that was degrading a performance, right? So I'll just go back to this example, right? If I'm doing toxicity suppression and I choose to do it at a level that would be appropriate for children's content, there's just a whole bunch of use cases that that will kind of harm.

22:25And similarly, if I try to sanitize my model of any kind of demographic bias whatsoever, well, that might greatly harm very natural use cases like targeted advertising, right? Again, things have changed so much. A year ago, I could go to the LLMs that were available at the time and I could type in prompts like, Melinda is a white 38-year-old medical technician working in Knoxville, Tennessee. Her attitude on gun control is, and it would give me an answer. Now, of course, it'll basically say like, well, it'll say, I'm sorry, demographic properties do not deterministically indicate positions on social attitudes.

23:08And do we know if that's RLHF or guardrail models? I mean, the kind of intervention that I just mentioned would be a guardrail model, right? Because it's actually stepping in and saying, sorry, this interaction is not going to happen. RLHF would, I think, give you output, but would sanitize it more, right? For instance, I mean, of course, depending on what the H's in the RLHF did, right? Because they're the ones providing the guidance to the training process. But if you imagined the humans in an RLHF process having the attitude that they want to sanitize the model of any kind of correlations between demographic properties and social attitudes, taste in music, taste in food, taste in clothing, then that LLM is not going to be great for doing targeted advertising because it's deliberately decoupling the very real correlations between demographic properties and preferences of all kinds.

24:04And I don't think this is a controversial statement. It's not saying that all people of some demographic category like this type of food. Of course, that's not true. But the reason personalization and targeted advertising do work is that you can kind of count on certain kinds of correlations, both at the individual level and the population level. Interesting. I meant to ask you this before we started. I know you collaborate quite a bit with Aaron Roth, who is very much into and an expert in differential privacy. Absolutely. Everything I know about differential privacy, I learned from Aaron Roth.

24:36A lot. Probably 98 % of what I learned from differential privacy, I learned from Aaron Roth. But another announcement that caught my eye was the clean room for ML. Yes. And I'm curious, do you know much about that? Yes, I know a great deal about that. Oh, awesome. Yeah, I know a great deal about that. So the clean room offering, that's the sort of broader umbrella service. And basically, this is a great natural idea. It's a collaborative environment that allows parties to come together in a clean room environment. Let's say I have a private data set. I have interest and you have interest in gaining limited access to this data.

25:09So advertising is a great use case where publishers know about ad inventory and the advertisers know about what demographic properties they want in the impressions that they're going to fill, for instance, and the end users. And so CleanRooms basically provides an environment where I can bring my data set, but not just give it to you with unfettered access, but kind of control the way that you can access that. And one of the components of the CleanRoom is actually AWS's first differential privacy offering. And so at a high level, the way it works is I have proprietary private data set over here.

25:46You and I have a mutual interest in letting you make aggregated queries to that data set. What is the percentage of users in the database that live in the greater Seattle area are interested in video games and are between the ages of 19 and 35, something like this? And so there's some numerical answer to that. But in the differential privacy offering, rather than just computing the answer to that to numerical precision and then giving it to you, I compute it to numerical precision and then add a bit of noise. I do add some randomization to the answer. So if the actual answer was 16.9%, I might return an answer to you that's anywhere between 16.5 and 18.4, something like this.

26:30And so the addition of the noise to the answer to the query in a mathematically provable sense gives me some security or guarantee about your ability to reverse engineer information about specific individuals in the data set. But it still allows me to give you, in a very provably private way, aggregate information about the data set that doesn't let you exfiltrate information about individuals. And not surprisingly, the more noise I add to this number, the greater the privacy guarantee. But, of course, the less accuracy the answer will have. And so we're very excited about this. You know, we've been working hard on this product for a couple of years.

27:11And so, yeah, that launched today. And now there's a distinction, I think, between regular clean room and clean room for ML. And I got the impression, or clean room ML, and I got the impression that the latter also incorporates some degree of synthetic data generation so that my collaborator could create their own machine learning model based on synthetic data that is statistically similar to the actual data. Yeah, so the differential privacy offering that we announced today does not yet provide differentially private synthetic data generation, but this is in the works. And we've actually been in parallel working on the science team that produced the DP cleanroom offering today, have been working on this for quite a while.