How Microsoft Scales Testing and Safety for Generative AI with Sarah Bird - #691

0:00So first of all, you want to start with a system that has defense in depth, has sort of layers built in by design, because each of the technologies have their own strengths and weaknesses, right? So I think it's often kind of the pictures of like stacking layers of Swiss cheese. So there's no hole that goes all the way through. And so to do that, we use.

0:33All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. And today I'm joined by Sarah Bird. Sarah is Chief Product Officer of Responsible AI with Microsoft. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Sarah, welcome back to the podcast. Yeah, thanks for having me back. I'm thrilled to be here. It's great to have you back. It is very hard to believe that it's been, what, four and a half years or so since we last spoke? Yeah, a totally different time in AI, right? It just feels like a lifetime ago.

1:10It sure does. It sure does. The last time the focus of our conversation was on operationalizing responsible AI, making it kind of concrete and practical for folks. And I think we'll be talking about a lot of the same things thematically. But as you just mentioned, it's a very different world. It's a very different context. And a lot of what organizations have to deal with now is changing as a result of the shift from quote unquote traditional AI, if we can call it that, to generative AI. So we'll be digging into all of that and what it means for enterprises in our chat today. But to get us started, I would love to have you share a bit about what you've been working on for the past four and a half years.

1:55Yeah, you know, I think to kind of what you were saying, in some ways, it's the same thing, right? How do we take principles, right? You know, fairness, transparency, accountability, reliability, safety, privacy, security, how do we actually put those into practice in AI applications, in how you build an application, how you operate an application, how you use an application? But what has completely changed is actually doing this for generative AI, because the sort of set of tools and techniques and the way we do it is different. And so we're all inventing a new toolkit, we're learning a new toolkit.

2:36And so, you know, that's really what I've spent the last couple of years doing, starting with GPT-3 and when we put it into production for GitHub Copilot and really figure out how do we build a large scale generative AI application. And so we've been going at that for a while and have really started to develop kind of new tools and techniques. But it's still the very early days. So I think, you know, there's a lot more to go do here. You know, I'd like to start the conversation from the perspective of just thinking through and talking through really risks and challenges. When you're out talking to customers as well as on your internal projects like Copilot and others, what are the risks and challenges that you face and how do you think about those?

3:29I mean, a lot of those we hear about all the time, hallucinations and jailbreaks and things like that. But do you have like a mental framework or taxonomy, I guess is a better word, for thinking about the various risks presented by generative AI? Yeah, it's a great question. And it's something that we've been working on for a while. Actually, one of the most, let's say, memorable moments in my career was when we first got GPT-4. So this was like they had just finished training the base model. There was no post-training yet or anything. And actually, the first people at Microsoft that got to touch it along with some of the senior leadership were responsible AI experts to actually red team the technology to figure out what could it do.

4:16And it was a wild experience to have something that was so like obviously so much more powerful than previous versions of the technology. When you like the moment that I touched it or anybody else on the team touched it, you knew like, wow, this is something different, but you didn't really know what was possible. And so we pulled experts across a large variety of areas looking at things, as you said, like adversarial inputs, like now what we call jailbreaks or prompt injection attacks, looking at errors like hallucination, looking at the ability to produce copyright material, but also other things like can it generate novel malware?

4:55can it generate harmful content? So we took that and sort of built out a taxonomy, and that helped also guide a lot of our work, and I think what we'll see in kind of future generations as well. And so certainly there's the list of harms that we see, or let's say risk that we see that we're focusing on right now. So adversarial inputs being one, we call this prompt injection attacks, errors. Hallucination is obviously the one that sort of captures people's attention, but omission is actually, can be like equally important. So for example, if you're summarizing a medical, you know, transcript or medical information, if you omit a key symptom, then you might actually be sort of totally changing the meaning of that or changing what like the diagnosis might be or something.

5:43And so we look at kind of both sides of those types of errors. We look at the ability to produce harmful content, harmful code, the ability to produce this copyright type material or IP material. And then some more abstract things like this is a totally new interface, right? And the thing that's exciting about the interface is that it is, it does like one of the most human like things. It speaks human language, but it is not human. And that's, that is a meaningful distinction. And so looking at, you know, making sure that the people understand the system. The system is not sort of confusing and manipulating them.

6:26So we look at sort of questions of how should that interface look. And like one of the challenging problems with this, for example, is the phrase, I think. We use the phrase, I think, to signal that we might not be certain of something, right? That's like one usage of it. But does that imply the system is thinking? Does that imply that it's conscious or something, right? And so we kind of dig into all of that. It's a pretty loaded term. Yeah. And so those are like kind of the places where we're spending our time. There's like a much, much larger sort of list of taxonomy that we look at. But there's probably one key insight we had early on in this Red Teaming that helped us sort of realize like, what are we trying to address?

7:08And one was a lot of these risks are sort of content types that are being produced that may be undesirable, like harmful code. And then the other type of risk is sort of model capabilities or model limitations that are a problem. So like hallucination, it like accidentally misusing data or something, that's a model behavior. And so we kind of take an approach where we look both at which types of content do we have to go after and which model behaviors or model tendencies do we have to go after. And so we look at kind of those two dimensions. In the past, when we spoke in particular, and we as a community talked about responsible AI, a lot of the focus was on concepts like fairness and bias, which we would try to make concrete in measurements and things like that.

8:06but they feel like different types of concepts than hallucination and jailbreaking and adversarial attacks, which feel more like security. Like, do you feel that, you know, has there been a shift from, you know, one type of concept to another or is, you know, fairness and bias kind of a, an umbrella to, you know, the other types of concerns that we're dealing with today? How do you think about the relationship between these types of ideas? Yeah, I think, so we see them as, you know, side by side, like fairness is one principle, sort of security is another principle we have. Certainly fairness and bias is really important in these applications.

8:50We say that, like one of the top types of fairness you see in this is representational fairness. How are people represented by the AI system, right? You don't want content to be stereotyping, or you don't want it to be like over-representing a group or under-representing a group, right? And we see that a lot as a challenge still today in like text to image generation, where you put in a particular, you know, query, I want a woman in address, and often you end up with kind of images that are all very similar to each other. So it looks like it's over-representing a group and sort of under-representing everyone.

9:23And so these issues are still very much something that we need to work on and we address in the systems. But I agree with your point that we are spending a lot more time than we used to on the adversarial elements and the security pieces and ensuring that the systems can only be used as intended and that they can't be misused and looking at things like, you know, secure by design for that. And so there's definitely a much bigger emphasis on that part than it used to be. But I don't think that people think that fairness and bias is less important. It is an equally important concern. There was a recent example that put the idea of representational bias and imagery on a lot of people's radar.

10:15I'm wondering if you can talk a little bit about what we can learn from kind of these, you know, big public Gen AI failure cases. Certainly, Microsoft is familiar with this. You had one of the first with Tay kind of pre-Gen AI, but then there was like the Kevin Roos article on Bing around the time that the Bing with AI launched. Google recently had the whole pizza glue fiasco thing. Like what is there to learn from these public failures? I would say everything, right? Like the part, I mean, like all of this is a learning journey. We're taking a very new technology and then using it in like totally novel ways in many different applications in the world.

11:01So, you know, we're going to be learning for a really long time, I think. And there will be more mistakes as well. I think that the thing that's important is to make sure we are sort of contained in the learning, right? Trying to sort of take calculated risks. Like there's some failures that are acceptable. There's some failures that are not. And so I think if we take the two sort of Microsoft examples, you know, first, Tay. Tay was actually even before our ResponsiVet program had really started. Certainly there was some kind of rumblings of the idea of AI ethics and research. but it wasn't as, you know, as a significant thing in practice in companies.

11:44And so I think people were just like shocked to kind of just that this had happened, right? I think people did not sort of go in prepared for that. And, you know, people inside Microsoft and people, you know, involved as well. And so I think that's like a good experience to have in Microsoft's history because it is a reminder of just how important this is all the time and really sort of what is at stake. And so I think we have changed a lot since then. We started our office of response by I implemented a response by a standard. You know, we don't put that into practice. And so it's just sort of night and day from how it was with with Tay.

12:28Now, Kevin Roos' example was something that this was right after we launched Bing Chat. Kevin Roos went and spent a lot of time interacting with Bing Chat and ended up with a, let's say, kind of more emotional, deranged, I think many people found highly entertaining experience. because I believe that was actually one of the most read New York Times articles of all time. So one thing we learned is like, wow, like exactly. One thing we learned is people are really paying attention, right? Like the entire world is really interested in this technology and how it behaves. Another interesting learning from that is we knew when we were launching Big Chat and GPT-4, like one of the things that was very top of mind for me the entire time was such a new interface.

13:26and it's so much more powerful than the previous version of the technology. So we knew people were going to interact with it differently, but we didn't know how they were going to interact with it differently. We had some ideas. We had done a lot of internal testing, so we had a feel for what people were doing. But the technology can do a lot of different things. It's a pretty broad surface area. Combined with a search engine, there's a lot there. And so we had built a testing system, And we had been testing conversations, but we really focused a lot of our energy on testing conversations that were around 10 turns, were a little bit shorter.

14:06Because most search engine interactions were like, single turn, maybe you'd like refine a query and that's it. And so even 10 turns, wow, that's a long conversation. And we did not expect people to be talking for like three hours in the middle of the night or whatever, going, you know, many, many turns deep. And so it sort of showed a limitation in how we had been testing the system. But on the other hand, you know, we were able to fix it really quickly. And that's something we did design. Like, as I said, we knew, like, it was very top of mind for me that we were not going to know every way people are going to use the system.

14:42So as we put it on preview, we're going to have to learn. We're going to have to adapt. So we had built a system so we could make very quick adjustments to the different responsible AI controls and actually be able to adjust. And so actually one of the quickest things we did there was just limit how long the conversations could be so that you couldn't have this very long form conversation where the system kind of contextually drifted over and over into a stranger place. But, you know, there's still, we're still going to keep learning from these kind of experiences. And then I think, you know, to the Google example, right, all of us are learning and I think there are challenges with this technology.

15:22and what I was saying of like the technology, it's a big surface area. There's a lot of different things there that people are going to use them for many different queries. People use Google for a lot, right? So they're pulling from all sorts of sources on the internet. Of course, some of those are going to have inaccurate information. And so that's a real challenge, I think, in terms of how to use information effectively to sort of summarize things. And it's not a challenge that anyone is sort of immune to right now. It's something that we're very actively working on because the way to make these systems really successful is putting high quality data into them.

15:59When you think about maybe from the perspective of the way that you've kind of architected these deployed systems there or what customers need to have in place so that when one of their deployed products does things that they weren't intending, like what is what is the system need to look like so that you can rapidly respond to you know the types of examples that we just discussed you know you know clearly like if you weren't planning to be able to address those things in advance it would take you a lot longer than it took to you know turn off whatever you know the change of length in this case or you You know, turn a filter, I guess, is another common example.

16:51Like, is there a way to kind of generalize the systems or layers that need to be in place in order to be able to respond quickly? Yeah, there's a there's a lot there, I think. So first of all, you want to exactly as you said, you want to start with a system that has defense in depth, has sort of layers built in by design, because each of the technologies have their own strengths and weaknesses, right? So I think it's often kind of the pictures of like stacking layers of Swiss cheese. So there's no hole that goes all the way through, right? And so to do that, we use obviously safety built into the model itself through techniques like reinforcement learning with human feedback.

17:38We use an auxiliary safety system that looks at all the inputs going in, looks at all the outputs coming out, can block those in real time. And that, of course, uses a mix of technologies like classifiers, but also things that are faster twitch so you can respond quickly, like block lists, right, or embeddings or things where we can very quickly add an additional assertion. The model of post-training is something you can do maybe on the weekly basis if you want to, but certainly not kind of incident level. Same thing with the classifiers. It's like we have fast twitch in the safety system and then stuff that we can update more in the speed of a week.

18:17And then one of the most powerful elements is actually the system message where the program you're giving the model. Because I think we've all seen the models will behave very differently depending what instructions you put in there, including like small changes in which word. And so that's a very powerful thing that you can change very, very quickly. Maybe for us, that's about something like adding to a block list we can do in a minute, and it's in the production system. Something like changing the meta prompt, we can do it in about a day. And the reason is it's pretty easy to go in and change the meta prompt or system message.

19:01We say meta prompt more often internally, but system message, meta prompt, same words. You can go and change it really quickly, but it's going to change your whole application behavior. And so it's not something that you want to do without rerunning your test. And so to be able to effectively change the system message, we had to build both automated quality tests and automated tests for all the different types of risk we wanted to be looking at as well so that when we make that intervention, we can rerun our testing suite, see if the system's still performing well before we come back. kind of ship that update.

19:37And so to be able to move fast there involves not only having the mechanism to change the system prompt, but the testing that allows us to know that that intervention is actually safe. And so we actually see that kind of pattern at every level where you need to have both the intervention mechanism, but the testing that ensures that the intervention doesn't break something else. For example, you don't add a word to the block list that is going to cause a lot of overblocking it unexpected way. So for example, in GitHub Copilot, one of the words that was on an early block list was race, and they meant it in the natural language sense, but then it blocks everything where they had written race condition or things like that, and people hadn't thought about that use of it.

20:27And so we build in testing that allows you to understand what would be blocked and what Not so that you don't have as much of that kind of collateral damage. So that, billing all of that is one part of it. And then the other part is actually... I'm envisioning there, you know, something akin to kind of traditional DevOps CICD. Someone adds an assertion to a block list or changes the system prompt and it automatically kicks off some set of regression tests. and there's a red light or a green light at the end and then someone pushes a button to deploy? Is it essentially that same idea? Yeah, it is exactly the same concept with one difference, which is often like, what are you testing, right?

21:14In this case, what you really need to do is look at your data distribution and understand the behavior over the specific examples. And so a lot of the ways that we need to build these tests are ensuring that you have sort of representative data samples or you're sampling your production traffic so that you know this intervention is not going to sort of really change how the system responds to, let's say, your most common prompts or something. And so it's exactly the same. Is the implication that at the end of this testing pipeline, a human needs to look at the results and do some analysis and that there can't be or you're not trying to get to a kind of red light, green light type of a situation?

21:57or just that the nature of the tests are different. It's not break, not break. It's, you know, distributional, but you can still automate it to a determination of whether it's good or not. I guess the question I'm trying to get at is like, for folks that are building these kinds of systems, like how far should you be trying to get with, you know, this type of automation in the testing loop? Yeah, it's a really great question. I think for the testing for these incident interventions where you want to go really quickly, as much as possible, we do try to make it procedural. So let's say you add an intervention and now the block rate goes up from 0.2 % of your traffic to 0.5 % of your traffic.

22:47You probably want kind of rules that say, okay, 0.5 is acceptable for an incident intervention. But we certainly have experienced cases where the, let's say the incident is very high stakes. We want to take a very quick intervention, but the intervention is also very high cost. It might have much more collateral damage where it blocks a lot more or it totally shuts down a used case. Yeah, let's see. it's usually just where I don't think I can think of a good one. It would, it's mostly just cases where like what the thing you need to block is like a really common word or something, right. Then that would probably take out like a lot more.

23:33Um, and so then, then you do need sort of human judgment in terms of who's, how is this really the right trade-off and is it a longer term decision or is it just for the next hour until you can like ship a classifier update that's really going to more surgically solve the problem. And so we, you know, you're still going to need it there. I think the other thing we've seen in all of this kind of testing is you need human judgment to go in and say, what are the appropriate test results? Like what thresholds, what defect rate, for example, is acceptable, which types of defects are acceptable, which are not.

24:12But once you've kind of decided for a particular application what that looks like, then you can rerun the testing really automatedly. But it's not that we can say, okay, for, let's say, you know, like hateful content, the appropriate defect rate is, you know, 1 % errors and no more, right? Like it just varies depending on applications. Actually, probably a better example there would be hallucination. How much hallucination is acceptable is going to vary a lot depending on the application. So like an application where you're brainstorming, totally fine, potentially, if it makes up a lot of content.

24:46In fact, probably desirable. An application where you're doing a factual lookup, let's say, in a search engine, then you want it to really stay focused and grounded on that content. And so the amount that would be appropriate is going to be different. On that theme of hallucination, you alluded to this earlier in your comment about the word think and thinking, but an element of both response, but also the initial delivery of these types of systems is like conveying the right expectations to the user and, you know, crafting that as part of the user experience. And I don't know that we've gotten much more advanced than chat GPT can make mistakes.

25:33Please review your results when you're using it for something important um you know have you seen do you think that there are you know deeper opportunities to integrate um conveying the probabilistic nature uh or the potentially hallucinated nature of results to to users are we just you know not there yet but there are opportunities or are you not bullish on a lot of progress there? No, I think absolutely there's opportunities, right? We're, as I was saying earlier, we're in just the early days of all of this. I think there's so much opportunity to do many things better. But I, and then we'll learn, it is still an innovation space.

26:24So we're still figuring out how to do this, inventing new approaches. I think that we did spend a lot of time on the user interface when we were designing BingChad, which is now Microsoft Copilot. And a couple of things we did, for example, is very much on the landing page. We did have the original statement that said, you know, this is powered by AI. Surprises and mistakes are possible. And because we wanted to emphasize that you also get kind of those amazing, unexpected, good experiences like, wow, I didn't know it could do that or it would respond that way. But we also, for example, added examples of what the technology could do.

27:12Like you can ask it this or it can do that because we knew it was just so new that people weren't going to really understand what it could do. And then we also added, shortly after launch, a feature that allowed users to signal a little bit more about the tasks they're trying to achieve exactly to the hallucination example I was just giving earlier, which is users now can select if they want their answer to be creative. creative, and that would allow it to be less based on the search engine results, more sort of brainstorming and creative. Or if they want it to be balanced, kind of just the middle option, or they want it to be precise, where we've tuned the system to really focus on the search engine results when it's answering and not deviate or add its own information.

28:00And so, for example, giving the users some amount of control so that they can sort of have the system behave aligned with what they're trying to achieve. We also, you know, put references in from the beginning and the responses so that you can go and say, I feel that I should have confidence in this answer because it comes from a source that I think is, you know, high authority or not, right? And so we, you know, looked at quite a few different UI elements in addition to those that I've listed to help people really sort of better understand what the technology is and how to engage in it. And so I think there's still, as I said, a lot more to do there, but it is something that is incredibly important, probably something we're not talking about as much, like a lot more interest in like alignment and, you know, things that we're doing with the model itself than, hey, how do we design better user interfaces so that people can use the technology effectively?

28:57When you talk about the idea of allowing the user to articulate their intent or degree of precision, does that slider influence the meta prompt or the model in some way, a combination of both? Like how is that ability implemented? Yeah, it influences both the meta prompt and the model. So one of the things we did when we first built precise mode, for example, is we worked with OpenAI on post-training RLHF for GPT-4 that really emphasized the model should only use sort of the grounding data in the context and it should only answer based on that. And so, you know, we gave it those types of examples.