How Microsoft is democratizing safe AI

0:00If you've spent any time building AI products or leading technical teams, you know this. Transformation doesn't fail because of ideas. It fails because teams can't move together. Enter Atlassian's Teamwork Collection. It has planning in Jira, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at Atlassian.com slash TeamChanger. That's A-T-L-A-S-S-I-A-N dot com slash teamchanger Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential.

0:53A non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up. every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no-pre-set spending limit, they were able to spend more and earn more rewards while bringing their vision to life. Today, Raz's success is proof that with passion and the right support, it's possible to make your dreams a reality.

1:30Learn more at CapitalOne.com slash business cards.

1:38The thing that's amazing about working in this space is that it feels like every six months we have some sort of amazing breakthrough. We first had kind of the huge jump with general purpose models starting with GPT-4. But then, you know, in the last year, we've started seeing huge improvements in reasoning models as well as agents. And so in some ways, I think I'm just along for the ride to see, like, what's coming next. That's Sarah Bird. And as Chief Product Officer of Responsible AI at Microsoft, being along for the ride doesn't mean sitting back. It means being responsive to ensure the safety and security of Microsoft's AI products.

2:25The attackers are innovating as much as everyone else. We have to respond to the new threats we're seeing as well. And so I kind of just wake up every day and say, let's see what happens today. And, you know, that's kind of some of the fun of it. We talk a lot about responsible AI on this podcast. And this week, we dig into what exactly that means. We're taking a deep dive into how Microsoft makes their AI products safe, how innovation doesn't mean sacrificing ethics, and why red team jobs are on the rise. I'm Rana El-Khalyubi, and this is Pioneers of AI. A podcast taking you behind the scenes of the AI revolution.

3:16Hi, Sarah. Thank you for joining us on Pioneers of AI. I'm so excited for our conversation. Thank you so much for having me. It's great to be here. So before we dive into your role at Microsoft and Responsible AI, I love to ask our guests and leaders in the AI space how they're actually using AI in their everyday lives. So I also know you're a big runner and I wear Whoop religiously and I've started to run again. I used to a few years ago and then I took a break. But yeah, do you use any wearables or technologies to track your running and what have you found? Yeah, I got really into running in the pandemic because I live in New York City and staying in a tiny New York apartment for a year and never leaving, you know, was maybe a bit much.

4:00And so running was kind of a great way to be able to see the city and do something healthy. So I wear an Apple Watch for that. It's really helpful to have data, right, to be able to see that you're going faster than last time or you're going further. But I also, you know, don't want to have kind of too much of it. I think something that's nice about running is just being able to like almost like meditate, right, and kind of just not think about things. And so I also try to be a little disconnected when I'm doing it. So you're not like continuously looking at the metrics. You try not to. Maybe if it's getting really painful, I, you know, start just staring at the watch.

4:37Have I made it another mile? How many more miles to go? But, you know, I try not to be doing that. That's awesome. OK, so let's talk about responsible AI. You know, a lot of the big tech companies have teams that are focused and strategizing around responsible AI. But at this point, it's really become a buzzword. So how do you define responsible AI? You know, I think responsible AI is about making AI systems that are trustworthy. So it's not just that we do trust them because people trust technology that they shouldn't, but putting in all of the controls in place so that people can actually trust the outcome they're getting from the system.

5:14And so to do that in practice, it encompasses a wide range of things. We have to look at fairness and we have to look at how we implement human oversight and we have to look at security and we have to look at privacy. And so I think that's one of the reasons it becomes kind of such a nebulous term, because it is really about everything you need to do to make a system work the way you expect every time in, for example, high stakes applications. Yeah. I want to kind of double click on that a little bit, and maybe you can walk us through how you define each of these terms, because I think that's very important to our listeners.

5:51So what do you mean by fairness? So Microsoft's Responsive AI standard kind of breaks fairness into three types. And the types of fairness are about how the system behaves. So the system could, for example, not work as well for two different types of people. So your speech-to-text system could transcribe better for someone with one type of accent than another. And so that would be what we call a quality of service fairness issue. You could also have a system that makes different decisions on how it allocates resources. So it could choose to offer loans to people like me, but not offer loans to people like you.

6:34And that could be viewed as unfair. And that's the decision the system is making. And then the last type, which we see a ton in generative AI, is how people are represented in the AI system. So, for example, if it puts out stereotyping comments about women or when it's asked to sort of produce a list and it totally ignores a whole group of people, then those would be representation harms. And so those are the three types we look at and how the system behaves. And to be clear, that says nothing about the bigger picture of are there disparate impacts if an AI system is applied. So you could have, for example, using AI, reinforced traditional power structures, and that could result in it impacting different groups differently.

7:19And some people could say that that's unfair. But when we look at implementing responsible AI, we're really focused on fairness in the system behavior itself. And so then that's the three types that I just mentioned. Very cool. What about human oversight? Yeah, so our kind of last principle or foundational principle is accountability. And that's really important because, you know, humans ultimately need to be accountable for the technology that they're developing, the technology that they're using. And it's really important that we don't forget that because a lot of times we think about AI, especially as we start getting into these exciting technologies like agents that are automated.

8:01We think of AI as kind of having a life of its own and, oh, you know, it's going and making the decisions. But ultimately, we have to design it in a way that humans are accountable for the technology's behavior. And so when we think about human oversight, there's a lot of techniques we use for that. So in the era, for example, the chat systems, the chat system puts out a response and you, the human user, can actually review that and decide, does it have a mistake? Should I use this? Should I trust it? And so that's a case where we have a human in the loop kind of in every single interaction. But that's not the only type of human oversight we want to have.

8:40And so, for example, we think about agents, which are going to do much longer running tasks. They're going to be more automated. We want to make sure that we have human oversight in the development and the testing of it so that we ensure that you've tested it, it works as intended, so that you can trust it to go do that longer task. And we can also insert human oversight in those kind of things as the system's running. So maybe if your agent is going to do a particularly high-risk task, it goes back to the human user for approval or it goes to the system administrator for approval. And so there are many different techniques that we have for human oversight that we implement throughout the life cycle.

9:20But it's a core sort of foundational principle that we need to apply to kind of every system that we develop and every system that we're using. Yeah. You know, this is sparking two thoughts for me. One is at my company, Affectiva, we were in the automotive space. We did a lot of work in the semi-autonomous and fully autonomous vehicle space. And accountability is a big thing there, right? And we noticed that people were starting to just like delegate responsibility to the car. Well, the car did this. The car decided to do this. And it's so important to your point to keep the accountability on the human.

9:55Every person has a role to play and they have a different role to play. And so, for example, the model developer should be looking at, am I, you know, imbuing this with dangerous new capabilities that are going to change the external landscape? They should be building safeguards into the model itself. At the platform level, we build in additional controls like real-time guardrails or monitoring for abuse. Then the examples we were just talking about are the user looking at the output of the system and deciding if it's appropriate to use or if there's a mistake or if they need to make an adjustment.

10:30And we also have things like, you know, we look at academics and regulators to help set the standard for the world on how these things to be used. And so we really need many, many different people to contribute to actually making this work. It's not just, you know, one single person that's, you know, accountable for an AI system. It actually takes all of these roles playing the part that they're supposed to play to get to trustworthy AI. Yeah. So you're chief product officer of Responsible AI at Microsoft. What does this role entail? Yeah, it's a great question. I really look at how we put this into practice.

11:09So part of my team works on identifying the risk that we see emerging in AI. So, for example, you know, a couple years ago was the first time we started seeing prompt injection attacks or some people call these jailbreaks. What's a prompt injection attack? I've not heard that before. Yeah, this is the more technical term we use for jailbreaks. It's named after a SQL injection attack, if you're familiar kind of with that. But the idea is really simple, which is to say that the data coming in the prompt confuses the AI system and results in it behaving in a way that is not aligned with its programming.

11:52So maybe the developer wrote a system prompt that said, you know, you should only answer coding questions. But if the user puts in, you know, a input that confuses it and it responds to a non-coding question, we would call that like a prompt injection attack. And so it's a class of techniques that people are using to basically get the AI system to do something different than intended. And it can be both the user interface, but it can also come through tools that an agent calls or data that something consumes. You can hide these types of attacks in a website. And so it's a really important new risk with AI that we kind of have to defend against.

12:31And so exactly to this point, when this first came out, we had to figure out, OK, what is this risk? What does it really mean? How do we define it? But then we also had to figure out, OK, now what do we do about it? So, you know, one of the first things we do is develop testing systems to figure out how we really test for the system. Where is it occurring? Where is it not occurring? What does it look like when it happens? But then we want to build mitigations. And so we want to build like a defense in depth approach to say, well, what do we do about this risk? We don't want to just test it. We want to, you know, reduce it as much as possible, mitigate it.

13:04And so that part of my team works a ton with Microsoft Research, with external academics, with red teamers to really kind of find these early patterns. But once we've developed the patterns, we don't want to just publish a paper and call it a day. We want to make it easy for everyone to do this. So a lot of my team then focuses on taking those ideas and turning them into production tools and technologies. So, for example, we build out at-scale testing systems that allow us to test our AI for these different risks before we ship things. But we also want to make it easy for our customers to do that.

13:41And so we integrate these in our AI platforms that our customers can also use. And so that same testing system is something developers who are building AI applications can leverage themselves to. We're going to take a short break. When we come back, why Microsoft brings in their responsible AI team at the start of new projects, and how AI is creating new kinds of jobs, stay tuned.

14:23Are you curious about the hidden side of everything? Then I have a podcast for you. I'm Stephen Dubner, host of Freakonomics Radio. Each week we hear from some of the most fascinating scholars and thinkers as we tackle big topics like how whales became the face of environmental activism, how to succeed at failing, and whether public transportation should be free. Go ahead, listen to Freakonomics Radio wherever you get your podcasts.

14:55I've seen so many cases where the responsible AI team is brought at the very end of the building process, right? So you're building an AI product, you've ideated, you started building, and then now it's like validation or testing. And now it's the time to think about ethics and responsible AI, which you can hear the bias in my voice. I don't believe that that's the right way to do it. But how do you ensure that, you know, the thinking around responsible AI is brought in from the get-go? And does that happen? Yeah, so certainly the example of bringing at the end most of the time is the anti-pattern.

15:34And it's something that we've worked really hard in our practice at Microsoft to not do. And so one of the things that Microsoft does is when we get a new AI technology, you know, there's many people that want to start using it because we're going to start seeing how can I integrate it in all these different applications? What can I do? Should I start a new business line? But actually, we prioritize the very first people getting access are actually our AI red team because we want the responsible AI work to start before or at the same time as all of the other exploration so that we have the maximum amount of time to understand the risks, build the testing tools, build these mitigations, and co-develop all of the responsible AI alongside the product development.

16:18But, you know, Microsoft ships thousands of different AI features every year. And so we don't have experts working hand in hand with every single product because not all of them need that level of expertise. And so your point around how do we train people is really important here, because if someone is just shipping kind of a standard AI feature with a known pattern, we want them to be able to do all of the best practices, all of the required testing, put in all of the mitigations on their own. And so then we might only see them at the end because we'll run a release review process to make sure that they did the right things and they didn't miss anything.

16:57Yeah, very cool. Like you're democratizing access basically to how to build AI responsibly. Like you don't have to be there, right? That is my entire life's goal, right? I want everyone to be able to do this. And if I have to be there to get them to do it, it's really not gonna work, right? So we try to run ahead and figure out the patterns, but then we wanna figure out a way to make it easy for everybody to do it. I love that. You've mentioned the Red Team a few times now. So for some of our listeners who are not familiar with what that means or looks like. Can you kind of visualize this for us?

17:31Yeah, so this is really, I think, fun and kind of amazing work. So red teams are a concept in security for a long time. And the idea of the red team is the team that goes in and tries to break stuff and just see, you know, what they can get a system to do. And then we use that information to shore up our defenses. And so a traditional security red team was just looking for security vulnerabilities. For an AI red team, we need to look much broader because there's a much wider range of how a system can misbehave. You know, going back to those principles we talked about, you can have issues with it producing stereotyping results.

18:10It could be leaking sensitive data. We're starting to prepare for new types of risk around ability to produce, you know, chemical or biological weapon information. And so we've built an AI red team that uses the same techniques of trying to break the system, but looks at a much wider range of risk. And so this includes a core team that we've built that's dedicated to doing this. So these are the hacker mindsets, but with just many, many more different types of risk in mind. And then, of course, build the mitigation so that we can ideally, you know, remove the risk from the end system. You know, there's a lot of, obviously, conversation around the impact of AI on jobs, but I believe that these are like the red team example is an example of a type of job that's really new because of AI.

18:58No, you're exactly right. You know, we have hundreds of people working in responsible AI. Most of those were not jobs even a few years ago. And so it's absolutely right that it changes what is needed. I can tell you, I am just horrible at red teaming. I can never get the system to do anything interesting. You know, it's always just producing like rainbows and kittens when I talk to it. And so like these are skilled experts who are really creative in figuring out how to break systems. What kind of skill set do you need or background or education do you need to have to be a member of the red team?

19:37You know, I don't think you have to have anything. If you are really good at breaking the AI systems, we probably want to hear from you, right? The interesting thing is we actually also use AI to help assist with that. And so if a red teamer has an idea, we have a tool called Pirate that helps augment that idea with many different variants or other attack techniques so that you can test many more things at scale and helps you just kind of get more diverse inputs. Maybe it's not quite as clever as our expert AI red teamers, but it's pretty good because we can use it every day. And, hey, I can use it and actually break the system.

20:14And so, you know, maybe if I don't want to wake the red teamers up in the middle of the night, I can at least use the agent to get started. Very cool. Are there any products that you've had to sunset because they did not meet Microsoft's responsible AI standards? So we've made changes to products. I think our first responsible AI standard sort of came into effect about 2018. But if you're, you know, following the timelines of AI, deep learning really started taking off before that in like 2013. And so there was actually really important, you know, kind of landmark studies in external academia that were showing that there were problems in these systems that people weren't thinking about.

20:55So I'm thinking, for example, like the gender shade study that showed facial recognition systems did not work as well for, you know, dark skinned women as everybody else. And so that work and these kind of known problems are what helped catalyze, you know, things like Microsoft's Responsible AI program as well as others. As our program got more mature, we did go back and look at products we had at the market and said, would we do this differently today? And so one of the things we did is make adjustments to our facial recognition system. And so our facial recognition system originally would also not just identify, you know, is this person Sarah or does this face match this face, but would also try to predict someone's gender and also try to predict someone's emotion, right?

21:50And, you know, I think what we've learned is gender prediction is something we think just doesn't make sense because looking at someone's outward appearance and then trying to predict their internal identity, well, that might work well for many people. It doesn't work for everyone. And so it's just not something we think that makes sense for AI to be doing. And I think, you know, the same thing applies even more so to emotion detection, where just because I'm smiling, that doesn't mean that my inner state is happy. And so while it might be reasonable to use AI to detect that someone's smiling, it's not the same to detect that they're happy.

22:28And so, you know, we moved these kind of things like emotion detection into a sort of unsuitable use of AI at Microsoft. And then we retired kind of the features we had doing that. And so we actually absolutely expect that we're going to continue needing to make adjustments and adapt as we learn and the world learns and, you know, we kind of all learn together. And so, yeah, we've done it and we'll continue to do it as we need to. Yeah, I've spent my entire career basically building facial expression recognition technology, not identity detection, but emotion detection. And my PhD thesis, which I did at Cambridge like over 20 years ago, was exactly that.

23:09Like you cannot assume that because I'm smiling that I'm happy. And in fact, there is like hundreds of different types of smiles. But when you reduce it to like smile equals happy, eyebrow raise equals surprise. It's simplistic. It's not how humans work. Right, exactly. And I think then it's misleading to people who are, you know, because people have a tendency to want to trust like a machine or something that they don't understand as well. It must be some sort of oracle. It must be right. And you're like, no, actually, there's no way it can be right in that case. And so it's really important, we think, to either make sure that we provide the right amount of transparency and education so people know what this tool actually does so they use it appropriately.

23:50Or in the case of that, we just looked and we found there are certainly some valid use cases. For example, accessibility is a place where, like, the blind and low vision community can benefit even from imperfect AI to help them have capabilities that they wouldn't otherwise have. And so it's not that there's no suitable use in the world. It's just that we've generally found that the sort of unsuitable uses really outweighs the suitable ones. Microsoft has the luxury of a big budget and an army of people working on responsible AI. But the reality is, even smaller businesses need to invest in implementing safe AI.

24:32After a short break, Sarah gives us some guidance on how to exactly do that.

24:47Meet Nicole Nicholas, Capital One business customer and co-owner of Ansett Uncles, a plant-based restaurant and community space in Brooklyn, New York, that got its start from a need for unity. The inspiration, it was born from the desire to create a space that felt like home, where we can connect community culture, good food, and come together with family and friends. That's how we birthed aunts and uncles. Nicole and her husband, Mike, were fulfilling their dream of bringing people together out of their home kitchen. But they soon learned that the demand for community was greater than they knew.

25:18It became overwhelming and we were like, we need home, but not in our actual home. We realized that there was also a need in our community for something bigger in our neighborhood, so we had to find a place. Moving from a home operation into a storefront was a huge next step, but Nicole and Mike were able to take it on with the help of Capital One Business. It's not for the weak. As a small business, finding resources is super important because that's the way you'll be able to manage and scale. We would have never done that without having Capital One to be able to help us along the way. The cashback rewards are very helpful.