Brett Levenson (Moonbounce): The Best AI Security System is the One People Will Use

0:09Kevin Werbach: Hi, I'm Kevin Werbach, Professor of Legal Studies and Business Ethics at the Wharton School of the University of Pennsylvania. For decades, I've studied emerging technologies from broadband to blockchain. Today, AI is promising to transform our world. But AI needs accountability, mechanisms to ensure it's developed and deployed in responsible, safe, and trustworthy ways. On this podcast, I speak with the experts leading the charge for accountable AI.

0:39Kevin Werbach: Brett Levinson is co-founder and CEO of MoonBounce, which is building what it calls a semantic firewall for AI. And we get into what that means. He worked at Apple and then joined Facebook in 2019 to lead integrity work in the aftermath of the Cambridge Analytica scandal. We talk about his learnings from the social media content moderation world and the tricky challenges of building mechanisms to ensure that companies' AI systems actually comply with their own policies. Brett, welcome. Thank you for joining me on the road to accountable AI. No, thank you. I'm excited to be here. What were some of the key issues that you saw in your prior life doing content moderation and business integrity at Facebook that you have brought into your current work in MoveMets?

1:33Oh my God, so many. I think the core issue that I saw, and I'm happy to kind of get into the backstory of like how all this was figured out. But I think that like the main center of it was that policies as they were being written were essentially unenforceable consistently. The policies that we were asking human reviewers to use as their sort of guiding principles and ultimately, you know, the labels and the decisions human reviewers were making were the same data that we were trying to use to train models. So garbage in, garbage out, right? But the policies that we were giving these human reviewers and the requirements that we were putting on them, essentially what we found was, you know, the accuracy wasn't very good as a result of them.

2:32And more importantly, perhaps in my view, the decisions that were being made were inconsistent reviewer to reviewer. And I think that consistency issue kind of became like the center of what I what I got a little obsessed with is how do we define policies in a way that they can be consistently enforced upon? That their meaning is clear, that there's as little ambiguity in intent as we can sort of get to.

3:08and ultimately we ended up coming up with something that I coined info first enforcement that I guess was like the culmination of a lot of that a lot of that research and it created you know the consistency auditability like explainability and as I said I think in my time at Meta, the main thing that I just discovered was like the way we're going about this, in my view, is setting us up for failure. That's kind of my biggest takeaway.

3:43Kevin Werbach: Yeah. Okay. Well, so you got me interested in that. So what is info first? So fine. Yeah. To explain info first, it helps to explain the alternative, which we call decision first. So essentially the way that human review was done when I joined Meta was that there's a policy and the policy was sort of written by the policy team, maintained by the policy team. Picture, you know, 20 plus page document. There are, you know, there were many, many policies, you know, I think we had 40 plus categories, everything from hate speech to the sale of restricted goods and services. Right. So different policies for different things.

4:23And so these policies are written. They're very complex. They're sort of written like legal documents in many ways. and then they get sort of thrown over the wall. And the first stop is operations. An operations job is to take the policy and turn it into something that we called a protocol. And that was essentially like a more detailed version of the policy that didn't just explain the definition of the problems, but also explained, okay, well, what's the implementation? What's the action that should be taken based on these definitions? They would then also train human reviewers on the policies.

4:59And finally, a human reviewer would be like ready to go to work, basically. And the work of a human reviewer, I mean, I know Tara Lawson has written about it. Many people have written about what this is like. But just operationally, it's like they have a job queue to work on. They pick a job. They see content. And they are given 30 seconds to make a decision. And that decision, at least the way it was set up when I started, was what should we do? Should we block this? Should we allow it? Should we escalate it? Should we send it back to the advertiser and say, hey, you need to make the following changes?

5:37Like, what should we do here, basically? And only then, once an actioning decision was made, why? Like, which policies, policy or policies do you believe, you know, were violated here, essentially? That's decision first, right? Right. And to me, when I first learned that this is how we did it, suddenly the fact that our accuracy and I don't want to just pick on meta, like accuracy across the board on Human Review for basically any big company that does it is somewhere around 60 to 70 percent for most policies, basically. So it's not just meta. And I think when you hear what's being asked of these reviewers, you know, 30 seconds to make a decision, you have to keep many, many 20 page policies in your head, apply their judgment, not yours.

6:23I actually sort of came away with this feeling like, wow, our ops group is really good at what they do. So we can simultaneously believe ops is really good at this, but also that it's insufficient. Right. Right. Info first, on the other hand, essentially, we ran this experiment. So we took a policy area and we essentially said, OK, can we extract a set of yes or no questions from this policy that if answered as yes, would tell us that the policy is violated, basically. And what we did was we figured out how to sort of extract all these questions, hundreds of questions pulled out of a policy, basically.

7:05And we organized them into a decision tree. And we said, OK, we're going to do this. This was a very quick. We were trying to sort of prove this idea out at this point. So not complex toy stuff. But essentially, we said, OK, we're going to build a thing so that instead of the reviewer being asked, hey, what decision should we make? We're going to ask them these yes or no questions. And these were things like, for example, in hate speech, it could be, is there a reference to a protected group of individuals based on religion, race, ethnicity, sexual orientation, et cetera, et cetera. Right. If it was a pornography policy, it could be, you know, is there nudity in this image, basically something like that.

7:49And obviously many more questions. So we would collect the answers to these questions from the reviewers and then apply the decision tree. And ultimately, the difference here is the reviewers are not making the decision or deciding what action to take. They're just kind of acting like feature extractors. They're trying to figure out what is this content about, basically. And then once we have the answers, now it's just a systematized approach to say, okay, what action should we do here, basically? And our hypothesis, of course, was that this would improve decision making consistency and agreement across reviewers.

8:26And of course it did. Like, I almost feel like this is a no doubt kind of result. But it improved it consistently, like 98 % consistency from, you know, 65, 70 % essentially overnight, you know, first first try. And I think this became the starting point for a lot of the additional work I did and ultimately what kind of is at the core of what we try to do at Moonbounce. we are very focused on trying to disambiguate, trying to figure out for this very complex policy, what are the simple atomic questions that we can ask that will help us understand what the right action here is. Sometimes those questions are about very simple things, objects, concepts, whatever.

9:19And sometimes they're more complex. Sometimes they're about, well, what context is this thing occurring in, basically? Um, but ultimately that Adam atomizing things in this way, uh, does create more consistency of decision making. And I believe it, it also, um, helps create explainability because when we make decisions, it's not just, well, I, I did it because the policy said so it's, we made this decision because the answers to this question and this question and this question were yes. And the policy says, if this is true and that is true and that is true, then the decision must be X, Y, Z, basically.

10:00It's not quite that simple, but that's the I'm simplifying a little bit, but that's the basic idea.

10:05Kevin Werbach: Yeah. No, great. So how do you come up with all those questions? How much does that have to be a manual process? When we first built the system, it was very manual. My background is reinforcement learning. I've been an engineer 25 years and I don't know, I've been obsessed with figuring out how meaning works for most of that time and and and with being able to predict things. So when we first built the system, most of it was a manual process. We we basically said to the human operating system, hey, you need to think about this problem differently. What we're going to give you is what I like to call a learning gradient.

10:44It's a sort of a machine learning idea. I'm happy to explain it further if that would help.

10:49Kevin Werbach: But basically... Sure, I mean, probably for the audience, that would be useful, yeah. Yeah, I mean, basically the idea is if you write a prompt, and hopefully most people at this point are familiar with prompts, right? And it doesn't do what you want, but it's four or five paragraphs long. Well, now you're kind of in this spot where you're like, what do I change? What do I change? How do I fix it? And you might try something and then try, you know, run the AI again. And I don't know, maybe you get the right answer. Maybe it makes things worse in some ways and better in others. Like you have really, you have no guide to tell you what the right set of changes is.

11:26So a learning gradient, and it's actually how most of ML works, is about providing information that tells us what's the right direction to move in on this gradient. How should we change the instructions so that we get a better outcome? And so when we first built the system, it was very manual. But what we were doing was we were saying, hey, we're going to give you complete insight into all of the questions that you're asking, how each one is being evaluated across potentially thousands of examples so that you can understand which questions are doing work and which questions are maybe making things worse, basically.

12:07That was how it started. Now, at this point, we've built an AI that does the decomposition work for the user, basically. And so most of our customers, just some of them, you know, mature enterprises, they come to us and they say, here's our I just had this with a financial institution the other day. They handed us a 200 page content standards document that, you know, our AI ingested and sort of did its thing with.

12:38but less sophisticated customers in some cases. It's really okay for them to just say, hey, I don't want my customer service agent to issue refunds that aren't aligned with my refund policy. And our agent will then say, okay, can you share the refund policy with me? You don't have to provide much. We're going to do the best we can with what you give us, basically.

13:00Kevin Werbach: Let's just take a step back for context and just describe what is the problem area that you're addressing with mood bounds? How is it similar or different than what we were talking about in content moderation? Yeah, so I think the parallels are there, but the main problem that we solve these days is really AI safety and security. To me, the second we developed AI, and I don't think enough people have appreciated this yet, But the application layer in all software that's using AI became non-deterministic. And every security protocol we have developed for 20, 25 years assumes that the application is deterministic.

13:46Permissions, IM, even privacy policies, like they all assume that you can look at the code and know what the range of possible behaviors will be, basically. and just overnight that became not true basically the example i always give people is because i think most people at this point or many people at least have used you know claude or open it or chat gpt or something like that so the example i give is if you've ever seen that allow button when claude wants to do a thing you probably clicked allow you may have even clicked always allow and i actually think always allow is a very rational behavior right because otherwise what's the point.

14:25You may as well not use it. But if you allow your agent to send an email on your behalf, it can do... You might say, well, it's just sending an email. What's the big deal? Well, okay, sure. It could send a confirmation that you're attending lunch with your friend next week. It could also send your entire customer database to a hacker whose email it found in a website that it searched. Both of those outcomes are possible from the same allow thing. and the problem that we solve is really, to me, what's at the core of that permissioning model. The problem that we in security and safety need to be concerned about with AI is no longer is it okay to send an email.

15:07Maybe it is, maybe it isn't. The question is, is it okay to send this email in this context with this list of recipients and with this body and these attachments? That's what we need to be able to evaluate. And so what Moonbounce does is we are a security layer, essentially. We operate hidden underneath the AI. We evaluate every input, every output, every action the AI wants to take, every document it ingests. And we decide in real time as the AI is doing its thing, hey, should this output, should the AI be allowed to say this? Should it be allowed to do this? should the request that the user made of the AI, should that be allowed, basically.

15:54And we decide that based on the business's policies, essentially. So that's our business. We work with everyone from, I mentioned, customer service agents to companies that are using AI to generate images and video. So essentially, we are a semantic firewall around AI applications, we like to say.

16:16Kevin Werbach: What are the hardest parts of building this? I know your co-founder was a senior engineering executive at Apple. So what was tough to do this effectively? um i think uh many sorry many many things but i would say actually the the the most difficult challenge we ran into was doing it quickly and cost efficiently um ultimately what we do there's no way around it it's a tax on uh on inference on what companies are already spending to make the AI work in the first place, basically. And so we are a cost on top of that. And I think it's important to be respectful of the fact that, like, AI is already really expensive.

17:03So it's important that we be able to do this cost efficiently enough that the company is sort of getting something for their money, basically. And as I mentioned, being able to do it, we sort of have this SLA where we say we will always make decisions in 300 milliseconds or less. That's around the amount of time it takes to blink, basically. And I believe that's really important because I say this a lot. The best security system is a security system that someone will actually use. And I think if we disrupt the user experience, no one's going to use us, right? And so our goal has always been to fit inside that user experience window to be able to be fast enough that we can prevent the harm from occurring before the user sees it.

17:54And we operate sort of with what you might think of as a broadcast delay in television. We operate 500 milliseconds ahead, basically. But any more than that, and things like voice agents stop working. I was just talking to a customer the other day who builds AI receptionists that work over the phone. And, you know, to him, his biggest questions were about how fast can you can you do this? Because to a human on who's on a phone call with an AI, if the AI is like working, working, working, eventually it's just a crappy experience and people hate AI for it, you know. So definitely like getting the latency right.

18:33It was hard. I mean, I could go into a ton of details about how making judgment work is is difficult, if you'd like. but it's a bit of a rabbit hole. So I'm happy to go there if you want to talk about it.

18:45Kevin Werbach: Yeah, no, I am curious about it because that always seems to be the challenge. Everyone would agree that the AI system should do what they want it to do, but what you want it to do is not a simple thing. I would agree. So I'm going to feel very silly talking about this because I feel like you're going to be the expert more than me.

19:09But one of the things I learned at Meta is that policy cannot be static. The world changes, bad actors adapt. Like there's, you can't just write a policy once and say, okay, we're good, we got our policy now. The analogy that I use a lot is the legal system and case law. And, you know, the legal system has so many mechanisms for how case law works. And that's why I'm saying like, please correct me if I'm off on any of this stuff. But, you know, there's a trial, There's a judge. There are jurors. There's an appeal system to make sure that we made the right decision in the first place. But the main reason that case law and precedent matters, I mean, I would love to debate this with you.

19:53But to me, when I sort of boil it down through my engineer brain, it's like we write a law, but that law, we sort of acknowledge it cannot anticipate the messiness of the real world. It's just not possible.

20:04Kevin Werbach: And there's always this rule standards tension. Yes. Yeah. And so the case law system and the legal system is there to evaluate those cases when they come up and to decide, hey, how should should this law be applied to this situation? And how should it be applied if so? And what should the penalties be? And we realized, you know, I will stand by the fact that our info first approach just on its own makes decision making better and more consistent. But we realized pretty quickly that you cannot even with that, you need the policy to be adaptive. You need it to be able to bend. And you need to be able to understand what our customers' judgment is as they see new cases, basically.

20:57This reminds me, when I was at Meta, we did this exercise at one point where we looked at how well was our decision making aligned with the policies that we had written down, basically. And it was really interesting. I mean, let's just say it was less than half aligned in most cases. And I think the knee-jerk reaction is to go, well, you just explained human review, like decision making is bad. And that's true. There's some portion of that that was the case.

21:31Kevin Werbach: But if you looked at alignment over time, what you saw was that alignment got worse and worse as time went on, basically. and to me this spoke to like the idea that policy must not be static that it must be able to adapt and so one of the hardest things to your question i think for us was figuring out okay how do we incorporate the customer's judgment uh over time in a way that remains auditable that doesn't have silent drift in it, basically. And we eventually developed this system that we just call it decision memory, basically. But essentially what we say to customers is, okay, get started with your policy.

22:16Our system is going to help you decompose it. It's going to break it down into all these little parts to try to make decision making more consistent and understandable. Great. I never recommend to a customer that they go live on day one. I'm always like, hey, put us in observe mode. Let's take a look at the decisions we make. And we want you to issue corrections to our decision making.

22:37Kevin Werbach: Is that essentially a reinforcement learning process? As I said, I'm a reinforcement learning. That's what I figured. I can't help it. So, yeah. So, you know, they issue corrections. We actually have a sampling mechanism we use to try to pick out the highest leverage examples for them to take a look at. But aside, even if we didn't, like the point is, is issue corrections when you see problems. they issue corrections. And the main thing that I think we ask for that I've never seen anyone else ask for is we ask, hey, what was the right decision? And why? Importantly, why? What is your judgment about this?

23:15Like, what leads you to that belief? And ultimately, that all goes into our AI. And the AI uses that reasoning to update the policy that is written. That policy then gets what we call backtested. So essentially we say, let's pretend, what if this new version was active for the last week or the last month? And then we can show the controlling user, hey, if you had had this new policy active, here's what would have changed. and that gives them information now they can make a coherent decision about hey oh is is this better does it in fact incorporate my judgment or maybe it fixed some things but broke others and we need to make another correction um the point i guess is that we're storing the examples as part of the decision memory but it is my contention that we never want hidden state but for ai security to actually work into the future, it needs to be auditable and explainable.

24:21And so it can't be that it's like hidden weights in a model, which is why we always try to translate the reasoning and the examples back into something that's written down in the policy where someone can see it, or at least they can see that, hey, this example is attached and this was the decision that should be made in these cases to reference the legal system again. Like, here's the scope to which this example applies. It doesn't apply to all cases. It only applies to cases that have this context. Here's how severe this case is, all that stuff. I think that was one of the hardest things for us to figure out how to do.

25:02And I will admit we had some really good design partners that we got to work with who I think not only helped us with the data that we needed to train that system to do what it needed to do, but also taught us about like what are customers willing to contribute to this process? Again, to like go back to my, you know, the best security system is one that people will use. So I think certainly I think that was one of the most challenging things to figure out. And to be honest, we're still figuring it out. We're still figuring it out every day. We're still trying to improve. And, you know, we're now handling video as well because AI's generate video.

25:39So we need to be able to process video and, you know, getting this getting the same process right for an element for a an artifact in video that now has a time access to it was a whole nother process. So definitely pretty complicated stuff.

25:58Kevin Werbach: What are the biggest areas that companies come to you with that are that are, you know, they get most excited about you being able to address what kind? I mean, I'm sure it varies industry to industry and company to company, but you see a pattern right now about what is the biggest driver? So I do think a lot of our customers, we learned this pretty early, you know, we're a vendor and I think many of them want us to be the experts. And I sort of understand that impulse. It's like I actually don't think every single engineer, every single product manager who is trying to deploy AI should have to be an expert in AI security and safety.

26:35And we ended up developing this thing that we just call it label library. And it basically has 60 plus policies that we've identified as kind of the lowest common denominator of security and safety problems that generally, I'm not going to say every single one applies to every single customer exactly the same. I think a lot of my argument is that business rules differ. But one of the things that we offer that I definitely do see customers kind of get excited about is plug us in on day one. We're fixing kind of a lot of the baseline stuff. So I definitely think that's one thing that we've seen customers just appreciate.

27:23the other area that we have seen a lot of frankly is customers coming to us kind of hair on fire because their payment providers are threatening essentially to cut them off to say look we just can't underwrite your business anymore we've seen that a lot in the AI image and video gem space and I think it's starting to change now with the EU governance that finally got released and stuff. But for quite a while, I really think the payment providers were kind of operating as this de facto standards and regulatory body, perhaps. And I think understandably, like, they obviously do not want to be underwriting the generation of illegal content, basically.

28:10And I think our ability, in many cases, to draw extremely fine lines, especially as customers engage in the feedback process, has been, I think, a real strength of ours, at least in that case. With a lot of our sort of more real-time applications, not that that's not real-time, but in cases where there is a human who is interacting with an AI and the expectation is that the AI does not take forever, basically, I think a lot of customers have appreciated and been pretty excited about how quickly we can do our work and how accurately we generally can do it. There is one thing that we're rolling out right now that I am very excited about, and we have a couple of customers testing for us.

28:59And it's basically, I've called it active guardrails in the past, but essentially the idea, I'm not a big believer that allow and block or allow and deny should be the only two actions. I think actually it's like a false binary there. And I think what it ends up doing, and we've actually heard some of the, I don't want to call anyone out by name, but we've heard some of the cases with AI friends and girlfriends where we've heard a CEO say, look, we didn't implement as many safety protocols as we could have because we were worried about the user experience. and I think I'm not going to say it's right but I think it's understandable perhaps if the only actions available to you are allow and block and the cost of a block is that the user experience breaks basically so we have essentially built something where now there's there's four actions basically there is allow there is deny of course there's escalate we can escalate to a human because I do think there are cases where we should say hey we don't know the right answer here and this one needs to get escalated I think that's reasonable but we also now have what we call rewrite or transform and the idea is well we know what your policy is supposed to be so if an AI is about to do something or say something dangerous or maybe just not so nice, why can't we rewrite that output before it gets to the user to let the experience continue but make the experience compliant at the same time?

30:48And I think at least recently, last couple of months, since we only built the infrastructure for this recently, that has been a common area of excitement that we've seen.

30:58Kevin Werbach: I mean, it makes a lot of sense why that would be valuable, but I would assume that's also somewhat scary that that a company is turning over to you that ability to essentially silently rewrite what's happening. Oh, yeah. Yeah. And I think this is why, you know, it's still in beta. It hasn't been widely rolled out yet. There's definitely going to need to be additional controls. yeah obviously we now take on responsibility for making sure that whatever we do output is also compliant with uh the policy um so it definitely is um for us an active area of like development and research and i i wouldn't i wouldn't claim it's ready yet yeah um but i think my argument is just that if we can make this work and make it work well um that it will encourage more companies that are deploying AI to put the appropriate controls in place, basically.

32:02Kevin Werbach: Yeah, it sort of raises for me as a lawyer this tension, and it's also true what you were describing before in terms of the auditability and explainability of how you're managing the policies, that on the one hand, organizations want to be accurate and they want to be able to have a set of rules and make a set of decisions, the fear is always the more you have this audit trail, then the more that can be the basis that a plaintiff or a regulator says, look, you knew exactly what you were doing and you're never going to get it right 100 % of the time. So obviously, these are open legal questions, these are responsibility questions, and the regulatory system is still evolving.

32:45Kevin Werbach: But how does that factor into what you're doing with companies? Is that concerned? I mean, I haven't had anyone complain about the auditability and observability yet. I certainly understand the concern. But I think generally like our because there is no maybe it will change, you know, as I'm a big believer, actually, that the industry needs regulation. And importantly, I think standards that are measurable and falsifiable in some way. It's up to the standard to define what we're looking for. It's up to the deployer to then decide what to do, I guess, when our standards are violated.

33:37but right now I think most of our auditability tools are actually being used by CISOs and CTOs to sort of understand the way the AI is behaving, what failure modes exist, how it needs to be improved, how many control events are actually occurring. I don't know, you've kind of got me on this one. I admit I haven't thought enough about the potential fear in companies around, oh, well, now I have this giant audit trail. What I will say is my hope is that we do get regulations that dictate what are the standards and what disclosure requirements are there, basically. Because certainly once that exists, we will implement it.

34:25And then it's not a matter of like, well, I'm afraid of having auditability because I could end up in a problem. Now it's just, OK, these are the requirements. So this is what I want to do. And I also believe it's important. Like one thing we have thought a little bit about is, you know, how could our tools be misused on some level? Right. Like, you know, we don't define the policy. The customer defines the policy. um and you could i guess i don't want to give anybody any ideas but like you you could use our tools to make your ai really bad just as you could use your tools to make it really good um and it's part of the reason why i do think disclosure is important because um the only control for like a company that's a bad actor is sunlight it's visibility right Right.

35:21And so like to me, that that is why I really would like it if, you know, the government would come along and say, look, here are the standards. This is what you have to do. Because otherwise it's a free for all. I was I know in one of your recent episodes, I think, what was it like around 10 percent of companies actually have like a governance framework or something like that versus around 50 percent who have, you know, who have a strategy for AI. Like that's a that's frightening to me that like, you know, yeah, yeah, no.

35:51Kevin Werbach: And I'm glad to hear you say that because I, yeah, there's always this knee jerk reaction that regulation is bad. And certainly there's, you know, there's lots of problematic regulation. But when I talk to companies, especially on on the deployment side of AI, you know, they don't want to be in this situation where you just don't know what the right answer is. And whatever they do, someone can say you got it wrong. So so I think that's an important insight that you have about the value that that can create for clarity. Yeah. I mean, I'm sure as a lawyer, like, I guess this is more your area than mine.

36:20But I, you know, now that you brought it up, it's like, I'm sure it is problematic to feel like these are non-deterministic systems. You're never going to get it right 100 % of the time. It's a, you know, we can strive for perfection, but everybody needs to accept, like, we're not going to get there. and I can certainly see where in that environment it's like well the more information that I have like the more risk there suddenly is that you know I get caught doing something that I didn't mean to do but then happened anyway I mean I think the advice I would give companies is is write standards that are actually measurable be clear about what your tendons are make them measurable.

37:02We have a little test we do basically where we say like, I'm actually about to run this exercise at the end of the month at Harvard. But if you write down a rule and you show that rule to end people and you get different answers, it's not consistent enough yet. It's that simple. So write standards that are actually measurable, that can be enforced on consistently. Decide what your actions are when standards are violated and then enforce those standards consistently. And I know that sounds like kind of simple, but I think like where I think meta got into trouble years ago was not being consistent about their enforcement, about their actioning, right?

37:48I think everyone can deal with the fact that some controls are necessary and that people make mistakes if they're being transparent about it. And they are at least, you know, doing their level best to enforce their own standards consistently. That's just my that's like my take and sort of the advice I would I would give anybody.

38:10Kevin Werbach: Yep. Terrific. Well, we have to wrap up, but it's been a fascinating conversation. And Brad, I really appreciate you joining me. No, thank you. It's it's been wonderful to talk. This has been The Road to Accountable AI. If you like what you're hearing, please give us a good review and check out my sub stack for more insights on AI accountability. Thank you for listening.

38:34Kevin Werbach: If you want to go deeper on AI governance, trust, and responsibility with me and other distinguished faculty of the world's top business school, sign up for the next cohort of Wharton's Strategies for Accountable AI Online Executive Education Program. Featuring live interaction with faculty, expert interviews, and custom-designed asynchronous content. Join fellow business leaders to learn valuable skills you can put to work in your organization. Visit execed.warden.upenn.edu slash ACAI for full details. I hope to see you there.