Hamilton Mann: Separating the Powers of AI

0:08Kevin Werbach: Hi, I'm Kevin Werbach, Professor of Legal Studies and Business Ethics at the Wharton School of the University of Pennsylvania. For decades, I've studied emerging technologies from broadband to blockchain. Today, AI is promising to transform our world. But AI needs accountability, mechanisms to ensure it's developed and deployed in responsible, safe, and trustworthy ways. On this podcast, I speak with the experts leading the charge for accountable AI.

0:40Kevin Werbach: Underneath all the controversies about AI risks and regulation is a deeper question that is too often overlooked. What makes the algorithmic power of AI systems legitimate? My guest, Hamilton Mann, takes on that topic in his new book, Trias Algorithmica, What Code Rules? And he's also host of the Hamilton Mann Conversation podcast about AI for good and a frequent author on AI's implications. Welcome, Hamilton. Great to have you on the road to accountable AI. Great. Great to be here. Thank you very much for having me, Kevin. I actually first met you when our roles were reversed, and I was a guest on your podcast, The Hamilton Man Conversation.

1:25Kevin Werbach: So first, I wanted to ask you about what your experience has been. What do you think are the biggest things you've learned from the opportunity to go and talk to people in this format? Yeah, I think this is a good question to start. I think, you know, what has influenced my views most is not so much any individual interviews per se, taking in isolations, right? but the very much ongoing process of seeking out and exchanging with leaders and peers from very different fields. And I think this is where, let's say, you know, the value is quite very much tremendous, you know, talking with different people coming from different disciplines and, you know, asking the same questions so that you start to, let's say, you know, see emerge some kind of a relation between the thinking from one to another, even though they are, let's say, you know, navigating in very different disciplines.

2:13And sometimes this is very much a kind of opportunity to, let's say, start to say some kind of a new framework in a way or some kind of a new way of thinking about an issue. And it has been kind of a, let's say, a kind of a practice that I, let's say, you know, trying to, let's say, have a kind of a, as a kind of a EGN each and every day in my practice, whether as an executive or as an academic or so as well. So very much, let's say, you know, that kind of zoom in and zoom out ways of looking at issues, problems, and challenges from different fields, thanks to the help of interaction with peers coming from different disciplines.

2:50Kevin Werbach: We're going to get into your books in a moment, but I wanted to follow up a little bit on that and ask, what do you think then are the conversations that we're not having enough around AI in particular? Well, I think there is two, actually, and maybe we will have the opportunities to go deeper with the, let's say, you know, other kind of questions that you think about. But I think there is two, basically. One that has started to becoming mainstream, which is related to what we call alignment AI or misalignment AI, which was not mainstream at the very beginning. Or let's say when I'm thinking very beginning, I'm thinking 2022, 23.

3:32when the, let's say, you know, Chad GPT wave was, let's say, coming in our world, was not very much, let's say, something that we were talking about that much. Of course, we, you know, you had some academics and experts, ethical experts, governance experts that was very, let's say, eagerly searching and looking at that subject, but was not an headline and was not a kind of a public kind of attention at that time. And so this is very much something that starts to becoming quite a public, let's say, subject, topic. And this is good, actually, because from my perspective, this is part of the, let's say, big elephant in the room since the very, very beginning.

4:18The more powerful artificial intelligence becomes and the more powerful should also be the way it is, let's say, designed so that it can be safe, so that it can be aligned with the objective that the developers think about in the very first place. so that's the part that is quite let's say you know starting and becoming obvious for everyone still on that same category that's the subject that i'm calling integrity which is something that goes beyond alignment because alignment is very much about let's say you know thinking the way of prescribing from a development point of view and from a design point of view what is it What is it that is going to be the objective, the preference, the threshold, maybe the rules sometimes that the systems needs to comply with and needs to be aligned with?

5:12And so that we can, let's say, observe from a laboratory perspective that this alignment is maintained from the different ways the system is used and from the different ways the system is behaving, right? integrity go a step further because integrity is basically let's say this element of alignment with in addition the capabilities of having a system able to challenge its very own reasoning and its very own processing so we could say its very own intelligence which a question is very own, let's say, functioning and processing so that it can, let's say, be able to, from its, let's say, you know, way to reason to be capable to, let's say, deliver an outcome not just based on what it can do, but also based on what it should do.

6:14So it is like mimicking, basically, let's say, that very qualities that we may, let's say you know exhibit sometimes and i hope most of the time which is there is many things that we can do in societies as a human being we have agency for doing many things as long as you still have your freedom and your liberty you you really have let's say the potential to do many things in life but there is things that let's say you know we prevent ourself in a very autonomous way from doing it just because reflecting against the potential and the capabilities that we can exhibit versus the consequences of that potential in real life, whether it is for ourselves or for, let's say, people we love or people, let's say, we interact with, we make the rational and the very decision to not use our potential and not use our capabilities.

7:13And so this is something that work ex post and ex ante, right? Ex ante in the sense of, let's say, you know, before acting in something where we, let's say, you know, foresee that there will be a kind of irreversible, let's say, effect. We, let's say, you know, take the time to think about it and then the act, knowing that the action that we take, we are going to assume it. We are going to take that action for, let's say, you know, being accountable for it. And ex post is basically where we are acting on something, we're deciding on something, and we observe consequences. Sometimes part of those consequences has been, let's say, you know, something that we was thinking about from the very first place.

7:58And sometimes those consequences are very much surprising, right? This is something that was not expected. And unfortunately, it may happen that those consequences are quite, let's say, not fortunate, not the things that we wanted to, let's say, see. And so we learn from that experience. We think like, okay, next time if I had to take the same decision or something related to, I'm going to act differently. I'm going to behave differently. I'm going to ask for some more data and so on and so forth. so that capacity of let's say meta reflection it is what is i'm calling artificial integrity and this is basically what is quite missing nowadays whether it is in our conversation when it comes to the use of those ai systems and of course when it comes to let's say the public awareness and the headlines that let's say you know happens to organize some of the noise that who are seeing on AI today.

8:54But I think that the more powerful AI is going to become, and we think about, let's say, you know, the claim and let's say, you know, the point that has been raised recently by a few leaders in the Silicon Valley, the more the AI systems will be powerful and the more those capabilities, not just from a performance perspective, but from an integrity perspective, will need to be proportionate, will need to be even more proportionate, It will need to be beyond that, like what I'm calling artificial integrity over intelligence, right? Yes. Because again, and we know this for sure, right? And history has been kind of a good learning for that.

9:35This is not the more intelligent people or organization that, let's say, guarantee in societies to delivering the better and the good outcomes, right? and sometimes it happens that when you see people or organizations very clever, it happens that sometimes that cleverness might be used for doing very bad things. So I think this is part of the story where we still have to learn collectively in terms of being aware, in terms of being demanding from those people that create those systems. And I hope this is happening, but this is, let's say, you know, something that we are more laggard, I would say, nowadays.

10:14And the last but not least category of subject, which is the subject of my second book, it is, let's say, the responsibility that we all collectively have when it comes to organizing the integration of an AI system in societies. not only that the systems should be designed in some of specific ways, and I think regulation have a work to do in order to organize what is it going to be the standards in terms of how we design those kind of a system in a safe way, but also it also comes with the fact that we also need to think about our different institutions and look at what it means to have that new capabilities entering in our life and entering in our societies.

11:01And when I say entering in our societies, I mean, I'm thinking about entering in each and every industries with no exceptions, which means, in a way, challenging and pushing in some kind of a limits our economic systems or social systems or political systems, et cetera, and so on and so forth. So we need to think about what needs to be changed, what needs to be adapted, what needs to be adjusted in all different systems with a S to take into account that integration because it is not neutral and it comes with a kind of a different powers that we need to, let's say, organize and let's say, in some ways, think about the equilibrium that we need to, let's say, you know, navigate through.

11:47Kevin Werbach: So let's try to put this in practical terms. Your new book, Trias Algorithmica, looks back to the concept of separation of powers and the political philosopher Montesquieu. How does that actually manifest in practice when we're looking at these AI systems? Yeah, so I think in practice, this is quite, let's say, both surprising in a way and, let's say, very quite questioning also at the same time. Because it happens each and every day and we can observe and we can, let's say, experience that each and every day. but in a kind of a questioning ways with i feel like we are kind of a very timid and some somehow a bit naive about what is happening so in a very concrete way for example what we can experience is let's say a chatbot organizing let's say the information that we can let's say you know scroll and that we can grab for a kind of a prompt or different question that we ask and this answers this let's say deliverable that we consume through that interface is basically following some rules some rules that are basically form of let's say ways form of avenues that have been coded in the first place by the developers.

13:22Those rules can, let's say, you know, include some form of a safeguard, for example, saying that, okay, you, if we, let's say, flag this word or this term in the chat, we want the systems to have this kind of a behaviors, okay, which is kind of a good rules in a way, because we don't want to be, let's say, you know, overwhelmed with such a kind of, let's say, bad behaviors. But rules goes beyond that because it can also include if, for example, you have such a kind of a political, maybe sensitive questions, we want the systems to answer in a kind of a neutral way. And neutral way doesn't exist.

14:04So it means that someone has defined what is neutral from his perspective. And this is where I'm, let's say, saying that at that very moment, we define this through the code, through the process of training the system. We make the same kind of, let's say, you know, exercise as defining a law, as defining a rule, right? Only difference is that you don't happen to be able to read what has been defined as a law, as a rules. it is uncoded in the functioning of the systems okay and that the first let's say you know power that happens to be what i'm calling the algorithmic power the legislative like power there is a second thing that we can observe each and every day in fact shifting gears looking at for example what happened in social networks which also happens to be since a long time ago actually AI systems or so as well with the moderation type of, let's say, you know, very kind of a sophisticated mechanism that exists.

15:12You also have a kind of an executive-like power in motion in such a kind of a systems. And actually, this is what we love. This is what we feel like being the sherry in the cake, basically, right? As opposed to the legislative-like power that we don't see, we don't get the opportunities to have some scrutiny in terms of what are the rules that have been encoded in the first place. The executive power is basically why we adopt those systems. We adopt those systems because they are performant, and so we gain some kind of a way to save times. So it is like, you know, oh, please do me a favor and do that synthesis for me or do that, let's say, writing text for me.

15:56If you are in a kind of a social network, Oh, I'm going to post a message. So please reach out to all my contacts with that message. So that executive-like power is also relying on the algorithmic power. And the clue, I mean, let's say, you know, the beauty of the things is that the more that executive power is into play, is in motion, and the more we are happy. Because it is, let's say, you know, coming as a service, right? Right. So when you see, you know, in your favorite, maybe social network, for example. Or even if you are in the business case, you know, in the business, then you're sitting right at work.

16:36Right. And you see in your favorite, let's say, software, a kind of announcement from the provider saying, oh, look, guys, here's the point. We are we have been making a lot of work. Right. Now, I mean, those past four months and we have been upgrading the features. Right. So the features now is going to do this and that. So we hope that we guys are going to be happy with this because it's going to be more performance for you. We don't, let's say, stop and question and think about, okay, more performance, but in exchange of what? We don't ask questions. We don't, let's say, raise doubt, let's say, reflection, because it very much happens to be a service, a convenience, a way to achieve more things.

17:21but this is an executive-like power because this is a kind of a way to enforce the rules to enforce the legislative-like rules that i was talking about in the first place through algorithm right and the more the algorithm is performant the more it has that capacity to enforce the rules so for example in the social networks that enforcements can come from let's say the very fact that you know when you look at your let's say you know threads of message there is something that you will see something that you're not going to see right the thing that you're not going to see is basically let's say part of let's say the work of that algorithms making that work in terms of executive like power and organizing basically the set of information that needs to be presented in front of your eyes and so you feel happy because it looks like your preference it looks like the sub the subjects and the topic that you wanted to see about that you want us to know about but at some point you start to becoming let's say you know navigating in the loop that you didn't choose right because those preference start to like say becoming not very yours not very something that you have been asking for but something that you have been presenting with and do you and you adopt those preference as yours but at the very beginning this is not your preference this is the choice that has been made by the systems so this is the executive like power and last but not least the algorithms also simulate what i'm calling the judicial like power which is that very power that we see right in our societies where there is a conflict where there is let's say a need to let's say you know validate or invalidate that things has been done and things has been, let's say, you know, behave or act the right way, and right way being the legislative-like power that we have been establishing among the rules that we need to comply with.

19:20And so that judicial-like power is basically there to, let's say, you know, make some arbitration and, let's say, you know, make some kind of evaluation. And we hope that evaluation is basically not neutral, but somehow legitimate in a way, because we have been organizing that setting so that it comes with, let's say, a very separation between all the different other powers that are emotional in societies. the point is that when it comes to algorithms, more and more and more advanced algorithms fusing those three like power, legislative executive and judicial within the code and within their functioning and their behaving imposing to billions of people when you think about those most advanced systems that we are all using I'm saying all, of course this is let's say, you know, not all and each and every human in the earth, but billions of people are using it.

20:23And we are basically imposed rules that we have not decided for, that we did not vote for, some enforcement of those rules through those algorithms, and some judicial power from sometimes, like say, you know, things like, oh, I'm posting this, and I'm surprised because the assistance has decided by itself that what have been posted, for example, is not, let's say, meeting the compliance rules of the very system. So the post or, let's say, the message has been, let's say, removed, right? There is no contestation kind of mechanisms. There is no appeals. There is no, let's say, way to claim or, let's say, have a debate about it.

21:11The judicial power just executes, let's say, you know, based on the rules that he has said there to execute. So this is basically part of the conversation that is, from my perspective, not part of the table, not part of the conversation that needs to happen nowadays. And I think this is critical because the more, again, those systems, I mean, is becoming powerful, and the more those three power that is fused within algorithms are also becoming powerful.

21:43Kevin Werbach: What does that perspective lead to in terms of AI governance and regulation? Does it mean that we need to have systems that require that kind of contestation and democratic participation for everything that AI companies are doing? I think, so first, it means two things and two very simple things. The first thing, it means that what we have been doing since centuries of, let's say, you know, looking at those three powers, legislative, executive, and judicial, and thinking about how to separate those powers, right? This is something we need to look at when it comes to the design of algorithm and more broadly when it comes to the way we want to govern the algorithmic power in societies.

22:30We need to not, let's say, be naive and look at, let's say, you know, those systems for what they are, which is also a power, a new kind of a power, and look at, let's say, the way to, let's say, maintain a kind of equilibrium and a kind of a separation so that those three forms of power doesn't fuse in a code applying and deploying to billions of people. So that's number one. Number two, because separation is also, I mean, is necessary, but is not sufficient. We also need to think about the counter power, right? And the counter power is like also an invitation for us as societies to think about new roles that needs to be, let's say, you know, include in some existing institution.

23:23new institution as well because we might let's say also you know observe that at some point we need some very specialized institution that will act as a counter power when it comes to let's say the behaviors of some some some very advanced systems and also think about the constitutions right in terms of what does it mean in terms of right for human being what does it change or what needs to be adjust when it comes to, let's say, the constitutionalisms in terms of the organization of that power. I think this is the very topic to which we need to, let's say, advance and continue to, let's say, work and accelerate, actually, because if we look at, let's say, the advance that we see with those different kind of LLM models that we are using nowadays, it starts to becoming urgent, right, to think about the way to govern that power.

24:25Kevin Werbach: The structures that you talk about, though, traditionally have been thought of as ways to constrain public power and governments. And it's certainly true that especially these hyperscalers and frontier AI labs are extraordinarily powerful among any measure. but does it make sense to completely obliterate that distinction between how we think of governance and governments and how we think about private companies and and if so where do you draw the line because there's only a handful of frontier AI labs but there are thousands of AI companies and millions of companies deploying AI yeah yeah no but that's that's a very that's a very interesting point because I think like what we say in France we we call it doing things without knowing that while doing it, right?

25:12She's for sure. And we call it doing like Mr. Jordan, right? Separation of power is something that is a very foundation, not only that you see at, let's say, you know, public and government, let's say, you know, organization, but that you also see in corporation. So in private sector, in business, right? And this is critical. And actually, I will say that the separation of power, when it comes to business and when it comes to corporation is very accurate, is very, let's say, you know, very well working. So the first point is that the notion of separation of power is not limited to public and not limited to government.

25:55Let me take an example of corporation. For any business, no matter the industries, when you are, let's say, in private sectors, within your companies, you have a board of directors. And what the board of directors is, basically, those people are basically, let's say, you know, addressing and defining what the legislative power needs to be. What is the frame within which the companies have to thrive? This is a form of legislative power. And then you have the CEO and his team. and as the naming said ceo chief executive officer is basically what it means executive like power and so executive like power in the companies is taking let's say you know recommendations orientations coming from the board and starting to let's say make sense about this figuring out What is it that we need to execute to reach out different kind of goals that is expected by the board, actually?

27:06And so this is executive. So executive, when it comes to companies, it will take different forms. But basically, it is very vertical, right? This is what I'm saying, that it works very well. You have the CEO, the team of the CEO. Then you have business unit, leaders of business unit. The separation of power is clear. You have business unit A, B, C. Everybody is minding their own business, which is also a kind of a separation. And of course, if you take, for example, the leader of a P &L, he don't have, let's say, you know, even though he has a large autonomy to drive the destiny of its business or of its P &L, he don't have the power to decide by itself what is it that is going to be the rules to which he has to comply as a business what is it going to be the different kind of a decision that he needs to take from an executive point of view because some of those decisions doesn't happen to be in his perimeter sometimes he has to comes from the ceo or the excom members and he don't he don't have the power also as well when it comes to the judicial kind of a power think about the audit department the compliance department the legal department in any companies even though they are sitting within the wall of the companies they are very much acting and behaving as judicial power right when sometimes we feel like oh there is some kind of an issue whether it is a hr issue or business issue, and we need to, let's say, have a kind of a second view, very, let's say, detached from the day-to-day business and operation, we call for the audit department to go and scrutiny and see what happens, right?

28:54Most of the time, business don't like them, right? But this is kind of a separation, right? So that's one point. I think when it comes to, let's say, you know, algorithmic power, two takes. The first one is that if you, let's say, you know, sit at any company level, you're a CEO, you're members of the executives, let's say, you know, committees, I think leaders need to acknowledge the fact that when it comes to embracing that AI's, let's say, revolution, adopting AI systems, deploying those systems within the companies, whether it is for augmenting XYZ processes, right? or XYZ competencies within your workforce.

29:37This is not just, let's say, you know, a way to make the technological, let's say, stack entering into, let's say, the companies. It is also a change and a challenge of the authority stack. And so it means that it is, let's say, reshuffling the, let's say, card in terms of the distribution of power. And we've seen that, right? When you look at companies, the power is distributed because of that very separation of power that also exists at corporate level, right? We call it governance, actually. And basically, you've seen companies adopting AI, deploying systems, whether it is for their supply chain, for their customer services, and so on and so forth.

30:30and in the middle of their journey start to let's say you know stop pause reflect and say oh there is something happening there we feel like we don't master anymore the operating model that is going on here because the processes the procedures that we know and that describes how things need to be done, how things need to be achieved, start to becoming not the only reference point in terms of how things need to be delivered and how things need to be done. And at the point where what you know as the norm, as the standard, as the processes, as the procedures, start to not becoming the only reference points to which each and every people, let's say, you know, look out and orchestrate their different activities, you start to, let's say, you know, see your operating model drifting from the mastership that you need, right?

31:35Kevin Werbach: Yes. So concretely, it might be having, you know, a world department. I like to take some time that example, right? Your marketing department is very keen and is very eager to use the AI system. So what they did, they took some licenses, whether it is Cloud, ChatGP, or Copilot, whatever. They took some licenses and they generate images and videos. Before, they was, let's say, doing the same work, but it was taking two months, three months, for example. Now with those, let's say, you know, new capabilities, it takes seconds. So what happens? They are very happy because they have, let's say, you know, a whole new capabilities of creating images, videos for doing campaigns.

32:19The thing is that in the process, they need to, let's say, at some point, go to maybe communication departments that needs to validate whether those different, let's say, you know, images and ways to present the companies comply with the branding standards. Right. And this is a nightmare for those guys. Why? because they're not staffed to let's say you know look at look at all those images and videos right it was doing the work for let's say you know a pipeline of maybe 10 videos per month now you overwhelm this very same department with the very same workforce with hundreds so this is where you start to see your operating models drifting because what happened sometimes the management is not aware about that situation right at least not aware at the very moment that situation is happening and you see behaving uh let's say you know some kind of a behaving starting to becoming a kind of a very problems in a ways one example is that okay the communication departments which look at let's say the different hundreds of videos they don't know how to do it with two ends so they are going to rely on air right so they are going to go for licenses deploy some ais to check whether it is compliance with the brain's attributes the issue is that ais sometimes hallucinate right sure so sometimes i will say yes of course green light this is compliant with the attributes but it's not first let's say you know issues second issues they don't have the budget to do so because you know in corporation budget is not equally equally let's say you know distributed and so they are going to be the bottleneck of that very systems and so what happens is that you will see many campaigns or many let's say you know things that is plans to be delivered so to support the business to be stuck in the pipeline just because we don't have time to deal with the control of those things so just simple real life example to just see that at some point as a leaders when you deploy such a kind of a systems you need to be aware not only from the kind of a performance and capabilities and productivity gain perspective right about what is it going to be the advantage but you need also to look at what is it going to be the redistribution of power between all departments, all activities.

34:55If you give, if it gives more autonomy at one side of the organization, what does it mean as a kind of a, let's say, Givens effect in the other sides of the organizations and how from an end-to-end perspective, you look at the total cost of it, whether it is a total cost in terms of culture, a total cost in terms of cost, of course, in terms of, let's say, of course, a way to address the different challenges from a kind of a business perspective. Total cost also in terms of, let's say, perception, whether it is satisfaction of your customers and so on and so forth. And so to me, the point is that the authority stack is as much at stake than the technological stack when it comes to deploying such a kind of a systems.

35:47And so you need to revisit in a way the, let's say, organization and the distribution of the power of your organization when you do such a kind of a move and when you deploy AI systems.

36:02Kevin Werbach: Fascinating. There's so much more I want to go into with you, but unfortunately, we don't have any more time for this podcast. But you've given us a wonderful taste and I hope people will buy your new book and get into it more deeply. So I just want to thank you, Hamilton, for being with me. Thank you very much, Kevin, for that conversation. This has been The Road to Accountable AI. If you like what you're hearing, please give us a good review and check out my substack for more insights on AI accountability. Thank you for listening.

36:34Kevin Werbach: If you want to go deeper on AI governance, trust, and responsibility with me and other distinguished faculty of the World's Top Business School, sign up for the next cohort of Wharton's Strategies for Accountable AI Online Executive Education Program. Featuring live interaction with faculty, expert interviews, and custom-designed asynchronous content. Join fellow business leaders to learn valuable skills you can put to work in your organization. Visit execed.wharton.upenn.edu slash ACAI for full details. I hope to see you there.

37:11Thank you.