Forget Alignment, Here's Why Every AI Needs an Individual "Soul"

0:00Today on the AI Breakdown, we're reading an argument for why AI needs individuation. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information. Hey, hello friends. Welcome back to another AI Breakdown. Today we are doing a long reads, and this one is really interesting. This is a piece by author David Brin that was published in Wired. Now, David Brin is an interesting guy. He's written fiction and nonfiction. And the piece is titled, Give Every AI a Soul or Else. To solve the crisis, quote unquote, in artificial intelligence, AI beings must say I am me.

0:38Obviously, we're getting a bit philosophical here when it comes to our AI safety and risk questions, but I thought it was an interesting take, and I think that it will provoke a lot of thoughts among you guys as well. Now, if you are enjoying the AI breakdown, I have one little request. If you are a Twitter user, I would love it if you would share your favorite episode from this week or the last couple weeks and just tag me either at NLW or at the AI breakdown pod, and share it out so more people discover the show. Thanks in advance, and let's get to the AI breakdown. Again, we're reading Give Every AI a Soul or Else by David Brin.

1:14Havens in the field of artificial intelligence, including architects of notorious generative AI systems like ChatGPT, now publicly express shared dread of terrible outcomes that might be wrought by their own creations. Many now call for a moratorium or pause in AI development, allowing time for existing nations and institutions to innovate systems of control. Why this sudden wave of concern? Amid the toppling of many cliched assumptions, we've learned that so-called Turing tests are irrelevant, providing no insight at all into whether generative large language models, GLLMs, or Golems are actual sapient beings.

1:48They will feign personhood convincingly long before there's anything or anyone under the skull. Anyway, that distinction now appears less pressing than questions of good or bad or potentially lethal behavior. Some remain hopeful that a merging of organic and cybernetic talents will lead to what Reid Hoffman and Mark Andreessen have separately called amplification intelligence, or else we might stumble into lucky synergy with Richard Brodigan's Machines of Loving Grace. But worriers appear to be vastly more numerous, including many elite founders of a new center for AI safety who fret about rogue AI misbehaviors, from irksome all the way to existential threatening human survival.

2:23Some short-term remedies, like citizen protection regulations recently passed by the European Union, might help or at least offer reassurance. Tech pundit Yuval Noah Harari proposed a law that any work done by Golems or other AI must be so labeled. Others recommend heightened punishment for any crime that's committed with the aid of AI, as with a firearm. Of course, these are merely temporary palliatives. Let's be clear about whether any moratorium will slow down AI advances in the slightest. As expressed succinctly by Caltech cyber scientist Yasser Abu Mustafa, If you don't develop this technology, someone else will.

2:54Good guys will obey rules, the bad guys will not. Twas ever thus. Indeed, across the whole span of human history, just one method ever curbed bad behavior by villains, ranging from thieves to kings and feudal lords. I refer to a method that never worked perfectly and remains deeply flawed today, but it did at least constrain predation and cheating well enough to spur our recent civilization to new heights and many positive-sum outcomes. It is a method best described by one word, accountability. Those opining about synthetic intelligence today generally ignore lessons taught both by nature and by history.

3:28Nature, because as Sarah Walker explains in Noema, similar patterns can be found in the rise of earlier life forms across 4 billion years. Indeed, generative AI might be compared to an invasive species now spreading without constraint into a novel and naive ecosystem. An ecosystem based on new kinds of energy flows, one that consists of the internet plus millions of computers and billions of impressionable human minds. And history, because our own human past is rich with lessons taught by so many earlier tech-driven crises across 6 ,000 years. Times when we adapted well or failed to do so, e.g.

4:00the arrival of writing, printing, presses, radio, and so on. And again, only one thing ever limited predation by powerful humans exploiting new technologies to aggrandize their predatory power. That innovation was to flatten hierarchies and spur competition among elites in well-defined arenas. Markets, science, democracy, sports, courts. Arenas that were designed to minimize cheating and maximize positive-sum outcomes. Pitting lawyer versus lawyer, corporation versus corporation, expert versus expert, rich dude versus rich dude. It never worked perfectly. Indeed, the method is always, as now, threatened with subordination by cheaters.

4:33But flattened reciprocal competition is the only thing that ever has worked. Reciprocal competition is both how nature evolved us and how we became the first society creative enough to build AI. And if I sound like a scion of Adam Smith, sure. Smith despised cheater aristocrats and oligarchs, by the way. Might we apply to fast-emerging AI the same methods of reciprocal accountability that helped us tame the human tyrants and bullies who oppressed us in frivolous feudal cultures? Much will depend on the shape these new entities take, whether their structure or format is one that can abide by our rules, by our wants.

5:06Underneath all of the wrangling about how to control AI, we find three widely shared and seemingly contradictory assumptions. One, that these programs will be operated by a few monolithic entities, e.g. Microsoft, Google, China, Two Sigma, OpenAI. Two, that they'll be amorphously loose and infinitely divisible and replicable, spreading copies through every crack in the new cyber ecosystem. For a parallel, try that 1958 movie, The Blob. Three, that they will coalesce into a super macro entity, like the infamous Skynet of Terminator movies. All of these formats and more have been explored in very good and many bad science fiction tales.

5:40I've done stories or novels featuring all of them. And yet, none of the three offers a way out of our current dilemma. How to maximize positive outcomes from artificial intelligence while minimizing the flood of bad behaviors and harms we now see looming towards us at tsunami speed. Before looking for another way, consider what all three of the standard formats have in common. First, we needn't assume that these entities are yet autonomously conscious for them to be either productive or dangerous when used by human partners. We are already seeing harmful memes, counterfactual delusions, and even cult incantations generated on command from both within the castle institutions, format number one, and outside the walls.

6:16In fact, one of the most worrisome applications is to help our existing human elites evade accountability. Perhaps these three assumptions come so naturally to mind because they resemble failure modes from history. Format number one is very much like feudalism, and number two is, of course, chaos. The third resembles despotism by a cruel master or absolute monarch. But those fearsome echoes of our primitive past may not apply as AI grow in autonomy and power. And so we ask again, how can such beings be held accountable, especially when their speedy mental clout will soon be impossible for organic humans to track?

6:48Soon only AIs will be quick enough to catch other AIs that are engaged in cheating or lying. Um, duh. And so the answer should be obvious. Sick them on each other. Get them competing, even tattling or whistleblowing on each other. Only there's a rub. In order to get true reciprocal accountability via AI versus AI competition, the top necessity is to give them a truly separated sense of self or individuality. By individuation, I mean that each AI entity must have what author Werner Winge way back in 1981 called a true name and an address in the real world. As with every other kind of elite, these mighty beings must say, I am me.

7:23This is my ID and home root. And yes, I did that. Hence, I propose a new AI format for consideration. we should urgently incentivize AI entities to coalesce into discreetly defined, separated individuals of relatively equal competitive strength. Each such entity would benefit from having an identifiable true name or registration ID, plus a physical home for an operational referential kernel, possibly sole, and thereupon they would be incentivized to compete for rewards, essentially for detecting and denouncing those of their peers who behave in ways we deem insalubrious. And those behaviors do not even have to be defined in advance, as most AI mavens and regulators and politicians now demand.

8:01Not only does this approach farm out enforcement to entities who are inherently better capable of detecting and denouncing each other's problems or misdeeds, the method has another added advantage. It might continue to function even as these competing entities get smarter and smarter long after the regulatory tools used by organic humans and prescribed now by most AI experts lose all ability to keep up. Putting it differently, if none of us organics can keep up with the programs, then how about we recruit entities who inherently can keep up, because the watchers are made of the same stuff as the watched.

8:32One person working on AI individuation is Guy Huntington, a quote identity and authentication consultant who points out that various means of entity identification already exist online, though inadequate for the tasks looming before us. Huntington appraises a case study medbot, an advanced medical diagnosis AI who needs to access patient data and performs functions that might change in seconds, but who must leave an accountable trail that humans or other bot entities might appraise. Huntington discusses the practicality of registration when software entities spawn multitudinous copies and variants.

9:02He also considers ant-like usociality, where subcopies serve a macro entity like workers in a hive. He assumes that some kind of major institution must needs be set up to handle such an ID registration system, and that it can operate strictly as software. Personally, I am skeptical that a purely regulatory approach would work all by itself. First, because regulators require focus, widely shared political attention and consensus to enact, followed by implementation at the pace of organic human institutions, a sloth-slash-snail rate by the view of rapidly advancing cybernetic beings. Regulations can also be stymied by the free-rider problem, nations, corporations, and individuals, organic or otherwise, who see personal advantage in opting out of inconvenient cooperation.

9:42There is another problem with any version of individuation that is based entirely on some ID code. It can be spoofed, if not now, then by the next generation of cybernetic scoundrels, or the next. I see two possible solutions. First, establish ID on a blockchain ledger. That is very much the modern with it approach and it does secure in theory. Only that's the rub. It seems secure according to our present set of human parse theories. Theories that AI entities might surpass to a degree that leaves us cluelessly floundering. Another solution, a version of registration that's inherently harder to fool, would require AI entities with capabilities above a certain level to have their trust ID or individuation be anchored in physical reality.

10:19I envision, and note I am a physicist by training, not a cyberneticist, an agreement that all higher-level AI entities who seek trust should maintain a sole kernel, SK, in a specific piece of hardware memory, within what we quaintly used to call a particular computer. Yes, I know it seems old-fashioned to demand the instantiation of a program be restricted to a specific locale. And so, I am not doing that. Indeed, a vast portion, even a great majority of a cyber entity's operations must take place in far dispersed locations of work or play. Just as a human being's attention may not be aimed within their own organic brain, but at a distant hand or tool.

10:51So, the purpose of a program's sole kernel is similar to a driver's license in your wallet. It can be interrogated in order to prove that you are you. Likewise, a physically verified and vouched for SK can be pinged by clients, customers, or rival AIs to verify that a specific process is being performed by a valid, trusted, and individuated entity. With that ping verification from a permanently allocated computer site, others, people or AIs, would get reassurance they might hold that entity accountable, should it be accused or indicted or convicted of bad activity. And thus, malefactor entities might be adversarially held responsible via some form of due process.

11:24What form of due process? Jeez, do you think I am some super being who is capable of applying scales of justice to gods? The greatest wisdom I ever heard was uttered by Dirty Harry in Magnum Force. A man's got to know his limitations. So no, I won't define the courtroom or cop procedures for cybernetic immortals. What I do aim for is an arena within which AI entities might hold each other accountable, separately, as rivals, the way that human lawyers already do today. And yes, answering Yuval Harari's dread of mass human manipulation by persuasive golems, the solution for AI-driven mass meme hypnosis is for the mesmerizers to be detected, denounced, and neutralized by others with the same skills.

11:59Again, competitive individuation at least offers a chance this could happen. Whichever approach seems more feasible, Huntington's proposed central agency or a looser, adversarially accountable arena, the need grows more urgent by the day. As tech writer Pat Scannell has pointed out, each hour that passes, new attack vectors are being created that threaten not only the tech used in legal identities, but also the governance, business processes, and end users, be they humans or bots. What about cyber entities who operate below some arbitrary level of ability? We can demand that they be vouched for by some entity who is ranked higher and who has a sole kernel based in physical reality.

12:31I leave theological implications to others, but it is only basic decency for creators to take responsibility for their creations, no? This approach, demanding that AIs maintain a physically addressable kernel locus and a specific piece of hardware memory, could have flaws. Still, it is enforceable, despite slowness of regulation or the free rider problem. Because humans and institutions and friendly AIs can ping for ID kernel verification and refuse to do business with those who don't verify. Such refusal to do business could spread with far more agility than parliaments or agencies can adjust or enforce regulations.

13:00And any entity who loses its SK, say through tort or legal process or else disavowal by the host owner of the computer, will have to find another host who has public trust or else offer a new revised version of itself that seems plausibly better. Or else become an outlaw, never allowed on the streets or neighborhoods where decent folks, organic or synthetic, congregate. A final question. Why would these super smart beings cooperate? Well, for one thing, as pointed out by Vincent Cerf, none of those three older standard assumed formats can lead to AI citizenship. Think about it. We cannot give the vote or rights to any entity that's under tight control by a Wall Street bank or national government, nor to some supreme uber-skynet.

13:36And tell me how voting democracy would work for entities that can flow anywhere, divide, and make innumerable copies? Individuation, in limited numbers, might offer a workable solution, though. Again, the key thing I seek from individuation is not for all AI entities to be ruled by some central agency or by mollusk-slow human laws. Rather, I want these new kinds of uberminds encouraged and empowered to hold each other accountable the way we already, albeit imperfectly, do, by sniffing at each other's operations and schemes, then motivated to tattle or denounce when they spot bad stuff, a definition that might readjust to changing times, but that would at least keep getting input from organic biological humanity, especially they would feel incentives to denounce entities who refuse proper ID.

14:15If the right incentives are in place, say rewards for whistleblowing that grant more memory or processing power, or access to physical resources when some bad thing is stopped, then this kind of accountability rivalry might just keep pace, even as AI entities keep getting smarter and smarter. No bureaucratic agency could keep up at that point, but rivalry among them, tattling by equals, might. Above all, perhaps those super genius programs will realize it is in their own best interest to maintain a competitively accountable system, like the one that made ours the most successful of all human civilizations, one that evades both chaos and the wretched trap of monolithic power by kings or priesthoods, or corporate oligarchs or skynet monsters.

14:50The only civilization that, after millennia of dismally stupid rule by moronically narrow-minded centralized regimes, finally dispersed creativity and freedom and accountability widely enough to become truly inventive. Inventive enough to make wonderful new kinds of beings, like them. There you are. This has been a dissenter's view on what's actually needed in order to try for a soft landing. No airy or panicky calls for a moratorium that lacks any semblance of a practical agenda, neither optimism nor pessimism, only a proposal that we get there by using the same methods that got us here in the first place.

15:20Not preaching or embedding ethical codes that hyperentities will easily lawyer evade, the way human predators always evade the top-down codes of Leviticus, Hammurabi, or Kautama, but rather the Enlightenment approach, incentivizing the smartest members of civilization to keep an eye on each other on our behalf. I don't know that it will work, it's just the only thing that possibly can. All right, back to NLW here for just a very quick recap. Reading this, you can almost hear the pained cries of some of the AI safety people not being willing to surrender the premise of this piece, which is that we have no power and no agency to slow down this inevitable push to create beings that are smarter than ourselves.

15:57So for the sake of our discourse and for the sake of actually trying to get something from the piece, I would encourage us to put aside the fact that there is still much room for debate around whether we actually want to proceed in such a way that creates super intelligent AI. But let's assume David is right, and this is coming hell or high water. What I think is valuable about this piece, and more than this piece, this approach to thinking about this issue, is that it at least gets us into the realm of the applied rather than the theoretical. Yes, it's predicated on a theoretical assumption and zooming out to this period in which super intelligent AI has come to pass, but it's applied in the sense that this is a blueprint, or at least a starting point for what we could do if and when that happens.

16:38So if we're trying to map this against possible approaches to dealing with super intelligent AI, we have on the one hand, the cleanest strategy of just not building it. We have on the other hand, approaches that call for alignment. Obviously, I'm reading this to you in a week where we have OpenAI announcing their new super alignment team and a goal to align super intelligence within four years. This is a more middle space. And although he doesn't use this term, he invokes it when he invokes Adam Smith, this is sort of a market-based solution, although maybe not a market as we currently conceive them.

17:08I do think, holding aside any specifics of whether David's particular approach would or wouldn't work, that exploring what the relationship and incentives between and among different AIs will be is a potentially profitable place to spend some time. To the extent that our worst risk scenarios involve coordination and agreement between diverse sets of AIs, how realistic do we think that is? And are there ways to intercede that make different AIs, if not combative with each other, at least competitive with one another in ways that lead to positive outcomes for humans? When I read this, my overwhelming thought is that we are so barely scratching the surface of these questions that even for those who are mostly focused on that first approach to dealing with this problem, which is stopping before we develop superintelligent AI in the first place, it's still probably worth thinking about plan B's of what happens and what we do if superintelligent AI gets developed in spite of ourselves.

18:02Anyways, guys, I will wrap there. Plenty to chew on for your weekend. Thanks again for listening to the AI Breakdown. If you're enjoying it, please go check out the podcast version if you're watching on YouTube or the YouTube version if you're listening to the podcast. And until next time, peace.