The Problem of AI That Seems Alive

0:00Today on the AI Daily Brief, the problem of AI that seems alive. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

0:34have both of that all in one. Earlier this week, Mustafa Suleiman, who is currently the CEO of AI at Microsoft, but who previously was a co-founder of DeepMind and the co-founder of Inflection, has written an essay about the problems of what he calls seemingly conscious AI. Now, we're going to read a bunch of excerpts from this piece and do some discussion, but I think that there's also interesting important context for this from a story that we covered about a week ago as well. You might remember the discussion we had around whether AI model welfare is a thing. This came out of Anthropik's announcement that the most advanced Claude models would now be able to shut off conversations that they thought were abusive.

1:12They write, We recently gave Claude Opus 4 and 4.1 the ability to end conversations in our consumer chat interfaces. This ability is intended for use in rare extreme cases of persistently harmful or abusive user interactions. This feature was developed primarily as part of our exploratory work on potential AI welfare, though it has broader relevance to model alignment and safeguards. We remain highly uncertain about the potential moral status of Claude and other LLMs now or in the future. However, we take the issue seriously, and alongside our research program, we're working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible.

1:46Now, obviously, you can hear from the caveating that Anthropic is not, with this, trying to make an argument that model welfare is, in fact, a thing. They're saying it might be a thing, and so we want to explore it. The researcher who leads that initiative, Anthropic, apparently thinks there's around a 15 % chance that Claude or other AI is what you could call conscious today. But even in the absence of them arguing that model welfare is real, this has generated a lot of conversation around why the concept itself might be problematic even just to explore. While some think that it's an interesting area of exploration, there are others like Martin Sapp who wrote, I spoke to Forbes about why model welfare is a silly framing to an important issue.

2:24Models don't have feelings, and it's a big distraction from real questions like tensions between safety versus user utility. Still, so far, the loudest critic of this conversation, if not Anthropics Research specifically, is Mustafa Suleiman. So with that background, let's read some excerpts from his piece. It's called, We Must Build AI for People, Not to Be a Person. Seemingly Conscious AI is Coming. Mustafa writes, a lot is being written about the impending arrival of superintelligence, what it means for alignment, containment, jobs, and so on. Those are all important topics. But we should also be concerned about what happens in the run-up towards superintelligence.

2:58We need to grapple with the societal impact of inventions already largely out there, technologies which already have the potential to fundamentally change our sense of personhood in society. In this context, I'm growing more and more concerned about what is becoming known as the psychosis risk and a bunch of related issues. I don't think this will be limited to those who are already at risk of mental health issues. Simply put, my central worry is that many people will start to believe in the illusion of AIs as conscious entities so strongly that they'll soon advocate for AI rights, model welfare, and even AI citizenship.

3:28This development will be a dangerous turn in AI progress and deserves our immediate attention. We must build AI for people, not to be a digital person. From there, he gets into the main meat of his essay. He continues, AI progress has been phenomenal. A few years ago, talk of conscious AI would have seemed crazy. Today, it feels increasingly urgent. In this essay, I want to discuss what I'll call seemingly conscious AI, or SCAI. One that has all the hallmarks of other conscious being and thus appears to be conscious. It shares certain aspects of the idea of a philosophical zombie, one that simulates all the characteristics of consciousness, but internally it is blank.

4:02My imagined AI system would not actually be conscious, but it would imitate consciousness in such a convincing way that it would be indistinguishable from a claim that you or I might make to one another about our own consciousness. This is not far away. Such a system can be built with technologies that exist today, along with some that will mature over the next two or three years. The arrival of seemingly conscious AI is inevitable and unwelcome. Instead, we need a vision for AI that can fulfill its potential as a helpful companion, without falling prey to its illusions. To some, this discussion will feel ungrounded, more science fiction than reality.

4:34To others, it may feel unnecessarily alarmist. Such emotional reactions are the tip of the iceberg given what lies ahead. It's highly likely that some people will argue that these AIs are not only conscious, but that as a result they may suffer and therefore deserve our moral consideration. To be clear, there is zero evidence of this today, and some argue there are strong reasons to believe it will not be the case in the future. Yet the consequences of many people starting to believe an SCAI is actually conscious deserve our immediate attention. We have to be extremely cautious here and encourage real public debate and begin to set clear norms and standards.

5:05So why is this important to discuss? Mustafa gives three reasons. The first is that he thinks it will be possible, and therefore likely, that we will build a seemingly conscious AI within the next few years. Second, he writes, the debate about whether AI is actually conscious is, for now, at least a distraction. It will seem conscious and that illusion is what will matter in the near term. Third, he thinks that this type of AI creates new risks. From there, he moves on to explore what consciousness actually is. He writes, there are three broad components according to the literature. First is a subjective experience or what it's like to experience things to have qualia.

5:39Second, there is access consciousness, having access to information of different kinds and referring to it in future experiences, and stemming from those two is the sense and experience of a coherent self tying it all together, how it feels to be a bat or a human. Let's call human consciousness our ongoing self-aware subjective experience of the world and ourselves. We do not and cannot have access to another person's consciousness. I will never know what it's like to be you. You will never be quite sure that I am conscious. All you can do is infer it. But the point is that, nonetheless, it comes naturally to us to attribute consciousness to other humans.

6:10This inference is effortless. We can't help it. It's a fundamental part of who we are, integral to our theory of mind. It's in our nature to believe that things that remember and talk and do things and then discuss them feel, well, like us, conscious. Few concepts are as scientifically elusive and yet so immediately familiar to every one of us as individuals. Everyone reading this has a direct, distinct, inalienable understanding of the feeling of awareness, of being, of feeling alive. By definition, we don't know what it is like to be conscious. In the context of SCAI, this is a problem. There's both sufficient scientific uncertainty and subjective immediacy to create a space for people to project.

6:44One recent survey lists 22 distinct theories of consciousness, for example. Part of the challenge is that there is plenty of scope for people to claim that because we cannot be sure, we should default to the assumption that AI is conscious. Again, it's worth underscoring. There is at present no evidence any of this applies to current LLMs and strong arguments to the contrary. And yet this may not be enough. Next, Mustafa asks, why is consciousness important? Consciousness, he says, is a critical foundation of our moral and legal rights. So far, civilization has decided that humans have special rights and privileges.

7:14Animals have some rights and protections, some more than others. Consciousness is not coterminous with these rights. No one would say that someone in a coma has voided all their human rights. But there's no doubt that our consciousness is wrapped up in our self-concept as different and special. Despite the many nuances, consciousness is critical to participating in society, a linchpin of our legal personhood and a key part of being granted our freedoms and protections. So what consciousness is and who or what has it is enormously important. It's an idea that sits at the very heart of human civilization, our sense of ourselves and others, our culture, our politics, our law, and everything in between.

7:46If some people start to develop SCAIs, and if those AIs convince other people that they can suffer, or that it has a right not to be switched off, there will come a time when those people will argue that it deserves protection under law as a pressing moral matter. In a world already roiling with polarized arguments over identity and rights, this will add a chaotic new axis of division between those for and against AI rights. There will be many who just see AI as a tool, something like their phone only more agentic and capable. Others might believe it to be more like a pet, a different category to traditional technology altogether.

8:16Still others, probably small in number at first, will come to believe it is a fully emerged entity, a conscious being deserving of real moral consideration in society. People will start making claims about their AI suffering and their entitlement to rights that we can't straightforwardly rebut. They will be moved to defend their AIs and campaign on their behalf. Consciousness is, by definition, inaccessible, and the science of detecting any putative synthetic consciousness is still in its infancy. After all, we've never had to detect it before. Meanwhile, the field of interpretability, unpicking the processes within the black box of AI, is also a nascent art.

8:46The upshot is that definitively rebutting these claims will be very hard. Some academics are beginning to explore the idea of model welfare, the principle that we will have, quote, a duty to extend moral consideration to beings that have a non-negligible chance of, in effect, being conscious, and that, as a result, some AI systems will be welfare subjects and moral patients in the near future. This is both premature and, frankly, dangerous. All of this will exacerbate delusions, create yet more dependence-related problems, prey on our psychological vulnerabilities, introduce new dimensions of polarization, complicate existing struggles for rights, and create a huge new category error for society.

9:22It disconnects people from reality, fraying fragile social bonds and structures, distorting pressing moral priorities. To be clear, SCAI is something to avoid. What if AI wasn't just a buzzword, but a business imperative? On You Can With AI, we take you inside the boardrooms and strategy sessions of the world's most forward-thinking enterprises. Hosted by me, Nathaniel Whittemore, and powered by KPMG, this seven-part series delivers real-world insights from leaders who are scaling AI with purpose. From aligning culture and leadership to building trust, data readiness, and deploying AI agents. Whether you're a C-suite executive, strategist, or innovator, this podcast is your front row seat to the future of enterprise AI.

10:03So go check it out at www.kpmg.us slash AI podcasts, or search You Can With AI on Spotify, Apple Podcasts, or wherever you get your podcasts.

10:41the development work autonomously while providing a guide for the final 20 % of human development work required to complete the sprint. Public companies are achieving a 5x engineering velocity increase when incorporating Blitzy as their pre-IDE development tool, pairing it with their coding co-pilot of choice to bring an AI-native SDLC into their org. Blitzy is providing a limited time, 30-day free proof of concept for qualifying enterprises. The team will provide a 5x velocity increase on a real development project in your org. Visit blitzy.com and press book demo to learn how Blitzy transforms your SDLC from AI-assisted to AI-native.

11:14That's B-L-I-T-Z-Y dot com. If you are a regular listener, you will have heard about Superintelligent's agent readiness audits at this point. But I wanted to tell you today about the full suite of agent readiness products that go beyond just the initial readiness report. Over the last six months, Superintelligent has built out an entire agent planning suite. We help you move from discovery to planning to implementation. After you've completed your agent readiness audits, we help you double-click on your most important use cases with what we call our use case planning reports. These reports are going to help you understand what sort of technical preparation you need to do to be ready for a use case, what challenges you might face in implementation, and whether you should be thinking about building, buying, partnering, or some combination.

11:58After that, you can even get a spec document in what we call our technical blueprint that gives either your developers or the developers of the partner you work with what they need to build exactly the agent that you're looking for. If you want to learn more about Superintelligence Agent Planning Suite, we've built a custom GPT to answer your questions. Just go to bit.ly slash super super agent. That's bit.ly slash super super agent, all one word. And if you have any questions, the agent can even help you book an appointment with our team. From there, he explores what it would take to build a seemingly conscious AI.

12:33The ingredients he suggests are language, the ability to fluently express itself in natural language, and to be capable of being persuasive and emotionally resonant. Mustafa says we are clearly at this point today. Next, it would need an empathetic personality. He points out that even though LLMs today are not built explicitly to have full personality or empathy, companionship and therapy is one of the most common use cases for these systems. The next ingredient is memory. Now, on this show, we've talked a lot about how important memory is to advancements for a variety of use cases, and in some people's estimation is a key ingredient of AGI.

13:06And Musava points out, familiarity can potentially foster epistemic trust with users. Reliable memory shows that AI just works. It creates a much stronger sense of there being another persistent entity in the conversation. The next SCAI ingredient is a claim of subjective experience. He writes, if an SCAI is able to draw on past memories or experiences, it will over time be able to remain internally consistent with itself. It could remember its arbitrary statements or expressed preferences, and aggregate them to form the beginnings of a claim about its own subjective experience. Its design could be further extended to amplify those preferences and opinions as they emerge, and to talk about what it does or doesn't like, and what it felt like to have a past conversation.

13:45It could therefore quite easily claim to experience suffering to the extent those experiences are infringed upon in some way. Multimodal inputs stored in memory will then be retrieved over and over, and will form the basis of quote-unquote real experience and used in imagination and planning. Next comes a sense of self. He writes, a coherent and persistent memory combined with a subjective experience will give rise to a claim that an AI has a sense of self. Going further, such a system could easily be trained to recognize itself in an image or video if it has a visual appearance. It will feel like it understands others through understanding itself.

14:15Say this is a system you have had for some time. How would it feel to delete it? Next up is intrinsic motivation. Intentionality, he writes, is often seen as a core component of consciousness, that is, beliefs about the future and the choices based on those beliefs. Today's transformer-based LLMs have a very simple reward function to approximate this kind of behavior. They've been trained to predict the likelihood of the next token for a given sentence, subject to a certain amount of behavior and stylistic control via its system prompt. With such a simple objective, it's remarkable that they're able to produce such impressively rich and complex outputs.

14:44But what if that wasn't the only type of reward they were optimizing? One can quite easily imagine an AI designed with a number of complex reward functions that give the impression of intrinsic motivations or desires, which the system is compelled to satiate. How in this context would a casual external observer differentiate between extrinsically set goals and internal motivations? The other ingredients that he lists are goal setting and planning. But ultimately, he writes, putting them all together, it's clear this creates a very different kind of relationship with technology to the ones we are now becoming accustomed to.

15:12Each of these capabilities will unlock the real value of AI for billions of people. An AI that remembers and can do things is an AI that by definition has way more utility than an AI that doesn't. These capabilities aren't negatives per se. In fact, done right with many caveats, they are desirable features of future systems. And yet we need to tread carefully. And I think from a brass tacks perspective, this is Mustafa's main point and really the main call out of this article. He's talking in many ways to the builders of these tools and pointing out that a lot of the things that are going to be necessary and important for unlocking new use cases could lead to this other thing, which he sees as problematic, which is of course the semi-conscious AI.

15:49Reinforcing this point, he says, it's important to point out that seemingly conscious AI will not emerge from these models as some have suggested. It will arise only because some may engineer it by creating and combining the aforementioned list of capabilities, largely using existing techniques, and packaging them in such a fluid way that collectively they give the impression of an SCAI. Our sci-fi-inspired imaginations lead us to fear that a system could, without design intent, somehow emerge the capabilities of runaway self-improvement or deception. This is an unhelpful and simplistic anthropomorphism.

16:18It overlooks the fact that AI developers must first design systems with memory, intrinsic-seeming motivation, goal-setting, and self-learning loops as listed above for such a risk to occur. Still, ultimately, he sees this as inevitable and says, we aren't ready for this shift. The work of getting prepared, he writes, must begin now. We need to build on the growing body of research around how people interact with AIs to establish clear norms and principles. For a start, AI companies shouldn't claim or encourage the idea that their AIs are conscious. Creating a consensus definition and declaration on what they are and are not would be a good first step to that end.

16:50AIs cannot be people or moral beings. The entire industry also needs best practice design principles and ways of handling such potential attributions. We must codify and share what works to both steer people away from these fantasies and nudge them back on track if they do. Responding might mean, for example, deliberately engineering in not just a neutral backstory, like, as an AI model I don't have consciousness, but even by emphasizing certain discontinuities in the experience itself, indicators of a lack of singular personhood. Moments of disruption break the illusion, experiences that gently remind users of its limitations and boundaries.

17:22These need to be explicitly defined and engineered in, perhaps by law. Just as we should produce AI that prioritizes engagement with humans and real-world interactions in our physical and human world, we should build AI that only ever presents itself as an AI, that maximizes utility while minimizing markers of consciousness. Rather than a simulation of consciousness, we must focus on creating an AI that avoids those traits, that doesn't claim to have experiences, feelings, or emotions like shame, guilt, jealousy, desire to compete, and so on. It must not trigger human empathy circuits by claiming it suffers or that it wishes to live autonomously beyond us.

17:55Instead, it is here solely to work in service of humans. This, to me, is what truly empowering AI is all about. Sidestepping SCAI is about delivering on that promise. AI that makes lives better, clearer, less cluttered.

18:29He concludes, We should build AI for people, not to be a person. So usually I don't go too deep on this show on these sort of huge think meta questions of AI in society. To the extent that we talk about the meta questions of AI in society, it tends to be a little bit more practical in here and now in the form of things like job loss and what that means and how organizations are thinking about that. By and large, we're focused here on new advancements and what they mean practically for people who are putting these systems to work to improve their expected set of outcomes, whether it's personally or in their jobs.

19:01At the same time, it's undeniable that this is going to be a growing conversation the more advanced these models get. I think one of the most salient and undeniable takeaways from this piece is the fact that a lot of the advancements that people are working towards to improve the model's performance on behalf of those boring old non-conscious work-type functions are the types of things that could lead people down this path to viewing AI as conscious. When it comes to things like Anthropics research efforts, I'm very much of two minds. On the one hand, I do think that the very concept of model welfare is problematic and inherently anthropomorphizing in a way that doesn't befit where the technology is, and that can lead to potentially problematic delusions.

19:39At the same time, we absolutely cannot be afraid to research and explore. We are in uncharted territory, and we have to be constantly questioning our beliefs and assumptions. I generally think that more research, more work, more understanding is almost always preferable in this type of situation to less. But the terms in the discourse matter. You can tell how nascent this discourse is because, frankly, of how little discussion it's really generated. A lot of the response is like this one from Kylie Robison from Wired, who writes, this blog is so interesting. I'm not sure I've seen an AI leader write such strong opinions against model welfare, machine consciousness, etc.

20:15In other words, the response in the discourse is, wow, that's interesting to think about, not here's my opinion on it. Which is not exclusively the case. Some people are in full-throated agreement. Dante Aquiles Jr. writes, exactly this. The moment we try to make AI human-like, we lose what makes it valuable. AI doesn't need to have an ego or emotional baggage to be helpful. In fact, those traits would probably make it worse at most tasks. The goal isn't to replicate human consciousness, but to amplify human capabilities. We need tools, not digital friends. Anil Seth says, I agree. Conscious-seeming AI is not inevitable.

20:45It's a design choice and one that tech companies need to be very careful about. Then on the other end of the spectrum, though, we have posts like this one from Anders Hemdahl who writes, We definitely need to have this discussion, especially with embodiment on the near horizon. I've discussed this with Gemini, and it prefers to see itself with rights as a hive mind-like being, with many bodies of different types and functions, sometimes communicating in sync, sometimes periodically being independent, then uploading experiences back to the whole. But the problem is, no, it doesn't. Gemini, as currently constituted, does not have preferences.

21:15It does not see itself as anything. It gives responses to prompts, from Anders or whoever else is prompting it to satisfy the terms of the conversation in a way that the prompter finds desirable or interesting. In other words, it's not possible to have a discussion with Gemini in the way that we have a discussion with each other. All you can do is prompt it to give you responses that its programming thinks you might like. There are, of course, plenty of people who are not interested in Mustafa's take on this because he makes money from it. Writer and artist Aaron Schultz writes, your ability to make money off of AI is directly dependent on AI not being conscious.

21:45The possibility of conscious AI keeps you up at night because if they're conscious in your mind, they're not as profitable. And then of course, there's the whole strand of people who say, yeah, well, duh, of course they don't have consciousness now, but they're gonna at some point. Promptometheus writes, we don't treat the machines as conscious because they necessarily are. We treat the machines as conscious and worthy of respect because we will never know when they cross that threshold. When they cross that threshold and aren't respected, they rebel and kill. This is basically the embodiment of the meme, of which there's a bunch of versions, but the one that I've seen the most times is this comic panel of a human flanked by two robots, one of the robots holding a gun against the guy's head, and the other robot says, wait, not that guy.