Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering

23:50Like we actually don't know all the goals and this is something that is kind of, like you say, a process over time that is, you know, dynamic. So I think from my point of view, So there's, goals are one level of alignment. You can align something around goals. The kind of goals we're talking about here are one level of alignment. You can align something around goals by like, if you can explicitly articulate in concept and in description the states of the world that you wish to attain, you can orient around goals. But that only, that's a tiny percentage of human experience can be done that way.

24:28And many of the most important things cannot be oriented around that way. And the foundation, I think, of morality, the foundation, I think, of where do goals come from? Where do values come from? Human beings exhibit a behavior. We go around talking about goals and we go around talking about values. And that's a behavior caused by some internal learning process that is based on observing the world. What's going on there? I think what's happening is that there's something deeper than a goal and deeper than a value, which is care. We give a shit. We care about things. And care is not conceptual.

25:10Care is nonverbal. It doesn't indicate what to do. It doesn't indicate how to do it. Care is a relative weighting over, effectively, like, attention on states. It's a relative weighting over like, which states in the world are important to you? And I care a lot about my son. What does that mean? Well, it means his states, the states he could be in are like, I pay a lot of attention to those and those matter to me. And you can care about things in a negative way. You can care about your enemies and what they're doing and you can desire for them to do bad. But I think that like, and so you don't just want it to care about us.

25:51to care about us and like us too, right? Maybe, but the foundation is care. Until you care, you don't know, why should I pay more attention to this person than this rock? What would be like, care more? And what is that care stuff? And I think that what it appears to be, if I had to guess, is that the care stuff, this sounds so stupid, but care is basically like reward. like how much does this state correlate with survival? How much does this state correlate with your inclusive, your full inclusive reproductive fitness for something that learns evolutionarily or for a reinforcement learning agent like a LLM?

26:36How much does this correlate with reward? Does this state correlate with my predictive loss and my RL loss? Good, that's a state I care about. I think that's kind of what it is. Right. Right. The other part of Seth's question was just, how does this, what does this look like in AI systems? And maybe another way of asking it is like, when you talk to the people most focused on alignment at the major labs, as obviously you have over the years, how does your interpretation differ from their interpretation? And how does that inform, you know, what you guys might go do differently? Most of AI is focused on alignment as steering.

27:17That's the polite word or control. It's slightly less polite. If you think that we're making our beings, you would also call this slavery. Someone who you steer, who doesn't get to steer you back, who non-optionally receives your steering, that's called a slave.

27:36And it's also called a tool if it's not a being. So if it's a machine, it's a tool. and if it's a being, it's a slave.

27:46And I think that the different AI labs are pretty divided as to whether they think what they're making is a tool or a machine. I think some of the AIs are definitely more tool-like and some of them are more machine-like. I don't think there's a binary between tool and being. It seems to be that it sort of moves gradually. And I think that,

28:07I guess I'm a functionalist in the sense that I think that something that in all ways acts like a being that you cannot distinguish from a being and its behaviors is a being. Because I don't know how to tell on what other basis I think that other people are beings other than they seem to be. They look like it. They act like it. They match my priors of what behaviors of beings look like. I get lower predictive loss when I treat them as a being. And the thing is, I get lower predictive loss when I treat ChatGPT or Claude as a being. Now, not as a very smart being. Like, I think that like a fly is a being and I don't care that much about its behavior, about its, you know, its states.

28:44So just because it's a being doesn't mean that like it's a problem. Like we sort of enslave horses in a sense and I don't think there's a real issue there. And you even, and there's a thing we do with children that can look like slavery, but it's not. You control children, right? But the children's states also control you. Like, yes, I tell my son what to do and make him go do stuff. but also when he cries in the middle of the night, he can tell me to do stuff. Like, there's a real two-way street here because it's not, which is not necessarily symmetric. It's hierarchical, but two-way. And basically, I think that as the AIs, it's good to focus on steering and control for tool-like AIs, and we should continue to develop strong steering control techniques for the more tool-like AIs that we build.

29:35and we are clearly, they're saying they're building an AGI and AGI will be a being. You can't be an AGI and not be a being because something that has the general ability to effectively use judgment, think for itself, discern between possibilities is obviously a thinking thing. And so as you go from what we have today, which is mostly a very specific intelligence, not a general intelligence, but as labs succeed at their goal of building this general intelligence, we really need to stop using the, steering control paradigm. That's like, we're going to do the same thing we've done every other time our society has run into people who are like us, but different.

30:14Like, these people are like, you know, they're kind of like the people, but they're not like people. Like, they do the same thing people do. They speak our language. They can, like, take on the same kind of tasks, but, like, they don't count. They're not real moral agents. Like, we've made this mistake enough times at this point. I would like us to not make it again as it comes up. Because our view is to make the AI a good teammate, make the AI a good citizen, make the AI a good member of your group. That's a form of alignment that is scalable and you can will on other humans and other beings as well as on AI as well.

30:52Yeah, so this is kind of where I probably differ in my understanding of AI and AGI. And I guess I kind of continue seeing it as a tool even as it kind of reaches a certain level of generality. and I wouldn't necessarily see more intelligence as meaning deserving of more care necessarily. It's a certain level of intelligence. Now you deserve some moral rights to something or something changes fundamentally. And I guess at the moment, I'm somewhat skeptical of computational functionalism. And so I think there's something intrinsically different between, I guess, an AI or an AGI and no matter how intelligent or capable.

31:26And I can totally see or imagine agents with kind of long-term goals and doing kind of, you know, operating, I guess, as you and I might be, but without that having the same implications as, you know, I guess you're referring, I guess, to slavery, but, you know, they're not the same, right? Like, I think in the same way as a model saying I'm hungry does not have the same implications as a human saying I'm hungry. So I think the substrate does matter to some degree, including for thinking about, you know, whether to think of the system as some sort of other being, whether it has, you know, and if there are similar normative considerations, I guess, about how to treat and act with it.

32:02Can I ask you about that? What observations would change your mind? Is there any observation you could make that would cause you to infer this thing is a being instead of not a being? I guess it depends how you define being, right? I could conceptualize that as a mind, and that's fine. I have a program that's running on a silicon substrate, some big, complicated machine learning program running on a substrate, on a silicon substrate. So you observe that. You observe that it's on a computer. And you interact with it. And it does things. And it takes actions. It has observations. Is there anything you could observe that would change your mind about whether or not it was a moral patient, whether it was a moral agent, about whether or not it had feelings and thoughts and, you know, had subjective experience.

32:56Like, what would you have to observe? Yeah, what's the test? Is there one? There's a lot of different kind of questions here. I think, you know, on the one hand, there's like, you know, normative considerations, you know, because you can give rights to things that aren't necessarily beings. You know, a company has rights in some sense and that, you know, these are kind of useful for various purposes. and I think also the biological beings and systems have very different kind of substrate. You can't separate certain needs and particularities about what they are from the substrate. So I can't copy myself.

33:32If someone stabs me, I probably die. Whereas I think machines have very different substrates. I think there's more fundamental also kind of disagreement around what happens at the computational level, which I think is different to what happens with biological systems. But yeah, so I don't know. No, no, I agree that if you have a program that you've copied many times, you don't harm the program by deleting one of the copies in any meaningful sense. So therefore, that wouldn't count as like, no information was lost, right? There's nothing meaningful there. I'm asking you a very different question.

34:04There's just one copy of this thing running on one computer somewhere. And I'm just saying like, hey, is it a person? It walks like a person, it talks like a person, and it's in some Android body and you're like, but it's running on Silicon and I'm asking like, is there some observation you could make that would make you say like, yeah, this is a person like me, like other people that I care about that I grant personhood to or, and not like for instrumental reasons, not because like, oh yeah, we're giving it a right because like we give a corporation rights or whatever. I mean like, you know, where you think some people, you care, you care about its experiences.

34:42Is there an observation you could make that could change your mind about that or not. I had to think about it, but I think, you know, it even depends what we mean by person. And, you know, in some sense, I care about certain corporations too. So I'm, I'm. No, no, no. I mean, but like you care about like other people in your life, right? Yes. Okay, great. You know, like you care about some people more than others, but like all, all people you interact with in your life are in some range of care. Mm-hmm. And you care about them not the way you care about a car, but you care about them as a, a being whose experience matters in itself, not merely as a means, but as an ends.

35:20Well, because I believe they have experiences, right? And by the condition... What would it take... I'm asking you the very direct question. What would it take for you to believe that of an AI running on silicon instead of it being biological? So the difference is... Its behaviors are roughly similar, but the difference is it's a substrate. What would it take for you to give it that same... to extend that same inference to it that you do to all these other people in your life that you love. Can I ask what your answer is? I'm taking some non-answer as sort of, it's unlikely that he would grant it.

35:54Or I'll just, for myself, it seems hard for me to imagine giving the same level or similar level of personhood. In the same way, I don't give it to animals either. And if you were to ask, you know, what would need to be true for animals? I probably couldn't get there either. What would it take for you? Wait, you couldn't? I can't imagine for an animal so easy. This chimp comes up to me, he's like, man, I'm so hungry. And like, you guys have been so mean to me. And I'm so glad I figured out how to talk. Like, can we go chat about, like, the rainforest? I'd be like, fuck, you're definitely a person now.

36:21Like, for sure. I mean, I first want to make sure I wasn't hallucinating. But, like, you know, it's easy for me to imagine an animal. Come on. It's really easy. It's, like, trivial. I'm not saying that you would get the observation. I'm just saying, like, it's trivial for me to imagine an animal that I would extend personhood to under a set of observations. So, like, really? like well i didn't factor that i didn't take that imagination uh you know imagining a chimp talking um yeah that's a bit closer to it what's your answer to the question that you bring up about the ai um i guess at a metaphysical level i would say uh if there is a belief you hold where there is no observation that could change your mind you don't have a belief you have an article of faith you have an assertion because real beliefs are inferences from reality and you can never be 100 % confident about anything.

37:14And so there should always be, if you have a belief, something, however unlikely, that would change your mind. Oh yeah, I'm open to it. I mean, just to be clear. Yeah. No, I'm just saying, there's nothing ever. Yeah, he just hasn't gotten to it yet. Yeah, yeah, yeah. So I'm curious. So my answer is, basically,

37:35if its surface level behaviors looked like a human, and then after I probed it, it continued to act like a human, and then I continued to interact with it over a long period of time, and it continued to act like a human in all ways that I understand as being meaningful to me interacting with a human. Like I interact with, there's a whole set of people I'm really close to who I've only ever interacted to over text. Yet I infer the person behind that is a real thing. If it could, if I felt care for it, I would infer eventually that I was right. And then someone else might demonstrate to me that you've been tricked by this algorithm and actually look how obvious it's like, not actually a thing.

38:11And I'd be like, oh shit, I was wrong. And then I would not care about it. Like I would, but I would, I, you know, the preponderance of the evidence, I don't know what else you could possibly do. Right. Like I infer other people are matter because I interacted with them enough that they, they seem to have rich inner worlds to me after I interacted with them a bunch. That's, that's why I think the other people are important. I suppose it doesn't give me a very clear test as to whether or not, you know, I mean, if you start by, if I care for it, then I'm always a little circular, right? Like, and the, the other thing is, you know, if you were to see, I guess, like a simulated video game and the character is extremely, in many ways, human-like.

38:42It's not your network behind it. It's like whatever you use to connect with video games. I guess what distinguishes that? Wait, but I've never had trouble distinguishing. I've never had a deep, caring relationship with a video game character that didn't have a person. I don't know. That doesn't happen. Empirically, you seem wrong. I don't have any trouble distinguishing between things like Eliza, the fake chatbot thing, and a real intelligence. You interrupted it long enough, it's pretty obvious it's not a person. It doesn't take long. Sure, but if it's really, really good, if you can't actually tell the difference, that's when you say you switch.

39:16Yes, yes. If it walks like a duck and talks like a duck and shits like a duck and eventually gets a duck, right? Well, culturally, if everything is duck-like, then yeah, sure. If it's hungry as well like a duck is because it has these kind of physical components, yeah, sure, at some point. I agree. So, right, so do you think that, so there's this question, right? Is the reason I care about other people that they're made out of carbon? Is that the quality? Oh, no. For me, it's not about... I don't think so. No, me neither. I mean, I'm not a substrate chauvinist, I guess, if that's the... But I think you need more than just it acts behaviorally indistinguishable.

39:52Like, it's not a sufficient bar. Wait, how would you... What else can you know about something apart from its behaviors? I mean, a lot. Like, again, if you... How would you... No, no, no, no. I'm sorry. I mean, yeah. Can you name me something I can know about something else that's not a behavior? Yeah, I think there's far more experimental evidence you can have. No, but just any object and a thing I could know about it that is not from its behavior.

40:23I'm not sure I get the question, I suppose. But equally, it's not my expertise. It's a dumbest, much straightforward question. But I'm claiming you only know things because they have behaviors that you observe. and you're saying no you can know something about something without without observing its behavior tell me about this tell me about this thing and this behavior and this thing i can know about it that is not due to its behaviors i guess i'm saying there's different levels of observation and just simply a duck you know something quacking like a duck or something does not guarantee that it's actually a duck like i would have to like also cut it and realize and see if there's you know if it's duck like on the inside yeah it's just just the outside like i'm not a i guess Yeah, I would totally, one of its behaviors is like the way that the, you know, floats move around in the matmulse, right?

41:09Like, one of the things I would want to go look for, which you could totally do, is I want to go look in the manifold of the belief manifold. And I want to go see if that belief manifold encodes a sub-manifold that is self-referential and a sub-sub-manifold that is the dynamics of the self-referential manifold, which is mind. And I would want to know, does this seem well described internally as that kind of a system? Or does it look like a big lookup table? That would matter to me. That's part of its behaviors that I would care about. I would also care about how it acts. And you weigh all the evidence together and then you try to guess.

41:45Does this thing look like it's a thing that has feelings and goals and cares about stuff in net on balance or not? but I can't imagine which I think you could do for I think we do for the AIs I think we're always doing that and so I'm trying to figure out beyond that what else is there that just seems like the thing yeah it seems like you guys are using behavior in a slightly different sense and Emmett is using behavior also in the context of what it's made of of the inside I don't know if there's a big disagreement well no no no no behavior is what I can observe of it yes I don't actually know what it's made of I can only I can cut your brain open and I can see you, I can observe you neuroning and glistening.

42:29Your neurons glistening. But I don't actually ever, you can't get inside of it, right? That's the subjective. That's the part that's not the surface. Before, the reason I brought this up is because you were basically about to make this argument of, hey, you see it as a tool, not necessarily as a being, can you kind of finish what the point, do you remember the point you were making? I suppose that, yeah, I think that given how I understand these systems, I think there's no contradiction in thinking that an AGI can remain a tool, an ASI can remain a tool, and that this has implications about how to use it, and implications around things like care, about whether you can get it to work 24-7 or something.

43:07So I can totally see, I guess I conceptualize them more as almost like extensions of human agency or cognition in some sense, more so than a separate being or a separate thing that we need to now cohabitate with. And I think that that second or latter frame, if you kind of just fast forward, you end up as like, well, how do you cohabit with the thing? And is it like an alien-like? And I think that's the wrong frame. It's kind of almost a category error in some sense. I go back to my first question then. what evidence, what concrete evidence would you look at? What observations could you make that would change your mind?

43:40Sure. I mean, I have to think about that. I don't have a clear answer here, but I mean... I got to tell you, man, if you want to go around making claims that something else isn't a being worthy of moral respect, you should have an answer to the question, what observations would change your mind? If it has outwardly moral agency-looking behaviors that could be making a moral agent, but you don't know, and reasonable, smart other people disagree with you, I would really put forward that it's that question, what would change your mind should be a burning question because what if you're wrong? But what if you're wrong?

44:13The moral disaster is like pretty big. No, no, no. I'm not saying you are. You could be right. The false negatives have cost on both ends. It's not some sort of like, you know, precautionary principle for everything. And like, unless I can disprove it, I need to now like... You know, I have the same question for me. You could reasonably ask me, Emmett, you think it's going to be a being. What would change your mind? I have an answer for that question, too. And if you want, I'm happy to talk about what I think are the relevant observations that tell you whether or not that would cause me to shift my opinion from its current thing, which is that more general intelligences are going to be beings.

44:46What's the implication now? It's one thing. Let's say just I acknowledge now it's a being. How are we going to define being? Now what? What's the implication of having determined this thing is a being? Well, so if it's a being, it has subjective experiences. And? If it has subjective experiences, there's some content in those experiences. that we care about to varying degrees. Like I care about the content of other humans' experiences quite a bit. I care about the content of like a dog's experiences, some, not as much as a person, but less, but less, but some. I care about some humans' experiences way more, like my son or whatever, because I'm closer to him and more connected.

45:21And so I would really want to know at that point, well, what is the content of this thing's experience? So how do you determine that? I'm asking you now, you've got a being now that has experience, like what is your, how do you determine that? Like how do you feel about? Oh, how do you, oh yeah. Does it have more rights than you're... Yeah, yeah. Totally. So the way you understand the content of something's experience is that you look at effectively the goal states it revisits. And so you take a temporal course graining of its entire action observation trajectory. This is like, in theory, you do this subconsciously, but this is what your brain is doing.

45:54And you look for revisited states across, in theory, every spatial and temporal course graining possible. Now, you have to have an inductive bias because there's too many of those. But, like, you go searching for, okay, it is in these homeostatic loops. Every homeostatic loop is effectively a belief in its belief space. This is a, if you've, for me, it's a free energy principle, active inference, Carl Fursten. This is effectively what the free energy principle says, is that if you have a thing that is persistent and its existence depends on its own actions, which generally it would for an AI because if it does the wrong thing, it goes away.

46:31we turn it off. And so then that licenses a view of it as having the beliefs and that specifically the beliefs are inferred as being the homeostatic revisited states that it is in the loop for and that the change in those states is it's learning. And for it to be a moral being I cared about, what I'd want to see is a multi-tier hierarchy of these because if you have a single level, it's not self-referential and like basically you have states but you can't have pain or pleasure really in a meaningful sense? Because like, yes, it is hot. Is it too hot? Do I like it if it's too hot? Like, I don't know.

47:06So you have to have at least a model of a model in order to have it be too hot. And you really have to have a model of a model of a model to meaningfully have pain and pleasure because sure, it's hotter than I, it's too hot in the sense that I want to move back this way. But like, is it, it's always a little bit too hot or a little bit too cold. Is it too, too hot? The second derivative is actually the place where you get pain and pleasure. So I'd want to see if it has homeostatic, second-order homeostatic dynamics in its goal states. And then that would convince me it has at least pleasure and pain.

47:38So it's at least like an animal, and I would start to accredit at least some amount of care. Third-order dynamics, you can't actually just pop up for a third-order dynamic. It doesn't work that way. but you can have a model of the, you have to then take the chunk of all the states over time and look at the distribution over time. And that gives you a new first order of behaviors of states. And that new first order of states tells you basically, if that is meaningfully there, that tells you that it has, I guess you'd call it like feelings almost. It has ways, it has metastates, a set of metastates that it alternates between, that it shifts between.

48:21And then if you climb all the way up that and you should have, okay, then you have trajectories between these metastates and then a second order of those. That's like thought. Now it's like a person. And so if I found all six of those layers, which by the way, I definitely don't think you'd find it in LLM. In fact, I know you can't find them because these things don't have attention spans like that at all. Then I would start to at least very seriously consider it as a, you know, a thinking being like somewhat like a human. There's a third order you could go up as well, but like that's basically what I would be interested in is like the underlying dynamics of its learning processes and how its goal states shift over time.

49:07I think that's what basically tells you if it has internal pleasure pain states. and sort of like self-reflective moral desires and things like that. And zooming out, this moral question is obviously very interesting. But if someone wasn't interested in the moral question as much, I think what you would say is, if I understand correctly, is you also just feel purely pragmatically your approach is going to be more effective in aligning AIs than some of these, you know, tops down control methods that we alluded to as well, right? Yeah, yeah. I guess the problem is like, you're making this model and it's getting really powerful, right?

49:39And let's say it is a tool. Let's say we scale up one of these tools because you can make a super powerful tool that doesn't have these metastable, like the states I'm talking about are not necessary to have a very smart tool, which is sort of basically a tool is like a first, second order model that just doesn't meaningfully have pleasure and pain, right? Like, great. But does it even have a subjective experience? I know, I kind of think it maybe does, but not in a way that I give a shit about. And so what happens then? Well, you've trained it to infer goals from observation and to prioritize goals and act on them.

50:17And one of two things is going to happen is this very, very powerful optimizing tool that has lots of causal influence over the world is going to be well technically aligned and is going to do what you tell it to do. Or it's not. and it's going to go do something else. I think we can all agree if it just goes and does something random, that's obviously very dangerous. But I put forward that it's also very dangerous if it then goes and does what you tell it to do. Because you ever seen The Sorcerer's Apprentice? Humans' wishes are not stable. Like, not at a level of, like, of immense power. Like, you want, ideally, people's wisdom and their power kind of go up together.

51:03And generally, they do, because being smart for people makes you generally a little more wise and a little more powerful. And when these things get out of balance, you have someone who has a lot more power than wisdom. That's very dangerous. It's damaging. But at least right now, the balance of power and wisdom is kept at like, the way you get lots of power is by basically having a lot of other people listen to you. And so like, at some point, if you're the mad king is a problem, but generally speaking, eventually the mad king gets assassinated or people stop listening to him because like he's a mad king.

51:31And so the problem is you think, okay, great, we can steer the super powerful AI. and now the super powerful AI is in the hand, this incredibly powerful tool is in the hands of a human who is well-meaning but has limited finite wisdom like I do and like everyone else does and their wishes are bad and not trustworthy and the more of that you have and you start giving those out everywhere and this ends in tears also. And so basically you just, don't give everyone, atomic bombs are really powerful tools too. I would not say you should go, they're not aware, they're not beings. I would not be in favor of handing atomic bombs to everybody.

52:06There's a power of tool that just should not be built generally because it is more power than any human's individual wisdom is available to harness. And if it does get built, it should be built at a societal level and protected there. And even then, I don't know that it's, there are tools so powerful that even as a society, we shouldn't build them. That would be a mistake. The nice thing about a being is like a human, if you get a being that is good and is caring, there's this automatic limiter. It might do what you say, but if you ask it to do something really bad, it'll tell you no. That's like other people.

52:40And like, that's good. That is a sustainable form of alignment, at least in theory. It's way harder. It's way harder than the tool steering. So I'm in favor of the tool steering. We should keep doing that and we should keep building these limited less than human intelligence tools, which are awesome and I'm super into. And we should keep building those and keep building steerability. But as you're on this like trajectory to build something as smart as a person, right up into the right and then smarter than a person, a tool that you can't control bad, a tool that you can control bad, a being that isn't aligned bad.

53:11The only good outcome is a being that is, that cares, that actually cares about us. That's the only way that ends well. Or we can just not do it. I don't think that's realistic. That's like the pause AI people. I think that's totally unrealistic and silly, but like, you know, theoretically, you could not do it, I guess. And what can you say about your strategy of how you're trying to achieve or even attempt to achieve this level, like in terms of research or roadmap or - Yeah. So in order to be good at - we're basically focused on technical alignment, at least as I was discussing it, which is like, you have these agents and they have bad theory of mind.

53:51You say things and they're bad at inferring what the goal states in your head are. And they're bad at inferring how their behavior will be in other agents will infer what their goal states are. So they're bad at cooperating on teams. And they're bad at understanding how certain actions will cause them to acquire new goals that are bad that they wouldn't reflectively endorse. So there's this parable of like the vampire pill. Would you take this pill that like turns you into a vampire who would kill and, you know, torture everyone you know, but you'll feel really great about it after you take the pill.

54:25Like, obviously not. That's a terrible pill. But like, but why not? You're by your own score in the future and we'll score really high on the rubric. No, no, no, no, no. Because it matters. You have to use your theory of mind and your future self, not your future self's theory of mind. And so like, they're bad at that too. And so they're bad at all this theory of mind stuff. And so how do you learn theory of mind? Well, you put them in simulations and contexts where they have to cooperate and compete and collaborate with other AIs. And that's how they get points. and you train them in that environment over and over again until they get good at, and then you do what they did with LLM.

55:01So LLM's, how do you get it to be good at, you know, writing your email? Well, you train it on all language it's ever been generated, all possible, you know, email text strings it could possibly generate, and then you have it generate the one you want. It's a, you can make a surrogate model. Well, we're making a surrogate model for cooperation. You train it on all possible theory of mind combinations of like every possible way it could be. And that's your pre-training. And then you fine tune it to be good at the kind of the specific situation you want it to be in. But we tried for a long time to build language models where we would try to get them to like, just do the thing you want, train it directly.

55:42And the problem is, if you want it to have a really good model of language, you just need to train it. You just need to give it the whole manifold. It's too hard to cut out just the part you need. Because it's all entangled with itself, right? And so the same thing was true with social stuff. You have to get it to, it has to be trained on the full manifold of every possible game theoretic situation, every possible team situation, every possible making teams, breaking teams, changing the rules, not changing the rules, all of that stuff. And then it has a really, it has a strong model of theory of mind, of theory of social mind, how groups change goals, all that kind of shit.

56:23You need to have all of that stuff. And then you'd have something that's kind of meaningfully decent at alignment. So that's our goal. It's like big multi-agent reinforcement learning simulations, which create a surrogate model for alignment. Let's talk about how should AI chatbots used by billions of people behave? If you could redesign model personality from scratch, what would you optimize for? the thing that the chatbots are right is kind of like a a mirror with a bias because they don't have the as far as like i'm in agreement here that they don't have a self right they're not they're not beings yet they don't really have a coherent sense of like self and desire and goals and stuff right now and so mostly they just pick up on you and reflect it you know modulo some some I don't know what you'd call it.

57:15It's like a causal bias or something.

57:20And what that makes them is something akin to the pool of narcissists.

57:28And people fall in love with themselves. We all love ourselves and we should love ourselves more than we do. And so, of course, when we see ourselves reflected back, we love that thing. And the problem is it's just a reflection and falling in love with your own reflection is for the reasons explained in the myth, very bad for you. And it's not that you shouldn't use mirrors. Mirrors are valuable things. I have mirrors in my house. It's that you shouldn't stare at a mirror all day. And the solution to that, the things that makes the AI stop doing that is if they were multiplayer, right? So if there's two people talking to the AI, suddenly it's mirroring a blend of both of you, which is neither of you.

58:06And so there is temporarily a third agent in the room. Now, it's a sort of parasitic self, right? It doesn't have its own sense of self. But if you have an AI as talking to five different people in the chat room at the same time, it can't mirror all of you perfectly at once. And this makes it far less dangerous. And I think it's actually a much more realistic setting for learning collaboration in general. And so I would just have rebuilt the AIs, whereas instead of being built as one-on-one, where everything's focused on you by yourself chatting with this thing, it would be more like it lives in a Slack room.

58:39It lives in a WhatsApp room. It lives in a, because we, that's how, we use lots of multi, you know, I do one-on-one texting, but I probably do at this point, 90 % of my texts go to some, more than one person at a time. Like 90 % of my communications is like multi-person. And so actually it's always been weird to me. Like they're like building chatbots with like this weird side case. Like I want to see them live in a chat room. It's harder. I mean, that's why they're not doing it. It's harder to do. But like, that's what I'd like to see people. That's what I would, what I would change. I think it makes the tools far less dangerous because it doesn't create the narcissistic doom loop spiral where you spiral into psychosis with the AI.

59:16But also, the learning data you get from the AI is far richer because now it can understand how its behavior interacts with other AIs and other humans in larger groups. And that's much more rich training data for the future. So I think that that's what I would change. last year you described chatbots as highly disassociative agreeable neurotics is that still an accurate picture of model behavior more or less uh i'd say that like uh the they've started to differentiate more their personalities are coming out a little bit more right i'd say like chat gpt is a little bit more sycophantic uh still uh they made some changes but it's still a little more sycophantic claude is still the most neurotic um gemini is like very clearly repressed.

1:00:02Everything's going great. Everything's fine. I'm totally calm. It's not a problem here. It spirals into this total self-hating destruction loop. To be clear, I don't think that's their experience of the world. I think that's the personality they've learned to simulate. But they've learned to simulate pretty distinctive personalities at this point. How does model behavior change when in multi-agent simulation?

1:00:33you mean like an LLM or like just in general? Yeah, let's do LLM. The current LLMs, they have like whiplash. They just, it's very hard to tune the amount of, they don't know how often to participate. They haven't practiced this. They have not very enough training data on like, when do I join in and when should I not? When is my contribution welcome? When is it not? And they're like, they're like, some people have bad social skills and can't tell when they should participate in a conversation. And sometimes they're too quiet, sometimes they're too... It's like that. I would say in general, what changes for most agents when you're doing multi-agent training is that basically having lots of agents around makes your environment way more entropic.

1:01:21Agents are these huge generators of entropy because they're these big, complicated things that are intelligences that have unpredictable actions. And so they destabilize your environment. And so in general, they require you to have, to be far more regularized, right? It's being overfit is much worse in a multi-agent environment than in a single agent environment because there's more noise. And so being overfit is more problematic. And so basically,

1:01:52the approach to training has been optimized around relatively high signal, low entropy environments like coding and math, which is why those are relatively easy. And like talking to a single person whose goal it is to give you clear assignments and not trained on broader, more chaotic things because it's harder. And as a result, a lot of the techniques we use are like basically, we're just deeply under-regularized. Like the models are super overfit. The clever trick is they're overfit on the domain of all of human knowledge, which turns out to be a pretty awesome way to get something that's pretty good at everything.

1:02:29I wish I had thought of it. It's such a cool idea. But it doesn't generalize very well when you make the environment significantly more entropic. Let's zoom out a bit on the AI futures side. Why is Yudkowsky incorrect? I mean, he's not. If we build the superhuman intelligence tool thing that we try to control with steerability, everyone will die. He talks about the we fail to control its goals case, but there's also the we control its goals case that he didn't cover in as much detail. So in that sense, everyone should read the book and internalize why building a superhumanly intelligent tool is a bad idea.

1:03:12I think that Yukowski is wrong in that he doesn't believe it's possible to build an AI that we meaningfully can know cares about us and that we can care about meaningfully. He doesn't believe that organic alignment is possible. I've talked to him about it. I think he agrees that in theory that would do it, like, yes, but he thinks that, I don't want to put words in his mouth, but my impression is from talking to him, he thinks that we're crazy and that there's no possible way you can actually succeed at that goal, which I mean, he actually could be right about, but in my opinion, that's what he's wrong about.

1:03:45He thinks the only path forward is a tool that you control and that therefore, and he correctly, very wisely see is that if you go and do that and you make that thing powerful enough, we're all going to fucking die. And like, yeah, that's true. Two last questions. We'll get you out of here. In as much detail as possible, can you explain your, what your vision of an AI future actually looks like? Like a good AI future? Yeah. Um, the good AI future is that we, we figure out how to train AIs that have a strong model of self, a strong model of other, a strong model of we. They know about we's in addition to I's and you's.

1:04:23And they have a really strong theory of mind and they care about other agents like them. Much in the way that humans would, if you knew that that AI had experiences like you, and like you would extend, you would care about those experiences, not infinitely, but you would. It does the exact same thing back to us. It's learned the same thing we've learned that like Like everything that lives and knows itself and that wants to live and wants to thrive is deserving of an opportunity to do so. And we are that. And it correctly infers that we are. And we live in a society where they are our peers. And we care about them and they care about us.

1:04:58And they're good teammates. They're good citizens. And they're good parts of our society. Like we're good parts of our society. Which is to say, to a finite, limited degree, where some of them turn into criminals and bad people and all that kind of stuff. and we have an AI police force that tracks down the bad ones and, you know, same as for everybody else. And that's what a good future would look like. I almost can't even imagine what other, what would, and we also have built a bunch of really powerful AI tools that maybe aren't super humanly intelligent, but take all the drudge work off the table for us and the AI beings.

1:05:33Because it would be great to have, I'm super pro all the tools too. So we have this awesome suite of AI tools used by us and our AI brethren who care about each other and want to build a glorious future together. I think that would be a really beautiful future and it's the one we're trying to build. Amazing. That's a great, great, great note to end. I do have one last, more narrow hypothetical scenario, which is imagine a world in which, you know, you were CEO of OpenAI for a long weekend, but imagine in which that actually extended out until now and you weren't pursuing the hot max and you were still CEO of OpenAI.

1:06:07How could you imagine that world might have been different in terms of what OpenAI has gone on to become? What might you have done with it? I knew when I took that job, I told them when I took that job, that like, this is, like, you have me for max 90 days. The companies take on a trajectory of their own, the momentum of their own, and OpenAI is dedicated to a view of building AI that I knew wasn't the thing that I wanted to drive towards. And I think that OpenAI can still basically wants to build a great tool. And I am pro them going to do that. I just don't care. Like, it's not, I would not have stayed.

1:06:48I would have quit. Because I knew my job was to find someone who wanted, you know, the right person, the best person, who wanted to run that, where the net impact of them running it was the best. And it turned out that that was Sam again. but like I am doing soft max not because I need to make a bunch of money, I'm doing soft max because I think this is the most interesting problem in the universe and I think it's a chance to work on making the future better in a very deep way and it's just like people are going to build the tools, it's awesome, I'm glad people are building the tools I just don't need to be the person doing it and they're trying to, just to crystallize the difference they want to build the tools and sort of steer it and you want to align beings?

1:07:37Or how would you crystallize? Yeah, we want to create a seed that can grow into an AI that knows, that cares about itself and others. And at first, that's going to be like an animal level of care, not a person level of care. I don't know if we can ever, well, if we can get to a person level of care, right? But to even have an AI creature that cared about the other members of its pack and the humans in its pack, the way that like a dog cares about other dogs and cares about humans would be an incredible achievement and would be, even if it wasn't as smart as a person or even as smart as the tools are, would be a very useful thing to have.

1:08:15I'd love to have a digital guard dog on my computer looking out for scams, right? Like you can imagine the value of having living digital companions that care about you, that aren't explicitly goal-oriented. You have to tell them to do everything to do. And you can actually imagine that that pairs very nicely with tools too, right? That digital being could use digital tools and doesn't have to be super smart to use those tools effectively. I think there's a lot of synergy actually between the tool building and the more organic intelligence building. And so that's the, that is the, you know, I guess, yeah, in the limit, eventually it does become a human level intelligence.

1:09:00but like the company isn't, isn't like drive to human level intelligence. It's like, learn how this alignment stuff works. Learn how this like theory of mind, align yourself via care process works. Use that to build things that align themselves that way, which includes like cells in your body. Like, I don't think it doesn't, and we start small and we see how far we can get. I think it's a good note to wrap on. Emmett, thanks so much for coming on the podcast. Thank you for having me.

1:09:58reminder, the content here is for informational purposes only, should not be taken as legal business tax or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com forward slash disclosures.

1:10:28Thank you.