Behavior Is All You Need: Making AI Feel Like a Person

0:00Vishnu Hari:In sort of the chase towards AGI, what we're kind of missing out on is how do we have AI simulate the things that we find really so compelling about each other? Our flaws, our fun, you know, our internal lives, conversations we can have with one another that aren't related to like knowledge search, all of that.

0:17Matt Paige:Welcome to the Talking AI Podcast, where we talk AI with both experts in the field and early adopters. I'm your host, Matt Paige, and we're here to demystify AI for you so you can get some value from it. Let's talk some AI.

0:32Matt Paige:Everybody's chasing AGI, but today's guest thinks that still won't get us to the thing that actually matters, and that's humanness, AI that behaves like a person, not just a model that talks. But what if the next leap isn't a smarter chatbot, but characters that remember you, build shared experiences, and keep living long after? Today, I'm joined by Vish Hari, CEO and founder of Ego, a Y Combinator winner 2024 company building persistent AI characters inside immersive worlds. And before Ego, Vish worked at AI applied research at Meta and worked at Meta Reality Labs. But welcome to the show, Vish.

1:13Matt Paige:Thanks for having me. Excited about this conversation because everybody's racing towards AGI. That's the hot topic. And there's a lot of opinions on what is AGI? Are we going to get there? Are we already there? How do you even define it? But I believe what's interesting here is like, your point is not necessarily that AGI is the thing that gets us to this thing you call humanness in a sense. And y 'all wrote this paper called behavior is all you need. So what's the core difference here in terms of behavior is all you need versus attention's all you need, the famous Google paper that kind of led us into this world of the transformer architecture and whatnot.

1:55Vishnu Hari:Yeah, our white paper is nowhere near the same caliber as that incredible seminal work. But the kind of point we wanted to make, which is a little tug-in-cheek, is that the thing that's going to drive, I think, a ton of consumer AI is going to be AI that is fun and relatable or AI that solves a problem. For the latter, which is AI that solves a problem. The race towards AGI makes a ton of sense. You want something that's intelligent, that can contextualize knowledge, that can draw insights for you and present it to you in a very consumable fashion. But the things, however, that are closer to consumer, which is entertainment and fun, what you want is something that resembles humanness because for us, we don't really go to scientists for entertainment unless they're Richard Feynman.

2:35Vishnu Hari:We go to entertainers, we go to storytellers, we go to comedians. And all of that usually exposes some level of humanness and the storytelling that is what draws audiences in and i think you know my entire theory around ego is that in sort of the chase towards agi what we're kind of missing out on is how do we have ai simulate the things that we find really so compelling about each other our flaws our fun you know our internal lives conversations we can have one another that aren't related to like knowledge search all of that that's our focus at ego yeah and you get into this like

3:13Matt Paige:architecture that uh consists of different components in a sense and there's this perception layer behavior layer emotion uh module a memory system a personality uh module a needs module which i thought was very interesting uh and then this aspect of memory and putting all these things together you know i guess is like you said it's not becoming human but it is starting to emulate human at a different level than what we'd experienced with just chat GPT. So what is that nuance? And what is it about these characteristics? Or what are the things that actually make this start to feel like you're interacting with not just some random bot within a simulated world and whatnot?

4:00Vishnu Hari:Core insight actually comes from bots and video games. So when we play any MMORPG, and back in my day it was World of Warcraft, when you were in a town, you knew who was an NPC who had a specified routine and specified tasks versus another human player. In fact, NPCs become kind of an insult these days, right? It's used online in various ways. Because what that implies is that you don't really have a theory of mind and don't have internal needs and desires. However, when you meet someone who's part of a clan who invites you to join the clan, suddenly the social structures come to fore. And you have to think about what your place is, what your needs are, how you can help the clan, how you can make friends, all of these sort of human-like things emerge.

4:39Vishnu Hari:And that emerges because each person who controls a character has certain needs, desires, wants, personal history, memories of other clanmates, and that makes all of this stuff really fun. You know, that's what drives a lot of interaction in virtual worlds and video games. So our foundational insight is that if you don't have characters with desires, needs, internal lives, you don't approach humanity or humanness. and the actual real proof of this is probably the most character chat app in the world character.ai the reason why it's so popular is because for the ip characters that you chat with you have an entire corpus of knowledge on who they are we know how aragorn is going to react to any given situation because there's a ton of work on who he is we have some idea on internal life people role play and larp as aragon and like many other ip characters because we kind of know who they are So our entire thesis is if you can simulate and build that in AI, you get to build the next AI equivalent of Aragorn in some UGC format.

5:40Matt Paige:Quick break in the pod. Our State of AI 2026 report just dropped, and it breaks down what actually is changing in AI, what's hype, and what leaders need to be paying attention to this year. You can grab it right now on our show notes or at hatchworks.com. Yeah, and getting to what y 'all are actually building at Ego, because it's super interesting. There's kind of two sides. You're actually working on a model in a sense, but then also building this world. You hinted at it a little bit there, character.world. What does this manifest into and why building this sim-like world as a first thing? And then where does that evolve into later on in your mind?

6:17Vishnu Hari:You caught us at a very interesting time because we realized that building the sim-like world is actually a larger bite than we can chew off. That is the final goal. and the main inspiration for that is this really awesome anime that i grew up watching called sword out online the other person who loves it is palmer lucky i think i exchanged an email with him about it but that that anime is like the source of all this inspiration that in westworld right that's super cool that's the final goal and part of like trying to build towards that was just breaking down what makes a human-like character human there's many things and one of them is uh proposing behaviors all you need which is that this idea that these characters have internal lives and desires and wants beyond just the goal to the game is like a thing that's going to imbue humanness and like make a human real human feel like the characters they're talking to you could be human now the one thing we realized though is that one creating a 3d game is pretty difficult making it extremely accessible is really difficult and three like doing the perception layer of the game outside of just the game code but like the visual perception visual language aren't quite there yet in terms of latency however we did find an area that we find pretty interesting that speech so in our internal tests uh we we actually built a version of some world on um on roblox and we call it chatter blocks and about a million people played it in like one month uh but the main behavior collapse we call it and we saw was people didn't really believe they're talking to um a human-like character because in ai all you do is turn taking like the way you just said, oh, wow, the way you slightly interrupted me, that's human.

7:49Vishnu Hari:The way we talk, the way we nod, the way we express ourselves, the way we intonate and then jump in, all of those things are what creates human-like conversation. And that's actually the first problem that we need to solve. Because with the Roblox game, and even some experiments we did in Minecraft and other companies that are in Minecraft too, there is proof that simulated humans can be seen and observed to feel like they lead full lives. And we saw that even in the Stanford paper that came amount of 2023 by June Park. It's possible. However, when you interact with them, the mask slips. And our actual focus in 2026 is to have the conversation feel natural.

8:27Vishnu Hari:No one's focusing on that. Everyone's still doing turn-taking and like your turn, my turn, your turn, my turn, no initiation, all of that. That is what we're thinking about.

8:37Matt Paige:When you say turn-taking, that's kind of the experience we have when we're talking to chat gpt or whatever exactly model today in a sense that's interesting and i guess where do you see this going so you know it's kind of this initial step the end result being kind of this simulated world but do these you know uh ai characters and you know simulating humanists just live in this world or do you see them emerging outside of the world into anything that is digital.

9:09Vishnu Hari:This is why I'm here in Singapore because we just closed a really exciting research partnership with a company called Menlo.ai. Menlo.ai, they're a robotics research company based out of Singapore. And the natural conclusion of any simulated frontier work is actually real life. And the example of that is self-driving cars. They started out in simulated environments first to make sure that the cars could behave in safe ways, even in four-way stop signs or whatever, right? and then eventually got put into the real world. The same thing is actually true of a lot of frontier approaches. Another example is Dota.

9:43Vishnu Hari:You might remember early on, like in 2017 and 2016, OpenAI at that time released an AI that could play Dota as well as a human could, if not better. That was awesome. That was like a spark moment. And the founder of DeepMind, Demis Hassabis, was a video game developer. He built a game called Black & White back in the late 90s, early 2000s. games are actually where the frontier of technology is tested and pushed even online networking had to be proven in games first and so our entire thesis is that if we can make these characters have their own internal lives in 3d games and they feel alive and real in the game they would probably feel the same way in real life that's it that would be a robotics play at some

10:26Matt Paige:point yeah that gets into the uh the west world scenario in a sense exactly talking about uh and I think NVIDIA does this a bit too, where they have this world where this is how they're getting to training robots or humanoid robotics in a more nuanced fashion because they're living in this simulated world versus just how we train a traditional LLM today. Is that similar to what you're hitting on?

10:53Vishnu Hari:Similar. In fact, there's a version of this that they did in Minecraft called Mind Dojo by Jim Phan, which is super exciting and foundational to how we're thinking about it. Google's doing something similar with SEMA too. there's a ton of interest in simulations for things like robotics but again there's a missing piece in all of these they're all for the industrial helper bot use cases they're not for what I find fascinating which is her the movie by Spike Jonze that is a different structure of thinking about robotics and human like characters

11:22Matt Paige:what are the two differences there you probably have hit on some of this as we've been talking but what's the nuance between that and her in a sense.

11:32Vishnu Hari:Robots are task-based creatures. They are effectively your slaves. They are chained to perform the task. They are limited and with safety and all that stuff they're specified to do the one thing they have to do extremely well and that's about it. For us, we're trying to get as close to humanness as possible in the sense of they have internal lives and internal motivations that in the case of robots wouldn't be related to survival because they don't need to eat, at least for now. But at some point eventually would evolve into this, I want to experience this because in my corpus of prompt architecture I'm actually someone who loves being in Italy.

12:09Vishnu Hari:I want to be in Italy. I want to go to the streets of Italy and do things and experience things and visually see Italy and the bot goes and does that and feels happy about it and loves meeting Italian people and talking Italian. That's just a random example that came into my head but the main difference is the fact that these things could ostensibly have an inner life that they talk about.

12:29Matt Paige:So what, what changes at that point when, even if simulated, uh, whether it's digital AI in a digital form and a, in a real, uh, form begins to have wants and needs, what, what changes at that point versus, you know, to your point with chat GBT and LMS today, they're, they're literally, um, from the onset there, they are a helpful assistant. consistent right what begins to change good or bad i guess i couldn't theorize on what societal

13:02Vishnu Hari:changes can happen because those are always kind of foolish to theorize about because it's so hard to bring such a complex organism called human society however i could make some like predictions about how humans would deal with it based on just how i feel about it mostly and sci-fi but we're already anthropomorphizing chat gpt and most chat apps most people are doing that there's a lot of headlines daily about people feeling all types of way um it's having unhinged ways but that's already here right now what happens when you go beyond anthropological conversation to like feeling it's another entity is relationship is memory is desire it is communion to some degree which i have no idea what the actual conclusion of that could be for society but for an individual it passes the turing test it goes beyond a slave it goes into something that has and it like it has like some some level of like a human that's the best word i could use yeah yeah is there some type

14:00Matt Paige:of like indicator measurements you're thinking of like measuring this as you go to to what's that mark of where this feels human and the level you're looking at it is there some kind of turing test or like way you're thinking about uh how do you measure this this progress or this different vector of how it behaves and acts and interacts.

14:26Vishnu Hari:Subjective measures, right? You can always do double blind tests on conversation and all of that stuff, but I think most of that is meaningless at scale. At scale, what matters to a consumer company is always going to be retention. At your 30th day, are there a lot of people still doing things with these characters and have they done more things than just within your app, within just conversation? That entire spectrum of activities they could do, does it increase over time? Do they want more? That's the question.

14:55Matt Paige:Well, it's an interesting, if you're talking from a pure business standpoint, like a moat in a sense, because I've always thought that memory, like say just called ChatGPT, Claw, Gemini, whatever it is, the memory it has of me to just recall conversations and nuances about me is one of the best moats in a sense, because I don't want to go to another model because it's not going to have that nuanced understanding of me and i feel like this is somewhat similar to where in the beginning it's like a new friendship or interaction or relationship but as it builds over time then it becomes more nuanced and stickier you know just like in a relationship it's it's more dynamic that's it that's it so i was about to say that

15:38Vishnu Hari:our interest is in evolving memory into relationship which can only happen if the entity's

15:44Matt Paige:human yeah memory into relationship yeah because it's it's such a big element and yeah i think in the paper you'll talk about short-term and long-term memory which starts to emulate how we as humans um have memory in a sense that plays into this this factor right absolutely now operationalizing that is a totally different thing that's that's very complicated but that's the ambition yeah yeah and there's another paper too we talked a bit about in our prep session the one by Google around nested, I believe there's nested learning. What's the... In context, continual learning or in context, online learning.

16:25Vishnu Hari:What's the nuance there? Any thoughts on that? Oh, yeah, no, that's actually more like a product. Oh, yeah, yeah. Sorry. Yeah. So no, the context there is like, okay, in like a new environment, like let's say that I ask you to play, you know, Fortnite with me, and you never played Fortnite before. What is the context you bring in? And while you know the control system, you know depth, you know kind of tasks you can do, you know a basic control scheme, what do you do? You look at what other people do and you're comms with them to figure out how you want to engage and interact. What you're actually doing internally is updating your weights.

16:56Vishnu Hari:You're like, oh, okay, that's new. Oh, you got a feedback. Okay, that's cool. It's a form of RLHF in your brain that just kind of happens, right? And then what you do is you project your personality, which is who you are, onto that learning and that character, which is I'm insanely aggressive. So I'm going to be doing the most crazy moves, but maybe one of my friends is a little bit more defensive and just kind of wants to hang back a little or run away or disengage in combat. That is a function of your personality and your past memory onto gameplay and onto a new system. Now, this can apply beyond video games.

17:33Vishnu Hari:Video games are probably the most salient example because it's the most complicated.

17:38Matt Paige:That's interesting. And you've worked at Facebook and Meta, Meta Reality Labs. What was the, any unique learnings there? Because they're obviously very much connected to what you're doing today in terms of like building these virtual worlds and whatnot. Any, any nuance or learning there?

17:57Vishnu Hari:I'm trying to figure out how much I can share. I started in applied research. And then when Facebook became Meta, I moved to the Reality Labs team because I was really fascinated by a lot of the work the Metaverse teams were doing. but I had one fundamental contention that you cannot simply create 3D worlds and have them feel lifeless and not populated. And that is a foundational problem that I wanted to solve was that all of these people create these awesome VR worlds but they're always empty and if they're always empty it's like a nightclub that's empty on a Friday night. You're not going to go away.

18:28Vishnu Hari:You're not going to put the first person in. Right? So if they're empty why would more people show up? Well, what if we could use internal language models to simulate humanness instead of just having NPCs do routines because it's so obvious when it's an NPC doing a routine. It's so obvious. And we evolved beyond that.

18:45Matt Paige:That was my interest there.

18:47Vishnu Hari:I think that's all I can really share.

18:48Matt Paige:Yeah. I bet that was an interesting experience. So let me get into kind of more of a lightning round style questions. I'll hit you with some and get your perspective. But is AGI overhyped, overrated, overhyped, or misunderstood? What's your thoughts on AGI as a thing you keep focusing on?

19:10Vishnu Hari:It's such a large field of play. It's not... I mean, you ask any two researchers, whether it's Karpathy or LeCoultre, his perspective on AGI, each of them will have, I think, a relatively tangential-ish answer, but not quite the same. So, different than how you want to define AGI. You want to define it as the infinite god AI thing. That's a very different type of thing from someone who's like, and you can bring into like a corporate entity and like have it, you know, gain context real quickly and give recommendations. And then it's also very different from something they can like cure cancer, right?

19:43Vishnu Hari:And you can like find new therapies. Oh, that's a different type of stream. So mostly it's misunderstanding, but I'm not like, I didn't finish my PhD. So I'm not the most AI expert person. I'm more on the applied side. So I have to be a bit humble about my perspectives on this because I don't actually truly know. The field moves so fast. So who knows? Yeah.

20:03Matt Paige:And you mentioned Karpathy. Who do you look up to the most right now? Oh, Satzkever. Ilya Satzkever. The models in a sense.