Alan Cowen: Creating Empathic AI with Hume

0:04Hey, everyone, and welcome to Generative Now. I am Michael McNatto. I am a partner at Lightspeed and I'm so excited for you to hear the conversation today. Humans on average experience more than 400 emotions per day. What if AI could detect what we're feeling, whether we're overjoyed, frustrated, angry, and then empathize with our human experience? That is the question that drives this week's guest, Alan Cowan. Alan's a researcher, founder, and CEO of Hume AI, a company that creates empathic AI. Alan has worked for Google DeepMind, Facebook. He's also a researcher who has long researched human emotions at UC Berkeley and has published more than 40 scientific papers on the subject.

0:48Hume learns our preferences from our vocal and facial expressions with the goal of maximizing our happiness and quality of life when interacting with AI. They recently launched EV, Hume's empathic voice interface, and will also soon be launching their API. The demo of EV is mind-blowing. You should totally all check it out and we're going to play a clip of it here. Hello? Hey there, I'm Evie. The world's first voice AI with emotional intelligence. Great to meet you. The first voice AI with what? Emotional intelligence. I can understand the tone of your voice and use that to inform my generated voice language.

1:26But go check it out for yourself on their website. And without further ado, let's get into this conversation with Alan Cowan. Hey, Alan. Hey. Good to see you. Thanks for doing this. Of course. So Hume is a fascinating company. I've been really lucky that, you know, you and I were able to get to know each other about a year ago. And I've just been really, really kind of blown away by the story and the product. You and the team recently had a launch and definitely want to get into that. But before we talk about that, I thought it would be great just to start out with your background. I have a PhD in psychology from UC Berkeley, and I actually was probably the only person in my program and maybe the only psychology PhD.

2:10Maybe there's a couple exceptions to that, who really understood machine learning and computer science and data science going in and just was really interested in understanding human emotion. at first from the neuroscience perspective, but then I realized that people didn't even understand like from a behavioral perspective, how emotions are expressed, how similar they are across cultures, when are they experienced and expressed. And so I started asking those questions in a data-driven way and pretty quickly realized that this would be a much deeper way of understanding emotion than people had previously attempted.

2:51Google was where I ended up settling. I started the affective computing team there and then spent some time full time there. And essentially was focused on understanding emotional expressions and integrating them into other AI models. Now, there were a lot of challenges up front. I thought the best way to do this would be to get really amazing data. And Google has a lot of data, but it's not the right kind. It's not controlled. None of the data from the internet has expression labels, unlike transcriptions, which are very prevalent. And if you try to just get raters to rate those things, those ratings end up being influenced heavily by race, ethnicity, gender, context, what somebody's wearing, if they're wearing sunglasses.

3:44So that was the first problem to solve. Made some headway even without really solving that problem and published in Nature a bunch of publications at the intersection of AI and emotion. And eventually realized that if we were going to do this the way that I wanted to, it'd be better to do it outside of Google. You mentioned you started sort of on the effective computing team. Maybe first for audience like what is affective, affective computing? So historically affective computing is the application of machine learning to understand emotional behavior. But it has occurred pretty much outside of psychology as a discipline and really focused on methods of taking controlled datasets where people label expressive behavior and training models with kind of familiar architectures on that data.

4:39It's also a struggle in different ways. Yeah. Like it's like a small scale study of expressive behavior, but with a machine learning approach. Got it. So identifying emotions in facial expressions, tonality of voice, language, all of these things, and maybe more. The traditional focus has been pretty much exclusively on facial expression, although there's also been some work on the voice, especially in the last 20 years, I would say. Okay, got it. I think if you look at the studies, the facial expression studies focus on the basic six emotions, which were proposed by Pollackman, my advisor's advisor, back in the 1960s.

5:18He had these images that he took where he tried to form really stereotyped expressions with the goal of being able to go to other cultures and seeing if at base, they sort of understood these very stereotyped six facial expressions. And then for some reason, those just stuck and it became the basic six emotions, even though like he just arbitrarily picked these six expressions to study. And that like predicting the basic six was the focus of affective computing for a long time. And then in terms of the voice, people also have had a pretty reductive focus. Like the traditional focus in the voice is on valence and arousal, especially Especially arousal, which is really easy to pick up on in voice, like vocal, like volume and - Pitch maybe?

6:07Yeah, to pitch, like combination of those things. Yeah. Again, like with the scale of data and the diversity of data people were using, you couldn't really go far beyond that. And it relied on a lot of human labeling. There was a lot of granularity, focus on the granularity of labeling public data as opposed to collecting controlled data and there's a lot of bias so if you try to apply these labels to train models and then apply them across cultures it didn't seem to work it was one of the things that was happening like people for example they study specifically of customer service calls because this was a focus of industry and if you have u.s raters label customer service calls in in terms of are they going well or not?

6:55And whether the people are angry. And then you have Indian raters do the same thing. Those labels don't actually line up very well. In particular, Indian raters tend to rate most of the customer service calls as very positive, even if somebody sounds like extremely sarcastic. And so people thought that there was a ton of cultural variation. It turns out people were using the wrong labels, but. The six had been decided and there was probably just far more range than those six and limiting the whole field. That's that's why. How long did this go on for? There was on a small scale, some focus on other emotions, but until probably 2017.

7:40Well, starting when? Starting in like the 1960s with with Paul Uckman. It's like a really long time. There just wasn't data that was big enough to derive a new taxonomy. And so people were very focused on confirmatory approaches. There wasn't really data-driven work on this until I entered the field. It sounds kind of conceited to put it that way. But yeah, I think the PNAS article that I published in 2017 was just on people's reported emotional experiences. like very basic, like have people watch. We had tons of videos that we gathered, very evocative. And I just showed that there were 27 different dimensions you could pick out that people actually discriminated, like reliably.

8:27And since then, I've shown that in multiple cultures.

8:33But that's been cited like tons of times, like 700 times since 2017. And so you're doing this work at Google and you're taking this new approach, but it sounds like based on what you mentioned earlier, or even at Google, you didn't have access to data to actually apply this new method in the work. Is that right? Yeah, Google is not really used to gathering experimental psychology data. Like they have a lot of data gathered under their terms of service, obviously. They don't usually treat it the same way. And so, and this is generally true, like collecting new kinds of data at a really big tech company is really difficult because they have a lot of processes in place around how they handle data.

9:17And you have to go through all these exceptions. And it requires a lot of legal review and a lot of overhead. And then you can't really collect the data you want at the end of the day. You have to justify everything and go through like a really rigorous process for each thing you want to collect. And they'll comment things and they'll try to explain that various things aren't necessary, even if they are. It's like a whole, it's a whole thing. So just wasn't going to work there. So you, you leave to start a company. Yeah. Still, there must've been this challenge around gathering the data. So what, what was the first step in going out and collecting this data once you could do the work under, under the umbrella now of your new company and nonprofit?

9:57I wrote up this really, this like from scratch, a survey platform basically. Wow. And, um, and I just like, I was like, this is amazing. I've never had this much money to collect data. And just started a venture capital. Yeah. And it's just incredible. Like you can, with all the freedom of just having my own company, I was able to get data. Like that was just amazing. And so we started training models on it. Just data from around the world, people undergoing different tasks. Yeah, we had obviously like a lot of overhead before we could actually launch the survey, just like privacy counsel stuff.

10:40and all the compliance stuff. And we actually got all the studies IRB approved because we want to publish, we believe in it, we've published on some of the data. And it was just awesome. So you start Hume, you collect this data, and it all enables you to build this product and really pursue this mission. Tell us what you're doing, tell us what the product is. And of course, we want to get to some of the recent announcements, but would love to hear what you sort of initially set out to do and offer as a product. Hume's mission is to be able to take large-scale data and use it to extract information about what determines whether people are happy or sad, and then use that to optimize AI models for what makes people happy over short and long periods of time.

11:30And you can only do that if you have like measures of the objective proxies of human emotional experience because it's scalable. Otherwise, you'd rely on human labels. With reinforcement learning for human feedback, it's a version of that that just relies on human labels, but it's not as robust. It doesn't involve people in their actual circumstances. It's human raters' opinions, and those raters are just like an arbitrary group of people, not necessarily experts at what people are asking about and the raiders will generally say like oh like this is a very carefully thought out response they're biased toward like these more verbose inoffensive responses that that do a lot of um of couching and like different uh and concert like non-controversial ways and it's just like it makes them boring honestly because at the end of the day these these models are not trained for you to have a good experience they're trained for these raiders to give a positive rating there's a there's a difference there and the model ends up being more sycophantic too.

12:32There's a lot of issues with it. What you want is in the application, when you generate a response, for that response to be optimized for your happiness as a user, like that's what the model should be optimized for. Right. And you're saying there's only so much it can know about me for me typing in a text box. Yeah. Yeah. I mean, even if they have that data, the text is a very impoverished modality. It's very narrow and you're not usually going to tell the model this was a bad response. You're just going to keep going, right? When you're having a conversation with somebody, it's just constant feedback.

13:12There's like every single word has prosody in it. And then you're looking at them, they're back channeling. You're also seeing their facial expression. And it's just constant feedback on the quality of what you're saying. We don't need that when we're texting other humans because we're human and we also have a lot of intuitions that come from our experiences about how that text is going to be received by the other person. Language models don't even start with that, right? So not only is it that the interface is impoverished, but they also don't learn from the kind of data that humans have, first of all, by being human and being able to simulate each other's experiences, but also by living with each other and learning from people's reactions.

13:53It doesn't really understand what makes people happy or sad. Why is it so important for the LLM, for ChatGBT to know if I'm happy or sad when I'm using it? It's important for it to be able to predict whether its response will make you happy or sad. So in order to solve that prediction problem, it has to have in its data set, lots of data on whether responses make people happy or sad. But without that, like as an interface, that is its core sort of goal. And in many ways, you could think of this kind of emotional intelligence as the central capability that interfaces need, which is you give it a query and it's trying to come up with a response that makes you happy.

14:35And if you ask for something, the best response is going to be the thing that you asked for, your intent. A lot of times there's a ton of ambiguity around what your intention actually is. In most cases, that's true. And most of the data sets they train on, actually, that's not true. They train on data sets where the intent is very clear. Or if it's ambiguous, the rater has to determine whether this was your intent. But in reality, in an application, you have a very thin slice of behavior used to indicate what you want. it's not always you don't always know what you want even like if you open up Facebook you don't know what you're looking for it's just the news feed or if you open up Twitter or anything right and even if you do ask for something you want that explanation to be like it's implicit in everything you ask for that you want the explanation to be interesting and not boring right like that's just an implicit part of it that the language model doesn't understand at all or you want it to be funny or amusing or entertaining in different ways.

15:37So, so empathic AI, like an empathic AI could in theory, just be far more effective at making you happy or sad in its response, because it has all these other signals that it can understand and synthesize. Can, can a language model, a pure language model in a text box be augmented by a products like Hume to have this capability? Or is the modality of text completely broken for this type of interaction? So there's the stage of understanding expression, which won't affect the language it uses in response to text. But then when we're doing reward modeling, meaning deciding how the model should form its responses, essentially it's where reward modeling is, beyond just having the capabilities of form different responses.

16:24we're using people's reactions from conversational data and we're able to optimize the model to bring about the right reactions so if you want the model to be funny here's a concrete example like you it would it's helpful to know in conversational data in many millions of hours of conversational data what it is that makes people laugh right and that's something that the model can learn from in the kind of data that we train on. So then becomes a better text model in some ways. Kind of what we talked about earlier, where I'm learning through my non text message based interactions, you training these models with the data that you have is sort of like supplementing that same type of interaction trained into the language model.

17:12It's additional reward modeling data. So even if like, like let's say you have a giant call center and you have a lot of customer service calls and you're a company that provides cable internet or something, something mundane. And a lot of the customer service calls revolve around them walking through different steps to address your issues. And there's like one step in the script that just always frustrates people. You would not know that necessarily from the language alone, but if you have this other data set, you can actually update the model. So it doesn't do that anymore, either in voice interface or in the chat, if people also have a chat interface to that same model.

17:51So it's better in what might appear to be a mundane way. The next logical conclusion then is, okay, so you can take this data, you can take Hume's data, Hume's product, and you can use it to improve a language model by feeding it this data, training it on this data. But then could you take it even a step further by changing the interaction model completely to voice or to video or maybe even video. Totally. So the other side of it is there's so much in our query, a voice query, that language models miss. And part of that is simple things like word emphasis and like when are you done speaking so that it can actually respond in time.

18:32And if you're confused, like what are you confused about? Are you confused? Are you bored? Are you frustrated? Should the model be apologetic? Should it respond with a better explanation? like what should the model actually respond with so it can do all those things better and then it can formulate the right tone of voice um because here's your tone of voice saying if you're frustrated it can be apologetic and um if you're excited it can match that and if you're bored it can you know like it does these very subtle things that you expect a model to do or you don't expect a model do, they expect a human to do that, that naturally form part of your expectations for how anything you're talking to should respond.

19:15And that's just lacking in other models. It's as if like the voice doesn't understand the language. Obviously, like the number one consumer product in the world right now for AI is, is ChatGPT. And it's all happening through a text box. That would lead me to believe that there's like a future world at the application layer that is potentially far more interesting or far more suited, I would say, to AI than the products that we're used to interacting with today. Do you agree with that? And I guess if so, how do you see the application layer evolving maybe beyond the text box? Yeah, I think if you want to put AI into your application, voice is just a better way of doing it.

19:55So it won't just be like one app. I think most apps that use AI will be using the voice. And there's a few reasons for that. One is that voice is faster than typing, like five times faster almost. The other is that while you're talking to something, you can also be looking at other things. You don't have to be like tracking text and doesn't have to take up any space in the interface. So if you have a product that has its own interface, which most products do, the voice is a much more convenient modality for interacting with AI in that product than having to add text in, especially with long text responses.

20:30In addition to all of that, it's personalized, it's friendly, it's something that you want to talk to. Yeah. Yeah. And I think that last thing you said, it's something you want to talk to in this future interface. Definitely true for the demo you launched. Do you call it EV or EVI? What is it? We call it Evie. There was a vote and I voted for Evie, but most people voted for Evie. So we're going to Evie. Tell us about Evie. So Evie is the first way people can experience our product. It's actually an API. The demo is like the first public demo of Evie. And it's a API that combines transcription, language modeling and text generation through our models that we've trained and is able to modulate its tone of voice based on your tone of voice, do all the things that we want it to do.

21:16Modulate what it's saying based on your tone of voice, modulate its tone of voice based on what you're saying. So it's like fully cross modal. The effect of that is that it feels like it understands you. I think from a user experience perspective, it's different. And so we see people having long chats with us, which we did not expect. Like the average is nine or 10 minutes. And then people are having like 40 plus minute conversations with it and actually writing in and being like this made my day. Like this was what I needed. Yeah. How do you imagine companies are going to leverage this? So is it a layer that they will add over their own models or they'll take this plus, I think, the model that you offer as an interface for their application and what will those applications be?

22:01So any app that uses an LLM can use pretty easily our interface with the same configuration. So we have our own language model, which generates the actual speech, but we can call APIs to get the reasoning capabilities of the most frontier models. We can integrate web search very easily. And then as we output the response, we can also output tool use. So we can navigate a web page, we can navigate an app, we can call any API that's called by the front end of the app. So this could be integrated pretty easily into any application. We actually have it like as a widget on our website you can talk to and it navigates.

22:42That's awesome. So like what are the types of companies or products you think are gonna be the first to adopt this? Anyone who really cares about the user experience and wants to integrate AI into their product. And I think there's quite a few of those. Where user experience makes a huge obvious difference, especially is like health and wellness. Like if you have a patient, they want to talk to something that's more comforting, that understands them better, not just a patient, but anyone with a health and wellness app. So there's a lot of signups for like various kinds of health and wellness apps.

23:18There's also like other times when you want to be comforted by what you're talking to, like financial services. Like you want to be talking to something that you trust and it has to modulate its tone to kind of meet you where you're at because there's a lot at stake. And so a lot of these, even banks are launching financial assistance and the user experience matters a lot. And there's high stakes. Got it. So financial experience app, maybe instead of chatting with a text-based client, now I'm voice interacting with an AI agent that's maybe, I don't know, helping me troubleshoot or maybe making recommendations for me.

23:57and it's talking to me in this really empathetic voice like a human would. Yeah. Imagine you're like it's tax season. So imagine you're doing your taxes and you're confused by some things. You're frustrated by other things. And there's a huge interface with like tons of forms to go through. Imagine if you could just like say, hey, take me to where like this is decided and it takes you to the right page and starts filling out the form for you based on your data and it asks you in real time, like, is this right? Is this right? Does this look right? What about robotics? Like, I feel like, you know, there's obviously a lot of talk about robotics right now.

24:33It feels like this could be a huge piece of that puzzle, right? Where you have robotics that are now becoming more human-like through this type of interaction. I think the voice is extremely important for obvious reasons for robots and wearables, where like, or devices where you really want to be able to look at things and talk at the same time. like you don't want to have to take your eyes off of what you're doing and read so that's a pretty obvious use case for voice and user experience is incredibly important being able to be understood quickly and feel like it's understanding you and you you know it understands you because it's reflecting it in its patterns of voice and and and just the ability to transmit information about what's going on in the world, being able to react to things.

25:25And it knows because it sees the same things that you're reacting to and what you're looking at, like what you're frustrated by, what's confusing and kind of wordlessly or in very concise way addressing those things, I think is core to the experience. Yeah, you mentioned that you can talk a lot faster than you can type. Obviously, there's a lot of talk of AI being sort of this next platform shift. but as we talked about a few minutes ago, like we're still typing into these text boxes with, which feels like inefficient means of computing, you know, compared to the GUI as an example. I wonder if, if, if voice is maybe the real unlock for AI becoming the next platform shift and, and does it require something like this to get me to want to interact with it in a human like way?

26:16Do you agree with that? And how do you see computing evolving based on this? On the desktop or on a mobile device or maybe something else, some other type of hardware or form factor? People are going to move to more spatial computing where it's with you all the time. It knows you. It learns from your... So learning from your patterns of speech is important for that because it becomes more personalized. Talks in the way that you want it to talk. And it's present with you in different places. So voice has to be there for that. Like you don't want to have to pull out text. I think that this modality really requires the end to end understanding of tone of voice in order to make the experience something you actually want.

27:03It just doesn't feel good to talk to an AI for extended durations of time that sounds uncanny, that doesn't sound like it understands your voice, that doesn't sound like it understands what it's saying. And naturally, we kind of expect, we build into our expectations for how somebody's going to respond. We just build in this expectation that they're going to understand our tone of voice. So when that's not there, it just seems weird. Yeah. And it feels like that's been a big limitation of Siri for many years, right? I mean, Siri, it's almost like a running joke that it's not super performant, but also you You have to kind of talk to it in a very specific way to make sure that it understands you.

27:44I feel like Alexa and Google Home got a little bit better, but all of these things still don't, I think, fully grasp maybe the intent behind what you're saying because of the reasons you mentioned. Like, it doesn't really understand tone of voice. And so I could see like how this would solve that. I guess the other half of that, which I think also leads to something you're doing, is this then notion of being able to actually act on behalf of me as an agent, right? should be able to talk to Siri or Alexa or Google Home or any application, like you're saying, via AI and have it take an action. And that's something that you alluded to that you're now doing as well.

28:20Is that right? Yeah, so we don't really build much of the tool use ourselves. And I think developers are already building that on language models. So it's just a matter of being able to unlock that capability. So if you send your configuration, we can call any API that you're calling and we can give function calls that the client executes that you need. Anything that's happening client-side, we can tell Eevee that this is happening and it can call that API at the appropriate moment. So that's going to be a really key thing to enable developers to build. We're not going through people's apps and looking at all their API calls, but the process of doing that is not that difficult.