Teaching AI To Read Our Emotions — With Alan Cowen

0:00Let's talk with the CEO of the leading AI startup for emotional intelligence, looking at where this exciting new discipline of AI is heading right after this.

0:12you're used to hearing my voice on the world bringing you interviews from around the globe and you hear me reporting environment and climate news i'm carolyn beeler and i'm marco worman we're now with you hosting the world together more global journalism with a fresh new sound listen to the world on your local public radio station and wherever you find your podcasts

0:40Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. We have a great show today going into one of the more interesting and emerging areas of AI research and application. I think it's going to be fascinating. We're probably going to see this stuff packed into basically every AI that we touch going forward. And we're going to speak with somebody who's really at the cutting edge of where this is all going. Alan Cowan is here. He is the CEO and founder of Yume AI, which is a company with a bot that we've talked about on the show here in the past, one that can sense the emotion in your voice and reply in kind.

1:15It's very fascinating. And I cannot tell you how excited I am to have Alan on the show. Alan, welcome to Big Technology. Hi, Alex. Thanks for having me on the show. We're going to speak with Dwarakish Patel in the next couple of weeks, but he had a podcast episode with Mark Zuckerberg basically right around the introduction of Llama 3. And they were talking about the cutting edge of AI research. And Zuckerberg says emotional AI for some reason is something that I don't think a lot of people are paying attention to, but there's a lot of potential there. One modality that I'm pretty focused on that I haven't seen as many other people in the industry focus on this is sort of like emotional understanding.

1:55Like, I mean, so much of the human brain is just dedicated to understanding people and kind of like understanding your expressions and emotions. And that's like its own whole modality. What do you think he was getting at when he said that? I'm not entirely sure what's in his mind, but it's definitely the right problem. I think the reason it's the right problem is that when you're trying to address people's, let's say, concerns with AI, and also when you're trying to just meet their requests, the most important thing is emotional intelligence. It's the ability to take people's preferences into account in your response.

2:32And so emotional AI is, first of all, all AI is emotional in the sense that like all AI has to do this. But AI that's specifically trained to do this is going to learn to produce responses that make people happier. And that's the fundamental objective for AI. And why is that important in terms of advancing AI? So if you're talking to ChatGPT, you're implicitly saying, okay, I'm going to give you a way to make me happy, which is to fulfill this request. Oftentimes, you don't even have an explicit request. And when you do, it's actually fundamentally ambiguous, right? So the goal of an interface is to take this ambiguous slice of behavior.

3:13And it could be language or it could be language plus audio plus facial expression plus eye contact, whatever behavior it has access to. And it's trying to take contingent on that the best estimate of what might make you happy if it responds in a given way. So that's literally what it's trying to do. It should be able to learn that by understanding human emotional reactions to things. That's the fundamental way that humans learn this stuff. Like we can either simulate what your emotional response will be to me doing something for you based on putting myself in your shoes, or we can look at your emotional reaction based on that, figure out whether I did the right thing or not and adjust.

4:00And those are really the ways that we learn to make each other happy. So much of that is actually done in voice, done in expression. And in text, it's so much harder. Let's start with the hardest problem and then kind of work back from there. How do you assess out emotion in like a text exchange? In text, there's some cues, right? And most of the time, we're not actually sussing out emotion in texts. We just have certain conventions that we use those conventions to dictate how we communicate. And so we know that this is a very narrow channel through which we can express our desires. And so we know that there's fundamental ambiguity.

4:37So we just basically limit what we try to do. We don't try to do too much with text. We don't try to bite off more than we can chew. What are some of the conventions that you'd say we rely on with text? Punctuation and capitalization and where you put periods and when you want to add an extra Y to the hay or like all these different conventions. And they just have like these kind of stereotyped meanings. So it's a message passing system. It's very frustrating to correspond by text if what you actually are trying to say is anything more complex than that. What's coming across here is that it's actually remarkable how limited this current set of large language models are, given that it's all text and it's still blowing people's minds.

5:24But there's really a limited range in which it can be useful. Is that right? Yeah, and it's never seen people's emotional reactions in any other modality. It learns some things from text about what people want and don't want, but it actually doesn't have that layer of understanding of how is this going to make people feel. So even via text, it seems to be missing something. And yeah, it can solve problems. If you specify the problem in exactly the right way, for example, in the way that these LLMs are evaluated, you just literally ask it a multiple choice question. If you specify it with that level of constraint, then they're really good.

6:00But anytime you want to do something more open-ended, it's a relatively unsatisfying experience. So this is why people are gauging how good these things are by asking them to take the LSAT. It's like not an accident. It's like this is sort of the only way to evaluate them within their constraints. And that's more or less how we evaluate. So like the biggest eval is MMLU that people use. And it's literally you can get each question right or wrong. And then you just look at the percentage they got right. It's basically a big multiple choice test with some other kinds of questions in it. When I first started using your product, you have a bot.

6:33It's called Evie, E-V-I. And when I first started using it, it's kind of in demo mode now. And the API just opened. We can talk about that. But I was like, oh, this seems like a large language model in some ways. And that like it's giving some of those kind of wink wink, like as an AI assistant, this is my feelings. As an AI system, my purpose is not to replicate the full human experience, but to engage in meaningful and empathetic conversations. But it's voice. And it's now coming into focus that you decided voice because there's so much more signal and also so much more realm of possibility in terms of interacting with this than you can with a text bot.

7:19there's so much more data that comes in via voice and can go out via voice, then we can really take a next step in terms of the way that generative AI is moving. Previous technology relied exclusively on transcribing the voice and taking away all that extra meaning that's in your vocal modulations. So this technology didn't exist before Hume. When you add that in, what's remarkable is that the model can actually learn what vocal modulations mean on its own. You just need to add it in the right way. We have these expression measurement tools that we've trained that are extremely accurate. They can actually encode 756 dimensions of embedding of vocal modulation and 756 dimensions of facial expression per word that you're saying.

8:03So there's these incredibly rich embeddings that we're adding in that add a lot more information. And the model can actually learn that. And it turns out that it's just more satisfying to use when it has that data. Take us through very briefly a sort of ride through the study of emotion in humans, where it was lacking and how it's now being applied into robots. Or maybe not that briefly. I feel like that's a tall task to do in a minute. I mean, it starts with Charles Darwin. I won't go back that far. But basically, he was sketching out facial expressions in both man and animals. And he has a book called The Expression of Emotion in Man and Animals.

8:41And he recognizes the complexity of it and the nuance. but somehow that kind of was lost over time nobody really talked about that book for a while in the mid 20th century there was Paul Ekman who decided he would tackle the question of what is it about facial expression that may or may not be universal and so he cataloged these six facial expressions as an arbitrary test more or less of what these broad expression stereotypes might mean to different people. And he chose them because they're very different. He only chose one positive expression. He chose five negative expressions. Wait, which ones are they?

9:20Anger, disgust, fear, happiness, sadness, and surprise. Wow. Surprise could be good, but it's definitely a largely over-indexed on bad feelings rubric. It was bad in his conceptualization of it. Yeah. So positive surprise is good. negative surprise is bad. And then there's also related epistemic emotions like awe and interest, which are good, or confusion, which is bad. So to varying extents with varying levels of arousal and excitement. Anyway, but none of these distinctions are in those images. And for some reason, people just stuck to those six categories for a while. And there was this countervailing narrative that actually there are fewer granularities in emotion that people recognize, that actually it's just valence, like positive, negative versus arousal, calm or excited.

10:10I don't know why that was the countervailing argument, except that maybe people were just limited by the amount of data that they could collect. And this was like the closest thing to data driven that they had, which is like, if you do dimensionality reduction on a hundred samples, you get positive, negative, uh, low arousal, high arousal as your dimensions. And wait, arousal in terms of what is arousal mean? Arousal means like excitement, not like physiological arousal, meaning like activation of the autonomic nervous system. Okay. Good to clarify that. Okay. Yeah. And that's how, I mean, there's lots of dimensions to that.

10:46Obviously, like one of them is actually sexual arousal. Like that would be different. Like we recognize that it's very different from the arousal that goes with fear and like running for your life. Right. And yet in this theory, they're compressed. I know. Well, it depends who you ask, but okay. Continue. you can kill that one can cause the other i don't know well anyway that was it's true this the the not to go too far in the digression but there is scientific literature that says danger and arousal are connected i mean esther pearl had a whole book on it called mating in captivity that's excellent but yeah well yeah it might not be danger but not our specialty novelty right yeah and there was there's misattribution of arousal which is this old experiment where you take somebody on a bridge and is very exciting.

11:36And then you test whether they basically are aroused. And yeah, they are. Anyway, or whether they're attracted to you. And this is not something that necessarily replicates in every condition. OK, so there's lots of different kinds of emotion. They get reduced down to these six basic categories or two dimensions for a very long time in research. And a lot of that has to do with the fact that you can only collect so much data. The data analysis tools that were available to people for a long time were very coarse. For example, there weren't even really computers that psychologists were using to analyze data.

12:14Incidentally, psychologists did, like the field of stats sort of came out of psychology, but that was earlier. And then, you know, when it came to actually applying it to lots of data, that was much harder and that didn't come until much later. So now data science comes around sort of in the 2000s and becomes a really big thing. And psychology was one of the last places that it was applied. So it starts to get applied to neuroscience and biology and genetics and all these other areas. But people didn't really have the tools to analyze human behavior until relatively recently. and part of that is like even when you have the the data analysis techniques the data collection techniques are difficult so what i pioneered in my phd in psychology and while i was working for google and facebook was a new kind of data collection that you could then use to collect enough psychologically controlled data to apply data science to psychology data and that's that's ultimately what gave us a lot of surprising results about the dimensionality of emotion.

13:29What were the surprising results? What you can see clearly is that people make really granular distinctions between lots and lots of different expressions pretty reliably. So there's differences between an expression of awe versus surprise versus fear versus confusion, interest. All of these expressions are reliably recognized and so we just started to map them out. We realized they're not discrete categories, that they're actually continuous and can be blended together. We realized that the number of dimensions that it takes to represent these things is large. It's not like you can reduce it down to valence and arousal.

14:09That actually captures like 20 % of the variability in people's ratings that's consistent across different raters. you actually need you know in facial expression over 30 dimensions and speech prosody the tune rhythm and timbre of speech you need over 18 different dimensions to represent how people are able to conceptualize speech prosody and in vocal verse like laughs and sighs and screams you need at least 24 different dimensions and this is just what people explicitly recognize it actually goes beyond that when you look at what's implicit in people's responses to things that is not well-verbalized, or what's implicit in people's conversational signals that's not verbalized explicitly.

14:52So there's a lot of different dimensions that had never been classified before. So you've never classified these dimensions of behavior or been able to measure them. There could never have been a science of what they mean. And so this sort of opens up the door to actually understanding human expressive behavior in the natural world, in the real world and understanding what it means. And we got a ton of publications out of that pretty quickly. And then you also seem to have been able to study how much small variations in these expressions or even tone of voice can tell you. And talk a little bit about that, especially the tune and the timbre of the voice and what we can learn from each of those.

15:34We created new machine learning models that can reliably distinguish among all these different dimensions of vocal modulation that people form and make it distinct from the language, the phonetics of what they're saying. But that was a challenge before. So we solved that. And when you can do this, you start to see a lot of meaning in people's voice modulations that is just completely not present in the phonetics or present to a much lesser extent. Like, are you, and it can be simple things like, are you done speaking? And obviously, like there's a lot of emotional dimensions that get added to a lot of words or every word is every word carries not just the phonetics but also a ton of detail in its like tune rhythm and tamper that is very informative in a lot of different ways you can predict a lot of things you can predict whether somebody has depression or parkinson's to some extent really perfectly yeah so like mental health conditions can be predicted to some extent you can predict in a customer service call, whether somebody's having a good or bad call, much more accurately by incorporating expression than just with language alone.

16:41So pretty much any outcome, like it benefits to include measures of voice modulation and not just language. And what is phonetics? Phonetics meaning like the underlying words, but specifically when we form words, there's a phonetic representation that gets converted to text or that's converted to semantics, basically. And that's what the transcription models are doing, is that when you convert something to text, you're actually only relying on the phonetic information, meaning the stuff that conveys word forms, and you're throwing out everything else. You're throwing out stuff that might convey, for example, emotionality or tone of voice, et cetera.

17:31It's so interesting because I'm thinking now about like, so often what I'll do is I'll take the transcripts of podcasts and dump them into Claude and start talking to Claude about it. But of course, it's only getting the words and not the emotions in there. So there might be parts where I thought, or it might be interviews which I thought were like particularly like rough or really exciting but it can't fully pick that out because it's just seeing the text and there is so much meaning contained in the words that are said uh that that you need deeper tech and deeper expertise to pick out yeah i think some people forget like there's a lot of density to the information in audio that like if you transcribe a conversation sometimes it doesn't even make sense to read it.

18:17Like you can read the words and you're like, this actually does not, my brain cannot make sense of this. But then when you hear the audio, like it makes perfect sense to you. Totally. Like even speaking with the media, I remember I was on the first time I was on NPR, I was very excited. I went to the studio, recorded my interview and from the audio, it sounded like a normal conversation. But then I read the transcript and I'm like, man, what were you saying there? None of this makes any sense but it's it is such different meanings when you just look at the text versus like hear like the full express conversation in the meaning so yeah um if you look at political speeches often that's the case i'll say more about that even if the writing's really good you look at like mlk's speeches if you transcribe them and read it it just doesn't have the same effect at all not the same no we're close no emotion to them or less people yeah there's certain people who don't use the right grammar or linguistic forms that would be traditional in writing um either because they're just not good at it or because you know they can depart from that purposely like if you look at like trump's speeches yeah they really when you transcribe them a lot of them are completely incoherent well when you listen to them yeah i mean maybe some of that is true going out on a limb here but but yeah i i do think that it is interesting because when I'll write for spoken word, it'll always be different than the written word.

19:44We are going to talk about how this technology is going to be applied and already is being applied in the business world, but just still a few more theoretical questions for you. Is this type of technology something that we can use to understand animals? Like people talk about using AI to understand whales. Is this something that can be used in that nature? so the same types of machine learning models that we're training can be used for for that our data would not really be that relevant to that right because it's a totally different language yeah and maybe some of the methods the sort of the training approaches that we're using could be helpful for that yeah um but it's a different problem um maybe to you know we have an interest from evolutionary biologists who want to measure the similarity between expressions and different mammals and human expressions.

20:38And so there, maybe there's some, there is some relevance because we can, we can treat it like a human sort of morph it to look like a human face and then analyze the expression and see if it predicts the things that we want to predict. Are you going to do that type of work? We can, but like, it's not really that strong of a model to actually perform inferences. It does show similarities. Like there's dimensions of similarity between humans and chimps. Chimps have an open face model, which means something similar to when humans are laughing and playing, and they use it when they're playing, both physically and kind of in a non-physical touch scenario.

21:18Mice laugh. For real? Yeah, hypersonically. So like you can't ultrasonic, so you can't like hear it at all if you don't turn down the frequency of it so that it's something humans can hear. But then it kind of sounds like a laugh. No way. And you can elicit this by tickling them. If you tickle mice. They laugh? Like this. Yeah. Kind of makes me feel bad that we use mice so often in scientific experiments knowing that they can laugh. Yeah. I mean, they also have a lot of empathy for each other. Like if one mouse is trapped, the other, without any ulterior motive, will hear that mouse screaming and then like go and try to investigate it and get them free.

22:03And if there's a lever, they'll figure it out. Really? Yeah, they're pretty smart, actually. Yeah. Mice and rats are also more related to humans than they are to dogs. Right. Well, that's why we do all this testing on mice. It's because they are close to us. Yeah, exactly. It does make me feel bad. And then, um, do you think that we're going to be able to read like, uh, using technology to be able to understand animals? As someone so close to this, what's your prediction? I think that the, the animals will have some kind of quasi language that they use. And so we know this for, for like, for like primates, for example, that there's different calls that mean like there's a snake below us.

22:49And then if you like play that through a speaker, all they're all look down or like there's an eagle above us and you play that through speaker, they're all look up. Right. And so there's like a quasi language there. I don't think they have the same level of syntax that humans have. And syntax is our ability to form sentences with nouns and verbs and relationships between entities and logic and stuff. And I don't think they have as much of that. Some people speculate that dolphins might have that. which would be pretty surprising i don't know for sure um because they don't have like hands and so the value of that ability like maybe they use it to coordinate like hunting and stuff but it's hard to imagine like why that's important for the dolphin brain to have that capability yeah but you know if they're they do seem to be really smart there's that like scene in blue planet i don't know if you've seen it where the the dolphins and the false killer whales are first they're like fighting like the killer whales are chasing the dolphins and then there's a scene where the dolphins like turn around and they all just stop and they just seem to be squeaking at each other for a while and then after that they hunt together for real i haven't seen yeah it's insane so that is suggestive to me that maybe there is something to the language potentially and ai's ability to decode that it's within our lifetimes we could it's what people are working on it people are working on it it's difficult because you don't really have a grounding for it you kind of just have to like like ideally you'd have really detailed explanations of what's going on to accompany these this language that's being exchanged but like you don't have that that's like that's how we train image models we have captions for the images right If you just had to teach a computer to understand images without captions, it would be difficult to ground that in anything.

24:51Like you'd have a computer that embeds images and can tell whether they're similar. That's fine. But at the end of the day, you want it to be able to take the image and explain it to you. Then you need some grounding for that. You might not need that much grounding if you've done enough compression of the similarities between images first. which is sort of maybe that's what we're hoping to do with dolphins. Fascinating. Okay, I want to talk about the business applications here, including why Facebook and Google would want to employ you to put some of this stuff into work inside their product. So let's do that right after the break.

25:23We'll be back right after this. Did you know your credit card points and miles can lose value to inflation? Credit card companies often reduce the redemption value of your points and miles. Now, imagine a credit card with rewards that can grow in value. With the Gemini credit card, you can earn Bitcoin or one of over 50 other cryptos instantly with no annual fee. Every swipe at the store or gas pump earns you instant rewards deposited straight to your account. Plus, sign up now for a$200 Bitcoin bonus to kickstart your rewards. Visit Gemini.com slash card today. Check out the link in the description for more information on rates.

26:01Again, if you're looking to invest in Bitcoin but don't know where to start, the Gemini credit card makes it easy. The Gemini credit card is issued by WebBank. In order to qualify for the$200 crypto intro bonus, you must spend$3 ,000 in your first 90 days. Some exclusions apply to instant rewards in which rewards are deposited when the transaction posts. This content is not investment advice and trading crypto involves risk. The Gemini credit card cannot be used to make gambling-related purchases.

26:32You're used to hearing my voice on the world bringing you interviews from around the globe. And you hear me reporting environment and climate news. I'm Carolyn Beeler. And I'm Marco Werman. We're now with you hosting The World Together. More global journalism with a fresh new sound. Listen to The World on your local public radio station and wherever you find your podcasts.

26:59and we're back here on big technology podcast with alan cowen the ceo and founder of yoom it is a company you should check out that has this very interesting new bot called evie and also an api that's going to allow companies to take advantage of this ability to determine human emotion through voice and maybe one day through facial expressions. So let's talk briefly about your work at Facebook and Google. So you've done this work. You've basically expanded the number of human emotions that we sort of acknowledge exist and can study and have worked to build that into machine learning models that can recognize them.

27:44What did Facebook and Google want with that type of knowledge and technology? You know, interestingly, there's some intersections among all the tech companies and sort of their interest in this. And they've had teams working on this now for a few years. I think it was more research driven in that, you know, there were obvious applications, but it wasn't really clear which applications would be the most successful for them. early on it was pretty clear to me when i saw these language models that could communicate with people and you could talk to them like you were human that this is where the technology would be most relevant um once i saw those language models before that i was more interested in recommendations and being able to optimize recommendation algorithms for people's well-being based on any implicit signals you you can get of well-being including expressions but now it's It's really obvious that the right thing to do is to fine tune large generative models to produce the right things, because there's even more flexibility in what they can produce that make people happy.

28:52And I don't know if it was ever explicit at the big tech companies that this is what this would be used for, but that was what was most present to me.

29:03So obviously I left in 2021 to start Hume, where we could collect the data that was needed to do this. and at the time I think it was clear to stakeholders at some of these companies how incredibly powerful this would be invaluable and important for the future of AI but I don't think there was broad stakeholder alignment across these companies. You think it might have been a different case? I mean I imagine if you were in one of them after ChatGPT came out they would immediately assign you to the problem. yeah um so i you know the chat gbt was i think it came out in 2021 or 2022 22 22 google internally had stuff like that a lot sooner wait when you were when you were at google did they have lambda up and running and were you able to play with it yeah that was like part of the inspiration oh talk about a little more internally at google it was very clear what these models uh where some of these models were going, although the business model was not clear of how to use it.