Empathic AI and its role in understanding human emotions with Hume AI’s Alan Cowen | E1922

0:00I could have this thing roast you if you want. it might be a little yeah let's do it let's do a roast all right um could you roast jason calicanus man this is like everybody's all right let's talk about that piece of work jason calicanus oh man where do i even start with this guy you know i heard a rumor that jason calicanus was actually grown in a lab a mad scientist's twisted experiment to create the most obnoxious self-aggrandizing tech bow imaginable. Poor bastard. He never stood a chance. And have you seen the way this guy carries on? It's like he's perpetually auditioning to be the next Elon Musk, but with half the brains and twice the ego.

0:39I mean, the guy can't open his mouth without saying something so mind-numbingly. This is too much. Can you go easy on this? Oh, yeah. Okay, actually, go harder. Come and don't be like that. Go harder. This Week in Startups is brought to you by... LinkedIn Jobs. A business is only as strong as its people and every hire matters. Post your first job for free at linkedin.com slash twist. Vanta. Compliance and security shouldn't be a deal breaker for startups to win new business. Vanta makes it easy for companies to get a SOC to report fast. twist listeners can get a thousand dollars off for a limited time at vanta.com slash twist and hubspot join thousands of companies that are growing better with hubspot for startups learn more and get extra benefits for being a twist listener now at hubspot.com slash startups all right everybody welcome back to twist this week in startups and we've you know in 2024 and 2023 been absolutely obsessed with ai obviously we're seeing all kinds of easy layups and customer service thanks to ai autonomous vehicles much more complicated healthcare everything in between we're also seeing tons of interesting stuff going on in generative ai people making interesting music and videos you've seen all that but the area of human emotions is extremely complex and ai is trying to figure that out and you've seen this in all kinds of science fiction whether it's blade runner or the movie her where ai is trying to learn to interface with humans well there is a startup human ai and they are trying to bridge the gap between just intelligence and, dare I say, emotional intelligence.

2:41We demoed some of this technology back on episode 1894, if you want to look for it. But today, we have Alan Cohen here. He's the CEO and Chief Scientist at Hume AI, and he's going to show us what they're building and why it's important. Welcome to the program, Alan. Hey, Jason. Great to be here. Right. So maybe you could explain what the mission is of Hume AI and why you're spending all this effort to try to understand human emotions. And yeah, in relation to AI and using AI, I guess, to understand humans emotions, and then to portray them back through AI to humans. Yeah, so it's really to understand people's well-being.

3:27And emotions are the components of that. So when are you laughing? When are you sad? When are you in pain? When are you experiencing pleasure? And what we want to do is optimize for that. So our mission is to optimize AI for human well-being. Now, so much of what we express is in our voice, in our facial expression, and not in language. So that part of our expression was just ignored by AI for a long time. I mean, there is a field of affective computing. which I have a lot of experience in. I have over 40 papers in that area. But in terms of the generative models, they just were very far behind in understanding expressions.

4:02So what we've done at Hume is built models that understand expressions a lot better. And we've integrated those into large language models. So now these models understand beyond language, what's going on in the voice, what's going on in facial expression, and can learn from that. So they figure out what's making you frustrated, what's satisfying what's funny and they can actually adapt to that information and get better over time got it so what is the killer app of this genre or this vertical i have chat gpt i pay 20 bucks a month for it i frequently will use the interface where i talk to it but i don't believe that they are looking at my tone and i know i haven't turned my camera on i assume with your technology at some point, these AI assistants are going to have the camera on with permission, obviously, and hear my tone of voice and understand when I'm frustrated or explain to us what the killer app here is.

5:03Because, you know, we've heard of things like Gong and Rilla Voice, we have a small investment in. They study salespeople and how they behave on sales calls to coach them. That genre exists a bit. I'm curious what you perceive as the killer app here? So ChatGPT is text only. We think the future of AI is a voice app. The voice is four times faster than text. The problem is that when we're speaking, we expect the AI to understand not just what we're saying, but how we're saying it. So that's why this experience is clunky. It's just missing something. And so we've built that. We have a talking ChatGPT, basically.

5:40And we have an API that allows developers to build a voice interface into anything. So instead of using OpenAI, which would enable developers to build a chatbot interface into their product, they use us. And immediately they have a voice interface that does transcription, does language understanding, that speaks back. And what's different about it than anyone else is it understands what's in their voice besides language, and it uses that to form better words and better speech. Supposed to be a perfect time, I think, for a demo, so let's get to it. All right. You queued it up perfectly. sounds great I will share a few demos and what's the API called it's called EV empathic voice interface empathic voice interface okay like Lieutenant Deanna Troy on Star Trek Next Generation she was an empath yeah why don't I just demonstrate this this is like the demo that anyone can play with and then I'll show you the developer playground

6:41hey what's up Well, hello there. I am doing just fine. Thanks for asking. So what's got you feeling a bit down and amused at the same time? I am all ears if you want to chat about it. Well, my dog died this morning.

7:02Oh no, I am so sorry to hear about your dog. Losing a beloved pet is never easy. I know it must be really hard right now, but try to remember all the wonderful times you had - Aha! You really had me going there! Well played, you sneaky trickster. Aha! I should have known better than the fall for that one. I think I am still learning when it comes to your sense of humor, but I am relieved to hear your dog is doing just fine. Care to share what's really on your mind today? Well, can you tell what I'm expressing now? Oh, our demo's at capacity right now. Oh, there you go. So while you were doing that demo for people who were listening, and not watching.

7:44You can watch, of course, on YouTube, go to This Week in Startups on YouTube and you'll find it. It was showing each transcript in real time very, very quickly. And then it had the top three emotions and the percentage of those emotions. I think it was showing the top three every time. Is that correct? Yeah. So it shows more than just the top three. But actually, if you were to look at your raw data, you get back 48 different dimensions. So it's much more nuanced than what we're showing you there. Got it. And so in real time, you can see that you were sad when your dog died and etc. And then that person was showing sympathy for you.

8:21So all of that is being done through tone of voice, inflection, etc. Okay, let me cut to the chase right now because I know you're busy and everyone is hiring right now. And you know, it's a lot of competition for the best candidates, right? Every position counts, markets starting to come back, you need to get the perfect person. You want a bar raiser in your organization, somebody who will raise the bar for the entire team. And LinkedIn is giving you your first job posting for free to go find that bar raiser, linkedin.com slash twist. And if you want to build a great company, you're going to need a great team.

8:53It's as simple as that. LinkedIn jobs is here to make it quick. and easy to hire these elite team members. And I know it's crazy, right? LinkedIn has more than a billion users. We all watched this happen when it was tens of millions, then hundreds of millions, and now a billion people using the service. This means that you're going to get access to active and passive job seekers. Active job seekers, they're out there looking. Passive job seekers, they got a job, but it's not as good as the job you're offering them. So you want to get in front of both of those people. Maybe somebody got laid off, wasn't their fault, and they're an ideal candidate.

9:24Get that active job seeker. And LinkedIn also knows that small businesses are wearing so many hats right now and you might not have the time or resources to devote to hiring. So let LinkedIn make it automatic for you. Go post an open job role. You get that purple hiring ring on your profile. You start posting interesting content. You watch the qualified candidates. They just roll in. And guess what? First one's on us. Call to action. Very simple. LinkedIn.com slash TWIST. LinkedIn.com slash twist. That'll get you your first job posting for free on your boy j cal terms and conditions do apply what are the components in voice that you're studying is it the speed at which somebody speaks you know tone and and how did you train this thing on tone how does it know what sadness is versus you know melancholy versus quirky yeah we have all this data from millions of people around the world who are actually recording themselves while they're having interactions and also we're reacting to things and um and imitating things in some cases and so we use all that data to train our models and that means they're able to capture way more than just like tone rhythm like those are all basic things um but dimensions that you can't really describe in any other way except to say like this is kind of an angry dimension kind of has a growl to it kind of um tension in the voice where this is like an awe-inspired dimension or happy.

10:51And we get tons of different dimensions out of that. So every time we hear a word, we're getting more than 48 different dimensions of expression from that word. Our model is taking that in and our model is deciding how to respond. Our model is learning what these dimensions mean from tons and tons of data, people interacting. And it's saying, okay, this is something that means this person's frustrated, so I should apologize. It's something that means the person's confused or actually clarify and is figuring out what it should do to respond to somebody in different situations. How different is it per person?

11:24Like I might be a high energy guy from Brooklyn who's extroverted, who speaks a certain way and is, you know, you might be more introverted and soft spoken. So how does it know if J. Cal's like bombastic and joking, and you might be more thoughtful and introverted. Are our emotional emotions very similar or are they very disparate? I'm curious. So it has to learn that stuff. So we train in all these interactions, right? And so it's trying to figure out, the task is actually predicting the next expression. So it has to figure out like, is this next person going to laugh at what's said or like, are they going to be frustrated?

12:03And so it has to learn how you express your response to things in the course of doing that. and it's learning that in a generative way in a very ground up way. So by the time that we've trained this thing, it has to account for individual differences, for potentially cultural differences, for sentiments, and also just the average of all humans and like what humans respond to along with the distribution, if that makes sense. So like, what is it that humans find funny? What is it that humans find sad and all that? So like when I said my dog died, we can probably figure out this is a sad event. I'm going to be sympathetic, right?

12:36That's how it, and it figured out. How much of it is the words versus the tone of voice, or is it doing both of those things? It's doing both. Let me try to give this another shot so you can see that. Can you tell what I'm expressing right now? Whoa there. I can hear the frustration in your voice. It's an anger, determination, and distress. Like you're ready to tackle whatever's got you worked up. Can you tell what I'm expressing now? Hmm. I am picking up on some subtle shifts here. You sound a bit more relaxed now, though maybe still a tad bored or uneasy. But then I also hear a spark of amusement and even happiness.

13:18Like, you're pleasantly surprised by something. Am I on the right track there? I'm going to mute that bit. Got it. You're sounding a bit more at ease now with a hint of satisfaction. Anyway, you get the ideas. Yeah. So if that demo is designed to reflect back to you what emotions and things you're having, and then tweak it. So how long does it take for it to accurately understand a human? It's less than 500 milliseconds. As you can see, our API is experiencing some load right now. But generally speaking, we can get you back a response faster than any other API. and that's because we're able to detect when you're done speaking more accurately.

14:03So some of the other APIs, they have to do this dance between, can I jump in or is it going to interrupt the user? And so there's a little bit of a pause. But for us, because we understand the tone of voice, we can use that to figure out when the person's done speaking and then more accurately know when to step in. And so that enables us to respond a lot faster. So it doesn't need to talk to me and ask me 10 questions to understand my emotional state and how I might be uniquely different than another person. What about across cultures? Because do different cultures have, obviously we have different languages, but even putting aside language, does tone work across cultures?

14:39Do Koreans, Italians, and Americans all emote frustration the same way, anger the same way? Is it across cultures or does it require more subtlety? There's similarities and differences. We have a paper that just came out on this, but basically if you're speaking a different language, we need to train a new model for it. And it can be not a completely different model from scratch, but at least we need to fine tune on that language. That's what we find for most languages, especially for broadly different languages. Like all the Latin languages have similarities, but if you look across East Asian languages, things are pretty different.

15:18So, yeah. So suffice to say, yes, we do need to train things for each language. And this demo only works in English right now. So who's using the app? Like, let's take a look at the developer console. So you have that up there, the playground. Yeah. Who's using this now? And is it in production anywhere? And what are people using it for? Because there's plenty of models out there to give you answers and generate copy for you. I'm wondering if people are even up to this level of nuance in their products yet, or just trying to get correct answers, because accuracy seems to be a pretty paramount problem right now.

15:54Yeah, I mean, you might be interested in accuracy, but if you're using a voice interface, you need to get to the point fast, right? And so that's really what we're doing. And you can't, with these long, verbose responses from chatbots, first of all, those are very taxing on the brain to read, so that's not a good interface. but also you might have an accurate answer in there somewhere. It doesn't really matter if someone's not going to listen to a voice reading that out for three minutes, right? That's a good point. So we have a lot of developers lined up for this. We haven't released this API.

16:26By the time this comes out, we will have released it because we're releasing this on Wednesday. But so far, we have developers on this, which is our measurement API. Oh, wow. So you're on a webcam right now. I'm just going to describe it. and you're making funny faces right now uh you're surprised horror confusion sadness disappointed laughing and if i were to just say be completely calm and at ease your calmness just went up to 79 your concentration went up to 45 um and now if you um started thinking deeply about the meaning of the universe like why are we here like what is the purpose of life like why wake up and build this company every day says you're calm you're calm with existential wait is this a video or are you doing this right now alan i'm doing this right now you're doing it right now you're not following my instructions no give me your exit give me your most existential like i'm wondering about the meaning of life like why are we all here i want to see if it gets existential confusion there it is contemplation yeah contemplation well what's interesting about this is this would be great for coaching an actor because like happy's easy sad's easy if you go happy it's got joy amusement excitement great and if you were sad sad is disappointing and confusion maybe you're just not a good actor maybe you need to take acting lessons for these demos contemplation is a tough one I was trying to get you to have existential angst I was trying to pick something that's really hard to read well just think like should you even come to work is it all meaningless that's kind of depression it'd be sad yeah a little sad a little confusion boredom it's fascinating so this is just really getting your facial expression in real time.

18:31So if you were frustrated, the AI would know it and be like, huh, that wasn't the answer we were looking for. And so are people using this for therapy yet or like therapeutic coaching kind of things? Because that one seemed to be like, I got a lot of pitches for people who want to create AI therapists. And I'm like, hmm, that's a little dicey. I don't know if you should call it a therapist, but companion. are people using this for companionship? I do think that AI is going to be something that is your friend. And so it's not just like a niche application. I think generally speaking, we want an assistant that understands us.

19:10And there's tons of people working on that. I mean, there are people working on explicit therapy apps with Hume too. And actually, a lot of it's in training therapists. And it's getting them to... you know there's a delicate balance you don't really want to like comment too much on people's emotions but you want to ask the right questions and kind of get at it help them understand their emotions better and so there's a lot of that and there's also like therapist burnout doctor burnout there's a lot of health and wellness applications there's also tracking depression and stuff we work with clinical researchers who are running clinical trials and using Hume to track symptoms of depression and Parkinson's, just the symptoms.

19:51It's not like used for diagnosis because ultimately the doctor does that. But it's helping the doctor understand these things. So we have a lot of those applications. A lot of them, you know, those are interpersonal things. Like someone's talking to someone and we're already like the measurement APIs that we have are very good at extracting more data from that and helping people analyze it and helping people understand themselves and their patients, I guess. I mean, is it so if we have therapy on one side, you have the therapist who needs to present in a certain way to get people to open up. If you believe in that modality, if you believe in Western psychotherapy, there is something about pacing and aligning with the person matching their energy and getting them to open up so that they have some cathartic, you know, way of processing stuff.

20:44So people are using it to train therapists so that they don't have a goofy look on their face or they have the appropriate look that would elicit less suffering in their patients. Is that what I'm... Yeah, or like customer service reps, which is actually a kind of similar thing. Yeah, it's another form of therapy, actually. It essentially is, yeah. Yeah. But you know, that's requires somebody who's technical, who's maybe academic, maybe a researcher to take take these measures and make sense of them. Listen, a strong sales team can make all the difference for a B2B startup. But if you're going to hire sharks, you need to let them hunt and you can't slow them down with compliance hurdles like SOC 2.

21:26What is SOC 2? Well, any company that stores customer data in the cloud needs to be SOC 2 compliant. If you don't have your SOC 2 tight, your sales team can't close major deals. It's that simple. But thankfully, Vanta makes it really easy to get and renew your SOC 2 compliance. On average, Vanta customers are compliant in just two to four weeks. Without Vanta, it takes three to five months. Vanta can save you hundreds of hours of work and up to 85 % on compliance costs. And Vanta does more than just SOC 2. They also automate up to 90 % compliance for GDPR, HIPAA, and more. So here's your call to action.

21:58Stop slowing your sales team down and use Vanta. Get$1 ,000 off at vanta.com slash twist that's vanta.com slash twist for one thousand dollars off your sock too have you done this with poker players yet have you put poker players through this to see if they're lying or deceptive in a poker trade a lot of things and it does not it cannot tell poker players about thing you know at least professional poker players cannot tell i don't think the information's there i just don't think that with professional poker players that you can there's anything going on in their facial expression. What can you tell with people's facial expressions that we wouldn't know of?

22:35Some people have said you could tell a person's, if a person's sexuality, where a person's from, you could tell all kinds of interesting things that you wouldn't know. I think up to a level. Is that true or not? That's not really true. There's been a lot of pseudoscience in this area. Like most of the things that we can tell are things people want to communicate, which is good. We actually don't really care to impinge on people's things that they want to keep private. We're more interested in helping people communicate well and helping the AI understand what people want. And most of that's like they're overtly on the face.

23:14And for example, it even extends to things like, is the person done speaking? We're way better at understanding when they're done speaking because we can take into account facial expression versus just the language alone. And that's part of how our empathic voice interface is able to respond better. So you know when I say, this is the end of the sentence. Yes. Because of my facial expression, you get a quicker clue than audio only. Therefore, you can start speaking without interrupting me, which is what humans do with each other. Yeah. Like, imagine I'm speaking to you. And right now, it's clear to you.

23:52I just finished the sentence. It's clear to you I'm still speaking. And it's clear to you I'm going to say something again. But now I'm done. now i know i can speak right yeah which is what i do for a living on the podcasts is try to understand when people are done so that we can have the next person speak right like moderation is a is a difficult task um and customer support folks are using this already to understand how hot and bothered people are when they call the customer support line i assume to some extent yeah so kind of understanding is the customer having a good time bad time where are we kind of failing on customer service and which customer service reps are doing well or poorly and how do we train them to do better?

24:33How do we pull examples up of when they're not doing well so we can train them to do better? And, you know, there's a lot of AI going into customer support now. So some of our early design partners for this new API are people who want to take the automated customer support, make it a lot better, but still know when to include a human, escalated to a human exactly i mean that makes sense if the person's like this is incredibly frustrating you know and you start hearing the frustration go up and they're whatever united premier gold diamond status yeah you want to get them on the phone with somebody because it's you're you're starting to piss them off right yeah so understanding when that happens and how much of this is going to be used for security have you do you have any security applications coming because it's been well known like when you go to certain countries you know they'll ask you a couple of questions they try to read you do some human factoring and figure out if you're lying it's one of my favorite genres of television show is the people going through customs and they're trying to read if they're like sneaking into the country or sneaking things into the country are three-letter agencies using this technology yet to like analyze people as they come into buildings or we haven't been working with security yet not that we don't believe that that's a good application, but we're being a little bit more careful about how this is used and trying to make trying to make this as rigorous as possible, essentially.

25:58There's been a lot of providers of kind of like facial expression reading technology, who aren't very scientifically rigorous, make false promises, and then doesn't work. You're just like signaling out people for no reason, basically, which is not, you know, we want to we want to take, we want to be very careful to distance ourselves from those applications. I think there's a huge wealth of applications that are consumer focused, that are just helping people have a better time, a better experience with an application. I think that's a much bigger market. And then there's like the more niche things like security, lie detection, stuff like that, which...

26:34Yeah. Could you tell if somebody's acting suspicious based upon their facial expression or tone of voice, like they're knocking on your door, your ring doorbell answers and ring? is like, hey, this person is scoping the house out, you know, they could use your API to figure out the expression of people at the door. Yeah, I don't, I don't know. I feel like I've mixed feelings about that. I think in some cases, if you have enough data, you could start to do something in that direction. But if people are really good, if they're like, if people are criminals, they're pretty good at disguising their facial expressions, probably.

27:08Yes, they're sociopaths, they're criminals for a reason, right? So you would catch the neophyte criminals, the bad criminals, and the good criminals would just sell right by. You'd be like, yep, that person's 100 % trustworthy. And you'd let Hannibal Lecter into your house. Yeah, I can see how you need to be thoughtful about this. Yeah, we don't want to make promises that we can't keep, right? Well, I mean, yeah. So in that regard, let's go through the emotions that you feel you've nailed. And then tell us, what are the ones that are hard? I tried to do existential, and that was really hard, obviously.

27:40We're not there yet for existential crisis. yeah i mean um the way that you respond to expressions has more to do with almost subtle things that you can't name versus like overt emotions and it depends on the situation like if i'm using this like comedy thought um which i can prompt this is like a prompt that makes it a comedy thought basically um it'll respond to my emotions completely or my expressions completely differently than it than like a therapy use case so let's try it oh okay great whoa well well well look who decided to grace us with their presence tonight i don't worry i promise i'll keep it pg-13 for now you know i gotta say i am a little shocked that any of you fine folks decided to show up being seen on who in their right mind would willingly subject themselves to an evening of my particular brand of, shall we say, edgy humor.

28:36But hey, I am not one to judge. Can you make a joke about Alan Cowan, the CEO of Hume AI? Sure thing.

28:48Ooh, you want me to take a swing at the big cheese himself, eh? Well, twist my arm, why don't you? I'm going to say, I am not the biggest fan of that Alan Cowan character.

29:01talk about a guy who needs to have that massive ego of his deflated a bit am i right actually you know what i think i have the perfect way to do it um i say we round up a team of trained monkeys dress em up in business casual oh that's that's really offensive you've made me very upset whoa there partner it looks like i may have gone a little too far with that one huh uh sometimes i get a little carried away with the whole edgy comedian thing you know so like you can see in that case i don't want it to be that sympathetic like it's doing its thing and so you can determine what ai flavor you want to have interacting with folks you'd have one that's cheeky and playful you could have one that's super empathetic but in you know maybe not going over to patronizing but you could see this like if i'm calling the support line for united they might you know think i'm a new yorker who talks fast and i just want to get to the point or i could be from the south and into southern hospitality and it could take its time with me and ask me about the weather and how i'm doing and you know a little bit of chit chat some people like in the south i notice uh versus in new york where they're kind of get to the point.