Replay: Could a robot fix the loneliness epidemic?

0:00Hey, folks, Jeff Berman here, co-host of Masters of Scale. I am thrilled to share some of the new names who will be joining us at this year's Masters of Scale Summit. They are the leaders driving the most pressing conversations in AI. Replit founder and CEO Amjad Massad, Cloudflare's Matthew Prince, Signal president Meredith Whitaker, and many, many more who will take the stage this October 20th through 22nd in San Francisco. We want you there with us, too. Join us at mastersofscale.com slash pioneers. That's mastersofscale.com slash pioneers.

1:00and the bottom line, how they navigate uncertainty, pressure, and high-stakes moments. I'm Bob Safian, former editor-in-chief of Fast Company, and I'll be your host as each episode breaks down what you need to know right now. You can find Rapid Response wherever you get your podcasts.

1:32Rana el Kaliouby:Abby is a social humanoid robot. She's about 110 centimetres tall, size of like a young child, very bright and colourful, and she has a lot of personality. So she's very inquisitive, very curious, very childlike in nature. When people think humanoid robots, they think the Terminator, that is not Abby at all, and people just kind of like lighten up a lot and realise that robotics doesn't have to be something that's aggressively masculine and intimidating. What are you most excited about this year? I want to walk down the street one day and see more abbeys in the streets than people walking their dogs.

2:11Rana el Kaliouby:That's what I'm excited for. Oh, wow. Tell us where you are in your journey. Everyone, when they talk about the robotics industry, they talk about it like, it's coming, it's coming, it's going to happen in these next few years, it's around the corner, etc. But it's not, like, for us, it's not coming, like, it's here already. On today's episode of Pioneers of AI, I'm interviewing Grace Brown. She is the co-founder and CEO of Andromeda. They are a company building a social humanoid robot designed for the aging population. You know, this isn't your typical Terminator-looking social robot. This is more pint-sized, and it even blows bubbles.

2:48I'm so excited to share my conversation with her, so let's dig into it. I'm Rana El-Khalyubi, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

3:08Hi, Grace. Welcome to Pioneers of AI. It is so great to have you on the show. Thanks, Rana. I'm super excited to be here. You've been building robots since you were 15, which is so cool. How did you get started? it?

3:21Rana el Kaliouby:So I was very fortunate. I went to a really good high school and we had like an engineering subject as part of like year 10 and year 11. And that was really my introduction into robotics and you know, engineering as a whole discipline. Prior to that, like I'd always, I think like most engineers, I just thoroughly enjoyed like maths and physics. Like I did quite well in school in these subjects. I found them to be a lot of fun. When I was really young, I wanted to grow up and be a mathematician. But then, yeah, when I took that first engineering class, we had to build this like three degree of freedom hydraulic arm.

3:57Rana el Kaliouby:And it was just so much fun. So that's kind of what got me down. It also introduced me to like Arduinos and MATLAB and writing code. And so, yeah, it was, yeah, that's where it all started. That's so fun. I love Arduinos. When my kids were younger, we would all do like Arduino projects together. So then you went on to start a company, your company Andromeda, and Andromeda builds social robots and your robot is called Abby. So tell us about the origin story of the company. Why did you start the company? Yeah, so I was doing my mechatronics engineering degree at the time. I had my degree during the COVID pandemic.

4:34Rana el Kaliouby:So a lot of my classes were online. I was living in a student dorm at the time. And it was during that period that I really the idea to build Abby really started. So you wanted to build a robot friend or a robot companion? Like what was the impetus? At the time I had really, really modest ambitions. So I really wanted to just build myself a system that could ask me how my day was going and to give me a hug. Because I'd realized for about four or five months I didn't have a hug from someone like a friend or a family. So it started off as just like, you know, with very modest kind of goals, as well as partly like wanting to like exercise my muscles in being a robotics engineer and working on a project that could really upskill myself and push myself technically.

5:18Rana el Kaliouby:So it was a combination of all of these things. When did you realize that, oh, there's a real business opportunity here. I need to like officially found a company. It was when I was coming out of the pandemic and I was thinking about the experiences that I had of like, you know, feeling really isolated and disconnected from the world. And, you know, the pandemic really shined this light on like the loneliness epidemic that was, you know, coming about and the health implications of it as well. Like that's the equivalent health effects of smoking 15 cigarettes a day. But the people who are most affected by this are our older generations and older adults and seniors.

5:57Rana el Kaliouby:And so, and I realized looking into the market, like many people living in aged care homes today, they experience what I experienced during COVID on a regular basis, not just during a lockdown or a pandemic. And so, yeah, so I just started going to local like aged care homes. I brought Abby in in a basket at the time, just seeing if anyone, you know, just seeing if she could really add some value for this industry. What are some of the ways Abby's being used today? Every individual uses Abby ever so slightly differently, but how I'd break it down is there's really like two areas of interaction.

6:35Rana el Kaliouby:There's group interactions or there's one-on-ones. In the group activities, Abby does like more, what I call more performative activities. So she like does dances and exercises, meditations, quizzes, all the activities that get people a bit more socially engaged in like a home. But then in the one-on-one interactions, it's very much like building a relationship with the individual. So having like, you know, regular check-ins with them, asking them how their day is going, speaking with residents in their native language, which is so, so important for this demographic because depending on your cognitive decline, if English is your second language, you'll often revert back to your native language.

7:13Rana el Kaliouby:So communication is like this massive, massive barrier in the industry. And so what we found is Abby being able to go into a home and just check in on a resident and ask them how their day is going in their native language has dramatically increased the quality of their life. I want you to bring Abby to life for us? Because you were very intentional about your design decisions and how you built Abby. So what does Abby look like? Yes. So we've modeled her off like the size and physique of like a young child. So she's about 110 centimeters tall. I think that's like four feet, I think. About 110 centimeters tall.

7:50Rana el Kaliouby:She's bright and colorful. She's anthropomorphic. So she's got arms and legs that mimics like human movements. She's got these bright, like large oversized like purple eyes that kind of like stare at you um so all of her designs were um meant to be very human-like in terms of like being relatable in the movements and the gestures but not so human where it crosses the uncanny valley curve explain the uncanny valley for folks who are not familiar with it yeah so the uncanny valley is basically when something that tries to mimic something to be very human-like but you can tell something's off so then it makes you or makes the audience a little bit uncomfortable creepy a little uncomfortable a little creepy something's a little bit off so lots of people who are building like humanoid robots with fake prosthetic skins or you know faces like it often makes people really uncomfortable because they can identify that it's not a real person so we wanted to make Abby look very robot-like but also still make her feel real so like real as in like you know when you interact with her you feel like she's like this brought to life character that you're having a real interaction with and when you look around the market as well like a lot of other humanoid robots, even if they're not crossing the uncanny valley curve, they're kind of copying science fiction really closely, building these really tall, militant like terminators.

9:05Rana el Kaliouby:And we didn't want to go down that area either, because if we're going to go into people's homes, people have to feel comfortable having this product and this system in their homes. And so people have to have a lot of trust with what they're adopting. And so that's also what we were optimizing for, something that was very approachable, non-threatening, and quite endearing. I want to double click on that because it was so refreshing to see a social robot that is warm, right? There's a lot of funding going into the physical AI space in general, but specifically humanoid robots. But I look at some of them and I'm like, wow, like whoever, you know, designed these things that it's so hard to imagine this thing roaming around my kitchen, you know even if it's the most helpful robot ever I think a lot of these robots are missing the human element yeah and kind of the social and emotional capabilities yeah no exactly I mean when I was thinking about Abby like all of my inspiration actually came from like Disney and Pixar characters from you know from childhood like people have always connected with these characters on such a deep level and we know that they're fictional like we know that they're not actually real but they still invoke these very real emotions from their viewers and the audience and you know different people see different parts of themselves and all these characters and and that connection that we have with these fictional characters that's exactly the connection that I wanted people to have with Abby so a lot of what I'm trying to build through Abby and through Andromeda is actually what animators have been doing over like the last century of just creating these storylines with arcs and narratives and these characters with such vibrant personalities and bringing that to life through robotics.

10:47Rana el Kaliouby:Like that's been my mission from day one. That's very cool. So what is Abby's personality like? So I recognized very early on there isn't a one size fits all approach. Like she adapts to the audiences that she interacts with. The out of the box Abby is, you know, quite inquisitive, quite curious, very childlike and very playful. But she does tailor and adapt her personality based on who she's interacting with. because everyone there isn't a one-size-fits-all approach some people want you know to talk deeply about philosophy or history and other people just maybe want to do more like playful things like do a dance or do tai chi or other people might want to speak in different languages and we even have a version of Abby that's in a nursing home it's a catholic nursing home and so that version of Abby is actually a christian Abby so wow she adapts to like the different environments that she's in and who she's interacting with.

11:45There's a version of Abby, I believe, that has like a bubble, like her hand is like a bubble. Tell us more about that.

11:53Rana el Kaliouby:Yes, yes. So she's got her left hand is actually a bubble gun. I quite like it. This is when we had her like in the, we had her in like a children's hospital. So we thought bubbles would be something that would be nice and playful. To me, Abby is always meant, what she stands for is contradicting the industry. Like when people think humanoid robots they think the terminator they think you know something that's like quite intimidating that is not abby at all and when they see her bubble gun they think the same thing they're like what is what does she have on her left hand and then she blows bubbles and people just kind of like lighten up a lot and realize that robotics doesn't have to be something that's like aggressively masculine and intimidating it can be something that you know is a lot more playful and enjoyable and can you know it can really improve the quality of life for people and enrich people's lives.

12:39Rana el Kaliouby:And that's been, you know, what I've wanted to work on as well since day one. Yeah. You know, there's not a lot of women in the robotics slash physical AI space. And why do you think it's important to have more women and more diversity in general designing these robots? If you look at all of the robots in market today, I really think they're a reflection of the teams that build them. Say that again, Grace. Say that again. It's true. it's so true I mean I get the feedback all the time or questions all the time like people say compare me to Abby a lot and they always say that they see my personality in her they also think that she has my voice but she's got an American accent I don't but um but but but if you look at even even now like I've got lots of engineers who are working on Abby now and we still get comments where if people who know my engineers and then they look at Abby and they go oh I can see like this is like Tom's personality coming out of Abby or this is Erica's personality coming out of Abby Like they start seeing the developers in Abby.

13:40Rana el Kaliouby:And so it comes down like the software, but also like the hardware and the design as well. Like all of it is just a reflection and an embodiment of the values and the personality of the developers and the teams that are building these systems. Unlike a lot of the robots out there, Abby has social and emotional intelligence. How Grace house trained her robots after a short break.

14:18On Masters of Scale, iconic leaders reveal how they've beaten the odds.

14:22Rana el Kaliouby:Asking really strong questions is a superpower. You want to show up with something radically different and how they've grown companies to incredible heights. The greatest rewards always come from the greatest risks. That's hit the gas. Airbnb, Zillow, Microsoft, Liquid Death, and more. Hear from the founders who've changed the game. It's anything but business as usual. Find Masters of Scale on Apple Podcasts, Spotify, YouTube, or wherever else you get podcasts.

14:58Welcome back to Pioneers of AI. You can also watch this episode and see what Abby looks like. by heading over to our YouTube channel. So Abby obviously has eyes. That's the computer vision part. And I'm guessing she has ears. So that's like she's listening, right? So people can converse with her auditory. Exactly. What other sensors does Abby have?

15:19Rana el Kaliouby:She's got three main inputs at the moment. So she's got her microphones, her visual sensors, her cameras. And then she's also got a LIDAR for navigation as well. Although later generations, we may move to a fully like vision based navigational stack. So she's got like this microphone array that kind of, you know, when you speak to her, she understands like where the voice is actually coming from, voice detection as well. So who's speaking to her. Diarization in like group settings is actually a really complicated challenge. There's so many things in building something like Abby that is so complicated.

15:53Rana el Kaliouby:it's part of like the enjoyment of actually really building abby is like solving all these deeply like hard technical challenges but everything when it comes to building abby from the design hardware software everything boils down to the fundamental questions of almost like like what does it mean to be human like how do you coordinate your arms and legs and head in such a way that you're expressing a very specific emotion or or how you want to communicate very specific things or, you know, even when it comes to a conversation, like it's never just like a back and forth, like a very old school chatbot, but there's so many nuances in understanding who's speaking when they're pausing versus when they're waiting for you to say something.

16:34Rana el Kaliouby:There's a lot of nonverbal cues and communication as well. There's a lot of like nuances. And even if you said something to me, like, oh, look over there, like that's really complicated for a robotic system to really break that down and understand, you know, look at where they're pointing, et cetera. So anyways, there's a lot, but it is a lot of fun. I spent 25 years of my career building emotional intelligence, right? Emotion AI into machines. And one question I have for you. So Abby does have vision, but can she see facial expressions? Can she understand if I look excited or scared or sad or mad?

17:08And can she hear it in the voice as well?

17:11Rana el Kaliouby:Yes. So that's all of the things that we are aiming to be world-class that is that kind of I put that all under the back bracket of like the psychology of interaction so she's looking at all of your micro gestures and micro expressions she's looking at the warmth and tonality of your tone of your voice um and and so that's like her input and then when she expresses things her output we're doing the same as well like what is her voice prosody the pitch um you know and and sometimes also just like the silence of when to gesture versus when to sit a But still, like even all of that matters as well.

17:45Rana el Kaliouby:I think right now the industry is definitely over-indexed on functionality and capability. And that's very important. Like it's so important. Like that's going to like revolutionize so many different industries. But when you think about going into a domestic environment in people's own homes, it's going to be so important to understand the nuances of, you know, exactly how the other person is responding and reacting and their emotional state. Yeah. It's actually also pretty complex because I remember from the work we were doing with some of the roboticists, even deciding when to mirror a person's emotion versus trying to kind of nudge the person out of an emotional state.

18:24That's all like, right, when to, as you said, when to pause, when to respond. These are all like dynamics that are kind of hard to incorporate.

18:35Rana el Kaliouby:Oh, it's so complicated. It's so, so complicated. and like the things that I'm most excited about is around like you know the the personalization of uh these these home companion robots like it's not just about like understanding the facts of like the individual like if you interact with Abby like she might learn your name you know that you know you have you know two or three children that they go to school in this suburb or this area and these are all the facts but what becomes really like what's to me is going to be so powerful and actually magical is when they start understanding all like the nuances of like, oh, like if, if Abby's in a nursing home and she talks to someone called Margaret, she learns that Margaret prefers to be called Maggie and that when Abby actually talks too fast, that's overwhelming for Maggie, even if Maggie never explicitly said that.

19:20Rana el Kaliouby:It's going beyond just learning the facts of an individual, but starting to really understand them as a person. Well, at the end of the day, the majority of that, like over 90 % of how humans communicate is nonverbal, right? And so if these robots are going to be in our homes interacting with us like on a day-to-day basis they need to kind of incorporate these non-verbal signals into their input channels which is exactly what you're doing and I love that what is the underlying like kind of large language model or world model that Abby uses um so she uses quite a number of different models that are available online today so you know she's got like we talked about earlier all of her like sensory input from like her microphone array to her cameras to her lidar and all of that is fed into like her behavior tree which then has uses different models and different parts of her architecture to then give a response whether it's a behavior or a verbal response or a gesture or suggesting an activity so um I think a lot of people have the misconception that and understandably so that um you know companion robots today are just like almost LLM rappers but um when it comes to building like a social robotic system or a social interface like we talked about earlier on there's so many nuances that are being brought into like you know like the like we talked about like the voice prosody the micro expressions the sentiment analysis and then also then like structuring the personality of Abby as well like hardware is inevitably going to reach a failure like failure points are going to happen and so for me it's like also how does the system how does Abby or any other robot respond to failure points like you know will it actually recognize it's made a mistake and apologize for it for example like a very simple one that we do there's a few simple things we do so like if the internet connectivity is really low like very very common technical challenge for robotics Abby will actually respond and say things like I'm really sleepy today like please don't ask me too many questions but that's not because she's actually sleepy and people find that really funny and then and then they're also more forgiving of mistakes so they're more like usually when the internet slow, there's like slower latency, slower responses, and people are more forgiving when you can actually, if you have that like social operating system layer and you can actually apologize for the mistakes or the technical hindrances that have just occurred.

21:42Does that mean that all of the processing that Abby, like all of the compute happens in the cloud or does she do any of the processing locally?

21:50Rana el Kaliouby:We have a hybrid where there's some in the cloud and some local, you know, in the environments that we go into, like connectivity sometimes can be unstable. So for us, we've actually got a hybrid where if the connectivity is low, she uses more local models, which are obviously less engaging and less interactive. Another thing we do as well is actually when you interact with Abby, we get her to say things like ums and ahs, just like we do as humans. And there's really no reason for a robot to do that. But firstly, it makes it more natural. but secondly it also kind of masks the latency that she has does Abby have memory yes yes she does so yeah so Abby has memories of you know who she's talking to like all the basic facts from that she's collected from their interactions um but then also like like I said like that deeper level understanding of you know understanding that maybe maybe Maggie she goes for a very specific football team um and that Maggie doesn't like people interrupting her in the morning before breakfast, but is very, very chatty after lunchtime.

22:52Rana el Kaliouby:So Abby might not go in before breakfast, but will always make sure she does a check-in after lunch. I think memories is going to be one of the next really major, I don't know if it's going to be a major breakthrough, if that's the right way to put it, but it's definitely going to be something very significant for the field in this next wave of robotics. Yeah, I think of actually memory as a potential competitive moat if you can crack how to represent that in the right way. But again, And it's one of these things which is quite complex, I believe. It definitely is. It definitely, definitely is.

23:23Rana el Kaliouby:You've also got to consider privacy and compliance and how you're managing the data. So it's not a small task, that's for sure. And it's true. There are some real security concerns when it comes to these kinds of systems. But it's not just about the data. It's also about getting the physicality right. Have you seen these viral videos of humanoid robots toppling over? Honestly, they're kind of comical, but they're also quite concerning. We'll get to that and more after a short break.

24:05I want to talk about like safety a little bit and accessibility, right? Like, especially that you're deploying Abby with elderly individuals. What are some of the considerations that you've had to think about? Like, for example, you know, what if Abby, I don't know, runs out of battery or is about to run out of battery? Like, what does she do or what does she say?

24:25Rana el Kaliouby:Yeah, no, there are major, major ethic, like ethical and philosophical, you know, questions around this space that my team have to consider building something like Abby. um going to like the physical safety like that's like front of mind for us when we're designing her so like right now even though she's anthropomorphic so she's got arms two legs and her head she's actually on like a wheeled base so when she powers down like you know if the power was cut out for whatever reason she actually doesn't fall over so like her arms and her legs will kind of maybe go down but she doesn't actually fall over or topple over one of my concerns around the use of AI chatbots for now is around safety, right?

25:03And guardrails around like what kind of recommendations this chatbot or this AI system can give you, especially around mental health, right? So how does Abby handle a situation where a person is potentially considering harming themselves?

25:19Rana el Kaliouby:So we work really closely with all of our customers and our partners in how we design Abby's guardrails and how she manages the data, how the conversations are stored and exactly what happens next. So usually when conversations reach fairly sensitive environment, like sensitive topics, Abby will always default to making recommendations to speaking to like one of the healthcare workers or the homes. Even the Catholic Abby that we have, like the Christian Abbey that we've deployed, my team tested like the boundaries of the Christian Abbey quite rigorously. They actually had the time of their lives that afternoon, just seeing how far it would go.