How to build robots that can live alongside us, with Cynthia Breazeal

0:00Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential. a non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up, every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no preset spending limit, they were able to spend more and earn more rewards while bringing their vision to life.

0:36Today, Raz's success is proof that with passion and the right support, it's possible to make your dreams a reality. Learn more at CapitalOne.com slash business cards.

0:51It was literally a moment where NASA land Sojourner on Mars. A faint signal from the red planet and confirmation that Earth's latest envoy has arrived. And we're all watching the field of robotics, huge celebration, right? Oh, my God, like achievement, achievement, you know. And then I remember thinking to myself, OK, now wait a minute. We send robots into oceans. We send robots into volcanoes. Now we even have sent one to Mars. Why are they not in our homes, right? It's like I grew up, you know, with Jetsons and this vision, Star Wars, with this promise of social robots walking among us, you know, interacting with us as part of our everyday lives.

1:31That was just that moment where I'm like, you know, nobody is actually really working on that problem when we think about a fully autonomous robot that anyone could interact with or collaborate with. Cynthia Brazil took that challenge on. Cynthia is a professor at the MIT Media Lab, and she's MIT's dean for digital learning. She's a pioneer in the world of social robots, the kind of robots that are socially and emotionally intelligent. They can interact with humans in a way that feels natural. And today, I am so excited to share my conversation with her. Not only is she a pioneer in this field of social robots, she's been a big inspiration to my own work as a computer scientist developing emotionally intelligent AI.

2:23Cynthia and I are going to talk about how to build robots that can live alongside us, the wide-ranging applications of social robots, and why we need AI literacy now. I'm Rana El-Khalyubi, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

2:54Hi, Cynthia. Welcome to Pioneers of AI. Hey, Rana. So great to be here. I am so excited for our conversation today because, as you know, I'm a huge fan. And we've had the pleasure to work together when I was a post-talk at MIT and then at Affectiva. So it's so awesome to have you here. I'm so excited for our conversation. Yeah. Fan club is mutual. So proud of everything you've accomplished, Rana. Seriously, it's been so wonderful to see your career just blossom. Thank you. Before we dive into your career as a robotics pioneer, I just want to take a moment and acknowledge kind of where robotics is at today.

3:32I mean, I feel like robots are having a moment. There's been like, I don't know, over a dozen humanoid robotic companies started in the last couple of years. So what do you think of this moment we're in? Yeah, I feel like robotics is a field that feels like we're either always having a moment or about to have a moment. it's like it's one of those fields that turns out very very hard robotics are hard yeah robotics is hard you know as i say you know mother nature doesn't care about your algorithm or how hard you've worked so um yeah i mean it's exciting to see humanoids start to you know again get a lot more interest obviously i you know pioneered this area of of social robotics and human-robot interaction, you know, envisioning this day when everyday people would be able to interact with fully autonomous robots in a supernatural, engaging, productive, but also uplifting and rewarding way.

4:30So it's, you know, it's been quite a journey. I mean, that work started, as you know, like in the 1990s when I was a graduate student with Kismet. Tell us about Kismet, because Kismet actually inspired a lot of my research, building emotional intelligence into machines. So tell us a bit about Kismet and how, you know, the origin story, what inspired you? So just historically, so Kismet is recognized as, you know, kind of the world's first social robot. So I was kind of inspired by, you know, the personal computer when it went from these huge, you know, machines, you know, back rooms to like, what did it mean to put a computer on every desk?

5:07So I'm like, so what is the universal interface of robots? And, you know, know, we had certainly seen people completely anthropomorphize and interact in social interpersonal ways with our autonomous robots. At the time, and we're talking about 1990s, they were very much inspired by models of insects. Like these were literally bug-like robots, six-legged robots, you know, very simple robots compared to like what we can build today. But people, of course, would imbue, you know, psychology to them, you know, treat them, you know, as a living thing. So we already saw that. So it was like the social interface is the universal interface.

5:45And so what does it mean now to create a socially and an emotionally intelligent machine, a robot, right? That was kind of how it really got started. And the vision was about to be natural for people to interact with in our human-centered ways. and I started as I started to kind of dig into that question I read like ton of psychology took a lot more classes and you know understanding not just the social behavior of people but also animals right I was you know ravenously trying to look at any literature on models mechanisms how to understand what it would mean to create a a first robot right that could engage people in this natural face-to-face interaction where it was clear there's no way we're going to do an adult level interaction.

6:28I mean, we are the most socially sophisticated, emotionally sophisticated creatures on the planet. So I thought, can we start by where it starts with us, which is literally the mother-infant dyad. And from those interactions, right, when you look at all the development psychology work, like from those interactions is where, you know, we, we both have our biological basis for developing, but those interactions are absolutely crucial that the caregivers, that those adults, you know, interact with the baby as if it's already socially and emotionally intelligent, arguably, although it's not quite, it's not there yet.

7:02I mean, it has to learn and develop that. I mean, we take years actually to develop into our full social and emotional intelligence, right? We're incredibly sophisticated. So that was where we started. You know, Kismet was in many ways designed to be like a young creature, infant-like creature, where the whole focus was on, you know, when you read the, again, the psychology literature, the development of psychology literature, they talk about the dance of communication. And this is actually really important because I think despite all the advancements we've had in large language, we are still not in the dance of communication with our technologies.

7:40You know, it feels very much like playing chess. You take a move, the AI takes a move. It's like communication with people, it is a dance. It's mutually regulated. We're constantly observing each other. We're responding to subtle cues and nonverbal cues, gestures, tones of voice. And so like Kismet was asking that question, like what are the mechanisms? What are the ways that we can basically start to design a machine, an autonomous robot that could engage in the social and emotional back and forth rhythm and dance of communication. So we looked at models of emotion. We looked at social models.

8:17I looked at motivations. I mean, I basically took a lot of models from, again, animal and human behavior and adapted them to an autonomous robot. So, I mean, the robot didn't model exactly what a child is, but it was inspired by that. It's the first robot that actually successfully did that. So you know what is interesting about, I want to go back to this dance of communication and this idea that a lot of our communication happens through nonverbal signals. Yes. But I look at examples like figure AI and Tesla's Optimus robot and, you know, physical intelligence's robot. They're all trying to build these robots that are going to work and live alongside us.

8:55Do you think they are factoring in this kind of social and emotional intelligence? I have to confess, like, I can't really imagine, you know, figure AI or Optimus in my kitchen roaming around. But yeah, do you know or think they're working on this kind of key component of a humanoid robot? Well, so I would say not to my knowledge, you know. And so when I started the work with Kismet, the field of humanoid robotics was just getting started as well. And so, I mean, looking way back now, we're talking about the 1990s, right? Right. So Japan was building these robots because, you know, one of the main use cases, even back then, they were very concerned about was their growing aging society.

9:41So they were very cognizant and concerned about this. So at the moment that Kismet and the social aspects of robots came out, I would say back then in Japan, again, the 1990s, a lot of it was still around what I think you're alluding to around it, which is more about the physical capability of the robot, the ability for a robot to walk, open doors, manipulate human tools, do physical tasks, right? But then this work from Kismet came in. And of course, they were like, yeah, these robots have to be able to interact with people who know nothing about robots, right? It needs to be able to interact with everyday, ordinary people.

10:17So that was enough of a community, I would say, that it propelled the field, now the field of social robotics and human-robot interaction forward. I kind of want to make a distinction because humanoid robots and social robots overlap a lot, but they're not the exact same thing because social robots don't have to be humanoid. Exactly. Social robots is more about an intelligence and a capability set that can be applied to physical robots, virtual, I mean, anything really, when you think about the core tenets of it. To me, it's really about how do you build socially and emotionally intelligent technologies that are really able to partner with people in a very natural human way.

10:58And then if you can do that, there's many, many, many, many applications for that. So there's humanoids as one kind of technology that could take advantage of that. So anyway, back to your point, I don't think people building humanoids today are thinking deeply about the partnership question with people. I think they're still focusing on the physical capabilities. But at the end of the day, the ability to do that with people is going to be crucial to any like real world application. In a minute, we get into those real world applications and how social robots could be the key to democratizing access to early education.

11:35That's after a short break.

11:54If you've spent any time building AI products or leading technical teams, you know this. Transformation doesn't fail because of ideas. it fails because teams can't move together. Enter Atlassian's teamwork collection. It has planning in Jira, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at Atlassian.com slash TeamChanger. That's A-T-L-A-S-S-I-A-N dot com slash teamchanger.

12:43So what are some of the applications of social robots? Yeah, so this was very interesting, right? So, you know, if you look at the progression of the field itself, so in the early days, you know, it was, you know, and we're still, of course, trying to advance this question of algorithms for creating more socially and emotionally intelligent machines. But a lot of it was at that level, right? We're trying to explore algorithms and so forth. Then we started to think about, okay, what are the use cases, right? So, you know, we started looking at areas where social and emotional support were actually really important for human outcomes.

13:16Well, it turns out there's lots of applications like that. And what we saw in areas of like particularly health, chronic disease management, mental health, education, you know, we are not able to actually train professionals at the rate to meet the growing demand. So there was a question of how do you build these technologies that can augment and support the human social systems around how do we thrive? How do we flourish? How do we become lifelong learners? So those became a lot of the first really, I would say, anchor use cases of the field of social robotics, at least the ones that I certainly focused on.

13:57You know, if you design these robots in a way that engage not just people's cognitive abilities or physical abilities, but their social and emotional as well, we as human beings engage much more deeply. And not surprisingly, we end up doing better. We are more successful with the technology when we do that. Can we give the example of Tagus, I think was the robot's name? Absolutely. So we were looking at early childhood education. So, you know, first of all, young children don't read and they don't type. So like the normal forms of interacting with computers, like that ain't going to happen. Right.

14:35And so we thought that was, from an AI challenge, a great use case because the robot would have to literally be able to engage in social and emotional, playful encounters with children in a very natural way because children are just going to be kids. Let's face it. Kids are going to be kids and it's wonderful and it's messy and it's complicated and it's delightful. And it's like, if you can do that, you must be doing something right. You have to have figured something out, right? So but then from a social standpoint, you know, we look at, you know, kindergarten readiness across different social economic classes of United States.

15:10We see a huge disparity. The more affluent kids are far more ready to enter kindergarten than kids coming from less affluent families. There was a huge inequity there. So the question is, you know, if AI, if robots are scalable, affordable, human-centered, if you could design them in a way that supported the teacher, could engage the students, could you actually have a much more equitable early childhood education? So the Tega robot was designed as one of our earliest platforms to explore that. So Tega was designed, interestingly, not to be a human-like tutor like a lot of what you see today. Tega was designed to be a peer-like learning companion.

15:55Peer-like because we did not want the teacher feeling that Tega was competing with their role or value in the classroom in any way, shape, or form. And also we wanted the robot to basically serve as this practice partner, right? This playful practice partner that would engage children in these learning games and activities, things like dialogic storytelling or educational games as a peer where it's kind of like if your dog became really, really smart and you could play games. It was like that, right? So children delighted in the robot. They were not confused. They didn't think this robot was like their human friends.

16:31I mean, this was a big concern that we heard quite a lot is what is this going to do to children's social development? We studied that too. We're like, even if children can associate a kind of social relationship-like thing with the robot, they knew it was not like what they have with other people or their best friends. Like there was no confusion in their minds about that. But they found value in what the relationship between, you know, what the Tega was, right? It was a different kind of relationship. So this was really fascinating, right? But there was also this companion animal dimension, which was fascinating because children could make mistakes in front of Tega.

17:08Like, it was okay. There was no sense of being embarrassed or losing face, that they may be more worried if, you know, the teacher was doing this, feeling they're being evaluated, right? They didn't feel any of that with the social robot. So it's just the different kind of relationships supported a different kind of engagement, even with their human peers, where we could see that it really did support engagement and learning outcomes.

17:35Cynthia is talking about a critical design decision when making social robots. You're not trying to replace human-to-human interaction. You're trying to augment it. Back in 2012, Cynthia founded a startup that made a small tabletop social robot built for the home called Jibo. They kind of went viral a few years back. So when Jibo came out, we were one of the first lucky families to have a Jibo in our home. And my son at the time, he was probably like six or seven. He's now 16. And he would, he loved Jibo. I mean, we had Alexa in our home. We had Google, whatever. But Jibo was different because it had a personality and a character.

18:16And I remember when you had to sunset the company and Jibo was going to go offline, my son was actually in tears. And I have to say, I thought about that a lot because I liked that they had this special connection. But also as a parent, I'm like, hmm, he's getting too attached to this robot. So was that something you saw with other families using Jibo at home? I mean, of course, we were all heartbroken about, you know, the company. We have continued to use. So basically, my research group has continued to use Jibo as a really one-of-a-kind research platform that we've gone on to explore many, many different really great applications.

18:54So Jibo lives in that context. But it was really, I mean, as you know, Ronna, you do a startup and you put everything into it. And it was just, I mean, it was heartbreaking for people not just because of the company but because of Jibo. So like what you're talking about, I mean, everybody felt that, you know, really. And I was shocked, honestly, like there continued to be coverage of Jibo a year or more after the company ended, reflecting and talking about exactly what you're talking about, like what Jibo actually meant to people. We definitely achieved that aspect of the design. And we tried to do it in a super responsible and mindful way, right?

19:36So we made some very conscious design decisions of if people wanted to ask Jibo questions about topics that a robot, frankly, has no business answering, like religion or what like, Jibo would basically say, hey, I'm a robot. I think people are awesome. That's a really important topic. You probably really need to talk to another person about that because I'm just a robot. So the robot intentionally tried to set up healthy boundaries of like what was what it could engage you around. But really what like if there's human domain stuff that really it was important for you to talk to people about that.

20:10So we were very, very mindful about things like that. And so fast forward, you know, the company had to shut down. We were using Jibo as a research platform. We were starting to look at the application of social robots for mental wellness and mental health. You know, so there was some kind of dawning of the mental health of students at universities as being like a big issue, you know, particularly at very like top universities. So we started to look at Jibo as a social robot positive psychology coach for freshmen in the dorms. So that's where we started, MIT freshmen in the dorms, right? And we saw some really provocative, encouraging results around that, where we saw students, you know, spite as the semester gets on, it gets more challenging.

20:54the students who had Jibo reported having, you know, more positive, you know, mood and optimism and, you know, all of that. And they also commented on the companionship. They talked about it as light companionship. So fast forward, you want to do a follow-on study. We're preparing to now ship Jibos all across the country to do this larger study. And the pandemic hit. And so we ended up doing this study in the height of the pandemic with social distancing, where people were literally not leaving the house. And we, again, from the data wise, we looked at different conditions. One was like the more classic kind of coach, the kind of paradigm.

21:35The other one was like the companion paradigm where Jibo also had a emotional positive goal and would do all these activities with you. And these are like well-established positive psychology exercises that you typically would do with a professional. Now you're doing it at home with the robot, but these are all very well-established kinds of things, like, you know, being able to think about three good things in your life. So anyway, the punchline is we saw the companion condition was stickier, more lasting results, even in a delayed post-test, and better results in general. And people again would comment again and again, that social presence of Jeeba when the home was so important for them at that time when they were literally, like, completely socially isolated.

22:20So, I mean, I think, and this raises, again, this bigger question, like, what is the appropriate healthy relationship between us and these increasingly conversational kind of AI systems? So, like, this was long before ChatGPT, right? ChatGPT came out, right? And then people are like, oh, my God, I can have all these conversations and blah, blah, blah. And we're like, yeah, we've been studying this literally for decades. You know, we could have told you that. How important is it? Because now people, of course, are turning to Chachipti for everything, right? How important is it that this thing is embodied?

22:59Well, I think it's, okay, so I will tell you this is kind of the overarching finding that we have. And, you know, there's always going to be, it depends, right? But when you want to look at the social and emotional engagement, much stronger and lasting when you actually anchor it in a physical interaction. So if you're doing an activity that has nothing to do with the social and emotional qualities of the interaction, you're not going to see that much of an effect because it's irrelevant. event. But for encounters and outcomes and interactions where the social emotional actually does matter, well, so the physicality definitely boosts it up.

23:37Because again, we as human beings are able just to engage more richly with it, provided you design it well. So I think now, chatbots, people having these long conversations, I just feel like now that that genie's out of the bottle and many more people are talking to AIs in this much more interpersonal way, the ethics of it, what is the appropriate healthy relationship is so important for us to design into these systems because without like AI literacy, without being aware of what these technologies are, how you should be thinking about them, people can over-rely on them. They can be potentially manipulated in ways through the social emotional channels that they wouldn't even be aware of.

24:24I mean, there's plenty of things that we need to be very mindful of on the ethics and responsible design aspect of it. But I will just say, I mean, that is also, and this is always the case, I think, with any technology, that is balanced by the huge positive outcomes. You can also see if you do it responsibly well and be really mindful about what do people need to thrive and flourish. If you start there always, you're going to end up with solutions that I think we can all feel good about. I want to take our listeners behind the scene and just kind of unpack what does it take to build a social robot?

25:02Like what are the different components? Because it is complex, right? It is so complex. Yeah. So let's start with like the perceptual abilities of the robot. One, of course is vision. So it has computer vision. You've explored a lot of like touch capabilities. So walk us through some of the different kind of boxes that you need to figure out when you're building a social robot. Yeah. So, I mean, you know, again, it always depends on the application and the use case. We have certainly looked at a lot of different perceptual channels. So some are very intuitive to us. So obviously, like you said, vision, the ability to perceive social cues, you know, here.