The making of an AI clone, with Natalie Monbiot

0:00Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential. a non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up. Every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no preset spending limit, they were able to spend more and earn more rewards while bringing their vision to life.

0:36Today, Raz's success is proof that with passion and the right support, it's possible to make your dreams a reality. Learn more at CapitalOne.com slash business cards.

0:49Have you ever wished that you could be in two places at the same time? Or maybe that there could be a second you, some carbon copy version to finally get through your to-do list. Well, in the near future, something like this can be possible. I'm talking about AI clones, a digital version of ourselves that sounds, looks, and thinks like us. For the past several months, I've been developing different AI versions of myself, using a couple of the big platforms out there. And I got to share the experience on stage with one of the experts in this emerging field of virtual humans, Natalie Monbiot. Today, we're sharing that conversation with you live from the On Air Fest stage in Brooklyn, New York.

1:39I'm Rana El-Khalyubi, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

1:57Hi, everyone. This is Rana El-Khalyubi and my guest today, Natalie Montbeau. And we have an awesome show for you today. I want to kick us off with a video, so let's watch. Okay, so at this point during the live taping, we played a video demo of one of my AI twins. You can watch it by heading over to our YouTube channel, but here's the audio version. Welcome to On Air Fest. I am so excited to be here with you. Today we're going to talk about AI clones. Let's get started. Bienvenidos a On Air Fest. Estoy muy emocionado de estar aquí con ustedes. Hoy vamos a hablar sobre los clones de EA. Comencemos.

2:43欢迎来到OnAirFest ,我很高兴能和你们在一起。 今天我们要讨论人工智能科隆 ,让我们开始吧。

3:09Yes? Okay. All right. Okay. Great. Well, I can kind of judge my facial expressions. They kind of look like me, but something's a little off and the voice doesn't exactly capture, I like to say, my soul. But that was my AI clone. That was my digital twin. And I've been experimenting with all sorts of AI cloning technology. And today we are going to go behind the scenes to uncover what an AI clone is, what are some of the applications, and of course, what are some of the ethical considerations. But before we got into all of that, I just want to take a moment and introduce the two of us. I'm Rana El-Khalyubi.

3:47I'm an AI scientist, entrepreneur, and now investor, and also the host of the podcast Pioneers of AI. And my guest today, Natalie Mambio, is a dear friend, and she is, in my opinion, the world expert in the virtual human economy. And yeah, Natalie, do you want to introduce yourself real quick? Sure thing. So just to explain the virtual human economy. Maybe I am the world expert because I coined that phrase, so not much competition so far. So I define the virtual human economy as a new era in which we each have AI twins that we can put to work on our behalf to lead better lives. And in terms of where I got that from and what my inspiration was, I've been a longtime emerging media strategist.

4:30And then about six years ago, I joined a fledgling AI avatar startup, Hour One, and helped build that business from the ground up back in the day when AI avatars and basically were only known as deepfakes. So kind of getting us out of the rut of the deepfake era into kind of positive use cases of AI twins and avatars and different commercial use cases. And through that journey, I got really excited about this idea that behind every AI avatar or any AI twin, there can be a human being behind it that can benefit from this technology. So to me, AI twin technology is an enabler for human beings in the era of AI.

5:12And it's a way that humans can collaborate with AI in a way that they can win. So define what an AI clone or an AI twin is very simply. And what did you think of mine? I mean, we've known each other for over a decade now. Did it sound or look like me? What do you think? I mean, to an extent. I've seen so many versions of AI avatars and AI twins. I think there's a certain level of kind of expectation and expectation gap. I mean, it's pretty good, right? I mean, at delivering to the camera a functional message, especially when it's doing things that you can't do. I think there's sort of like, there's this kind of aha moment.

5:46Whenever I saw my AI avatar speaking languages I didn't know, I'd be like, wow. actually I do speak that language. Like I kind of like started to believe it. And so it's kind of this weird feeling. It's like, no, but actually it's also true because in a way I do speak those languages. So it's starting to kind of feel, you know, that empowerment through AI. But yes, there's still definitely a gap. And I think the visual aspect of an avatar is very ambitious to actually communicate. As you say, it doesn't represent your soul. And is that really what we should be trying to do anyway? are there actually more like functional ways that we can use our AI twins to actually benefit us?

6:25And I would say something that people come to me quite a lot, especially sort of towards the end of last year is, so I created my AI avatar, I cloned my voice, but now what? How do I profit from this? Like, how do I benefit from this? And I think there's been this kind of a bit of a bubble in that respect where people haven't really reaped the benefits from their AI clone. So I think we're transitioning into an era now where people really got over the fact that you can do this. It's cool, but not. Right. And kind of like it's sort of settled. And now it's kind of like, well, now what? And so I think we're entering a really interesting phase of being able to clone aspects of ourselves that are more valuable and that we can use to profit.

7:05So let's go behind the scenes and kind of explore what it takes to make an AI clone. And I want to start with kind of products that mimic your likeness. And in this case, it was my, you know, visual appearance and my voice. There are so many products out there. And to be honest, you know, they're all still evolving and none of them are perfect yet. My video clone, I used a company called Hey Jen. And the process was actually pretty simple. I logged on, you know, signed up for the, on their website. I turned my video on on my laptop and created a three-minute video, just kind of pomficating for three minutes, uploaded it.

7:45And then basically, I am now able to type in whatever script, in whatever language, and my clone will present it on my behalf. So you've been in this space for many years. What is the technology behind this kind of video cloning? Right. So what we're essentially doing is we're taking a little bit of video footage. So in your case, just a couple of minutes of video footage. In some cases, it's just a photo. You're seeing maybe if you're following the kind of AI news stream or firehose, you'll see kind of different breakthroughs from different, you know, companies from all over the world showing what kind of lifelike videos you can write, even from a photo.

8:27So essentially what it's doing is it's taking your characteristics, the data points in the training data, which is essentially whatever you gave the AI to train on. And it is replicating that. And you breathe life into this replication of you. And then it mimics how you might have said those words yourself. And what is also really cool is it does a lot of lip syncing. How does it do that? Well, that's kind of the magic of the AI algorithms and something that our category of AI avatars for the last few years have been extremely focused on because it's actually very difficult to nail. Just a little tidbit, at the very beginning of hour one, so 2019, we actually used a different technique.

9:10We actually had a human being behind the AI avatar, moving the AI avatar kind of in real time. And the lifelikeness and expressiveness was much more accurate and compelling because actually you had a human being kind of driving it, but that was not scalable. So we actually had to take a big step back in terms of sort of the effectiveness of the experience in order to be able to build a product that actually was more scalable. Yeah, that is kind of interesting, because that's like motion capture in the movie production space, right, where you have actually a person driving the actions, driving the movements, and the quality of the output depends on that human being, which is a great segue to the next company we want to feature.

9:55It's another big company in the space, they're called Respeacher, and they make voice clones. It's actually the company behind the movie The Brutalist, where Adrian Brody spoke perfect Hungarian, thanks to AI. They basically essentially used Respeacher to have him speak Hungarian. And we'll show a demo in a minute. But Natalie, how does that technology work? Right. So it's actually a little bit like what we were just talking about. There's a voice, a real person's voice that's driving your voice, right? And so it's giving it the emotion that is desired, but with the mask of the voice that you want it to actually be communicated in.

10:35And so similar to the earlier technique that I mentioned at the beginning of hour one, it actually creates a much more lifelike effect, but isn't scalable, which is perfect in some use cases. And in others, it's not when, you know, scale is the objective.

10:53After a short break, we will get to hear what this technology sounds like in action. Stay with us.

11:19fails because teams can't move together. Enter Atlassian's teamwork collection. It has planning in Jira, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at Atlassian.com slash team changer that's a t l a s s i a n dot com slash team changer we're going to talk about the use cases of some of this in a second but let's um listen i guess to a demo of re-speecher so the way this worked is essentially we had our producer rachel um record a promo script in her voice so let's listen to that.

12:15What does the AI revolution mean for jobs, for education, even friendships and dating? Who are the people creating this technology and what do they think? So this is Rachel's voice and she recorded the script. So now let's listen to how Reese Beecher transforms her voice into my voice. And to train this model, we used excerpts from episodes of Pioneers of AI. So my voice as a podcaster, if you like. So let's listen to that. What does the AI revolution mean for jobs, for education, even friendships and dating? Who are the people creating this technology and what do they think? OK, then we trained a different version of this using data from my audio book.

13:04So I recorded my book. That was a whole week worth of data. I think we only used a short clip. So let's listen to that. What does the AI revolution mean for jobs, for education, even friendships and dating? Who are the people creating this technology and what do they think? Natalie, what do you think? I like that one. You like that one? I like that one a lot. Yeah, that really sounds like you. Yeah, no, no, no, that's great. And it's like your intonation and, but it's not, it's Rachel's. Yeah, exactly. Which is incredible. Yeah, well, Rachel has gotten to know me really well. So I think she's also channeling a bit of it.

13:40She is your twin. What I find interesting about, actually, the first one is a little like, you know, news anchor Rana kind of thing. And sometimes I'm like, ah, maybe I should sound like these people, these alternative versions of Rana. I still think it's missing some of my energy, though. I don't know how we capture that. I think that we shouldn't. I think we should leave that part to you. We should delegate our AI twins to have very specific roles and to execute specific functions. And it's your job to use that free time that you have to enhance your soul. Okay, I love that. I'm down for that.

14:19Let's talk about some of the use cases. One of the applications we're exploring in our podcast is using Respeecher and my voice clone for voiceover scripting. So instead of me kind of carving out time every week to do some of the voiceovers, we can just have Rachel recorded and read the speecher kind of reformulating it in my voice, which would be a huge time saver. And I think it makes just our team more productive and efficient. I think a lot of people can easily see the use cases in entertainment and media like movie production, podcast production. What are some of the other use cases that you see for these types of technologies and clones?

14:58Are you talking about voice in particular or kind of more broadly? Voice and video, both. Okay. So I'll just give you an example, because I'm always thinking about how can we actually use these technologies to actually benefit and quite literally profit. Is everyone here or anyone familiar with Eleven Labs company? Okay. So Eleven Labs allows you to clone your voice or access millions of cloned voices and, you know, generate endless amounts of content with these cloned voices. also allows you to do so with your own voice and you can actually put it in the 11 Labs marketplace and you can make it available for use and you can make money.

15:36So actually sort of every month I get some random like thing in my account, like 20 bucks, 30 bucks, and I'm just making money through making my voice available for commercial projects. Do you know who's using your voice? I actually don't. And maybe I should be concerned about that. It's on the bottom of my to-do list to kind of work that out or sort of manage that. But you can do things like set your price and you can kind of, there's no commitment on my part in terms of how long I would make my voice available. So I could just remove my voice any day. So the value of my voice is lower, but I can increase the value of my voice by setting parameters to make it like I will, you know, you can, I'll commit for a year.

16:17And so therefore, you know, companies will invest in your voice and pay more for it. So it's kind of interesting to think about kind of separating out different aspects of you that you can clone and put to work on your behalf. And then I think it comes back to kind of what aspects of you are valuable that you want to put to work on your behalf. And there are increasing number of ways that you can kind of separate yourself out, I guess. Yeah. I mean, it's the British accent, Natalie. That's why. It's not really doing it for me on the 20 bucks a month thing, though. Yeah, I've got to work on that.

16:51But do you have any idea? I mean, do you think you're reading advertisements somewhere for some brand? Yes, I think I probably am. Yes, and again, I should probably look into it. But in the terms of use, as with all these AI avatar companies, so at Hour One, the thing that really kind of inspired this idea that any human can create their AI twin and kind of profit from it came from the fact that behind every avatar that we had, And we, at this point, you know, cloned thousands of people, and many of whom made their AI twins available on the platform and kind of made money passively while they slept.

17:28And governing these contracts are terms of use. So such things as it's not so much that you can select exactly where you want your likeness to show up, whether that's you in video or in voice. It's more about like where you're not going to show up, right? It's like we don't do business and anything, you know, any sexually explicit content, anything illegal, no gambling, no political content, etc. So those will be the platform sort of parameters. And you can read those terms of use and, you know, see if you feel comfortable with that. Can you talk about the example with Berlitz? Because that's also an amazing use case.

18:06Yeah. So actually, this is our first ever enterprise customer that probably broke, nearly broke us, but in the end actually propelled us forward. which is often what happens in startups. So this is back in 2020. We got approached by Berlitz, which is 150 year old language school. Japanese owned, like not what you would think is kind of going to be the most innovative company, the first to use AI avatars and generative AI or synthetic media as it was called back then. But they had an absolute need to digitally transform, scale their classroom-based learning into a platform and product that people could access all over the world for a lower price point.

18:50And they just needed to, you know, do what every other company had done at that point. And they tried filming their instructors, you know, the usual way. But tens of thousands of videos needed to be captured to have all these different clips in different languages. And they couldn't manage doing that with a human workforce. It became completely untenable. Not just the volume. Very expensive too, right? Very costly, but actually also just the management of the takes. And then it's like, oh, we need to redo that. It's like, bring them back the next day. They're wearing something slightly different.

19:25Or like, it just was unmanageable. So they looked for, it was almost like the last resort, but the only one, which was to use AI instructors, some of whom were based on their actual instructors and other ones were based on other actors. they were able to kind of fulfill their diversity kind of objectives as well as a global brand by introducing AI instructors that could teach multiple languages that, let's say, the actual instructor in real life wouldn't be able to teach all of those. So that's the kind of thing that got me really excited. Like, when can you create outsized value and deliver something that you couldn't possibly do before because of this technology and create a better customer experience as a result?

20:08We need to take a short break, but more of my live conversation with Natalie in a minute.

20:32Meet Nicole Nicholas, Capital One business customer and co-owner of Ansett Uncles, a plant-based restaurant and community space in Brooklyn, New York, that got its start from a need for unity. The inspiration, it was born from the desire to create a space that felt like home, where we can connect community culture, good food, and come together with family and friends. That's how we birthed aunts and uncles. Nicole and her husband, Mike, were fulfilling their dream of bringing people together out of their home kitchen. But they soon learned that the demand for community was greater than they knew.

21:03It became overwhelming and we were like, we need home, but not in our actual home. We realized that there was also a need in our community for something bigger in our neighborhood. So we had to find a place. Moving from a home operation into a storefront was a huge next step. But Nicole and Mike were able to take it on with the help of Capital One Business. It's not for the weak as a small business. Finding resources is super important because that's the way you'll be able to manage and scale. We would have never done that without having Capital One to be able to help us along the way. The cashback rewards are very helpful.

21:38So, you know, it just gave us that runway to be able to breathe a little bit. Then you get to focus on the cooking of the food and making the experience great. To learn more, go to CapitalOne.com slash business cards. So let's move into a different type of AI clone. So far, we've looked at visual and auditory clones. But one area that I know we're both super excited about is how do we capture our knowledge, our brain trust? And what does that look like? Wisely is one of the companies we are going to feature next, and you are currently working quite closely with them. Yes. So tell us a bit about their vision and what are they trying to achieve?

22:16Yeah, absolutely. So what they're doing is, as I mentioned earlier, is that we're sort of transitioning from this period where we're sort of like, okay, this technology exists, but now what? How do we actually put it to work on our behalf? And what are the most valuable aspects of us that we can actually scale and augment? And for many people, that's not how they look or sound. That's actually their body of work or their expertise. And so what Wisely is doing is they're building a verticalized platform for experts to be able to clone their expertise and manage their expertise in an AI platform and to then make that expertise available to people anytime.

22:58time. And so the benefit to the human expert is that their time is freed, right? So they can have their AI twin trained on their knowledge, interact with potential customers, customers, and then also use AI in the background as an assistant to help actually manage all of their content, actually have the AI brain trained on a firehose of your own data. So connect it to LinkedIn, connect it to Notion, connect it to Google Docs, and have it constantly be updated with your expertise. And so this thing is constantly becoming more and more useful. And I guess for the rest of us, it democratizes access to expertise.

23:44It's like having Andrew Huberman in all of our pockets kind of thing. Right. And actually sort of relating back to the Burlitz case study, I think a really interesting future use case is making your expertise available to more people at a lower price point and in different languages that you don't speak. So imagine different markets where they can't, you know, an individual can't usually afford to access their expertise and to now be able to not just afford it, but also have it communicated in their language, in their native language. So I think that the opportunities there are exciting. Very powerful.

24:19So I signed up to be a guinea pig with Wisely and we created an AI clone of me. Do you want to talk a little bit about what kind of data we use to train my wisely clone? Absolutely. So as we mentioned before, we're not trying to sort of clone Rana's soul here. We're trying to be very targeted. Just my brain. Yes, your brain and aspects of it, right? So you're a very multifaceted person that's done a number of different things. So for this experiment, we're like, well, why don't we take, you know, a big body of data that already exists, which is Rana's memoir. And why don't we train her AI twin on that memoir and the memoir tracks her experience as a founder and the founder of Emotion AI and the startup Affectiva.