Breaking the Sound Barrier: Spotify's AI Voice Cloning Venture

0:00Spotify, the popular music streaming giant, announced a game changing move on Monday. It's diving headlong into AI to translate podcasts into other languages. So this revolutionary feature comes as part of a new alliance with OpenAI, which is, of course, the most famous company in the AI field. And the partnership puts Spotify at the cutting edge of an industry-wide trend to essentially leverage generative AI technology for enhancing user experiences. So let's talk a little bit about translating voice and style, how this is going to work. So I think in a fairly significant step forward, Spotify has launched a pilot for its what it's calling, quote, voice translation feature aimed at providing multilingual translation that doesn't just convert words, but also mimics the original speaker's voice and style.

0:47So it's kind of interesting. I actually speak French. I know a lot of people don't know this. I lived in France for a couple of years. I'm from Canada originally, did not learn French in Canada, so I do not have a French-Canadian accent, mais je peux parler un petit peu de français, c 'est pas terrible, mais voilà. In any case, what I was thinking recently of doing is I'm like, dang, just to reach more of an audience, it'd be really cool if I could just sit there and do a French version of this podcast, and I was like, man, it's a lot of work to do that because, yeah, it would just be a lot of work.

1:20It's twice as much work. I don't know how big the audience in France is compared to English, which I have a ton of different countries that speak that. And so I ended up not doing it. However, I definitely saw the appeal. And I think this is what Spotify is doing here. They're not just going to translate my voice, for example, into other languages and just using the same words. It would literally be my same voice. This is really, really cool AI technology. I've seen this demoed a couple other places. And it seems that Spotify is jumping headfirst into it. So this really innovative capability emerges from OpenAI's release of a new voice and image capabilities for its AI chatbot, of course, ChatGPT.

2:00Users are going to be able to interact with ChatGPT in far more lifelike manners thanks to the generation of human-like audio from just text and a few seconds of sample speech. This is something that ChatGPT has recently announced they will be doing. There's some really cool use cases they have here. but it's really cool to me to see that Spotify is going to partner with them and really pull this off in a big way. So I think the underpinning technology of this feature is OpenAI's voice transcription tool which is Whisper. Of course this has been announced for a while, not a lot of people talk about it or know a lot of its capabilities but it's known for its ability to essentially transcribe English speech and translate other languages into English.

2:37Whisper really is kind of the cornerstone which Spotify's voice translation feature is being built. The pilot will initially make three podcast episodes available in Spanish, including Lex Friedman's podcast, Armchair Expert, and The Diary of His CEO with Stephen Bartlett. So both subscribed and unsubscribed users can access those episodes. French and German translations are also going to be coming in, I guess, the coming days and weeks, Spotify said, but I think first they'll kick it off in Spanish. This is so, so cool. I'll be curious to see when the rollout is, when they, you know, if they charge for this, what the broader rollout looks like.

3:12I mean, personally, I'd be willing to, you know, pay for something like this if I could see, you know, I could double or triple or even 10x my audience being able to translate it to all the languages in the world. So very, very interesting. I think as these AI-driven features become kind of ubiquitous, ethical and privacy considerations take on kind of a heightened significance. the OpenAI team has kind of explicitly noted that with these new functionalities, there are some different risks. You know, they said, quote, the potential for malicious actors to impersonate public figures or commit fraud is definitely one of them.

3:47I think given that the voice translation feature is going to mimic the speaker's voice and style, the potential for misuse is certainly a factor that is interesting. But like the other thing is, you know, if you if you know, like a voice like Joe Rogan or someone like, you know he only speaks English and so if all of a sudden you see like Joe Rogan speaking Spanish telling you to invest in some shady crypto like I would just hope that people have enough sense to say this person does not speak that language obviously obviously this is AI generated so I mean yeah they're they're saying that people can like use this for bad but of course like this technology isn't completely um new this has been around for a little bit there's different companies that have done versions of this technology, deep fakes and cloning people's voices and whatnot.

4:33So, I mean, definitely if they make a kind of a broader market that is now using it because it's, you know, more famously found on OpenAI, I understand that. But at the same time, it's not like they're inventing something that has these new risks we never had before. They are there. And I'm not saying they're not risks, but, you know, take that with a grain of salt. I think beyond some of the ethical considerations, Spotify's move also has important implications for it's standing in a really intensely competitive streaming market. So while Apple Podcasts and Amazon's music are definitely significant players, neither has announced anything as radical as translating podcasts while preserving the original speaker's voice and style, right?

5:11So for Spotify, I think the ability to really differentiate itself is such a unique and compelling way to prove to be a significant strategic advantage. Now, I will admit for everyone listening that But I think I have 70 % of my listeners are listening on Apple Podcasts. So that's where a majority of my listeners come from. If you're listening, it's probably where you're coming from. And I think it's only like 8 % to 10 % that actually listen on Spotify. However, Spotify does have some really unique features. For example, not this episode, but a lot of episodes, all interviews, I'll record video of the interview.

5:45And you can actually watch the video of the interview on Spotify, which is a really cool feature. You can't do that on Apple. and I wonder if Apple will roll this out because I know Apple recently just rolled out, I believe like the custom thumbnails for every episode. There's like this big email they sent me about how innovative it was. And, you know, Spotify has been doing that for a couple of years now. So there's just a few different things. I would hope Apple jumps on the video bandwagon. That would be really cool if they do this kind of AI translation thing. I think that would be really cool.

6:13It'll be very interesting to see if Spotify is able to take some market share by being aggressive with some of these new features that Apple is not looking at. I think, you know, with all eyes right now on Spotify's pilot phase, the real test is going to be how effectively the new feature can actually meet the expectations of a global audience. Of course, it sounds really cool, but like conceptually, but will they be able to actually pull this off is a real question. Will it help break down language barriers or, you know, is it going to sound cheesy or bad? There's all sorts of questions we have.

6:42Either way, I think the partnership between Spotify and OpenAI represent a really ambitious leap into this AI space. I think, you know, as Spotify and opening eyes collaboration kind of unfolds, it provides us not just with a new kind of way to experience content, but also a case study on how innovation can be rolled out in these larger companies. I think it's an exciting time to be a consumer, a technologist, or just an observer of kind of this really rapidly evolving digital landscape. And who knows, maybe next time you're listening to this, you'll be listening to it in Spanish or French or German or wherever else you come from.

7:15So definitely a very cool prospect that I'm very excited for with this topic.