ElevenLabs’ Mati Staniszewski: Why Voice Will Be the Fundamental Interface for Tech

0:00Late 2021, the inspiration came from Peter was about to watch a movie with his girlfriend. She didn't speak English, so they turned it up in Polish. And that kind of brought us back to something we grew up with, where every movie you watch in Polish, every foreign movie watch in Polish, has all the voices. So whether it's male voice or female voice, still narrated with one single character, but not on a narration. It's a horrible experience and it still happens today and it was like wow We think this will change this will change we think that technology and what will happen with some of the innovations Will allow us to enjoy that content in in the original delivery in the original incredible Voice and and and let's let's let's make make it happen and change it

1:04Greetings. Today we're talking with Maddie Stanishevsky from 11 Labs about how they've carved out a defensible position in AI audio, even as the big foundation model labs expand into voice as part of their push to multi -modality. We dig into the technical differences between building voice AI versus text. It turns out they're surprisingly different in terms of the data and the architectures. Maudi walks us through how 11 labs has stayed competitive by focusing narrowly on audio, including some of the specific engineering hurdles they've had to overcome, and what enterprise customers actually care about beyond the benchmarks.

1:46We also explore the future of voice as an interface, the challenges of building AI agents that can handle real conversations, and AI's potential to break language barriers. Maddie shares his thoughts on building a company in Europe, and why he thinks we might hit human -level voice interactions sooner than expected. We hope you enjoy this show. Maddie, welcome to the show. Thank you for having me. I have first question. There's a school of thought a few years ago when 11 labs really started ripping that you guys are going to be a roadkill for the foundation models. And yet here you are still doing pretty well.

2:26What happened? Like how were you able to stave off the multi -modality, you know, big foundation model labs and kind of carve out this really interesting position for yourselves? It's exciting last few years and it's definitely true. We still need to keep on our toes to be able to keep winning the fight of foundation models. But I think the usual and definitely true advice is staying focused and staying focused in our case on audio both as a company, of course, the research and the product, but we ultimately stayed focused on audio, which really helped. But the, you know, probably the biggest question on the old question is, is through the years we've been able to build some of the best research models and and out compete the big labs.

3:14And here, you know, credit to, to, to, to, to micro funder, who I think is a genius Piotr, who has been able to both do some of the first innovations in this space and then assemble a rock star team that we have today at the company that is continually pushing what's possible with Nodio. And as I, you know, when we started, there was very little research done in all of you. Most people focused on LLAMS. Some of you focused on image, you know, a lot more easy to see the results, frequently more exciting for people doing research to work in those fields. So there's a lot less focus put on to audio and the set of innovations that happened in the years prior, the diffusion model, the transform and models, were really applied to that domain in an efficient way.

3:57And we've been able to bring that in in those first years where for the first time, the text to speech models were able to understand the context of the text and deliver that audio experience in just such a better tonality and emotion. So that was the starting point that really differentiated our work to other works, which was the true research innovation. But then fast following after that first piece was building all the product around it to be able to actually use that research. It's, you know, we've seen so many times, it's like, it's not only the model that matters, it also matters how you deliver that experience to the user.

4:34Yeah. And in our case, what is narrating and creating audio books, what is voice -alvers, what is turning movies to other languages, what is adding text to speech and in the agents or building the entire conversational experience, that layer keeps helping us to win across the foundational models and hyperscalers. Okay, there's a lot here and we're gonna come back and dig in on a bunch of aspects of that, but you mentioned your co -founder Peter I Believe you guys met in high school in Poland. Is that right? Can you kind of tell us the origin story of how you two got to know each other and then maybe the origin story of how this business came together?

5:11The I'm in the I'm probably in the lockiest position ever. We met with 15 years ago in high school and the the we We started an IB class in and Poland and Warsaw and took all the same classes. So, so kind of everything and we headed off pretty quickly on some of the mathematics classes. We both love mathematics so we started about sitting together, spending a lot of time together and now kind of morphed from outside the school and time together as well. And then over the years we we kind of did it all from from living together, starting together, working together, traveling together and now 15 years and we are still best friends you know with the time is on our side.

5:56I was building a company together strengthen the relationship or there are ups and downs for sure but I think it did I think it did I think it's it's a battle tested it definitely battle tested it I know it's like you know when when the company started taking off we it's hard to know how long the horizon of this intense work will will happen. Initially it was like, OK, this is the next four weeks, we just need to push trust each other that will do well on different aspects and just continue pushing. And then there's another four weeks and another four weeks. And then we realized actually this going to be in the next 10 years.

6:36And there was just no real time for anything else. We would just do 11 laps and nothing else. And then over time, and I think this happened organically, but looking back at it's definitely helped. We now try to still stay in close touch in what's happening in our personal lives where we are in the world. And spend some time together still speaking about work, but outside of the work context. And I think this was very healthy for now. I know here for so long. And I kind of seen him evolve personally for those years. but I can still stay in close touch too. It's important to make sure that your co -founder and your executives and your team are able to bring their best self to work and not just completely ignoring everything that's happened on the personal front.

7:26Exactly. And then to your second question, part of the inspiration for 11 Lops came. So maybe the longer story. So there are two parts. First, for the years, when he was at Google, I was a parent here. We would do a Huck Weekend projects together. OK. So like trying to explore new technology for fun, and that was everything from building recommendation algorithm. So we tried to build this model where you would be presented with a few different things. And if you select one of those, the next set of things you're presented with gets closer and optimizes closer to your previous selection. Deployed it a lot of fun.

8:02Then we did the same with crypto. We tried to understand the risk in crypto and build like a risk analyzer for crypto. very hard. They're fully work, but it was good attempt in the first, one of the first crypto heights to try to provide like the analytics around it. And then we created a project in audio. So we created a project which on the last how we speak and gave you tips on how to win was this early 2021. Okay, early 2021. That was kind of the first opening. This is what's possible across audio space. This is the state of the art. These are the models that do with the erasation, understanding of speech, this is where the speech generation looks like.

8:42And then late 2021, the inspiration came from, and like the more of the aham moment, from Poland, from where you're from, where in this case Peter was about to watch a movie with his girlfriend, she didn't speak English, so they turned it up in Polish. And that kind of brought us back to something we grew up with, where every movie you watch in Polish, every foreign movie, or it's in Polish, has all the voices. So whether it's male voice or female voice, still narrated with one single character, like none of that is narration. It's like a horrible experience, and it still happens today. And it was like, wow, we think this will change.

9:21This will change. We think that the technology and what will happen with some of the innovations will allow us to enjoy that content in the original delivery, in the original, incredible voice. and let's make it happen and change it. Of course, they're an expanded since then. It's not only having realized the same problem exists across most content not being accessible in audio, just in English. How the dynamic interactions will evolve. And of course, how the audio will transmit the language barrier to. Was there any particular paper or capability that you saw that made you think, okay, now is the time for this to change?

10:02Well, attention is all you need is definitely one which which which which you know was so so crisp and and clear in terms of what's possible. Yeah, but maybe to to give like a Different angle to the to the answer I think the interesting piece was was less than the paper. There was this incredible open source repose That was like slightly later and as we started discovering like is it even possible and And there was a Tortoist TTS effectively, which is a model, an open source model that was kind of created at a time. It provided incredible results of replicating a voice and generating speech. It wasn't very stable, but it kind of have some glips into like, wow, this is incredible.

10:52And that was already as we were deeper into the company. So like maybe a first year in So in 2022 and but that was that was like another element of like Okay, this is this is this is possible some great ideas there and that of course we've spent most of our time like what other Things we can innovate through start from scratch bring the transforming diffusion into the audio space and And that that kind of heal that's just another level of of like human quality where you could actually feel like it's a human human voice. Let's yeah, let's talk a bit about how you've actually built what you've built as far as the product goes.

11:34What aspects of what works in text port directly over to audio and what's completely different, different skill set, different techniques. I'm curious how similar the two are and where some of the real differences are. The first thing is, you know, that there's kind of those three components that come come into the model that there's the compute, there's the data, there's the model architecture. And the model architectures is high -sem ideas, but it's very different. But then the data is also quite different in both in terms of what's accessible and and how you need that data to be able to train the models.

12:13And then compute the models are smaller, so you don't need as much compute, which allows us to, given a lot of innovations, need to happen and on the model side or the data side, you can still out compete foundation on models rather than just the use of data. They compute disadvantage. Exactly. But the data was, I think, the first piece, which is different, where in text, you can reliably take the text that exists and it will work. In audio, the data, first of all, there's much less of the high quality audio that actually would get you the result you need. And then the second, it frequently doesn't come with transcription or with a high accurate text of what was spoken.

12:59And that was the kind of lacking in the space where you need to spend all of time. And then there's a third component. Something that will be coming across in the current generation of models, which is not only what was said, so the transcript of Dario, but also how it was it said. What emotions did you use, who said it, what are some of the nonverbal elements that were said? That kind of almost doesn't exist, especially at the high quality. And that's where you need to spend a lot of time, that's where we spent a lot of time in the early days too, of being able to create effectively more of a speech -to -text model, and like a pipeline with additional set of manual laborers to do that work.

13:41And that's very different from text where you just need to spend a lot more cycles. And then the model level, you effectively, you know, you have this step of in the first generation that takes the speech model, you have understanding the context and bringing that to emotion. But of course, you need to kind of predict the next sound rather than the predict the next text talking. And that both depends on the on the on the on the prior but can also depend on the on the what happens after. Like an easy example is like, you know, what a what a wonderful day. And let's say it's a passage of a book.

14:22Then you kind of think, okay, this is positive emotion. I should read it in a positive way. But if you have a what a wonderful day. I said sarcastically, then suddenly it changed the entire meaning and you kind of need to adjust that in the audio delivery as well. Put a punchline and different in the different spot. So that was definitely different where that contextual understanding was a tricky thing. And then the other model thing that's the very different you have, the text to speech element, but then you have also the voice element. So the kind of the other innovation that we spend a lot of time working on is how can you create and represent voices to a higher, accurate way of what was in the original.

15:01And we found like this decoding and coding way, which was slightly different to the space. We weren't hard coding or predicting any specific features. So we weren't like trying to optimize is the voice male or is the voice female or was the age of the voice. Instead we effectively let the model decide what the characteristics should be. And then I found a way to bring that into the speech. So now of course when you have the text to speech model, it will. will take the context of the text as one input, and the second will take the voice as a second input. And based on the voice delivery, if it's more calm or dynamic, both of those will merge together and then give the end output, which was, of course, very different type of work than the text models.

15:50Amazing. What sort of people have you needed to hire to be able to build this? I imagine it's a different skill set than most AI companies. There's a... and it kind of changed over time, but I think the first difference, and this is probably less skill set difference, but more approach difference. We've started fully remote. We wanted to hire the best researchers wherever they are. We knew where they are. There's probably like 50 to 100 great people in audio, based at least on the open source work or the papers that they release or the companies that they worked in. That's a bit of admire. So you, it's at the top of the funnel, it's pretty limited because so much fewer people worked on the research.

16:36So we decided let's attract them and get them into the company wherever they are and that kind of really helped. The second thing was you know given, given we want to make it exciting for a lot of people to work, but also we think this is the best way to to run a lot of the research. We try to make the the researchers extremely close to to deployment to actually seeing the results of their work. So the cycle from being able to research something, to bringing it in front of all the people is super short. And you get that immediate feedback of how is it working. And then we have a kind of separate and research.

17:14We have research engineers that focus less on like the innovation of the entire kind of new architecture of the models, but taking existing models, improving them, changing them, deploying them at scale. And here frequently you've seen other companies called our research engineers researchers, given that the work would be, would be, ask, ask complex in those, in those companies, but, but that kind of really helped us to create a new innovation, bring that innovation extended and deploy it. And then the layer around the research that we've created is probably very different where we effectively have now a group of voice coaches, data labelers that are trained by voice coaches to understand the audio data, how to label that, how to label them motions and then they get re reviewed by the voice coaches, whether it's good or bad.

18:03because most of the traditional companies didn't re -support audio labeling in that same way. But I think the big difference is you need to be excited about some part of the audio work to really be able to create and dedicate yourself to the level we want. And we're a special at the time, small company, you would be willing to embrace that independence, that high ownership that it takes, that you are effectively, you know, working on a specific research theme yourself and, and of course there's some some interactions, some guidance from others, but a lot of the heavy lifting is individual and creating that work, which takes a different mindset.

18:51And I think we've been able to now we have like a team of 15 research and research engineers, almost, and they are incredible. What have some of the major kind of step function changes in the quality of the product or the applicability of the product been over the last few years? I remember kind of early, I think it was early 2023 -ish when you guys started to explode or maybe late 2023, I forget. And it seemed like some of it was on the heels of the Harry Potter, Bollinciaga video that went viral where as an 11 labs voice that was doing it, it seems like you've had these moments in the consumer world where something goes viral and it traces back to you, but beyond that, from a product standpoint, what have been kind of the major inflection points that have opened up new markets or spurred more developer enthusiasm?

19:39You know, it's there, what you've mentioned, it's probably one of the key things we are trying to do, and continuously even now, we see this is like one of the key things to really get the adoption out there, which is have the prosumer deployment and actually bringing it to everyone out there when you create a new technology, showing to the world that it's possible, and then kind of supplementing that of the top down, bringing it to the specific companies we work with. And the reason for this is kind of twofold. One is these groups of people are just so much more eager and quick to adopt and create our technology.

20:14And the second one frequently when you, when you, when we create a lot of the, the, the, both the product and the research work, the set of use cases that might be created. We have of course some predictions, but there's just so many more that we have what wouldn't expect like the example that we didn't have come to our mind that this is something that people might be, be creating and trying to do. And that was definitely something where we continuously even now when we create new models, we try to bring it to the entirety of the user base, learn from them and increase that. And it kind of goes in those waves where we have a new model release, we bring it broad, then kind of the prosumar adoption is there, and then the enterprise adoption follows with additional product, additional reliability that needs to happen.

21:05And then once again, we have a new step release in a new function and kind of the cycle repeats. So we tried to really embrace it. And through the history, the first one, the very first one was when we, when we had our beta model, so that you, you're right, just like when we released it publicly early 2023, okay, late 2022, we're iterating in the, in the beta with subset of users. And we had a lot of book authors in that subset. And we had this like literally a small text box in our product where you could input the text and get this speech out It was like a tweet length effectively and we had one of those book authors copy paste his entire book inside this box download it Then at the time you wouldn't you it was most of the platforms bandai content, okay, he managed to upload it They thought it's human He started getting great reviews on that platform and then came back to us with set of his friends and other book authors saying like, hey, we really need it.

22:10This is incredible and that kind of triggered his first like many virality mind with book authors. Very, very keen. Then we had another similar moment around the same period where we that there was one of the first models that could that could laugh. Okay, we released this blog post that the first AI that can love. And people picked it up like wow this is incredible this is really working we got a lot of the early users. Then of course the theme that you mentioned which was a lot of the creators and I think there's like a completely new trend that started around this time where it shifted into no um no face channels effectively you don't have the the creator in the in the frame and then you have narration all that creator across something that's happening.

22:52And that started going like like, wildfire and then the first six months of the work where of course we were providing the narration and the speech and the voices for a lot of those, a lot of those use cases and that was great to see. Then late 2023, early 2024, we released our work and moved other languages. That's one of the first moments where you could, you could really create the narration across other most famous European languages as an our dubbing product. So that was kind of back to the original vision. We created finally a way for you to have the audio and bring it to another language while still sounding the same.

23:33And that kind of triggered this other small virality moment of people creating the videos. And there was this, you know, the expected ones, which is just the traditional content, but also unexpected ones where we had someone trying to dub singing videos, which the model we didn't know will war on. And it kind of didn't work, but it gave you like a drunken singing result. So that is what Vyutai is viral too for that result, which was just fun to see. And then in 2025, at the early time now, and we are seeing I kind of, it's recurrently now, everybody is creating an agent. We started adding the voice to out of those agents.

24:17and it became both very easy to do for a lot of people to like have their entire orchestration, speech to text, their alarm responses, text to speech, to make it seamless. And we had now a few use cases which started getting a lot of traction, a lot of adoption. or most recently we worked with Epic Games to recreate the voice of Darth Vader. Yes, well done. Which players, there's just so many people using and trying to get the conversation of Darth Vader and Fortnite, which is just immense scale. And of course, most of the users are trying to have a great conversation, and use him as a companion in the game.

25:03Some people are trying to like, stretch whether he will say something that he shouldn't be able to say. So you see all those attempts as well. But luckily the product is holding up and it's actually keeping it relatively both. Performative and safe to actually keep him on the rails. And thinking about some of the dabbing, in the case of the one of the viral ones, was when we worked with Lex Friedman and he interviewed prime minister Narendra Modi. and we turned the conversation which happened between English, Prolex and Narinamodi spoke Hindi, and we turned the conversation into English, so it could actually listen to both of them speaking together.

25:40And then similarly, we turned both of them to Hindi, so we had a lot of legs speaking Hindi. And that went also extremely viral in India, where people were watching both of those versions, and then you asked people were watching the English version. So that was like a nice way of tying it back to the beginning. But I think they, especially as you think about the future, the agents and like just seeing them pop up in new ways is going to be self -requient. Like both early developers building, building everything from stripe integration and being able to process refunds through to the companion use cases all the way through to the true enterprise is kind of having probably a few viral moments ahead.

26:22Yeah, same more about what you're seeing in voice agents right now. It seems like that's quickly become a pretty popular interaction pattern. What's working? What's not working? You know, where are your customers really having success? Where are some of your customers kind of getting stuck? And then before I answer, maybe a question back to you. Do you see, do you see a lot more companies building, building agents across the companies that are coming through, Sikra? We, yeah, we absolutely do. And I think most people have this long -term vision that it's sort of a Agents style avatar powered by an 11 labs voice where it's this human -like agent that you're interacting with and I think most people Start with simpler modalities and kind of work their way up So we see a lot of text -based agents sort of proliferating throughout the enterprise stack and I imagine there are lots of consumer applications for that as well But we tend to see a lot of the enterprise stuff It's it's similar definitely what we are seeing both on the new startups being created where it's like everybody is building an agent And then enterprise like too, it's like, it's so helpful for the process internally.

27:24And like taking a step back, what we think and believe from kind of the start is, voice will fundamentally be the interface for interacting with technology. It would be one of the most, you know, it's probably the modality we've known from when the human, when genre was born, was the kind of first way the human's interacted. And it carries just so much more than text does. Like, it carries the emotions, the intonation, the imperfections, we can understand each other, we can, we can, based on, on the emotional cues responding in a very different ways. So that kind of, that's where our, our start happened where, where we think the voice will be that, that interface and build not just the text to speech element, but seeing our clients try to use the text to speech and do the whole conversation application can we provide them a solution that helps them abstract this away.

28:19And we've seen from the traditional domains, and to speak for a few, it's like in healthcare space. We've seen people try to automate some of the work they can do with nurses, as an example, a company that's a bit of a critic will automate the calls that nurses need to take to the patients to remind them of taking medicine, ask how they are feeling, capture that information back, So then the doctors can actually process that in a much more efficient way and And voice became critical where a lot of those people cannot be reached otherwise in the voice calls just the easiest thing to do then Very traditional probably that the the quickest moving one is customer support so many companies Both from the call center and the traditional customer support Trying to build the voice internally in the companies companies, whether it's companies like Deutsche Telekom, all the way through to the new companies, everybody is trying to find a way to deliver better experience and now voice is possible.

29:24And then what is probably one of the most exciting for me is education where they could you be learning through having that voice delivery in a new way. I'm a, I used to at least be a chess player or like a amateur chess player and and we work with chess .com where you can, I don't know if you are a user of chess .com. I am, but I'm a very bad chess player. Okay. Okay. So maybe, so that's a great cue. One of the things is we are trying to build effectively a narration which guides you through the game so you can learn how to play better. And there's a version of that where hopefully you will be able to work with some of the iconic chess players where you can like have the delivery from Magnus Carlson or Garrikas Parvarki, Karna Kamura to guide you through the game and get even better while you play it Which would be phenomenal and I think this will be one of the common things we'll see where like everybody will have their personal tutor for the subject that they want with voice that they relate to and they and they can get closer And that's on the enterprise side.