#281 What Is AI Audio Separation with AudioShake CEO Jessica Powell

0:00Jessica Powell:Basically what we're doing is we're separating audio and giving it to you in its components so that then you have much more control and usability over that audio.

0:13Hey everyone and welcome to SlaterPod. Today on the podcast we welcome Jessica Powell. She's the CEO of language tech platform AudioShake. So for those of you who don't know AudioShake, they specialize in AI audio separation. And I had to do a bit of digging and reading up on audio separation before this podcast to get myself up to speed. Very, very interesting company story, use case. So yeah. Hi, Jessica. And thanks so much for joining. Hi.

0:38Jessica Powell:Nice to meet you. We always ask where in the world our guests are recording is from today. So where are you based, Jessica? Where are you recording this from? I'm in San Francisco. San Francisco, the heart of it all. All right. So before we start, for those of you and for those of our listeners who don't know what AudioShake does. Give us the kind of elevator pitch, the 30-second, one-minute elevator pitch to set the scene. Oh my gosh, being in San Francisco, you have to elevator pitch all the time. So how about I not give you the elevator pitch, but some other version of it. What we fundamentally, what we do is we help make audio more usable for both human and machine workflows.

1:15Jessica Powell:That may still sound a little abstract. My parents always say it is. But what it essentially is, is we use audio separation. It's a field of AI known as source separation to separate mixed audio into its component parts. So that could mean in a broadcast that you're separating out the crowd noise, the sound of the squeaks of the shoes on the basketball court and the commentators. And meanwhile, there's a Kendrick Lamar rapping in the background. We can separate all of those elements out. And that just gives you a lot more control over what you do with that audio. So everything from workflows where you're just needing to boost or lower certain sounds, or you need to clean audio before it goes through other pipelines, think of transcription and captioning, of course, or where you simply need editing control in superhuman DAW heavy editing workflows.

2:08Jessica Powell:For example, you're separating dialogue and M &E for a production because you want to take that program into other languages. So basically what we're doing is we're separating audio and giving it to you in its components so that then you have much more control and usability over that audio. Now, there's a pretty interesting origin story behind the product. So maybe tell us a bit more about that, how you came up with that idea originally, and also just a bit about your background before co-founding AudioShake. I think you were with Google for quite some time. So just tell us a bit more about the background and then the origin story there.

2:45Jessica Powell:Sure. Yeah, I was at Google for most of my career. Started at Google in London and then ended up in Tokyo and then finally out here in the Bay Area with Google. Did a whole bunch of different things there. But my final years there, I was running communications across the company. And then when I was in Tokyo, though, did and my co-founder was there as well, did a lot of karaoke. You do a lot of karaoke when you're in Japan for business, for social, whatever it might be. And it was very, very fun. I discovered I love karaoke. But we'd always wish that we could sing along to any song, not just the songs, the re-records that are in those karaoke books that they give you.

3:28Jessica Powell:and so one of a thousand silly ideas that we had probably drunk while doing karaoke was like oh wouldn't it be cool if you could rip the vocals off of any song so you could karaoke to any song I don't want to pretend that the like heavens opened up and then all of a sudden we were like let's start a company to create no it wasn't anything like that it was just this idea that we had and then we both ended up back in the Bay Area I was at Google uh Luke my co-founder was at Platt, which is a fintech company out here. And he was already working on the AI side, leading data science there. And one day he came to me, he said, remember that, that silly idea that we had in Tokyo about being able to separate songs.

4:08Jessica Powell:Like, I think deep learning has gotten to the point where we could do that. And so just for fun, we started experimenting with that problem. And the first few passes that we had at it were, it sounded pretty terrible, But we were still really excited because very quickly we thought, well, if you could do this for a song, say remove the vocals, why couldn't you separate all the instruments? If you could separate all the instruments, why couldn't you separate all kinds of sound? And we got really, really excited because sound is working in sound. You're dealing with a ton of noise, a ton of frequency overlap.

4:40Jessica Powell:And if you can figure out, for example, how to distinguish a piano from a guitar, that means that you have a pretty sophisticated system that can then be applied elsewhere to other kinds of sounds. I want to take a step back even further before we go into the product in more detail. Like, I looked up your LinkedIn profile and you studied at Uni Fribourg in Switzerland. So I'm like, how did that happen? Tell us more about that. I feel like the good answer would be something like as a little girl, I always wanted to, I don't know, I read Heidi or I don't know, right? Like it'd be some, but it had nothing to do that.

5:16Jessica Powell:It had to do with like a terrible relationship and a boy, which was that I was dating a guy in Spain and we broke up. And I think I had like a hundred dollars and this was right after college. And I used that hundred dollars to buy a train ticket to where my aunt and uncle live. And they're in kind of in the Clia region. and um and I sort of showed up on their doorstep which I'm sure they loved and it was just like this moody heartbroken person and traipsed around the pre-alps for like two weeks just you know like wallowing in my misery and then my aunt was kind of like um you're 21 and you need a job like what are you what are you doing what are you doing here and uh and I was like I don't know like I didn't know.

6:03Jessica Powell:And, um, and she, she worked in Fribourg and so she had gone by the university there and seen that there was still a program that was open and it was in pedagogy, which, um, I'm sure for the European audience, everyone knows what pedagogy is, but it's not a word. Well, it's English. It's not a word you use that much in like in English in your day to day, at least in America. So I didn't even know what pedagogy meant. I didn't know it was just like another way of saying like an education degree. And I also didn't really speak any French. Um, Which was also a problem for doing a degree in French.

6:34Jessica Powell:But I did speak Spanish and the test to get in was a written test. And so I relied very heavily on cognates and somehow squeezed into a program where I ended up doing a do, which I guess is kind of like a. Anyway, I did some sort of degree or certificate at the university. It was a great experience. And I did learn French as part of that humiliating journey, being in a program to teach French when you didn't know French. I mean, yeah, Google's got to be present here in Zurich. Obviously, it's probably one of the top employers. So not in Fribourg, but here, close, right? One hour by train, one and a half.

7:12All right, let's get back to AudioShake and the product. So, like you mentioned the original use case that you had in mind in the karaoke, but how, like at what point did you realize that this was like bigger when you started the company, bigger than maybe the music use case? And then, yeah, tell us a bit more about like the other areas where it found traction. Well, we started again, I think from the very start, sitting out here in the Valley and

7:42Jessica Powell:working so like every single day with code, right? I think our heads pretty immediately went to what could you do with sound if you could work with it in a programming environment, right? If you had essentially structured data that would allow you to deal with audio in a reliable way. That was always sort of front and center in our minds. But we did start with music because we're both musicians. We were really interested in music. And again, we weren't really trying to start a company. It was like a fun hobby kind of thing. um but when we started first working uh we and again actually maybe this is relevant when we started we weren't we weren't thinking we were starting a company we thought it was just a fun party trick we showed it to friends that were in the music industry and we had already thought about things like remixing and stuff like that but they were like yeah but have you heard about sync licensing or you know anything the move that's happening towards like 5.1 mixes and immersive mixing that stuff was all again we played music as hobbyists but that stuff was all kind of industry type stuff.

8:42Jessica Powell:It wasn't anything that we were actively thinking about. But that got us excited that there actually was a market and an interest in stems and like separated sound. And so we went and spoke to some sync companies or sync departments at like the majors and everyone was immediately interested. And that was our entry point was music. And then, you know, from the sync, it then spread into other departments like A &R that would be working on things like creative remixing. And while we were doing that, the film people found us and they said, well, wait, if you could separate film, like if you can separate music, couldn't you separate like M &E tracks?

9:20Jessica Powell:And we were like, well, yeah, it's not entirely the same, but there's a lot of overlap there in that kind of problem. And so we very naturally got pulled into like M &E separation as well. And the very first project we did was actually Doctor Who, the kind of iconic BBC show, they wanted to bring it into, somebody purchased the rights in Germany, wanted to bring it into German, and all they had was the director's cut. So director's commentary, the original dialogue, and all of that great sci-fi soundtrack, all on the same track. And so that's where we came in, is we isolated the different components so that they could then put a German, a human German dub on top of that.

10:05That's so interesting. Can you just dwell on that a little bit? So basically the challenge was they just had the master giant file and yeah, it was just, okay, it was very, very hard. So technically for you, and I want to talk about kind of the streaming component here as well, but this obviously is not like mission critical that it's fast. It just has to be super high quality first, right? And I heard you talk on another podcast initially was really that the focus was on super high quality and then some of the other use cases where the speed was relevant came later.

10:39Jessica Powell:Yeah. I mean, quality is still our North Star on everything. When you're talking about our artistic workflows in particular, well, artistic workflows or a workflow in which separation is some component of a larger pipeline. And if you don't separate properly, you've all these downstream effects. We can talk about transcription and ASR accuracy in a moment, but that would be a really good example of that. Quality is paramount, right? That's why people use AudioShake is that we're the best at what we do. And we have our own models. And we've spent years investing in the research to build really, really great separation.

11:19Jessica Powell:But the initial workflows we were in were all post-production workflows. So again, on the music side, the track was already out there. And now you're just trying to find a new way to license it and splitting that track into the instrumental and the vocals or the different instrument stems now opens up that track for sync licensing, which is not always an open opportunity for older music, for example, kind of pre-2015. And then similar parallels, of course, on like the film and TV side, right? Then as we got better and better at building these models, we did start to get a lot of incoming around faster workflows.

11:59Jessica Powell:So examples of that would be kind of on the extreme end, you would just think of anything that's in real time where you're taking a streamed file coming in and being able to separate that. So for example, now we work with broadcasters where we are isolating dialogue. And there we can go as low as like 11 milliseconds. So we can take a live stream, like a live feed, isolate the dialogue and allow you to then, again, control the levels. And if you're ever watching a sports match, for example, and you can't hear the ref's call because the crowd is so loud. That would be a good example of where you might use it.

12:32Jessica Powell:Or there's bleed from, again, the crowd into the dialogue track. You can now split all of that. We can also remove music in real time. So music that's caught in the background. Again, if we're thinking about sports and broadcast, all of those things we have now models, special models, that allow it to work with streamed audio. So there, the performance aspect or the speed aspect is really, really important. You have to do different kinds of things to these models to get them to be like sufficiently lightweight to work in those environments. But we now today work both in post-production and live.

13:11Jessica Powell:So but they're sort of different model architectures depending on the use case. Yes. So when we did the research for this podcast, we also saw that you have a partnership with AI Media, the Australian company. So what you just described, that would be, I think you're integrated into the Lexi voice workflow. So the whole broadcasting, would that be where part of it would go via like an AI media partnership? They're unveiling a bunch of different things at NAB coming up. So I think I should leave, let them speak to that. But what I can talk about more generally that we do with them, which is already public, is we, this actually goes back to the ASR point that I've mentioned earlier.

13:50Jessica Powell:One of the big challenges with captioning media is that media, again, we think of it as really rich and beautiful and creative sound. From a sound separation perspective, it's a lot of noise, right? It's not a clean feed the way you and me talking right now on a podcast that's going to be tracked out where I have my recording and you have your recording. We're both mic'd up. We both have headsets of some kind. that's sort of like an ideal recording environment right but so much that happens in the media world is not like that so you can think of on set sound or again think of a sports stadium there's so much uncontrollable sound and frequency overlap if you try and take that raw audio and feed it into a transcription and captioning system quite a bit you're going to have a lot of errors and accuracy that you wouldn't have in a clean environment like this podcast right now.

14:46Jessica Powell:Even now, if you and I started talking over each other, that can create problems. But if we, again, think about where AI media is doing a lot of work, it's in these really rich media environments. And so if you can isolate the components of audio before it goes through transcription and captioning, you're going to have a clean input that means then that your ASR system is operating at its very, very best level. And so that's where we contribute. Got it. And I want to explore the film side as well and maybe the localization use case. So the Doctor Who you just mentioned before, but can you tell us a bit more like how, what do you do in localization?

15:25How is AudioShake relevant to these localization general use cases beyond having a rich catalog that needs to be backed up or, yeah.

15:37Jessica Powell:I mean, a lot of your listeners are bigger experts in this than I am, but I'll just speak to it from the perspective of the workflows that we get involved in. I'd say the most common ones we see are certainly dialogue and M &E separation. Certainly, that's kind of obvious why you would need to do that for older content, right? And that it's all on one track, that Doctor Who is a good example. There's no way, the way that, as I understand it, localization used to happen, is that you would basically silence the audio. So the way you would handle the fact that you didn't have multi-tracks or stems is that you would just silence all the audio and then have Florian speaking Swiss German across the track.

16:20Jessica Powell:And in some countries, as I understand it, they'd even have Florian doing all the voices of the track. um now you now are able to separate the dialogue the music and the effects from there it kind of depends on your workflow um i think if we're looking at um entirely human or primarily human workflows which is still the majority of what we see through audio shake is um uh even if ai is being used there's human in the loop but in these anyway and in these workflows you would have, you have your dialogue separated, you have your M &E, you're going to retain the M &E and then you're going to take say the Swiss German and you're then going to like stitch together the Swiss German dub and the M &E together.

17:07Jessica Powell:If you were doing an AI workflow, you would take that dialogue isolation, you would run it through transcription and voice cloning and so forth. And then again, you're going to take that outputted, cloned Swiss-German, stitch it back together with the M &E. And then I know that there's some that are like 100 % AI doing that and everything, but I think the majority of the work that we see tends to be either still 100 % human or human, but AI-assisted at different points. Very interesting. Have you ever come across like an AI interpreting interpreting use case, like live interpreting? For example, we did a research project some time ago for a client and it was a little early, maybe like one and a half, two years ago.

17:58But the problem was that when you have like real life, like remote interpreting sessions, sometimes the AI was like trying to translate the dog barking in the background and it just kind of produced gibberish. So, have you ever come across anything in that area? Has anybody ever approached you for that? I haven't seen anything on live interpretation, but from the way you're describing it, that feels very solvable.

18:21Jessica Powell:You need to be able to tell the machine what to listen to and what to ignore and then give it a clean input. Can you walk us a bit more through the products when we did the research? You have a developer platform, you have AudioShake Live, you have an enterprise platform. Just try to untangle this a bit. How would somebody come into you with a problem? How would they interact with you? Would they be on a platform? Would you, is it kind of a, is it a SaaS? Would they log in and use it? Just give us a bit more insights there. Sure. We have an on-demand platform that is largely used by enterprises and companies working across the media and tech space.

19:01Jessica Powell:Tends to be used more on the, I guess, media side. It's like a drag and drop. You're pulling in your asset. You're telling us what you want to do with it. And then we're separating it for you. Again, we have models that can separate instruments, sure, but also can isolate dialogue, music and effects, can remove music from an environment, but keep all the natural sound. So again, that's like a kind of a, how do you take older catalog where the licensing has expired and then make it so that you don't have the copyrighted music in it and you can put in new freshly licensed music. And then we have models that can separate multiple speakers.

19:38Jessica Powell:We also have models that are designed to boost the intelligibility of a speaker's voice. And so that's particularly useful in really noisy recordings. Got one through last night. Normally, this is all happening. I don't see the files and everything. But we had a special outreach from a customer that does a lot of work in unscripted and documentary. and they had some very sensitive recordings tied to different kinds of audio that was recorded via an Alexa or a Siri type device and wanted to see, could we separate all the different voices so that you could extract some very sensitive kind of material from that.

20:24Jessica Powell:So that kind of thing can all just be uploaded to the site. We separate it. We don't retain that audio. We always can do that on the API. And we also have, again, these real-time streaming technologies that people can implement as well. When you say you separate it, like there is a human in the loop at that point still somewhere? Or like that, and you just delivered a, no, not at all. We don't have any humans working at AudioShake, including me. Sorry. No, no, I'm human. The employees are human. And there's no human in the loop in the audio, like audio shake process. That's entirely deep learning.

21:07Jessica Powell:All of our human resources are going to building those models. When I was referring to human in the loop earlier, I was just talking about a lot of our media partners. You know, I think there was the hope when we first started, there was a lot of talk in the industry and customers that we would have about being able to fully automate pipe, like say localization pipelines. And I think that some of the companies we worked with that were in that space started off with 100 % automation and found that you couldn't get the quality that you needed because there were just too many points of failure in orchestrating all of these different AI systems together.

21:48Jessica Powell:And a lot of them moved into kind of human in the loop type scenarios where you then have, you might have audio shake at the start where it's isolating it. Then it's going to go through transcription. Your transcription is going to be boosted. But ASR still, even in clean, clean environments like you and me talking right now, ASR can still fail. Transcription can still fail. So having a human check on that, having a human check before, say, like an AI voice gets involved and after the voice is involved. So I think today what we primarily see are workflows that are either 100 % human, meaning the separation is happening and they're keeping the M &E file and then they're having a human Swiss German dub on top of it, or the human in the loop AI workflows.

22:33Jessica Powell:But yeah, the audio shape component of it is an automated component. When you're doing the separation of the voices, does the language matter at all or it really doesn't matter at all? Like if it's like a bunch of English speakers and you're separating or there are a bunch of, I don't know, Quechua speakers, would that matter? It can matter. Yeah. Because so it really matters when you're doing voice cloning, right? Which we don't do. But if you're doing voice AI, yes, like you would want to be able to train on Quechua as well as like Quebecois as well as Sutsu Deutz, right? Like you'd want those accents and those inflections that might matter a little bit less when you're talking about frequency overlap.

23:12Jessica Powell:But it does still matter. There are still cultural norms in how we speak and frequency differences in how we speak. And so data diversity also matters when you're training models and source separation. So are there like any kind of technical metrics and like eval benchmarks that you're looking at, that you're tracking and like, you know, benchmarking your progress? Because I think you posted somewhere that you came top in an evaluation of like models that Meta ran. But like what type of eval benchmarks are those? Yeah. Well, so generally, Meta created its own benchmarks. As they should. Well, so that's kind of a separate issue.

23:59Jessica Powell:But in terms of benchmarks that are used widely in the field, yes, we're state of the art. So we benchmark in source separation. It's something called the SDR score. That's what you use. We're state of the art on SDR.

24:14Jessica Powell:It's always helpful to have benchmarks because then you have some common thing that everyone's using and there's some good to benchmarks because there's something that a benchmark is evaluating that probably is important to your system. At the same time, if you're talking about sound separation, and I would imagine the same would be if you were thinking about other kinds of audio components. If I think about, again, voice AI is not our field, but if I was thinking about voice AI, I imagine there are similar constraints. When you think about sort of separation, of course it matters how another machine is telling you, you know, how did you score on this task?

24:53Jessica Powell:The problem is, is that there's also perception and what a human perceives as high quality may be different from what a machine perceives as high quality, no matter how robust you try and make those benchmarks. So it's totally possible to achieve great SDR scores, but not have output that sounds as good. So we've had models that scored higher on SDR, but that we don't deploy because we found another, like a better sounding model that doesn't score as high SDR, but that we know in all of our listening tests has much better perceptual quality. So we use both of those. Like, I think it's just sort of kind of, I think, part of running a business.