#320 Carter Huffman: Exploring The Architecture Behind Modulate's Next-Gen Voice AI

28:33It was very, very cool. They're like, you know, I don't know if people are going to want this. Like, I don't know. You know, Mike and I were like, we really want this. We think it's cool. But, you know, nobody really like, you know, hey, we'll definitely want it. And there have been voice changers out there like apps and they never really took off or anything like that. So it's like it's not clear that people actually want this. But one of the gaming companies we talked to earlier took that tech and they were like, hey, can you make everybody sound the same so people stop harassing each other based on the sound of their voice?

29:05And we were like, that's an interesting idea. It's kind of weird. You don't want to like hide yourself in order to play a game without being harassed. Like it's kind of weird. but it's also like you could do it and it has merit and most importantly you care a lot about preventing harassment in your game you'll do anything to try to fix it even this tech isn't built for that but you still want to try to use it to solve that problem that's the problem you really care about and that's what got us turned on to let's solve harassment in gaming hearing that from our customers and how much they cared uh yeah for people who are not familiar with multiplayer games and i've never played one but i've my kids are grown now but i used to listen to them uh you know and they're talking and you know they're wearing a headset i can't hear the other side but uh uh you're not only talking to your friends right you're talking to anybody that that logs onto the game and on the server that you're connecting to is that right and so is the is the uh i didn't realize there was i mean i've heard about the swatting for example uh but i i didn't realize that harassment is uh is that big of a problem um it happens all the time sadly yeah it's a huge huge component and it's one of the largest drivers of attrition from a gaming community across all factors one of the biggest reasons people leave is they're like Like this community is too toxic.

30:40People are harassing me or people are harassing other people and I don't want to listen to it. And if that's what the community is like, you don't want to engage. Most people don't want to engage with that. And so you stop engaging. You're not participating in the community. And eventually you're just like, I'm tired of listening to this stuff. And you drop off and you go play something else. Happens all the time across all the big platforms. It's not unique to any particular game. It happens everywhere. And again, the problem was that there weren't really the tools to do anything about this. Because if you were going to say, I'm going to listen to every single thing that every person says in every online game and see if they're harassing somebody or not, that would have cost billions of dollars a month.

31:26That's such a big scale. There's so many people that game, so many people that talk. Completely impossible. But when we came along with these ensemble models and we said, what if it's a hundred or a thousand times cheaper, then would you do that? The studio said, yeah, yeah, we do want to do that. We want to solve that problem. Yeah. To jump back to your original voice getting tech. I was at a conference in Armenia, and I met a company that had a tech that I thought was incredibly cool, but it sounds like you guys were doing it quite a while ago. And it was for call centers in places like India or the Philippines where the agent, the human agent, it would normalize, not normalize isn't the word, but it would turn them into a Midwestern accent.

32:26So no matter how heavy their accent was on the other side, you wouldn't be able to identify the person as a non-native speaker. Did you guys do anything with the voice skinning? Do you still offer that? Yeah, we still have it as a product. And we actually have a couple customers and some pretty cool amounts of revenue from that product. The thing about that product that caused us to focus more on the analysis piece was that I think the use cases for voice skinning, there's twofold. There's the business use case that can range from normalizing accents, which makes people more understandable. That's a good goal.

33:18To sometimes it's important that you're not easy to identify in a phone call if you're doing like an investigation or something like that. There are other kinds of applications for this sort of thing from the business side. And then the other side is purely commercial for just like, hey, would people think it's cool to change the sound of your voice? And we were really heavily into that kind of consumer side of it of like, we want to build a technology that people think is really cool to use and makes their experience better, makes their experience more fun. And I do think those other applications, like they do provide value.

33:56They make things better for sure. But we were really like, I want to build tech that people love, love, love using. And people get really engaged by and just kind of makes their experience better. and that was really how we wanted to focus which is why we started with the voice conversion and style transfer voice skins for consumer first like you're going to buy skins you want to use because you want to sound like this person and then fighting toxicity you want to participate in the community but this small subset are ruining it because they're being super toxic and nobody can do anything about it we're going to go solve that problem we're going to make our tech make make this community better so people have more fun and people enjoy this more and people have a better experience.

34:43And that's been a real driver for us. Not just can we find all the use cases, but can we find those use cases that really make things better? So that's why we focused in that direction. Yeah. And so you do this real-time analysis and when you identify

35:05harassment or grooming is another issue I'm sure you're paying attention to. Does your system, what happens then? Does it just flag the gaming company and they take care of it with their own strategy? or do you mute the offender? I mean, what do you do with that data once you've analyzed it? It's really important that there's one central expert for any community that is the final arbiter of what do you do when something bad happens? That has to be a single centralized source. otherwise you risk these pieces getting out of sync it would be a really awful experience for me to say something in call of duty for example and then type something in call of duty and i get banned for saying it but not for typing it or vice versa it's just like why you know it doesn't make any sense for this to be to be inconsistently applied and so these studios especially in gaming And then also if we've expanded out into anti-fraud, security, like keep people from breaching passwords through calling the help desk, all of these kinds of things.

36:26In all of those places, there are multiple modalities, text, email, voice, video, all these sorts of things. But you have one group that is responsible for safety or security or community or trust that applies to all those different domains. and everything has to feed into them or you're going to get inconsistent and you're going to miss stuff. And that's going to be a bad experience for everyone. And at worst, a security breach or people are going to get hurt or that sort of thing. It all has to feed into one centralized group. So our role is cost-effectively, scalably and accurately find stuff and then make it as easy as possible to feed into that centralized group who can be the single expert to take the action.

37:13So that's always our model.

37:21And what is TalkSmod? Is that your model or is that the primary product or what is that? It's an application. So it's an application that primarily social and video gaming companies can use to help moderate their voice chat at scale. It uses our models under the hood. And in fact, it was our first product that did this sort of thing. And so it started out using kind of an early preliminary version of our models, which we're kind of retroactively calling Velma 1. It was our first ensemble of voice understanding models. And we've now kind of published the second bigger, more flexible, more dynamic version.

38:09But it kind of used the first version of the models. And we've since updated it to use the more sophisticated models. But Toxmod is sort of the purpose-built application that you hook into your voice chat stream and it helps you moderate this stuff. And under the hood, it's using the models, Velma 2, to do that analysis cost-effectively and scalably.

38:32And you said that the same strategies can be applied to other modalities. Where are you guys going? You said that you're moving beyond gaming. Can you talk about other applications for the technology? And then are you moving into other modalities? Big expansion into different applications is to provide these models in a very flexible way to do any kind of understanding you want. So you could run the models on this conversation we're having right now and feed the data that we extract into any sort of system that you want to. So people can get a whole readout, not just a transcript, but everything about like, you know, what is like, like, what is Carter really trying to say in this conversation?

39:23You know, where, where is it going? Like, like, like, how did they feel? Are they excited? You know, is Carter tired? Like, did he not drink enough coffee? You know, these kinds of things, like, like get all of that information from just this conversation. You can do that to any conversation on the planet. I want our models to be helping my grandmother stay safe from fraud. She's 91 years old. And she got scammed, I think it was nine, 10 months ago, because somebody called her and pretended to be me. And she's 91. You know, there's not that much you can do to protect against it. Our technology could help protect against that stuff.

39:59Same thing, you're deploying a voice AI. It goes off the rails, does the wrong thing. People get hurt. Happens all the time with new technology. We can be watching that and keep that from happening. Like, is it appropriate for an AI to say this thing in these contexts? All of these sorts of applications. And maybe most exciting of all, we're transitioning from being this product company that builds products like Talksmod that have one use case to being this model-first company that provides the models for anyone to use. So I think the most exciting place we're expanding is offering the models directly through an API, and then people can build them into more applications wherever it's useful to understand voice that I haven't even thought of yet.

40:43So I think that's the most exciting piece we're going. Yeah. It's interesting, the scamming industry, which is just, who knew, you know, that it would be a billion, multi-billion dollar industry. Yeah. And I have a close friend who can't believe it but he got taken in a romance scam basically lost his life savings uh who would apply that then is that uh i mean your your tech because it seems to me that But, for example, he claimed he was talking to one person and showed me a photo. And it was like, it was obviously a photo taken from the internet. And I was, you're not talking to that person. But I couldn't convince him.

41:40But presumably, I guess my question is, who would be implementing it? Is it the telephone service that you're using, or would individuals have to download it onto their phone? That's one thing. But also, how subtle could you match voices to faces? I mean, there is a certain physical structure to your face that influences what your voice sounds like. I mean, those sorts of things. How sensitive is this to those kinds of subtleties? Great questions. I'll answer in order who implements it and then how sensitive is it. So who implements it? It could be anyone in that chain. That's what we're excited about is the telephony companies themselves, the backbones, they could implement this.

42:47Before today, it's just completely impossible. I mean, there are 13 and a half billion phone calls that happen across the world every single day. You want to monitor every single one of those? Like, how on earth would you? Now it's a thousand times cheaper? Okay. Maybe you can start to at least scan for fraud, right? Something like that. It's possible, right? So the telephony companies could implement it. The OS systems for the phones, Apple, Google, could implement this at kind of an operating system layer before you even get to the app layer to scan that stuff. They do some of that for like the explicit photos, right?

43:24So like you can turn on features. If somebody sends you an explicit photo, it can double check and say like, hey, you want to receive this? Yes or no? and you could also do it on the app level so that people making the communications apps you could implement the scanning there it could go anywhere in the stack because all we need is the voice signal we extract all of the nuance we extract everything just from that voice signal you don't need a ton of other inputs coming in to figure out is this a likely fraud or not is this likely scam is somebody using a deepfake right we have the best deepfake detector for voice in the world?

44:00Is somebody using a deepfake? Like all of these things, you can do that from just the voice signal. So it's so flexible. You can implement it anywhere in the stack. So for the sensitivity question, we are completely voice specific and we are voice native. So I see our niche, our role is like there are a bunch of AI companies out there. And many of them focus on different aspects of voice. Many focus on different aspects of video or imagery. tons of people are focused on text, for example. We are really, really deep experts in voice. We know what we're doing. We know how to make it efficient. The architecture can apply to other media.

44:37And it's not impossible that we go and apply it one day. But we have built so much expertise in voice. We understand the data. We understand the medium. And I think the biggest advantage you can have as an AI model company today is to really deeply understand the data you're working with. And we deeply, deeply understand conversational voice data. So that's where we want to stay because we want to build the best, the highest quality models that exist in the entire world. And in order to do that, we need to be experts and we're experts at voice conversations. So that's where we're going to stay.

45:12Yeah. I mean, voice is maybe not as unique as a fingerprint or face structure, but it is an identifier, a unique identifier. Can you, I mean, the example of someone calling your grandmother and pretending to be you, and you said you guys can spot deepfake audio or voice. can you if you have a chronic abuser can you identify their voice and then you know that becomes a profile that that whoever is trying to prevent that person from harassing people would recognize and maybe alert somebody hey he's back she's back we recognize the voice print We have technology that can do that with over 99 % accuracy.

46:19The thing that's more difficult than just building the model to do this in a capable and accurate way are the privacy concerns. Because on the one hand, you want to catch common scammers and you want to keep them from hurting other people. But on the other hand, there are different levels to the amount of data you extract and store from an interaction. So for example, scanning for toxicity or scanning for harassment is one level of analysis and you need to have the proper safeguards in place and the proper consent in place to do that. Building a fingerprint of someone and tracking their conversations across calls is another level of privacy and another level of consent that you need to get to be able to do that.

47:05And there's a lot of variation across countries, across regions, across even states in the U.S. as to what you're legally required to do. And there's also a layer of what you ethically should do, and we pay attention to both of those things. So we have the technology capable of doing that, but there's a big jump to saying we can apply that technology in a specific situation, in a specific application, on specific kinds of data with specific kinds of consent. that's where things get very nuanced and you have to be very careful so for our end we've only deployed that technology in a small number of cases where it's unambiguously value providing to everybody involved and we get completely unambiguous consent to be able to do that that has an opt-out path.

48:03Yeah. And, you know, this drifts into sensitive areas because it depends on who's deploying it, but it seems like this would have tremendous application in the national security space. And are you working with national security agencies that are, you know, scanning voice data looking for criminals or terrorists? We're not working with those agencies. I do agree it could have a lot of impact there. And I think the thing from my perspective to make sure we're being super, super careful about is once again, our goal as a company is to both comply with the law and do what we need to do from a regulatory perspective.

48:59and also do what we need to do from an ethical perspective. And I think there's once again, as always, a ton of nuance. Nobody's against catching the bad guys who are out there hurting people. But being able to do that in a way where you're unambiguously having neutral to positive ethical impact, that can get a lot more nuanced very quickly. So it's absolutely a very applicable technology to that kind of problem statement. and I do not at all doubt that you could do a ton of good doing that right now, but it gets tricky. So we're not doing that today. Yeah. And do you see, I mean, I started out saying that I kind of felt voice was commoditized, but as you pointed out in this conversation, you're going much, much deeper besides the other advantages in latency and cost, but you're going much deeper in analysis.

50:03How large, I mean, and the fact that you're not working with national security or law enforcement, is how big is that specialized market? I mean, you know, below the commodity market. Is it just, from your point of view, as an early stage company, is it sort of bottomless? Or do you see a pretty definable market, approachable market? There are tens of billions of hours of voice conversations that happen over some digital channel, telephony, VoIP, etc. every single day. Multiple people, the participants, the platforms can get value from every single one of those conversations that they aren't getting today.

51:02Maybe it's just kind of taking notes about how you felt. Maybe it's assisting a voice AI agent. Maybe it's moderating and making the community safer. But only a tiny fraction of those conversations are analyzed at all. And a very, very tiny, currently today, sub-fraction of that fraction are analyzed with the level of depth where you can get the complete amount of value out of that conversation and do what you want to do with it. That can apply to every single conversation happening over a digital channel on Earth. So I think it's a big enough market. And it doesn't have to be a one-to-one conversations over telephony.

51:47It could be analyzing public speaking. for, I mean, can you, how can it detect, obviously not with absolute accuracy, but can it detect when someone is not being truthful? Lie detection is absolutely a component of what we're able to do from the voice signal and from the context and how folks behave in a conversation. It's a really difficult field, even the best trained humans aren't perfectly accurate. And technology will never be 100 % accurate at everything. But what I can tell you for sure is that just trying to analyze a text transcript and see if somebody is lying is a much, much, much more difficult problem than trying to understand the full context and emotion and tonality and sentiment and cadence and everything about that audio signal to try to figure out if there's a lie going on there.

52:50And that's, in fact, one of the behaviors that we look for and detect. And that's only going to get better over time. Yeah. And beyond lying, just sentiment analysis, I would guess in, well, the obvious thought is of finance industry where they're trying to pick up signals about the direction of the economy, the direction of markets or policy. do you have customers that are using it for that yeah actually we we both mike and i have a little bit of background in finance and so those were some of the first kinds of customers that came to us when we started talking about how we're doing more than just text analysis because they have some of the strongest needs the ceo of a company is going to say the exact same talking points no matter what they're they're never going to be like hey everybody company sucked this quarter, things were bad.

53:48But they're going to talk about the future and they're going to be say exactly the same words and be more or less excited compared to how they are in other quarters, right? Because each CEO is different. So you have to be really, really nuanced and you get a massive payoff if you get it right. And you're able to be a little more nuanced than everybody else. So they were actually some of the first people that came to us and said, I'm very interested in even the tiniest amount of extra signal you can give me beyond just the transcript. Yeah, it is really fascinating how data that we once thought was held no real information has all these signals.

54:30So is Modulate, where are you focused right now? Is it in safety? Is it in or is it in just making the system available through API and people can try it for whatever application they see fit? I mean, where do you see the market going or your market? Our company focus is right now very heavily concentrated on making the model available to everyone. We've never been an API company before. We've never really been a model company before. We've been an application company powered by the best voice analysis models on the planet. And our recent transformation has been, let's give those best models on the planet to everybody through this API access.

55:24So it's completely new for us. As of like last week, that's where our big focus is. But we are doing that while still making sure that we maintain and improve and iterate both on the core models themselves and on the applications that are already out there delivering value. We know monitoring voice conversations for safety, for child safety, for extremist propaganda, for potential fraud, for potential security incidents, for deepfakes. We know all of those provide value because we have customers doing that through our applications. So we're maintaining those applications, continuing to grow that.

56:00We're offering our models for the first time to everyone to use and build other applications that we couldn't even think of. And then the thing that joins all of that together is we are constantly improving our models to make sure we are shipping the best and most cost-effective voice AI models in the entire world. That's what joins our whole mission. Yeah. And I would guess, I mean, it's early days in the API journey, but you can see who is making an API call, presumably. So over time, will you be able to see, oh, there's this industry that we hadn't thought of that's starting to use the API and maybe there's a market there that we should talk to?

56:49Or, I mean, do you have that kind of visibility? One of the most important priorities for us is understanding who's using our models for new purposes that we haven't really thought of or haven't understood the scale of before. So a big part of this push for us and a big part of the learning process is how can we go out and see who are using our models, what they're using them for, and how we can optimize our models and our experience to make them more useful and better for those use cases. One of the cool areas that I experienced recently was going to a hackathon in San Francisco. And you can just provide your technology to the hackathon participants and they'll build crazy stuff that you didn't think of, like you had no idea.

57:34And then you sit down and you judge the projects and they come up to you and you get like, what, you know, 50 teams of extraordinary developers who just spent six hours building something. And you've, you know, maybe seen 20 of the 50 projects before, 10 of them you've thought of, but you weren't sure if it would work. And 20 of them are totally new. And you're like, wow, I, you know, I've spent I've spent a decade thinking about voice AI and I never thought of this. It's so cool. So between that and kind of real world customer usage, that's a top priority for us. What are people actually doing with these incredible models that we've provided?

58:10Yeah. And Modulate is separating the voice from the content initially, right? I mean, you're analyzing the voice. You're not necessarily analyzing the content, are you? Or do you have different, do you do both? You analyze, I'm just thinking about sentiment analysis, you know, where you could analyze the emotion in the voice, but you can also analyze the transcript. Do you do those both and do you do them separately? I mean, can you talk about that? Fantastic question. we do both of those things and then we fuse all the data together to figure out what's going on because you don't want when you're calling this is one of the big things where I'm like okay you know voice was kind of commoditized and then we have this new explosion right the part of that new explosion of voice AI is the realization that you don't want to get the output of one model you don't want me to come back and tell you well my voice specific emotion model that's tuned primarily towards English speakers with American or British accents, thinks that there's a 73 % chance that this person is happy and a 20 % chance that they're neutral.

59:32That's not what you want. What emotion is this person experiencing? That's the information you want. And there are so many different components to understanding that. So we look at the content, the text transcript. We also look at the tonality and the emotion extracted from that voice. We look at the context and how it evolves over time. So like someone like me, I just naturally sound excited. And so there's some signal in that. But also it means that if I sound neutral, that's a deviation from the norm. And that's important for you to know about. So we take all of those different signals and we're trying to get you the understanding of the conversation.

1:00:11We're trying to get you the answer. And also, if the tone of my voice doesn't match the text transcript of what I'm saying, if I'm like, wow, yeah, that was really fantastic. Super cool. I'm totally blown away. Right. Like that tells you a ton more about the content of what I really mean than looking at just the text or just the vocal emotion. You got to fuse it to really understand. I mean, you're focused on real-time analysis. Can these models be put on edge devices or do they have to be called using an API? Now the way we're serving these models is through an API, but deployed to a variety of different regions so that we keep the latency as low as possible.

1:01:01Something we've done in the past and explored is breaking up the processing to be partially on the edge and partially in the cloud to be even lower latency and even more efficient. That's an area I'm super, super excited about. It adds complexity because different edge devices have different amounts of capability. And also there are different applications. So we tried this early in the gaming days. but when you're playing a video game, you're using the GPU that would run the neural network to render the video game. So your computer's already very busy. You don't have a ton of time to go process a little neural net running in the background.

1:01:38But for other applications, that's not the case and you do have spare resources. So breaking the processing flow up into part on the edge and part in the cloud is a really exciting way to get lower latency while still having the same level of accuracy. It's an interesting direction. The one thing we won't compromise on is accuracy. So no matter what approach we take, we have to be able to deploy the full conversation understanding model. And if that can be broken up between edge and cloud, awesome. If that can only be done by keeping all the stuff localized together in the cloud, so be it, but we won't compromise on the accuracy.

1:02:17Yeah, and you mentioned that this is, you handle multiple languages. I think I saw 18, is that right? how many languages? It's 18 language families, which is a weird, you know, maybe I should kind of revisit that marketing language, marketing speak, because it's 18 language families that handle approximately 100 different individual languages, and then multiple different dialects within those languages. So it's kind of a hierarchy. And you get some weird kind of combinations of languages or branches that it's actually very hard to enumerate, a specific number of languages your machine learning models can handle.

1:02:58But it's pretty broad. We process underneath those 18 language families. Approximately 99.3 % of all the voice traffic that goes through our models fall into one of those specific language categories that we can understand effectively. Yeah, and this is, I would just keep on thinking of interesting things you could do. You could also identify origin, where a person is from based on their accent, couldn't you? That's absolutely possible from a voice. You can trace what kind of accent does this person have, what kind of tonality, and what are some of the demographics that you can extract from their voice.

1:03:39And you can fuse all that data together. Once again, it's always a privacy question and a legality question, but it's possible to do that with the models. Yeah. And the other question is, you can analyze all of these nuances in voice. Can you generate those nuances? Can you go the other direction? Synthesis is absolutely a part of our voice AI mission, but we've started out just focused in understanding. So right now, today, the models we're deploying and the applications we create are hyper-targeted towards understanding what's going on in a conversation. And as I alluded to at the beginning of this call, the voice skins where we started, that's sort of a transformation or synthesis application.

1:04:28But the standard voice synthesis task, doing something like text-to-speech, we haven't built our own individual models around that yet. We wanted to focus on building the absolute best voice understanding capability in the world. But that's a place where I'll go next. Yeah, yeah, because I can see huge. I mean, the voice synthesis models that I've used, it's very hard to get them to express an emotion. But anyway, if someone wants to follow up with this, I mean, I have to say this is far more interesting than I expected it to be. I'm thrilled to hear that. Fantastic. Yeah, and I'll follow you guys.

1:05:12How does someone find the API? How does someone find you guys? Is it modulate.ai? Is that your URL? Correct, yes. You can go to modulate.ai. You can find us on LinkedIn. You can find us on X. Follow our company account. Feel free to follow my account if you're so inclined. And we're actually going to be doing a ton of announcements over the next couple weeks, launching access to more facets of our models, providing out free credits to people who want to build on the back of our models. So if you're listening to this and you're interested to try them out, come follow us. Come message us and let us know what you want to build or what you want to try or just that you're interested in.

1:05:58We're going to be sending out a lot of free credits and a lot of access to our different models and capabilities over the coming weeks. So just hit us up. We'd love to let you in and see what you built. Okay, great. Carter, it's been fascinating. And I'll be in touch. This episode is brought to you by Tasty Trade. On Eye on AI, we talk a lot about how artificial intelligence is changing how people analyze information, spot patterns, and make more informed decisions. Markets are no different. The edge increasingly comes from having the right tools, the right data, and the ability to understand risk clearly.

1:06:42That's one of the reasons I like what Tasty Trade is building. With Tasty Trade, you can trade stocks, options, futures, and crypto all in one platform with low commissions, including zero commissions on stocks and crypto, so you keep more of what you earn. The platform is packed with advanced charting tools, backtesting, strategy selection, and risk analysis tools that help you think in probabilities rather than guesses. They've also introduced an AI-powered search feature that can help you discover symbols aligned with your interests, which is a smart way to explore markets more intentionally.

1:07:28For active traders, there are tools like Active Trader Mode, One-Click Trading, and Smart Order Tracking. And if you're still learning, Tasty Trade offers dozens of free educational courses, plus live support from their trade desk reps during trading hours. If you're serious about trading in a world increasingly shaped by technology, check out Tasty Trade. Visit tastytrade.com to start your trading journey today. I'm going to myself. Tasty Trade Inc. is a registered broker-dealer and member of FINRA, NFA, and SIPC.