AssemblyAI CEO Dylan Fox on 120 Million Voice Conversations a Week

0:00The amount of weekly conversations that Assembly handles through our APIs every week is up over 800 % over the last three years. On a peak week, there'll be something like over 120 million voice conversations going through our platform, over 2 million hours of voice. It's a little over 4x the amount of daily volume that's going to YouTube. So they're getting more accurate, they're getting faster, more controllable, more capabilities. The newest voice models that we just launched, you can give them context about the environment they're operating in. If you drive through a McDonald's and you order with your voice, that voice AI system has no clue that it's taking a McDonald's order.

0:33But with our models, you can give them context on, hey, you're taking a McDonald's order. You want to focus on the person that's ordering. Ignore the kids that are screaming in the background. And so we're seeing this huge inflection now of enterprises, small businesses that can build with API infrastructure now. Our TAM has just increased by 100x because we're not just selling to engineering teams within product companies. It's now like anyone.

1:06Dylan Fox:Dylan Fox, welcome to Sorcery. Yeah, thanks for having me here. So we're going to have a very fun conversation on voice. We were just gabbing off camera about how much I love voice and how I've used it to start this podcast and do all these interviews. specifically like use the transcripts that I capture from recordings into different kinds of formats. I started it for investing and so I would take that I would repackage it I would make like open source expert calls and open source memos and all this kind of stuff just from capturing that data. Now you have built the infrastructure for how that even works so I want to get into that I want to get into everything from the business model how you built out the products You've been doing this for a very long time and there's a lot of new entrants to the category.

1:51Dylan Fox:So it'd be great to learn what is BS and what's not, as well as like areas that you're most excited about. But before we start, I guess it'd be great to kind of capture and illustrate just how big Assembly AI has gotten, how much data you train on. Yeah. So the demand for voice applications and the applications that companies are building on our infrastructure is over the last two years in particular really started to explode. One stat I was just looking at before I came over here was like the amount of weekly conversations that Assembly handles through our APIs every week is up over 800 percent over the last three years.

2:32Oh, my God. And so now, you know, on a given week, on a peak week, there'll be something like over 120 million conversations, voice conversations going through our platform, over 2 million hours of voice, which as of December of this past year was 4x or a little over 4x the amount of daily volume that's going to YouTube.

2:55Dylan Fox:Which when I found that was like, wait, I need to double check this because, you know, that can't be right. But yeah, the volume is huge. There's almost 100 million API calls a day coming against our API, about a million developers, a little over a million developers on the platform now. And 40 % of those developers signed up to the API last year. And so while we started this a long time ago, it's really over the last two, three years where voice and it's accelerating and inflecting voice is becoming just a core part of software and increasingly hardware too. And so we're seeing that through our platform because we're the infrastructure really under all of it.

3:34Dylan Fox:So if you look back over time, you guys went through YC. Yep. Did you think that this would be the reason why voice would take off and people talking to their phones and talking to their computers and that kind of thing? Like, what did you think? Yeah, I mean, I did, which is why I've spent so much time on this because we were actually the very first AI batch in YC when we went through YC. And so it was Daniel Gross, who, if you know Daniel now at Meta, he started the AI batch at YC back when we went through in 2017. And it was me and like five other companies. And we got, you know,$100 ,000 in GPU credits.

4:13Dylan Fox:That was like our perk, which now seems like cute, right? Right. It's like, wow, you know, that's nothing. But back then it was like, wow, you know, $100 ,000 of like NVIDIA K80 usage. Like, this is amazing. But the, you know, the AI ecosystem back then was just like in its infancy. Like I was going to the very first like TensorFlow meetups, just to put it into perspective. Like it was, no one was really using AI in production yet. But for me personally, I had gotten an Amazon Echo and was like really into voice interfaces and talking to this hardware that worked well. Because my experience with voice prior had been, everything was terrible.

4:51And so then I went looking to find like APIs that I could just in my spare time back then like build with and I couldn't find anything. And all the technology kind of sucked at the time. But I felt like, okay, over the next 10 years, this is going to get so much better. And when it does, if it's easily accessible to developers like a Twilio or a Stripe, you're going to have so many creative people build really cool things. And so now, you know, that's what gets us excited when we see, you know, our customers like Granola, right? Granola building these like amazing products that are helping companies and professionals like run their businesses.

5:29You know, we have customers like Tollands that are building, we were talking about it before, like AI companions that hundreds of thousands of people are talking to for like hours every day in the healthcare space. Athelis, Commure, it's a big health tech company building AI scribes for doctors and And it's really cool to see what people are building. So I think that to go back to your question, like I always viewed this journey as similar to self-driving cars where it wasn't a question of like, oh, is the product market fit gonna come? It was like, no, the technology just sucks. And as it gets better and better, you're gonna continuously cross these thresholds where you open up more tranches of the market.

6:07And that's still happening. And there's new macro trends that are happening that are accelerating that further beyond just the core technology getting better. I can talk about those, but that was very much the thesis behind the company. For me, it was like, this technology is going to get better. It's going to enable so many new applications and voice to be something you can really build with and a core modality in our lives. And it just felt like something really cool to work on.

6:39Dylan Fox:So what are those core macro trends? So there's really three things that we see that are causing, you know, everything, all our metrics like inflect. The first, which is what we control, is voice models are getting just much better. So they're getting more accurate. They're getting faster, more controllable, more capabilities. You know, the newest voice models that we just launched, for example, you can give them context about the environment they're operating in. And to give you an example of that is like, you know, today, if you drive through a McDonald's and you order with your voice and there's a voice AI system there, that voice AI system has no clue that it's taking a McDonald's order, right?

7:18It's like, it could be, it has no context. It's like fresh. But with our models, you can give them context on, hey, you're taking a McDonald's order. You want to focus on the person that's ordering. Ignore the kids that are screaming in the background. And those types of capabilities make them much more accurate. And then they're also faster and they're lower cost, so they can be more widely deployed. So you have these voice models that are just getting better across these key dimensions. That's like what we control. But the other two macro trends, the one is number two in this list of three is like you have this ecosystem of AI infrastructure getting built out.

7:55So you have these reasoning models, you have vector databases, you have other AI models across other modalities. And so now when you're a company wanting to build something like drive-through voice ordering or AI note-taking or healthcare, you don't just have the voice models. You have the rest of the infrastructure you need to go build the thing you're trying to build. And that has been accelerating over the last couple of years. And then the third, which is new, is coding agents. So we're now seeing companies sign up, spending a lot of money on API that like not even two years ago, like 18 months ago, like I would never think of them as our target customers because we're API platform.

8:42So you would think like, oh, you're selling to engineering teams, product teams. But now everyone's an engineer because they have coding agents. And so we saw this small business lawn care chain sign up. And we're like, what do they do with the API? And a lot of these small businesses are automating parts of their back office using Lovable or Cloud Code or Cursor or whatever, Replit. And those coding agents are building our infrastructure. And so we're seeing this huge inflection now of like you have enterprises, you have small businesses that can build with API infrastructure now. So I think about this as like our TAM has just increased by 100x because we're not just selling to engineering teams within product companies.

9:31It's now like anyone. anyone. And we're seeing some global enterprises now come in and build their own software for a bunch of different use cases. And they're doing that on top of our infrastructure. And when I go talk to them, like, why are they doing this? It's like, well, they have two people working on this with a bunch of coding agents. And so those three things, it's the voice models are getting better, like the broader ecosystem of AI technology. And then the third is anyone can build this stuff now because of coding agents are really causing voice to be way more widely deployed and across just like so many different use cases.

10:11Dylan Fox:So for people that are new to this space, it would be helpful to explain your differentiation between the likes of like 11 labs. Obviously, Sierra is in a different bucket too. But like, how do you fit in within that world? Yeah, so we are 100 % focused on voice AI infrastructure. So we don't do anything at the application layer. Where we focus is the models. So we create models that are amazing, voice models are amazing for all those use cases I spoke about. So healthcare, we have medical-focused models, drive-through voice ordering, contact center, and note-taking. We then build out the inference around those models.

10:54So to handle 4x the amount of YouTube volume on a day, 120 million conversations a week, and growing over 100 % year over year, there's a lot of infrastructure we have to build out to make sure everything scales, is available, is super fast, is low cost for customers. So we build out a ton of infrastructure on inference around our models. And then we also do a lot around the orchestration layer. So if you want to build a voice agent, if you want to understand speakers, if you want to translate data, we have a ton of software at the orchestration layer that companies can leverage. And then above that, we have this amazing developer experience, agentic coding experience, so agents can easily build with our stuff.

11:42And when we say infrastructure, I think a lot of times people think like, oh, you're just creating the model weights, right? Like you're just creating like models. That's a part of it. But those other parts are equally important to us, the inference, the orchestration, the developer and agent experience. And so we're 100 % focused on that. And so and across like all the different use cases that we focus on and making sure everything's like tailored for those. And so the reason most AI note takers are using Assembly as the voice infrastructure is because our models are the most scalable. If you have tons of people, millions of users that you're doing live note taking for, we have the most scalable models in the market that are super low cost, high accuracy, very fast, can burst to huge amounts of traffic.

12:32You don't have to stand up servers as a customer. So all of that is like the area that we're focused on. And we're enabling companies to build on top of that.

12:41Dylan Fox:How did you get it to become so efficient? You know, years of engineering work. People like Ben at our company. We have an amazing team of engineers, of researchers that just operate so closely to customers that they really understand like how this stuff is being deployed. I think that's the biggest difference between us and a lab at a bigger company. So we have a lot of researchers, research engineers that will come to assembly from a bigger company. And they're so far removed from the customer. So they just create their model, they benchmark it, and then it's like, okay, I'm done. But you have to know, all right, who are the customers?

13:21How are they using this? What do they care about? What errors break their application and what errors don't matter? And then how do you optimize both your models and your infrastructure for that? So we've just spent years and years building out the infrastructure to make this stuff work. So infrastructure that's like cross-region and cross-cloud and stuff that just will really scale. But it's probably like half our work is spent just making our infrastructure more and more scalable.

13:50Dylan Fox:This episode is brought to you by Brex, my favorite. You become what you spend on. And I refuse to spend my time on work that shouldn't exist. Expense reports, receipt chasing, and manual closes. The companies building what's next from Vercel, OpenAI, Anthropic, Granola, and Deepgram all made the same call. They all run on Brex. Brex is the intelligent finance platform that combines cards, expenses, and banking into a single stack with agentic finance built in. AI agents that handle expenses automatically, enforce policy before spend happens, and close your books in minutes. That's why Sorcery runs on Brex, so I can spend time on building and not busy work.

14:35Dylan Fox:It's time to get Brex AF. Learn more at brex.com slash sorcery. That's B-R-E-X dot com slash S-O-U-R-C-E-R-Y. Bye. I feel like between like, well, the SaaS apocalypse is one thing, but even leading up to this point with AI helping companies restructure, whether it's on the kind of internal side with developers or on the outward facing side in marketing, that kind of thing. I'd be really interested to hear how you re-architectured the company internally to do that. Yeah, it's a great question. I mean, we're, we're AI native company. Like everything we do is, is AI native. And so what I mean by that is like, I have even me personally, like I have my own agent that like I built, right.

15:28I call it like my Dylan claw on my computer and it has my meeting notes and it can make slides for me and it can do it, do everything. And so I just onboarded someone new to the company and I was going through my slide deck and it's like, Oh, I was talking through a part of the deck that was like a bit clunky. And then because my agent has access to the slides and can make new slides and was, you know, has the notes from that meeting powered by assembly. I was like, Hey, go create a new version of the deck. That is a bit easier to walk through and like, look at the transcript from that most recent onboarding to make some changes.

16:04And I want to like merge these two decks together. And then like 20 minutes later, it's done and it's perfect. And I'll have to edit it. We have an agent that has access to all the company information. So our metrics and the whole company uses it. It can submit PRs in GitHub across our products. And so I think like, you know, we've always stayed very small. You know, we're only, I think we'll be 80 people at the company pretty soon. So we have huge scale for the size of company that we're at. and we try to be intentional about that. But, you know, I think for us, like, we've just really leaned in to try to, you know, use AI, like, across everything.

16:43Like, you know, the most recent example is we had Cloud rebuild our whole website off of Webflow and just deploy it on Vercel. And now anyone, you know, people in Slack can just make changes to the website. So you, like, get off a customer call, you're like, oh, this part needs updating. And it's like, boom, you can just change it. So it's things like that that have, I think, helped us leverage AI across the company.

17:05Dylan Fox:Sovereign AI has been a huge topic. And you mentioned some of your customers that are in categories that are a bit more critical and sensitive for data. But we are seeing this across the board because you can get copied by the labs. Yeah. So you can deploy our platform on-premise. You can run it all self-hosted, and then it's totally private. And we do see an uptick in that for sure. I think that it's this mix like in voice is so early where you know we we the models and the capabilities are getting better so quickly that it's still hard to like to branch out you know what I mean by that like like we've seen some customers maybe six months ago or a year ago they'll take an open source model and fine-tune it or something and then you know they're now like okay this is like shit this is like really behind and it's like nightmare to maintain and I don't want to work on this.

18:01And so, you know, they'll come to assembly. I think voice is much earlier where we haven't seen as much of that as you're, you know, you're probably talking about in like, yeah, if you're a bank and everything's going into cloud or something, it's like, you know, wait, all my proprietary kind of knowledge is going out. But we are working on ways for companies to create like their own custom models, deploy them on our inference or run them, you know, on their own cloud but it's it's earlier there for voice for sure i was thinking about it because when we were

18:27Dylan Fox:at the raise summit this was a hot take that max cook who's a sector head at code 2 said he thinks the keyboard and mouse is over we've been through so many different form factors of computers yeah human like machine interface that's like no we gotta close the books on the computer keyboard and mouse it's funny when you go into offices and everyone has like the gooseneck microphones and they're like whispering over. I think you're going to have, so here's like what I genuinely see, like voice is a, I would think of the word like modality or interface, but I don't even think those capture it. Like voice is a reliable form of, let's just call it data capture now.

19:11Like for the first time probably ever in the past year, it's like a reliable form of data capture. And so what does that mean? That means like you can actually talk to your computer now. You can actually have it listen in. You have your computer listen in on your meetings and take notes for you. Doctors can use it during your patient encounter to update the charts. We have a startup. It's called Ciro AI. I don't know if you've heard of them. They're building on the platform. They're creating an AI for field service tech. So if you're a plumber, HVAC technician, They have an app you can run and it will listen in on your service visit and then give you feedback after for like sales coaching.

19:49And it's helping those field tech salespeople be so much more successful because like they're actually getting feedback and they never were before. So they're earning like 20 % more take home pay or something. You know, we're seeing it with like toys and with hardware where you can like talk to toys now, which is cool for kids. You know, me as a parent, seeing that stuff is really cool. So voice is this reliable form of data capture. And I think when people talk about voice AI, a lot of times they're talking about like AI voices, right? And that's a part of it. That's a big part of it. Like you have voices that sound great and they're amazing and they're so much better.

20:25But the other part is like, can the computer or the machine understand you? And that voice capture part, is that reliable? Is that solid? Does that work at scale? But that is the case now. and so I actually think you're going to see you know like touchscreens didn't replace keyboards it's like you have both you have the you know on a coffee machine like there's the rotary dial like and there's the button the touchscreen menu and I think voice is going to be another dimension that's probably the way the way I think about it voice is going to be a new dimension and but it's going to be an additional dimension and so for me like I like to type but I also like to talk to my computer I like to do both things and I think for you know where I get excited is like consumer electronics that you're going to be able to talk to.

21:11So like your TV, like, it's still crazy that like most TVs, you can't just be like, Hey, put on like open up Apple TV and put on, you know, this, this episode of whatever.

21:20Dylan Fox:They are always listening. I don't know if you saw the succession. No, I didn't see that. No, always listening. Yeah. But like, it's crazy that doesn't work yet. And so I don't think that means that computers are just going to to be like a screen and you just talk to it because actually we'd get tired of talking but but I think it's going to be a dimension that's added to everything and so that over the next couple years will happen and I think that's really cool because I think that will allow for us to be freed of like our addiction to screens right because your input to most computers now is like the this like touchscreen it's like smaller and smaller touchscreen that like just captures you.

22:04But with voice, that enables more passive hardware and more passive computing experiences where it's like you can talk to things and you can be walking around and you can be free from being like a prisoner to the device. So that's what I think is really cool. And that's for sure coming. We have some like big consumer electronics companies like actively engaged with us, like implementing our technology right now for things like that, which is like really cool.

22:33Dylan Fox:What about humanoid robotics? Those two. You know, I think that, you know, every time I think about, like, the Neo or the figure, I think about, like, my, I have a four-year-old and a three-year-old. And I think about, like, one of those in the house with them. Because we have a Matic robot. And they, like, love to mess with the Matic robot. They broke one. No. It was, like, because they always go run over it and, like, turn it on. Because they realize, like, you can press the button and turn it on. and then they like sit on it and they mess with it and try to ride it around. And yeah, I don't know what the weight is of a Neo or something.

23:07But yeah, I think you'll have humanoid robots. Obviously, the way you're going to want to interact with a humanoid robot is by talking to it. But this is where what we do is so important is because I'll tell you one of the main problems the humanoid robots face today, because a lot of them are using our APIs. If you have three people standing next to the robot, it doesn't know who to listen to. and it has a hard time disambiguating who's saying what. And so it all just gets kind of jumbled and merged. This is a big problem.

23:36Dylan Fox:Because it's using listening data, not video data? Yeah, exactly. And even if it's video, like if you're not facing it, maybe you can't see, right? So like it does have the cameras, but you really need to be able to, like for you, like for me, like if I close my eyes, I can still tell who's saying what, right? But for voice agents, even if they're over the phone, this is a big problem with them today. you know i i last week or last month like called a voice agent and i called a restaurant it was a voice agent that answered right so it's like ai that i'm talking to and i know it's an ai even though it tries to trick me because like i build this stuff but my son's in the back and he's talking and it's like tripping up the voice agent because it doesn't know that that's a background speaker it just hears speech and it's like i need to respond to this and those problems are still not solved yet these voice models and that's going back to like the sovereign ai stuff i think right now we're still in the phase of like, I need this stuff to work.

24:29And I just need to like, cause there's, there's such an opportunity if you can be the first to get it to work. And we're still in that, like, get this stuff to work phase. Probably that will be the next, you know, 18 months. And, uh, and yeah, I think that like, yeah, across humanoid robots, consumer electronics, like you're going to see voice as this dimension that you just like expect. And what the example I think about is, you know, when you see pictures of like kids, kids trying to swipe on TVs or something because they just expect everything is a touch screen. I think it's going to be similar where in five years, kids are just going to talk at things expecting them to be able to understand them.