Building Jarvis: A Voice AI Startup Making Computer Interaction Invisible |Tanay Kothari (Wispr Flow)

0:00We connected it to Chad, GPT, Siri, Alexa, and they all sucked. We called five contestants and they had to try to beat Whisperer. So they were trying different things to make it break. And the award was to get a Porsche 911 G3 RS. When you double your team size, the output does not double.

0:27Anne Dwane:okay everyone welcome um and we are pleased today we're going to talk with tanae kathari of whisper flow but he's running late but we have a secret insider to give us the background And that is his brother, Jacell. And so let's give him a hand. Portfolio founder, working on Gallium, which you should all get to know if you're marketing your company. But let's go back to your childhood. And what was it like? Was Tanay building stuff as a teenager? Yeah. So Tane and I, by the way, have a seven-year age gap. And so when I was five years old and he was 12 years old, he used to secretly stay up every night because my parents only gave him one hour of screen time every day.

1:26And he used to sleep alternative nights. And he just used to build apps and go throughout the night. He was only 12 years old. And I think by the time he was out of high school, he published like 50 total apps on the App Store.

1:41Anne Dwane:Okay, and one of them famously got quite popular and was banned by Google. Can you talk about that app? Yes, so that was like an early version of Sajam or if you've used any of those YouTube 2 MP3 apps. So he built one of the first ones. It got to 2.5 million users with obviously no marketing. He was, I think, 15 when he did that. And then Google sent him a cease and desist. Okay, and what did your parents think of all this technology and building and things like that? So for a long time, they didn't know. Because, you know, he kept it secret. And as much as he loved doing all these things, he was living like 20-hour days here, like twice the amount of time as any other person or any other kid.

2:34So he also loved, you know, playing Pokemon and all of the other things that kids love. So he was also a normal child. But, you know, they didn't know it. But, you know, I'm very grateful for our parents because they were super open-minded. You know, they gave us resources. I got my first laptop when I was nine years old. So, you know, it was a lot easier to get started and, you know, find your way into the tech world.

3:00Anne Dwane:We're going to come back to you later. Do you recall, was there a movie that Tane watched that is kind of formative? Oh, yes, of course. So I think the biggest inspiration for him was he watched Iron Man, and he just fell in love with everything he saw. And to him, he was a child. It wasn't unachievable. He almost believed that that was a real thing that could exist. So his lifelong dream kind of became to build Jarvis. and actually before Siri existed, before Alexa existed, he built the first version of a voice assistant. It was called Aria and it worked flawlessly and that's kind of what he is now doing at WhisperFlow.

3:46Anne Dwane:Yeah, it's an amazing arc. Okay, so you were seven years younger. When did you get an iPhone or a phone? You got a laptop at nine. My first phone was an iPhone 3, which I'll have to calculate, but very early. A little too early, I would say. Okay, that's what I was going to say, because many parents in the room, and they're thinking about this now. You turned out okay. I hope so. Okay, and what was it like? Because sometimes the older sibling being into something like technology can cause the other sibling to say, I'm not into that but you were you were more inspired to go into technology or or how did that work for you?

4:33A hundred percent. I mean everything I know today he has taught me almost forcefully but you know it's it's like just seeing him what technology enables you to do is just to think of something and and then make it feasible and you have it in your hand and and that's just something that I feel like is so influential to everyone. It's like you see someone else do it, and it's like, I want to do this as well.

4:59Anne Dwane:Okay, and without stealing Taneh's thunder about his whole journey of Whisper Flow, you were an important part of it because he called on you when you needed some help. Can you tell me, like, what was that time like in the company and the journey for you? Of course. So at the time, you know, Whisper, I'm sure Taneh will explain more on this, was a hardware company, and they had just, you know, built their very first prototype and they wanted to actually do some form of action with it. And so I was a freshman at Stanford and so I joined Whisper as their first software engineer. And then my kind of intern project was kind of to take up what Tanay had built in middle school, which was Aria, the assistant, but, you know, pair that with the hardware that they have.

5:45And, you know, one of the first things I built over there was voice dictation as part of the city of Whisper, which today is Whisper Flow.

5:54Anne Dwane:Amazing. Amazing. And great. Okay, so Tanea is here. So, Jaisal, thank you so much. Yeah, it's great seeing you. So we heard the inside upbringing story. That's very inspiring. Awesome.

6:15Anne Dwane:Great. Okay. Thank you for coming. Thanks for having me. And so sorry for being a few minutes late, guys. I account for one accident and today we had two. Oh, no. Okay, good. But you were not involved. No. Okay, good. So we heard a little bit of the origin story of Tanay as entrepreneur. Let's talk a little bit about the origin story of Whisperflow. And it was a journey. It's quite a long one. Yeah. So tell us a little bit about the journey and then the metrics today. How many of you here use the product? Okay, that's almost everyone. So we started Whisper Flow or Whisper the company in early 2021 was when my co-founder Sahaj gave me a call and said, then I'm thinking about leaving my job and I want to start a company.

7:03And I was working at another place at that point. And this was my college roommate, one of my closest friends. And I was like, okay, this is not an opportunity I want to give up. And so he and I started working on what we thought were some of the most important problems that nobody else was working on. And the biggest one that came up was we saw LLMs start to come out. And in a world where computers could understand natural language, you want to be able to talk to them just as naturally as talking to a friend. And the voice layer just did not exist. So for us, you're like, OK, you know, some other companies are going to build the voice layer.

7:42And so what we need to build is the hardware device that lets you use voice everywhere. And so over the next three years, what we built was this wearable device that looks like a little Bluetooth earpiece that you could put on. And it could go from your thoughts to text. It was a team of 40 people, mostly PhDs in neuroscience, electrical, mechanical, that built the world's first silent speech device. and 2024 June was the time when it first started to work we connected it to chat GPT Siri Alexa and they all sucked and so then we realized we needed to build something that goes from your kind of rambly thoughts to text uh which is what Jess was just talking about and he was the one software engineer intern on the team who who was hacking this together what we saw then was actually something really interesting, which was when we were giving people kind of the software and hardware product together, they just love the software experience and they would just use it without the hardware, which is really funny.

8:49And that was basically the signal to us to switch over.

8:53Anne Dwane:Amazing. And you had a great quote, which is you said, sometimes the best product decisions come from humility, not conviction. So can you talk us through that decision to basically cut off the hardware so a big thing that you know like most of first founders in the room and one of the things that lets you do the things that you do is a level of delusion that you have about like nope this is a thing that i want to build and you just blaze through everything and the world is going to say no no but you keep going through it but the thing i've learned is you need to pair that up with a deep level of truth seeking as well and then taking the signals for what they were and that is what often lets you make those bigger decisions which is like you know you can have the grandest vision in the world but there are steps to get that and there is a reality that you need to deal with and so this big grand vision that we had the answer was not to go there directly, but actually realized that, hey, for us, the realization was habits change one step at a time.

10:05And for us, we needed to get people on that journey with us, the users, society overall. And the first step of that was just wise dictation. And that got us to make that jump.

10:20Anne Dwane:That's right. And because it is, entrepreneurs have to deal with a lot of timing issues, right? Because you foresee a world where we're not locked behind keyboards, which are the weirdest interface in a sense, right? And screens, which are also kind of suboptimal. And how do you think about, you just think that's the inevitable future, but this is the next step or talk us through that. The goal for me is one day you go outside and you see nobody stuck on their phones doing this all day long. People are looking up. They might have something in their ear that they're talking to but for most part technology phase into the background so you as a person can be a lot more present so okay that requires a lot of habit changes for people to do and what would it look like if we stage it one by one well so the first thing you need to build is voice input that just works it's reliable and people trust it blindly if you don't have that you can't build anything else on top of it which is where siri elects all of these products failed okay so we have this thing what people want to get to is actions and whisper not just writing what you say but doing things for you now that is really hard and there are hundreds of companies that are trying to build agentic workflows and none of them have gotten to a point where you or me are just using it like that on a daily basis and there's a number of problems one is trust two is uh when it says like hey i can do these 50 things like siri does you don't really know know what 50 things it can do and so that is this kind of expectation reality mismatch on what the users know it can do and whatnot okay so you need to solve that problem and finally you need to make it very seamless to use because my dad is not going to go in and install 10 mcps and so okay Now that is the next hard problem to solve.

12:12And so that is what we've been doing for the last six months in solving that. Then you need to build the ability for it to proactively reach back out to you, which again, Clippy is the most prime example of what that used to be. And that should tell you everything that's hard with that because Clippy didn't respect you as a person, you didn't respect Clippy overall. And so there needs to be some level of context and competence that you need to build in the system overall. But if you actually build this, if you then get to this point where, and this is where we want to be by the end of this year, where you're whispered as a system that just gets you, you trust you to do things.

12:54It helps you at times proactively that gets closer and closer to this vision of Jarvis. And again, this is like one step at a time, really simple.

13:04Anne Dwane:Amazing. And to that point you backed that up with a metric you created yeah that um can you talk about that metric we are true north metric right now so for the wise dictation product and this is what i think for for most metrics is step away from there's there's scientific metrics that are really helpful for research then there's actually product metrics that that matter for us for wise dictation the single biggest pain point that people had with siri was it just always keeps making mistakes I was like, okay, that kind of defines the metric for us. What percentage of the times does it not make a mistake?

13:38And it's a very strict metric where if you dictate a three-paragraph email and it makes a single comma wrong, that's a zero for whisper. And that is what we call zero edit rate. And so now that we're thinking about actions and so on, there's going to be similar metrics that we create for that. But it doesn't come out of a vacuum. you. It mostly comes by giving it to people and figuring out what is the highest predictor of success for this product. Right.

14:06Anne Dwane:And this is a brilliant insight because word error rate is the industry standard for transcription. But you said, let's make zero edit rate. So I, as a user, don't just say, oh, it's good. It's like, I don't need to edit it. Like that is a much more user friendly metric that you created so it's a really fine line distinction okay and you're an amazing promoter and i'll just say people should look at your um competition for the porsche can you tell a little bit of story about this okay so marketing for me is just like a really fun uh it's it's the more fun creative like random part of the company so with this one it was like okay you know what would be really funny is if we do a mr beast like ad but actually make it very tasteful for the audience that we have.

14:54And so this was our Android launch that we were doing. And so we had this whole video where we called five contestants and they had to try to beat Whisperer. So they were trying different things to make it break. Now, what they were trying was, again, this is all scripted, right? What they were trying was what were all the features and edge cases that we have, which was like language switching and whispering silently into it and changing your minds 10 times while you're saying the thing. And the award was to get a Porsche 911 GD3 RS, which again, like no one was actually going to win. But it made for a fantastic hook.

15:41The engagement on this video was through the roof. The number of people who watched the whole thing was also fantastic. fantastic and for me kind of when I was thinking about that video I didn't want to do another video you know where like there's a founder sitting behind a table and it's like hey guys let me tell you about my new feature I was like no that's it's it's done it's like what shows up on my Twitter feed every day uh let's do something different and for me at the end of the day like good marketing is really simple you have a message that you want to deliver to people the right people at a reasonable cost.

16:20And engagement on social media is a huge part of that. And so kind of, yeah, I think this got about 50, 100 million views.

16:31Anne Dwane:It's so funny. You're a good straight man too in there. Okay, so you personally ran every one of the first 500 onboarding calls, right? So first of all, what advice would you have for founders about doing that? And then when did you know, okay, I got it. It's time. I can stop. I never stopped. Oh, okay.

16:56Onboarding is a really interesting part of the user's journey. It's the first impression that your product creates. And I think 90 % of the work happens there. so for the first 500 or so it was really important to do that because what I was looking at in every single one of these onboardings was not what they're doing on the screen but their expressions across every single page are they confused are they excited about something or are they trying to read something is it starting to feel tedious or they're starting to feel impatient and that differed by person because again we're onboarding everybody from like people like you or me to people like again my dad less tech savvy completely different kind of persona who's gonna go through the exact same experience and even now um when i'm standing in like an airport check-in line or i'm standing in a in a coffee shop i just uh i started raving about the product and I start showing the person beside me the product and like nothing to do with like me being the founder of the company everything to do with my like oh you should check this out this is so cool and then they download it on their phone and then I see them go through that experience and then my team gets a bunch of messages from me like this these these things are the things that we need to fix and it's a never-ending exercise because the more we build on the product the more we need to teach people and like the whole product and onboarding evolves.

18:27So I think I'll, I don't know when I'll stop because even today I find something new every single time.

18:33Anne Dwane:I'm sure you'll get questions after this. Okay. So congratulations. Cause whisper whisper flow is three times more accurate than open AI models today and all the other frontier labs, but I hear they're working pretty fast. So how do you think about building your own model at a time when there's behemoths investing either in your space, maybe adjacent to your space? Voice AI is actually a large space. It's funny. Like, this is the same way where, you know, in 2021, everybody was like, we're investing in AI. And you're like, no, no, AI is massive. Think about a certain segment of it. Voice AI, very similar.

19:13And so we're in the space of human to computer interaction, very different from 11 Labs, which does phone calls, very different from OpenAI's advanced voice mode, which is a totally different thing. And at this point, you know, if you asked me four or five years ago, I would have told you like, hey, we have the best voice model. It's our mode gives us five years of durability. At this point, an AI edge gives you say maybe three months of durability. So all the modes that we used to have before, they do still exist, but they're just a lot smaller. So what you need to do as a company is you need to stack these modes on top of each other you need to have the best ai model you need to have a lot of product love and then deep enterprise workflows and then high switching costs and if you have a lot of these together that is how you build resilience so now coming back to the note of okay other people are building these models the thing we realized is one building really good voice models is actually really really hard and if somebody wants to come in and compete in this space it needs to be one of your top priorities the same way for anthropic coding was one of their top priorities which is why cloud code exists and it's so good and they came in out of nowhere when github copilot and open ai had a massive leap of years on it and so at the end of the day focus is one of the biggest things And what we did most recently is we raised a quarter billion dollar round.

20:50And now we have one of the best Voice Frontier Lab teams out there that is led by Aria, who was the founding researcher for Alexa, grew on to be the SVP at Amazon, built the whole org, and was most recently leading all of Voice and Multimodal at Meta. And so now I think our ML team has grown from five to 50 and will probably be 40 people by the end of the year, which is more than half our engineering team because we need that.

21:22Anne Dwane:Okay. And let's nerd out for a second. You are not a fan of small edge models. You're doing everything in the cloud. Is that right? I'm not a fan of small edge models because they don't solve the user problem. like i would uh so the the thing people want at the end of the day and anybody who's used whisper and the voice experience you know you want it to be insanely accurate you want it to be insanely fast edge models can achieve only one of them not actually not they can't achieve the accuracy because you need a lot more intelligence and the second thing is i want to not just have this be stuck to Silicon Valley techies.

21:59I want this to be built for everybody. My uncle uses it on an old Dell Windows laptop, which doesn't have a GPU. My grandfather uses it on his old Samsung phone. Again, I don't know how old that thing is, but it still runs. It's very simple to use the product on it. And so I know in Silicon Valley, it's very easy to say, oh, let's build an edge model. Let's run everything offline, but then you just delete 99 % of the world as your customer.

22:27Anne Dwane:Got it. And then finally, do you train on individual users? So you have an individual user model and your overall? So a couple of things there. So one is privacy is a huge pillar that Whispers is built on top of because people use it for their most confidential, personal, and professional messages. But 15 % of the people opted to share in the data with us to improve the models so it's like great so that is what comes in that's how we're able to make whisper be so good across accents across languages across countries and then that that trains the overall model now for individuals what we have is the way where your data is just stored locally but it learns things over time for you builds your profile and so then for example it learns little nuances for example that my email style is very casual and this is how i write emails to people who are external this is how i write emails to customers all right people emails to people on my team and it builds all of that locally and you again now we're going in a level deeper into how the model works if you try to do this at a model level where you're trying to have kind of a custom adapter for every single person for every single context it's going to be extremely inefficient and so you're going to try to minimize the number of kind of customizable layers you can build but if you have other ways to be able to do that then it makes things a lot easier and so this is where we had to build our own model because every single speech model that's out there it doesn't let you put in a lot of context openai's model for example lets you put 250 tokens of context, which is tiny.

24:15And so we're building our promptable speech models that let you put in a lot more. And so it can adapt to every single situation, no bars. Amazing.

24:28Anne Dwane:Let's talk a little bit about, so you are in a tear now, 10X revenue in like months, 40 % month over month growth, 7X the team, but you have a history of changing your team to be more efficient, right? And can you talk a little bit about, you don't have to go into the whole backstory, but how do you run a team for efficiency in terms and efficient communications, which is a passion of yours? Yep. By the way, changing the team does not mean firing people. This is, um, no, I am, I'm insanely particular about overall efficiency. So here's one of my biggest pet peeves is when you double your team size, the output does not double.

25:14Right. And we've all noticed this a thousand person company doesn't produce a hundred times as much work as a 10 person company. Oftentimes it's like, like significantly lesser. and the thing I was curious about is like why do companies start like have this sublinear growth with the number of people it's kind of unintuitive got me to a couple of key realizations the first one is you actually have communication overhead that starts to happen for some reason all slack channels that were three person slack channels become a hundred person slack channels and everybody starts to have an opinion about everything else.

25:52The second thing that happens is you get a lot more people who get in the decision making. You need to get alignment with a lot of stakeholders. I was like, what on earth are you guys doing? This was like literally my reaction when we went from 10 people to 25 people and I was horrified looking at the kind of inefficiencies that grow on. And so that has gotten us to apply some principles within within Whisper overall, which is number one, for every single project that we work on, it should have the smallest number of people possible, which means even when we're building a new product surface, there's only two or three people working on it.

26:34That's it. the second thing is slack channels have to represent your actual uh team meetings and so if for a certain project you have three people who meet every couple of days to chat about it that is your slack channel otherwise uh you just start thinking about your slack channels as kind of physical spaces you're not going to go have a loud discussion in the middle of your entire office where everybody can hear no that's a waste of everybody's time and attention like people those attentions is extremely extremely extremely precious and so if they're working on a couple of things i just want them to keep working on a couple of things the one problem this creates is of how do you share information and how do people know what's happening this is why you start to see this and this is like now we're going to like very specific like organizational details of like how slack channels work where where people jump into your project slack channels to get information about what's happening because they have no other channel.