Your keyboard is the real bottleneck | Wispr’s Sahaj Garg

0:04Sahaj Garg:Today, we're exploring the way engineers are relearning how to build software from scratch with Sahaj Garg, co-founder and CTO of Whisper Flow. And Whisper is a company that's been on everyone's lips lately, and for good reason. They're at the center of a shift that many teams are feeling, moving from voice to text, from never worked to a possible workflow that people now use every day. all day. And this change in the bottleneck going from different parts of the process to actually being your keyboard has led many leaders and senior engineers to discover that they can express their context, taste, and intent faster, and help their teams move quicker with voice.

0:45Sahaj Garg:Because what if the limiting factor in your organization isn't actually like a context window, but how quickly you can get the right context out of someone's head? So today we're going to be talking about shared context as being a valuable currency, and all of the different changes that this introduces to how people can communicate with software. And we're going to get really practical about that as well. So Sahej, welcome to Dev Interrupted.

1:10Andrew:Thank you, Andrew. It's fantastic to be here. Really excited for this conversation.

1:14Sahaj Garg:Me too. And I want to start at the top by just mentioning a little bit about using Whisper. I've been using Whisper Flow very recently, and I've totally fallen in love with the type of software that it is. It's a true delight to use. And I can see why it's dismantling the way that engineers have traditionally approached working with code on their keyboard. I myself have definitely written like a whole novel at this point. And Whisper even tells me as much. And so I'm actually so blown away by how much I can trust this technology to express what I'm trying to say. And that's what I really want to explore today.

1:46Sahaj Garg:Because I know me and our listeners, too, we've all been burned in the past by like bad transcripts. or like, you know, that voice of text didn't really capture what I said, or that voice note had a crazy typo in it. And those little tiny burns, they add up over time and people walk away from the technology. But, you know, we're seeing a shift now where you're rebuilding trust in a technology that many had dismissed. So what has that been like for you at Whisper and how did you approach that challenge?

2:15Andrew:Yeah, it's a fantastic question. And it's actually one of the reasons why we almost never built the product that we did. Because in some ways, I actually thought using voice for communication, for typing, for interaction in a computer was like a fundamentally doomed thing after 20 years of being disappointed by every product there was in the market. And I think the thing that we learned as we built it, and especially the first version was like, oh my God, when this works, it's magical. And that is the thing that we have so consistently heard from people who use the product over time, even now, which is when it works, it's magical.

2:48Andrew:and all the work that we try to do here is to expand the settings and the context and the places in which you just get it right for the first try. Because like, as a user, what I want is a system that just gets me intuitively. Like, I don't have to explain myself a bunch of times. It should know, like, if I'm talking to Cloud Code and I'm talking about a.end file, what that actually means, right? Not show up in a completely nonsensical way. I'd say, like, the biggest framing for us of this problem, right, is we want to build you a voice interaction where you never have to go back and fix the mistake.

3:22Andrew:For us, we call this zero edit rate, right? Something where you have to fix no mistakes with what you're doing. And that means both getting everything you said right and figuring out what you actually meant to convey so that we can help you fix that up on your behalf in a way that sounds just like you and that you can actually use downstream.

3:42Sahaj Garg:So really, it's like a two-part equation because you have like the traditional layer of like, Like, oh, yes, we can turn this voice into the words it is. But then there's also this contextual layer of understanding what are you operating in? What are the words around you? And what have you said recently? What are the things that matter to you as the user? And combining those two is like the formula, I think, that whispers is getting right and is what's letting people work very quickly with it. So I'm curious in how when you solve the problem of like rebuilding trust and doing it in this two part way and acknowledging that it's a nuanced engineering problem.

4:18Sahaj Garg:And so you're approaching this with your teams. And, you know, we're now in a green field because we've acknowledged that there's a lot more nuance in how we solve this. So what assumptions there in that world do you throw away to ultimately arrive at an application or a system that's a delight that gets that zero edit rate? You know, I'm curious, like what becomes the secret levers?

4:39Andrew:Yeah, yeah, yeah, yeah. There's a bunch of different ones. I think the first one is this idea of like, don't treat speech models as dumb things. Like right now, speech models are mostly dumb. They just take an audio. They just produce text. And it's kind of like listening to like a three second voicemail from somebody who you don't know. It's like very hard to actually figure out what a person's saying. and so the really fundamental assumption to break down is like speech models actually need the kinds of things that people use to make sense of each other which is that context that memory that understanding of the other person which helps make sense of so much more and so there's this part of it and then there's like how do we actually build products like this and i think products in general right now is there's two types of things that we do there's one type of work where we do where we just really really care about precision and scale right so this is like build the world's best speech models and spend and pour all of our energy into making it better and better and better and this is like a monumentally difficult effort in terms of how we actually approach it and requires very sustained persistence it's not the kind of thing where you can like vibe code in a day and have the thing work and like it's going to solve all the problems it's very much you know Use the coding agents to build out all the infrastructure, to train these models, to create a self-reinforcing feedback loop to make it better and better.

6:03Andrew:That's one half of our work. And the other half is the user experience, right? How do we help people build this habit? How do we make it really good? And that's where we actually do a lot of experimentation. Build a version of a way you can handhold a user to learn how to do this. And ship that every two hours, right? Because with being able to go from we built a thing to we saw what happened, now we learned something to now I can express that idea of what I think we should do next into a coding agent, it completely changes the feedback loop of how we build a product like this. So much of that is experiment to learn, experiment as quickly as possible, and then get that to be something that billions of people use.

6:44Exactly.

6:46Sahaj Garg:And you're calling on something that's really smart that actually a recent guest, we had an engineering lead from Codex at OpenAI talk about by capturing the effects of research and using that to drive downstream engineering decisions and then feed that back into the research to create this really amazing feedback loop. And we recently talked with Jeffrey Huntley, who talked about with the Ralph loop, it's all about capturing the back pressure of working and finding the outputs that are most useful for that next input. So what you've described here is fascinating because it's like you identified and created that compression event.

7:19Sahaj Garg:You are doing the iterative work, that persistence, you said, which I love, on the model level, which is necessary. But then you're approaching the UX level with the level of experimentation and rapid iteration that's needed to really survive and be effective as a user tool. And those two things are a constant, probably, balance. And it's a new kind of engineering dynamic that I think a lot of product leaders are still kind of wrapping their head around. I think you've you painted a really clear picture of how those things work together.

7:51Andrew:The way I think about it is like, what's the hat to be wearing right now? Is this the kind of thing where we know a certain approach is going to work and we just have to hammer away at it to figure out how to make it work? This is what we call like no brainer bets or scaling here internally. Or is it an experiment? Because if you treat an experiment the same way that you treat the no brainer bets, like you're going to get nowhere. You're going to get nowhere at all. And so the thing that I always ask myself is like, hey, put on the right hat. What's the hat to be wearing right now? And what's the right way of working to accomplish that?

8:26Sahaj Garg:Absolutely. So I want to dive into a bit how this shifts the technology itself. I want to talk a little bit about Whisper as something that now has been created in this compression chamber capturing event that's taking all of the best of research, all of the best of UX experimentation and combining it into what we know as Whisperflow, that's transforming how people work. And if we take that one step further into the downstream effects for how teams are now communicating and sharing their context and engineering, both like in an IDE or with an agent, along with what you've built. And I want to frame that actually in a really fascinating way.

9:06Sahaj Garg:It ties back to an article I just recently read from Steve Yage about how the economics of the software era are changing. And the types of software that are useful and survive now in this new era are just fundamentally solving a problem of cognitive burden and couldn't possibly be replaced. And voices like that, like conveying your thoughts and being able to do so effectively and accurately, most people are not going to have the throughput or the energy to solve that problem. So there you discover your moat in this new AI era. So for folks that are now using these types of tools, like how do you think they would utilize your tool compared to someone who wouldn't to get further ahead and to share their context best?

9:54Sahaj Garg:Like when you model your users and your downstream engineers, for example, what does that look like for you? Yeah.

10:01Andrew:So I'll talk maybe high level about how things are changing and then for different types of people what this means. So in terms of how things are changing, right, the better and better AI gets, the more of the work it's doing. But the one thing it can't do is figure out like what's in my head, what's in your head and the unique things that we observe in the world. Right. And so that's the one core thing where we want to help amplify people's voice. We want to amplify the thing that they can get into communicating with an AI or into communicating with other people. The two types of places where people are going to be communicating right now.

10:33Andrew:there's a lot of different ways in which it amplifies different people so if you're a developer and you're building software the most important thing to actually get these tools to build a software that you want is to express with clarity what you actually want to achieve and to go back and forth on brainstorming with it if you don't do that then you're not able to actually build the right thing and build the right plan and actually go and execute on that plan and so fundamentally right that system is bottlenecked by like your ability as a person to express the things that are on your mind into the tool.

11:04Andrew:The other like really powerful thing about voice is even if you don't have your thought fully formed, you can still express it into the tool because it'll help you think through what you haven't fully understood yet, right? So that's an example for developers where it's like you vibe code the wrong thing, debugging it and fixing it takes weeks, right? But if you get it right on the first try because you gave it all the context upfront, it like saves you all of that downstream pain. The other example is kind of managers and leaders. Like a lot of what managers and leaders do is they have context across an entire company or an entire organization.

11:38Andrew:That's one of the unique vantage points they have. They know a lot of different dots in the team that they're leading. And a lot of their work is connecting those dots. And connecting those dots often means just getting the right information to the right person at the company at the right time. And so if you can do that faster, right? Because you can get a message and instinctively just say the reply instead of having a meeting, right? Avoiding the meeting when you don't need to and avoiding the pileup of like 100 messages in your Slack, it becomes an extremely strong amplifying force, right? Because then one leader can help unblock people so much faster and keep doing more.

12:15Andrew:Those are like two examples. There's plenty more for people who communicate with external clients or customers or legal industries and so on and so forth. But I think those two kind of give a sense of what it looks like.

12:28Sahaj Garg:totally and so for you know engineers doing for example async engineering work and where does voice meaningfully improve that uh like things like around reasoning and exploring your code and architecture and where does it possibly introduce ambiguity because there's obviously lag in communication between teams and people and kinds of async and remote coding environments i'm curious too i've seen the really cool posts of people like in their engineering offices with the gooseneck microphone you know everyone's whispering in the engineering room i love that like i want to be there and i want to be like it's so quiet too is what i hear so like i love that story but then there's also the engineers plenty of us who like me are you know work in our living room and so how does exploring and molding with voice in this way like what are things they should

13:14Andrew:keep in mind yeah versus like in person um i actually think they're the if you work remotely or in a hybrid capacity it's like the easiest to adopt tools like this because when you're not around other people, it's so easy to just speak to your computer all day. And it's really delightful too, right? And so there's like that aspect of doing it. And there's that aspect of using it to kind of catch up on async quicker when it comes to communication and so on. The one pitfall is I've seen times where people on our team are both using flow back and forth to each other, DMing each other back and forth in a Slack thread.

13:46Andrew:And it's like, I'm speaking to you, you're speaking to me, but like, we're both using a tool to turn it into text back and forth in for real time. And that's not good. Like at that point, you should hop on a call, right? And I think it's really easy, especially in hybrid and remote environments to accidentally fall in that trap sometimes. So that's the one thing to kind of caution against when you start using tools like this is, hey, what's the way in which it could impede really important live and productive conversation. But outside of that, it's like, I use the tool definitely the most, It's like between 10 p.m.

14:19Andrew:and midnight when I'm just able to work at home and really get in my full state. Yeah.

14:27Sahaj Garg:And I think it's funny, too, that you talk about these communication anti-patterns that kind of emerge. I think, I don't know about you, but I've definitely at some point in my life been at the receiving end of like a stream of consciousness voice note from somebody. And you're trying to follow and trying to understand. And now it's like we're in a world where that could be a Slack message that comes to you across from somebody. maybe even your boss and they're trying to get you to do something so it's more important than ever that like these tools don't act as obstacles they don't make work for the recipient you talk about a zero edit rate but also like a zero work rate for the recipient if they are on a receiving end of one of those to understand because like we talked a little bit about like the obviously talking to your computer total delight but being voice to text it can be used in a whole bunch of ways so you have to think about a really wide user span right as like a as a pretty big challenge especially as like a cco like how do you how do you you know bucket and understand that it goes beyond and just like you know when you go to onboarding flow and you're like oh i'm a coder i do engineering work like it there's actually more to that it's like it's like how do you tackle

15:32Andrew:that problem it's actually tremendously tremendously nuanced so there's two parts of it right so i think about it as okay right now we're a tool to amplify our communication eventually it'll be a tool, but also takes actions for you. But let's just talk about communication right now, right? The way we think about the product spec for our language models that are going from what you said to what you want to communicate is there's two tasks, right? One is to make it something that represents what you said in a way that's true to you and true to the communication you want to convey. And the second is to make sure it's intelligible as the person receiving it, right?

16:08Andrew:Because I don't know about you, but I've never spoken for two minutes straight and been able to produce a perfectly coherent email when I speak. But the recipient of this kind of a conversation, you'll understand me when I speak for two minutes, right? People listening here will understand a two-minute dialogue. And so there's definitely a way to satisfy both of these constraints. And where it gets even more kind of challenging is like the way that this should happen depends so much on who's communicating with who. Like when I text my co-founder, some of the employees on my team, my wife and my parents, it's all going to look pretty different.

16:46Andrew:And the shared understanding, the shared context, they're super, super, super different. And so what we really try to do is basically build ways to automatically infer and learn your intent. Understand what you're trying to do right now. Give you as a user control. right so if you wanted to be more like verbatim to what you said or if you want to be more interpreted on your behalf give you that control and then learn your preference kind of automatically over time right because the the way that i see it is like hey you fix a mistake that we produce we should learn that and not do it again right why should you have to tell us multiple times we should be able to learn all those things about how you might want it in different settings like different from how somebody else does.

17:34Andrew:And so maybe the biggest challenge from an actual engineering and tactical perspective is not trying to do everything for everyone all at once. Because if we do that, it's very hard to actually tackle all these problems. But really methodically working through it in a way that's both specific and then over time generalizable so that we can make it work actually for everyone. Right.

17:59Sahaj Garg:Right. And in thinking about the two, you know, and thinking about your wide breadth of users and the different contexts that they find themselves within, then even within that, there's a level of granularity of the type of handoff that it is because there's, you know, in cases where there's a human talking to their agent. And then, of course, like hearing something back from the agent and then working with it versus working with a co-worker. Right. But in the world we live in right now, all of that text swims together. And so it's a fascinating kind of problem to solve for. But I'm curious to know, too, like when you are able to pull the context out of people's head, what form do you think it should best live in?

18:39Sahaj Garg:Because obviously when we capture those thoughts and then they become markdown documents and these things accumulate, right? How do you, as somebody who's, oh, I'm capturing my thoughts and working with them, not create garbage or instead of creating useful artifacts?

18:54Andrew:um it's a great question and i think there's like garbage and two ways that you can create right one is the stream of consciousness that goes to somebody which is that's really dangerous as you said uh and then there's a stream of consciousness that you output into your notes app and i think everybody everywhere has never had a great experience with finding a way to like dump all of the crap that's going on up here into something that uh is useful over time right and so I think one of the unique things about voice is the thing it's best for is frictionless capture of the jumble of ideas in your brain.

19:35Andrew:It's so much better than anything else for that kind of problem. And then the question for us will become, hey, how do we help you organize that information for yourself? How do we help you proactively make sense of that kind of information for yourself? And those are two problems that we're very actively exploring right now and prototyping different solutions because, you know, this idea of being able to offload my thoughts into a second brain is something that people have, I'd say, tried quite a bit and never really found a way to make it stick. And like voice is that interface to which we can do that.

20:15Andrew:And yet today, like our best tools kind of just tack on some AI after the fact to like try and help you maybe do something which is not really what people ultimately need or want to solve that problem.

20:29Sahaj Garg:Which is why going back to perhaps what you alluded to, the idea that you solve this problem now for voice and in understanding that context and then the, you know, the idea of what someone's invoking, you can then go one step further and solve the action problem too. And because you understand the context in which they're operating in, in a much more intimate way, because of that's the, that's the benefit of voice. So is that where you see this type of technology evolving, where, you know, I could today, of course, talk to any sort of type of tool to operate things for me. But do you think that maybe even that collapses one level further?

21:07Andrew:I think it does, right? If I think about five years from now, what does computing look like, right? I'm probably just going to be either expressing my intention to my computer in response to a decision that it asked me to make or like spontaneously because there's something that I want to put into it. And like the things that I want to happen and will actually happen, right? Like that's the magic that technology is supposed to promise us. And unfortunately we're now in a like world where we're like twiddling away tiny thumbs on a tiny screen, far, far from that part of that promise, right? But that's what I expect it to be.