LiveKit and OpenAI with Russ d’Sa

0:00LiveKit is a platform that provides developers with tools to build real-time audio and video applications at scale. It offers an open-source WebRTC stack for creation of live, interactive experiences like video conferencing, streaming, and virtual events. LiveKit has gained significant attention for its partnership with OpenAI for the advanced voice feature. Russ Dussa is the founder of LiveKit and has an extensive career in startups. In this episode, he joins Sean Falconer to talk about his startup journey, the early days of Y Combinator, LiveKit, WebRTC, LiveKit's partnership with OpenAI, voice and vision as the future paradigm for computer interaction, and more.

0:42This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

1:00Ross, welcome to the show. Hey, Sean. Thanks for having me. Yeah, absolutely. I'm looking forward to this. I'm excited about all the things that you're doing at LiveKit. I know that LiveKit isn't your first company. You had a couple of swings at the entrepreneurial bat, so to speak, and you were even part of an early YC batch. Can you talk a little bit about your background and perhaps your experience with YC? Yeah, for sure. So background wise, my dad was an entrepreneur as well. And in technology, he was starting companies in the semiconductor era and early GPUs in like the kind of late 80s and early 90s.

1:38And so I grew up in the Bay Area. And I've kind of been around people starting companies and around technology for pretty much my whole life. and Y Combinator was kind of interesting. I was in college and a friend of mine ended up dropping out and joining a company that was just down the street from YC's Mountain View office. I think this was, YC started in Boston, more in Cambridge, and then opened up an office in Mountain View. And so my friend walked over one day and said, Hey, how do we be part of this? And they invited us to a dinner. I think that was for the second batch. So we went to dinner and met a bunch of founders and decided we were going to apply to YC.

2:21And so we didn't get in our first time, but then the second time we applied, I think it was for the fifth batch. Summer 07 was when we ended up getting accepted into YC and joined a group. I think it was 18 companies in that batch. Everyone was scared because it was a jump from the previous batch size. I think the previous batches, one through four, were all maybe like 10 to 12 companies. Maybe even the first one was eight. and this was 18. And so we weren't sure how that was going to work. And so we moved to Boston, to Cambridge, and I think we were the second last batch in Cambridge and started a company there.

2:58And in my batch, who was in that batch? Drew and Arash from Dropbox were in that batch. Imad from Mercury was in that batch. And so I think YC today and YC back then, both valuable propositions for an entrepreneur, but also very different. This was in 2007. There wasn't a lot of information on the internet, but also it just wasn't normalized outside of maybe Zuck about being like a young founder coming out of school. And how do you actually start a company and build a company from the ground up? At that time, the iPhone platform was just announced. Sorry, actually, the iPhone just came out that summer.

3:36There wasn't even a development platform for it yet. You couldn't build your own apps for mobile. And so everyone at that time was just working on a website. I guess Drew was working on a desktop app for storing files, but everyone else was pretty much working on a website. And so it was a different time. And I think what was interesting about my YC experience that maybe isn't necessarily the same as it is now just because it's a different beast. But back then, you kind of found your tribe in YC. You know, it's 18 companies. And when you're in college, there weren't a lot of people at that time who wanted to start companies or were thinking in that way.

4:12Everyone was, I'm going to get a job at Google. I'm going to get a job at Microsoft. And so being part of YC, it was much more like a small family of like-minded individuals who were trying to go and do this thing against the grain of what young kids out of college are supposed to do. Amazing experience, one of the best of my life. But yeah, YC now is different, valuable, but different. Yeah, absolutely. I vividly remember those days. I remember like Dropbox in the YC batch. And that was a time when you could really, like, if you were sort of deep in this world, it was easy to pay attention to like everything that was going on in YC because it wasn't that many companies.

4:45And that was also not a hot time, but 2006, 2007 to 2011, 12 was not a, like a hot time to do a startup. I started a company in 2009 and it was really a time where we were like, really, you know, is that a good idea? Yeah. We were hitting kind of like a startup winter in 2007. And I remember that as well. I think like, I want to say one of the top VC firms, maybe it was Sequoia. They definitely published like a RIP, the good times, but that was similar sentiment in 2007 as well that was happening. I think another interesting part about YC during that time was the biggest hit that YC had during that period was Reddit.

5:26It was a$10 million or so sale to Condé Nast, and that was kind of the big exit for YC. And so I would say that there was more of a sentiment back then around, oh, YC is this kind of startup school that is taking bets on kids, but who knows if this actually works? And maybe it's actually a negative signal that you're funded by YC, positive signal that it's Paul Graham, but negative that it's this new, unproven thing. And then over time, of course, you had Heroku and other companies that came out that started to do a bit better in the market and have bigger exits. And then YC became legitimized over time.

6:02And Dropbox, of course, helped with that too, and Airbnb. But it was a very different time. Another interesting tidbit about YC was that we were on the East Coast and we were pitching all these East Coast investors. And I remember Paul saying like, okay, well, this is a warmup for the real kind of pitch, which is the real demo day, and that's on the West Coast. And so after the East Coast demo day, we all flew to the West Coast. And Paul kind of recommended that founders stay on the West Coast, like move to California and build your companies there. And so at that time, Silicon Valley was still, I would say, largely South Bay Peninsula kind of focused.

6:40There was, of course, Twitter in the city and Salesforce was in the city, but a lot of the smaller startups weren't. And I think Justin Kahn and friends from Justin TV, they had this apartment in, I think it was called Crystal Tower, somewhere in the city. I don't exactly remember where, But from my batch, all of these startups, I think Drew and I think Daniel at Discuss and a bunch of them moved into Crystal Tower. And it became called, I think, like YC Tower, I think was the name or something like that. And I've thought about it back now in retrospect. And in some ways, I feel like our batch of YC kind of created the San Francisco startup ecosystem.

7:20Not intentionally, it just kind of incidentally happened because all of these founders from our batch were moving to the city and all living in the same place. And everyone knew about this being like the YC startup kind of epicenter after you graduated from YC. So kind of interesting to look back. Yeah. And I think that, of course, as companies that did well, that maybe had their start in San Francisco, it creates a situation where people take the wrong interpretation of that and associate, okay, well, if I want to do well, I also have to be here and stuff like that. It becomes this, people like cloning what they believe to be the recipe of success and so forth.

7:56But so now you're at, you know, you started LiveKit. How did that end up? What led to you starting that company? Well, LiveKit is my fifth company. And the one I did in YC was the first one. And I've had a few in between. It's interesting because my YC company back in 2007 was trying to do real-time streaming a video over the internet, introducing two people to one another that I've never met before. I don't like to use the chat roulette three years before chat roulette kind of analogy, but effectively that's what it was. And very few people were doing real-time audio and video through a web browser at that time.

8:30And so we kind of rigged something together with glue and tape, and there wasn't great support within the browser for it. Kind of fast forward to 2012, WebRTC, which is this protocol that was designed specifically for streaming real-time media, started to get built into Chrome and then slowly expanded into other browser implementations as well. And now in 2021, when I started LiveKit, I'm kind of returning back to the same similar type of technology that I was working on in 2007. Now I'm much more mature, and I think the use cases are a lot more clear too. So we started off trying to connect people over the internet in 2007, and then pandemic happens in 2020, and everyone's building software to try to connect people over the internet because that's the only place you could go to find other people and interact with other people that weren't in your house with you.

9:21And in that world, it's kind of shocking how little had happened in terms of having scalable infrastructure that made it easy to build an application that connected people in real time. Yes, you had something like Zoom, which they'd been working on it for a decade and it was very mature and performed very well, but that was at the application level. If you wanted to be able to build something like Zoom or take Zoom's features and put them into your application, that still wasn't easy to do. And there was actually no open source infrastructure for doing it in 2020, 2021. And so I was working on a side project that was trying to do something similar, which is connect people over the internet in real time.

10:02This time, though, for audio and no video. And I struggled with the same thing. There was not really great infrastructure to do it. There were commercial providers, but they were really expensive. And they didn't scale to really large numbers of people. And so my application needed to be able to run economically. It needed to be able to have thousands of people in a session together, potentially. And that's when I started to look at open source, realized there was nothing there. And so pinged my old co-founder from the previous company, company number four. And fun fact, we met during company number one, we started separate companies in YC and we met in that batch.

10:37But we started LiveKit together as an open source project to build infrastructure that any developer could use to create real-time experiences. In terms of that infrastructure and that open source project, I guess, where did you start with that from an engineering perspective? And then how did you actually get from this open source project into it turning essentially into a company? So the way that it kind of happened was the original impetus for working on it was a side project. So I was working on my previous company had been acquired by Medium, and I was now leading product at Medium. And I was working on a side project that was kind of best described as clubhouse for companies, where you could have kind of drop in casual audio conversations with your coworkers during the pandemic.

11:26And when I went to go actually integrate the streaming audio piece of it, what we found out was when we were looking at open source to integrate into the application, there wasn't anything that was easy to deploy that had SDKs on every platform. And that really felt kind of consumer or production grade to use. I kind of liken what LiveKit was trying to do and is trying to do as what Stripe did for payments. LiveKit is trying to do for communications. So Stripe didn't invent payment processing. There's already an underlying network of payment processors and gateways. And what Stripe did was they effectively took all of this infrastructure, which is very difficult and a bit obtuse to integrate into your application, and made it into a very simple piece of software that you can go and write.

12:15And they handle all of the complexity or the undifferentiated problems of handling payments in an app. LiveKit is doing the same thing or an analogous thing for communication. Handling the undifferentiated pieces of how do you actually connect one or multiple people or machines together with ultra-low latency anywhere in the world, where those nodes in this graph are located anywhere. The interesting part about the technology that existed before LiveKit existed is that the protocol itself is already there. I mentioned it earlier. It's called WebRTC, Web Real-Time Communication. And the way to think about that protocol is it's a higher level abstraction built on top of UDP.

13:00Most of the internet itself is actually when you use a browser, when you interact with applications, what you're interacting with underneath is really TCP. TCP and UDP are these two different protocols, and TCP is not really designed for real-time media streaming. The main reason for that, there's one key difference between TCP and UDP, a lot of differences, but one key one. The key one is that TCP requires every packet that is sent over a network to be ordered. So the receiver must wait for those packets to arrive in the order that they were sent. That's the main difference. So why is that a problem for real-time media streaming?

13:44Because with real-time streaming, you only really care about the latest packet. Something that someone said a second ago or a video stream capturing someone from five seconds ago doesn't really matter. You really just care about the absolute right now edge kind of packet. And so with a protocol that requires that all packets arrive and are handed to the application in order, if a packet gets delayed while it's transported through a network, or if it gets lost somehow and not delivered, you have to halt the entire application before you can actually touch any of the packets that are coming in real time.

14:26You have to go and get that old packet before you can do any of that. And so TCP is not well designed and the internet in general is not really designed for real time audio and video. UDP gives the infrastructure provider or the lower level software layer, it gives you full control over what to do when you miss a packet or whether a packet's delayed. You can decide to actually fetch that old packet if you want. You can decide to try to conceal it by interpolating between packets you have and closing the gaps that way. You can add error correction so that if you have a damaged packet or missing information, you can reconstruct the information that you were missing from kind of the pieces that you do have.

15:06There's all kinds of techniques that you can use to deal with packet loss or delayed packets over the network. And UDP gives you that control. You don't have that control in TCP. And so WebRTC is just another layer on top of UDP that gives you kind of nicer facilities. But at the same time, the problem with WebRTC by itself is that it's peer-to-peer. So you're sending your data over the public internet, and there's no server in the middle when you use vanilla WebRTC. And so if I'm talking to three people in a video call, I'm sending three copies of HD video to all of those people at the same time over my home internet connection.

15:43It just doesn't really scale. And so what LiveKit does is LiveKit builds server infrastructure and client SDKs where you take the server, you deploy it somewhere in a data center. And then from the SDKs, the client SDKs that you integrate into your app, they all connect to that server. And the server is like a router that everyone only sends one copy of their information to that one server. And that server is determining who should get what at what frame rate and resolution. And so it's the mediator here. LiveKit Cloud is the next step above that. And that's kind of the commercial thing you were asking about, like, when did we start a company from this?

16:17I think we were actually on a more accelerated timeline than most companies are in the commercial open source space. So folks like Elastic and Mongo have had a longer amount of time to kind of build out communities and continue to improve the open source kind of by itself in isolation before they have the pressure to commercialize. For LiveKit, it was a little bit different. And the reason it was a little bit different is we put out the open source project in July of 2021. It became a top 10 repo on GitHub across all languages for six months very quickly, I think within three weeks. And then we had companies like Reddit and Spotify and X and Oracle who were already deploying it internally and starting to experiment with it.

17:01And when we had conversations with them, they said, hey, look, we're coming from Twilio or we're coming from another one of the larger real-time streaming companies. We love your code. We can see your code. You guys designed a good system, but we don't really want to deploy and scale this ourselves. even though we know we can, we would rather you do that for us and operate your own network. And we're happy to pay you for that. And so I think a combination of large companies that were pinging us and offering to pay us right out of the gate was one kind of piece of pressure that caused us to commercialize very rapidly.

17:33The second was, it was 2021. And so the demand for real-time audio and video infrastructure was very, very high because of the state of the world in the macro environment. And so we made this decision that we were going to continue building an open source and supporting that and giving free support and helping people in the community build. But we also had to raise some money and build a team up a bit bigger and start to actually work on a commercial product, which is called LiveKit Cloud. Yeah, I mean, that's a fantastic story. It's actually, I had a similar conversation with the CEO and founder of Crew AI recently, where kind of similar idea, he built something for a side project that he was interested in, open sourced it.

18:15suddenly Fortune 500 companies were calling him and asking him questions and how do they manage it and so forth. I mean, that's like the dream scenario for anybody that wants to start a company. Totally. In terms of going from sort of this peer-to-peer WebRTC world to then making it so that you can have this essentially server-side router that is going to proxy these calls to the various people who are trying to interconnect, What were some of the engineering challenges that you ran into with changing from this peer-to-peer model into this more client-server model? So I think that there's kinds of two pieces to this or two sort of scaling mechanisms that you have to tackle when you do this.

18:56So going from a peer-to-peer model to a server-mediated model, that I would say is relatively straightforward. word. There's this term used in this world, not super important to know, but just for completeness, it's called an SFU. So a selective forwarding unit, just a fancy name for a router, a router of media. The selective forwarding unit is your single server system. You deploy it somewhere in a data center. And then what it's doing is it's acting like a peer in WebRTC. So just like my client device is a peer, this server acts like a peer as well. It speaks the same language as if it's just another human somewhere else in the world.

19:38A client device sends its media to that server, and then that server makes a decision or it's aware of who else is connected in the session to that user, and then starts making routing decisions about, okay, send them this byte, send them only audio because they only want the audio track and they muted that user's video, et cetera. Building that server, fairly straightforward. There's other people who have implemented SFUs. LiveKit was not the first one. Now it's the most popular one, but it definitely wasn't the first one. And that system works to kind of surmount the first scaling challenge of peer-to-peer WebRTC, where everyone's sending multiple copies of their media to everyone else.

20:16The problem with that model, though, and the scaling wall you hit relatively quickly with this single server mediated WebRTC, we call it single home architecture, is that there's three problems. There's a reliability problem, there's a latency problem, and there's a scaling problem. So the latency problem is that when you send your packets from point A to point B, what you really want to do is you want to minimize the amount of time that your packets spend on the public internet. When I send data from San Francisco to Germany in a peer-to-peer world, I am effectively sending my packets over the road system of the world.

20:59Networks are like the road system. There's all kinds of bridges and ditches and traffic and rush hour happening over here. Routers and ISPs can hold your packets for certain amounts of time. It depends on what other data people are sending. So it's a pretty complicated web that a packet has to traverse to get to where it needs to go. And what you want to do is you want to be able to bypass that kind of web. It's not a big deal if someone in San Francisco is sending packets to someone else in San Luis Obispo. That's not a big deal. But sending packets from San Francisco to the UK, it's a much longer haul, and you're going to encounter more of this kind of mess of road system on your way between those two points.

21:40So what you want to do is you want to try to terminate the user's connection as close as possible to them and then use the private internet backbone to route the packets. What is a private internet backbone? All the data centers of the world and all the cloud providers, they basically have interconnects and they have kind of these private fiber networks on the back end that are wired up and not as noisy as like the public internet. And so you want your packets to spend as much time as possible on these super fast routes. These, I don't want to say information super highways, but like you want them to like be on that kind of good network for as long as possible.

22:17The Audubon of the internet. Exactly. I love that. The Audubon of the internet. And you want them to spend as little time as possible on like the residential streets. So you want to have a server as close as possible to San Francisco and a server as close as possible to the UK and then route between those two servers over private backbone. And so that's something in a single server system that you don't get. Everyone connects that single server wherever it is located in the world. And you're going public internet to the server and then public internet out of the server to wherever the destination is.

22:48So latency penalty. Reliability penalty because that server is on commodity hardware in a data center. It's going to crash. How do you resume a session kind of seamlessly without a user interruption or with minimal user interruption? Unclear how you do that. So kind of the failover recoverability or reliability is another issue with the single server model. And then the third issue is scale. You can vertically scale that machine, right, add more and more connections to it, more and more users. But ultimately, you will run out of resources of the physical device underneath. And you need to be able to kind of horizontally split and scale out horizontally.

23:25And so these are the three weaknesses of the single server system. And you have to be able to scale past that. So how do you scale past that? Well, you scale past that by creating a multi-server sort of mesh network. What you want to do is you want to have servers all around the world, be able to spin up as many nodes as you want anywhere in the world. And then any user that is trying to connect, regardless of where they are, they connect to the closest server to them. So you minimize that time on the public internet. And then this mesh network on the back end, all of these servers and data centers communicate with one another.

23:58And they form kind of a fabric for routing information between them. And you have software that is doing all of that kind of connecting different points. And packets go over that kind of mesh on the back end and then exit out and travel along the shortest path to their destination. And so that's kind of the system that we built with LiveKit Cloud. when these larger companies were asking us, the open source piece is all of the SDKs and our single server. And then the part that is not open source with LiveKit Cloud is this kind of orchestration system on the backend that allows you to spin up these LiveKit servers just kind of infinitely, as long as you can find machines.

24:37And they all get registered as part of this fabric and there's software that does all the routing. The challenges that you have to solve there, there's a lot of them. So one in particular that I think is quite interesting is state synchronization, right? What you don't want to have in a system like this, where it's kind of a mesh and all of these data centers are connected to one another, is you don't want to have a single point of failure. If you have a single point of failure or one single coordinator or a few of them, when those coordinators go down, all of a sudden a session is severed if the session spans multiple people all around the world.

25:10So you need to have kind of this, you know, almost like a truly distributed system where any data center can kind of run independently. And then there's a type of a quorum that is formed when they want to intercommunicate. And so how do you manage state synchronization properly across all of these data centers? And you're aware and measuring the network in real time for are there connectivity issues between two data centers? And does a connection get severed? We had to deal in the early days with these undersea cables getting severed between Europe and Asia. And how do you deal with situations like that?