AI Gets Eyes and Ears | LiveKit’s Russ d’Sa

0:05Welcome to Dev Interrupted. I'm your host Ben Lloyd Pearson. And I'm your host, Andrew Ziegler. This week has some pretty weird news. We're talking about a new problem called slop squatting, introduced by Code Generation, the AI turf wars, and buyers literally lining up to buy Google Chrome, and some new vibe coding techniques that we've been reading and trying out. What do you want to talk about first, Ben? Well, as much as I love talking about vibe coding, particularly because I just started doing it this week for the first time, this word slop squatting just really has me going. So let's talk about that one first.

0:42Okay, this one's really interesting. So slop squatting is a variation of a vulnerability, a security vulnerability during production called typo squatting. This happens when you think you're installing a package, maybe like a really popular or big package into your project. but maybe you make a typo or you forget that hyphen or you use an underscore instead of kebab case. Whatever happens, you know, you can end up downloading the wrong package and inside of that could contain malicious code from a third party. And slop squatting is the newest variant of that. It happens when you ask an LLM to generate code and it does so so diligently, but in doing so, it maybe hallucinates a package that before didn't exist.

1:25and it can do this using really common combinations of really popular packages during everyday development of common things that you might ask it to do and when these things slip into your project suddenly you're opening the door to a malicious user or you know even like a hacker to get into your application so ben you know what do you think of this kind of phenomenon evolving as people start exploring vibe coding like yourself i really think this is like a cautionary tale for anyone that's running a software team. You know, as more developers and frankly, non-developers to get access to AI tools for scaffolding and creating code, the surface area for attacks like supply chain attacks is exploding.

2:08Unlike traditional attacks like typosquatting, for example, this doesn't rely on human error. You're potentially doing everything right. It's just exploiting something that or an assumption that you can make about how AI operates? And it raises some pretty critical questions for this AI-driven agent-native future. If agents are writing our code, what guardrails are we implementing to stop them from doing things like importing malicious packages? I think what we're really seeing is there's still a lot of unknown unknowns in the security space when it comes to AI. We're learning a lot of new things about how it can be applied to take advantage of you.

2:48And, you know, it's you got to stay up to date on what's happening. And of course, I started vibe coding this week for the first time. Like I am now taking such a fine tooth comb to every single dependency that it tries to bring into my project, which is a good practice. You should always do that. But it also made me realize that even when you know that this is a challenge, it still can be kind of difficult to validate that these packages are legit. Like it's pretty easy to fake a package page on a repository. you know so just like until we have tools that make this a lot more consistent and automatable i think we do really need to be conscious of when we use ai and when it does things that bring in new dependencies into our projects you know we'll talk a little bit more about this later too but it really highlights the skill sets and the in the mindset shift that you have to adopt when working with code generation in this way because you're spending a lot of time now looking up the packages that are going into your project and really understanding the building blocks of it.

3:47And that's because, you know, maybe some of your time that would have been dedicated to coding is now freed up to do this better, higher level understanding of your application. So it all plays together and really highlights the importance of understanding the code you're shipping. Yeah. So I want to talk now about the AI wars, because this is really starting to seem to heat up. So what do we have for that this week? Oh, yes. So there's a whole rumble right now in the tech world because Google has to sell or divest from Chrome. Judges ruled that as part of a monopoly that Google can no longer keep Chrome as part of its portfolio.

4:23And this is causing a lot of large tech companies, people adjacent to search and AI to swarm the scene, literally somewhat like vultures, trying to immediately buy the Chrome browser. Now, why is this important? Obviously, we all know Google is a large company, very large reaching tech portfolio, powers a lot of the modern world that we live in. And when you talk about a fundamental tool like Google Chrome, which is almost ubiquitous now with accessing the Internet, there's a lot at stake. It's a large user base. And this is happening at a critical time when we're completely re-evaluating what it even means to go online and use an application or search for information.

5:04and the ways in which people are doing this are kind of flipping on its head. You have AI that's kind of coming in to search. You know, we've been covering this a lot on the pod. We've been talking about how there's these encroaching wars between going to ChatGPT to search a query and going to Google and getting your AI-generated responses. They're both playing in each other's ball yard. So all of that is to say is that you've got companies like Yahoo, Perplexity, DuckDuckGo, all lining up. OpenAI, of course, wanting to obtain some of this for themselves. So Ben, what do you think of this circumstance?

5:39Yeah, well, first of all, as someone who used to work for Yahoo, I love that they're trying to find a way to make sure they remain relevant in the AI era. And also, I just want to say, I just want to have it out there. I would also like to buy Chrome. I might end up being like TuckTuckGo and not actually being able to afford it. But that's besides the point. I would like to buy it if that's possible. You know, we'd have to talk with them and ask. Yeah. But personally, you know, I would love to see it hosted like under a foundation similar to Mozilla, but it's probably like an unreasonable thing to expect.

6:10And maybe they'll become like an independent company. But it really does not surprise me at all that we have all these vultures circling to scoop it up once Google is forced to sell it. I mean, I think everyone would like to own Chrome. So it's kind of like, yeah, you and everyone else. Thank you. I think, too, is that the key thing here is that the tool we're going to use to search or go online and go to websites, I think the tool we're going to be using in a few years from now probably hasn't even been invented yet. The Internet and the way we access it completely reinvents itself very frequently.

6:41The browser wars, like that used to be a thing. We weren't sure which was going to emerge ahead. You got like Mosaic. Then you had Netscape. Then you had Internet Explorer. Then you had Firefox. And you had Chrome. And then Chrome became huge. And, you know, what's next? I don't know. but I'm sure it'll be something different. Yeah, and I would definitely love to hear more about OpenAI's vision about an AI-first browser because that was something they mentioned as a part of this. So like, it's probably going to be a cool experience. I would really love to see that happen. But this wasn't the only thing that we saw in the AI wars, right?

7:12Wasn't that we have another story involving Cursor and Microsoft? Oh, yes, we do. So there's something that happened very recently on the VS Code extension for C and C++. This is an extension that powers a lot of the tooling within VS Code to work within that language as a very popular part of the ecosystem, right? But recently, Microsoft released Agent Mode for GitHub Copilot. And so this is GitHub Copilot starting to move into things like Cursors territory. Going back to our turf wars, what we just covered with search and with AI, you have this happening as well with code assistance. And so by blocking the VS Code extension from working in things like Cursor, which is entirely within their realm and their right, because this is an extension they have developed.

7:57And if they want to make it for their own applications, they can. They haven't stopped Cursor from using or adopting a different type of tool within the ecosystem because there's lots of open source alternatives. And they have since done so, right? So it really shows that Microsoft might try to close a door to make less of an opportunistic space for their competitor. But because they also control so much of the ecosystem, they get to make moves like this that are quite fascinating that other companies can't quite do. Yeah, I think the AI space is going to get really cutthroat. And we're just going to see this competition heat up, especially with how much money is on the line.

8:34Also, given that nobody has established a defensible moat in this space yet. So everyone is prone to being disrupted that operates here. and I doubt it's a coincidence that Microsoft times this blocking with or blocking the VS Code extensions with the release of their agentic mode for co-pilot and I think on one hand it's a lesson that all startups should learn like it's Cursor's fault for relying on a competitor's product to build upon I also understand why they made that decision it's a tiny company they don't have very many employees they bootstrap their way to to where they are so it's a natural that they're going to encounter growing pains like this.

9:10And it also makes sense to me why they would adopt open source as the solution, you know, because it's a lot lower barrier to entry. Yeah, completely. Yeah, so this week in VibeCoding, we have yet another story on VibeCoding. So what do we have, Andrew? Okay, so this is a really great one that I read. I also shared it on LinkedIn. This comes from Pete Hodgson, and it talks about a prompting method within an agentic coding tool like Cursor or GitHub Copilot agent mode called Chain of Vibes. And really the thesis here is that AI isn't ready for unsupervised coding. And Pete found that a workflow called Chain of Vibes lets him maximize how much he can lean on the AI for coding while still having a really good rigid process with a human in the loop.

9:55The approach consists of driving the AI through a series of very separate and very distinct but fully autonomous vibe coding changes that build upon each other. And between each of them, there's a thorough human review to make sure that things haven't gone off the rails. I myself have had a lot of luck with this kind of technique and definitely encourage anyone experimenting with agentic coding to check that out. But Ben, have you tried anything like that? Yeah, I know you've been using agentic coding for a few weeks now. It's been fascinating to me just seeing secondhand, like learning from you what you've been doing with it.

10:29And as I mentioned earlier, like I just got into like vibe coding or agentic coding this week with Cursor. And it's frankly been an eye opening experience. You know, I still haven't figured out really how to maximize the use of things like rules to facilitate better agentic coding. Like this is kind of what Pete is describing in this article, right? It's like having more discrete chunks of work that have rules that guide it and a human overseeing the process. but it really does like mirror a lot of the approach I've taken with AI, just even outside of coding. You know, my focus has always been on breaking down complex work into a series of smaller steps that can easily be handled by individually purpose-built GPTs.

11:14And the reason I do this is because it improves consistency, but it also makes it much easier for you to inject human judgment into the loop to make sure that the output from one GPT is good enough for the input of the next. And there's a couple of other tips that really stood out to me that are things that I've already adopted. Clear context frequently. Like personally, I treat almost every interaction with a GPT as ephemeral. Like the moment that I'm done with it, it is deleted from existence where I never think of it again. Or if I feel like it's going off the rails, like I just start over from scratch with a fresh GPT prompt.

11:50And then he also mentioned using the right AI for each task. Like when you start breaking down these complex chains into discrete tasks, you might end up finding out that one model works for a certain task much better than another model, but then for a different task, the reverse is true. So, you know, I really, like this approach really makes it easy to set yourself up for success, but also experiment with the tools that are out there. Yeah. And this article does a really good job of kind of setting the perspective correctly of what you should be thinking when you open a tool like cursor or when you go to amazon queue in your command line when you open an ide now that has these capabilities you know you're putting on more of a product manager hat than you are a developer hat instead of focusing on the exact code lines that you're going to write you're instead focused on what is the purpose of what i'm building what's the intended impact and what do i need in order to get there.

12:48And then your goal becomes really as that product manager to rally all of that information together and get everyone on board. And when I say everyone, I mean all of these ephemeral little chats that you're about to have, like what Ben just described with the IDE. And another guide that are two sets of guides actually that go really well in depth with this. They're from Joffrey Hunter. They're on his website. We've actually linked them before in the download. So we'll be sure to link back to those as well if you haven't checked those out. It breaks it down into a really repeatable method, similar to what Pete Hodgson is kind of touching on here.

13:24And you start really by understanding what it is that you're building and writing very clear documentation for yourself and for the agent to use. Then you spend some time writing the bounds and the rules and the constraints of what you're building. In cursor, this might even mean writing rules for the project you're doing. And then finally, you're going to iterate. across all of those ephemeral sessions. You're going to reference that well-written spec. You're going to point to those strong, well-defined rules. And you're going to iterate until you get the results you want, reviewing every step of the way to make sure that you don't, of course, get slop-squatted, like we mentioned earlier.

13:59So it really kind of changes your whole perspective about what you're doing when you go into the IDE. And it really makes me wonder, to our listeners, are you experimenting with agentic coding right now too? I think a lot of people are trying it out for the first time and we want to hear about your experiences. I'm personally posting on LinkedIn every week and learning with others in the open about how to use things like Cursor, Amazon Q, Windsurf, all these other tools. So come learn with us and share your experiences too. Yeah, everyone's learning right now. And this is a good time to remind our audience that we have a Substack newsletter where we share a lot of these stories as well as a LinkedIn community where we're sharing this as well.

14:39So if you want to be the first to find out about all these great guides and get these tips delivered straight to your inbox, make sure you're subscribed to our sub stack. It's the best way to get all this news. So we've got a bonus story, too, and I wanted to include this one because I like to walk and I use crosswalks frequently. What is the story this week, Andrew? Okay, so this is a really weird one. Recently in Seattle, some of the crosswalks appeared to have been hacked. And people at first were not quite sure how this happened. But when you would go to the crosswalks on some intersections and press the button, instead of it telling you to wait or to cross, instead you were greeted by a deep-faked voice of Jeff Bezos or Elon Musk or other tech billionaires.

15:24And they'd be talking to you about all sorts of stuff. And this is a neat assortment of weird technology all coming together to create what was ultimately a viral moment. Because when you boil it down, it's a very simple vulnerability, actually, that they took advantage of. It turns out that a crosswalk is a lot like a router. It comes with factory settings, including a factory login over a technology like Bluetooth. So it really was probably just as simple as somebody who knew the model of the crosswalk and had their Bluetooth open and was able to connect with a default password and username to it.

15:58And then once they're in, They could upload, you know, an MP3 of what it should play when the crosswalk button is pressed. But what makes this a very impressive and very interesting usage of technology is ultimately they used modern tools like AI to deepfake these voices. They used very modern tools and the things that we're all worried about in terms of like misinformation to rapidly create a high impact message that then hacked something more than the crosswalk. it hacked social media because it became a viral story suddenly you had people going out to these crosswalks and recording it and sharing it on x and on blue sky and all these places look what this crosswalk is saying and then the whole world was talking about it now we're talking about it and that's ultimately what this kind of hacking in the wild is all about is making a message known using the tools available so quite a fascinating one what do you think ben yeah so first of all i want to just say put this out there like safety first like if you're going to do something like this please don't screw up the crosswalk so that like a disabled person can't use it like it's already dangerous enough being a pedestrian in many american cities but you know this is a tale as old as time like somebody's setting up a piece of digital equipment and just not bothering to change the default password like come on that's it's like security 101 like nobody should ever be doing that in this day and age.

17:22But the AI brings a very new twist to this. And you and I have been talking about this concept of code as art. As code gets more accessible to more audiences through AI, I really get the sense that we're going to see more emergencies of people just using AI and using code in unique ways that we had never really considered before. Again, going back to our slop squatting example, there's lots of unknown unknowns out there. AI is going to keep opening up new attack vectors. Stay tuned to Dev Interrupted. We're going to keep you on top of all the stories as they develop. So, Andrew, who's our guest this week?

17:58Oh, yes. This week's guest is a really interesting one. We have Russ DeSah, the CEO and co-founder of LiveKit. And he's really pulling out the curtain on where the future of AI is heading. And I don't mean just in the chatbots that we interact with now or even the agents that people are building and everyone wants to talk about. but instead interacting with us on our terms, being able to see, hear, feel, and understand from the real world using multimodal inputs. And the impact of this technology is profound. Talk about saving lives even on 911 calls. So stick around. You don't want to miss this one.

18:34Are your code reviews slowing you down? With Linear B Automations, you can transform the way your teams review code. With automatic AI-powered PR descriptions and code reviews, your developers get instant feedback on every pull request. Combine that with smart AI orchestration, and you can cut the noise and boost your productivity. LinearB AI flags bugs, suggests improvements, and keeps your team focused on what really matters, building great software. Say goodbye to review fatigue and hello to faster, higher quality delivery. Head over to linearb.io to learn more about incorporating AI into your code reviews.

19:12Hey, everyone. Joining us today is Russ DeSau, the co-founder and CEO of LiveKit, where they're building the nervous system for multimodal technology that interacts with the world through voice, vision, and real-time understanding. And what they're building at LiveKit, it's not just about its applications in AI. It's about how all modern systems are starting to sense, interpret, and act in real time. And to give our listeners a sense of this technology's impact just right off the bat, you know, right now, LiveKit delivers voice to millions of ChatGPT users. It even saves lives on 911 emergency calls, and we're going to talk a bit about that.

19:51It's also used in live streaming and robotic systems across the world. And in today's chat with Russ, we're diving into the major shift that's happening in software development that's currently underway and learning how your team can embrace those multimodal opportunities in your work. So Russ, welcome to the show. Thank you so much, Andrew. It's lovely to be here and hi to all of the listeners as well. It's great to have you. And you know, Russ, you and I had such a great chat and preparation for this. You know, your take on technology, it was really refreshing. It stuck with me. I've been really excited to bring you onto the show, especially this idea that you put in my head about how we're outgrowing a traditional request response model.

20:33And I want to dive into that a bit in our conversation today. But first, I think to benefit our listeners, let's zoom out a little bit because we've been talking about the shift into a real-time multimodal world. There is this huge shift happening with technology and not something that I foresaw when we started LiveKit, but then something that kind of ended up happening. For me, that was what we started to do with OpenAI and building voice mode with them for ChatGPT. And it was at that moment that I kind of like looked up a little bit and I was like, well, where does all this go? Right? Like, this is a really cool feature.

21:08I can talk to an AI, you know, using my voice, but where is all of this headed? And I think the thing that I realized back then, this was like August, 2023 is that the internet or let's say the web really was not built for multimodal real-time audio video, right? When you type into the browser, HTTP colon slash slash, like what you're typing in is you're typing in a protocol there. And that protocol, HTTP stands for hyper text transfer protocol. What you're doing is you're taking text and you're transferring it from one computer to another computer. It's not hyper voice transfer protocol or hyper video or vision transfer protocol, it's hypertext.

21:59And so, you know, just that word is telling, like the way transferring high bandwidth data, like audio and video over a network, it fundamentally requires a different approach, a different paradigm than transferring text over the network. The way that we interact with computers today, for the most part, is we open a browser, not every single use case, right? Like we use Zoom sometimes and we use Discord and all of that stuff. So it's, we are definitely transferring audio and video for some of these applications that we use. But predominantly, what you're doing is you're in a web browser, you're navigating to a web page, you're clicking into a form, you're filling out some fields, you're clicking submit or other buttons on that page.

22:43And that kind of paradigm or interaction model is the stateless web application model, right? So you click a button, some text is transferred to a server somewhere, that server receives that text and what you want to do, it looks up who you are in a database, it has some side effects that are generated. So it runs through some business logic, say that you're trying to get a reservation at a restaurant, right? And it's looking up like, are there spaces available at that restaurant for your party size? And then if there are, then it's going to put a record in the database for that time slot for that restaurant saying that you have the reservation.

23:22And then it's going to send back some text that gets rendered on that website that you're on saying, you have booked your table, show up at this time. And so that's not really a latency-sensitive application. And most of the applications on the web today are not. The way that you're going to interact with computers in the future, right, at a high level, what we're trying to build, I guess, you know, in society now is AGI, right? If we're trying to build AGI, like what is AGI? In my opinion, humans are tool builders, right? We create these things like hammers and nails and screwdrivers and planks of wood and all this stuff so that we can construct things and solve problems.

24:04And what we're building now is we're kind of building the ultimate tool, which is a tool builder. We're building ourselves, right? The mirror in some ways. And it's a computer that can behave like a human being, talk like a human being, maybe in its ultimate form is indistinguishable from a human being. I'm not trying to do any scare-mongering or anything like that. It's a computer that very much behaves like and mimics a human being. And when computers were kind of these not as intelligent machines. We had to create something like a keyboard and a mouse and adapt our own behavior. I know you're probably a bit younger than me, but I'd take a keyboarding class in school, right, where I need to learn how to type on this layout of keys.

24:53We have to adapt our behavior and learn how to give the computer information so that we can use it as a tool to help us, right? The bicycle of the mind. Yes. To use Steve Jobs' kind of phrasing, now we're almost creating the mind itself. And if you are going to build a computer that is as smart as a human, you no longer have to adapt yourself. The computer actually will adapt itself to you. And the way that you give information and communicate with other human beings is the way that you're probably going to communicate and behave or give information to that computer that is very human-like. And so humans use eyes, ears, and mouths.

25:35The computer of the future is going to use cameras, microphones, and speakers. Those are the equivalent sensors to kind of the humans, eyes, ears, and mouths. If you're going to build a computer that takes in natural human input and output in the same way or a similar way, you have to almost change the way that you build applications for that computer, right? That run on that computer. It's not where I click a button and then I wait and then like a database kind of looks up information and some actions are taken and then a response gets generated some number of seconds later. This is more similar to how you and I are communicating with one another right now.

26:17Like you are constantly listening to me. I have your attention. You're figuring out whether I'm done speaking or not, or whether you should interrupt with the next question. You're keeping this rolling context growing in your mind of everything that I'm saying, and you're committing some of it to memory, and you're going to come back to maybe something that I said or focus in on it. It's this constant connection, and the data is streaming to you throughout this entire time that we're talking, and you're processing it in your mind and also at the same time listening to new information that's coming to you from my mouth and through my movement.

26:53And so if you're going to build applications for that future, they're going to look just very different. So when we're talking about shifting how people are looking at, are we going to be using technology in the future? It's really a fundamental shift from getting out of the box that we put ourselves in, of using text and converting everything in our world to text, to talk to a machine, which is the traditional model, and more about opening it up and bringing, elevating the model or rather the computer up to our level, allowing it to participate in our own senses, have that same kind of understanding of the world.