The era of the Small Giant (Interview)

32:24Now, I think the only reason we're kind of stuck in this old process, because many of them are set in stone, but also because LLMs aren't quite smart enough yet. They still make stupid mistakes. Right. You still need a human in the loop and on the loop. Yeah, I mean, they're still a bit dumb and they get done with silly things and they do stuff. They've gone the wrong direction for a while. And I'm like, no, hang on a second. That's a great thought here, but let's get back on track. This is the problem we're solving. And you've side quested us. It's a fun side quest if that was the point, but that's not the point.

33:01But this is going to change, right? And this is one of the hard things is trying to put ourselves in the mind of like set of what it's going to be like in a year. And I think I've only been, you know, after us being able to play with LLMs for several years, it feels like i've i can i can feel the velocity of it now right because i've felt chat tpt3 four five claude code codex and now i can go oh okay that's what it feels like for it to get better and it's gonna keep getting better for a few more years and so it's kind of like self-driving cars right they're like not very useful while they're worse than humans but suddenly when they're safer than a human like why would you have a human yeah and i think it's the same with coding like all this process is to stop humans making mistakes we make mistakes like our mistakes are not special better mistakes they're still like we stuff in code we call security incidents and so i think I think as soon as the LLMs are twice as good, five times as good, ten times better at outputting good code, that doesn't cause these issues.

34:24We're going to start to let go of this concern, like these things, right? We're going to start to trust them more. Something I leaned on recently, and it was really with Opus 4.5. I feel like that's when things sort of changed because I'm with you on the trend from chat GPT or GPT-3 on to now and feeling the incremental change. I feel like Opus 4.5 really changed things. And I think I heard it in an AIE talk or at least in the intention of it, if it wasn't verbatim, was trust the model. Just trust the model. As a matter of fact, I think it was one of the guys, man, they were building an agent.

35:04and in the pro it was maybe agent layer layer agent something like that maybe borrowed something from your name layer code um i have to look it up i'll get the talk i'll put in the show notes but i think it was that talk and i was like okay the next time i play i'm gonna trust the model and i will sometimes like stop it from doing something because i think i'm trying to direct it a certain direction and now i've been like wait hang on a second and like this code's free basically it's just going to generate anyways let's see what it does worst case i'm like you know roll it back or worst case is like just generate better you know i mean like ultra think right you know what's the worst that could happen because it's going faster than that can anyway so let's see even if it's a mistake let's see the mistake let's learn from the mistake because that's how we learn even as humans i'm sure ellen's the same and so i've come back to this uh this philosophy or this this thought almost to the way you describe it like falling into this hole slipping in via gravity, not excited at first, but then kind of like excited because like, well, it's good in there.

36:08Let's just go. Just trust the model, man. Just trust the model. And it can surprise you. And I think that still gives me that like dopamine hit that I would have coding, right? When I was coding manually, you'd get a function, right? And you'd be like, ah, it works. Yes. And now it's like you've got the whole application, right? And you're like, ah, I just did a prompt and the whole thing works. That's right. Yeah. It's really exciting. And yeah, it's fun right now. And I mean, it's going to keep changing. This is just a bit of a temporary phase we're in now. But I think for many of us building software, we love the craft of it, which you can still do but also the like the making a thing is also one of the exciting bits of it and and the world is full of software still like you think about so many interactions you have with like government services or whatever not saying that they're going to adopt coding agents particularly quickly, but there is a lot of bad software in the world.

37:28And software has been expensive to build. And that's because it's been in high demand. And so I don't think we're going to run out of stuff to build. I think even if we get 10 times faster, 100 times faster, there's so much useful software and products and things and jobs to be done. Close this loop for me then. You said SaaS is dead. or dying. I'm paraphrasing because you didn't say or dying. I'm just going to say or dying. I'll parentheses. That's my parentheses. I'll add it to your thing. How is it going to change then? So if we're making software, there's still tons of software to write, but SaaS is dead.

38:07What exactly are we making then if it's not SaaS? I know that not all software is SaaS, but you do build something, a platform and people buy the platform. Is that SaaS? What changes? you mentioned interfaces like where do you see it going? I think we're moving and so this is the next level, the next kind of revelation I had was I started using the CRM. I was like this is cool, this is super fast, this is better than the other CRM and I can change it, cool, I'm doing some important sales work, I'm enriching leads and then I kind of woke up a few days later and was like why am I doing the work?

38:47like what's going on here i create an interface for me to use right why can't claude code just do the work that i need to do for me i know it's not going to be with the same taste that i have and i know it's going to make mistakes but i can have 10 of them do it at the same time and i it's not a particularly fun idea fully automated sales and what that means for the world in general. But it's the particular vertical where I had this kind of... Well, the enriching certainly makes sense for the LLN to do, right? The enriching is like, come on, I'm just the API, I'm copying things. And a lot of it is so manual still.

39:31And so the revelation was just waking up and then going, okay, Claude Code's going to do the work for me today. Like it does for software, It builds the software for me. I'm going to give it a Chrome browser connection. That's still an unsolved problem. There's a lot of pain in LLMs chatting to the browser, but there's a few good ones. And I'm going to let it use my LinkedIn. I'm going to let it use my X. And I'm going to connect it to the APIs that I need that aren't pieces of software, but are like data sources. where I can get enriched and search things. And then I just started getting it to just do it.

40:16And it was really quite good. It was slow, but it was really quite good. And that was a kind of, that was like that moment where we typed in, build this feature in clawed code, build this. But it was suddenly like this thing can just do anything a human can do on a computer. The only thing holding it back right now is the access to tools and good integrations with the interfaces, like the old software it still needs to use to do what a human does. And a bigger context window, and it'd be great if it was faster, but I can run them in parallel. So the speed's not a massive, massive problem. and in the space of a week I built the CRM and then I got Claude Code to just do the work but I didn't tell it to use the CRM, I just told it to use the database and I just ended up throwing away the CRM and now we have this little Claude Code harness that overrides the Claude Code system prompt sets up all the tools and gives it an Escalate database and I've just got like, I need to vibe code a new CRM UI, but I've just got like a database viewer that the non-technical team used to kind of look at the leads and stuff like that.

41:42It's just a kind of beekeeper kind of database viewer. And now Claude Code is just doing the work. We've only applied it there, But this is just like Claude Code is like this kind of little innovation in an agent that can do work for a long time. And we already know people use ChatGPT for all sorts of different things beyond coding, right? And so suddenly I think these coding agents are a glimpse of all knowledge work can be sped up or replaced. administration work can be replaced with these things now. These non-technical folks, why not just invite them to the terminal and give them CLI outputs that they can easily run and just up arrow to repeat or just teach them certain things that maybe they weren't really comfortable with doing before and now they're also one step from being a developer or a builder because they're already in the terminal and that's where Cloud's at.

42:47Yeah, I mean, that's what we've done now. I've seen some unexpected kind of teething issues with that. I think there's just the terminal feels a bit scary to non-technical people, even if you explain how to use it. Like when they quit Claude Code or something, they're just kind of like lost. They're like, oh my gosh, where did Claude Code? Yeah, and I was onboarding one of our team members. Like, open the terminal, and then I'm like, okay, we've got a CD. What if the terminal was just Cloud Code, though? What if you built your own terminal that was just that in Cloud Code? Yeah, that's what I think.

43:28I actually think that specific UI, whether it's terminal or web UI, it's kind of neither here nor there. But the magic is a thing that can access everything on your computer. Right, and they're doing that with, I think it's called Cowork. Have you seen co-work yet? So it's like, I haven't played with it enough to know what it can and can't do. I think I unleashed it on a directory with some PDFs that I had collected that was around business structure. And it was like an idea I had like four months ago with just, you know, a different business structure that would just make more sense, primarily around tax purposes, you know.

44:08and I was like hey you know revisit this idea I haven't touched it in forever and it was a directory and I think it went and it just did a bunch of stuff but then it was like coming with ideas I'm like nah those are not good ideas so I don't know like if it's like less smart than cloud code is in intent or whatever but it's kind of I think that's what they're trying to do with co-work but you know you could just drop them into essentially a directory which is what cloud code lives and it lives in a directory of maybe files that is a application or knows how to talk to the database as you said your crm does and they can just be in a cloud code instance just asking questions show me the latest leads yeah that could use a skill if you want to go that route or it can just be smart enough to be like well i have a neon database here uh the neon ctl cli is installed i'm going to query it directly maybe i'll write some python to make it faster maybe i'll store some of stuff locally and he'll do it all behind the scenes.

45:06But then, then it gives this non-technical person a list of leads. All they had to do was be like, give me the leads, man. You know? And then you, you mentioned enabling them as builders. I think it, it then it, it, it is a window into that because then when they want something. Oh, they get curious, right? They'll be like us. They're just going to be like, Hey, build me a report for this. Build me a web app for this. Help me make this easier. Yeah. You'd be surprised how easy that is. like help me make it easier is one of those weird ones and cloud code will also autocomplete and just let you tab and enter and i've noticed that those things gotten more terse like uh uh like maybe uh i think the last one i did was like that's interesting it was like super short it was like i like it comma space implement it was the was the completion for them i like it comma space implement it i was like okay is that how easy it's gotten now to like just spit out a feature that we were just riffing on that you know the problem you understand the bug we just got over and now your response to me to tell you what to say because you need me the human to get you back in the loop at least in in today's REPL is i like it implemented you know what i mean i found myself just responding with the letter Y.

46:24And a lot of the time it just knows what to do, right? Even if it kind of like is a bit ambiguous, you're kind of like, you'll work it out. So I think it's very exciting that Anthropic released this co-work thing because they've obviously seen that inside Anthropic, all sorts of people using Claude code. And when we think about, okay, so it starts there for non-coding purposes. But stuff is done with code and CLI tools and some MCPs or whatever, APIs. And then the user says, well, make me a UI to make this easier. So, for instance, I had to review a bunch of draft messages that I wrote. I was like, okay, this is kind of janky in the terminal.

47:09Make me a UI to do the review. And I just did it. And I think that's right where software is changing. because when the LLM is 10 times faster, I mean, if you use the Grok with a Q endpoints, right, they're insanely fast. It's going to be fast. Then if you can have any interface you want within a second, why have static interfaces, right? Yeah, I'm camping out there with you. What if everything was just in time? I think you can be. Like that interface. What if I didn't need to share it with you? Because you're my teammate. But what if you could do the same thing for you and it solves your problem?

47:57And you're in your own branch and what you do in your branch, it's like Vegas. And it stays there. It doesn't have to be said anywhere else, right? Like just leave it in Vegas, right? Right. What if in your own branch, in your own little world, as a sales development representative, for example, an SDR who's trying to help the team, help the organization grow and all they need is an interface. What if it was just in time for them only? And it didn't matter if it was maintainable. It didn't matter how good the code was. All it mattered was that it solved their problem, get the opportunity and enable them to do what they got to do to do their job.

48:31And you just take that and multiply it or copy and paste it on all the roles that make sense for that just-in-time world. It completely changes the idea of what software. It also completely changes how we interact with a computer and what a computer does and what it is for. I just love this notion that every user can change the computer can change the software as they're using it, as they like it I think that's a very essentially everyone's a developer yeah, I mean, it's the ultimate way to use a computer all the gates are down there's no geeky anymore if I want software the way I want software so long as I have authentication and authorization, I got the keys to my kingdom I want to make.

49:27And I think also the agents can preempt, right? I haven't tried this yet, but I was thinking of giving it the little sales thing. We have a little prompt where it says, where it's like, if a web UI is going to be better for the user to do this, review this, then just do it. So then instead of you asking it, You ask it to do some work and then it comes back and be like, oh, and I've made you this UI where I've displayed it all for you. Have a look at it. Let me know if you're happy with it. I mean, this is getting kind of wild, this idea. But it's kind of how we can... Think about how we communicate with each other as humans, as employees, right?

50:08We have back and forth conversations. We have email, which is a bit more asynchronous. us you know we we put up a preview url of something like i i think all of those communication channels can be enabled in in the agent you're chatting to and like i haven't liked this kind of like product companies of sell you know the initial messaging where people are sort of like digital employees right but some something like that's going to happen and and i don't think it's the The exciting bit for me is the human-computer interaction. It's right. And this is how it's kind of exciting in the context of layer code and why we love voice.

50:52It's like voice is this OG communication method. Whereas humans, we started speaking before we were writing. And it's kind of quite a rich communication medium. And it's a terrific way. If your agents can be really multi-medium, whether you're doing voice with them, text with them, they create a web UI for you, you interact with the UI with them, there doesn't have to be these strict modes or delineations between those things. Well, let's go there. I didn't take us there yet, but I do want to talk to you about what you're doing with layer code. I obviously produce a podcast, so I'm kind of interested in speech to text to some degree.

51:38because transcripts, right? And then you have the obvious version, which is like you start out with speech, you get something or even a voice prompt. What exactly is layer code? And I suppose we've been 51 minutes deep on nerding out on AI essentially, and not at all on your startup and what you're doing, which was sort of the impetus of even getting back in touch. I saw you had something new you were doing. And I'm like, well, I haven't talked to Damien since he sponsored the show almost 17 years ago. It's probably a good time to talk, right? So there you go. That's how it works out. Has your excitement and your dopamine hits on the daily or even minute by minute changed how you feel about what you're building with LayerCode?

52:19And what exactly are you trying to do with it? Well, we've talked a lot about the building of a company and the building of software now. and and i and i think founders today have that is that is a as important as the thing that they're building right because if you just head into your company and and operate it like you did even a few years ago right using no ai using all your kind of slow development practices using our slow sales and marketing practices, you're going to really get left behind. And so there is a lot to be done in working out and exploring what the company of the future looks like, what the software company of the future looks like.

53:13I'm very excited about the idea that we can build large companies with small teams. I think a lot of developers will. I mean, there is a lot of HR and politics and culture change that happens when teams get truly large and companies get truly large. And this is one of the kind of founding principles when we started our startup was, let's see how big we can make this with a small team. and that's very exciting because I think you can move fast and you can keep culture and keep a great culture. And so that's why we invest a lot of our energy into the building of the company. And what we build and what we provide right now and our first product is a voice infrastructure, voice API for building real-time voice AI agents.

54:13and this is currently a pretty hard problem we focus a lot on on the real-time conversational aspect and there's a lot of kind of um wicked problems in that right conversations are dynamic things and there's a lot of state changes and interruptions and back channeling and and everything that happens. And if you're a developer building an agent, it could be your sales agent, it could be a developer coding agent, and you want to add voice AI, there's a bunch of stuff you're going to bump into when you start building that. And it's interesting. We kind of see our customers. We can kind of predict where they are in that journey, right?

55:00Because there's a bunch of problems that you don't kind of preempt, and then you just quickly slam into them. And so we've solved a lot of those problems. And so with layer code, you can then just take our API, plug it into your existing agent backend. So you can use any backend you want, and you can use any agent LLM library you want, and any LLM you want. So the basic example is a Next.js application that uses the Vercel AI SDK. We've got Python examples as well. and you connect up to the layer code voice layer and put in our browser SDK. And then you get a little voice agent microphone button and everything in the web app.

55:45We also connect to phone over Twilio. And then for every turn of the conversation, whenever the user's finished speaking, we ship your backend, that transcript. You call the LLM of your choice. You do your tool calls, everything you need to do to generate a response. like you normally do for a text agent. Then you start streaming the response tokens back to us. And then as soon as we get that first word, we start doing that, converting that text-to-speech and start streaming back to the user. And so there's a kind of bunch of stuff you have to do to make that really low latency, make that a real-time conversation where you're not waiting more than a second or two for the agent to respond.

56:23So we put a lot of work into refining that. And there's also a lot of exciting innovation happening in the model space for voice models, whether it's the transcription or the text-to-speech. And so we give you the freedom to switch between those models, right? So you can try out some of the different voice models, some that are really cheap and got really casual voices, and some like 11 Labs that are much more expensive, but they're very professional, clean voices. And you can find the right for your kind of experience that you want. Trade-off, there's a lot of trade-offs, right, in voice between latency, price, quality.

56:57So we let users explore that and find the right fit for their voice agent. That is interesting. So Next.js, SDK, streaming, latency, you're meant to be the middleware between implementation and feedback to user? Yeah, we handle everything related to the voice, basically. and we let you just handle text like a text chatbot basically No heavy mp3 or wave file coming down. Yeah, and everything's streaming. And so a very interesting problem to solve because the whole system has to be on real-time. So the whole thing, we call it a pipeline. I don't know if that's a great name for it because it's not like an ETL loading pipeline or something, but we call it a pipeline.

57:53But the real-time agent system, our backend, when you start a new session, it runs on Cloudflare workers. so it's running right near the user who clicked to chat with your agent with voice and then from that point on everything everything is streaming so the microphone input from the user's browser streaming in that is then getting streamed to the transcription model in real time the transcription model is spitting out partial transcripts we send that partial transcript back to you so we can show you what you're saying if you want to show them that and then the hardest bit in this whole thing is working out when the user is finished speaking it's it's it's so difficult because we pause we make sounds we we we pause and then we start again and And conversation is such a dynamic kind of, it's like a game almost, right?

58:59Yeah. So we have to do some clever things, use some other AI models to help you detect when the users end speaking. And when we have enough confidence, like there's no certainty here, but we have enough confidence, we think the users finished their thought. Then we finalize that transcript, finish transcribing that last word and ship you that whole user utterance, like whether it's a word, a sentence, a paragraph, the user's spoken. The reason we have to kind of like, we can't stream at that point, right? We have to like bundle up this user utterance and choose an end is because LLMs don't take a streaming input.

59:38I mean, you can stream the input, but like you need the complete thing, the complete question to send to the LLM to then make a request to the LLM to start generating a response, right? There is no duplex LLM that takes input and generates input, output at the same time. Technically, what if you constantly wrote to a file locally or wherever the system is, and then at some point it just ends and you send a call that sends the end versus packaging it up and sending the whole thing once it's done? Like you incrementally, just line by line, it's maybe even like a... I don't know. I'm not sure how to describe it.

1:00:19That's how I think about it. What if you constantly wrote to something and you just said, okay, it's done, and what was there was the done thing? Yeah, yeah, yeah. So we can do that in terms of like, because we have the partial transcripts, yeah. So we can stream the partial transcripts and then say, okay, now it's done. Now make the LLM call. Then you make the LLM call. But interesting, sending text is actually super fast in the context of voice conversation, right? And actually the default example, this is crazy i didn't think this was this would work until we tried it but it just uses a webhook when the user finishes speaking the basic example sends your next js api root a webhook with the user text and turns out the webhook sending sending webhook with a few sentences in it that's like that's fine that's fast it's all the other stuff like then waiting for the llm to respond yeah that's actually not the hard part I mean you have maybe a millisecond-ish or a few milliseconds but it's not going to be a dramatic shift the way I described it versus how you described it and we've got a WebSocket endpoint now so we can kind of shave off that HTTP connection and everything but yeah then the big heavy latency items come in so generating an LLM response most LLMs we use right now they're optimized the ones we use in coding agents right they're optimized for intelligence, not really speed.

1:01:45Then when people optimize for speed, the LLM labs, they tend to optimize for just token throughput. Very few people optimize for time to first token. And that's all that matters in voice, is I give you the user utterance. How long is the user going to have to wait before I can start playing back an agent response to them? And time to first token, is that right? How long before I get the first kind of word or two that I can turn into voice and they can start hearing. The only major LLM lab that actually optimizes for this or maintains a low latency of TTFT is Google and Gemini Flash. OpenAI, most voice agents now doing it this way are either using GPT-40 or Gemini Flash.

1:02:37GPT-40 has got some annoying, the open AM points have some annoying inconsistencies in latency. And that's kind of the killer in voice, right? It's a bad user experience if it works, you know, the first few turns of the conversation are fast, and then suddenly the next turn, the agent takes three seconds to respond. And you're like, is the agent wrong? Is the agent broken? But then once you get that first token back, then you're good. Because then you send that text to us, you start streaming text to us, and then we can start turning into full sentences. And then again, we get to this batching problem.

1:03:12The voice models that do text to voice, again, they don't stream in the input. They require a full sentence of input before they can start generating any output. Because again, how you speak, how things are pronounced depends on what comes later. And so you have to then buffer the LLM output into sentences ship them off sentence by sentence to the voice model and then as soon as we get that first chunk of of 20 millisecond audio we chunk it up into we stream that straight back down web sockets from the cloudflare worker straight into the user's browser and can start playing the agent response friends you know this you're smart most ai tools out there are just fancy autocompletes with a chat interface they help you start the work but they never do the fun thing that you need to do, which is finish the work.

1:04:06That's what you're trying to do. The follow-ups, the post meeting admin, the I'll get to that later tasks that pile up into your Notion workspace looks like a crime scene. I know mine did. I've been using Notion agent and it's changed how I think about delegation, not delegation to another team member, but delegation to something that already knows how I work, my workflows, my preferences, how I organize things. And here's what got me. As you may know, we produce a podcast, it takes prep. It's a lot of details. There's emails, there's calendars, there's notes here and there. And it's kind of hard to get all that together.

1:04:39Well, now my Notion agent helps me do all that. It organizes it for me. It's got a template that's based on my preferences and it's easy. Notion brings all your notes, all your docs, all your projects into one connected space that just works. It's seamless, it's flexible, it's powerful, and it's kind of fun to use. With AI built right in, you spend less time switching between tools and more time creating that great work you do, the art, the fun stuff. And now with Notion agent, Your AI doesn't just help you with your work. It finishes it for you based on your preferences. And since everything you're doing is inside Notion, you're always in control.

1:05:13Everything Agent does is editable. It's transparent. And you can always undo changes. You can trust it with your most precious work. And as you know, Notion is used by us. I use it every day. It's used by over 50 % of Fortune 500 companies. And some of the most fastest growing companies out there like OpenAI, Ramp, and Vercel. They all use Notion Agent to help their team send less emails. cancel more meetings, and stay ahead. Doing the fun work. So try Notion now with Notion Agent at Notion.com slash changelog. That's all overcase letters, Notion.com slash changelog to try your new AI teammate, Notion Agent, today.

1:05:51And we use our link. As you know, you're supporting your favorite show, The Changelog. Once again, Notion.com slash changelog.

1:06:03you chose Tyscript to do all this we're pretty set on cloudflare workers from day one and it just solves so many infrastructure problems that you that you're going to run into later on I like I don't think we'll need a devops person ever it's such a that's interesting it's such a wonderful there are constraints you have to build to right you're using V8 JavaScript browser JavaScript in a Cloudflare worker right tons of node APIs don't work there is a bit of a compatibility layer you do have to do things a bit differently but what do you get in return your your application runs everywhere 330 locations around the world there is essentially zero cold start Cloudflare workers start up in the time while the while the ssl negotiations happened happening the worker has already started and you have very few limitations to your scaling extremely high concurrency every instance is very kind of isolated that's really important voice as well there's a there's kind of often quite big spikes like 9 a.m everyone's calling up that everyone's calling up somewhere that's got a voice agent and asking to kind of book an appointment or something you get these big spikes you want to be able to kind of scale and you need to scale very quickly because you don't want people waiting around and if you throw everyone if you throw tons of users on the same system and you start kind of overloading it then suddenly people get this problem where the agent starts responding in three seconds instead of one second and it sounds weird but um yeah cloud flag gives you an incredible amount of that for no effort and i think compared to kind of lambda and stuff it's also pretty like the interface like it's just a http interface to your worker there's there's nothing in front and you can do web sockets very nicely and there's this crazy thing called DurableObjects, which I think is a bad name.

1:08:19And it's also kind of a weird piece of technology, but it's a, you know, little JavaScript runtime that is persistent, basically, and has a little SQLite database attached to it. And it is, I don't know what the right word of it it's kind of like it's not the right word for javascript but it's basically think of it like thread safe so you can have it take a bunch of web socket connections and do a bunch of sql writes to its sql light database it has attached and you can you don't have to do any kind of special stuff dealing with concurrency and atomic operations so you can you know the simple example is just have like a implement a rate limiter or a counter or something like that you can do it very simply in durable objects you can have as many durable objects as you want each one of them has an SQLite database attached to it you can have 10 gigabyte per one and you can then kind of like do whatever you can kind of shot it however you want right you could have a durable object per customer that tracks something that you need to be done real time you could have a durable object per chat room as long as you don't kind of like it does have a set amount of computer durable object but you can use it for all sorts of magical things and i think there's a real under-known thing that cloudflare has that um you know coming from pusher it's like it's like the kind of real-time primitive now and a lot of the stuff we'd reach for something like pusher durable objects especially when you're building a fully real-time system is really grateful.

1:10:03Yeah. You chose TypeScript based on Cloudflare workers, it sounds like, because that gave you 330 locations across the world, durable objects, great ecosystem, no DevOps. For those who choose Go, or I don't think you'd choose Rust for this because it's not the kind of place you'd put Rust, but Go would compete for the same kind of mind share for you. How would have the system been different if you chose Go? Or can you even think about that? I haven't actually written any go. So I don't know if I can give a good comparison. But from the perspective, what we do have out there is similar real-time voice agent platforms in Python.

1:10:48And I think because a lot of the people building the models, the voice models, then built coordination systems like LayerCode for coordinating the real-time conversations. Python was the language they chose. And I think what's more important is the patterns rather than the specific languages. And so we actually, we wrote the first implementation with RxJS. And that has an implementation in most popular languages. I hadn't used it before, but we chose it because it was for stream processing. thing it's not really for real-time systems but it gives you subjects channel these kind of has its own names for these things but basically it's like a pub subby kind of thing and then it's got this kind of functional chaining thing where you can then kind of pipe things and filter things and filter messages and split messages and things like that and that did allow us to build the first version of this kind of quite dynamic system we we didn't touch on it but interruptions is this other really difficult dynamic part where whilst the agent is speaking its response to you if the user starts speaking again you then need to decide in real time whether the user user is interrupting the agent or are they just going yeah and like agreeing with the agent oh gosh or are they trying to say oh stop i bet that's a hard problem to solve we have to still be transcribing audio even when the user's hearing it and we got to deal with background noise and everything and then when we're confident the user is trying to interrupt the agent, we've then got to do this whole kind of state change where we tear down all of this in-flight LLM request, in-flight voice generation request, and then as quickly as possible start focusing on the user's new question.

1:12:47And especially if their interruption is really short, like stop. Like suddenly you've got to like tear down all the old stuff, transcribe the word stop then ship that as a new llm request to the back end generate the response and then get the agent speaking back as as quickly as possible you know and that and not it's all happening down one pipe as it were at the end of the day right it's like audio from the browser microphone and then an audio replaying back and we would have bugs like you'd interrupt the agent But then when it started replying, there'd still be a, you know, a few chunks of 20 millisecond audio from the old response snuck in there.

1:13:27Or, you know, the old audio would be interleaved with the new audio from the agent back. And you're kind of in the, you know, audacity or something, some audio editor trying to work out like, what's going, why does it sound like this? and you're like rearranging bits of audio going, ah, okay, the responses are taking turns every 20 milliseconds. It's interleaving the two responses to try and work out what's going on. Real pain in the ass to the bar. When you solve that problem with the interruption, do you focus on the false examples, the true examples? So do you like have these, if it is an interruption, you can tell it's an interruption by these 17 known cases.

1:14:11How do you direct that interrupt? It really depends on the use case. How you configure the voice agent really depends on how the voice agent is being used, right? Like a therapy voice agent needs to behave very differently than a vet appointment booking answering phone agent. Yeah, a lot of dogs barking in the background. Yeah, there's that. And when we call that audio environment, that's often an early issue users have. Interesting. They're like, well, my users call from cafes, and it kind of really misunderstands them. And big problem with audio transcription, it just transcribes any audio it hears, right?

1:14:55So if someone's talking behind you, it doesn't know that, the model doesn't quite know that's a relevant conversation, it's just transcribing it all. But if you imagine the therapy voice agent, it needs to actually not respond too quickly to the user. It needs to let the user have long pondering thoughts, long sentences. Big pauses. Emotional. You know, like maybe tears or crying or just some sort of, you know, human interrupt, but it's not a true interrupt. It's something that you should maybe even capture into parentheses. And so you can choose a few different levels of interruption, right? You can just interrupt when you hear any word.

1:15:38By default, we interrupt when we hear any word. That's not a filler word. So we filter out, um, uh, things like that. And then if you need some more intelligence, you can actually just ship off the partial transcripts to an LLM in real time. So let's say the user's speaking and starts interrupting agent. every kind of word you get or every few words you fire off a request to gemini flash and you say here's the previous thing that the user said here's what the agent said here's what the users just said yes respond yes or no do you think they're interrupting the agent and you get that back in about 250 300 milliseconds and you just you just can't as you get new transcripts you cancel the old ones you just constantly try and make that request until the user stops speaking then you get the response from that and then then you can kind of make quite an intelligent decision but these things feel very hacky but they actually work very well well the first thing i think about there is that gemini flash is not local so you do have to deal with an outage or latency or downtime or in in the in clod i would say cloud web's case most recently a lot of downtime because of uh usage like really heavy usage in the last two days i've had more interruptions on web than ever and i'm like the rouse because yeah it's the it's the ralph effect and i'm like okay cool i get it you know i'm not upset with you because i mean like i empathize with how how in the world you scale those services.

1:17:15So why does your system not allow for a local LLM to be just as smart than Gemini Flash might be to answer that very simple question? Like an interrupt. It's a pretty easy thing to determine. I think smaller LLMs can do that. Gemini is just incredibly fast. I think because of their TPU infrastructure, they've got an incredibly low TTFT, time to first token, which is the most important thing. But I agree, there are smaller LLMs. And actually, I think probably maybe one of the grok with a Q llamas actually might even be a bit faster. We should try that. But you make a point about reliability. People really notice it in voice agents when it doesn't work, right?

1:18:03And especially if a business is relying on it to collect a bunch of calls for them. And so that is one of the other helpful things that platforms like ours provide. Or even just cost. I imagine over time, cost. I mean, right now you're probably fine with it because you're innovating. And maybe you're finding out like customer fit, ability, reliability, all those things. And you're sort of just in time building a lot of this stuff. And you're maybe okay with the inherent cost of innovation. But at some point, you may flatten a little bit. And you're like, you know what? If it had been running that locally for the last little bit, we'd have saved 50 grand.

1:18:38I don't know what the number is, but the local model becomes a version of free when you own the hardware and you own the compute and you own the pipe to it. And you can own the SLA latency to it as well and the reliability that comes from that. And there's some cool, there's a new transcription model from NVIDIA and they've got some voice models as well. And so there was a great demo of fully open source local voice agent platform That was done with Pypecat, which is the Python coordination agent infra open source project that I was mentioning. And they've got a really great pattern. They have a plugin pattern for their voice agent.

1:19:17And I think that's the right pattern. And we've adopted a similar, and other frameworks have done that. We've adopted a similar pattern for ours when we've rebuilt it recently. And the important thing is the plugins, you know this this is kind of the the plugins are independent things that you can test in isolation that was that was the biggest problem we had with rxjs is the whole thing was kind of you know those mixing um kind of audio mixing things where you have like cables going everywhere it was kind of like that right with rxjs subjects going absolutely everywhere it was kind of hard it was hard for us as humans to understand it was the kind of code where you come back to a week later and go what was what was happening here yeah and and things like you know oftentimes we'd end up writing code where the code at the top of the file was actually the thing that happened last in the execution of it and you know basic stuff like that just because that's how the rx.js was kind of telling us to do it or kind of like guiding us and how we had to initialize things but that was one of the key things we we moved to a plugin architecture we moved to a very basic we've got no kind of rxjs style stream processing plugin it's just all very simple javascript with async iterables and we just pass a waterfall of messages down through plugins and it's so much better and we can take out a plugin if we need to and we can unit test a plugin and we can write integration tests and mock out plugins up and down.

1:21:01And we're about to launch that and that's just such a game changer. And interestingly, tying back to LLMs, we ended up here because with the first implementation, we found it hard as developers to understand the code we'd written. The LLMs were hopeless. They just could not hold the state of this dynamic, crazy multi-subject stream system in their head. Well, context was everywhere, right? Like it was all, it was here, it was there. Yeah. Even if you, I would do things like take the whole file. I was like copying and pasting files into ChatGPT Pro, being like, you definitely have all the context here, fix this problem, and they still can fix it.

1:21:43Solve the problem. Yeah. Fix it. And part of the problem was that complexity. I mean, not having the ability to test things in isolation then meant we couldn't have a kind of TDD loop, whether it was with a human or with an agent. And because we couldn't use agents to add features to this platform, to the core of the platform, it was slowing us down. And so that's when we really started to use coding agents, Claude Code and Codex, like really properly and hard. is we were like, I spent like two weeks just in code coding codecs. And the mission was, if I can get the coding agent to write the new version of this, it was kind of not even a refactor.

1:22:36It had to be rewritten. Start from scratch, yeah. First principles. Then it will, by virtue of it writing it, it'll understand it. And then I'll be able to use coding agents to add features. and I started with literally the API docs for our public API because I didn't want to change that and the API docs of all of the providers and models we implement with like the speech-to-text and text-to-speech model provider endpoints and just some ideas about I think we should just use a simple waterfall pipe like pass messages through the plugins and that experience was really interesting because it felt like molding clay because I did care about, I really cared about how the code looked because I wanted humans as well as engineers.

1:23:26The agents aren't quite good enough to build this whole thing from a prompt, but I think they will be in a year or two, right? But it did an okay job and it needed a lot of like re-prompting, refactor this, re-architect this, but it felt like clay in one sense because it was like, and you mentioned this earlier, like you can just write some code. And even if it's wrong, you've kind of learned some experience. Yeah. I was able to just say, write this whole plugin architecture and do it. And it would do it and I'd be like, oh, that seems a bit wrong. That's hard to understand. I was like, write it again like this, write it again like this.

1:24:05And I suddenly got that like experience of throwing away code because it hadn't taken me weeks and weeks to write this code. It had taken me 10 minutes and I was there. So it doesn't matter. Just threw it away. And you still have your chat session too. So even if you had to scroll back up a little bit or maybe even copy that out to a file for long-term memory, if you needed to, you still have that there as a reference point. Yeah, I find myself doing similar things where it's just like, just trust the model, throw it away and do it again. If you need to learn the mistake, go down the wrong road for the learning.

1:24:40Make the prompt better. And it did a terrific job. And then the bit that really got it over the finish line was then I said, I gave it this script that we used to have to do manually to test our voice agent. You know, it's like connect to the voice agent, say this to the voice agent, tell it to tell you a long story. Now interrupt the story. You shouldn't hear any leftover audio from the long story. Like all these things. There's like 20 different tests you had to do. I gave it that script and I was like, write the test suite for all of these tests. and then it did and I gave it all these bugs we had in our backlog I was like write tests for this and I just started doing TDD in our backlog and it was great then I was like write a bunch I did like a chaos monkey thing I was like write a bunch of tests for like crazy stuff the users could do with the API found a bunch of bugs and issues security issues and then it kind of got it working had a bunch of unit tests and I was still having to kind of do a bit of manual testing.

1:25:44And then one day I was like, oh, you know what? I really want like a, no one's made an integration test thing for voice agents. There are a few observation platforms, observability platforms and eval platforms. So I was like, I just, I want it to simulate conversation. And, you know, so it's so like, that's part of the magic is trying something that you're like, this is a pain in the ass to build. Or like, how is this even going to work? Well, I just got it to build it and I recorded some WAV files of me saying things and I gave it to them and I was like, make an integration test suite for this and feed the WAV files like you're having a conversation and check the transcripts you get back.

1:26:22Wow. And it did a great job. And then it was actually able to fully simulate those conversations and do all the tests. And then that, I mean, we've got these practices like TDD, which are going to hold value, right? It was so valuable for the model, for the agent to be running the test, fixing the test, running the test, fixing your tests. And that feels a bit like magic when you get that working. So much to cover in this journey. Wow, I'm so glad we had this conversation. I kind of feel like a good place to begin to end, not actually end, is back to this idea that is on your about page. And I just got remarkable because I love to write and I really hate paper.

1:27:10because this thing has Linux on it. And I wrote an API that I now API to my remarkable pro tablet. So amazing. I'm loving this little hacking thing. You need to be able to code codex from your tablet. That's next. I just got it. I just got it. So like, that's the next thing is I'm going to have this. It's a little playground for me basically, but it's, it's real time. So if you see me looking over here, writing audience or even you, Damien, I'm not, not paying attention. I'm writing things down. And the one thing I wrote down earlier from your about page was the era of the small giant, which you alluded to, but you didn't say those exact words.

1:27:44And the reason why I think it might be a good place to begin to end is I want to, I think you might be able to encourage the single developer that maybe in the last couple months, they've just begun to touch and not resist falling into this gravity hole or however we describe this resistance we've had as developers loving to read our own code and code review and all the things as humans. and now to not resist as much or if at all and just trust the model to give them this word of encouragement towards, hey, you're a single developer and in your case, Damian, you don't need a DevOps. It's not that they're not valuable or useful, but you chose a model, a way to develop your application to solve your problem that didn't require a DevOps team.

1:28:30Give them that encouragement. What does it mean to be in this era of the small giant world? I think the hardest thing is our own mindset right i just found this with coding agents you you start off putting in things where you kind of have an idea you know what to expect out of it and then you start just putting in stuff that just seems a bit ridiculous and ambitious and oftentimes it fails but more and more it's working that's a very magical feeling and and it's a very revealing kind of experience and so I think we can all be more ambitious now. I think we all, and especially as engineers, we know how the whole thing works, right?

1:29:21There is a lot of power. Everyone's been given with Vibe coding, being able to Vibe code. There are a lot of security issues, I think, that will be solved over time. But as engineers, we have the knowledge to be able to take things fully through, deploy things, scale them, fix the issues that the LLMs can't still get stuck on. But we can do so much more now. We can be so much more ambitious now. And I think the thing that, you know, everyone should, every engineer should be doing now is not only trying out Cloud Code and Codex and doing something new and fun. I mean, the great thing is it's so, there's so low risk there's it's so easy to do that you can build something ridiculous and fun that you've always wanted to do right like yeah you can just you can just and you can build something for for a friend for your wife it's like that that's really exciting and i think this um Ralph Wiggum thing, very kind of basic idea is give a spec.md, a to-do.md, just an ambitious task or a long list of tasks in a markdown file.

1:30:45And you run a shell script and all it does is it just says to Claude Code, do the next bit of work. When there's no more work to do, return complete. And the shell script just greps for complete. And if it hasn't seen that word in some XML tags, complete, it just calls Claude Code again. Yeah. And like many of these things, it seems like a terrible idea. It seems ridiculous. But it is also incredible what it can do. and so I think that's probably one it like to feel what the future is going to be like I feel like you write down something very ambitious in a markdown file or transcribe an idea you have that you've been thinking about for a while and you set a Ralph Wiggum script off in it and you just go for a long walk or go and have lunch and when you come back I mean it's a very exciting feel.

1:31:44And as a developer, it's very fun because then you get to go through all this code and be like, why did it do that? And you're like, oh, that's pretty smart that it did it like that. Okay, that was quite a good idea. And then it messed up this bit. But I just feel like that's a very exciting experience. Very cool. I definitely agree with that. I'm looking forward to writing that to do.md or spec.md and just going for that walk because I haven't done it yet. I've only peeked at some of the videos and some of the demos, but I haven't tried the Ralph Wiggum loop yet. I'm going to post on X a one-liner Ralph as well because I think you can just, then you can just copy and paste and go.

1:32:32There's no blog post to read. Yeah, well, I feel like with everything, I want to make it more ceremonious because I want to, it's not because it needs to be, because I want to know. I want to give myself space to think of something that I think will be challenging for me even, you know, and then give it to the thing and go away, like you said, and come back happy. I want to save space to do that when I can give it full mind share versus the incremental 20 minutes or 10 minutes or whatever it might be that I have available to give it. I kind of want to give it a bit more ceremony not because it deserves it because i want to actually do it for myself um so i'm just in this like constant learning scenario it's uh it's pretty wild to it's a pretty wild era to be a developer and to be an enabled developer you know like these non-technical folks that may get introduced to a terminal like thing that basically just clawed in a directory where they can ask questions and get a just-in-time interface that matters to them only like that's a really really really cool world to be in and it doesn't mean that software goes away just means there's gonna be a heck of a lot more of it out there and i do concur that maybe maybe code review doesn't matter anymore maybe it won't in a year maybe it won't in six weeks i don't know how many weeks will take let's uh let's truly end with this what's over the horizon for you what's over the horizon for layer code what what is coming so the show will release next Wednesday.

1:34:05So you've got a week given that horizon and no one's listening now. It's a week from now. What's on the horizon for you that you can give us a peek at? Is there anything? We are working really hard to bring down the cost of voice agents. There is a magic number of$1 an hour for running a voice agent where suddenly a huge, huge number of use cases open up, whether it's consumer applications, gaming. There are so many places where voice AI will be super valuable, super fun, and isn't implemented yet. And with the choices we made being on Cloudflare with the system we've built, we're going to be able to bring out the lowest cost platform.

1:34:56I'm very excited for that. and most of all, very excited just to see voice AI everywhere. Voice AI, voice is just such a wonderful interface, right? I find myself dictating all the time to code code and you can kind of get out your thoughts so much better. And I'm excited to see how many applications we can enable adding voice AI into their application. and then we get an insight into the future of voice AI as well with the companies that are working for it. A lot of them are startups and they're building some crazy, crazy new things with voice AI on our platform. So there's going to be some amazing stuff with voice coming out this year.

1:35:45What's the sweet spot for Layer Code right now that you can invite folks to come and try? Well, the great thing is we've got a CLI, single command you can run, and you'll get a Next.js demo app, all connected to your layer code voice agent, and you can get a voice agent up and running with a minute. So it's super fun, worth trying. And then from that point, you can use Claude Codex and just start building from there. Well, friends, right here at the last minute, the very last question, Damien's internet dropped off or something happened. I'm not sure, but it was a fun conversation with Damien.

1:36:28Kind of wild to be talking to somebody 17 years later after being one of the first, if not the first, I'm pretty sure the first sponsor of this podcast. What a wild world it is to be this deep in years and experience in history in software and to just still be enamored by the possibilities. these. I hope you enjoyed today's conversation with Damien and we'll see you next time. Well, friends, the YOLO mode philosophy is out there. The code review is a bottleneck, maybe non-existent. SaaS may be dying or dead. It's time to trust the model, building a CRM just in time. What kind of world is this we're living in?

1:37:11Did you think that the beginning of 2026 would be this kind of year? Now, I know if you're listening to this podcast at the very end and you're a Spotify hater, well, guess what? AI is here to stay. You should read the tea leaves. And that's just me being honest. But seriously, you can't deny the impact that AI is having. Everyone is talking about it. Everyone is using it. And those who aren't, well, we'll see. I know our friends over at Depot, our friends at Notion, and our friends at Tiger Data, they're all loving this podcast, just like you. Check them out, depot.dev, Notion.com slash changelog.

1:37:47and of course, TigerData.com. Much thanks, much love. Appreciate the support. But hey, friends, this show's done. This show's over. I'm glad you listened. We'll see you again real soon.