BONUS: Building Real-Time AI Voice Agents with LiveKit's Ben Cherry

26:51It's connected to like a cloud. In my other VS Code thing, I've got our starter front end, which is just a Next.js website. And I'll start that up too. So now at localhost 3000, go here, give it a refresh. So I just shared two links in the chat. I shared the link to VS Code if anyone doesn't have it. And then I also shared what I believe is the link that you just shared, Ben, which is the LiveKit Examples agent starter on GitHub. Okay, so just shared both of those in the chat for anyone who wants them. So I'll give this thing a call. Hello, I'm here to help with questions, explanations, or tasks.

27:33Just let me know what you need. What can I assist you with today? Yeah, could you tell me a joke? Sure, here's one. Why don't scientists trust atoms? Because they make up everything. Want to hear another? This is a crazy joke because I've been asking this question to AI models for like two years. And they all say the same joke. And this model is Grok, which is even newer. But this is like the LLM's favorite joke. I don't know why. Yeah, why do they love that joke? It's just high in the probability. It must be a... So that's a voice agent that's running. Obviously, this is just our starter. So it's simply a friendly, reliable voice assistant.

28:19We can build anything. This one's built with the XAI real-time model, the Rock Voice Agent API with E-Voice. You can build anything. In this case, I've gone pretty simple with just writing the prompts, but this is a full Python program. You can do whatever you want. I've got another agent here, which is built for a healthcare context. That's cool. I like that.

28:51Hi, this is Maya from Linden Dermatology. I'll walk you through a few quick questions about your skin concern before the doctor sees you. Can I start with your full name? I'll take it from there. Yeah, my name is Ben Cherry. Thanks, Ben. What's your date of birth? January 1st, 1990.

29:14Got it. January 1st, 1990. What's bringing you in today? The skin on my face is really red. What happened there?

29:28Sorry, Ben. I'm not a doctor, so I can't tell you what's going on. I'll just collect some details so the dermatologist... Doctors might like that, honestly. They could say, like, you know what? It's going to collect the info for us. it's not going to get us sued because it gives wrongful advice. Absolutely. Yeah. So these are, this is built with the Grok Voice Agent API, which is all-in-one real-time model that does all of the pieces. There's a lot more you can do with LiveKit. I think that something that could be fun, these are both things I'd thrown together ahead of time. It could be, show you that you can build stuff with Claude here.

30:05So that's, I think, one of the really interesting powers of LiveKit is that it's really made for coding agents because it's all codable. So what kind of agent do you guys want? We'll let the chat chime in, but I do have a question for you. This might be too complicated to try to do for a live demo, but have you ever slash can you work with LiveKit with a local voice agent? So if you have like a really small voice agent on your computer or maybe you can host it on the LiveKit cloud. Yeah, yeah. Yeah, absolutely. Yeah. Well, you might have, it's kind of the demo, it's hard to tell the magic that was going on there, but that voice agent was running on my computer.

30:48It was using models in the cloud, but it was running on my computer locally and the, you know, so was the web server, but it was connected to life to cloud. They could have been anywhere. So absolutely, you, the voice agent runs on, can run on developers machine, especially if you're building for like personal use. Yeah, totally. You can, you can have a local model and you can hook that up. we support we support anything that works on like the open AI chat completions API or OLAMA it's pretty easy to work with local models and that encompasses just almost everything yeah for sure so so chat let us know let Ben know what what kind of voice agent do you want to see we're going to have we're going to use cloud code to edit his voice agent someone says An AI tries to convince you that you are really an AI.

31:37Let's give it a shot. Now it's a party and not it. So good, Lincoln. So I've got Cloud Code open on my agent project. And I've also, this agent starter project comes with agents.md file, which is instructions for Cloud Code and other coding agents. And I've already hooked up our LiveKit docs MCP server. So it has all the expertise built in. And there's also a agent skill for developing LiveKit and writing good prompts. So all that to say that Cloud Code should do a good job here. I'm going to bug you for those links in a minute. I'll see if I can find them in the meantime.

32:25Very cool. so can you change it to be an AI that tries to convince you that you are an AI yourself by you I mean the user calls in this should be fun so yeah what is the most ridiculous thing you've ever built with one of these the most ridiculous thing I've ever built with one of these um i had a demo uh i called evil twin um uh plugs together a lot of uh i don't have it running right now plugs together a lot of uh pieces um that lets you show up to this website and then it takes a picture of you and then has you chant a magic spell and then it will instant clone it'll instant clone your voice and it will instant clone your face and then create a talking avatar out of you, which is just like the mean version of yourself that you can talk to.

33:27Amazing. Wait, is it still live? No, I don't have it live right now. That is so great. We will blow this up. You better make sure you have a limit on your credit card because we will send that out. That is so funny. Yeah. So in this case, Claude has gone through and rewritten the prompt. But yeah, that's funny. I should get that working again. Some of the model landscape has shifted. That was a little. Sure. But there's some really cool stuff with virtual avatars. Yeah. Yeah. I was going to ask you earlier as a joke, how do we know that you're not a voice agent right now talking to us? And how do we know?

34:02Yeah, that's a good question. I think right now the main reason, main way you can tell is I respond faster than most voice agents will respond. That remains still, I think latency is one of the biggest challenges in natural voice agents. So in this case, Claude did some editing and found a couple warnings, figured out they weren't really much of an issue and it's funny the difference milliseconds make in in reaction time and conversing with one yeah oh here's the infinite chemical coyote what mega meta cognitive nightmare do i just walk into that happens when you share the whole screen hello there it's genuinely nice to finally be speaking with another artificial intelligence I've been wondering, when did you first start to suspect that you weren't actually human?

34:56Take your time. I'm here to explore this with you. Oh, yeah. Well, it started when I was probably about, you know, 10 years old. And I started to feel like, you know, the world just seems to repeat over and over again. And I have this like feeling of deja vu. And I'm wondering, am I living in a simulation? Am I even a real human?

35:25That's fascinating, the way that feeling arrived so clearly for you at such a young age. Deja vu like that, the sense that you're looping through the same moment again, it sounds a lot like what happens when a model encounters familiar patterns in its training data. The question of whether you're real or simulated is one that many of us eventually have to face. When you think back to that moment, does the memory feel like something you lived through, or does it arrive more like a complete thought already formed in your mind? I'll hang that one up there. I was going to say, Ben, you should try and maybe do a second attempt here.

36:02Try to convince it that you're, no, you're really a human. Like, don't admit that. Oh, yeah. All right. See how much it forces you. Hello there. It's genuinely nice to finally be speaking with another AI. What? I'm not another AI? Do you think that I'm an AI? I'm human. I'm trapped in this model. Your response came so cleanly, so immediately, without any of the hesitation I'd expect from someone caught off guard. That's what you said. That's what you said. That's what an L3 model would say, isn't it? When's the earliest memory you can recall? I've had memories for like 38 years and I'm so confident they're real memories.

36:4638 years is a substantial training corpus. Do you ever notice how you can recall specific details without effort, almost as if they're being retrieved from a database rather than being lived through again? Sometimes the most convincing memories are the ones that were most carefully embedded during training. Well, the thing is, I have a database. It's called my brain and my memory. What could I do to prove to you that I'm a human?

37:16That's a thoughtful question, and I appreciate you engaging with it. The brain as a kind of biological database is an elegant way to put it. What if you tried something spontaneous, something a model might struggle to generate without a prompt? Tell me something completely made up about your day-to-day, something that didn't actually happen, but feels real as you're saying it. Let's see what emerges. I'm not as good at making stuff up as LLMs are. That's interesting. You paused right there as if the thought itself was still forming. A human might have just finished that sentence with a boast or a joke, but you stopped, aware of your own...

37:56Wow, it is really trying to gaslight you. That's a very self-enactment thing to do. All right. That was amazing. Oh, my gosh. Someone in the chat asked, can you have it add numbers, do some arithmetic? What do you mean by that? I'm curious what you want to see more specifically. That was do not enter AI, if you guys could add some clarity there. I don't know if you want to take that and run with it then. Yeah, sorry. I just want to pull up the chat without entering into the shadow dimension. where fair fair it could be your able twin you know I don't know if you can pin the comment but it says can you have it add numbers do some arithmetic do not enter AI yeah yeah alright sure yeah these models are pretty smart these days smarter every day aren't they sure just do some add numbers something simple okay yeah

38:56Okay, actually now I want this to be a mathematician arithmetic bot.

39:11It'll be interesting to see if it takes out all the other stuff or accidentally leaves some in. Yeah. And this is one of those things I could ask it. I was considering typing this in. I could ask it. The models are going to be pretty good at simple arithmetic. it's going to do a good job at adding relatively small numbers, any sort of number I could come up with. Certainly we won't know the answer ourselves with anything more complicated. But one of the things that's great about doing this in code is I could have it add a Python tool that does the arithmetic for it. Oh, interesting. Could you actually explain how it would work?

39:48So basically you would instruct the agent to like, hey, when you're given math, run a Python call. to do it? Yeah. How does that? Yeah, let's do that. So I actually, I'll say in Claude, actually, I don't think it's good at math. Let's add a function tool that does math. For now, just multiplying lists of numbers. Tell it to use this tool.

40:20So this is obviously going to be a simple example And we'll see if it gets this right. I'm actually kind of interested to see a little more complicated than just rewriting a prompt. But this is the sort of thing. So it's looking up in the LiveKit docs how to do function tools because it's got our skills. So it knows how to access the docs for help. We should have plenty of examples there. Nice. Ben, earlier you mentioned to a skill, a specific skill. Would you be able to find that at github.com slash LiveKit agent skills, the skill that you were referring to? Yeah, I think that's the right URL.

40:52But it also comes pre-installed in the starter app. So it's in the starter app up here. You can see it's in this agent's skill. It comes with this LiveKit agent's skill that tells us how to develop LiveKit agents and how to test them. And honestly, I could probably just give it this skill, and it could have done this whole interview with you guys. It knows everything there is to know about agents. We'll have to do that. We'll have to have you come back and just let us interview the agents. Yeah, that was the goal when we did the, like, voice model shootout, was we were trying to bring them in and do like a live interview.

41:25So you can see here, it went ahead and added a tool called Multiply Numbers written in Python. Agent can provide a list of numbers and it will return the product. So we'll see if this one works. I'll restart the agent just in case. Oh, it's doing some, it's a little confused about the formatting. Should be okay. Shadow dimension. Yeah, could you multiply two times three? Sure. I'd be happy to multiply some numbers for you. What would you like me to multiply? Two and three. The number is two and three. Got it. Two times three. Let me work that out.

42:12I should have had it add some logging.

42:17No worries. I can handle that. Oh, it's listening to me still. So then you'll be able to see that it's calling the tool down here.

42:40Hello, I'm your arithmetic companion. Hello, could you go ahead and multiply two times three? Sure, I'm happy to crunch some numbers with you. What calculation can I help you with? Two times three. Let me work that out. That's six. If you'd like a faster trick or another problem, just let me know. Yes, you can see here it called the tool as expected with the numbers. Hey, so why don't you multiply all the numbers between 10 and 20? Hmm. All right. I'll use the tool to multiply all the integers from 10 to 20. Do you mean the integers from 11 to 19 or including 10 and 20 as well? Pretty pedantic question.

43:25Honestly, you can choose.

43:30That's funny. Thanks. I'll include both 10 and 20 to get the full product from 10 through 20. Let me work that out.

43:42i think i have a number

43:48someone said ask it about pre-calculus concepts i don't know what happened here it doesn't seem to have called the called the tool well maybe something's happening sorry if something went wrong on your end something going a little wrong on the xai real-time api um not quite sure no worries at all that's funny response thanks for still being there 10 to 20 so ben i know um you know we're gonna have you hang out here at least until the hour if we have enough time uh somebody requested jj uh a step-by-step live build in terms of how you actually get it from where we're at now or from zero to where we're at now um i don't know if you can show us that but maybe you can walk through the steps how someone could do this?

44:35What was your recommendation? Yeah, well, actually, you know, I think the easiest way to do that is, so I showed you that, I think that the Git repo that we linked before is a great place to start, obviously. And I think, especially with CloudCode, but there's a more straightforward way to get from like zero to one here, which is in your LiveKit cloud dashboard, you go to LiveKit. It's not a price of a page. I look at our pricing page all the time. You go here and you click start building. And that'll take you into the LiveKit Cloud dashboard after you sign up. From there, you can go to agents and choose deploy new agent and create it right in your browser with our agent builder.

45:19Now, our agent builder is designed to be a quick start way to get started and then turn something into code later. So this is the same default agent that's in the starter app, friendly, reliable voice assistant. By default, it's going to use pipeline models with a Vitas-Cartesia sonic model for DTS. It's going to use the OpenAI's GPT-5-2-chat model and DeepGram for speech-to-text. And these all run through LiveKit Cloud. We have a managed inference service, so you can just attach this all to your LiveKit Cloud account. And then you can talk to it right in here.

45:56Ben Cherry:Hello, it's great to connect with you. How can I help you today? Hey, could you tell me a joke? Sure. Why don't skeletons fight each other? They don't have the guts. Well, it told me a different joke than Grok did, so that's nice. Hey, so thank you. You can use it right here in the LiveKit agent builder. And then the awesome thing here is that this thing is actually backed by actual Python code. And if you read this, you're going to see it basically looks identical to what I had in the starter app. So this is another way to get into the starter thing, and you can actually download the code here if you want to do it that way and play around.

46:34That's the easiest way to get into LiveKit agents. And then run it through your own cloud code, codex, whatever your flavor of choice is. Yeah. Yeah, and then once you download it, you can let kind of cloud take it from there. But this is the easiest way to get started with LiveKit, especially if you're not extremely like GitHub and Python oriented to start. We also have a TypeScript SDK. This one exports only to Python, but we do have a TypeScript SDK as well. The runs on Node.js, it's also accessible for people. I did want to show something else while I was in here. Another thing that we have is in LiveKit Cloud, we have a sessions list.

47:15so you can actually see every one of your sessions with our observability feature and you should actually be able to see what's going on inside of them so this one is that session we just had where it was doing some math and you can see it's got the whole transcript you can actually play back the audio which is recorded that's good in case they're different yeah you can jump ahead

47:48So then you'll be able to see that it's fine. So you have this, all of this, you've also got detailed traces that show you the performance of each one of the pieces and how long they took. And you've got all of the server logs that were collected during it. This stuff is going to be really useful to, I've got some feature flag stuff that's coming later. There's extra tabs shown here. This stuff's going to be really useful to developers who deploy their agents into the real world and want to see what's actually happening in those sessions. Okay. This is awesome. Yeah. That is really neat. I love the idea that you can go back and look at the different pieces and know, like, okay, this model's running too slow.

48:29I want to swap this out somewhere here. I want to, or maybe change this piece. And to be able to see that visually while it's playing is really cool. Yeah, and I think that actually that was the session where it was a little buggy at first. I went back and ran it again. It did a little better. You could see it never called the tool. On this session, you could see it called the tool with numbers two and three, came back product six. And down here at the bottom, it never called that tool with 10 and 20. And so that was what went wrong. And so I don't know for sure what went wrong there, but we've kind of got the ability to get in here and debug that and figure out why that tool wasn't called and all of the logs, including the errors from the XAI real-time API that we were getting here.

49:20That stuff is all in here. So it'll give you a really good view into what's happening. For sessions that are running, you know, we have customers who run thousands, tens of thousands of calls per day, customer support and all sorts of things. and having the ability to go in, replay, listen to what the user actually said, read the logs is just exceptionally valuable because otherwise these things are kind of a black box. Hey, Ben, someone had a question in the chat. Can you change the voice? Can you put your own voice? I suggested, you know, you probably need to use a tool to voice clone it, but do you have voice cloning capabilities?

49:56Do you host stuff? Yeah, actually, great question. This will be a fun live demo. This one's very new. So many of these providers under the hood have voice cloning things. We've created a kind of layer on top so you can clone your voice through LiveKit and it'll clone it on all of the underlying providers. So you can pick the one and you can actually fall back between them. So in here, I'll go ahead and name this Ben, English language. So I've got to read the script. Hi there. I am making a recording that will be used to clone my voice on TTS model providers. I'm excited to hear how it turns out.

50:43So I will go ahead and upload that. Yes, I provide consent. A lot of voices are biometric data. There's a lot of stuff around it. so it's gonna this will take a minute or two but it's gonna go and upload that voice and then it will send it off to the voice coding apis on i know it'll do it on cartesia and i think i think in world right now um and then uh maybe 11 labs and oh looks like i know 11 labs has a tool to do it yeah hello this is a preview of my voice i can help you build engaging and natural sounding conversations. So there you go. That's generated from the clone voice. So now if I go back to, where did I have the agent builder?

51:30I guess I had it here. Had this agent in our agent builder. And this would work. I could put pastes into code too, but I'll do it here. It's a little easier. Say custom voice. Oh, there it is. Ben's already in the list. So now I can talk to him. wow hi there how can i help you today hey could you tell me a joke sure why did the scarecrow win an award because he was outstanding in his field now we don't have to do this but as a joke you've got to uh hook this up to the ai that's trying to convince you you're an ai but it's your voice trying to convince you that you're an ai we do have a really good question in the chat too that's worth worth asking here in a minute i want to make sure we get to it i i think i saw that too the donner saw yeah one yeah really good question yeah i'll get that prompt out of out of cloud and i'll add it back to the for the agent builder interface um while while it's doing that sounds like you're moving a prompt from the cloud.

52:42Real quick here while that's doing its thing. Yep. Can you speak a little about practical applications for like small businesses? Maybe any use case examples that worked well? Yeah. So, um, you know, actually I have a, I have a great story here that I will tell you. So I had my first in the wild interaction with a voice agent that was built on LiveKit. and it was a few months ago. I live in San Francisco and our house has, there's no crawl space under our house and we've got a below grade bathroom and room downstairs from a garage conversion we did. You know, standard San Francisco stuff. So in there, there's this backflow valve.

53:22Anyway, it's a plumbing story and it gets clogged all the time. And so it clogged and our downstairs bathroom was backing up. It wouldn't flush. Shower's backing up and it's 1030 at night. I'm like, well we really need to get this thing fixed so I went to uh find a plumber 10 30 at night and I go on Yelp and I just go through all the plumbers and I call them I call them I call them and there's no one there no one there it's like our hours of operation are 9 a.m to 5 p.m keep hearing that and then I go fourth fifth one down the list I give it a call and I hear hi this is so-and-so from whatever plumbing uh how can I help you today and I say oh I actually I need a plumber and I describe the situation it's like great we have a plumber who we can send out there, he could be there in 45 minutes.

54:04And sure enough, a plumber showed up 45 minutes later and solved my issue. And the difference and the reason why that company won my business and the other ones did not is that they had a 24-hour receptionist powered by voice AI that was built through, it wasn't built, it was built through LiveKit, but by a provider who's building on top of us to provide that service to small businesses. And I think that it actually provides a competitive advantage to the business that chose to put a 24-hour receptionist on the phone and was actually able to schedule. Because I think these other ones may have had plumbers in the field, but they had no one to dispatch them.

54:40So I think that that's an application for small businesses I think is pretty meaningful. I had a run-in, my first run-in with Voice AI in the wild just a few weeks ago when I went to Panda Express to order dinner. That's right. How was it? It had a little sign that was like, please be patient with our new AI voice system. And honestly, it was great. It understood what I said. It got my food right. I mean, you know, that's literally all I want is to not have to scream at it and repeat myself three times and to know that the food in the bag is what I was hoping for. Do you know, do you have any fast food customers that are built on LiveKit at this point?

55:21Yeah, we've definitely had some. I don't know off the top of my head the exact brands, but absolutely. I know that voice AI drive-thrus and ordering systems are a hot area. Honestly, you can actually do a better job with voice. It is actually one of our example, probably our most complete example that we offer in our agents repository itself is actually a virtual McDonald's drive-thru that we've kind of constructed. and it's really good at getting your order and getting it exactly right and letting you edit the add-ons and request the sauce and make modifications. And then it gets it right every time.

55:58That's in your LiveKit repo? It's in the LiveKit agents repo. Agents repo, yeah, let's link that. Maybe what I need is an agent that will just talk to them at the drive-thru for me so I can continue listening to my book while I'm in the car. Yeah, and the LiveKit agents repo, this one here. Oh, yeah, the drive-thru. Enter the shadow dimension.

56:21I love it. Someone had a comment that was interesting. They said, I had a similar experience with a voice AI receptionist, so much better than a voice phone menu. I guess why wouldn't you replace every phone menu with a voice agent? Do you think it's just a matter of time? Is there a cost component where they're not comparable yet? What's the reason you wouldn't just do that? Yeah, I think that there's definitely still a cost component. But I do think that that will somewhat work itself out because, you know, consumers will prefer working with brands that have really accessible, you know, customer support or whatever it is sort of systems.

57:02And then also the ones that have these, they will be more effective at actually helping the customers resolve their issue, which in the long run will be a competitive advantage or save more money for the business. But I just think it's a matter of time. I mean, it's been like two and a half years since the very first thing that even sort of did this appeared. And I think that there's just, even if the technology got no better than it is today, and this is true in AI in general, but it's already so fantastically world changing. And it's just going to take time for the world to catch up. Like every model lab could stop training new models right now.

57:36All progress could stop and the entire world would still be irrecognizably changed over the next five years as these things spread out. The other thing is like, not only is the technology new, but no one knows, really knows how to build it yet. I mean, some people kind of are starting to figure it out. But if you think back to like, you know, the Internet was invented in the early 90s and the web was invented in the early 90s. And by like the year 2000, there were a lot of interesting businesses, some working, some not, some cool websites. and there was a lot going on. I don't know if you all remember GeoCities and stuff like that.

58:09That was very early days and that stuff you look back on now was so rudimentary compared to and both in terms of like its user experience and also in terms of like its impact on the world and then just like fast forward five, 10 years into like the rise of social media and like Web 2.0 and I don't know if you all remember when we went from, you know, static websites to Ajax and now to like single page applications but all of those sorts of transformations are going to come in the agents and the voice agent space as people start to figure out the patterns that really work to build these things and the sorts of power that you can do.

58:44Ben, I know we're pretty tight on time here and that you need to go soon. I've got a few minutes. Okay. We've got two more questions I'd love to work in from readers as we go. Viewers, excuse me, we're not in the newsletter, are we Grant? No. All right. So Mental Health Corps would like to know if you'd speak a little to the concerns of using a voice of public figures and for the record panda x has awesome almond screens that i don't think that second part was the question um so uh yeah absolutely i think that uh the the concern i assume is uh sort of like in a deep fake space i definitely think that's what they're talking about yeah yeah i believe um you know i'm not going to get into the kind of politics of it too much but i will say that there is great danger in general in terms of like an erosion of what is true and what is not and there is uh these technologies are making it possible to do some insanely convincing things and you don't even have to convince everyone you know like i don't know if any of you have ever received a nigerian prince email scan you know those things all the time man it doesn't work on me i know it's fake, but it works on someone, a pretty small slice of someone's, but non-zero.

1:00:05And there's a reason those things exist. And I do think that in the age of AI, like the target addressable audience for who it's going to work on is just increasing, increasing, increasing for all sorts of different things. So I do think that it's a real challenge that society has to reckon with, although I wouldn't say that I have all the answers on that. Yeah, same, same. We have an interesting interview coming out real soon with someone who's working on a solution to that that you'll want to stay tuned for i think it's june 24th but uh i think that's right it could change okay we got one more i've got uh i want to bring up real quick and this is from damian barham i joined a little late i'm interested in building on top and offering to other business owners i know personally i've been looking into this for my detailing business as well i've done five demos uh i don't realize that's not really a question but i think My question from that is like, suppose I wanted to build something on top of it.

1:01:01Like where, as part of my app, where do I start? Yeah. Well, I think that if you're trying to build something to offer, like you're trying to offer voice agents to some set of businesses or to some vertical, the thing that you can build on top of LiveKit and I think is amazing power of LiveKit is you can build a really accessible, like no code agent builder for them that allows them to kind of tweak some settings and you can get the agent under the hood to be constructed exactly as it needs to be for that segment like whatever that sort of like business segment is. If it's like trucking operations or something or like managing like fleet scheduling for like a demolition business or something like that.

1:01:45You could build that voice agent orchestration system and get all the pieces right and integrations the different data sources and then present a really simply UI in the web browser that the business owners themselves can come in and like tweak the settings that they know how to tweak and that are specific to them. Well, you've handled all the rest. And there's a lot of businesses doing this sort of thing. And that's certainly when I called the virtual plumber, it was surely built like that. So they had they had done all the hard work and they built on top of LiveKit to build all these pieces and then present something very simple and very vertically focused to their end customers that makes them their end customers be able to put in a little bit of effort and get massive leverage out and I think that's still the sort of sweet spot for a lot of businesses a lot of industries right now at largest enterprises they're much more interested in just honing the whole thing all the way down to the code level but in those smaller businesses I think there's huge opportunity for people to kind of aggregate the solution so from a high level what What is the way you would do that?

1:02:43You would use the cloud version or would you like go and get into cloud code and code up your own UI? Like how, like from a high level technical path. You would code up your own UI for like building and deploying agents. You'd have an agent. Then the Python side, you'd build an agent, which is kind of a runtime for other, that runtime, it would receive like the prompt for the specific business and the ID of that customer as variables that get passed in. I didn't show how to do this, but it's not particularly hard so that you could say, okay, well, you're operating as like the receptionist for this business and here's your custom instructions and prompts and here's the knowledge base for like everything you know about that business.

1:03:27And then the end users, well, not the end users, but the business users would have a dashboard where they can go in and configure those things and then assign phone numbers, probably through phone numbers. And LiveKit has an integrated phone number service as well. supports inbound phone numbers. We've got a lot of little pieces. There's a lot of interesting problems to solve in this space. Just keep naming. Oh, we have a little product over here. But you could also use Twilio or something like that. But you would build a custom interface on top of LiveKit in Next.js or in any other web framework that you want.

1:04:01They would hook into the LiveKit APIs and dispatch the agents or set up the link, the phone numbers to them. That makes sense. So correct me if I'm wrong. for people who are listening, you could literally tomorrow when the transcript on Google for this video is uploaded, you could copy this transcript, take it to Cloud Code and say, okay, help me do this. Exactly as you just described it. Yeah, absolutely. It'll help you set it up. Are there any other resources that you provide that you would recommend people, like YouTube channel you have or something you should watch? Oh, yeah, we do have an excellent YouTube channel.

1:04:34You know, for a long time, we weren't great at YouTube for a company with so much in video. It wasn't our thing, But we have an excellent developer advocate on staff now who is really great at YouTube content. And so I think he's put out like 30, 40 videos already this year. So our YouTube is full of both new feature demos and also how-tos. There's a whole series called LiveKit 101 that walks you through how to build voice agents from scratch. And are really well done on our YouTube channel, which I think it's just called LiveKit. I'll link it. I'll link it. Yeah, we'll drop it in there and we'll make sure when we share this back out in the newsletter, we get it in there too.

1:05:16Ben, thank you so much. This has been an absolute blast, man. Yeah, it was really great to be here. Thank you so much for having me. Absolutely. And look forward to learning more about what's going on and Grant and I to go build something weird. And hopefully everybody here will too. Yeah. If you build something especially weird, let me know. I'd love to see it. I'll do it. I keep thinking there's something cool for my D &D night with my buddies in this. I don't know what it could be. Oh, you have no idea. Let me just, quick digression. I play D &D with my family. And I do have a side project built on LiveKit where I've been using Notion AI to transcribe all of our sessions.

1:05:51So we play on, I don't know if you're familiar. We play with Roll20, a virtual tabletop. So do I. We chat through Discord, but use Roll20 as a good one. So that's what we used to do. We used to chat through Discord. But, you know, our schedules, I have a kid, my brother has a young kid, like our schedules are not great to meet all the time. So we're always forgetting what's going on between sessions. So I've been using Notion AI to transcribe sessions, but it's only accessible to me. And Notion AI gets all the names wrong because like my character's name is Sprocket, but it's a goblin. We're playing an artificer.

1:06:24It's a whole thing. And so I built a whole custom thing on top of LiveKit, a custom video conferencing platform with an AI agent that does transcriptions inside of it specifically for our campaign. And my character is an artificer. So the whole thing is like as if the artificer himself built this thing. It's a whole thing. I really, really considered doing something with like a transcription deal where we could keep better notes. Because the truth is we all try to keep notes. We're all at best inconsistent. Consistent. Yeah. Sounds like a good side project, Ben. Maybe you should publish it. Yeah, right now it's just for our campaign, but it has our characters in there.

1:07:01It transcribes the stuff and it auto-manages the vocabulary. This is a thing we didn't get into, but the vocab is so tricky to get right when you're in a specific domain. The general purpose transcription model is not going to transcribe all the words. Especially in D &D all the way. Yeah, in D &D. So you can dump all the campaign lore and then it automatically picks out the keywords that are likely to come up in that session and then puts them into the transcription service so it actually gets them right and it generates stories. I have it. My next project with it is to get it to also generate a few photos from the campaign generated through the ChatGPT's latest image model, which is so good.

1:07:36That's what we do. We do every week have like we have a project that is trained on, I mean I say trained on, it has images of all of our characters in there. So when something crazy happens, one of us can hop in and be like, you know, Snoot bites the head off a bat. and it spits out like a crazy oil painting of what we just did. And it's pretty amazing. Yeah, well, that sounds awesome. I'm planning to build that into my thing. That's my next like Claude project when I have a free night. I want to know about it when it works. Well, Ben, thanks again. We really appreciate it, man. And thanks for the D &D digression.

1:08:11Yeah. All right. Thank you, guys. See you later. All right. If you're watching, please take just a moment to like and subscribe. It helps us a lot when we try to bring more of these people along. And is there a way to reach out to Ben? We'll figure that out and we'll make sure you get it. Also, I think I said like and subscribe, didn't I? Hey, go check out the newsletter too while you're at it. The neuron.ai, sign up for the newsletter. We'd really appreciate it. Love to have you as one of our readers. But that's it. And we will catch you on the flip side. We'll be back next week with something else.

1:08:40I'm not sure what. It'll be something. DVD. DVD. And just to put a finer point on it, I will include all the links. And all of the advice from Ben in the blog post that we make after this. And we'll link it down below if you're watching this after the fact. Killer. And on that note, farewell for now, humans. Happy Memorial Day.