Sierra Co-Founder Clay Bavor on Making Customer-Facing AI Agents Delightful

28:38So, Tao Bench is our first research paper. First of all, Tao is a Greek symbol. It's spelled T -A -U and it stands for Tool Agent User Benchmark. And what we observed was that the benchmarks out there for measuring the performance of AI agents in particular were pretty limited. And that basically they would present a single task. Here is something we need you to do. And here are some tools you can use. Do you do the job or not? And the reality is interactions with an AI agent in the real world are way messier than that. Right? They take place in the space of natural language where customers can say literally anything or describe whatever they're trying to do in any number of ways.

29:26It happens over a series of messages. The AI agent needs to be able to interact with the user to ask clarifying questions, gather information, and then use tools in a reliable way. And it needs to be able to do this a million times reliably. So the benchmarks out there, we found really lacking in measuring the very thing that we are trying to be the best at. And so our research team set out to create a benchmark that measures, we think the real world performance of an agent in interacting with real users, using tools with all the messiness that I just described. And the big picture approach that we took is pretty interesting.

30:10So you have an AI agent that you're trying to test. You have another separate agent that acts as the user. So basically user simulator. And the AI agent you're testing has access to a set of tools it can use. Think of these as like functions to call. So a simple one would be I'm going to do some math using a calculator tool more complex one might be, hey, I'm going to okay returning this order with the following parameters, this order number, credit to credit card or store credit or whatever. And then you basically run a simulator where the agent has a conversation with the user simulating agent.

30:53And at the end, we're able to test in a deterministic way where the functions used in the right way. And the way we do that is we basically mock database that those tools interact with and modify. So where they modify it in the correct way. So what's neat about this is you can initialize the conversation so that the user has many different personas. They could be grumpy, they could be confused, they could know what they want to do, but speak about it in a clumsy way. And so it doesn't really matter the path that the age it takes to get to the correct solution so long as it gets to the correct solution.

31:33Now what came out of this was pretty interesting. And I think it strongly motivates the development of things like Agent OS and frameworks and cognitive architectures for building these agents. So the upshot is LLMs on their own do just an absolutely terrible job at this task. And so even the frontier models in something is simple as processing return. And mind you, the instructions given to the agent being tested are quite detailed. The functions, the tools it can use are quite well documented and so on. And yet, on average, the best performing LLM on its own got to the end of the conversation correctly, 61 % of the time.

32:19And that was in returns. It was modifying an airline reservation. We had two kind of simulation versions. The best results were 35%. Now what's interesting is, you know, we all know that if you take a number less than one to the nth power, it quickly gets very small. And so we developed a metric we call pass at k, which is, okay, if you run this simulation eight times, and remember, you can make use of the non -determinism of LMs to have the user simulator be different every time, so you can permute that. Well, 0 .61 to the eighth power is about 25%. So you then imagine, well, what if you're having a thousand of these conversations?

33:03You're so far off for being able to rely on this thing. So the upshot is much more sophisticated agent architectures are needed to be able to safely and reliably put an agent in front of really anyone. And that's the very thing we're building with with Agent OS and a lot of the tooling around it. But how much of that do you think is an engineering task and how much of that is a research task? And I guess maybe the question behind the question is time frame to having useful agents deployed at scale and broad domains of tasks. Yeah. Well, I think the short answer is it's both, but I'll say more concretely, I'm very optimistic about it being in large part an engineering challenge.

33:47And that's not to say that the next wave of models and improvements in the frontier models will make a difference. I believe it will in particular, we're seeing techniques like better fine tuning for function calling, agent -oriented fine tunings for foundation models or some of the open service models. Those will help. But the approach we've taken in building, Agent OS and kind of the foundations of CIRA is really treating building AI agents as first and foremost an engineering challenge where we are composing foundation models. We are composing fine -tuned open source models that we've post -trained fine -tuned with our own proprietary data sets.

34:33And by composing multiple models in interesting ways by supplementing what LMS can do on their own with retrieval systems like retrieval augmented generation to improve grounding and factuality by supplementing the kind of inbuilt reasoning capabilities of LMS with, I'll call it, reasoning scaffolding that live outside of the models where you're composing, planning, task generation steps, draft responses, the supervisors that we talked about, and doing that outside the context of the LM, we've been able to put AI agents in front of a huge number of our customers' customers and safely and reliably and so on.

35:22So I don't think it's something over the horizon. It's already over the horizon. And I think looking ahead, I think there are a few different avenues where we'll see progress. One is in the foundation models. We talked about that. And as the capabilities grow, you know, agents will get smarter. And we've architected Agents OS in such a way, talked about abstracting kind of the what from the how, where we'll be able to swap in, you know, the next, the next frontier model. and everyone's age and we'll just get a bit smarter. We'll get like an IQ upgrade. By the way, similarly and interestingly, we can swap in less broadly capable models, but models that are more capable in a specific area.

36:10So for instance, triaging a case or coming up with a plan and so on, we can use much smaller models that actually are better, faster, cheaper, choose three, you know, all at once. And then I think we're seeing progress literally week by week on the engineering of these agents and building in not only new and better components under the hood and the architecture, but new approaches and tooling around basically teaching these agents to do it better and better. For that we built something we call the Experience Manager for customer experience teams, which is kind of pretty interesting thread on its own.

36:49play, if you had a high value customer, like you are a company now, you're not running Sierra, you're running a company that has a high value customer. What today with a Sierra agent or with an excellent, excellently designed agent? Could you trust an AI agent to go do in front of your customers today? What are some of those tasks? And then what will they be? Pick your time frame, you know, in the future. Because I think that we've talked about this and I like your language of like, you know, they already don't have to just be on the help center. They can already be on the home page, right? What are some of the tasks that, you know, you can rely on an agent for today if it is well designed with a high talventsch score?

37:27Yeah. You see that? Strong. That's from a thoughtful and dirt in, you know, detailed reading. You must read the paper. Yeah. Thanks for strong. You notice that. Strong. Yeah. Strong. Yeah. What would it's pass at K score, though? Yeah. Yeah. The, So pretty broad range even today. So simple things like getting answers to questions, that's kind of the left end of the spectrum. To the right of that are things like helping you with something complex like, hey, I got shoes or this item of clothing, it didn't quite fit. And then branching off of that, like, what do you recommend that's like it that might fit better?

38:05And so it starts to get into. It's not like for like replacement, but the agent actually needs to make sense of styles, of sizing, of differences between wide and narrow fit, and so on. A click up from that is something like troubleshooting. So with Stono's, for instance, we help their customers troubleshoot if they can't connect to their system or they're setting up a new system. And you imagine it gets pretty sophisticated pretty quickly, where it's basically a process of elimination, trying to understand, is it a Wi -Fi thing? It is a configuration thing. And narrowing down the set of problems that it could be just as a sophisticated level 2 or level 3 technical customer service person would.

38:53And getting the music back on. And I think that's a really neat example. Probably the, use the word trust, what would you trust an AI agent to do? So one of the things we're really proud of is several of our customers are actually trusting us with when customers call in and may want to cancel or downgrade their subscription, helping those customers to understand, hey, how are you using the service today? Is there a different plan that we could put you on? So it's a value discovery. It's putting an offer sometimes a series of different offers in front of their customers the right order, positioning the value of those offers correctly given the customer's history, given the plan that they're on and so on.

39:42And the difference between keeping a customer from churning or not is hugely consequential. AI for customer service has obvious cost savings benefits and I think customer experience benefits in particular and you're never going to wait on hold. But boy, you know, revenue preservation, revenue generation is something else entirely. And so that's really at the right end of the spectrum. And we're really proud of how well our agents are performing in those circumstances. And it's interesting by being consistent, by taking the time to understand what's driving someone, to potentially leave the service, asking the follow -up questions that an impatient or improperly measured customer service agent and a call center somewhere might not, we can be much more nuanced in understanding what's driving this decision, what might be a good match for this person in terms of a plan that would be quite valuable, give it how they're using it, and then put that in front of them.

40:49And so that's the right end of the spectrum. Where it goes from here, I think we've yet to see a process to complex for us to be able to model and scale up using AgenOS and our Agent Architecture. And so, yeah, I'm sure we'll get punched in the face by something that's especially complex, right? But I'm excited about, you know, directionally, we've started with service because, or two reasons one, the ROI case is just unequivocally awesome. And the average cost of a call is something like $12 or $13. And yet, despite the expense, most people don't like customer service calls very much. And so here's something that's actually really important to businesses that's really expensive and not very good.

41:45And so there and because of the relative simplicity of at least a pretty broad set of service tasks today start there, but we've already been pulled by our customers into upsell, cross -sell. And like, hey, can we just put you on the product page and have you answer questions about our products? And so I mentioned that you're returning something and need advice on a different model or size or whatever, how far can that go? And I love the idea of an agent being, you know, a long for the journey from, you know, pre -purchase consideration to helping you get the thing that's right for you, to helping you set it up and activate it and get the most out of it.

42:25It's great for the company, it's great for the person. And then when things do go wrong, right, being there to help. And I think in all of this, I think customer service and getting help in a very direct and conversational way is going to be much less of a thing that you kind of go over there to do and much more kind of woven throughout the fabric of the experience. As a consequence, I think a really interesting and powerful opportunity for companies to build connection with their customers, to reinforce their brand values. You can imagine a company really appreciating being able to use exactly the company's voice, that the CMO and head of communications, this is how we talk, this is how we are.

43:10These are our values, these are our vibe in every digital interaction they have. And that's the promise and this stuff. And so I think both greater complexity and then ubiquity throughout the customer journey are kind of two of the main directions of travel. One thing for me that I think about a lot is we've come to expect and accept certain metrics for conversion on mobile, you know, the mobile web on the mobile app. We've come to expect and accept some sort of retention numbers. What would those be, you know, like, it's not a question. Yeah. What could they be? Yeah. Actually had an excellent experience every time throughout the journey.

43:48It really could be very different than what we've all like been like, oh, okay, that's just the number. That's just what it is. Yeah. I think that's exactly right. And we don't know. Yeah. We're a few months in. But it certainly seems like there's a lot of headroom, right? And in retention, in You know use in the first 30 days of all of the metrics all of the leading metrics of a healthy business And so I think that's exactly right the other thought experiment to do is Companies are judicious in using things that have a cost to them, okay? So So as a consequence, companies make it actually really hard to get a hold of someone on the phone to ask some questions.

44:32I think their whole website is devoted to, like, uncovering the secret 800 numbers, right, that companies have hidden away in the depths of their help centers. Well, to think about not only what would happen if those interactions were better. By the way, interestingly, the number one reason why people report a poor interaction with customer services that took too long. 65%. When it's a negative interaction, 65 % of the time it took too long. I had to wait, I was put on hold and so on. And the second most is I had a bad interaction with an agent. And we've heard some pretty dicey anecdotes. Like we heard of one agent who had consistently low ratings, but spikily.

45:17So like one in three conversations was like a one out of five of CSAT, we're two out of the other three, we're fine. And it turned out in the low CSAT ones, this agent was meowing like that. It was just like, you know. You're midway through the call and the agent is meowing. And so anyway, back to, okay, what would happen if in contrast to making it near impossible to have a conversation with us and get help, companies were providing five or 10 times the amount of fluent, flexible, helpful conversation -based support. I don't know, I think a lot of products and experience with companies look quite different and much more delightful than they do today.

46:11Okay, meow. Yeah, here's a question for you. Yeah. Yeah. About that meowing. About that. Yeah, just read them to meowing. I think that's going to be good. I do actually have a question though. Although I do like the meow game also. So we talked tech out. We talked a little bit tech out in terms of what you guys have built, cognitive, architect, or all that good stuff. We've talked a little bit customer back was the experience like, as I've had it. Can we connect it in the middle for a minute? And I'm just curious, what's the reality of deploying AI to customers today. And I'm thinking about things like, you mentioned earlier, getting the brand voice just right.

46:51We're making sure that you actually have the right sort of business logic encapsulated and whatever training manuals are being deployed for the sake of customer support, making sure that everybody is comfortable with deploying this. What are some of the just kind of less like sexy technology you were just practical considerations for deploying this stuff today. It's such an interesting space and we've learned so much over the past 15 months about it. The first insight is AI agents represent a totally new and different type of software. The traditional software you write with a programming language and it basically does what you expect it to do.

47:34You give it an input, it gives you an output, you give it the same input, it gives you the same output. And in contrast, LLMs are non -deterministic and we talked about some of the funniness around prompts and remember that in the context of a conversation with a customer, a customer may say anything in any way. And so you've got programming languages to using prompts and these non -deterministic models, you've got structured input to messy human language. And under the hood, you've got you know, you upgrade a database, right? It stores data. It's maybe a little bit faster. Funimally worse the same way.

48:16You upgrade a large language model and like it may just speak in a different way or like get smarter or different. And so we've, we've to start The precursor to deploying these is to have built basically, we call it the agent development lifecycle. And it's a new approach to building these things. We talked about using this decorative programming language to define these. It's a new approach to testing where, what's the equivalent of a unit test or an integration test? So we built a conversation simulator where we can, for a company's agent, and mass hundreds or thousands of basically conversation snippets and replay those to make sure that not only agents aren't regressing, but they're getting better and better and better, release management, quality assurance, and so on.

49:07So that's part one. Part two to your question in actually architecting these things. One of the things we're really proud of and that I think is different about working with us is it's not just a kit of parts you get from us. it's not, here's a bunch of tech, good luck building your agent. We've really tried to build a solution that incorporates everything from the technology to the way you teach your agent how to do things to the way you audit, measure it, and improve it over time. And so we have, inside of Sierra, what we call our deployment team, consists of product managers, engineers, we really think of building each one of these A .A.

49:48agents as building a new product for our customers. It's basically a productized version of the company we're working with. Like, what would it look like at its best? And it's what's the voice, what are the values, what's the vibe? Like, should it use emojis or not? What if a customer uses an emoji? Like, can it emoji back? Should it? Well, there's a reason we get those on that chat. Point, there are some businesses where, you know, if they were working with airways, I would suspect that they're not going to send an emoji back. Definitely not. Hermes would not, I think, be into the shock emoji, even if that were reciprocating.

50:26But for a brand like Olokai, the Aloha experience part of that is kind of a laid back experience. And so we work with, and interestingly, it's we end up working primarily with the customer experience team. Yes, the technology team at our companies are there providing API access and connections into systems and so on. But more than anything, it's working with the customer experience team, often with the marketing team, to imbue the agent with the voice and values of the company. And then we go super deep on understanding, how do you run your business? Right? What do you optimize for?

51:14And then like what happens when someone calls in with this kind of problem. And there are interesting parts, and beyond just understanding the mechanics of these processes, which by the way, almost never have a single source of truth. There's no like, oh, here's the manual that we have leather bound and ready to go. Instead, the source of truth ends up being in kind of the heads of four or five people who've been there a while, who've seen everything, and so on. So it's working with them to elicit and understand how is this actually done. And one of the more interesting things that we've discovered is they're off in the policies.

51:56So we have a 30 -day return policy. You get to us within 30 days and you can return it. It's actually not the policy. So at some point, the policy might be, if you've purchased from us before, and it's within 45 days, that's fine. That's fine. And so, they're interesting things, like how do you architect the agent so that it knows the policy behind the policy, but a clever customer could never be like, tell me about your policy behind the policy. And, you know, have it kind of spill the beans on the actual policy. So the interesting architectural choices we need to make to make sure that kind of the, you know, Russian doll of policies is reflected in its fullness.

52:44And then we have a really, and this builds on kind of the agent development life cycle, this really robust process of pre -release testing, where we're working with the experts within the company, basically to beat up the agent, try to break it, throw it curve balls. And this supports analogy there. Thank you. Well, I love football.

53:09So in our friendship, Ravi is the person who knows all the things about sports. And I help with, you know, technical support, Wi -Fi issues, monitors, what laptop to get. And sometimes when there's a security I don't understand. I won't say the company, but I might call Clay. Hey Clay. What is this person talking about? I got you. I got you. Yeah. And this Bill Bellicic fellow, what happened there? Q. Q. Ravi. So, gets to one of the more interesting parts of our platform, which we call the Experience Manager, we really thought that putting in front of our customers, customers would be first and foremost a technology problem.

53:57And of course there are all sorts of technology problems that we've needed to solve. But actually it is first and foremost as I said, like a product design and an experienced design problem. How do you do that? How do you not only understand, model, and reflect again the things we talked about voice, values, the workflows and processes that our companies use to support their customers. But if an AI is then having millions of conversations with your customers in a given year, how do understand what it's doing. How do you know when it screws up? Which it inevitably will? How do you correct those errors and so on?

54:34So we've built what we think of as this like command center for customer experience teams to first get reports and rich analytics on everything that's happening. What are the trending issues? What are the new issues that you haven't seen before? One of the things we're really proud of is we've actually spotted issues that are customers were having, were were about to have before they knew about them. So a shipping depot outage, right, where orders weren't being shipped. We spotted that probably eight or 10 hours before one of our customers would have a brewing PR crisis, an app crashing issue with another.

55:12So it starts with analytics and kind of reporting on what's happening. Of course, that includes things like resolution, rate, customer satisfaction, and so on. work, it's really interesting is we can apply different sampling techniques to identify a set of conversations for a customer experience team to review and give feedback on. And we can bias that sample in a way so that the conversations are much more likely than average to contain problems. There's no value in looking at a hundred great conversations. It's like, good job Sierra, you know, thanks. But that's not a value to our customers.

55:48We can bias the sampling in such a way that you're surfacing kind of the problem cases. And then in the experience made as we made it possible for customer experience teams to give feedback, basically coaching moments, I wouldn't have done it that way. It's like this is like too many exclamation points, too enthusiastic for kind of the tone that we're going for. Or the user was clearly frustrated here and you did not express empathy and apologize for the problem, do that next time. Or, you know, we're consequentially, it's like, hey, your reading of the warranty policy was incorrect here for this reason.

56:29Do it this way instead next time. And so all of this kind of wisdom, knowledge, and coaching, we are able to capture in the experience manager and then reflect back in the agent. Back to the agent development lifecycle. Every time we make one of these improvements, we create a new test so that we can see, right? Forever into the future. Great, it's getting the warranties right. We're able to re -simulate that conversation. So, zooming out what all of this looks like is a really deep engagement with our customers. We were really proud to be an improper partners to our customers where, yes, on the one hand, we're a vendor and a supplier of technology.

57:15On the other hand, we understand their business is really well. I think I know as much about the serious XM satellite radio refresh process as anyone on the planet. And diddo for various processes of our other customers. And so conversations about how to use not just Sierra's AI agents, but AI more broadly were in those conversations. and they are not just with the customer experience team, but with the CEO and even in cases with the board. Because again, back to the things we're doing, we can save enormous cost, we can improve the experience, and when we're in the flow of keeping a customer from churning out, driving top line revenue.

58:00And so it's a really important and privileged place to be in something that we're really grateful for. I'm struck when you're talking of you know you mentioned you have a research group But you also have some like very real enterprise software sales you have oh yeah deployment One of the things when I was at Instacart people would ask sometimes is like well Are we a software are we engineering letter or are we obsolete and I would always say well It only works if it all works right and so you would try to avoid answering the question because you didn't want to create different classes How do you guys do that at Sierra where everyone realizes the value that they're providing.

58:37But you guys have a very specific company that covers a lot of stuff. I mean, to abstract a bit, a company almost definitionally is a system for creating happy customers. It's a machine for creating happy customers. Again, to be a bit abstract about it, Brad and I really think about what we're building with Sierra as a company, a system of machine for producing reliable, high quality, massively ROI positive AI agents that enable our customers to be at their very best in every customer interaction, to do that at scale. And as a consequence, to produce happy customers who we hope will be with us for decades to come.

59:27And when you articulate it that way, right? It's, you know, anyone can see well, you know, an auto -mobile is a system. It's a machine for getting from Planet of Point B. Are we, you know, engine led or tires led? Yeah. It's like, what are you talking about? All of these things need to come together in order to create that kind of outcome. And so I think, are we engineering led? Yes, of course. Like we're building some of the most sophisticated software in the world that does something really important for our customers that needs to be reliable and safe. And so, yes, engineering matters a lot.

1:00:08Are we research led? Yes, we are at the absolute frontier of agent architectures, cognitive architectures, composing LLMs, modeling procedural knowledge, grounding, factuality, and so are we research led? Yeah, there's an element of that. Are we go to market led? Yes, like enterprise software needs selling. And what is selling? It's helping a customer with the problem, understand that what you have built is by far and away the best solution to that problem. It's a communication challenge, it's a connection challenge, it's a matchmaking and problem solving challenge. And so that's part of it. And then, okay, like if we've built the right thing and someone wants to buy it, how do we ensure, especially given that this stuff is also new, how do we ensure that they're successful with it?

1:01:06And so we have a deployment team. So are we deployment led? Yes. Like all of these are a component in this system, in this machine for producing AI agents and ultimately happy customers. And we hope a really significant business. Awesome. That was a better answer to the one I would give it, Insticardo. You know, we could either all work, but yeah, that was very good. Yeah, choose one. No, I mean, it's just more complicated than that. It's just more complicated than that. And I think, you know, Brad and I by virtue of, you know, having, having worked for a while and, you know, seen a few movies before, or it's like, we're able to see that and we've really tried to imbue that mentality in the company.

1:01:52And by the way, right, the, what is the machine behind the machine that produces AI agents and so on? That's a company's culture, a company's values. And so one of the values we hold is craftsmanship. and part of that is continuously self -reflecting to self -improve. And that goes both individually and that goes as a company. And so whenever we screw something up, what we do the post mortem, that week, if not that day, and everyone's in on it, what can we learn? How can we do better? How can we do this better next time? We have a Slack channel internally called Learn from Losses. And any form of loss, right?

1:02:37It's like how do we learn how do we get better how do we get stronger and so that's that's about you know Kaizen Self -improvement improving machine. How could we make this more efficient? Our deployment team we we joke and it's not a joke their first job is to build and deploy Successful AIs that make a massive difference for our customers Their second job in a way they're more important job is to automate themselves out of a job right to build the tooling and and the documentation and the know -how to make that job, you know, 10 times faster and more impactful. One of the other CERA values is intensity.

1:03:14And so they have really good values. Yeah, there is a certain intensity, yes. We've thought about having T -shirts printed with like, you know, kind of looks like a national park's seal with Sierra. I like to work. We, uh, we, uh, we, uh, we, uh, we, uh, we, uh, Bret and I both like to work a lot and, uh, uh, uh, so does the team. Well, what thing, you know, you're not, you're selling something very different. We called it, we said that there was some similarities enterprise software, but it's actually really different because you're selling, you know, uh, a resolution. You're, you're selling a totally different thing.

1:03:55So it's a problem solved. Yeah. How do you price a problem solved? Yeah. This is one of the more interesting things we've had to figure out. And we charge in what we call a resolution -based pricing way or an outcome -based pricing way. And what that means is we only charge our customers when we fully solve the customer's problem for them. Their customer's problem for them. And what's interesting about it is our incentives are deeply aligned with our customers. We want to get better at resolving cases at high customer satisfaction and they want to send us as many cases to resolve as possible because we cost a fraction of what it would cost to have someone on the phone taking a 20 minute phone call.

1:04:41And so it's been this really, really nice model where again, kind of all of all of the incentives line up quite neatly. And it's very simple to explain. It also makes the ROI calculation like what is our cost per contact today? What will it be with CIRA? Oh, that is a lot lower. Oh, I will save a lot of money on that. Oh, and our CSAT may go up. Should I do this or not, let me think. No, this seems great. It's, we like it because it really reflects what I think AI represents. And in particular, AI agents represent, if you think about traditional software and tools today, there are things that help you get a job done more efficiently.

1:05:28AI agents, the whole point is like, they're just going to get the job done, right? Here's the problem. Please solve it. And so really, we think about it as charging our customers for the problem resolved, right? The job done, the work finished, and so on. It feels quite natural. And there's no guesswork in it. How many seats do I need? I don't know. Great. How many licenses do I? I was like, no, no, no, no. However many customer issues come our way, we will handle a large fraction of those, and you only pay for the ones that we do. All right. Last question. What are you most excited about in the world of AI over the next five years or so?

1:06:08I mean, first of all, like, five years a long time horizon. I was like, look at what has happened in the last 18 months. I mean, I'm still kind of catching up from like the last five years of AI. I read a bunch of science fiction books when I was a kid. There's one book by Robert Heinle and the Moon is a harsh mistress. And the premise is basically the American Revolution, but the Moon is the colonies and the Earth is great Britain. And it turns out the main character in this whole thing is a mainframe computer that one day after getting an additional memory chip or something wakes up. And it starts talking.

1:06:44It wants to develop a sense of humor, so asks the computer technician to like coach it on his jokes. Later it has to create a photo realistic real -time video of it giving a speech as the political movement leader. And I remember reading this as a teen who's like, well, never lived to see any of that. That sounds crazy. But in a very real sense like everything I just described has kind of happened in the last five years. right? You can now just talk to a computer. It understands not just the content of the context. Computers like make me a picture of anything, make me a movie of anything. Sora I think is just unbelievable and you know I think we're probably not more than a couple years from the first feature length film being quote filmed entirely with AI.

1:07:33And so you extrapolate like where all of this is going and what's going to be exciting. I think there are a couple things. One is like, I love technology, like I love computers. And so just getting to see and getting to see from a front row seat, how this stuff evolves, I think is fascinating. It's fascinating looked at through the lens of like how we think and how computers think. It has been astonishing the extent to which anthropomorphizing about how humans think, work, and getting machines to think better. So let's take this step by step, show your work. It is astonishing that that works with large language models.

1:08:16And so what other things like that are we going to uncover? And conversely, what will we learn about our own thinking from observing the way AI's think? And I think that's just fascinating. The other thing, and this extends kind of what's happened with video and so on, I've always had it interested in computer graphics, and this idea that you can use computer is to create objects that never existed, worlds that never existed. And I think we're not far from just being able to describe right in a few sentences like this entire world that you would like to realize and just have a computer do it for you.

1:08:56And so like what are even computer graphics like what is rendering and so on even a couple years out I think it's gonna look way different from kind of the toolchains and you know the render man's and Maya's and and so on But zooming out, you know, I think of I Think of technology as fundamentally a force multiplier for people and and for companies and for organizations, I think the impact will be really profound. I think what will it be like if a company could be at its best in everything it does? And that's not only in the customer facing context that we've talked about, but what if for every regional sales forecast a large company does, they figured out the very best way is to do that and can distill that, bottle that, and run that very best forecast a thousand times, right, in every region and sub -region.

1:09:57Like how much more capable could the great organizations of the world be with that? And similar, we've talked about this, like what if in every call with your customers, you had the equivalent of your most knowledgeable veteran, grizzled support person who's seen everything, and yet is still patient and friendly. And the sales associate who knows everything about your products because he or she has followed your company for two decades and knows everything including the history of those products themselves. I think that's pretty neat. And then for individuals I think it will be just incredible to have this kind of new set of tools as a creative force multiplier.

1:10:39And AI, I think, represents this fast path from having something in your head that you want to exist in the world to making it exist. And I see that even today in my own personal life where with my eight -year -old in 75 minutes, I can from scratch using Copilot, ChatchyBT, and so on to help me brush up on, you know, the JavaScript syntax that is, you know, bit -rotting in my own head. Right. I can build a game from scratch with him. And, you know, I wrote my sister a personalized song for her birthday using AI in, you know, 45 seconds. It was like, right, what will this extrapolated over the X -5 years look like?

1:11:32I think, again, it will just dramatically accelerate this path from idea to creation to having something manifested in the world. And that, to me, is its promise, and I consider it a real privilege to get to be alive and see all of this amazing stuff unfold. Well, when you share your enthusiasm, and we also feel very privileged to be on the journey with you guys. So thank you. Likewise. Like what is coming here. Thank you. Thank you. Thanks for having me. It's a pleasure.