What OpenAI and Google engineers learned deploying 50+ AI products in production

27:10And I've seen that being a very distinguishing factor of companies that build products which are successful because you're kind of bringing in that top-down approach. It's almost always impossible for it to be bottom-up. You can't have a bunch of engineers go and get buy-in from the leader if they just don't trust in the technology or if they have misaligned expectations about the technology, right? I've heard from so many folks who are building that our leaders just don't understand the extent to which AI can solve a particular problem or they just wipe out something and assume it's easy to take it to production and you really need to understand the range of what AI can solve today so that you can guide decisions within the company.

27:51The second one is the culture itself, right? And again, I work with enterprises where AI is not their main thing and they need to bring in AI into their processes just because a competitor is doing it and just because it does make sense because there are use cases that are very right. Then along the way, I feel a lot of companies have this culture of FOMO and you will be replaced and those kind of things and people get really afraid. Subject matter experts are such a huge part of building AI products that work because you really need to consult them to understand how your AI is behaving or what the ideal behavior should be.

28:27But then I've spoken to a bunch of companies where the subject matter experts just don't want to talk to you because they think their job is being replaced. So as I mean, again, this comes from the leader itself. You want to build a culture of empowerment, of augmenting AI into your own workflows so that you can 10x in what you're doing instead of saying that probably you will be replaced if you don't adopt AI and stuff like that. So that kind of an empowering culture always helps. You want to make your entire organization be in it together and make AI work for you instead of trying to guard their own jobs, etc.

29:04And with AI, it's also true that it opens up a lot more opportunities than before. So you could have your employees doing a lot more things than before and 10x their productivity. And the third one is the technical part, which we talk about, right? I think folks that are successful are incredibly obsessed about understanding their workflows very well and augmenting parts that could be ripe for AI versus the ones that might need human in the loop somewhere, etc. Whenever you're trying to automate some part of a workflow, it's never the case that you could use an AI agent and that will kind of solve your problems, right?

29:46It's always you probably have a machine learning model that's going to do some part of the job. You have deterministic code doing some part of the job. So you really need to be obsessed with understanding that workflow so you can choose the right tool for the problem instead of being obsessed with the technology itself. And another pattern I see is also folks really understand this idea of working with a non-deterministic API, which is your LLM. And what that means is they also understand the AI development lifecycle looks very different. And they iterate pretty quickly, which is can I build something, iterate quickly in a way that it doesn't ruin my customer experience, at the same time gives me enough amount of data so that I can estimate behavior, right?

30:31So they build that flywheel very quickly. As of today, it's not about being the first company to have an agent among your competitors. It's about have you built the right flywheels in place so that you can improve over time. When someone comes up to me and says, we have this one-click agent, it's going to be deployed in your system. And then in two or three days, it'll start showing you significant gains. I would almost be skeptical because it's just not possible. And that's not because the models aren't there, but because enterprise data and infrastructure is very messy. and you need a bit to, even the agent needs a bit to understand how these systems work.

31:06There are very messy taxonomies everywhere. People tend to do things like get customer data, we want, get customer data, we do, and these kind of things. And all those functions exist and they are being called. And basically, there's a lot of tech debt that you need to deal with. So most of the times, if you're obsessed with the problem itself and you understand your workflows very well, you will know how to improve your agents over time instead of just slapping an agent and assuming that it will work from day one. I probably will go as far to say that if someone's selling you one-click agents, it's pure marketing.

31:41You don't want to buy into that. I would rather go with a company that says, we're going to build this pipeline for you and that will learn over time and kind of build a flywheel to improve than something that's going to work out of the box. To replace any critical workflow or to build something that can give you significant ROI, Quite easily it takes four to six months of work, even if you have the best data layer and infrastructure layer. Amazing. There's a lot there that resonates so deeply with other conversations I've been having on this podcast. One is just for a company to be successful at seeing a lot of impact from AI, the founder CEO has to be deep into it.

32:18I had Dan Shipper on the podcast and they work with a bunch of companies helping them adopt AI. And he said that's the number one predictor of success is the CEO chatting with ChatGPT, Claude, whatever. many times a day. I love this example he gave at the Rackspace CEO as like catch up on AI news in the morning every day. I was imagining he'd be like chatting with like the chatbot versus like reading news. With the kind of information you have as of today, you could just, I mean, you want to choose the right channels as well because everybody has an opinion. So whose opinion do you want to bank on?

32:53I feel like having that good quality set of people that you're listening to really make sense. So he just has a list of two or three sources that he always looks at. And then he comes back with a bunch of questions and bounces it around with a bunch of AI experts to see what they think about it. And I was part of that group. So I kind of know about the questions that he comes up with. So it's pretty cool. I was like, why are you doing so much? And then he says it trickles down into a bunch of decisions that we will take. Okay, let me talk about another topic that's very, it's been a hot topic on this podcast.

33:25It was a hot topic on Twitter for a while. Well, evals, a lot of people are obsessed with evals, think they're the solution to a lot of problems in AI. A lot of people think they're overrated. That way you don't need evals. You can just feel the vibes and you'll be all right. What's your take on evals? How far does that take people in solving a lot of the problems that you talk about? In terms of like what is going on in the community, I feel there's this false dichotomy of like there's either evals is going to solve everything. or online monitoring or production monitoring is going to solve everything.

33:59And I find no reason to trust one of the extremes in the sense that I will entirely bank my application on this or like that to solve the thing. So if you take a step back, think of what are evals. Evals are basically your trusted product thinking or your knowledge about the product that is going into this set of data sets that you're going to build in the sense that this is what matters to me. Like this is the kind of problems that my agent should not do. And let me build a list of data sets so that I'm going to do well on those. And in terms of production monitoring, what you're doing there is you're deploying an application.

34:39And then you're having this some sort of key metrics that actually communicate back to you on how customers are using your product. Like you could be deploying any agent. And like if the customer is giving a thumbs up for your interaction, you better want to know that. So that is what production monitoring is going to do. And this production monitoring has existed for products for a long time. Just that now with AI agents, you need to be monitoring a lot more granularity. It's not just the customer always giving you explicit feedback, but there is many implicit feedback that you can get. For example, in ChatGPT, if you are liking the answer, you can actually give a thumbs up.

35:17Or if you don't like the answer, sometimes customers don't give you thumbs down, but actually regenerate the answer. So that is a clear indication that the initial answer that you generated is not meeting the customer's expectation. So these are the kind of implicit signals you always need to think about, and that spectrum has been increasing in terms of production monitoring. Now, let's come back to the initial topic of, okay, is it evals or is it production monitoring? What does it matter? So I feel, again, we go back to this problem-first approach of what is it that you're trying to build? like you're trying to build a reliable application for your customers.

35:54That's not going to do a bad thing. Like it's always going to do the right thing. Or if it is doing a wrong thing, you are you are basically alerted like very quickly. Right? So the I break this down into two parts. Like one is you like nobody goes into deploying an application without actually like, you know, just testing that this testing could be wipes or this testing could be, okay, I have this like ten questions that it should not go wrong no matter what changes I make. And let me build this and let's call this an evaluation data set. Now, let's say you build this, you deployed this, and then you figured, okay, now I need to understand whether it's doing the right thing or not.

36:31So if you're a high throughput or like a high transaction customer, you cannot tactically sit and evaluate all the traces, right? You need some indication to understand what are the things that I should look at. And this is where production monitoring comes into the picture that you cannot predict the base in which your agent could be doing wrong, but all of these other implicit signals and explicit signals, those are going to communicate back to you what are the traces that you need to look at. And that is where production monitoring helps. And once you get this kind of traces, you need to examine what are the failure patterns that you're seeing in these different types of interactions.

37:10And is there something that I really care about that should not happen? And if that kind of failure modes are happening, then I need to think about building an evaluation data set for it. And okay, let's say I built an evaluation data set for my agent trying to offer refunds where explicitly I have configured it not to. So I built this evaluation data set and then I made my changes in tools or prompts or whatever, and then I deployed the second version of the product. Now, there is no guarantee that this is the only problem that you're going to see. You still need production monitoring to actually catch different kinds of problems that you might encounter.

37:48So I feel evals are important, production monitoring is important, but this notion of only one of them is going to solve things for you, that is completely dismissible in my opinion. All right, a very reasonable answer. And the point here isn't, it's not just as simple as do both, it's more that there are different things to catch, and one approach won't catch all the things you need to be paying attention to. Exactly. Awesome. I want to take two steps back and kind of talk about how much weight the term evals has had to take in the second half of 2025. Because you go meet a data labeling company and they tell you our experts are writing evals.

38:28And then you have all of these folks saying that PMs should be writing evals. They're the new PRDs. And then you have folks saying that evals is pretty much everything, which is the feedback loop you're supposed to be building to improve your products. Now, step back as a beginner and kind of think, what are evals? Why is everyone saying evals? And these are actually different parts of the process. And nobody's wrong in the sense that, yes, these are evals. But when a data labeling company is telling you that our experts are writing evals, they're actually referring to error analysis, or, you know, experts just leaving notes on what should be right.

39:02Lawyers and doctors write evals that doesn't mean they're building LLM judges or they're building this entire feedback loop. And when you say that a PM should be writing evals, doesn't mean they have to write an LLM judge that's good enough for production. I think there's also very prescriptive ways of doing this and plus one to KD, which is you cannot predict upfront if you need to be building an LLM judge versus you need to be using implicit signals from production monitoring, et cetera. I think Martin Fowler at some point had this term called semantic diffusion back in the 2000s, which kind of means that someone comes up with a term, everybody starts butchering it with their own definitions, and then you kind of lose the actual definition of it.

39:45That is kind of what is happening to evals or agents or any world in AI as of today. Everybody kind of sees a different side to it, I guess. But if you make a bunch of practitioners sit together and ask them, is it important to build an actionable feedback loop for AI products? I think all of them will agree. Now, how you do that really depends on your application itself. When you go to complex use cases, it's incredibly hard to build LLM judges because you see a lot of emerging patterns. If you build a judge that would, you know, test for verbosity or something like that, it turns out that you're seeing newer patterns that your LLM judge is not able to catch.

40:22and then you just end up building too many evals. And at that point, it just makes sense to look at your user signals, fix them, check if you've regressed and move on instead of actually building these charges. So it all depends. I think one statement that every ML practitioner will tell you is it really depends on the context. Don't be obsessed with prescriptions they're going to change. That's such an important point, this idea that especially that evals just means many things to different people now. It's just like a term for so many things. And it's complicated to just talk about evals when you see it as the stuff data labeling companies are giving you and things PM are right.

41:00And there's also benchmarks. People call benchmarks a little bit evals. I recently spoke to a client who told me, we do evals. And I was like, okay, can you show me your data set? And I said, no, we just checked LM Arena and artificial analysis. These are independent benchmarks. And we know that this model is the right one for our use case. And I'm like, you're not doing evals. That's not evals. Those are model evals. But it makes sense. It's like the word, you know, it could be used in that context. I get why people think that. But yeah, now it's just confusing it even more. Yep. Just like one more line of questioning here that I think that's on my mind is the reason this became kind of a big debate is Cloud Code, the head of Cloud Code, Boris, was like, nah, we don't do evals on Cloud Code.

41:37It's all vibes. What can you share, Kariti, on Codex and the Codex team of how you approach evals? So Codex, we have like this balanced approach of like, you know, you need to have evals and you need to definitely listen to your customers. And I think Alex has been on your podcast recently, and he's been talking about how you're extremely focused on building the right product, right? And a big part of it is basically listening to your customers. And coding agents are extremely unique compared to agents for other domains in the sense that these are actually built for customizability, and these are built for engineers.

42:12So coding agent is not a product which is going to solve these top five workflows or top six workflows or whatever, right? It's meant to be customizable in multi-different ways. And the implication of that is that your product is going to be used in different integrations and different kinds of tools and different kinds of things. So it gets really hard to build an evaluation data set for all kinds of interactions that your customers are going to use your product for. But that said, you also need to understand that, okay, if I'm going to make a change, it's at least not going to damage something that is really core to the product.

42:47So we have evaluations for doing that. At the same time, we take extreme care on understanding how the customers are using it. For example, we built this code review product recently. It has been gaining extreme amount of traction. And I feel like many, many bugs in OpenAI as well as external customers are getting caught with this. And now let's say if I'm making a model change to the code review or different kinds of RL mechanism that I trained with it. Now, if I'm going to deploy it, I definitely do want to A-B test and identify whether it's actually finding the right mistakes and how are users reacting to it.

43:28Sometimes, if users do get annoyed by your incorrect code reviews, they go to the extent of just switching off the product. Those are the signals that you want to look at and make sure that your new changes are doing the right thing. It's extremely hard for us to think of these kind of scenarios beforehand and develop evaluation data sets for it. So I feel like there's a bit of both. There's a lot of vibes and there's a lot of customer feedback. And we're super active on the social media to understand if anybody's having certain types of problems and quickly fix that. So I feel it's a how do I put this?

44:05It's like a domain of things that you do. That makes so much sense. Okay, what I'm hearing codex pro evals, but it's not enough. But also, just watch customer behavior and feedback. And also there's some vibes, just like, is this feeling good? Is this, as I'm using it, generating great code that I'm excited about that I think is great. I don't think if anybody is coming and saying that I have this concrete set of evals that I can bet my life on and then I don't need to think about anything else, it's not going to work. And every new model that we're going to launch, we get together as a team and test different things.

44:41Each person is concentrating on something else. And we have this list of hard problems that we have. And we throw that to the model and see how well they are progressing. So it's like custom evals for each engineer, you would say. And just understand what the product is doing in its new model. If you're a founder, the hardest part of starting a company isn't having the idea. It's scaling the business without getting buried in back office work. That's where Brex comes in. Brex is the intelligent finance platform for founders. With Brex, you get high limit corporate cards, easy banking, high yield treasury, plus a team of AI agents that handle manual finance tasks for you.

45:20They'll do all the stuff that you don't want to do, like file your expenses, scour transactions for waste, and run reports, all according to your rules. With Brex's AI agents, you can move faster while staying in full control. One in three startups in the United States already runs on Brex. You can too at brex.com.

46:08and then just walk us through what this is, how this works, how teams can shift the way they build their AI products to this approach to help them avoid a lot of pain and suffering. Before we go about explaining the life cycle, a quick story on why Kirita and I came up with this is because there are tons of companies that we keep talking to that have the pressure from their competitors because they're all building agents. We should be building agents that are entirely autonomous. And I did end up working with a few customers where we built these end-to-end agents. And it turns out that because you start off at a place where you don't know how the user might interact with your system and what kind of responses or actions the AI might come up with, it's really hard to fix problems when you have this really huge workflow, which is taking four or five steps, making tons of decisions.

47:00You just end up debugging so much and then kind of hot fixing to the point where at a time we were building for a customer support use case, which is the example that we give in the newsletter as well. And we just shut down the product because we were doing so many hot fixes and there was no way we could count all the emerging problems that were coming up. And there's also quite some news online. Recently, I think Air Canada had this thing where one of their agents predicted or hallucinated a policy for a refund, which was not part of their original playbook. And they had to go by it because legal stuff.

47:40And there have been a ton of really scary incidents. And that's where the idea comes from. How can you build so that you don't lose customer trust and you don't end up or your agent or AI system doesn't end up making decisions that are super dangerous to the company itself. At the same time, build a flywheel so that you can improve your product as you go. And that's where we came up with this idea of continuous calibration, continuous development. The idea is pretty simple, which is we have this right side of the loop, which is continuous development, where you scope capability and curate data, essentially get a data set of what your expected inputs are and what your expected outputs should be looking at.

48:24this is a very good exercise before you start building any AI product, because many times you figured out that a lot of the folks within the team are just not aligned on how the product should behave. And that's where your PMs can really give in a lot more information and your subject matter experts as well. So you have this data set that you know your AI product should be doing really well on. It's not comprehensive, but it lets you get started. And then you set up the application and then design the right kind of evaluation metrics. And I intentionally use the term evaluation metrics, although we say evals because I just want to be very specific with what it is, because evaluation is a process.

49:01Evaluation metrics are dimensions that you want to focus on during the process, right? And then you go about deploying, run your evaluation metrics. And the second part is the continuous calibration, which is the part where you understand what behavior you hadn't expected in the beginning, right? Because when you start the development process, you have this data set that you're optimizing for. But more often than not, you realize that that data set is not comprehensive enough because users start behaving with your systems in ways that you did not predict. And that's where you want to do the calibration piece, right?

49:40I've deployed my system. Now I see that there are patterns that I did not really expect. And your evaluation metrics should give you some insight into those patterns. But sometimes you figure out that those metrics were also not enough. And you probably have new error patterns that you've not thought about. And that's where you analyze your behavior, spot error patterns. You apply fixes for issues that you see, but you also design newer evaluation metrics to figure out that they are emerging patterns. And that doesn't mean you should always design evaluation metrics, there are some errors that you can just fix and not really come back to because they're very spot errors.

50:16For instance, there's a tool calling error just because your tool wasn't defined well and stuff like that. You can just fix it and move on, right? And this is pretty much how an AI product lifecycle would look like. But what we specifically also mention is while you're going through these iterations, try to think of lower agency iterations in the beginning and higher control iterations, what that means is constrain the number of decisions your AI systems can make and make sure that they're humans in the loop, and then increase that over time because you're kind of building a flywheel of behavior.

50:53And you're understanding what kind of use cases are coming in or how your users are using the system, right? And one example I think we give in the newsletter itself is the customer support. This is a nice image that kind of shows how you can think of agency and control as two dimensions. And each of your versions keep on increasing the agency or the ability of your AI system to make decisions and lower the control as you go. And one example that we gave is that of the customer support agent, where you can break it down into three versions. The first version is just routing, which is, is your agent able to classify and route a particular ticket to the right department?

51:34And sometimes when you read this, you probably think, is it so hard to just do routing? Why can't an agent easily do that? And when you go to enterprises, routing itself can be a super complex problem. Any retail company, any popular retail company that you can think of has hierarchical taxonomies. Most of the times the taxonomies are incredibly messy. I have worked in use cases where you probably have taxonomy that says some kind of hierarchy And then that says shoes and then women's shoes and men's shoes all at the same layer where ideally you should be having shoes and then women's shoes and men's shoes should be sub classes.

52:15And then you're like, OK, fine, I could just merge that. And you go further and you see that there's also another section on the shoes that says for women and for men. And it's just not aggregated. It's not fixed for some reason. So if an agent kind of sees this kind of a taxonomy, what is it supposed to do? where is it supposed to route? And a lot of the times we are not aware of these problems until you actually go about building something and understanding it, right? So, and when these kind of problems, real human agents see these kind of problems, they know what to check next. Maybe they realize that the node that says for women and for men that's under shoes was last updated in 2019, which means that it's just a dead node that's lying there and not being used.

52:56So they kind of know that, okay, we're supposed to be looking at a different node and stuff like that. And I'm not saying agents cannot understand this or models are not capable enough to understand this, but there are really weird rules within enterprises that are not documented anywhere and you want to make sure that the agents have all of that context instead of just throwing the problem at them, right? Yeah, coming back to the versions we had, routing was one where you have really high control because even if your agent routes to the wrong department, humans can take control and undo those actions.

53:30And along the way, you also figure out that you probably are dealing with a ton of data issues that you need to fix and make sure that your data layer is good enough for the agent to function. We do is what we said of a co-pilot, which is now that you've figured out routing works fine after a few iterations and you've fixed all of your data issues, you could go to the next step, which is, can my agent provide suggestions based on some standard operating procedures that we have for the customer support agent? And it could just generate a draft that the human can make changes to. And when you do this, you're also logging human behavior, which means that how much of this draft was used by the customer support agent or what was omitted.

54:13So you're actually getting error analysis for free when you do this, because you're literally logging everything that the user is doing that you could then build back into your flywheel. And then we say, post that once you figured out that those drafts look good. And most of the times, maybe humans are not making too many changes. They're using these drafts as is. That's when you want to go to your end to end resolution assistant that could, you know, draft a resolution that could solve the ticket as well. Right. And those are the stages of agency. where you start with low agency and then you go up high rate.

54:47We also have this really nice table that we put together, which is what do you do at each version and what you learn that can enable you to go to the next step and what information do you get that you can feed into the loop, right? When you're just doing your routing, you have better quality routing data. You also know what kind of prompts you need to be building to improve the routing system. Essentially, you're figuring out your structure for context engineering and building that flywheel that you want to write. And while I go through this, I want to also be very clear that two things. One is when you build with CCCD in mind, it doesn't mean that you fix the problem all for once.

55:27It's possible that you've probably gone through V3 and you see a new distribution of data that you never previously imagined. But this is just one way to lower your risk, which is you get enough information about how users behave with your system before going to a point of complete autonomy. And the second thing is you're also kind of building this implicit logging system. A lot of people come and tell us that, oh, wait, there are evals, right? Why do you need something like this? the issue with just building a bunch of evaluation metrics and then having them in production is evaluation metrics catch only the errors that you're already aware of, but there can be a lot of emerging patterns that you understand only after you put things in production, right?

56:17So for those emerging patterns, you're kind of creating, you know, a low risk kind of a framework so that you could understand user behavior and not really be in a position where there are tons of errors in You're trying to fix all of them at once. And this is not the only way to do it. There are tons of different ways. You want to decide how you constrain your autonomy. It could be based on the number of actions that the agent is taking, which is what we do in this example. It could be based on topic. There are just some domains where it's pretty high risk to make a system completely autonomous for certain decisions.

56:54But for some other topics, it's OK to make them completely autonomous and depending on the complexity of the problem. And that's where you really want your product managers, your engineers and subject matter experts to align on how to build this system and continuously improve it. The idea is just behavior calibration and not losing user trust as you do that behavior calibration, I guess. We'll link folks to this actual post if they want to go really deep. You basically go through all of these steps by step, a bunch of examples. And the idea here is, as you said, that like the reason everything about what you're describing here is about making it continuous and iterative and kind of moving along this progression of higher autonomy, less control.

57:36And this idea of even calling continuous calibration, continuous development is communicating. It's this kind of iterative process. And just to be clear, this this naming is kind of owed to CICD, continuous integration, continuous deployment suite. And the idea here is like that this is the version of that for AI, where instead of just like integrating to unit tests and deploying constantly, it's running evals, looking at results, iterating on the metrics you're watching, figuring out where it's breaking and iterating on that. Awesome. Okay. So again, we'll point people to this post if they want to go deeper.

58:11That was a great overview. Is there anything else before I go into a different topic around this framework specifically that you think is important for people to know? I think one of the most common questions we get is, how do I know if I need to go to the next stage or if this is calibrated enough, right? There's not really a rulebook you can follow, but it's all about minimizing surprise, which means, let's say you're calibrating every one or two days and you figure out that you're not seeing new data distribution patterns. Your users have been pretty consistent with how they're behaving with the system.

58:42then the amount of information you gain is kind of very low. And that's when you know you can actually go to the next stage. And it's all about the wipes at that point. Like, you know, you're ready. You're not receiving any new information. But also, it really helps to understand that sometimes there are events that could completely, you know, mess up the calibration of your system. An example is GPT-4O doesn't exist anymore, or it's going to be deprecated in APIs as well. So most companies that were using 4.0 should switch to 5, and 5 has very different properties. So that's where your calibration is off again.

59:24You want to go back and do this process again. Sometimes users start behaving with systems also differently over time, or user behavior evolves. Even with consumer products, you don't talk to chat GPT the same way you were talking, say, two years ago. just because you know the capabilities have increased so much. And also just people get excited when, you know, these systems can solve one task. They want to try it out on other tasks as well. We built this system for underwriters at some point, right? Underwriting is a painful task. There are agreements that are like, you know, loan applications that are like 30 or 40 pages.

1:00:03And the idea for this bank was to build a system that could help underwriters pick policies and information about the bank so that they could approve loans. And for a good three or four months, everybody was pretty impressed with the system. We had underwriters actually report gains in terms of how much time they were spending, etc. And post three months, we realized that they were so excited with the product that they started asking very deep questions that we never anticipated. They would just throw the entire application document at the system and go like, for a case that looks like this, what did previous underwriters do?

1:00:39And for a user, that just seems like a natural extension of what they were doing. But the building behind it should significantly change. Now, you need to understand what does for a case like this mean in the context of the loan itself? Is it referring to people of a particular income range? Or is it referring to people in a particular geo and stuff like that? And then you need to pick up historical documents, analyze those documents, and then tell them, okay, this is what it looks like, versus just saying that there's a policy X, Y, and Z, and you want to look up that policy. So something that might seem very natural to an end user might be very hard to build as a product builder.

1:01:17And you see that user behavior also evolves over time, and that's when you know that you want to go back and recalibrate. What do you think is overhyped in the AI space right now? And even more importantly, what do you think is underhyped? I am, as I said, like super optimistic in different things that are going in AI. So I wouldn't say overhyped, but I feel kind of misunderstood is the concept of multi-agents. People have this notion of like, I have this incredibly complex problem. Now I'm going to break it down into, hey, you're this agent, take care of this. you're this agent take care of this and now if I somehow connect all of these agents they think they're the agent utopia and it's never the case that there are incredibly successful multi-agent systems that are built right like there's no doubt about that but I feel a lot of it comes in terms of how are you limiting the ways in which the system can go off tracks and for example like if you're building a supervisor agent and there are like sub-agents that actually do the work for the supervisor agent, that is a very successful pattern.

1:02:24But coming with this notion of I'm going to divide the responsibilities based on functionality and somehow expect all of that to work together in some sort of gossip protocol, that is extremely misunderstood that you could do that. I don't think current ways of building and current model capabilities are right there in terms of building those kind of applications. I feel that is kind of misunderstood than overrated. Underrated, I feel it's hard to probably believe, but I still feel coding agents are underrated in the sense that I feel like you can go on Twitter and you can go on Reddit and you see a lot of chatter about coding agents, but talking to an engineer in any random company, especially outside of the area, you can see the amount of impact this coding agents can create and the penetration is very low.

1:03:17So I feel like 2025 and 2026 is going to be like an incredible year for optimizing all of these processes. And I feel that is going to be creating a lot of value with AI. That's really interesting on that first point. So the idea there is you'll probably be more successful building and using an agent that is able to do its own sub-agent splitting of work versus like a bunch of, say, codex agents where you do this task, you do that task? You can have agents to do these things and you as a human can orchestrate it or you can have like one larger agent that is going to orchestrate all of these things.

1:03:52But letting the agents communicate in terms of peer-to-peer kind of protocol and then especially doing this in a customer support kind of use case is incredibly hard to control what kind of agent is replying to your customer because you need to shift your guardrails everywhere and things like Yeah. Okay. Great picks. Okay. Ash, what do you got? Can I say evals? Will I be canceled? On which category? Which bucket do they go? Overrated. Overrated. Okay. Go for it. We won't let you get canceled. Just kidding. I think evals are misunderstood. They are important folks. I'm not saying they're not important.

1:04:27But I think just this, I'm going to keep jumping across tools and going to pick up and learn if new tool is overrated. I still am old school and feel like you would really need to be obsessed with the business problem you're trying to solve. AI is only a tool. Try to think of it that way. Of course, you need to be learning about the latest and greatest, but don't be so obsessed with just building so quickly. Building is really cheap today. Design is more expensive. Really thinking about your product, what you're going to build. Is it going to really solve a pain point? Is what is way more valuable today?

1:05:04And it will only become more true in the near future. So really obsessing about your problem and design is underrated and just rote building is overrated, I guess. Awesome. Okay. Similar sort of question. From a product point of view, what do you think the next year of AI is going to look like? Give us a vision of where you think things are going to go, say, by the end of 2026. Yeah, I feel there's a lot of promise in terms of this background agents or proactive agents who is like they are going to like basically understand your workflow even more. If you think of like where is AI failing to create value today, it's mainly about not understanding the context.

1:05:48And the reason that it's not understanding the context is it's not plugged into the right places where actual work is happening. right and as you do more of this you can give the agent more of context and then it starts to see the world around you and understand what is the what is the set of metrics that you're optimizing for or what are the kind of activities that you're trying to do it is a very easy extension from there to actually gain more out of it and then let the agent prompt you back we already do this in terms of chat gpt pulse which kind of gives you this daily update of things you might care about and And it's very nice to actually have that like jog your brain up in terms of, oh, this is something that I haven't thought about.

1:06:26Maybe this is good. And now when you extend this to more complex tasks, like a coding agent, which says that like, okay, I have fixed five of your linear tickets. And here are the patches just to review them at the start of your day. So I feel that is going to be like extremely useful. And I see that as like a strong direction in which like products are going to build in 2026. That is so cool. So essentially agents kind of anticipating what you want to do and getting ahead of you. And here's, I've solved these problems for you. Or I think this is going to crash your site. Maybe you should fix this thing right here.

1:06:56Or I see the spike here and let's refactor a database. Amazing. What a world. Okay. Ash, what do you got? I'm all in for multimodal experiences in 2026. I think we have done quite some progress in 2025. And not just in terms of generation, but also understanding. Until now, I think LLMs have been our most commonly used models. But as humans, we are multimodal creatures, I would say. Language is probably one of our last forms of evolution. As the three of us are talking, I think we're constantly getting so many signals. I'm like, oh, Lenny is not in his head, so probably I would go in this direction.

1:07:34Or Lenny's bored, so let me stop talking. So there's a chain of thought behind your chain of thought and you're constantly altering it with language that dimension of expression is not explored as well. So if we could build better multimodal experiences, that would get us closer to human-like conversation richness. And yeah, I think, and just you will also, just given the kind of models, there's a bunch of boring tasks as well, which are ripe for AI. if multimodal understanding gets better. There are so many handwritten documents and really messy PDFs that cannot be passed even by the best of the models as of today.

1:08:15And if it's possible, there'll be so much data that we can tap into. Awesome. I just saw Demis from DeepMind, AI, Google, whatever they call the whole org, talking about this, where he thinks that's going to be a big part of where they're going, combining the image model work, the LLM and also their world model stuff, Genie, I think is what it's called. So that's going to be a wild, wild time. Okay, last question. If someone wants to just get better at building AI products, what's just maybe one skill or maybe two skills that you think they should lean into and develop? I think we did cover a bunch of best practices for AI products, which is start small, try to get your iteration going well and build a flywheel and all of that.

1:09:02But again, if you kind of look at it at a 10 ,000 feet level for anybody building today, like I was saying, implementation is going to be ridiculously cheap in the next few years. So really nail down your design, your judgment, your taste and all of that. And in general, if you're building a career as well, I feel for the past few years, your former years, say the first two, three years of building your career is always focused on execution, mechanics, and all of that. And now we have AI that could help you ramp pretty quickly and post that. I mean, after a few years, I think everybody's job becomes about your taste, your judgment, and kind of, you know, what is uniquely you.

1:09:49I think nail down on that part and try to figure out how you can bring in that kind of a perspective. It doesn't have to, mean that you should be significantly older, have years of experience. We recently hired someone and we use this very popular app for tracking our tasks, right? And we've been using it for years and we pay a high subscription fee for it. And this guy just came with his own white coded app to the meeting. He onboarded us to all of it and he's like, okay, let's start using this. And I think that kind of agency and that kind of ownership to really rethink experiences is what will set people apart.

1:10:27And I'm not being blind to the fact that wipe-coded apps have high maintenance costs. And maybe as we scale as a company, we have to replace it or we have to think of better approaches. But given that we're a small-sized company now, I was really shocked because I never thought of it. If you've been used to working in a certain way, you associate a cost with building. And I feel like folks who grew up in this age have a much lower cost associated in their mind. They just don't mind building something and going ahead with it. They're also very enthusiastic to try out new tools. That's also probably why AI products have this retention problem because everybody's so excited about trying out these new tools and all of that.

1:11:08But essentially, having the agency and ownership, and I think it's also going to be the end of the busy work era. If you can't be sitting in a corner doing something that doesn't move the needle for a company, you really need to be thinking about, you know, end-to-end workflows, how you can bring in more impact. I think all of that would be super important. That reminds me, I just had Jason Lemkin on the podcast. He's very smart on sales, go to market, run Saster, and he replaced his whole sales team with agents. He had 10 sales people, now he has 1.2 and 20 agents. And one of the agents, it was just tracking everyone's updates to Salesforce and kind of updating it automatically for them based on their calls.

1:11:50And one of the salespeople is like, okay, I quit. And it turned out he wasn't really doing anything. He was just sitting around and he's like, okay, this will catch me. I got to get out of here. So to your point about it, it'll be harder to sit around and tootle your thumbs, I think is really right. Yeah, I think to add on to that, I feel like persistence is also something that is extremely valuable especially given that anybody who wants to build something is the information is like at your fingertips even more than like the past decade right you can learn anything overnight and become that sort of like iron man kind of approach so if you're like having that persistence and like going through the pain of like learning this implementing this and like understanding what works and what doesn't work and as you are going through this like pain of like developing multiple approaches and then solving the problem.

1:12:42I feel that is going to be the real moat as an individual. I like to call it pain is the new moat, but I feel that is exactly super useful to actually have this, especially in building these AI products. Say more about this. I love this concept. Pain is the new moat. Is there more there? Yeah, I feel as a company, I mean, successful companies right now building in any new area, they are successful not because they're first to the market or they have this fancy feature that more customers are liking it, they went through the pain of understanding what are the set of non-negotiable things and trade them off exactly with what are the features or what are the model capabilities that they can use to solve that problem.

1:13:24This is not a straightforward process. There's no textbook to do this or there's no straightforward way or a known threaded path to be here. So a lot of this pain I was talking about is just going through this iteration of, like, okay, let's try this. And if this doesn't work, let's try this. And that kind of knowledge that you built across the organization or across your own lived experiences, I feel that pain is what translates into the mood of the company, right? This could be a product of evals or something that you built. And I feel that is going to be the game changer. That is awesome. It's like turning a coal into diamond.

1:14:03Yes. Okay. I feel like we've done a great job helping people avoid some of the biggest issues people consistently run into building AI products. We've covered so many of the pitfalls and the ways to actually do it correctly. Before we get to our very exciting lightning round, is there anything else that you wanted to share? Anything else you want to leave listeners with? We're obsessed with your customers. We're obsessed with the problem. AI is just a tool. and try to make sure that you're really understanding your workflows. 80 % of so-called AI engineers, AI PMs, spend their time actually understanding their workflows very well.

1:14:40They're not building the fanciest and the most cool models or workflows around it. They're actually in the weeds understanding their customers' behavior and data. And whenever a software engineer who's never done AI before hears the term, look at your data, I think it's a huge revelation to them, but it's always been the case. You need to go there, look at your data, understand your users, and that's going to be a huge differentiator. It's a great way to close it. It's not the AI isn't the answer. It's a tool to solve the problem. With that, we have reached our very exciting lightning round. I've got five questions for both of you.

1:15:20Are you ready? Yay. Yes. All right. So you can both answer them. You can pick one which you want to answer either way up to you. what are two or three books you find yourself recommending most to other people? For me, it's this book called When Breath Becomes Air, Lenny. It was written by Paul Kalaniti. I think he was an Indian origin neurosurgeon who was diagnosed with lung cancer at 31 or 32. And the whole book is his memoir and just is written after he was diagnosed. And it's really beautiful, especially because I read it during COVID and all we ever wanted to do during COVID was stay alive.

1:15:57There are a bunch of really nice quotes within the book as well. But I remember one of them, he was kind of arguing against a very popular quote by Socrates, which is the unexamined life is not worth living or something like that. And which means you really need to be thinking about your choices. You need to understand your values, your mission and all of that. And Paul says, if the unexamined life is not worth living, was the unlived life worth examining? which means are you spending so much time just understanding your mission and purpose that you've forgotten to live? And I think everybody who's staying in the AI era and building and continuously going through this phase of reinventing themselves need to take a pause and live for a bit, I guess.

1:16:43They need to stop evaling life too much. I was going to say that. That's where my mind went. You've got to write some evals for your life. Oh, my God. We've gone too far. Yeah. Yeah. That's my favorite book. I like more of science fiction books. So I really like this three-party problem series. It's like a three-book series. It has elements of like grander than science fiction, life outside earth and how it impacts like human decision-making process. And it also has like elements of geopolitics and how much important or like valuable abstract science is for human progress. And then when that gets stopped, It's not noticeable in everyday life, but it can cause devastating effects.

1:17:27So I feel like AI helping in these areas, for example, is going to be extremely crucial. And that book is a nice example of what could happen otherwise. Completely agree. Absolutely. Love. Might be my favorite sci-fi book, except our series even. And it's three. I have to read them all three, by the way. I find that it only got really good about one and a half books in. So if anyone's tried it and like, what the heck is going on here? Just keep reading and get to the middle of the second one and then it gets mind blowing. yes uh if you love soft guy and you're an ai you got to read this book called a fire upon the deep by uh vernon vinge check it out it's incredible i saw noah smith on his newsletter recommend this book and there's like a whole there's like sequels to it but this is the one it's so incredible and it's actually turns out it's about agi and super intelligence and all these things and it's just like so epic.

1:18:19And no one's heard of it. There you go. I'm giving you one back. Okay, next question. What's a favorite recent movie or TV show that you've really enjoyed? I started rewatching Silicon Valley. And I think it's so true. It's so timeless. Everything is repeating all over again. Anybody who's watched it a few years ago should start rewatching it. And you'll see that it's really similar to everything that's happening right now with the AI wave. That's a good idea to rewatch it. I love that their whole business was like an algorithm to compress like a compression algorithm it's like maybe a precursor to lms in some small way oh my god all right kareet what you got uh i'm gonna drag this and say a lot a movie or a tv show but there's this game i picked up recently called expedition 33 uh it has nothing to do with ai but it's an incredibly incredibly well-made game in terms of the gameplay or like the movie and the story and the music uh it's been amazing i love that you have time to play games that's a great sign.

1:19:12I love that. Someone opened AI, I'm just imagining there's nothing else going on except just coding. Yeah, it has been incredibly hard to find time for that. That's good. That's a good sign. I'm happy to hear this. Okay, what's a favorite product that you've recently discovered that you really love? For me, it's Whisperflow. I think I've been using it quite a bit and I didn't know I needed it so much. The best part is it's a conceptual transcription tool, which means If you go to Codex and start using Whisperflow, it starts identifying variables and all of that. And it's so seamless in terms of transcription to instruction.

1:19:48You could say something like, I'm so excited today. Add three exclamation marks. And it seamlessly switches. It adds those three exclamation marks instead of writing, add three exclamation marks. And I think it's pretty cool. If you're not using it, you should try it. I'll do a plug. Get Whisperflow for free for an entire year. For a year. For free. By becoming an annual subscriber of my newsletter. That's how I got access to it, Lenny. There we go. It's like, I think I pitched this deal. I think people don't truly understand how incredible this is. They're like, no way, this is real. It's real.

1:20:22And 18 other products. Lenny'sProductPass.com. Check it out. Moving on. Kiriti. Awesome. I actually am a stickler for productivity. I keep experimenting new CLI tools and things which can make me faster. So I feel like Raycast has been amazing. I've discovered all this new shortcuts that you can use to open different things, type in shortcut commands and things like that. And Caffeinate is another thing that I've recently discovered from my teammates. It helps you prevent Mac from sleeping so you can run this really long codex task for like four or five hours locally, let it build the thing, and then you can wake up and be like, okay, this is good.

1:21:00I like this. That's hilarious, that combo. Codex and Caffeinate. You guys need to build that yourself, an open-air version of that. Or the Codex agent should just keep your Mac from sleeping. That's so funny. By the way, Raycast, also part of Lenny's product pass. One year free of Raycast. We weren't. Lenny didn't tell us these books. These are actually our favorite products. These are just two of 19 products. No caffeinate, though. I don't know if that's even paid. Okay, let's keep going. Do you have a favorite life motto that you find yourself coming back to in work or in life? For me, I think this is what my dad told me when I was a kid and it's always stuck, which is they told it couldn't be done, but the fool didn't know it.

1:21:44So he did it anyway. I think be foolish enough to believe that you can do anything if you put your heart to it, especially now because you have so much data at your hand that could be pointing towards the fact that you probably will be unsuccessful. How many podcasts made it to more than a thousand subscribers? Or how many companies hit more than one million Yara? And there's always data to show you that you want to be successful. But sometimes just be foolish and go ahead with it. That's great. Yeah, for me, I am more of an overthinker. So I really like this quote from Steve Jobs that you can only connect the dots looking backwards.

1:22:21So it's a lot of the times there are like numerous choices and you don't really know the optimal one to pick. but life's life works in ways that you can actually see back and be like oh these are actually beautiful in terms of how i would transition so i feel like that is extremely useful and like you know keep moving forward keep experimenting final question whenever i have two guests on the podcast at once i like to ask this question what's something that you admire about the other person i think with kiriti um it's about he's he's pretty calm and uh very grounded and he's always been my sounding board.

1:22:59I can throw a ton of ideas at him and he always comes up with, he's able to anticipate the kind of issues that might land into and he's extremely kind and lets his work speak instead of actually doing a lot of talking, I guess. If I had to pick one, I think he's the most incredible husband. Reveal, little do people know. So we've been married for four years and been the most beautiful four years of my life. Oh, wow. Okay. How do you follow that? Yeah, it's super hard to follow that. I would say I am extremely privileged in terms of working with like really smart people in great companies in the Silicon Valley.

1:23:43and I feel the unique thing that stands with Aishwarya across like any other smart folks I've worked on is like she has this really amazing knack of teaching and like explaining something in a very understandable and easy to comprehend way and that combined with persistence is like super useful especially in this fast moving AI world that we are in in the sense that there's so many new things coming up it feels overwhelming but when I hear her talk about like this is how you make sense of this entire thing this is where it plugs in i feel like oh that is so simple like i can also do that so she empowers a lot of people by simplifying things and you know like uh explaining things in the most understandable way so i feel that is like an incredible quality amazing how sweet i gotta do this all the time i need more more yes to that was that was great okay uh final questions where can folks find stuff that you're working on find you online talk about your share your course link and then just how can listeners be useful to you I write a lot on LinkedIn.

1:24:44So if you want to listen to Pragmatists who've been in the weeds working on AI products and what they're seeing, you can follow my work. We also have a GitHub repository with about 20k stars. And that repository is all about good resources for learning AI. It's completely free. and if you like what we spoke today we also run a super popular course we leave a link to it on building enterprise AI products and the course is a lot about unlearning mindsets and following like a problem first approach instead of a tool first or a hype first approach so you can check that out as well and if you don't want to do the course we write a lot we give out a lot of free resources we have free sessions so make sure you follow our work yeah I would also add that I you can also to find me on LinkedIn.

1:25:30I don't like write a lot, I guess, but I'm super excited to just talk to any complex product that you're building. And if you have thoughts on like how you can use coding agents to make your life better, or how, what are the problems that you're seeing? Always my DMs are open and like, we can have a great discussion. Awesome. Well, Kirithi and Ash, thank you so much for being here. Thank you so much. Thank you, Lenny. This was so much fun. So much fun. Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.

1:26:05Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny's podcast.com. See you in the next episode.