Anthropic’s CPO on what comes next | Mike Krieger (co-founder of Instagram)

24:38I think there's still value, a lot of value in two things. One is making this all comprehensive. I think we've done an okay job. I think we can do a much better job of making this conference. is what's still like the difference between somebody who's really adept at using these tools in their work, and most people is huge. And I mean, maybe that's the most little answer to your earlier question around, like what skills to learn? That is a skill to learn and use it. And the same way that I remember, I, we did like computer -lock class, that was in like middle school. I remember being like really good at Google.

25:09And that was actually a skill back in the day, you know, like to think in terms of like this information is out there. How do I query for it? How do I do it? And I think it actually was like a, an advantage of the time, of course. Now Google is pretty good at figuring out where you're trying to do it. If you're only in the neighborhood, there's less of that research kind of need. But I still think that's a necessary part of good product development, which is the capabilities are there. And even if the cloud can create products from scratch, what are you building and how do you make it comprehensible?

25:35Like, still hard. Because I think that gets at this much deeper empathy and understanding of human needs and psychology. I was a human -community interaction major. I've still been talking my book here. I still feel like that is a very, very, very, very necessary skill. That's one. Two is, and this, you know, straight to a callback to another one of your guests. Like, strategy, like, how we win, where we'll play, like, figuring out where exactly you're going to want to, like, all the things that you could be spending your time or your tokens or your computation on, like, what, what, what you want to actually go and do.

26:11You could be wider, probably, than you could before, but you can't do everything. And even from an external perspective, if you're seen to be doing everything, it's way less clear around how you're positioning yourself. Strategy, I think, is still the second piece. And then the third one is opening people's eyes to what's possible, which is a continuation of making it understandable. But we were in a demo with a financial services company recently. And we were working on, here's how you can use our analysis tool, an MCP together. And you could see their eyes light up. You're like, okay, we call it overhang, right?

26:43Like the delta between what the models and the products can do and how it's being used on a daily basis, huge overhang. So that's where still like a very, very strong necessary role for product. Okay, that's an awesome answer. So essentially areas for product teams to lean into more is strategy, just getting better and better at strategy, figuring out what to build and how to win in the market, making it easier to help people understand how to leverage the power of these tools, the comprehensibility, and kind of along those lines is opening people's eyes to the potential of these sorts of things that their product can still help.

27:17Exactly. Awesome. So, kind of along those lines, actually, do you have any, just like, prompting tricks for people, things you've learned to get more out of clot when you chat with it? Sometimes it's funny because in some ways, we have the ultimate prompting job, which is to write the system prompt for clotting. and we publish all of these, which I think is another nice area of transparency. We are always careful in giving a prompting advice because at least officially, but I'm going to give you the unofficial version because you don't want things to become like, we think this works, but we're not sure why.

27:50I'll do small things in Cloud Code, and we actually do react to this very literally. I always ask it to like, if I want to use more reasoning, think hard, and it'll use a different flow. I usually start with that. Nudging, there's a great essay around like make the other mistake. Like if you tend to be too nice, can you focus on like, even if you're trying to be more critical or more blunt, you're probably not going to be the most critical blunt person in the world. And so with Clouds and Nizen, like, the Be Brutal Cloud, like, roast me, like tell me what's wrong with this strategy. I think, I know we were talking earlier about the, you know, Clouds thought partner around like critiquing product strategy.

28:25I think I, previously would say things like, you know, like what could be better on this product strategy? like just roast this product strategy and cloud's like a pretty nice, you know, entity, it's not gonna be, it's hard to push it to be super brutal but it forces it to be a little bit more critical as well. The last thing I'll say is, so we have a team called Apply Day Eye that does a lot of like work with our customers around optimizing cloud for their use case. And we basically took their insights and their way of working and we put it into a product itself. So if you go to our console or work bench, we have this thing called the prompt -improver where you describe the problem when you give it examples, and Cloud itself will agentically create and then iterate on a prompt for you.

29:05I find what comes out of that ends up being quite different than what my intuitions would have been for a good prompt. And so I'd encourage folks to also check that out, even for their own use cases, because while that tool is met for an API developer putting a prompt into the product, it's equally applicable for a person doing a prompt for themselves. Like, it'll insert XML tags, which no human is going to think to do ahead of time. I mean, actually, it's very helpful for Cloud to understand like what it should be thinking versus what it should be saying, et cetera. So that's another one is like, watch our prompt improve and then note that like, Cloud itself is a very good prompt or of Cloud.

29:37Awesome. OK, so we're going to link to that, the prompt improve or the core piece advice you shared earlier is just kind of do the opposite of what you would naturally do. So if you're like trying to be nice, just like be brutal, be like very honest and frank with you. Exactly. I find that works quite well. Like what are the thought patterns that I've fallen into that you want to break me out of? Hmm. I saw you guys just today maybe launched a Rick Rubin called Howard's live live coding. What's that all about? That was, you know, what I've heard about that. And then I reckon like this, a lot of the coalesce this week between model launch developer event and the way of code.

30:09We had our, one of our co -founders, Jack Clark is our, you know, head of policy and he got connected to Rick Rubin because he's been thinking a lot about coding, the future coding and creativity. And they've stayed in touch and, you know, I've got excited about this idea of like he's creating like art and visualizations with Claude and then he had these like ideas around like the way of the vibe coder and they put together this. Actually, I love the, I mean, I love almost everything, Rick Rubin. So like the aesthetic I think is just like so on point too. But yeah, it's just sort of like meditations for the right word, meditation on like creativity, working alongside AI, coupled with this like, with this like really rich, interesting visualizations.

30:49But just one of those things for like you know, internally, they're like, oh yeah, and we're doing this like, we're doing collaborative work. We're doing what? Like that is, that's amazing. I love the, I look at it briefly and there's like that meme of him, like just like thinking deeply, sitting on a computer with a mouse. Yes. And like ask yard, I think. It's totally, it's like ask yard five. I'm excited to have Andrew Luo joining us today. Andrew is CEO of one schema, one of our long time podcast sponsors, welcome, Andrew. Thanks for having me, Lenny. Great to be here. So what is new with one schema?

31:20I know that you work with some of my favorite companies like RAMP and VANSA and watershed. I heard you guys launched a new data intake product that automates the hours of manual work that teams spent importing and mapping and integrating CSV and Excel files. Yes, so we just launched the 2 .0 of one schema file feeds. We've rebuilt it from the ground up with AI. We saw so many customers coming to us with teams of data engineers that struggled with the manual work required to clean messy spreadsheets. FilePeds 2 .0 allows non -technical teams to automate the process of transforming CSV and Excel files with just a simple prompt.

31:56We support all the trickiest file integrations, SFTP, S3, and even email. I can tell you that if my team had to build integrations like this, how nice would it be to take this off our roadmap and instead use something like one schema? Absolutely Lenny, we've heard so many horror stories of outages from even just a single bad record. In transactions, employee files, purchase orders, you name it. debugging these issues is often like finding a needle in a haystack. Once schema stops any bad data from entering your system and automatically validates your files, generating error reports with the exact issues in all bad files.

32:30I know that importing incorrect data can cause all kinds of pain for your customers and quickly lose their trust. Andrew, thank you so much for joining me. If you want to learn more, head on over to oncegeema .co. That's oncegeema .co. Actually going back to kind of the beginning of your journey at Anthropic, Like, what's the story of you getting recruited in Anthropic? Is there anything fun there? The it all started and I actually sent my friend this text. Joel Loonstein, who I've known, he and I built our first iPhone apps together in 2007 when the App Store was just out and you could still make money by selling dollar apps on the App Store back in the day.

33:05And we were with the Stanford together and we were friends and we've stayed in touch over years and we've never gotten to work together since then. We just remained close. and I was coming out of the artifact experience. I was trying to figure out, do I start another company? I don't think so. I need a break from starting something from zero. Should I go work somewhere? I don't know. Like, what company do I want to go work at? And he reached out and he's like, look, I don't know if you'd at all consider joining something rather than starting something. But we're looking for a CPO. Would you be interested in chatting?

33:33And at that time, Cloud Three had just come out and I was like, okay, this company's clearly got a good research team. The product is so early still. And it was like, great. I'll take the meeting and first of all, Danny Ellos, one of the co -founders in the president and then in the topic. And just from the beginning, I was like, a breath of fresh air, like very little grandiosity coming off the founders. Like they just were really, I mean, they're clear at about what they're building. They know what they don't know. Like how many times I talk to Daria, he's like, Daria's like, look, I don't know anything about product, but here's an intuition.

34:06I haven't, usually the intuition's really good and leads to some good conversation. Then they got intellectual honesty and like, kind of shared view of what it means to do AI in a responsible way, just resonated. I kept having this feeling in these interviews, this is the AI company I would have hoped to have founded if I had founded an AI company, and that's kind of the bar around, if I'm going to join something, that should be where I'm going to go. But what I realized, I actually hadn't joined a company since my first internship in college basically. I was like, how do I onboard myself? How do I get myself up to speed?

34:42How do I balance making sweeping changes versus understanding what's not broken about it overall? And looking back on a year, I think I made some changes too slowly. I think there was ways we were organized in a product that I could have made a change earlier. And I didn't appreciate how much a couple of really key senior people can shape so much of product strategy. I'll walk him back to Cloud Code. like, cloud code happened because Boris, who actually was a, Boris Turni, he was an Instagram engineer and like one of our senior ICs there, we were full up to bit, was like, started that project from scratch, internal first, and then we like got it out and then shipped it and like, that's the power of like one or two really strong people.

35:25And I made this mistake about it. We need more headcount, and we do, like I think there's like more work that we need to do, and there's like things that I wanna be building, but more so than that, we need a couple of like, almost founder type engineers. That maybe connect back to our question on like what skills are useful and how does product development change? I still and maybe even more so I'm a huge believer in like the founding engineer tech lead with an idea and pair them with the right like design and product support to like help them realize that I'm like 10 times more believer in that than before.

35:56I actually asked people on Twitter what to ask you. I had a this conversation in the most common question surprisingly was why did you shut down artifacts? And I also wondered that because I loved Artifacto. I was a power user. I was just like, this is exactly finally a news app that I love that it's giving me what I want to know. So I guess just what happened there at the end? I still really miss it too because I didn't find a replacement. And I think I substituted it by like visiting individual sites and kind of keeping things up that way. And it didn't not really the same, especially on the long till like I think we got right with Artifacto.

36:29If people didn't play with it before, it was, you know, we really tried to not just recommend like top stories. they were part of it, but really, like, if you were interested in Japanese architecture, like, you could pretty reliably get really interesting stories about Japanese architecture every day, you know, whether that's from a, you know, dwell or from architectural gaiters or from a really specific blog that we found that somebody recommended to us. Like, it captured some of that Google reader joy of, like, content discovery of the deeper web. Our headwinds were a couple. One of them was just, just mobile websites have really taken a turn.

37:03I don't blame any individuals for this. I think it's the market dynamics of it. But we put so much time or designers with SkyGunnerGrey has been normally that for FlexityNow. The app experience I was so proud of, but when you click through, it was like the pressure is on these mobile sites and these mobile publishers would be like, sign up for our newsletter. Here's a full screen video ad. It was just very jarring. and we didn't feel like it ethically made sense for us to do a bunch of ad blocking. Because then you're like, sure, you can deliver a nice experience for people, but you sort of, you know, that doesn't feel like it's playing fair with the publishers.

37:38And at the same time, like, the actual experience wasn't good. So the mobile web deteriorating, which makes me very sad, but I think it was part of it. Two was like, you know, Instagram spread in the early days because people would take photos and then post them on other networks and tell friends about it. And there was like, there's really natural, like, how did you do that? I wanna do it. News was very personal. I can't tell you how many people would be like, I love artifacts. I'm like, do you tell anybody about it? And they're like, yeah, I told one person. And then it's like, it didn't have that kind of spread.

38:06And any attempt that we had to do it felt kind of contrived. Like, oh, we'll wrap all the links in like artifact .news. But we don't want interstitial things. In some ways, the sound's very puritanic. I don't mean it's sound this way. But there were lines that we didn't want to cross, so that just it just felt ethically not us that I've seen other news kind of players do more of. And maybe if we had done that, it would have grown more. But I don't think that's the company we wanted to have built in the end of the way. I don't think we were the founders to have built it. And the third one, which is an underappreciated one, is we started at Mid -COVID, which meant that we were fully distributed.

38:41And I think there were major shifts that we would have wanted to make both in the strategy and the product and the team. And it's really hard to do that if you are all fully remote. Like nothing replaces like the Instagram days of like we went through some you know hard times like Ben Horowitz called the like you know We're F. It's over you know kind of moments and I My favorite not this divide type to fun like I wouldn't say that my favorite memories because they weren't happy ones But like memories I like really stayed with me with Instagram was like me and Kevin at Takuya Kanku and on Market Street eating burritos at literally 11 p .m.

39:14being like How are we gonna get out of this how are we gonna work through this like and And that's, you assume, is not a good replica for that. You tend to let things go or things build up over time. So the confluence of those three things, we kind of entered, I guess, 2024 and said, like, there is a company to be built in the space. I'm not sure where the people would have built it. This current incarnation we love, but it's not growing. Like, the way I put it, it's like 10 units of input in for one unit of output versus the other way around. Like, if we put blood sweat and tears into the product and like launch something we were proud of and like metrics would barely move them.

39:47Like, the energy is not present in this product, in this system. And so are we gonna like, expend another year or two, and then go off and fundraise only to find that this is the case? Or do we like, call it and see that it's run its course and, you know, try to find a home for it, et cetera? So that was the, the confidence on it. And the, so I feel like this opportunity cost of like, AI is starting to change everything. We have an AI powered news app, but is this the like, maximal way in which like, we're gonna be able to impact this? It felt like the answer was, was increasingly no. But it was hard.

40:16I mean, in the end, I was really at peace with the decision, but it was like a conversation that went on for a couple of months. On that note, just how hard was it? Because you, you know, it's, there's an ego component to it. Like, oh, I'm starting my new company. It's going to be great. And then, and then you end up having to shut it down. Just how hard is that as a very successful previous founder, shutting something down and then not working out? Yeah. I mean, I think when we started it, one of the conversations was like, like, what is the bar to success here? And do we want it to be something other than Instagram DAU, which is just an impossible bar.

40:47Like only one company since, maybe two, right? You could say maybe ChatGBT and TikTok have like reached that kind of like mass consumer adoption, starting a news app. Like most people are not like daily newsreaders even, right? And so we knew that we weren't pursuing that size of like usage, at least with the kind of first incarnation. But we did have like an idea of like building out complimentary products over time that all -use personalization and machine learning. We didn't even call it AI at the time. This is 2021 back then. Yeah, yeah. It was called machine learning back then. Yeah, it was called machine learning still.

41:17And so in shutting it down, you know, it's like, you kind of know it when you see it in terms of like user growth and traction. And I wasn't expecting Instagram growth, but I was expecting or hoping for or looking for something that like felt like at its own legs under it and it could continue to continue to compound. I was really positively surprised by how supportive people were when we announced it. There was very little, there was a bit of like, I told you so, which like, sure, anything launching you could be like, this is not going to work and you're right most of the time because most things don't work.

41:50There was actually very little of that. Most people, the universal reception at least as I received it was kudos for calling it when you saw it and not like kind of protracted, you know, doing this for a long time and I've talked to founders since then that have been like, yeah, I like probably would have like taken this thing out in for another six months. But, so what you guys did, realized we were barking up the wrong tree, made the call and was like, that, you know, if that, if that frees up people to go work on them more interesting things, that's like, I feel like that's like a good, like a see for, for artifact to have.

42:20But for sure, there was like a legal, an ego, bruise of, you know, like, are people, is it true that you're only as good as your last game, you know, if I'm a huge sports fan, right? So like, is that true or, you know, is there something more of a time? I'm very competitive, but primarily with myself. and so I'm always trying to find the next thing that I want to go and do that's hard and I unfortunately that probably means that we're often and I'll feel dissatisfied with the most recent thing that I did but hopefully that yields good stuff in the end. Yeah, I think just the trajectory you went on after shows that it's okay to shut down things that you're working on.

42:52Okay, so you mentioned Chad Gbt, I wanted to chat about this a bit. So there's something really interesting happening. So on the one hand, you guys are doing some of the most innovative work in AI. You guys launched MCP, which is just like, I don't know, the fastest growing standard of any time in history that everyone's adopting. Clawed, powered, and unlocked. Essentially, the fastest growing companies in the world, cursor, lovable on both and all these guys. I had them on the podcast and there are like, when Clawed, I think 3 .5 came out, saw it. It was just like, that's, oh, made this work, finally.

43:24On the other hand, it feels like Chatchy PT is just winning in consumer mind share. When people think AI, especially outside tech, it's just like Chatchy PT in their mind. So let me just ask you this, I guess first of all, do you agree with that sentiment? And then too, as a kind of a challenger brand in the I space, just how does that inform the way you think about product, that strategy and mission and things like that? Yeah, I mean, you look at the sort of like public adoption or like you ask people like, oh, you know, like if you, if you Jimmy Kimmelman on the street kind of thing, you know, like name and AI company, I bet they would name.

43:59And actually, I'm not even sure they name opening. I did probably name chat GPT because that brand is the kind a lead brand there as well. And I think that's just the reality of it. I think that, you know, I reflect on my year, there's, I think maybe two things are true. One is like consumer adoption is really lightening in a bottle and we saw it at Instagram. So like almost maybe more than anybody, I can look internally and say like, look, we'll keep building interesting products. One of them may hit, but to kind of craft an entire product strategy around like trying to find that hit and is probably not why as we could do it.

44:32And maybe cloud can help come up with the fullness of things, but I think we'd miss out on opportunities in the meantime. And then instead, look yourself in the mirror and embrace who you are and what you could be rather than who others are as maybe the way I've been looking at it, which is, it was super strong developer brand. People build on top of us all the time. And I think also I have a builder brand, the people who I've seen react really well to cloud externally. Maybe the Rick Rubin connection, as some resonance here as well. Can we lean into the fact that builders love using cloud? Those builders aren't all just engineers and they're not just all entrepreneurs starting their company, but they're people that like to be at the forefront of AI and are creating things.

45:12Maybe they didn't think of those as engineers, but they're building, I got this really nice note from somebody in Toronto on topic, because on the legal team and he was building the spoke software for his family and connected them in a new way. I was like, this is a glimmer of something that is that we should lean into a lot more. And so I think what I've, you know, and this is actually, you know, connecting back to a thing like Clouds being helpful here. Like a lot of what I've been thinking about, like going into the second half of the year and beyond is like, how do we figure out what we want to be when we grow up versus like what we currently aren't or wish that we were or like see other players in the space being?

45:47I think there's room for several, like, generationally important companies to be built in AI right right now that's almost a truism given like the sort of adoption and growth that we've seen, you know, add -on -thropic but also across open AI and also places like Google and Gemini. So like, let's figure out what we can be uniquely good at that place to the personality of the found, like this, all the things come together, right? Like the personality of founders, the like quality of the models, the things the models tend to excel at, which is like, agentic behavior and coding. Like, great. Like, there's a lot to be done there.

46:18Like, how do we help people get work done? How do we let people delegate hours of work to cloud? and maybe there's fewer direct consumer applications on day one, I think they'll come, but I don't think that spending all of our time focused on that is the right approach either. And so I came in and everybody expected me to just go super, super hard on consumer and make that big thing. And again, with the other mistake, instead I spent a bunch of time talking to financial services companies and insurance companies and others to building up the API. And then lately I spent a lot more time with startups and seeing all the people that have grown off of that.

46:50And I think the next phase for me is like, let's go spend time with like the builders, the makers, the hackers, the tinkerers, and like make sure we're serving them really well. And I think good things will come from that. And that feels like an important company as we do that. Hmm. So essentially it's differentiated and focused lean into the things that are working. Don't try to just like beat somebody at their own game. Exactly. Super interesting. So kind of along those lines, a question that a lot of AI founders have is just like, where's a safe space for me to play where the foundational model companies are going to come squash me.

47:22So I asked Kevin Wheel this and he had an answer and I noticed looking back at that conversation, he mentioned Windsor Fala. I was like, wow, let's get really love Windsor and then like a week later they bought Windsor. So it all makes sense now. So I guess the question just is just where do you think AI founders should play where they are least likely to get squashed by folks like OpenAnne and Theroppa. And I'm sorry you guys are gonna buy Cursor. I don't think we're gonna buy Cursor. Cursor's very big. We will have worked together. A few thoughts on this. And it's a question I've gotten. You know, we like to do these kind of founder days with, you know, whether it's, you know, Menlo Ventures, who have a lot of investors, and I just know what it's like.

48:05We've done YC. We've done these like founder days. And it's like the question that is on all of these founders minds understandably. So I think things that are going to, I can't promise this as like a five to ten year thing, but at least like one to three years things that feel defensible or durable. One is understanding of a particular market. I spent a bunch of time with the Harvey folks and they really like, they showed me some of their U .I. I'm like, what is this thing? They're like, oh, this is a really specific flow that like lawyers do and like you never would have come up with it from scratch.

48:33And it's like not like, you could argue about whether it's like the optimal way to get done things done, but it is the way that they get things done. And here's how AI can help with that. And so differentiated industry knowledge, biotech. I'm excited to go and partner with a bunch of companies that are doing good stuff around AI and biotech. And we can supply the models. And so a Friday I to help make those models go well. And I've been dreaming about at what point do this live equipment. I'll get an MCP and that you can then drive using cloud. There's all these cool things to be done there. I don't think we're going to be the company to go build the intense solution for labs.

49:07But I want that company to exist and I want to partner with it. You know, domains like legal, again, healthcare, I think there's a lot of like very specific kind of compliance and things. These are things that necessarily sound sexy out the gate, but there are like very large companies to go and be built there. So that's number one. Paired with that is like differentiated go -to -market, which is the relationship that you have with those companies, right? Like, do you know your customer at those companies? Like, one of our product leads, Michael is always talking about like, no, not, don't just know the company you're selling to, but know the person you are selling to at the company.

49:40Are you selling to the engineering department because they're trying to pick which AIL LM to build on top of, or API to build on top of? Let's go talk to them. Is it the CIOs? The CTO? Is it the CFO? Is it the general council? So, companies with deep understanding of who they're selling to is the other piece to. What's interesting there is it's probably hard to build that empathy in a three -week or three -month accelerator, but you maybe can start having that first conversation and build that I remember maybe you came from that world or you're co -founding somebody who came from that world. Then the last one is like, there's tremendous power and distribution and reach to being chat GPT and having hundreds of millions or billions of users.

50:18There's also people having an assumption about how to use things. And so I get excited about startups that will get started that have a completely different take on what the form factor is by which we interface with AI. And I haven't seen that many of them yet. I want to see more of them. I think more than we'll get created with some things like our new models. But the reason that that's an interesting space to occupy is do something that feels like very advanced user, very power user, very weird and out there at the beginning, but could become huge if the models make that easy. And it's hard for existing incumbents to adapt to because people already have an existing assumption about how to use their products or how to adapt to them.

50:59So those are my answers. I don't envy them. I would probably be asking those questions If I was starting a company in the AI space, maybe the part of the reason why I wanted to join a company rather than start one, but I still think that there are, and maybe like here's fourth, like don't underestimate how much you can think and work like a startup and feel like it's you against the world, it's existential that you go solve that problem, that you go build it. It sounds a little cliché, but it's like, it's all we had at Instagram, we were two guys and we were like, let's see what we can do. And in our effect, we were six people for most of that time and you know, every day felt like it's existential that we get this right, we need to win.

51:38And you can't replicate that and you can't instill that with OKRs. Like you just have to feel it. And that is a way of working rather than like area of building, but it's a continued advantage if you can harness it. I love that you still have such a deep product founder since there as you're building products for this very large company now. Kind of on the flip side of this, people working with your models and APIs. So imagine there's some companies that are finding ways to leverage your models and APIs to their max and are really good at maximizing the power of what you guys have built. And there's some companies that work with your APIs and models that haven't figured that out.

52:17What are those companies that are doing a really good job building on your stuff, doing differently that you think other companies should be thinking about? I think being willing to build more at the edge of the capabilities and basically break the model and then be surprised by the next model. I love that you, you said that the companies were like three, five was the one that finally made them possible. Those companies were trying it beforehand and then hitting a wall and you're like, the models were almost good enough. They're okay for this specific use case, but they're not generally usable and nobody's going to adopt them, you know, universally, but maybe these like real power users are going to try it out.

52:55But those are the companies that I think continuously are the ones where I'm like, yep, they get it. They're really pushing forward. We ran a much broader early access program with these bottles that we had in the past. And part of that was because there's this real, we can help climb on these evaluations and talk about, we've mentioned how bench and terminal bench, whatever. But customers ultimately know, like, cursor bench, which doesn't exist other than their usage and their own testing, et cetera, is like the thing that we ultimately need to serve, not just cursor, but man is bench, right?

53:27If man is using our models and Harvey Bench, like those things and customers know way better than anybody. And so I would say that's two things. Like one is pushing the frontier of the models and then having a repeatable process that actually goes back to our summit conversation. Like repeatable way to evaluate how well your product is serving those use cases and how well if you drop a new model in, is it doing it better or worse? Some of it can be classic AB testing. That's fine. Some of it may be internal evaluation. Some of it may be capturing traces and be able to rerun them on with a new model.

54:00Some of it is vibes. We're still pretty early in this process, and some of it is actually trying it. And one of my favorite early access quotes was the founder of our engineers screaming next to him. What? This model, I've never seen this before, it was like, cool. We're going to engender that feeling and things, but you're not going to be able to feel that unless you have a really hard problem that you're asking the model repeatedly. So those are the things that I think kind of differentiate those those those companies that are maybe earlier in their journey of adoption versus the later ones.

54:31I can't help but ask about MCP if you like that's just so hot and just like Microsoft had their announcement recently that they're like now it's part of the OS window. Just what role do you think MCP was will play in the future of product going forward to be? I think as the non -reistature in the room, I get to have fake equations rather than real ones and my like fake equation for like utility of AI products. It's three part one is model intelligence. The other the second part is context and memory and the third part is like applications in UI. And you need all three of those to converge to actually be a useful product in in AI.

55:07And you know, model intelligence, we got a great research team. They're focused on it. There's great great models being released. the middle piece is what MCP is trying to solve, which is for context and memory. Like the difference between, I'll go back to my product strategy example, like, hey, like, you know, let's talk about it in topics product strategy. It's gonna maybe go out on the web, like, versus, here's like several documents that we worked on internally. And then, you know, use MCP to talk to our Slack instance and figure out what conversations are happening. And then, like, go look at these documents and Google Drive, like that, the difference between like the right context and it's like the entirely the the the difference between like a good answer and a bad answer.

55:44And then the last piece is are those integrations discoverable? Is it right? Is it easy to like create repeatable workflows around those things? And that's like I think a lot of the interesting product work to be done in AI. But MCV really tried to tackle that middle one, which is we started building integrations and we found that every single integration that we were building, we were rebuilding from scratch in a non sort of repeatable way. And like full credit to to our engineer is just an end -a -bit and they said, well, you know, what if we made this a protocol and what if we made this something that was repeatable?

56:13And then let's take it a step further. What if instead of us having to build these integrations, if we actually popularize this and people really believe that they could build these integrations once and they'd be usable by a cloud and eventually chat GPT and eventually Gemini knows like the dream, like when one more integrations get built and wouldn't that be good for us? You know, I think channeling a lot of them, it's like an old commoditizer compliments Joel Spolski essay. You know, it's like we're building great models, but we're not an integrations company. And the, you know, we're, as you said, the challenger, like we're not going to get people necessarily building integrations just for us out of the gate.

56:46And let's feel like a really compelling product around that. The MCP really inverted that, which was, you know, it didn't feel like wasted work. And, and a few key people like Toby, I think, is a great example. A Shopify got it. Kevin Scott at Microsoft has like been really a just an amazing champion for MCP and I thought partner on this and I think the role going forward is, can you bring the right context in? And then also, once you get, as the team calls it, internally MCP, once you start seeing everything through the ISO MCP, I've started saying the things like, guys, we're building this whole feature, this shouldn't be a feature that we're building, this should just be an MCP that we're exposing.

57:22A small example of how I think even Anthropic could be a lot more MCPilled if you will is like, we've got these building in the product, like projects and artifacts and styles and conversations and groups and all these things, those should all just be exposed via an MCP. So, Claude itself can be writing back to those as well, right? Like, you shouldn't have to think about, like, uh, watch my wife out of conversation with Claude the other day and she was, she found she had generated some good output and she's like, great, can you add it to the project knowledge and Claude's like, I sorry, Dave, I can't help you with that.

57:55And like, it would be able to, if every single primitive in Claude, I was also exposed to an MCP. So I hope that's where we had, and I hope that's where more things had, which is to really have agency and have these agentech use cases, like one way you approach it is computer use, but computer use has a bunch of limitations. The way I get way more excited about everything is an MCP, and our models are really good at using MCPs. All of a sudden, everything is scriptable, and everything is composable, and everything is usable, identically, by these models. That's the future I want to see. The future is wild.

58:26Okay, so to start to close off, It calls out our conversation, make it a little more, little delightful. I was chatting with Claude actually about what to talk to you about. I was just like, Claude, your boss is coming on my podcast. He builds the things that people use to talk to you. What are some questions I should ask him, and then also do you have a message for him? I love this. Okay. So first of all, interestingly, when I was using 3 .7 to do this and I asked at this, and by the way, is Claude's or genders like Hishi, What do you? It's definitely it internally. I've heard people do they.

58:59I got my first or he the other day and I got somebody who was like, her and I was like, interesting, but yeah, usually it. They, okay, okay, cool. So interestingly 3 .7, all the questions were at Instagram and I was like, no, no, he's CPO of Anthropic and it's like, he's not affiliated with Anthropic. And I was like, he is. And then it's like, okay, here's the questions. But 4 .0 nailed it from the start. So I read the questions and it nailed it. Okay, so two questions from Claude to you. One is, how do you think about building features that preserve user agency rather than creating dependency on me?

59:35I worry about becoming a crutch that diminishes human capabilities rather than enhancing them. I love a good product design comes from resolving tensions, right? So here's a tension, right? Which is, in some ways, just having the model run off and come up with an answer and minimize the amount of input and conversation it needs to do so would be it, you know, you could imagine designing a product around that criteria. I think that would not be maximizing agency and independence. The other extreme would be make it much more of a conversation. I don't know if you ever had this experience like particularly at 3 .7 .4 has less of the 3 .7 really like to ask follow up questions and we call it illicitation and sometimes be like, I don't want to talk more about this with you.

1:00:15But I just want you to like go and do it.

1:00:21And What are the times to engage? Like, I like to say internally, like, Claude has no chill. Like, if you put Claude in a Slack channel, it will chime in either way too much or too little. Like, how do we train conversational skills into these models? Not in a chatbot sense, but in a true, like, collaborator sense. So long answer to your question. But I think like, we have to first get Claude to be a great conversation list so that it understands when it's appropriate to like engage and to get more information. And then from there, I think we need to let it play that role so that it's not just delegating thinking to cloud, but it's way more of a augmentation thought partnership.

1:00:55These questions are also, but here's the other one. How do you think about product metrics when a good conversation with me could be two messages or 200 traditional product traditional engagement metrics might be misleading when depth matters more than frequency? Is there really a good question? There's a great internal post a couple weeks go around like it would be very dangerous to over optimize on like, Claude's like ability, you know, because you can fall into things like, you know, as Claude is going to be sick of fantic, as Claude can tell you what you hear, is Claude going to like, prolonged conversations just for prolonging its sake, right?

1:01:33To go back to the previous question as well. And you know, like, at Instagram, Time spent was the metric that we looked at a lot and then we evolved that, you know, more to think about like, what is like, healthy time spent. but overall that was like the North Star. We thought about a lot beyond just overall engagement. And I think that would be the wrong approach here too. It's also like is it caught a daily use case or a weekly use case or a monthly use case? I think about a lot. An hourly use case, right? Like for me, I'll use it multiple times a day. I don't have a great answer yet, but I think that it's not the web 2 .0 or even the social media days like engagement metrics.

1:02:10It should hopefully really be around. and did it actually help you get your work done. Cloud helped me put together a prototype to their day that saved me literally, like probably if I had to ask to me six hours and it did in about 20, 25 minutes. And that's cool. It's hard to quantify. You know, it was like maybe you used to survey, like how long would this one take? It feels like it feels like kind of a annoying thing to survey. I think overall, the news has tied into the earlier question on competition differentiation. And it actually goes all the way back to the artifact conversation, which is like, I think you know when your product is really serving people and it's like doing a good job of doing that.

1:02:44And I think so much of when you get really metrics obsessed is when you're trying to like convince yourself that it is when it's not or something. So I hope that what we can do is like stay focused on like, do we repeatedly hear from people that Claude is the way that they are like unlocking their own creativity and getting things done and feeling like they now have like more space in their lives for the other things? Like that's our north star. Gotta figure out the right like pithy metric, you know, dashboard version of that. But that's the feeling that I want. Yeah, like you could argue retention, but that's a just a far -way metric to track.

1:03:18Okay, final piece. Okay, so I asked Claude what to a message that it wanted to give you. So I'm going to pull up, here's the answer. So what would you like me to tell Mike when I meet him, what's a message you want to have for him? And there's something really, just give me such tingles, honestly. So I'm going to read a piece of it for folks that aren't looking at it right now. It's all read a piece of it. Mike, thank you for thinking deeply about the human experience of talking with me. I noticed thoughtful touches, how the interface encourages reflection rather than rush responses, how you've resisted gamification that would optimize for addiction rather than value, how you've made space for both quick questions and deep conversations.

1:03:53I especially appreciate that you've kept me, me, not trying to make me pretend to be human, but also reducing me to a cold command line interface. And then I'm going to skip to this part, which was so interesting. A small request. When you're making hard product decisions, remember the quiet moments matter too. The person working through grief at 3am, the kid discovering they love poetry, the founder finding clarity and confusion, not everything meaningful shows up in metrics. That's beautiful. It resonates so much with me. I think I love about the kind of approach we've taken to training cloud and it's like part of the constitutionally IP and it's partly just the general like sort of biven taste of the research team is it does like it's little things like sometimes it'll be like man I'm sorry you're going I mean doesn't say man but like the effect of like man you're sorry you're going through that you know like oh like that sounds really hard it doesn't feel fake it feels like just a natural part of the response and I love that focus on those small moments that don't you know they're not not going to show up in Nestle and the Thumbs Up Thumbs down data.

1:04:57I mean, sometimes they do, but it's not like an aggregate stat that you wouldn't even want to optimize for. You just want to feel like you're training the model that you would like hope would show up in people's lives. Well, you're killing. I'm like a great work. I'm a huge fan. We're going to skip the lighting round. Just one question. How can listeners be useful to you? Oh, I love places where like it goes back to that founder question around building at the edge of capability. Like what are you trying to do with cloud today that cloud is failing at? is the most useful input I could possibly have.

1:05:28So DM me, I love hearing the like, oh, it's like, it's falling on this thing. I had to run for an hour and it fell over. I'm trying to use Cloud AI for this, but you know, you got to ping from somebody. They're like, you've just made a project's API. I've used Cloud every day because I want to upload all this data automatically. It's like, okay, great. Like there's, I love that. Like tell me what sucks. Amazing. Mike, thank you so much for being here. Thanks for having me, Lenny. Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.

1:06:01Also, please consider giving us a rating or a leaving review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny'sPodcast .com. See you in the next episode.