20Product: How Duolingo Build Product 10x Faster with AI | Duolingo's Biggest Lessons on Paywalls, Push Notifications and In-App Purchases | Why Small Teams are the Future of Product | Why PMs Will Become Extinct with Cem Kansu, CPO @ Duolingo

24:43When you have that unlock, the amazing thing is it means that bunny everyone can create product with any sort of thing. I think we're about to enter a wild time with this. Yes. Does that It's so exciting. It's, I'm so excited. Oh, worry. Because it now means that you have infinite supply of product. Yes. And that means that discovery becomes even more challenging. Yes. On a personal level, I'm super excited. And you know, for the discipline of product management, I'm also super excited. Maybe you've had multiple folks on this show and I've heard this come up actually. It's like, PMs do have imposter syndrome.

25:18They're, because they're like, when they're interacting with engineers, they're not technical enough. when they're working with designers, they're not creative enough. These tools level up everybody to have much, they're maybe not at the same level, but to a similar leveling field where we're all operating with like strong tools. So it helps PMs be a lot more productive and independent for themselves. So I think you up level product managers, which is good. Like for my function, I love that. But you up level everybody to be way more productive. This unlocks much easier product building. And I think this is very similar to just like a 10X productivity gain where the best ideas will eventually win.

25:55So creating 10X more prototypes doesn't, I mean, it creates maybe chaos at first, but then the users eventually get the best of 100 ideas rather than best of five ideas we can ship today. So I think just like creation goes up, and when creation goes up, the best things that go out to users are just going to be better. I agree completely. When you look at the tools again, who wins cursor or wins off? I'm gonna say cursor why because internally at Duelingle I'm seeing way more usage and way more creation come out of cursor That said maybe we have an experiment enough with windsurf can ask you everyone says to me I look at these tools and you know that neither of them have locking or defensibility and as we discovered from my Act due diligence on some companies we mentioned before I naturally go at something very interesting Tell me more, but I didn't actually understand why why did they not have defensibility and it's that wrong I feel like these statements come out of, is there a data lock -in?

26:50So, if you're feeding it more data, is the tool getting better? Or is there a social lock -in where I don't know, you're collaborating with folks and switching is painful? For example, a lot of enterprise tools have heavy data lock -in. Like, we use Gira for, for example, task management in the company. It's very hard to switch out of because all of our history has been now fed in there. And switching out means we kind of have to leave that data in those two. We use Gera for. I love Pubelle. I'm going to introduce you to Carrie at Linear after this. Okay. Sounds good. And he is amazing. And this is called Venture Value Ad.

27:30Here we go. And honestly, they're brilliant. They're much better than Gera. I mean, I've seen their designs. It looks awesome. I'll say the hardest thing is to switch out of the thing we have. So even if we love the product, switching out is going to be a big debate because there's a huge cost. And I think people bring this up because you can switch from cursor to wind surf. You don't lose anything, right? If the wind surf is better tomorrow morning, well, we'll start using that. Maybe these will change. Like, meaning the more code you build with one, we'll just make it so much better. For example, now chat GPT has memory.

28:02I think that is an amazing lock -in for chat GPT because now since it remembers our chats, it can now give me smarter answers. That is a big lock -in for me to not use a different product. I do have to ask, because you mentioned at the beginning, like being central to the monetization. And I spoke to Savrin and he said that bluntly it was your fault that they waited so long. Uh, Jokes aside, why did it take so long, do you think? And how do you reflect on that decision? The hindsight bias is either to apply to our past and say, oh, we waited too long. But I think at the time a few things were, I would say, maybe headwinds to starting to monetize.

28:42One of them is dualingos obviously has a very strong mission today and that certainly was the case then when it first came out Luis you can find Luis's talk. This was a lot easier to find on YouTube and I think it was on our corporate pages. I think we removed it now because we moved away from that. But the monetization plan for dualingo was going to be everything was going to be free. So as users went through did language exercises, they would do translation and then we would crowdsource those translations as a service to anyone who want to divide. So if CNN wanted to translate an article from English to Spanish, they would feed into the Olingo system.

29:18That article would be broken up into chunks fed into users and exercise. And as crowds translated that as an exercise, it would populate back a translation. That was the original plan. And as Luis announced to Olingo, he also said, we will never have ads, never in app purchases and never subscriptions. And and finally left that's the exact three things I did as part of my job at Duelinggo. And your lesson from that is never say never. That's lesson one that things change and your view on these change. But I think to the reason there was a lot of resistance in the company for monetization is we launched with the view that if you did these things, you would screw up your amazing, great consumer product.

29:59I think what honestly my job eventually turned into is obviously building these things, but also changing the organization's mind that there is a way to do these without going against our mission of providing access to free education and doing them well. The big unlock there was at the time and honestly mostly today, all education apps would lock all of their content behind the paywall because that's so straightforward. Like, why did you come to this app to learn? Well, what are we going to charge for? What's charge for the learning content? We did the exact opposite. We're like, learning content is completely free.

30:30But we are going to monetize the bells and whistles. We're going to do freemium. So most of the stuff will be free, but the premium features could be part of the subscription. Even our investors at the time was like, oh, that's not a good idea. I've never seen that work. If I was one of your investors, I'd say, hey, growth is everything. Engagement is everything. Retention is the fucking holy grail. Don't stick in at when we don't have a massive company yet. I think that was also the common wisdom at the time. There was the Facebook model. It just grow. Doesn't matter. Ignore monetization. And honestly, it was good advice.

31:00but I think what we ignored is you can still have high retention, you can keep growing your product, but also monetize at the same time. I think it felt like it would kill things, and it didn't. It was fine. I mean, putting ads on Duolingo, if you do it tastefully, doesn't kill your retention. So you can still have very high retention and monetize the bells and whistles, but have high growth. Can you just take me to the moment where you were like, okay, we're going to move from a zero revenue new company to trying to make money. Yeah. Was there a meeting? Can you take me to that moment? It probably wasn't one meeting, but there was a few key moments where I remember being like, how am I going to figure this out?

31:38One of them was sitting with Luis. In honestly, probably one of my first weeks, and I'm a very early product manager trying to figure stuff out. And I'm like, all right, Luis, like here's our ads roadmap. I calculated how much money this is going to bake. Here are the experiments I'm going to run. And he's like, all right, what is it going to look like? And I show him like the prototype or like we look at it. He's been looking at he's like These as look like shit shit is his first response is like I'm not sure if we should do it And I'm like okay, where did we I hear you? I understand that this doesn't look like the pristine every other page of the app It's different.

32:12It's another advertiser's property But we can improve the look we can change the design anyway It was an iterative process from there to be like he's like let's test Let's see what happens and the other worry was like oh, this is gonna kill retention. We test it It's fine It's like neutral on retention on the first experiment and we're like okay our users are not running away We are improving how the ads look and when they're surfaced and then I think over time it started snowballing to be like Okay, we can do this we can do ads once we did ads the second phase was well No ads was like a request people started sending us being like can I pay you guys to remove ads?

32:45We're like well, let's try that that's a good start before we move that into the paid then And do you accept that you cannot do ads without hindering user experience a little bit? There's certainly a cost. I had it the other day and you have the timer in the top left. Yeah. And I'm like, listen, it's annoying. I stay, of course. And I love dealing with just the same. I can't say I love ads as a user either. So I think you sacrifice because it is not a property you control. Meaning if I bring an advertiser, they have to show their asset. and that comes with a five second timer on, I don't know, a video for selling you shorts, whatever it might be, it's a cost.

33:24I think that you take. That said, there are many different ways to do ads. So I think treating ads as this black box carrier of like it's only bad and it can only hurt is a bit too simplistic. What's the lessons on what it takes then to do ads well? If it's too simplistic to just black box, I think ad load is the key thing there. We stick to one, you can only see one ad at the end of a lesson. and that's not mid -lesson, that's not started lesson, you finish your lesson and that's you take a breather, that's the only time we show you an ad is what we aligned on. Versus you can put a banner ad on top of, there's many tasteless ways to do it.

33:58Which optimize for cash though, because you can charge more for in -play versus post -experience. But I think that is short term thinking at the end, because you maximize a lot of maybe revenue now because you're putting a lot of ad clicks across the app. But the moment you ruin your free user experience, you kill your growth engine. Can I ask you, do you get paid on a views or on a clicks? We get paid by impressions, so views, I guess. So the rate that these end works with, you depend on how clickable these impressions are. It kind of both touch eventually the ad revenue you make if you're running in programmatic ads on your products.

34:36Okay. And so I then say, you know what? Fuck this. I want no ads and you're like, hmm, what happens then? A lot of users obviously were like, I love Duolingo, but I'll pay you guys whatever. Can I remove the ads? That no surprise because that's a common model. That was the idea for our subscription. So it's like we had ads, we had a no ad subscription. That was the start. And what we saw also at the time was there was a lot of user love for Duolingo. It sounds like you also love Duolingo. And we saw users just buying the subscription because a lot of our users was I was using Doolinggo for five years.

35:13I just wanted to contribute something back, so I bought the subscription, even though I didn't know what was really in it. No, I'm not that Christian. Definitely not. A $20 billion market cap made me feel. I was like, no. But yeah, so, but basically the no ads was a starting point, and it was feature light, honestly, at the time. The subscription was kind of a nice to have more than anything else. What was, if you were advising me as a founder on doing that, like premium subscription? Yeah. What would you say having been through that? It really depends on what business you're building. But if you are building a consumer app business, I would very, very highly advise you to build a premium model.

35:52Meaning you should have a very great freezer experience without asking anybody to pay. The reason for that is if you have a premium product, let's say it's all paid, you can and only use the app if you subscribe, or it's very limited, what you have to build is also a paid marketing engine. So you have to spend a lot of performance marketing to acquire subscribers, generate revenue from them, and spend it back on performance marketing. That's how paid products kind of have to grow. Free products, if you build an amazing free product, it takes way longer, but what you end up building is an organic growth flywheel.

36:24Meaning, you tell your friends about duolingo, I don't know if you do. I tell my friends about duolingo, let's go off of me. they can go experience the full thing for free, whether they pair or not doesn't matter. If they like it, they will also tell their friends. It just expands the entire pie of people who can use your product without having to pay. And we have a lot of users that use the app for free for five years. Eventually one January they're like, oh, I'm gonna go ham on my French this year. I'll subscribe, it doesn't matter, but they use dual -ing go. So to build basically a large user base, and if you wanna build a large business, it's way more defensible to build a great free user experience and do it freemium rather than full premium.

37:03The other problem with premium is competition. So like the reason Duolingo has grown quite a bit in the past 10 years or even gained initial success was because it had an amazing free user experience. If you created the exact equivalent of Duolingo today, what you would probably do is okay, like you want to make some money, apps have tried this actually. So you kind of copy Duolingo exactly as is, you make it fully paid. You're like, I'm going to go fully paid and spend on paid performance marketing. Well, what happens is the users who you can acquire will land on your app and say, okay, you ask me to pay $80, $100, whatever it is, or $10, or I can just go to Duolingo and do it for free.

37:42So your paid costs go really high. So it makes it really hard to compete with a free, amazing product. So this is a very around about way to say, if you're building a consumer app business, I think you want to make sure you have an amazing free growth flywheel, makes you defensible, and it kind of spreads your product without you having to spend marketing money. How do you think about the right time to hit them with a paywall? You want to build the trust, the love, the loyalty, without being like, okay, I love one paywall. Exactly. So I think it's twofold. One of them is honestly like hitting with a paywall is kind of the wrong approach.

38:15In the sense that I think you want people to discover or the paywall by experiencing features. That is the best way to honestly convince someone to buy something, meaning let's say I would like to convince our users to buy Duolingo Max and its key feature is a video call. The best way to do it is have them try video call, actually. So a lot of our ways to do that is maybe show them a video of how video call works and then say, if you're interested, here is Duolingo Max. Or have them go into the experience, have a call themselves, And then at the end say this is part of dual -ingomax. That is kind of the best way when it's like contextual and user discover that there is something to be paid to experience.

38:58I think just like having people showing kind of what I would call generic ads works way less better. And it looks spammy in the app. So we try to do everything contextual. For example, if we show you an ad, we will say right after there's an ad -free version if you'd like to upgrade. And that is in a way like you've discovered an ad. Okay, maybe you didn't love it. Okay, but you can also find an ad free upgrade. I just wasn't expecting for an hub premium to be an advertiser I was before a second. I was like oh shit

39:31It was the first useful ad I've had on julie It's the only ad you close turn out like sign up sign up sign The joys of doing the shows also like you hear the show and you're like, God, it's so serious. And then behind the scenes you're like, what a rise. He's born up for you. Yeah, it's much more fun that way. Can I ask you, when you look at a different paywall pages that you've experimented with, are there lessons on how to bucket different offerings? Oh, yeah. What are those lessons? There's a lot. The biggest one I will say is what you say in your copy and strings makes a massive difference.

40:12And obviously you don't want to kind of sell something that you don't have. But how you talk about entering that experience makes a massive sense. I'll give you an example. We have tested many ways to on the button that is that is going to start your subscription. We tested probably endless number of things. Turns out if you say try for zero dot zero zero, which is not a normal human. I wouldn't talk to you. Would you like to try something for 0 .00 pounds? I wouldn't say that. But in the string, if you hide multiple zeros and you hint at the fact that there's a free trial with multiple zeros, conversion goes up a significant amount.

40:48Over. Start your free trial. Over anything else. Start my free week. Start my free trial. Start your free trial. Or come up with any other combination. We've tested probably many different things. Turns that if you put multiple zeros, I think it like lights something up in the brain that you're like, oh yeah, I get it. It's very free. It's because it's like zero dot zero zero. So it copy matters. That's one because there's with subscription, there's natural aversion to entering a thing that might be hard to get out. We also have seen talking about how easy it is to cancel is like an unlock in conversion.

41:20Do you guys send reminders? We also send reminders because I signed up to Max and I signed up because of the reminder. There you go. I so So that actually makes a big difference. And honestly this is one of those changes I love because it's user friendly. If you don't like the experience, I want to remind you to cancel. We don't want people who forget and pay because of it. But that relieves the anxiety of entering a subscription free trial. And the thing about the different packages, like give people just one option, paid versus not paid, max versus premium versus free, and then you're like choice.

41:53I'm always fascinated by the paradox of choice. I think comparison is super important. What we've also seen is if you give one, it's for example, we have multiple packages now, there's Super, there's Family Plan, there's Max, we think you are a great fit for Super Individual. If we only show you one package and you have no anchor for different packages, it converts way less because it feels like if that's not for you, you're just going to decline. If we give you more than three packages on one page, it's too much. Like too much choice, you also convert way less. I think 2 to 3 is we've seen as a sweet spot.

42:26So you want some anchors to compare to. And most of the time, if you give a choice of 3, you want the middle option to be the option that you really recommend because that makes choice very easy. Humans love, I mean, on a menu too, if you think about it, if you have the lowest price, middle price, and highest price, middle price feels like the safe choice. If you don't know what to choose, very similar to if you're kind of offering someone a subscription choice. I think you want to make the choice easy by recommending one and recommended one being the middle one feels safe to choose. What percentage of people take the recommended?

43:00Very high. I don't I don't remember the exact percentage but certainly more than it's the majority more than 50%. Wow. Yeah. Any other big lessons on copy matters. I'm sure you've been through many different iterations of oh my god we've tested so many things. Your copy matters. You know how every company eventually puts like a checklist in front of of you where like, oh, if you do this package, you get like two check marks, but if you do this other package, you get like 10 check marks. This has become like a very common way of subscription purchase flows. Weirdly that also works because it's the psychological difference you can see visually on like, oh, what I have now has one check mark, what I don't have has like 10 check marks.

43:36Well, 10 check marks feel seven customer support. I lose better. It feels like you're getting a lot and I think that feeling of getting a lot feels better. I think the other thing that we've learned is there's a lot of emotion on Purchasing anything certainly something that is an upgrade to a consumer experience like Dueling going from free Dueling go to super Dueling go Or do all in go max and I think you want to convey that emotion in your design meaning Dueling go takes pride in like a fun being fun and Animated what are flows being animated on duo doing funny stuff through the flow it works well because it like conveys that this is not just a boring subscription as a service purchase.

44:17It's like part of your experience and it's an upgrade. So anyway, you can use illustrations, animation and things like that make it a fun flow that moves metrics at the end and increases conversion. What percent of revenue is subscription today? Today it's more than 80%. What revenue line does do it and go not have today that will be meaningful in five years. We do have this revenue line, but it's incredibly small today, but I think it will be way bigger is merchandising. We have a store you can buy dual -ingle merch. Our most selling item is the dual plushie. It's like a very funny plushie of duo.

44:52I think we are one of those brands where our characters are quite, duo is quite recognizable at this point and we're trying to build more IP around our characters. There's a lot we can do on creating better merch and expanding our merch offering. Surely content is it you want to become the Disney of the next generation. Why not? Exactly. We would love to become the Disney of the next generation. I think that is hard for sure, but that would be awesome. But I'm actually the founder of this idea and now company. That was my idea. So thank you Lewis. I'll take over. Would you say that you're an education company, an entertainment company, a content company?

45:31I know it's all semantics. Honestly, neither is the reality. We have maybe in somewhat of an identity crisis is also the reality. We don't identify as an education company, but we also don't identify as a game or entertainment. So we're kind of in our weird own league and own market that we've created. The reality is what we compete for, if you're like, okay, but what are you? Who do you compete with? We eventually compete for a screen time. We would like to make your screen time very useful for you to learn something and get better. The reason we don't fit into education very much is if you look at the education space, honestly, this is a bit of a hot take, but it's all boring -ass products.

46:13So I think we would like to not be a boring -ass product. As you can see, we try to do everything super fun. And I think if you say, well, you guys do a lot of fun, then there you are game. Well, we're not a game either. We also want to teach you something. So we're kind of in this zone, League of Our Own, is how we see it. But what we eventually compete for is your screen time. So everything that takes up your screen time that might be social media that might be gaming that might be streaming Whatever it is at the end is what dualing go competes for It's also a hard one on multiples gaming multiples a shocking Education tough and also that's not the product we're building They were not building just for entertainment But we're also not building just to educate and ignoring the fun element of it So that's why it's like right in the middle.

46:54Okay, the final element of this to we've got the ads we've got the premium and then we have in -app purchases. How do we think about what worked there and the biggest lessons to doing that so successfully? So in -app purchases for us was very similar to how games have designed in -app purchases and mechanics. So we have a virtual currency. It's called GEMs. It's not named after me. It's the actual diamond GEMs. But you were the one who came up with monetization. So yes, GEMs was one of the first features I built actually. So there's sometime there. Um, in 20 BC, we had our Harry stars. It's like I didn't come up with it.

47:32Actually, um, no connection with a connection at all, but it just happens to be that. And you can buy, you're going to listen to this and give this guy. It was a fucking joke. I am curious what you, what you cut and what you keep. So I was going to end up really being interested in. Oh, yeah, it's going to be fun. It's a surprise for you and me. So you give gems. We give gems and you can either buy gems or earn gems and you can spend those gems in virtual features. So you can, for example, one of the most popular features is a streak freeze. So streak as you might know is a very popular feature.

48:06People take pride in their streak. They don't want to lose it. But if for one reason or other you can't do a dualingo that day, you can use a streak freeze. This is a very valuable item. People love it. And if you earn gems, you can spend your own gems on it. If you're out of gems, you can buy gems to purchase a streak freeze. Items like that exist. And that's part of our gem economy. Have you ever had someone, sorry, to interrupt you? Have you ever had someone lose a streak? And then contact you and be like, listen, I was tracking up the Everest and I've dropped my phone and so I couldn't do my streak.

48:37All the time. What's your favorite story? High volume request. I think that most most of the stories are time zone switches and honestly this has happened to me So I actually have full empathy for this where I actually did my lesson But the reason is I flew to Australia I was on the plane for 14 hours and when I landed the day had already moved forward So I lost my streak so you had to cool up the head of product Completely get it and I we fixed though So like if you actually reach out to customers support and you're like I have this legitimate reason We're like lenient. We're like of course like we'll fix your streak.

49:09Don't know if you have to prove it Honestly, we really don't ask that many questions. But, and we're lenient. But yeah, so there's like time switches, and there's like, I was on a mountain with no connection. What was I supposed to do? And, of course, we understand. We're like, also fucking mountain. We understand, but also, yeah, like, use a streak freeze maybe, and then get a hop to mountain. By the freeze, that's why that's there. But okay, so it lessons on how to do that effectively. Is it just create addictive apps that people don't want? Well, like I think the core experience of course matters a lot.

49:42Addictive sounds like the bad version of this, but habit building. Like you want to build good habits. And the reason we use that term is like, if you want to learn the language, you have to practice. If you want to be fluent thousands of hours. And to get to thousands of hours, you need a daily habit. You can't like binge your way to fluency. So that's why everything at Toulingo is built around a daily habit. Once you've done an experience that people want to use daily, you have good daily retention, then you kind of layer on these benefits that are kind of extra, like streak and a streak freeze.

50:13You don't need it, but it is really core to your daily habit, and then freezes this extra benefit that you can monetize. That's how we've thought about it. Totally. Yeah, that's fascinating. What do you think people do wrong within that purchases? You see so many, you're a great product mind. What do people do wrong? I think people do really aggressive things and that ends up hurting the experience. So for example, streak freeze is so extra, right? We're talking about it because it's like, if you don't buy it, your doing your experience is going to be exactly the same. But it's this very geometric that you can choose to freeze.

50:44And if you don't, nothing really changes all up to you. Like your French is going to be the same either way. I think what sometimes apps do is they make really core functionality and in -app purchase. And I think then you start killing the fun, the retention producing elements of your app. So, imagine if Doolingo was every new unit of a lesson was in that person. It was like a dollar per unit. You would get pretty sick of it pretty quickly. It's like buying a chapter of a book for a dollar every time. You could do it. That's a monetization model, but it's a pretty stupid and bad one. So I think you want to find things that people find joy in buying that doesn't hurt the freezer experience.

51:22I think it actually goes to trust, which is when you do that, I actually lose trust in you. It makes it a transactional relationship. that I don't even feel like I'm a premium. Like the one -hit pain of like I'm a premium, fine, you pay the toll and now you enjoy the holiday. And you feel good, yes. But if it's like consistent pain inflicted, fuck that. No, it's very annoying. Like we talked about with ads the same way, Taseful Monks 'ation really matters. It builds user trust and I think user trust goes a long way for the business because you tell your friends, they download the app. What are the metrics that matter most to you?

51:56There's two metrics that drive the business. So if you really squint at a business like Duolingo, you have to look at two metrics. One of them is daily active users. And the second one is revenue. But if you just want to spreadsheet a business, you have to look at DAUs and revenue. But I think for me, what matters is retention. So because that's like company metrics. Correct. So if you think about product metrics, the inputs to drive the outputs retention. Retention is everything when you think about building product because it is the best metric to distill down if people are finding value or liking a feature or a new app, whatever it is, what is your daily retention?

52:35I mean, if it's a daily app, daily retention, if it's a weekly app, you look at weekly retention for us, everything we build is around daily usage. So it is daily retention. So if we say, for example, chess, we're, we just launch chess. The thing we're looking at day in day out is what is the daily retention? Because if it's really high, well, it's going to compound to high DAUs. We know it's going to, for example, you could look at DAUs, but do you have any idea has many inputs that are muddying the waters of is just a sticky habit building product. So daily retention is the king of all metrics for consumer products.

53:07And just so I understand daily retention is do I come back tomorrow? Correct. Hi bar. Hi bar. Exactly. With mobile products, that's why it's so important because you have many different things that can steal your time on your mouth. You don't maybe. I didn't have anything. You're like, I don't know what apps are. doing what apps are. Okay, so we have daily active users are like cool cool. What is good? I think it's going to really depend on your product. There's no like easy benchmark you can put and also really depends on what your user base looks like. If it's all new users, you have a different metric than if you have a bunch of users that are already settled users that I'm going to call them like dualingo or they've already been using for like three years or above.

53:50When they're settled users to those cohorts in terms of usage that look quite like candy crush users, like very, very settled gaming users. I think so. I'm not knowing candy crush metrics myself at all, but what we tend to see is if you've reached a 30 -day streak, so you built up a 30 -day streak, your retention is very close to 100%. It's not 100%, but it's very high because you've built a habit. It's not crazy because you've done it for 30 days without breaking your streak, you have built a habit, so you're very likely to continue with the habit. But if you look at retention of users or streak less than seven days, it's way lower because they actually haven't built a habit.

54:30Or maybe they're unsure of doing or they're just playing with it. One way or another, they haven't built a habit. Do you see wildly different retention numbers in paid versus free? We do. However, the difference is a little hard to parse on its causation versus correlation. Do they retain better because they are a subscriber or they were likely to retain because they were more engaged and that's why they decided to subscribe. It's hard to parse in the data, but if you look at just pure users who choose to subscribe their retention and users who choose not to subscribe retention is much higher.

55:03What caused the biggest change in retention from a product perspective? Three things. Wow, that was very... Yeah, this is something I've thought about a lot. You have that answer ready to go, didn't you? I was like, that's a good question. You're like, I got that one. So, one of them is the streak. The no surprise. Having the streak and making the streak exciting. So when you finish your first lesson, there's this flame animation that lights up. There's a haptic. There's a sound. We make the streak like one of the most important things on dual and go. The details being the details, it's exceptional really.

55:36The details matter a lot. So every tiny thing that might not even make sense to talk out loud, but when you see it that made the streak more exciting has helped improve retention actually. So the streak is this incredibly important mechanic that instills daily habit helps you learn a language or learn a subject. That is one of the biggest things for retention. I would say second as a feature was leaderboards turns out no surprise in hindsight, but if you see your name ranked on a leaderboard, there is this human tendency to be not at the bottom of that leaderboard, but instead move towards the top.

56:09So a lot of our learners, even though when they came to doing a, they didn't come to compete in French. We have a leaderboard feature where you, you see your name among cohort of similar learners. You want to rise to the top where a lot of users want to at least. So that drives a lot of usage, a lot of retention. Who's like, no, I'm happy languishing at the bottom. I think some people are like, I don't want to compete, but a lot of users when, you know, you see your name, you're like, I would rather move to the top, not to the bottom. I'm one of them for sure. For sure. Do not understand the people I'm happy to be in there.

56:43Like, well, you're not getting hired. Oh, you're like the worst tiring tray ever. Thoroughly uncompested, happy losing. Exactly, exactly. Okay, that's, you love that. So that's the second one. Third is push notifications, actually. For every app, it is an incredible mechanic that you have to figure out how to make amazing. Still. Still. Maybe 10 years ago it was email was a heavy channel email is a debt channel for most products at this point People dead is maybe an overstatement like we obviously invest a good amount of time and make our emails The email team we still love you don't we yes We do enjoy the dot basement with the leak in the corner.

57:19It's fine. I'll let you out next year Yeah, where the push notification team is like popping champagne. Yeah, that's a bell mod in full of heat Celebrating 20 B So yes, what does it take to push well? You can get incredibly complicated, and this was kind of news to me over my career, incredibly complicated with the limited real estate and copy you right in a tiny push notification. For example, timing is a key factor, content is a key factor, personalization of that content is a key factor, emoji usage is a key factor that moves metrics. So if you get into the full details of really optimizing push nobys which we have, you can do a lot with it.

58:02And every year we're like, okay, we invested a lot. Are you going to keep improving? There's a lot to do on improving push notifications. I'll give you one insight that was kind of like tested in and out. And then this turns out to be the winner. We schedule. So let's say you did your dual and go us in today at one PM. We schedule a notification almost exactly at 23 and a half hours later from that time. And we've tested many different timing And the notification will remind you to practice your Spanish because we you just did a Spanish lesson We've tested many different times But the 23 and a half hour works way better for retention because turns out humans do lessons if you're a morning person You do in the morning and if we schedule it almost 30 minutes before you're you're ready to do your morning lesson You're very likely to do a dual -ingolescent at the same time.

58:49So Based on your behavior we timed the next notification that has done really well for retention. So when you get into this type of experimentation, how do we actually figure out how to make it better for each of our users? There's a lot you can do. Wow, that's fascinating. Okay, so that's like timing. What emoji usage? Yeah. Well, emoji breaks up pattern is why I think it works. So a lot of notifications suffer from, they fall into a pattern and now a sea of every one sense so many notifications that it all blends in. So you wanna do something that is more interesting than the 85 other apps sending you notifications a day.

59:24emoji is one way to break pattern. The other thing we do is kind of go full unhinged unnotification. So we write them from duo's voice and we make it sound like duo is basically screaming at you or to do your not always screaming sometimes sending you a subtle message, but it's like a personalized message rather than do all in go saying do your Spanish lesson or practice now that kind of doesn't go very far. But if duo says yo what's up are you doing Spanish today and looks like almost a message from duo that triggers you to be like, okay, yeah, I'll respond to duo rather than this app that is sending me sanitized notifications.

59:57It's like a replacement girlfriend, isn't it? You don't need someone to shout at you. You've just got duo being like, your bitch. Get the spine. You would respond to that more. So you respond because it's different. It's different. Yeah, it's that's going to the purple cow. Yeah, yeah, it's different. And it doesn't feel like a company is sending you a thing. It's a human would send you a thing that way. We're going to do a grow session, me and you. Okay, so we've got a couple of fun programs we can do. We can do a duo -lingo school in New York, physical school, great community, great for like vibe and...

1:00:30Did this come from Severin? This is his like passion project of like, yeah, duo -lingo should have a physical school. I agree with him. But that shit... I don't. I'm one of the founders. You wouldn't know, but we were talking about this about the 15 -year -old. It's found to talk. Okay, got to talk. You can do a duo movie. Go in the Disney direction. Okay, I do like that idea for the record. Really, really cool. I think if we want to build characters, we have to expand more beyond the app. Or you can do a date. Do a date. Yeah, which is like dating, connecting people learning the same language.

1:01:02You've mentioned before people getting married, before who maybe met on meet -a -boards. Which one do you love? Which one do you hate? I hate the school one, if I'm being honest, because building a physical business seems like that's not our strength. I like the movie idea. I love the date idea. That's how I would pick. Wow, you love the data idea. I do love the data idea. I think it is the riskiest one because what could go wrong by just building a dating platform? However, there's already signs that people date on Duolingo. So it's the best way to build product is capture latent demand that users can't do on your product today.

1:01:41We have couples that got married because they met competing on a leaderboard, and then they found each other on Facebook and got married. This behavior is already happening, so it's not out of the question for us to figure out, well, how can we do matching better? Wow, that's pretty cool. I mean, it is out of the question for the record. We're not doing that, but I love the idea. There's so many interesting lessons to be had from, like, hindsight. When you look at the product decisions that you didn't do, what didn't you do with the benefit of hindsight you wish had done? We talked about the obvious one, so I'll start with that, which is we didn't monetize in 2014.

1:02:19We started monetizing kind of two years later than we could. So I think that's one. We talked about it. We don't need to unpack it. I also think we, this is in hindsight easier to say, we waited too long to apply the dual -ingle magic to new subjects. Meaning we first launched a subject that is not language that was math and then music followed fairly quickly, but math, we first started working on math three years ago. We didn't have to wait that long. Like we kind of had the dual -ingle mechanics in place. We could have honestly started our journey and figuring out other subjects earlier. In hindsight, this is very easy to say, but this is a long journey to figure out product market fit for new subjects.

1:03:01However, we had the competency. We kind of knew eventually we would do something like this, But instead, we were investing in various projects that now in hindsight, we shouldn't have, like for example, we were trying to build a school's dashboard for teachers to use, and that was a complete different product problem that is a very hard problem to solve. Instead of doing that, we shouldn't probably done math and music and chess way earlier. I was going to ask, what did you do that you wish you hadn't done? A school's dashboard. Yeah, there's another one. I think that at the VC level, this makes sense, but when you get into the product level, it does.

1:03:35And which is, why is Duolingo not catering to language teaching in the classroom? Like, why does Duolingo not have a product that fits into the classroom? So we want to solve that. So we tried to come up with a version of our product that teachers could assign, teach certain parts of a curriculum and assign homework through the Duolingo interface. And then they could track if assignments are being done, etc. The problem with this is every language teacher who wants to teach a different way and the dualingo curriculum does not fit any of it. So you end up having to customize how people should go through dualingo and that's not the product we have.

1:04:12So even if teachers could assign assignments, that's not that doesn't match the textbooks they're working on in the French classroom. Anyway, it didn't work. What, how long did you assign to it? Oh, we worked on schools. By the way, this product still exists. It's just like not a product mark. It doesn't have product market fit. So you can still use it. You can still use it. The school's dashboard exists. If you'd like to try it out, it exists. It's there. It just hasn't found the adoption we wanted to because it doesn't fit the classroom very well. But we worked on it for, I don't know, three to five years maybe.

1:04:45Why still have it live? Why not pull it? Because there's still usage. It's not zero usage. Some people find use that of it, but it's like thinking about the market of classrooms teaching languages. And you have no one on it. Right now, there's no investment. That doesn't mean we won't invest again in the future, but today we've chosen not to. It's a complicated problem. Anything else that we did that we should know? There's a big one now on hindsight. It feels like, oh my god, why did we do that? So right now we're talking about how video call and Lili are our speaking feature. There was a short -lived moment where we had a high tier subscription, which was basically an Uber for a Spanish tutor.

1:05:21So you could press a button to speak Spanish to a Spanish speaker immediately. So, rather than scheduling, you could press, so we figured out how to connect you immediately with someone online, and it would be like a two, three minute condensed conversation. Honestly, at the idea, I was like, oh, this is great. This is pretty great. Yeah, we saw. It's not so cool. Exactly. We have solved the touring problem, which is the biggest problem is scheduling, and it's like expensive, et cetera, we made it part of a subscription, so you could do unlimited calls on demand. Very cool idea. And also, users have been asking us to do this for years.

1:05:53Like, we're like, okay, finally, we figured out how to do it. There was two massive problems in hindsight. Now we know one is it is super scary to actually press a button and talk to a human immediately in the language you're not proficient in. Even though you think you want it, what we've seen when we observe users doing it, they will press up on they're like, okay, I'm ready. I'm going to do it. And then the person would dial in, start speaking Spanish and then they'd be like, I don't hang up. is what would happen to most users that ever tried it. So it was a scary experience to be thrown in.

1:06:29And that's why AI is way better for this because you can stumble, you can talk shitty Spanish, you're not embarrassed. A lot of users found it embarrassing to stumble when there's a human looking at them on video. That was problem number one. Problem number two is you have no quality control. So you can't run an A, B test and say, hey, let's do the opener in English. I mean, you do, but then you can't control if the human -delivered experience is what you thought it would be. So we found it kind of impossible to operate a human -based product experience. It's not impossible. It's very hard. And around that time, GPT -3 came out and we're like, oh, hold on.

1:07:06We can solve this. So we pivoted away from that. What would you most like to change in the product? But for some external reason, shareholders being public, exacts not wanting to do it, you can't do it. I would like to make Duolingo way more social. And I don't think there's any block or anyone against it. I think we need to figure out a way forward and the best way to do that. And then staff teams accordingly to solve that problem. So right now, Duolingo has social features. So I can add you as a friend. If you reach a certain streak, I can congratulate you. You can do it back. So we have the social layer, which is very congratulatory and nice.

1:07:43We can do a friend quest together. We can do friend streak. So it's a lot about collaboration and congratulations to achievements. That said, I mean, I think this is kind of why I loved your dating idea. It builds on the social potential of Duolingo. Duolingo today taps into this reality of you're doing something good for yourself. And there are others doing the same. But if you go through the experience, you feel like it's a single player experience. You're doing it solo. And language learning is inherently a social thing. You're most people are learning language to do something social with others.

1:08:15Maybe order their coffee in Italian when they go to Italy or date someone or whatever it might be. There's a social need. And I think social just fits into language quite well. So we have to figure out how to make Dueling go more social. We can go in many different ways. But I think that's one thing that we have to invest more. And which way would you most like to go? Dating is actually not the direction I would go first. But I would love to go in a way where we build better features that you can do with others. So for example, gifting is one area people know that I'm passionate about. So for example, if I can send you more virtual items when you do something well, you can send something back.

1:08:51We build this reciprocity. Imagine I send you a streak freeze. You send me a thank you back. When you're in need, I send you another one. We can kind of create more of a positive gifting environment, that would be awesome. But it's social is one of those hard things where right now not a lot of our users have friends. So when you build these features that are not easy to get right, they don't like kind of hit massive adoption because not everyone has friends, but it's just chicken and egg. We don't have a lot of features. That's why people don't add friends as much. So we have to invest in both.

1:09:22When you look at chess, music, maths, and languages is 85 % of the languages. Yes. Majority of our users are doing languages. Chess is very new, like counted as kind of zero still, it just rolled out like few days ago. So, but math and music are a small percent of our business. Which subject you can do one more? You've capped at five forever. Which one would you add? This will be the next business inside the headline. I think the one I would do is physics. Because, you know, as surprising as that might sound, it's very similar to math. Meaning, we can kind of apply a lot of the stuff we're doing on math and we're doing a lot of product development for teaching math.

1:10:00We can apply it to physics. I'm not sure if it's the largest market or it's certainly not the sexiest market. It's probably the nerdiest thing we can teach. But it's certainly we have a competency or we're building the competence for math that we can apply to physics. Got a fascinating ass in an audience grab, isn't it? Speed equals distance over time. Where do I sign up? Gem has. You sound excited. I'm so excited. I'm so excited. Okay, you ready to quit fire? Let's do it. Okay, so which other product do you look at? And you're like, Yeah, that's small. I like what you did there. The products I love are the ones I use a lot.

1:10:35So I'll mention two and they are everyone using this product a lot. One of them is I'm an Apple fanboy. iOS itself is a beautiful piece of software that is so well designed, so sleek, amazing animations, attention to detail next level. Another product I love is Spotify. Amazingly built product. Obviously, very successful business on top, but I think they also take care and attention details high. I'll mention a third one, which is, we actually were chatting about this, is bump from Ammo. It's just a high craft, incredibly creative app. So those are the apps I'm, when I use them, I get ideas for me to think about for Dueling Go.

1:11:14What was the biggest change PMs will have to encounter in the next five years? The biggest change will be how they do product is going to be significantly different than today, and the tools they use will be obviously a lot more AI -based. They will have to learn, basically, how to be a lot more independent, rather than reliant on other functions to build the thing they have a vision for, meaning they will be independently coming up with prototypes themselves, rather than today they rely on machines that involve a lot of other humans to even like visualize an idea and then turn into prototype AI tools will be the main thing they used to do there.

1:11:55Well if you change your mind on in the last 12 months the example for me I was like you know what chat GPT whatever fuck it will go to groc perplexity you name it I don't think there's any lock in there I'm like unwaveringly chat GPT focus now I have memory I love the brand. Similar to I've actually changed my mind about the same thing, but I'll pick a different one. I thought this change in how we build product was maybe five to 10 years out. 12 months ago, if you had asked me, like, oh, like, you know, how do you think the PM job will change? I like, well, maybe in 10 years, we'll be using more AI tools.

1:12:30I think it's, it will get there slowly. It would be my answer. Honestly, now, probably in 12 to 24 months, the job of a PM will look entirely different. So I think we're talking about five years because I feel safe. But honestly, my guess now, what what I've changed my mind on is it's going to be in two years, is my guess. Where do you most need to improve as a product, either? What I'm trying to improve on is quicker decision making. One of my fallbacks is I can debate something for a very long time because I would like to try to like in my head reach super 100 % clarity or perfection, but that is a massive mistake because you're slowing your decision making down.

1:13:07I think you want 80 % clarity and faster decision making. And you know, weirdly, everything comes back to AI, but it does. ChatGPD is actually really good at helping me do that, because it can be my sounding partner to bounce back and get to clarity quickly or enough clarity. So I'm actually doing that. Totally agree with you. It actually makes me think of the stat, which is you can learn a language. I think 95 % proficiency in like six months if already devoted, but it takes five years to get to 98%. Oh yeah, the last mile is high investment. Totally. Okay, final one. In 20 years, where's Jolingo?

1:13:4120 years is so impossible to think about. But here's what I imagine it will be. I don't know what platform it will be on. Assuming we're all still using mobile phones, it will be on mobile, but if it's if the different consumer platform emerges, it will be on that. If we're all wearing VR headsets and living in the metaverse, which I don't think it will happen, it'll be on that. But Duelingo itself is going to be incredibly personalized in 20 years, meaning rather than I think following standard patterns, I mean it's already headed there heavily, but this will compound. That's one, it will teach many more subjects.

1:14:15I think that is just like, that is just going to happen. It will teach you, hopefully, everything that's good for you, maybe dating by then will be a part of the thing we teach you how to date better, or maybe you'll be dating AI agents. Who knows? But it will teach you way more things. And since it's personalized, it will teach all of those things with much higher efficacy. So, So an hour of time spent will give you much more yield than it does today. That's what I assume we'll do. I think the other thing that will kind of emerge is these AI first experiences. And what I mean by that is now when people think of AI first, they think of, oh, it's chat GPT.

1:14:50It's a text box or a voice input. What hasn't emerged much is UI creation. That is also AI first. So I think you'll see a lot of on the fly UI generation off of what you're doing instead of hard -coded UI that comes into play So I assume in 20 years this will be a thing that we experience where let me paint a picture you enter a dualingo lesson You do a lingo has not built a world where you want to engage in not a hiking up a mountain However, that's where your conversation leaves. That's where your interest is you I can generate you hiking up a mountain and then kind of put you in that position and maybe you teach you the word from Mountain while kind of you immerse with the experience that, I think that type of experience will happen more, which makes the whole thing way more entertaining.

1:15:34I look forward to duodate. At that point, I'll take one sentence. That's fair enough. You're like, I want one future. I just, this is why I did the show. That's why I did it. Listen to it honestly. I do a lot of shows. I don't have this much fun in shows. You hear them? There's not this much fun. So I really appreciate you making it so fun and this has been such a joy. Love it. It's been a pleasure. That was one of the most fun shows to do. I've done it in a long time. If you want to see that episode, you can find it on Spotify by searching for 20 VC on Spotify. It makes such a difference if you leave a review or a five star.

1:16:12I would so appreciate that. But before we leave you today, are you struggling to beat model benchmarks or implement Gen AI in your product? But if so, you need churring. Churring is an AGI infrastructure company backed by incredible investors like Foundation Capital and Westbridge Capital. And they do two things. Number one, they help leading companies in AI labs. Like Salesforce and Thropic and Meta, enhance their LLMs with advanced reasoning, coding, multilinguality, multi -modality and more. Number two, they combine human and artificial intelligence expertise to deploy cutting -edge AI systems for awesome companies like Rivian and Reddit.

1:16:52Right now, Turing offers a free five minute self assessment to help you pinpoint your place in the Gen AI journey, get tailored next steps to optimize your model strategy and then finally learn how Turing can refine and implement your models for better performance. Take the guesswork out of Gen AI, visit Turing .com forward slash 20 VC to start your free assessment today. And once your Gen AI strategies on point with cheering. Now let's switch gears to the future of mobile and how one company is turning your screen into real returns. We spend nearly half our waking hours glued to our phones, upwards of 50 hours every week.

1:17:32Recently, one company transforming this reality stood out so much I personally became a shareholder, mode mobile, mode mobile created the earn phone. The smartphone that pays you for daily activities, instead of big Tech profiting billions from our attention, mode returns over $325 million, directly to users through earnings and savings. Modes revenues, surging incredible 32 ,481 % in 3 years, recognized by Deloitte as 2023's fastest growing software company in North America, and here's why I'm excited, Modes' equity offerings have raised over $30 million from 20 ,000 plus retail investors. One of 2025's standout public raises and you can now join me as a shareholder with as little as a thousand dollars at invest .modemobile .com.

1:18:21Fortsash20VC for a limited time unlock up to 100 % bonus shares and a free earn phone. Email us for the investor brief at 20VC at modemobile .com or check out invest .modemobile .com. Fortsash20VC. We've talked about making money from your phone. Now let's talk about making your product do exactly what you want it to. Whether it's the software you build or the software you buy, your text act should be creating results, not creating roadblocks. Well, Pendo's no -code software experience management platform makes your software better with tools to see where users get stuck, guide them within that messaging and constantly improving your UI.

1:19:01It's so easy that over 14 ,000 businesses use Pendo to increase revenue, lower costs and reduce risks. Businesses love the control, engineers love the freedom, everyone wins! Star for free today at pando .io -4 -slash -20 -product. As always, I so appreciate all your support and stay tuned for an incredible episode coming on Monday with Larry Asherbrook at G -Squad.