Episode #4: Consumer Tech Napkin | Building great products in consumer tech

0:00Features have never been difficult to copy. AI hasn't changed that. It's just made it so much more obvious because the speed at which it can be done is so much quicker now. But it's never been a moat. I really think especially now there's only three things that really classify as a moat. One, and even more importantly now, distribution. Two, whether there's actual genuine user love for your product and or your brand and three, your learning loop. I think whoever has a system that learns the fastest about what the correct, you know, what the optimal solution is, I think there's the people that win and then can put themselves in a defensible position from there.

0:36So that learning loop and whoever can go through that fastest, I think generally becomes the winner in the market. And I think fundamentally, distribution is great, but it's also pointless if the product isn't great, because you might be able to get it into the hands of 100 ,000 people. But if they try it and it's s*** and they never come back, then you've just wasted loads of money anyway. And you've just wasted that distribution point.

1:01Andreas Munk Holm:Entrepreneurship has always been cast as the story of the magical founding inside. The idea is so unique, you guard it behind an NDA, build in stealth, and pray no one figures it out before you do. But what if that insight was never really the thing? And what if the software you build around it was never the moat? In this episode of our Consumer Tech series, Neil Tana from HowBout, Graham Patterson from JITI, and Mark Martin from True join me.

1:31This show is not investment advice and the hosts of this episode may be invested in the funds and companies featured.

1:37Andreas Munk Holm:Welcome back to the podcast, everyone. Let me open with something Neil sat in our prep call that I want to put on the table. Software was never the defensible thing. People just thought it was. AI has just made that more visible. Graham, you told me that you rebuilt what took you three years and several million pounds in more or less five minutes with Claude. So if If the cost and time of building is no longer the barrier, what actually wins, Graham? So I think when you're building an uncertainty and when you're building something new for consumers, especially, you're always building on uncertainty.

2:08I think whoever has a system that learns the fastest about what the correct, you know, what the optimal solution is, I think there's the people that win and then can put themselves in a defensible position from there. So that learning loop and whoever can go through that fastest, I think generally becomes the winner in the market. Something that I think I've always believed is, especially like coming into the world and building in consumer social, like software, there's never been a moment. I think people are just really starting to say it like it's a massive statement right now. but like when I was at uni and excuse me and I was like looking at Snapchat and it was blowing up and then all of a sudden you know stories was copied and then lenses was copied you know like software has never been difficult to copy features have never been difficult to copy AI hasn't changed that it's just made it so much more obvious because the speed at which can be done is so much quicker now but it's never been a mo in my like you know I really think especially now there's only there's only three things that really classify as a mo one and even more importantly now distribution to whether there's actual genuine user love for your product and or your brand and through your learning loop i think right now ai has made it easier to learn but that compounding edge of like every experiment teaches you something about your users that a competitor with no users cannot learn they can ship a new product quickly they can copy a thing but they can't copy your experimentation cycle and that's fundamentally how you can build faster and faster uh but then ship things that actually are going to resonate and move your needles more like that's fundamentally the moat over someone who's just able to to copy your your software they've always been able to do

3:50Andreas Munk Holm:that i've promised you all three that we will spend the majority of today talking about the learning loop and fast iteration and so on because it is the the the real thing that that is truly changing right now but I think let's stay of course in distribution and use your love but just one second stay on that product development part because I want to pressure you a little bit on whether that has changed or to the extent that it has changed and also in the end like is there are not still this true founding insight behind a company that you then used to be able to build at least a year or something like that, a bit under the radar.

4:36Andreas Munk Holm:We all know that everyone says on LinkedIn, stealth startup is what they're building. Do you truly think that there's never been any truth to that, that getting started with a unique insight and building a way a bit in the darkness, so to say, was valuable and put you ahead of the competition? Mike, I'll let you answer as the VC guy with a grand perspective, and then we'll bring Neil and Graham in as well. No, I do agree in that, you know, ultimately AI is just software, but I think the speed of iteration, speed to market is what's changing. I mean, for us, it's one aspect of things that we're looking at, right?

5:13It's not the only thing. The speed at which founders can move is clearly like very important. I think it stems from something that we've talked on the show about before, which is like customer eccentricity and obsession about the customer problem. And so the signal that we're looking for in intensity of experiments, first for taking on feedback from customers and iterating the product is almost like a window into that founder obsession around that customer problem and the ambition to build the best product experience possible. Clearly, if the founders have personal insight or like a lived experience in that, that is definitely a head start.

5:54But I don't think it necessarily doesn't mean it's not copyable. Right. And so today you're still able to have people copying ideas. And so back to Graham and Neil's points around just that complete obsession around customer iterating quickly, feedback loops and effectively how how fast you can do that in a balanced way. And I'm sure we'll sort of come to it later in the podcast, but I'm very intrigued about how particularly founders today are balancing that speed of shipping and balancing that with like, how do we still land the best product experience? I think that's like, it feels to me like a very hard thing to do.

6:30So I'm sort of curious as we, you know, go through the pod, how you guys are thinking about that. I suppose it's actually made your life a little bit harder, Mike, as a pre-seed seed investor, because it's been, I mean, the fact that it's not difficult now for anyone with an idea to ship a MVP or be one of a product, it's probably that you're getting a lot more come through the door, especially a pre-seed. and so it's harder to I mean it's a good thing right fundamentally the lack of technical ability is no longer a reason for why people can't become founders but the the negative of that is you must see a lot more slop come through the door of people who just don't have the the determination the way of thinking about it uh and not just experimentation but just how to fundamentally build a massive business but how do you you must see a lot more come through your your mailbox Yeah, I agree with you as in not necessarily say it's making our jobs harder because that just is the job.

7:32And I guess the earlier you invest, the more you are backing the founders to figure that out. And so you're buying into their ambition, their mindset, their obsession about, like I said, again, the customer problem and how they're going to figure that out. I do think it's great, though, to your point, just on innovation for the consumer. like how exciting i think the next five or ten years is going to be for consumers like completely reimagine all experiences across different consumer categories um so that's very exciting yes for sure it makes people's jobs harder both both founders and and investors um i think it's it's hard for everyone but i think that fundamentally that's should be great for the consumer in the long term when i when i look at the really a bunch of the really great founders and I venerate.

8:19The thing, very few of them actually approach something with a kind of unique insight of, I lived this thing in this industry and I just wanted to solve it. Like the ones that I know of is like Will Shewitt, Deliveroo, got frustrated not being able to order nice takeaways to the office. It's like this crazy unique insight. It's more like a man on a mission. And Chris Rotava, transfer-wise, we're not international money people. They were international people who were like, oh, this should be better. And for us, we're the same inside of real estate is going, well, this sucks. There must be a better way of doing it rather than coming in and saying, right, here's our unique angle right now.

9:01I think almost the real entrepreneurial skill set is not having that inside of it, but it's being able to turn it into something. I guess to Neil's point is like, there are going to be so many more people now who can turn up with something looking really polished, but lack that whatever it is, skill set to say, cool, I'm going to hustle to get intros to VCs and make those pitches and do some sales calls and get some people along on the journey with me and take 100 no's without losing too much enthusiasm and motivation. Those are the weird bits that I think were always a bit more of a barrier to getting started.

9:39Whereas now I can create something called, literally like in the 10 minutes before i fall before i fall asleep whereas before it would take me to go like to a friend or someone be like hey can you go this thing it would take a couple of weeks rather than a couple of minutes but i think the main filter stumbling point will always be that first major one where it's like oh shit this is hard what's the hard bit and i'm guessing you're laughing you know because i'm guessing you took you took a lot of uh no's that's it man i just i agree with you wholeheartedly man like a lot of people out there like ideas are cheap ideas have always been cheap it's about the thing that how do i take this idea and turn it into a billion dollar business and uh it yeah the idea just i'm laughing because i remember in my naivety in my early days when i had this what i thought was such a revolutionary idea making it easier to find time with your friends and i would get like people to i would like ask if people would sign in da's and things like that but it's just what this just this idea that i thought would change the world and actually you realize that the idea is that is that is actually nothing about that the great thing is the fact that a thousand people have had that same idea just in london uh and actually you realize that the fact that everyone's had the same idea just makes it easier to share the idea you don't have to do as much selling but i totally agree with all the points you're making it's it's the determination it's what gets you to run through the door that you're you're facing not just the idea you've had and ai has made it easier for only part of that but definitely not the running through the door bar and i guess unlike the kind of the overall theme here i think that early stage of going i've got this idea and then over however long you turn that into the actual business the actual thing i don't know about for you but for us like it's such a different thing like the thing we've got now is just like in some ways almost unrecognizable from the initial idea and it was that early learning process of being really brutal with it that lets us now do the kind of our learning process is now more about refinement rather than like trying to work out where the opportunity lies yeah i think i think that's exactly and the the bar keeps moving right i remember um when we really started spending time in the us and usbcs and the phenomenal um partner it used to be a general catalyst nico said to us or he was saying to the a group of us you know he he doesn't really take any ideas serious now if they can't or in consumer if They can't already generate 100 ,000 users without marketing spend, without any spend.

11:59And it's because in this day and age in a world where distribution is kind of a moat, but it's also from the perspective of if you've built a good consumer idea, especially for a teenager, and you can't show me that you can already get 100 ,000 people using it through Instagram, TikTok, word of mouth, network effects, well, are you really building something that's going to change the world? And so the bar does keep moving. AI is only moving it further and further now because there are further things that you should just be able to do by yourself. But the determination side of what you're saying, I totally agree with that.

12:31That bar has always been there.

12:33Andreas Munk Holm:I think there's a long conversation to be had clearly about what makes a starter baggable and is there something about the fact that there is a unique insight and access to that insight that resides only with that founder and probably not that many else. also oftentimes you tie that to inflection points in the market and that's a bit what many VCs would describe that they're looking for but we're not here to talk only about about the early stages and how you think about about the founding inside though of course it does create a bit of a mode in the beginning the real focus here should be on on how we creating modes as as consumer startups and I think that we should before we go to learning loops then go to distribution and you should love and if we start on distribution because you just mentioned it Neil this concept of being able to generate 100 ,000 users on a whim almost or right off the bat without having any VC money raised and so on and so forth.

13:30Andreas Munk Holm:Mike maybe you can comment on that statement. It's a pretty radical statement. Do you agree with it? What does it tell you about distribution as a mode in today's market? Because in the past you would have said well if you can get to 100 ,000 users if we remember back to the good old Facebook days and the social network movie. Well, obviously distribution and the fact that you can show early traction, even in pretty small numbers, was a very significant achievement. But we are at a very different place today. And for that reason, maybe distribution is much less of a mode today. Look, I think I agree with distribution being really important because fundamentally, if you can solve for that without needing to spend loads of money fighting, you know, other people in Messery Google.

14:18That's a huge structural advantage. So I find the best products you want, like built in network effects so that you don't have to do that. So that is like, and that is a real mo as well, those network effects and also people being locked into the product. I mean, you hear about people who have kind of, they're like really good about like a moment in time, they found something that works like and whether it's about their tiktok videos going viral or whatever it might be and that's great but i don't think that's necessarily like a sustainable structural advantage because the distribution channels are changing all the time the consumers who are interacting with those channels are changing all the time so you need fundamentally you need to almost be your team needs to have the mindset of almost like trading and being really in touch with what is happening with those channels what works what doesn't work how to pivot when things aren't working so i think that's like a capability within the team that needs to be there and be built on over time and i think fundamentally like distribution is great but it's also pointless if the product isn't great because you might be able to get it into the hands of a hundred thousand people but if they try it and it's and they never come back then you've just wasted loads of money anyway and you've just wasted that distribution point so distribution is fundamentally a very important part of accessing customers and building a business but i don't think it's as simple as saying distribution is a mode i think it can be depending on your product and depending on the team is definitely a critical part of building a big business but you know if you if you build a software product or even a physical product and you have a consumer um sorry a celebrity alongside it right you might be able to reach millions of people they might buy it or download it but if it's not good enough then they're never coming back and you potentially haven't you know you might not have made any money on that on that first that first use of product anyway so it can be a waste of time i think you have to have the combination of distribution and the great product

16:20Andreas Munk Holm:experience to make it work yeah and i think that if we should just tie a knot on this distribution point i think the point that you're all saying here is it's not that distribution is not massively important. Obviously it is. But it's just that everyone will be fighting for distribution and everyone will constantly be getting better and better at it. And you want to have that network effect. You want to have all the things that make your flywheel of distribution get ever more powerful. But what really matters is whether you have that learning capability within the organization to constantly enhance and improve your distribution via your learning loops that you keep doing.

17:02Andreas Munk Holm:Am I right in saying that that's kind of the distinguishing factor here where because distribution has been has gotten somewhat acquirable both like non organically so to say or inorganically then and because it is now pretty well described how you unlock distribution then that can no longer be that's not the distinguishing factor of a very good founding team and a very good company building activity is really whether you have a learning loop that then empowers your distribution and make it more and more powerful. Am I somewhat correct in what I'm saying here? Yeah, I think that's spot on. I think we're all in competition, right?

17:46We're all in competition with other startups, incumbents, new players, et cetera, et cetera. And it is critical that our products are constantly getting better and better because we to my point we can't just rely on just distribution is always and but like the product thing exactly the same our products have to improve to stay relevant and so by building fast experiments flywheels allows us to have far more high quality shots on goals to find that next thing that is either going to make our product itself far better distribution better maybe both but that is what's ultimately going to either turn us into a unicorn faster or even at all but beat the incumbents or the other startups out there so for sure once you have got over that cold start problem ai in itself but also just generally an experimental culture can be the difference between you winning or losing i definitely agree with that okay guys so if we

18:45Andreas Munk Holm:accept this premise that the edge is really how well and how fast you learn, then that means that the conversation becomes what does a good experiment look like? And I think it's fair to say that run more experiments is very bad advice to come away from this podcast with. So if we try and get a bit more specific so that we spend the rest of the episode thinking about that and talking about how do you then build an organization that's very good at this. If we start from the foundation, let's just describe what is an experiment in this context of getting better feedback loops inside of a consumer company.

19:22Andreas Munk Holm:Neil, you had a good distinction in the pre-call for this conversation. So I'd love for you to start. Yeah. So the way we think about this internally is you have to separate what is an experiment versus just what's shipping and hoping. And an experiment for us starts with, right, so what is our, we set quarterly goals, like we have three quarterly goals and they're around growth monetization and retention. And so let's just, for example, take our growth goal. We have metrics we want to move. So the metrics we want to move, we want to move our word of mouth coefficient, we want to move our K factor.

19:56Then let's just say we're focusing here and now on our word of mouth coefficient. We will set hypotheses that we will hopefully prove or disprove that can move the word of mouth coefficient and then build experimental features, tests, et cetera, that look to move or disprove or prove that hypothesis. So everything must have a hypothesis and then we determine how we want to test it. Is it an A-B test? Is it A-B-C test? Like what metric, what market, et cetera, et cetera. But it all must start from a hypothesis, not just hey should we we've thought of this new feature what do we think about it it actually all start from a hypothesis which ladders down from a goal we try to apply that even earlier stages because as we're building new features i try to distinguish between let's say like an experiment and a and a bet and a bet is one of those things we like we will not be able to tell for a very long time if this has worked or not we're going to do it and assess in let's say six months rather than two weeks and we need to not worry about measuring it too much as we go but we do need to learn so a really useful way for us to learn is to do targeted line research and conversations with users with people and go you know even going online looking for those conversations and say what do people think about this thing i find framing even that research as a hypothesis of you know we think if we create a feature that lets people do abc they will do behavior xyz which will help us in the following ways as a business.

21:31And you can often tell quite early on, like get more and more validation that, yeah, if we do ABC, users will do XYZ because they keep complaining about problem XYZ in frame of life so we can do that. So I think even like from the earliest stages, having everything framed as that hypothesis keeps you aligned. I think especially as you're delegating, working with other people, asking them to do this research, it's a lot, and experimentation, it's a lot easier for them to come back to answer a single question because that's all the hypothesis is right it's kind of like a question saying if we do this we think this will happen is that right that's a lot easier than saying to someone here's a feature go do it go work on it go see if we should do it they can come back with a hundred different answers all of which are correct but don't answer the question when you actually care about so just like framing it in that way I think is incredibly helpful, like all across the board.

22:25Or there could have been three other features that could have actually been much better about moving that hypothesis. You thought about it from those aspects of the hypothesis. If someone, and we talk about this a lot internally, like someone might come with the best feature, but actually there might be an even better idea if we can understand what that feature is looking to achieve and then focusing on actually trying to do that than just trying to ship this feature. But we think about this and actually very similar to you. We don't call them bets. We call them, like the word we use internally right now is remarkability so like like using to your framework of a bet like what what do we have low confidence on but like if we get it right really could move the needle like those are really important in product because otherwise if you'd be too scientific with it you never like you know you would have ever if if you would only ever try to do incremental improvements you would have just got the the faster horse you would have never got the car that that mentality so like here you do need to focus on your bets and your bets by definition don't really have much behind them um we look at that from the sense of like how could we make our product at its core even more remarkable such that it may increase our word of mouth coefficient or our k factor so they're still working towards the same hypotheses but just have different legs behind them or different data points they need to prove about whether they should even make on our roadmap math or not so that's such a good point because i think like one of the bit that's quite hard is like as a founder and a vc right you just get really used to the i wouldn't even call it power law but of just like very few things will have this huge outsized impact and as a business you need to continually do them whereas when you you know when you explain to your friends and family like oh that's unlikely to work why would you do it and the answer is because if it does the payoff is so big understandably most people just aren't comfortable with that level of like uh placing bets on the long shots and so if someone accidentally optimizes for you know in this quarter i want to get as many hypotheses correct as possible you're like fantastic it's very easy to have almost no impact because you've proven yes to a bunch of easy stuff that doesn't make sense whereas if it's like here's a goal it's really outsized and almost unreachable see how what you can do to get there and embedding that kind of uh like embedding that culturally to people i think is is really useful like hypotheses mixed with uh this is real like long shot breakthrough mentality it's i think remarkable it's a good frame and this then then ties back into how we do these experimentations and sort of how we can teach it as a moment because like using the the better analogy the question that we always ask ourselves is like well what's the worst case if this were to go wrong because if for example we really don't do well and actually damages our word of mouth coefficient or k factor well then we want to start in a market that actually we don't care as much about so for us that's our australian market so like our power like you know our most important market to the us and the uk we will never do a bet in the us or the uk we will start in australia but then the flip side is that you hit a volume bottleneck right the the us and the uk are far larger markets so if it's something to your point graham you might and not see the results in a while, well then just starting in a tiny market isn't going to allow you to do that at speed.

25:38And so it's a constant debate about where the experimentation should be done, but the fact that the audience has already been built allows that to be faster. Does that mean that it becomes, or I guess it's a question, does it become harder to take those bets the bigger the platform or the bigger the user base becomes? because obviously in some ways the more users you have the better but to your point on damaging the kind of no no no no like i said like so like we have over seven million users right we can take a high risk experiment and still only do it to 70 000 of them but 70 000 people is a is an awesome test bed that is not going to move the the needles of our entire user base but it's still a mass a big enough audience to get statistical significance and not damage our network effects.

26:30The key with us is obviously we're a business built on network effects. And so we kind of need to do experiments to the entire network. Sometimes you don't have to, but if, for example, we are doing something that certain users in a network or a friendship group have and others don't, it can really negate the test as a whole because some of them are then complaining, well, why don't I have this feature? Or many of our features are used by the entire group. So that can make things more challenging, which is why we do our experiments. Geo-controlled rather than just a random sample of using my 70 ,000.

27:04And that's why Australia is so good because it's just the network effects within the country. But also it's a UK, sorry, it's an English-speaking country. It mirrors some of the, a lot of the behaviors of sort of other Western countries. And so it's a great testing bed for us. And sometimes we get it massively right and we put it over and sometimes we don't. and then we're glad we tested it in Australia. I guess the way I think about your question, Mike, is it depends on how much the business is willing to absorb that risk. Because I've been in plenty of businesses where as soon as they get their first 100 ,000 users, you can see like, okay, we don't want to upset anything.

27:42We are now very, very risk averse. Whereas I think that's why companies like Facebook that kept the founder off for so long are able to keep taking those big swings because it takes someone to say, I'm really comfortable with this risk, like at the top, whereas as soon as you hire, I don't know, a professional CEO, as soon as the culture starts to change. So I think that's more of a cultural thing than like technical capability.

28:10Andreas Munk Holm:Guys, now we've just defined what an experiment looks like, the importance of forming hypotheses and framing everything as an hypothesis rather than an execution problem for the sales team to solve. I think those are really important. But let's go to then running experiments at scale. Neil, I think you described it quite well in our pre-call as well. You have 7 million users across 125 countries. I'd love to understand from you, how do you use that as a testing environment? And what does that experiment pipeline look like? And I think that if we tie this back into the opening of this conversation as well, the AI moment is meaning that we're going from running five experiments at one time to running 5 ,000 experiments.

28:53Andreas Munk Holm:So I'm very curious to hear how your machines of experimentation have evolved. Yeah, we're definitely running far more experiments than we ever have before. One, because we can ship faster. Two, because we can analyze faster. And so it's all a compounding benefit of why we can run so many. But we grade our hypothesis and our features by their risk profile and everything is like has uh what is the what is the metric we're trying to move like what is the primary like goal or what are we what's the what a success look like and then everything has a guardrail metric right and so like if something is trying to shift k factor we will check to see if it will have a detrimental impact on retention and we will always assess that and then the markets we pick uh again depending on the risk profile and the grading we give it but australia like i was just saying is a high risk market english speaking behaviorally similar to the us uk but it's just enough to strategic priority for us so it's great for us to run really risky monetization tasks or anything like a damaged word of mouth or anything that is just so novel from like a user experience perspective you throw it to australia anything that is we want high volume for but we want it to be a more controlled test we will pick certain states in the u.s right there's 50 to choose from it's an awesome platform to enable us to go a b c d e etc etc test we pick the states with similar demographics and we just run our different variants there so for example before we launched our subscription paywall six months ago we ran a test to restrict certain of our existing features behind the paywall and we wanted to see which ones would lead to the most conversion to subscription without having a detrimental impact on k-factor word of mouth retention and so we just ran multiple different variant tests in multiple different states so for example florida texas california we will restrict different features and then we'll we'll analyze the the primary metric versus the guardrail and then occasionally we will use non-English speaking countries for lower sort of strategic because there's still lower strategic markets but we want to see the impact that maybe different languages or locales have on our different variance tests so like sort of Germany and Netherlands is a good one they still do have a great English speaking demographic versus for example France and then last of all will be the UK it is our core market so we will go there when we want to do sort of specific pricing tests for for example, because we've got a ton of volume and it's obviously where our network is densest, but it's the last market that we will go for risky changes.

31:38Saying all that with AI, the speed that will now ship to the UK is far faster, right? So it's not like we have to wait a long time. It's like, it's just we want to make sure that if there's anything that will damage or could damage metrics, we haven't launched it in the UK. That's how we got that way. So thinking. We had similar, when I was at Deliveroo, Whereas one of the jobs I had was product manager and the logistics algorithms. So your kind of backend service that picked which way to go through and had this massive, crazy like network effects problem in that you couldn't experiment by applying the test versus control algorithm to one writer versus another because you need to see how they act in aggregate.

32:22And whichever marker you tested on and when you tested on would be hugely influential on the outcome. So if you test it on a Saturday night in central London, the results would be insane compared to like a Monday morning in like in Bath. And so we had to do was like carve up, carve up like all of the world into these little zones that were kind of, I guess, like slightly discrete. So like just separate from each other and then apply it to each one day on, day off for each one for a two week period to make sure we kind of canceled out any effects of which zone it was. So like where it was, time of day, day of week, weather, things like that to get kind of the full picture.

33:02So that's how we approach the challenge of making sure that we gather as many data points as possible by including our kind of core markets like London and Paris. Without taking on too much risk of, you know, getting catastrophically wrong, which did happen once or twice. And you would turn it off quite quickly and eat humble pie quite a lot. it's funny because we are currently at such a kind of comparatively early stage so a lot of times for us experimentation is not about seeing measuring did this move the needle and are we all like you know 95th percentile confidence that it did and more like did this not something up and are we like are we really confident that it hasn't completely bottomed out the metric or can we see like a crazy big growth and I think of the earlier stages it's still really useful to kind of run those experiments, even if you're not going to see anything most of the time, because it does just give you a really great signal one way or the other.

33:56So you at least learn like that. So we're currently in that position where our experimentation is to like, just see, do we get any crazy strong signal in one direction?

34:06Andreas Munk Holm:Mike, you have a vantage point across the portfolio of consumer investments, of course. Can you talk a bit about how you've seen scale of experimentation change over these past two, three years? I wouldn't be able to give you specifics to be honest I know that as we've talked about before definitely like being able to utilize AI software just to accelerate stuff and just do more experiments so it's just I think it's just made it easier to do more I'm curious actually to understand from Neil and Graham to what extent you're either building your own software or using stuff off the shelf to be able to like do many more micro experiments or like which part of it like other ways of automating by getting feedback from customers or like i guess analyzing the qualitative versus the quantitative stuff yeah no so it's not that we've got no external software that's made like testing faster or easier like all of the ux and ui um experiments that we're doing it's not like for example we have a tool that's um got a paywall got a thousand versions of it and is shipping out anything like that it what like we internally are able to ship quicker because of ai and we're able to analyze quicker because of ai and that has enabled experimentation to to 10x but we've since we've started i've built this software internally so like there are external software tools that allow you to build feature toggle system experimentation toggles a b c d test experimentation toggles but we've always had those built internally because it allows enables us to be very very very specific with how we do our testing because every test we want to run differently sometimes we want to do it where we want to specifically target it just to specific networks other times it's countries other times it's bone numbers other times it's based on interest etc etc so it's a bit too nuanced uh to what was that i'm sure though So nowadays with AI, there are some incredible tools out there.

36:05But the main thing is that it's enabled us to ship and analyze so many more tests weekly. And then we review those tests weekly. But I think if you were allowing AI to just run off with the expensivity, it still needs to know the hypothesis and the metrics is trying to move. So still fundamentally, there would always be a sort of a UX UI angle to it.