The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

28:01Coming back to this kind of concern people have of experimentation keeps you from innovating and taking big bets. I know you have this framework, overall evaluation criterion. And I think that helps with this. Can you talk a bit about that? The OEC or the overall evaluation criterion is something that I think many people that start to dabble in A B testing miss. And the question is, what are you optimizing for? And it's a much harder question that people think because it's very easy to say we're going to optimize for money revenue. But that's the wrong question because you can do a lot of bad things that will improve revenue.

28:45So there has to be some countervailing metric that tells you how to improve revenue without hurting the user experience. So let's take a good example with search. You can put more ads on the page and you will make more money. There's no doubt about it. You will make more money in the short term. The question is, what happens to the user experience and how is that going to impact you in the long term? So we've run those experiments and we were able to map out, you know, this number of ads causes this much increase to turn. This number of ads causes this much increase to the time that users take to find a successful result.

29:28And we came up with an OEC that is based on these metrics that allows you to say, okay, I'm willing to take this additional money if I'm not hurting the user experience by more than this much. Right? So there's a trade off there. One of the nice ways to phrase this as a constraint optimization problem. I want you to increase revenue, but I'm going to give you a fixed amount of average real estate that you can use. So for one query, you can have zero ads. For another query, you can have three ads. For a third query, you can have wider, bigger ads. I'm just going to count the pixels that you take, the vertical pixels.

30:08And I will give you some budget. And if you can under the same budget make more money, you're good to go. Right. So that to me, turns the problem from a badly defined, let's just make more money. Right. Any page can start plastering more ads and make more money short term, but that's not the goal. The goal is long term growth and revenue. Then you need to insert these other criteria and what am I doing to the user experience? One way around it is to put this constraint. And another one is just to have these other metrics again something that we did to look at the user experience How long does it take the user to reach a successful like what percentage of sessions are successful?

30:49These are key metrics that were part of the overall evaluation criteria and that we've used I can give you another example by the way from you know the hotel industry or Airbnb that we both work that You can say I want to improve conversion rate But you can be smarter about it and say it's not just enough to convert a user to buy or to pay for a listing I want them to be Happy with it several months down the road when they actually stay there Right, so that could be part of your OEC to say what is the rating that they will give to that? Listing when they actually stay there and that's if that causes an interesting problem because you don't have this data now Now you're going to have it three months from now when they actually stay.

31:34So you have to build the training set that allows you to make a prediction about whether this user, whether Lenny is going to be happy at this cheap place or whether no, I should offer something more expensive because Lenny likes to stay at nicer places where the water actually is hot and comes out of the faucet. That is true. Okay. So it sounds like the core to this approach is basically have a kind of a drag metric that makes sure you're not hurting something that's really important to the business. and then being very clear on what's the long term metric we care most about. To me, the key here, the key word is lifetime value, which is you have to define the OEC such that it is constantly predictive of the lifetime value of the user.

32:17And that's what causes you to think about things properly, which is am I doing something that just helps me short -term or am I doing something that will help me to longer. Once you put that model of lifetime value, people say, okay, what about retention rates. You can measure that. What about the time to achieve a task we can measure that? And those are these countervailing metrics that make it make the OEC useful. And to understand these longer term metrics, what I'm hearing is use kind of models and forecast and predictions, or would you suggest sometimes use like a long term holdout or some other approach, What do you find is the best way to think and see these long?

32:58There's two ways that I like to think about it. One is you can run long -term experiments for the goal of learning something. So I mentioned that at Bing, we did run these experiments where we increase the ads and decrease the ads so that we will understand what happens to key metrics. The other thing is you can just build models that use some of our background knowledge or use some, you know, data science look at historical. I'll give you another good example of this. When I came to Amazon, one of the teams that reported to me was the email team that it was not the transactional emails when you buy something to get an email, but was the team that sent these recommendations, you know, here's a book by an author that you bought.

33:44Here's a product that we recommend. And the question is, how do we give credit to that team? and the initial version was, well, whenever a user comes from the email and purchases something on Amazon, we're going to give that email credit. Well, turn out, this had no countervailing metric. The more emails you send, the more money you're going to credit the theme. And so that led to spam. Literally, a really interesting problem. The team just ramped up the number of emails that they were sending out and claimed to make more money and their fitness function improved. And then so then we backed up and then we said, okay, that we can either phrase this as a constraint satisfaction problem.

34:26You're a lot to send user an email every X days. Or which is what we ended up doing is let's model the cost of spamming the users. Okay, what's that cost? Well, when they unsubscribe, we can't mail them. Okay, so we did some data science study on the side, then we said, what is the value that we're losing from an unsubscribe, right? We came up with a number of a few dollars, but the point was, now we have this countervalent metric. We say, here's the money that we generate from the emails. Here's the money that we're losing a long -term value. What's the trade -off? And then when we started to incorporate this formula, more than half the campaigns that we're being sent were negative.

35:11it. Okay. So it was a huge insight at Amazon about how to send the right campaigns. And this led and this is what I like about these discoveries. This fact that we integrated the unsubscribe led us to a new feature to say, well, let's not lose their future lifetime value through email. When they unsubscribe, let's offer them by default to unsubscribe from this campaign. So when you get an email, you know, there's a new book by the author, the default unsubscribe would be unsubscribe me from author emails. And so now the negative of the countervailing metric is much smaller. And so again, this was a breakthrough in our ability to send more emails and understand based on what users were unsubscribing from, which ones are really beneficial.

36:05I love the surprising results. We all love them. I mean, this is the humbling reality. People talk about the fact that AB testing sometimes leads you to incremental. I actually think that many of these small insights lead to fundamental insights about which areas to go, some strategies we should take, some things we should develop helps a lot. This makes me think about how every time I've done a full redesign of a product, I don't think ever has it ever been a positive result. And then the team always ends up having to claw back what they just hurt and try to figure out what they messed up. Is that your experience, too?

36:47Absolutely. Yeah, in fact, I've published some of these in LinkedIn posts showing a large set of big launches that redesigns and dramatically fail. And it happens very often. So the right way to do this is to say, yes, we want to do a redesign, but let's do it in steps and test on the way and adjust. So you don't need to take 17 new changes that many of them are going to fail. Start to move incrementally in a direction that you believe is beneficial adjust on the way. Right. The worst part of those experiences I find is it took like, I don't know, three, six months, three to six months to build it.

37:31And by the time it's launched, it's just like, we're not going to unlaunch this. Everyone's been working in this direction. All the new features are assuming this is going to work. And you're basically stuck, right? I mean, this is the SunCost fallacy, right? We invested so many years in it. Let's launch it even though it's bad for the user. No, that's terrible. Yeah. Yeah. So this is the other advantage of recognizing this humble reality that most ideas fail, right? If you believe in that statistics that I published, then doing 17 changes together is more likely to be negative. Do them in smaller increments, learn from, it's called OFAT, one factor at a time.

38:12Do one factor, learn from it, and adjust of the 17 maybe you have four good ideas. those are the ones that will launch and be positive. I generally agree with that and always try to avoid a big redesign, but it's hard to avoid them completely. There's often team members that are really passionate and like we just need to rethink this whole experience. We're not going to incrementally get there. Have you found anything effective in helping people either see this perspective or just making a larger, bit more successful? And by the way, I'm not opposed to large redesigns. I try to give the team the data to say, look, here are lots of examples where big redesigns fail.

38:54Try to decompose your redesign if you can decompose it to one factor of time to a small set of factors at a time. And learn from these smaller changes, what works and what doesn't. Now, it's also possible to do a complete redesign. I'm just as you said yourself, they'd be ready to fail. Right? I mean, do you really want to work on something for six months or a year and then run the AB test and realize that you've hurt revenues or other key metrics by several percentage points and a data driven organization will not allow you to launch. What are you going to write in your annual review? Yeah, but nobody ever thinks it's going to fail.

39:34I think no, we got this through talk to so many people. But I think organizations that start to run experiments are humbled early on from the smaller changes. Right. You're right. Nobody, I'll tell you a funny story. When I came from Amazon, the Microsoft, I joined a group and for one reason or another that group disbanded a month after I joined. And so people came to me and said, look, you just joined the company. You're a partner level. you figure out how you can help Microsoft. And I said, I'm gonna build an experimentation platform because nobody at Microsoft is running experiments and more than 50 % of ideas in Amazon that we tried failed.

40:19And the classical response was, we have better PMs here. Right, there was this complete denial that it's possible that 50 % of ideas is that Microsoft is implementing in a three -year development cycle, by the way. This is how long it took Office to release. It was a classical every three years we released. The data came about showing that being was the first to truly implement experimentation at scale and we shared with the rest of the companies the surprising results. And so when office was, and this was, you know, credit to Chilu and Satchinadella, they were ones that says, Ronnie, you know, you try to get office to run experiments, we'll give you the air support.

41:10And it was hard, but we did it. You know, it took a while, but office started to run experiments and they realized that many of their ideas were failing. You said that there's a site of a failed redesigns. Is that in your book or is that a site that you can point people to to kind of help build this case? It's on I teach us in my class, but I think I've posted this on LinkedIn and answered some questions. I'm happy to put that in the notes. Okay. Cool. We'll put that in the show notes because I think that's the kind of data that often helps convince a team. Maybe we shouldn't rethink this entire onboarding flow from scratch.

41:44Maybe we should kind of iterate towards and learn as we go. This episode is brought to you by Epo. Epo is a next generation AB testing platform built by Airbnb alums for modern growth teams. Companies like DraftKings, Zapier, ClickUp, Twitch, and Cameo rely on Epo to power their experiments. Wherever you work, running experiments is increasingly essential. But there are no commercial tools that integrate with a modern growth team stack. This leads to waste of time building internal tools or trying to run your own experiments through a clunky marketing tool. When I was at Airbnb, One of the things that I loved most about working there was our experimentation platform.

42:20Whereas able to slice and dice data by device types, country, user stage, Epo does all that and more delivering results quickly, avoiding annoying prolonged analytic cycles, and helping you easily get to the root cause of any issue you discover. Epo lets you go beyond basic click -through metrics, and instead use your North Star metrics, like activation, retention, subscription, and payments. EpoSupports test on front end, on the back end, email marketing, even machine learning clients. Check out Epo at GetEPPO .com, that's Getepo .com, and 10X, your experiment velocity. Is it ever worth just going, let's just rethink this whole thing and just give it a shot to break out of a local minima or a local maxima essentially?

43:03Yeah, so I think what you said is fair. I mean, I do want to allocate some percentage of resources to big bets. as you said, we've been optimizing this thing to hell, could we completely redesign it? It's a very valid idea. You may be able to break out of a local minimum. What I'm telling you is 80 % of the time you will fail. So be ready for that, right? What people usually expect is my redesign is going to work. No, you're most likely going to fail, but if you do succeed, it's a breakthrough. I like this 80 % rule of thumb. Is that just like a simple way of thinking about it 80 % your experience?

43:39That's my rule of thumb. And you had, I've heard people say it's 70 % or 80%. But it's in that area where I think when you talk about how much to invest in the known versus the high risk high reward, that's usually the right percentage that most organizations end up doing this allocation. But you interviewed Trey, so I think he mentioned that at Google is like 70 % you know the search and ads and it's a 20 % for some of the apps and new stuff and then it's the 10 % for infrastructure Yeah, I think the most important point there is if you're not running an experiment 70 % of stuff you're shipping is hurting your business Well, not hurting it may it's flat too negative some of them are flat And by the way flat to me if something is not static that's a no ship because you've just introduced more code There is a maintenance overhead to shipping your stuff.

44:39I've heard people say, look, we already spent all this time. The team will be demotivated if we don't ship it. And I'm not that's wrong, guys. Let's make sure that we understand that shipping this project has no value is complicating the codebase. Maintenance costs will go up. You don't ship on flat unless it's a sort of a legal requirement. right? When legal comes along and says you have to do X, Y, you have to ship on flat or even negative and that's understandable. But again, I think that's something that a lot of people make the mistake of saying legal told us we have to do this. Therefore, we're going to take the hits.

45:15No. Legal gave you a framework that you have to work under, try three different things and ship the one that hurts the least. I love that. Reminds me when Airbnb launched the rebrand, even that they ran as an experiment with the entire homepage redesign, the new logo and all that. I think there's a long -term holdout even. I think it was positive in the end, from what I remember. Speaking of Airbnb, I want to chat about Airbnb briefly. I know there's, and you're limited in what you can share, but it's interesting that Airbnb seems to be moving in this other direction where it's becoming a lot more top -down, Brian Vision oriented and Brian's even talked about how he's less motivated to run experiments.

45:58He doesn't want to run as many experiments as they used to. Things are going well. And so, you know, it's hard to argue with the success potentially. You worked there for many years. You ran the search team, essentially. I guess just what was your experience like there? And then roughly, what's your sense of how things are going? Where it's going? As you know, and I'm restricted from talking about AirBnB, I will say a few things that I am allowed to say. One is in my team, in search relevance, everything was a B test. So while Brian can focus on some of the design aspects that people who are actually doing you know the neural networks and the search everything was a B test to help.

46:42So nothing was launching without a B test. We have targets around improving certain metrics and everything was that A because now other teams some did some did not. I will say that you know when you say things are going well, I think we don't know the counterfactual. I believe that had Airbnb kept people like Greg really, which was pushing for a lot more data driven and had Airbnb run more experiments, it would have been in a better state than today, but it's the counterfactual. We don't know. That's a really interesting perspective. Yeah. Airbnb is such an interesting natural experiment of a way of doing things differently, there's like, de -emphasizing experiments.

47:22And also, they turned off paid ads during COVID. And I think, I don't know where it is now, but it feels like it's become a much smaller part of the growth strategy. Who knows if they've ramped it up to back to where it is today. But I think it's going to be a really interesting case study looking back, I don't know, five, 10 years from now. It's a one -off experiment where it's hard to assign value to some of the things that Airbnb is doing. I personally believe it could have been a lot bigger. and a lot more successful if it had run more control experiments. But I can't speak about some of those that I ran and that showed that some of the things that were initially untested were actually negative and could be better.

48:04All right. Mysterious. One more question Airbnb. You were there during COVID, which was quite a wild time for Airbnb with Sunshine on the podcast, talking about all the craziness that went on when travel basically stopped. And there was a sense that our being view is done and travel is not going to happen for years and years. What's your take on experimentation in that world where you have to really move fast, make crazy decisions, make big decisions? What was like during that time? So I think actually in a state like that, it's even more important to run AV tests, right? because what you want to be able to see is if we're making this change, is it actually helping in the current environment?

48:50There's this idea of external generalizability. Is it going to work out now during COVID? Is it going to generalize later on? These are things that you can really answer with the control of experiments. And sometimes it means that you might have to replicate them six months down when COVID is not as impactful as it is. Saying that you have to make decisions quickly, to me, I'll point you to the success rate. Like if in peacetime you're wrong two -thirds to 80 % of the time, why would you be suddenly right in wartime? Right? In COVID time. So I I don't believe in the idea that because bookings went down materially, the company should certainly not be data -driven and do things differently.

49:41I think if every and be stayed the course, did nothing, the revenue would have gone up in the same way. Fascinating. In fact, if you look in one investment, one big investment that was done at the time was online experiences. And the initial data wasn't very promising and I think today it's a footnote. Yeah, what a... Another case study for the history books, Airbnb experiences. I want to shift a little bit and talk about your book, which you mentioned a couple times. It's called Trust Worthy Online Control Experiments. And I think it's basically the book on A, B testing. Let me ask you just what's the price you most about writing this book and putting it out and in the reaction to it?

50:23I was pleasantly surprised that it sold more than what we thought, more than what Cambridge predicted. So when, when, first we were approached by Cambridge after a tutorial that we did to write a book, I was like, I don't know, this is too small of an inch area. And I, you know, they were saying, so you'll be able to sell a few thousand copies and helped the world. I found my call authors which are great. We wrote a book that we thought is not statistically oriented, has fewer formulas than you normally see, and focuses on the practical aspects and on trust which is the key. The book, as I said, was more successful.

51:21It's And so it's great to see that we helped the world become more data driven with experimentation. And I'm happy because of that. I was pleasantly surprised. By the way, all proceeds from the book are donated to charities. So if I'm pitching the book here, there is no financial gain for me from having more copies sold. I think we made that decision, which was a good decision. All proceeds go with the charity. Amazing. I didn't know that. We'll link to the book in the show notes. You talked about how trust, like it's, trust is in the title. You just mentioned how important trust is to experimentation.

51:55A lot of people talk about, how do I run experiments faster? You focus a lot on trust. Weiss trust is so important in running experiments. So to me, the experimentation platform is the safety net. And it's an Oracle. So it serves really two purposes. The safety net means that if you launch something bad, you should be able to abort quickly. Right? safe deployments, safe velocity, there are some names for this. But this is one key value that the platform can give me. The other one, which is the more standard one, is at the end of the two -week experiment, we will tell you what happened to your key metric and do many of the other surrogate and debugging and guardrail metrics.

52:39Trust builds up, it's easy to lose. And so to me, it is very important that when you present this and say, this is science, this is a control experiment, this is the resolve. You better believe that this is trustworthy. And so I focus on that a lot. I think it allowed us to gain the organizational trust that this is really, and the nice thing is when we built all this checks to make sure that the experiment is correct. If there was something wrong with it, we would stop and say, hey, something is wrong with the experiment. And I think that's something that some of the early implementations in other places did not do and it was a big mistake.

53:25I mentioned this in my book, so I can mention this here, optimizely, in its early days, we're very statistically naive. They sort of said, hey, we're real time, we can compute your p -values in real time. And then you can stop an experiment when the p -value is statistically significant. That is a big mistake. That inflates what's called type 1 error or the false positive rate. Materially, so if you think you've got a 5 % type 1 error or you aim for that p -value less than .05, using real time sort of p -value monitoring to optimize the offered, you would probably have a 30 % error rate. So what this led is that people that started using optimizely thought that the platform was telling them they're very successful.

54:15But when they actually started to see, well, I told them this is positive revenue, but I don't see this over time. Like, by now we should have made double their money. So their question started to come up around the trust in the platform. There's a very famous pose that some of you wrote about how optimizely almost got me fired by a person who basically said, look, I'm going to say, I'm going to say, I'm I came to the organ, I said, we have all these successes, but then I said something is wrong. And he tells of how he ran an AA test when there is no difference between the A and the B and optimize the tolem that it was statistically significant.

54:49Too many times, optimized the learned, optimized the, several people pointed. I pointed this out in my Amazon review of the book that the optimized the authors wrote early on. I said, hey, you're not doing the statistics correctly. other, you know, Ramesh, Johari, it's Stanford, pointed this out, became a consultant to the company, and then they fixed it. But to me, that's a very good example of how to lose trust. They lot of trust in the market. They lost all this trust because they built something that had very much inflated error. That is a pretty scary to think about. You've been running all these experiments, and they weren't actually telling you accurate results.

55:34What are signs that what you're doing may not be valid if you're starting to run experiments and then just how do you avoid having that situation? What kind of tips can you share for people trying to run experiments? There's a whole chapter of that in my book, but I'll say maybe one of the things that is the most common occurs by far, which is a sample ratio of mismatch. Now, what is a sample ratio of mismatch? If you designed the experiment to send 50 % of users to control and 50 % of users to the treatment, it's supposed to be a random number or hash function. If you get something off from 50%, it's a red flag.

56:15So let's take a real example. Let's say you're running an experiment and it's large, it's got a million users and you've got 50 .2. So people say, well, I don't know, it's not going to be exactly the same as 50 .2, reasonable or not. Well, there's a formula that you can plug in. I have a spreadsheet available for those that are interested in. And you can tell here's how many users are in control. Here's how many users have in treatment. My design was 5050 and it tells you the probability that this could have happened by chance. Now, in a case like this, you plug in the numbers and it might tell you that this should happen one in half a million experiments.

56:53Unless you run half a million experiment, very unlikely that you would get a 50 .2 versus 49 .8 split and therefore something is wrong with the experiment. Now, people, I remember when we first implemented this check, we were surprised to see how many experiments suffered from this. There's a paper that was published in 2018 and where we share that Ed Microsoft, even though we'd be running experiments for a while, is around 8 % of experiments that suffered from the sample ratio mismatch. And it's a big number. I think about this, you're running 20 ,000 experiments a year. So many of them, 8 % of them are invalid.

57:37And somebody has to go down and understand what happened here. We know that we can't trust the results, but why? So over time, you begin to understand there's something wrong with the pipe data pipeline. There's something that happens with bots. spots are a very common factor for causing a sample ratio of mismatch. So, there's a whole, that paper that was published by my team talks about how to diagnose sample ratio of mismatches. In the last, probably a year and a half, it was amazing to see all these third party companies implement sample ratio of mismatches. And all of them were reporting, oh my god, you know, 6 %, 8%, 10%, so yeah, they were where it's sometimes fun to go back and say, how many of your results were in the past or in valid before you had this ampleration, this much test.

58:32Yeah, this frightening. Is the most common reason this happens is you're signing users in kind of the wrong place in your code? So when you say most common, I think the most common is bots. Somehow they hit the controller, the treatment in different proportions because you changed the website, the Bob may fail to parse the page and try to hit it more often. That's a classical example. Another one is just a data pipeline. We've had cases where we were trying to remove bad traffic under certain conditions and it was skewed because of the control and treatment. I've seen people that start an experiment in the middle of this site on some page, but they don't realize that some campaign is pushing people from the side.

59:13So there's multiple reasons. It is surprising how often this happens. And I'll tell you a funny story, which is, when we first added this test to the platform, we just put a banner saying, you have a sample ratio of mismatch, do not trust these results. And we noticed that people ignored it. They were starting to present results that had this banner. And so we blanked out the scorecard. We put a big, you know, red can't see this result. You have a sample ratio of mismatch click okay to expose the results and why we do we need that okay? We need that okay button because You want to be able to debug the reasons and sometimes the metrics help you understand why you have a sample ratio of mismatch So we blanked out the scorecard we had this button and then we started to see that people pressed the button It's still presented the results of experiments with sample ratio mismatch So we ended up with an amazing compromise which is every number in the scorecard was highlighted with a red line.

1:00:18So that if you took a screenshot, other people could tell you how the sample ratio is less. Freaking product managers. This is intuition. People just say, oh my, it's the realm of small. Therefore I can still present the result. People want to see success. I mean, this is a natural bias. And then we have to be very conscientious and fight that bias and say with something looks to get to be true, investigate. Which is a great segue to something you mentioned briefly, something called, time and law. Yeah. You can talk about that. Yeah, so time is law, you know, the general statement is if any figure that looks interesting or different is usually wrong, it was first said by this person in the UK who worked in radio media, but I'm a big fan of it.

1:01:06And my main claim to people is if the result looks too good to be true, if you suddenly moved your normal movement of an experiment is under 1%, and you suddenly have a 10 % movement, hold the celebratory dinner. It was just your first reaction, right? Let's thank everybody to fancy dinner because we just improved revenue by millions of dollars. hold that dinner, investigate, see because there's a large probability that something is wrong with the result. And I will say that 9 out of 10 when we call a time is law, it is the case that we find some flaw in the experiment. Now there are obviously outliers, right?

1:01:47That first experiment that I should where we promoted the made -long -out titles, that was successful, but that was replicated multiple times and double and triple checked and everything was good about it. Many other results that were so big turn out to be false. So I'm a big, I'm a big, big fan of trying this law. There's a deck I can also give this in a note where I shared some real examples of trying this law. Amazing. I want to talk about rolling this out, accompanying things that you run into, that fail. But before I get to that, I'd love for you to explain just p -value. I know that people kind of misunderstand it and this may be a good time, I'm just, help people understand, what is it actually telling you, P value of say 0 .05?

1:02:29I don't know if this is the right forum for explaining P values, because the definition of a P value is simple. What it hides is very complicated. So I'll say one thing, which is, many people assign one minus P value as the probability that your treatment is better than control. So you ran an experiment, you got a P value of 0 .02, they think there's a 98 % probability that the treatment is better than the control. That is wrong. They say rather than defining p -values, I want to caution everybody that the most common interpretation is incorrect. P -value assumes it's a conditional probability or an assumed probability.

1:03:15It assumes that the null hypothesis is true and we're computing the probability that the data we're seeing matches the hypothesis. There's no hypothesis. In order to get the probability that most people want, we need to apply Bayes rule and invert the probability from the probability of the data given to hypothesis to the probability that hypothesis given the data. For that, we need an additional number, which is the probability, the prior probability that the hypothesis that you're testing is successful or not. That's an unknown. What we do is we can take historical data and say, look, people fail two thirds of the time or 80 % of the time.

1:03:58And we can apply that number and compute that. We've done that in a paper that I will give you the notes so that you can assess the number that you really want that what's called the false positive risk. So I think that's something for people to internalize that what you really want to look as this false positive risk, which tends to be much, much higher than the 5 % of people think. Right? So if you're, I think the classical example in the Airbnb where the failure rate was very, very high is that when you get a statistically significant resolve, let me actually call the note, because I know I have the actual number.

1:04:34If you're at Airbnb, where the success rate of, or Airbnb search where the success rate is only 8%, And if you get a statistically significant result with the p value less than 0 .05, there is a 26 % chance that this is a false positive result. Right? It's not 5%. It's 26%. So that's the number that you should have in your mind. And that's why when I worked at Airbnb, one of the things we did is we said, okay, if you're less than 0 .05, but above 0 .01, rerun, replicate. When you replicate, you can combine the two experiments and get a combined p -value using something called Fisher's method or Staufer's method and that gives you the joint probability and that's usually much, much lower.

1:05:19So if you get 2 .05 or something like that, then the probability that you got them is much, much lower. Wow. I have never heard it described that way. Makes me think about how like even data scientists in our teams are just like, this isn't perfect. like we're not 100 % sure this experiment is positive but on balance if we're launching positive experiments we're probably doing good things it's okay if sometimes we're wrong. It's true on balance you're probably better than 50 -50 but people don't appreciate how much that 26 % that I mentioned is high and the reason that I want to be sure is that I think it leads to this idea of the learning the institutional knowledge which you want to be able to say is share with the Oregon success.

1:06:03And so you want to be really sure that you're successful. So by lowering the P value, by forcing teams to work with the P value maybe below point, one, and do replication or hires, then you can be much more successful. And the false positive rate will be much, much slower. Fascinating. And also shows the value of keeping track of just what percentage of experiments are failing historically at the company or within that specific product. say someone listening wants to start running experiments, say they have tens of thousands of users at this point. What would be the first couple of steps you'd recommend?

1:06:37Well, see, if they have somebody in the org that has previously been involved with the experiment, that's a good way to consult internally. I think that the key decision is whether you want to build or buy. There's a whole series of eight sessions that I posted on LinkedIn where I invited guest speakers to talk about this problem. So if people are interested, they can look at how what the vendors say and what agency said about build versus buy question. And it's usually not at zero one. It's usually a both. You build some and you buy some and it's a question of you build 10 % or do you build a 90 %?

1:07:17I think for people starting the third party products that are available today are pretty good. This wasn't the case when I started working. So when I started building running experiments at Amazon, we were building the platform because nothing existed. Same at Microsoft. I think today there's enough vendors that provide good experimentation platforms that are trustworthy that I would say not a good way to consider using one of those. So you're at a company where there's resistance to experimentation and AB testing, whether it's a startup or a bigger company, whatever you found works in helping shift that culture and how long does that usually take, especially at a larger company?

1:07:57My general experience is with Microsoft where we went with this beachhead of Bing. We were running a few experiments and then we were asked to focus on Bing and we scaled experimentation and build a platform at scalehead Bing. Once being was successful and we were able to share all these surprising results, I think many, many more people in the company were amenable. And it was also the case that helped a lot. There's a usual cross -pollination people from being moved out to other groups and that helped these other groups say, hey, there's a better way to build software. So I think if you're starting out, find a place, find a team where experimentation is easy to run and by that I mean they're launching often right don't go with the team that launches every six months or you know office used to launch every three years go with the team that launches frequently you know they're running on sprints they launch every week or two sometimes they launch data I mean being used to launch multiple times a day and then make sure that you understand the question of the OEC is it clear what they're optimizing for right there are some groups where you You can come up with a good OEC.

1:09:09Some groups are harder. I remember one funny example was the Microsoft .com website, which this is not MSN. This is Microsoft .com. Has multiple different constituencies that are trying to determine this is a support site and this is the ability to sell software through this site and it's and and warn you about about safety and updates. It has so many goals. I remember when the team said we want our own experiments and I brought the group in and some of the managers and I said, do you know what you're optimizing for? It was very funny because they surprised me. They said, Hey Ronnie, we read some of your papers.

1:09:52We know there's this term called OEC. We decided the time on site is OEC. And I said, wait a minute. it. Some of your main goals as support site is people more spending more time in a support site, a good thing or a bad thing. And then half the room thought that more time is better and half the room thought that more time is worse. So, and always seems bad if directionally you can't agree on it. That's a great tip. Along these same lines, I knew you're a big fan of platforms and building a platform to run experiments versus just one off. Experiments can just talk briefly about that to give people a sense of where they probably should be going with their experimentation approach.

1:10:32I think the motivation is to bring the marginal cost of experiments down to zero. The more you self -service, go to a website, set up your experiment, define your targets, define the metrics that you want. People don't appreciate that the number of metrics starts to grow really fast if you're doing things right. And being, you could define 10 ,000 metrics that you wanted to be in your scorecard. Big number. So it was so big and people said it was computationally inefficient. We broke them into templates so that if you were launching a UI experiment, you would get this set of 2 ,000. If you're doing a revenue experiment, you would get this set of 2 ,000 if you're doing.

1:11:15So the point was build a platform that can quickly allow you to set up and run an experiment and then analyze it. I think you know one of the things that I will say to Airbnb is the analysis was relatively weak and so lots of data scientists were hired to be able to compensate for the fact that the platform didn't do enough and so and this happens in other organizations too where there's a straight -off but if you're building a good platform invest in it so that more and more automation will allow people to look at the analysis without the need to involve a data scientist. We published a paper again, I'll give it in the notes with this sort of a nice matrix of six axes and how you move from crawl to walk to run to fly and what you need to build on those six axes.

1:12:08So if one of the things that I do sometimes when I consult is I go into the organization and say, where do you think you are on these six axes? And that should be the guidance for what are the things you need to do next. This is gonna be the most epic show notes episode and we've had yet. Maybe a last question. We talked about how important trust is to any experiments and how even though people talk about speed, trust ends up being most important. Still, I wanna ask you about speed. Is there anything you recommend for helping people run experiments faster and get results more quickly that they can implement?

1:12:40Yeah, so I'll say a couple of things. One is if your platform is good, then when the experiment finishes, you should have a scorecard soon after. They mean it takes a day, but it shouldn't be that you have to wait a week for the data scientist. To me, this is the number one way to speed up things. Now, in terms of using the data efficiently, there are mechanisms out there under the title of variance reduction that help you reduce the variance of metrics so that you need less users so that you can get results faster. Some examples that you might think about are capping metrics. So if your revenue metric is very skewed, maybe you say, well, if somebody purchased over $1 ,000, let's make that $1 ,000.

1:13:27It Airbnb, one of the key metrics, for example, is NightsBook. Well, it turns out that some people book tens of nights, they're like an agency or something, hundreds of nights, You may say, okay, let's just cap this. It's unlikely that people book more than 30 days in a given month. So that various reduction technique will allow you to get statistically significant results faster. And a third technique is called Cupid, which is an article that we published. Again, I can give it in the notes, which uses the pre -experiment data to adjust the result. And we can show that you get the result as unbiased, but with lower variance and health, hence it requires fewer numbers.

1:14:11Ronnie, is there anything else you want to share before we get to our very exciting lightning round? No, I think we've asked a lot of good questions. Hope people enjoy this. I know they will. Lightning round. No, lightning round. Here we go. I'm just going to roll right into it. What are two or three books that you recommended most to other people? There's a fun book called Calling Bullshit, it, which is despite the sort of name, which is a little extreme, I think, for entitled, it actually has a lot of amazing insights that I love. And it sort of embodies, in my opinion, a lot of the time is love showing that things that are too extreme, your bullshit meter should go up and say, hey, I don't believe that.

1:14:54So that was, that's my number one recommendation. There's a slightly older book that I love called Heart Facts, Dangerous Half Truths, and Total Nonsense by the Stanford professors from the Graduate School of Business. Very interesting to see many of the things that we grew up with as well understood, turn out to have no justification. And then, so with Stranger Book, which I love on the verge of psychology, it's called mistakes were made but not by me about all the fallacies and that we fall into and the humbling results from that. The titles of these are hilarious and there's a common theme across all these books.

1:15:43Next question, what is a favorite recent movie or TV show? So I recently saw a short series called Chernobyl on the disaster. I thought it was amazingly well done. Yeah. I highly recommend it. You know, based on true events, you know, as usual, there's some freedom for the artistic movie that it was kind of interesting. At the end they say, this woman in the movie wasn't really a woman. It was a bunch of 30 data scientists, 30 scientists that in real life presented all the data to the leadership of what to do. I remember that. Fun fact, I was born in Odessa, Ukraine, which was not so far from Chernobyl.

1:16:29I remember my dad told me he had to go to work. They called them into work that day to clean some stuff off the trees, I think ash from the explosion or something. It was like far away where I don't think we were exposed. hoes, but yeah, we were in the vicinity. That's pretty scary. My wife thinks I've, every time someone's wrong with me, she's like, oh, that must be a Chernobyl thing. Okay, next question, favorite interview question you like to ask people when you're interviewing them. So it depends on an interview, but I'll give you, when I do a technical interview, which I do less of, but one question that I love that is amazing, how many people it throws away, for languages like C++ is, tell me what the static qualifier does.

1:17:16And for multiple, you know, you can do it for a variable, you can do it for a function. And it is just amazing that I would say more than 50 % of people that interview for engineering job cannot get this and get it awfully wrong. Definitely the most technical answer to this question yet. I love it. Okay, what's a favorite recent product you've discovered? Now you love it. Blink cameras. So, a Blink camera is this small camera. You stick in two AA batteries and it lasts for about six months. They claim up to two years. My experience is usually about six months. But it was just amazing to me how you can throw these things around in the yard and see things that you would never know otherwise.

1:18:08You know, some animals that go by, we had a skunk that we couldn't figure out how it was entering. So I threw five cameras out and I saw where he came in. Where did he come in? He came in under a whole defense that was about this high. I have a video of this thing just squishing underneath. We never would have assumed that it came from there from the neighbor. But yeah, these things have just changed. And when you're away on a trip, it's always nice to be able to say, you know, I can see my house, everything's okay. At one point, we had a false alarm and the cops came in and have this amazing video of how they're entering the house and pulling the guns out.

1:18:56You got to share that on TikTok. That's good content. Well, okay, blood cameras. We'll set those up in my house. Yes. What is something relatively minor you've changed in the way your team's developed product that has had a big impact on their ability to execute? I think this is something that I learned at Amazon, which is a structured narrative. So Amazon has some variants of this, which sometimes, but then the gov than anyone, six page or something. But when I was at Amazon, I still remember that email from Jeff, which is no lower PowerPoint, I'm gonna force you to write a narrative. I took that to heart and many of the features that the team presented instead of a PowerPoint.

1:19:42You start off with a structured document that tells you what you need, the questions you need to answer for your idea, and then we review them as a team. And Amazon, these were like paper -based. Now it's all based on Word or Google Docs where people comment. And I think the impact of that was amazing. I think the ability to give people honest feedback and have them appreciate and have it stay after the meeting was in these notes on the document just amazing. Final question. Have you ever run an A B test on your life, either your dating life, your family, your kids? And if so, what did you try?

1:20:20So there aren't enough units. Remember I said you need 10 ,000 of something. The run through A, B test. I do, I will say a couple of things. One is I try to emphasize to my family and friends and everybody this idea called the hierarchy of evidence. When you read something, there's a hierarchy of trust levels. If something is anecdotal, don't trust it. If there was an experiment, it was observational, given some bit of trust, as you get more up and up to a natural experiment and a control experiments and multiple control experiments, your trust levels should go up. So I think that's a very important thing that a lot of people miss when they see something in the news is where does it come from?

1:21:06I have a talk that I've shared of all these observational studies that people made that were published and then somehow a control experiment was run later on and proved that it was directionally incorrect. So I think there's a lot to learn about this idea of the hierarchy of evidence and share with our family and kids and friends and there's a now I think there's a book that's based on this is like how to read a book. Well, Ronnie, the experiment of us recording a podcast I think is 100 % positive of p value 0 .0. Thank you so much for being here. Thank you so much for inviting me and for great questions.

1:21:47Amazing. I appreciate that. Two final questions working folks, finding online if they want to reach out. And is there anything that listeners can do for you? Finding me online is easy. It's linked in. And what can people do for me? You know, understand the idea of control experiments as a mechanism to make the right data -driven decisions use science. Learn more by reading my book if you want. Again, all proceeds go to charity. And if you want to learn more, there's a class that I teach every quarter. On Maven, we'll put in the notes how to find it and some discount for people who manage to stay all the way to the end of this podcast.

1:22:30Yeah, that's awesome. We'll include that at the top. So people don't miss it. So there's be a code to get a discount on your course. Ronnie, thank you again so much for being here. This was amazing. Thank you so much. Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or a leaving review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny'spodcast .com. See you in the next episode!