Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor)

23:31Or do I only have one of these two or do I have neither? Okay, if you don't have both, you can call yourself whatever you want to call yourself, but at this moment in time, you're not a marketplace. All right, if you have one, congratulations, you've won the game on one side of the market and now you can, you can, if you won, you have a choice point. You can lean into growth on the side that you're doing well with, right? You go, a ton of user, ton of buyers, great, lean into it, get more buyers. That's one option. It's, there's no shame in not being a marketplace. There's, there's, you know, scaling a business, a scaling a business, if that's the way to do it, do it.

24:05If you decide you want to be a marketplace, then at that moment, when you've got a lot of buyers, but not a lot of sellers or a lot of sellers, not a lot of buyers, the choice you're facing is, how do I take advantage of having that one side scale to attract the other side? We can talk more about that, but there's a lot of ways to kind of hack that, to think about how, so, you know, to take Uber as an example, right? They would walk into New City and one thing that, that, you know, Uber was, was kind of commonly known for doing this was back in the days when really Uber Black was the only service is they just hand out coupons for free rides that kind of, events, parties, things like that, to take people home.

24:40And that was a way of saying, hey, we're subsidizing the drivers in this city. That's our scaled side. Now we're going to use that subsidized driver base to attract riders. Okay? So that's like how do you get that flywheel going? And, and again, you know, many people have written about how to take liquidity, scale liquidity on one side and use it to attract the other side. If you don't have either side, don't worry about it, don't worry about being a marketplace, worry about scaling one side. And in that world, it's opened up, it opens your visibility up completely into the advice of many, many, you know, startup advisors, right?

25:17People who have advice not so much about scaling a marketplace, but about scaling a startup. And I really, I want to say you got to let the ego go at that point, right? Like it's fine to articulate to people that your vision of the future is to be applied from a marketplace. As I said, virtually every business is going to have that option at some point and, you know, the modern tech enabled economy anyway. So you're not saying something people don't already know when you tell, you know, an advisor or an investor that. But I do think you need to be humble enough at the starting point to recognize that there's no sense to talking about a marketplace if you don't have scaling on either side yet.

25:51And then it becomes a question of a business model, you need economics of can I build say a door dash not as a marketplace? Can I just hire a bunch of people delivering right? It's like a, is this even possible in a different route? Yeah, that's a, that's a great point. Yeah, one of the things I think that's useful for people to think about here that you're raising at some level, it's kind of tied up, I think, with that question of whether I should have employees or kind of contractor freelance work on one side of the marketplace. And that's actually a pretty old question in economics. The way we talk about it often is a distinction between a market or a firm.

26:31And kind of one of the interesting puzzles in economics, Ronald coast is a famous economist who thought about this is, well, you know, if markets are so efficient, why do we need firms? Right? Because if markets are efficient in matching labor up with things that need to get done, why would you ever need a firm? And that's like one of the earliest recognitions that transactions can cost a real thing. And that's kind of one of the things that firms are solving for. And I love what you're saying because what it's recognizing is, hey, for your frictions, the best resolution to that might not be to have a marketplace, might actually be to have very tightly controlled labor.

27:05A good example of this actually, you know, stitch fix, I think one of the things that's cool about stitch fix is the experience that people had early on with stylists at stitch fix. Am I happy customer, by the way? I think, yeah, that's a big stuff. Yeah, I think, I think like one of the great things about that experience is it felt magical to have have someone who kind of got to know you, right? But that depends on a relationship that doesn't feel like a freelance relationship every single time you're going back. Another example that that I would pull out is pretty much any healthcare platform.

27:37So, you know, for example, for physical therapy, you know, if you reared, if every time you went to a physical therapy platform, you just got randomly matched to whoever happened to be available then. So, I think there's some there's some curation that needs to happen of that relationship. Does that mean full employee, maybe not, but it does mean you have to think a little bit about exactly as you brought up, you know, what's the what's the nature of curation of your of your labor pool? Awesome. Okay. So, let's come back to a point you made early on around the importance of data and the power of data in actually making your marketplace a lot more efficient and work more effectively.

28:12So, say that you have a data scientist or a data analyst or someone that is helping you optimize your marketplace. Where do you often find the biggest leverage and opportunity for a data person to help you make your marketplace more effective? This is an incredible question, right? Because I think I could answer it, you know, a number of different ways. Like one question I think there that's kind of basic is just what should this person be doing? And I'm not I'm going to actually kind of evade that question a little bit. I'm going to give some examples of what they could do, but I feel like that's one where context matters a lot.

28:45So, as an example, you know, at a ride sharing or grocery delivery marketplaces, pricing means actually what do you pay for that rider? What do you pay for that delivery? Right? So, that's actually the price that's set at the moment you actually place there just to be clear by the way, if you order from DoorDash, I don't mean the price of the restaurant. I mean, what do you pay to DoorDash, right? What's that? What's that fee? Is there a surcharge? Because it's, you know, surge or whatever, right? So, okay, so that's a thing, right? Now, but that's not really a thing in a marketplace where the platform's not setting the prices.

29:16So, Airbnb really hosts are the ones who are charged of setting prices for their listings. So, one answer to your question is, if I'm in a place like, you know, over a lift DoorDash, I want to have good data scientists thinking about pricing, because that seems like something which should be heavily dependent on the instantaneous state of supply and demand in my marketplace, right? So, that's one type of answer is, well, do I need people data scientists working on pricing? Do I need data scientists working on search? Why search? Because maybe in my marketplace, finding the needle in the haystack is really the biggest highest friction problem.

29:48So, maybe I need a lot more data science thing, that's what I'm going to evade, okay? I'm going to focus more on something completely different, which is just a more philosophical point about what a data scientist does. So, in a lot of companies today, especially, a main thing that you ask data scientists to do is build what's called the machine learning model. You know, machine learning model even already can mean a lot of things to a lot of different people. I'm going to focus on a very concrete. Your ass in predict something. When I started at ODSK, this is in 2012, one of the funny things about me is I started ODSK because I had an academic career up to that point about 10 years just building mathematical models of things.

30:30I was not really very much of a data scientist up to that point. What I expected would happen is I go to industry and I'd be told, hey, look how important data is, and definitely my eyes were opened. And one of the first things I was asked to think about is, well, okay, someone comes to ODSK post a job. Workers apply to that job. Predict which of these workers is most likely to be hired on that job. That was the narrow question. And so why is that a good question? Because we have a whole awesome set of tools now to solve that kind of a problem. Exactly. How do we do it? Take a lot of past data of past jobs, past applicants, past hires that were made.

31:08And then we ask, you know, these crazy big black box algorithms, all right, do the best job you can predicting who's going to get hired on this job with these applicants. And we use that data to test how well these applicants these algorithms are doing. That's like machine learning in 30 seconds, basically. So, you know, we're working on this problem. Great. And then I kind of poke to my head up a little about, like, why are we working on it? What's what is this going to do? Well, it turns out the reason these kinds of things are important is they get used to make decisions. So what kind of decision do you make with that?

31:37Well, one thing you do is you say, oh, well, if I could predict who's most likely to be hired, then I should just rank people based on that. And that would be a good matching algorithm, right? That would be a good way to sort and triage people applicants for employers when they're like screening, trying to figure out who to interview who to hire. Great. Sounds pretty natural, right? And then, you know, you think about it a little bit. And this, this to me is really it's such a such a passion project to get people to understand that this is why the humans in the loop that help us in businesses and making sense of data are so critical is the following problem.

32:15If you think about it a little bit, you realize, you know, what that algorithm is doing, it's really just picking up on patterns and past data, right? So yeah, that's great. This person is likely to be hired. But what we really want is something different. We're trying to add value by ranking people. So, you know, to give another example that's similar to this, when you're a marketing manager, and you've got, you know, a crack data science team that's built a long -term value lifetime value model for you, you're not going to get in trouble with anyone if you send your highest value promotions to the highest LTV customers, right?

32:49Who's going to, who's going to blame you for that? Because you're like, oh, yeah, you know, this person's worth a lot. And I set them this promotion, you know, say that in your monthly report, nobody's going to give you a hard time. But the problem with that way of thinking is actually predicting what their lifetime value is isn't really the question. The question is, how much more are they going to spend on my platform because I sent them that promotion? That's a very different thing. It's a differential rather than an absolute, I'm not interested in their absolute LTV. I mean, I'm interested in the difference in their LTV because I sent them this promotion.

33:21And when you look at it that way, what you realize can happen is picking up on patterns because of good predictions, right? Finding the people that have high LTV because you predicted that is very different than making good decisions, which is about saying that difference in their LTV is going to be higher because I sent them this promotion. I love this example because I taught a class here at Stanford. It was like an executive education class. We had, you know, all the executives for company in the room. And one of the people in the room was the chief marketing officer and I just asked this question like, hey, okay, let's see.

33:51You got this great LTV model. Who would you send the promotions to? It's like definitely the highest LTV people. And there's a CMO in the room. And so, you know, it's like, it's a, it's a little bit of a delicate situation like pushing back a little bit, right? I do want to be clear, there's reputational reasons you might do that anyway. I mean, I'm not trying to get away from that. But just to make the narrow point that predicting is about picking up patterns, but making decisions that is about thinking about these differences. Now, why is that important? Because we learn in high school, correlation is not causation.

34:21That's a phrase everybody has heard all over the place. What does it have to do with this? Well, when we teach people to build machine learning models, we're asking to make predictions. We're asking them to find correlations. Prediction is inherently about correlation. But when we ask people to make decisions, we're asking them to think about causation. If I make this decision, then will I actually increase the net value of my business, right? Well, I have, by sending the promotion, increase the likelihood that this person is going to spend more on my platform. And so, the first and most important thing that I feel very strongly about in what would I get a data scientist to do is no matter who they are, even if it was that person in the weeds thinking about building this prediction model for hiring, get them to be thinking in the back of their mind always that their goal is to help the business make decisions.

35:10And that distinction between causation and correlation matters a lot. We could talk a lot more about how does that play out in terms of their day to day work? But at least at a starting point, you have to recognize that the first step is always recognition that prediction isn't the same thing as making decisions. So the takeaway here is help as a data team and as a data scientist on the team is help the business make predictions. Are there a couple more examples you could share just like what is an example of a decision that you think they often should be making and using data to help them with?

35:41Maybe the right way, the right frame of reference for this and the word that an academic would use is causal inference. So what we're changing from is machine learning to causal inference. So let's think that through and a couple of different use cases that are related to that marketplace data science fly wheel I talked about earlier, finding matches, making matches, and then learning about matches. So finding matches, like you said, a core part of that is search and recommendation. And each of those relies on rankings. So I want to be able to rank order. Let's say I go do a search on Airbnb on a rank order of the different listings in the marketplace.

36:16At some level it's true that what I'm trying to do there is I'm trying to just predict what are you going to like the most. But I think there's an important piece of that also, which is that I want to think a little bit about the distinction between two different ranking algorithms. That's the real decision that's being made. And when I think about the distinction about two different between two different ranking algorithms, I don't want to be only comparing them in terms of how well they recreate the choices people made in the past. The way I'm really going to evaluate those is in my market, does one of those lead to better matches or more matches than the other one, right?

36:54So Airbnb as a business, like what are the most obvious core metrics? It's bookings and revenue. So you're going to want to ask a very basic question. If I use the ranking algorithm Lenny just developed last night versus the ranking algorithm mesh developed last week, does Lenny's ranking algorithm lead to more bookings than remesh's ranking algorithm? And it's so important to put it that way, starkly, because that's so different a question than does Lenny's ranking algorithm do a better job of predicting over the last two years what bookings people made than remesh's ranking algorithm? Okay, so that's I think like at that level.

37:24Then we talked a little bit about ranking at the point of making a match. And I think that's where this kind of hiring issue popped up, right? Because in the end, while we might have these predictive algorithms to rank who you're going to hire, that's not the important question. Interestingly, the important question is actually to evaluate the quality of the match that's made. And we would do that through the next step of that flywheel, we'd ask ourselves, you know, what ratings did they get back to that freelancer? Do they hire that freelancer again? So you're comparing two different algorithms not through their ability to recreate the past, but their ability to make matches in the future that can be objectively evaluated to say, hey, I increased the value of the business.

38:03I actually made better matches this way. You know, then rating systems, I think we could talk quite a bit about kind of a similar, a similar phenomenon there too. This episode is brought to you by Epo. Epo is a next generation A B testing and feature management platform built by alums of Airbeing, Be and Snowflake for modern growth teams, companies like Twitch, Miro, ClickUp, and DraftKings, rely on Epo to power their experiments. Experimentation is increasingly essential for driving growth and for understanding the performance of new features. And Epo helps you increase experimentation velocity while unlocking rigorous deep analysis in a way that no other commercial tool does.

38:42When I was at Airbeing Be, one of the things that I left most was our experimentation platform where I could set up experiments easily, troubleshoot issues, and analyze performance Epo does all that and more with advanced statistical methods that can help you shape weeks off experiment time and accessible UI for diving deeper into performance and out of the box reporting that helps you avoid annoying, prolonged, analytic cycles. Epo also makes it easy for you to share experiment insight through their team, sparking new ideas for the A B testing flywheel. Epo powers experimentation across every use case, including product, growth, machine learning, monetization, and email marketing.

39:19Check out Epo at getepo .com slash Lenny and 10X your experiment velocity. Let's get eppo .com slash Lenny. Yeah, I'd actually love to talk about reading systems, but there's kind of an implication in everything you're describing of running an experiment versus looking at what would have happened in the previous world. It's you made a change run experiment. See if it actually makes an impact on bookings and revenue. And that leads me to a question I wanted to ask, which is with experiments, there's kind of this classic challenge and always elephant in the room of if you just run a bunch of experiments, you're kind of going to micro optimize lead to these local minima, local maxima.

39:59And you may miss big opportunities and big unlocks if you're just like extremely experiment driven. You spend a lot of time thinking about experimentation. What if you learned or what advice do you have for people to either be less worried about micro optimizing and missing something big or just finding a balance with running experiments, but also creating opportunity to find a huge new opportunity. Yeah, first of all, I'm really glad you broached the e -word. I was dancing around it and I'm really glad that we talked about experiments because yeah, that's one of the big lessons of this recent conversation we've been having is just how could you possibly know that difference without doing something like experimenting.

40:38Okay, so yeah, I'm a big believer in experiments. I mean, I'll just lay those cards on the table. I love working with businesses that think experiments are important to helping make good decisions. Now, all that said, I am also someone who feels pretty strongly about this exact issue that you're raising, which is you can't experiment your way out of everything. And one way, you know, that one frame I like to give people is that although you might say you're an experiment driven business, I you know, some businesses will proclaim we literally test everything, but that kind of leaves aside a little bit is there's a lot of degrees of freedom in what it means to test everything because ultimately what's getting built and tested are choices that are made through the organizational structure, the data scientists, the PMs, the engineers, everybody's on the, you know, before we're running experiments, we're actually thinking about even what's experimenting, right?

41:31Like what designs are we coming in? So that's one. And the other big one is how long to be run these experiments. Okay, that's a big choice. And what I generally believe, and I think there's a paper I'll we can we can link to later that I'll point your readers to as well that not my paper from some folks at Microsoft, but I generally believe is that we're risk of worse on both these two dimensions that what people decide to test in a world that has promoted experimentation for everything tends to be more incremental by design. Okay, because and we'll come back to buy actually answer the because in a second.

42:07So that's one and two is people tend to run experiments for a long time. It probably longer than they should. Okay, now what do I mean by these two things? So what's interesting to me about this dynamic is experiments don't live in a vacuum, companies have incentives. And in companies that really go all in on experimentation, one of the things that gets wrapped up in that is the incentives around experiments because if you go all in on experiments, a common thing you'll see is data scientists get judged based on how many wins they had that quarter. And right, how do you get more wins? Well, it's easier to get wins when you're being incremental.

42:47And because it's important to have wins, you have to run them long enough to demonstrate that they're really wins. Right? You're less willing to cut something off in exchange for trying something riskier. So the big less of this Microsoft paper, it's called AB testing with what's called fat tails, which in lay terms just means you're running a business where there's potentially big opportunities out there if you look at kind of the effects of the experiments that you run. But there's a couple of lessons there about both trying a lot more stuff that's not all risk averse and not necessarily running everything for so long.

43:18So really getting velocity up. So you can see that there's a big incentive problem there, right? Because the culture that says it's okay to fail big actually requires changing the terminology of wins. This is one of the things I hate most in AB testing. I have to say I get where it comes from. You know, experimentation was never historically in science about winners and losers. It'd be weird if Ronald Fisher, who's kind of the father of experimentation with his agriculture experiments talked about winners. I experimentation is always very hypothesis -driven. It's about what are you learning? And that's really an important distinction because what it means is if I go with something big risky and it quote unquote fails, meaning that doesn't win.

44:03I never the less if I was being rigorous about what hypotheses that's testing about my business, I'm potentially learning a lot, right? So a great example of this kind of thing is, you know, that there's an important feature of marketplaces is badging, right? So sometimes it's really important to have badges on your kind of top -rated profiles or whatever when people are searching. And without going too far into the details, a common kind of finding it, you know, with badges is that badges you think are going to be great actually turn out to be terrible. And one reason they're terrible is they focus too much attention on the badged folks and pull too much attention away from the unbadged folks, right?

44:44And if we judge at only terms of winners and losers, you throw the baby out with the bathwater, you're like, oh, that badging, I think it was terrible. So you'll ditch that, you know, no badges. But that's not what it's telling you. It's teaching you something about how inventory is being reallocated, how attention is being redirected through the badges. And you really want to think not in terms of winning and losing, but learning. So learning is a win. And I feel that that's a cultural thing fundamentally. It's very hard to somehow attach dollars and cents at the top to data scientists running experiments that fail but learn.

45:17And ultimately, I think getting into that space where you experiment more, meaning you don't run all your experiments for quite as long. And you accept the willingness to try experiments that are into the tails where you might fail bigger is a cultural thing. It's about saying that, you know, we're allowing that to be part of our social contract with our data scientists or actually our employee contract with our data scientists that not everything is just about how many launches you had and how many wins there were. It's okay to say that's how I want to use experimentation. But if you're going to use it that way, then I would say don't be we experiment everything business.

45:52Because then I think you need some other way to deal with these big you know, changes that teach the whole company a lot, but maybe can't fall into the incentives you've created for your data scientists. This badging example is I don't know if you're referring to the Airbnb example, but actually led the launch of Superhost at Airbnb, which is like the ultimate badge on Airbnb. And there was a lot of concern from the data team that it would destroy the marketplace because they built as you describe this very well crafted ranking algorithm with just like a prediction of, you know, exactly as you described, which listings that guest is most likely to book and be successful booking.

46:32And then we're about to throw a badge on random listings in the results. And so this one data site just our teams like, no, we can't do this. This is insane. We're going to destroy it all. And we still went ahead with it. We ran an experiment showing the badge to some people and some not actually was no, no impact at all, which is like Superhost itself had no impact at all on the business as far as I we could tell initially, which is also bittersweet because it felt like with like, why did we even work on this thing? There was like a slight benefit where hosts felt better. They felt more satisfied with being a host, but I went exactly through which you described.

47:09So that's pretty funny. Without necessarily like going into the weeds on like the data science of Superhost, I think there's a lot wrapped up in what you said. I guess another thing I'll say is that I'm a big believer that you don't, you don't throw your understanding of the business out the window when you process experiment results. And it's partly, partly I guess what I mean by this is data science is really about accumulation of evidence. It's never about one finding an isolation. And so another kind of trap I think is to sometimes say, well, I hit Statsig on my A .B. Test, green light, it's all go like and you know, I think you had Ronnie Kohavi on your show and he made a similar point that there are different levels of evidence.

47:55And it's just having an outlier A .B. test that goes against everything you believe about your business doesn't mean that you somehow have contributed all your knowledge. And I think that's one side of the other thing is you can't always measure everything that's important that's needed to really develop like a full sense. So with Superhost, right? One of the things that's hard to measure is the long -term impact of Superhost. Because I mean the short run Superhost causes a rebalancing inventory. There's going to be winners and losers. Part of Superhost is actually about retaining hosts that get the badge over a longer period of time.

48:31Recognizing that hypothesis actually says something about maybe how long the experiment needs to be run or what kinds of data analyses need to be done. And in the end, if you can't do that, you can't run it long enough or you can't do that data analysis due to sparsity of data or lack of data to address the question, it matters what you bring to the table, right? What are your beliefs about that? So what I like to tell people to do there is I like to push people to be what's called quantified rather than data, which is okay, fine, some things we can't measure, right? But maybe you've got a leadership team with different beliefs about what they think the retention value of Superhost is going to be and they might be all over the place.

49:10You can process your experiment results in the context of these competing beliefs, it's almost like a prediction market kind of thing. And start asking, okay, like if this is what we believe about our business, this is what the data is telling us out of the experiment. Let's put those two together and ask, is this enough for us to make the bet that we're still going to go with it, even though maybe that short -term test you ran was flat. That's actually exactly how I think of Superhost looking back. It was a great idea. I'm really happy. I can't even imagine it being viewed without that, even though there's no evidence at least initially that it made any impact.

49:43I'm guessing they looked at it again and maybe there's something that came out of it. But even if it had no impact, it just feels like it made the marketplace better. That was a big learning for me. It doesn't need to always drive a metric that you can measure. This is the way it should work. One of the reasons the thing you said happens is because a little bit like a game of whack -a -mull. What I mean by that is, so narrowly in the context of Superhost, because you're redirecting attention to some hosts at the expense of it, it's not even obvious if bookings can really go up. Maybe you get lucky and maybe you get a bunch of bookings.

50:19One reason you probably wouldn't expect that in the first place is there's only a limited number of Superhosts. How many more bookings are they going to be absorbing because of all the extra attention? And you're taking attention away from other people without doing any data analysis. My prior would have been the bookings should probably go down. What example that I came across with a lot of the companies that worked with us that I love is we were working together over a period of time. In a month, we looked at some of the data and it suggested that our new supply side was having a pretty bad experience.

50:53We got to do something about this. What we decided to do is build some custom bespoke features that were really going to direct them to more experienced folks on the other side of the market. Good. Then, yeah, blow and behold, pretty soon those metrics start to look better, but then we're looking at it. We're like, wait a second. No, the existing folks on the other side are having a worse experience. You kind of whip lash around. You're like, oh, wait a second. We better do something about that. We take them. We try to match them up with the more experienced folks. Now, suddenly, month after that, you're like, wait a second.

51:23Your metrics just keep moving around. That's because the black and wall game here is ultimately a lot of marketplace management is moving attention and inventory around. Sometimes you get lucky and you really expand the pie for everybody. But I think Serba Salen, who was a CFO at Upwork that I got to know there and then went to Thumbtack later, he had this line when he came to visit our class that I love, which is you have to recognize when you run marketplaces that many of the changes that are most consequential create winners and losers. And rolling with those changes is about recognizing whether the winners you've created are more important to your business view than the losers you've created in the process.

52:03And it's a hard reality because nobody likes to articulate the idea that a feature change is hurting some of the people in your marketplace. But because of this fundamental constraint baked into how marketplaces work, many of the things that we would choose to do and the reallocation they create can't necessarily create observed pie expanding wins in the short run. You're often making bets that that's where you're headed partly through the reallocation that you're having right now. And so I think that's what's interesting about Superhost and me is it partly points to thinking about what's the objective you would have defined the metric you would have defined in the short run that captures this idea of a trade off.

52:42That's a great way to think about it. I wanted to come back to this idea you're sharing of maybe you should run experiments more quickly, not wait for stats, sig, have a culture of learning versus impact in practice. It's very difficult because people are measured by impact. There's performance reviews, there's promotions, there's how much impact at the steam drive. We're going to look at their experiments. You've worked a lot of marketplace companies, a lot of different companies. Is there anything you've seen about just like how to something you could do to help the company shift and actually work this way while also recognizing success and who's doing great who's not, which team's driving impact who's not.

53:22Interestingly, it's actually sort of an active area of research for me now. What I mean by active area of research is I care a lot about the incentives that we create for data science through how we set up reward mechanisms. There's a couple things I think that could be helpful that are maybe there may be a little bit less about like maybe I'm not going to directly answer the question you ask because I think that's a hard one. I think I recognize that measurement on impact is critical. Well, let me answer that actually for the most obvious way first. I think there's a cultural issue that's really critical.

53:56One of the things I often find is that my PhD students, our PhD students here often go off and get great data scientist jobs. In one sense, they're doing amazing stuff. They apply really technically sophisticated methods. When I look at the problems they're working on, they're often more at the margins of the business than they should be. It's a cultural thing. It's basically because if you're measured narrowly on impact and that's all anyone sees around you, then it's very hard to engage with the creative aspect of business change and the strategic aspects of business change. The cultural aspect there is I think it's partly incumbent on the leaders to expect something more of their data scientists.

54:35What I mean by expect more is that you expect them to do more than deliver narrowly defined statistically rigorous results to you in their reports. You're actually expecting them to talk also about what they're learning about the business in the process. Where that's headed is this concept of being hypothesis driven, which is like the technical phrase. What does that mean? Again, in a more lay sense, what it means is tests aren't going to be defined only in terms of winners and losers that each test should also say something about what will we learn about a business flow, a funnel, preferences of the guests, preferences of the hosts, what will we learn about their demand elasticity if we're changing prices around.

55:17These kinds of things. It's possible to articulate in an experiment doc, a launch doc, what are the hypotheses that are being tested? That's one thing I would say. It's just culturally setting the norms that learning is part of the discourse. It's expected actually, I think, is important. The other thing I would say that's maybe a little bit more about programmatically. What could a team, a data science platform team do? A funny thing about experiments is that we throw paths learning away effectively. This is just an artifact of how we analyze experiments. The methods use the statistical methods used typically.

55:54P -value, use confidence intervals. These fall in a branch of statistics known as frequentist statistics. The idea behind frequentist statistics, without being overly technical, is just, I let the data speak for itself. There's no beliefs brought to the table about where that data came from. But if you think about this in a company and a A -B testing and coming, it's a weird thing. I might have run a thousand A -B tests in the past on this exact same button or call to action or call there. Now, I'm going to completely ignore that and focus only on this. There's ways to take the past into account to build what's called a prior belief before I run an experiment.

56:31Now, take the data from the experiment, connect it with the prior to come up with a conclusion of like, okay, light of the past, plus this experiment, what's it telling me about the future? And that falls broadly under the category of what's called Bayesian A -B testing. So that's one of the things I think can help culturally, weirdly. It's like a super technical thing. But I think it can help culturally because what it's doing is it's now rewarding people for contributing information to that prior. I think it then becomes possible to say, oh, your experiment that failed actually moved our prior.

57:06And that's an important thing because by doing so, you're now altering how we're going to think about this flow or this pricing plan in all future experiments. So there's like an information positive externality, positive network effect that's generated for the rest of your business. If I can somehow encode what you learned into those analysis of future experiments. So this is one thing I, there's a strong connection between the culture and incentives of A -B testing and the ability to actually incorporate past learning into these prior beliefs. I love that you're doing research in this area. We should bring you back when you've completed it and have the ultimate answer for everyone to change how they operate.

57:42Yeah, one of the great things about professors is we never complete anything and never have ultimate answers. So, oh boy. Yeah, I'll do my best stuff. This touches on a really interesting concept that you share with me around how learning just learning isn't free. People think that they could just learn a bunch of stuff and there's not a cost to it. I'd love for you to just chat a bit about what that means. Let me start with an anecdote that I just absolutely love this anecdote. I use it every year in class. So yeah, I was talking to a real estate platform and they have a they had a marketing data science manager who's basically responsible as many marketing managers are for allocation of ads spend across different channels.

58:22And what they discovered had happened at the end of the year is on one hand, the team had done great. But the manager had held out some subset of arriving visitors not shown them any of the innovations they were making. Like a hold -out group. Yeah, that's exactly what's called a hold -out group in experimentation. And the thing we're thinking about this hold -out is, was authorized. That's not the way things are supposed to work. They've got their ads spend, allocate out your ads spend. Great. So at the end of the year, you know, they look to the hold -out. They're like, wow, that cost us like a couple of million dollars is something in that range.

58:58And it's like not a trivial amount of money. Like what's the deal with what we're thinking basically? And of course the answer was, well, I get that I cost you that much. But number one, not you know what my team's worth. And number two, you would never have had that answer unless I'd done that on my own. Right? Now, why is that so powerful? I think what's what I find so interesting about experiments is that when you don't know something, it seems not even a question that you would allocate some of your samples to all options. Right? Treatment and control. I got two different ways of doing something.

59:34I don't know which one's better. So of course, I'll give some samples to each of them. After the fact you're like, oh, treatment was better. What the heck were we thinking? Why'd we give all those samples to control? Right? That doesn't make any sense now. There's this great Seinfeld clip where you know, they get the bill at the, you mentioned getting a bill at the end of a like large luxurious meal and people stare at the bill. We're not hungry now. Why'd we order all this food? Right? So it's the same thing. Like, I mean, you know treatments better now. Why'd you waste all those samples on control?

59:59And I think that is such a powerful observation that you have to put yourself in the frame of reference if when you didn't have the answer. And at that moment, what you're essentially saying to yourself is that it's worth paying to learn the answer. I think it sounds obvious the way we're saying it now, or this anecdote of the marketing manager and the holdout sounds obvious, right? What's culturally not baked in, I think, is that idea and the reason I say it's not culturally baked in, by the way, is because of the language of winners and losers. Because if we use that language or implicitly saying is that we wasted time when we ran an AB test on loser, if I reward you for shipping winners, then what I'm really telling you is all the time that you spent testing out failures was wasted time.

1:00:46And I think, you know, of course, yeah, like you don't want to keep data scientists around who regularly are just generating failures. That's not my point. But my point is there's a disconnect there. On one hand, we can all look at the story of this marketing manager and chuckle at it, right? And yet every day we're instantiating language and processes that are reinforcing that same theme, which is essentially trying to say to you, if you're wasting samples on things that don't ultimately end up being a winner, then that is the act of doing so as a failure. So I really feel, you know, that that idea that you have to pay to learn is, again, it's a cultural thing, but it's also an education issue for, you know, businesses are populated by people of all stripes, not everybody comes from a data science or experimentation background.

1:01:31And this idea that learning is costly is not natural actually. It's not natural. You sound as a matter of human nature. It's certainly not natural as a matter of running a business. I love that example of the real state platform where it's like a very viscerally clearly like cost. There's like, they lost because they didn't roll out experiments to those groups for a long time. Yeah, example of this idea in action. You mentioned star ratings. And you spend a lot of time on designing rating systems. Sorry, I didn't mean to imply star ratings. That's just one implementation rating systems in general.

1:02:05So maybe just to keep it focused, say a marketplace founder is trying to decide and design how they do ratings and reviews and things like that. What's a couple pieces of advice you'd give them for how to do this correctly? And is there a model like a model marketplace? You'd point them to like, these guys really do it really well. And I know it's like super specific based on marketplace. But is there one just like, oh, they really nailed it? Oh, man, that's a tough one. I think I'll answer the second part first. I don't feel like anyone's really nailed this. Yeah, I think there's a lot of innovation that's happened.

1:02:38But I think fundamentally we're still playing with the same kind of tools that we had when, you know, eBay and Amazon like first started thinking about how to do rating systems ages ago. And part of the reason we haven't nailed it is because there's a lot of dynamics and play that lead to what's called rating inflation. Where if you look at ratings over time in a marketplace, one of my colleagues John Horton was a professor at MIT and, you know, has worked very closely with Alperic. We worked together when I was at ODSK. He was the staff economist there. He's written a couple of really nice papers with the empirical phenomenon that over time you see the median rating and flading, let's say on marketplaces like ODSK, like we've really got any of these, right?

1:03:17And there's a lot of reasons for this. But one of them is just that there's a reciprocity issue, right? Which is it's effectively, you know, from your perspective, it's kind of costless. If someone says to you, hey, like please leave me a nice rating, right? And you know, if you're seeing them or you're interacting with them, you know, most people don't want to be mean. So that happens. But there's another aspect of it which is norming. As the ratings in the marketplace go up, they get normed, right? So that now you're in the condition you're like, oh, a four -star rating, I'm really screwing this person over.

1:03:45Whereas maybe when the marketplace started, you didn't think that. So definitely one thing that we worked on in our research was to think about renorming the meaning of some of these labels and renorming could mean something like rather than, you know, the star ratings just being, you know, forward to excellent. The top rating is actually exceeded expectations, right? You could go one step further and you could say, how did this compare to this experience you had in the past that you rated really highly? And Airbnb had something like this in place where they would actually ask you to compare or, you know, or ask you questions about expectations.

1:04:19I find that that's really valuable because it's easier for people to say that was good but didn't exceed my expectations. I mean, that was good, but definitely not better than this amazing stay I had like two months ago. Then it is to say, well, you know, I'm going to ding this person and give him four stars. So that's one issue. And I think another thing I want to point out for any marketplace founder is that something you want to be really careful about is the concept of averaging and whether the implications of averaging. And that's because a default for many marketplaces is to just average the ratings that people get.

1:04:53It feels very natural, right? Like Lenny's got five ratings, let me average it. And that actually has some pretty important distributional consequences for the marketplace, distributional in the sense of who wins, who loses. And that's because if you're averaging and you're really established on a platform, take a restaurant on Yelp with 10 ,000 reviews, it's irrelevant with the next review is doesn't matter. Nothing's moving at that point. If you're new and you break into that market and your first review is negative, you might be completely screwed. In fact, there's some early work on eBay that show that your first, if your first ratings negative, that could actually immediately cause like an 8 % hit on, you know, your immediate expect review saying nothing of long -term consequences.

1:05:34Subsequent work has found that that's a significant indicator of potential exit from the platform just because now it's very hard to find work. And, you know, some platforms do things like maybe they won't show your ratings until you've accumulated a few. But in the end, this kind of distributional fairness aspect of averaging is pretty significant. And one of the recent papers that we've written is trying to get platforms to think a little bit about that, there's ways to address that interestingly through the same concept of a prior. And the prior basically says, hey, if someone comes into the marketplace and instead of averaging them, I average them together with a prior belief, then maybe what that prior belief does, it says, yeah, you got one negative rating, but maybe you got a little bit unlucky.

1:06:15And maybe my prior belief is something which actually pulls your rating up a little bit and allows me to still have you alongside others in the marketplace that give you a chance at getting work, you know, getting rides, et cetera. So I believe pretty strongly in this kind of like distributional fairness element of designing rating systems. I think it's been understudied. And, you know, I'll say in general, actually, I think rating systems are understudied, which to me is astonishing because the biggest change from those agoras and trades in market elements of, you know, those kinds of markets to me, the biggest change is that we get to see what happened with our matches.

1:06:51So if, you know, as a data scientist working on marketplaces, I feel like it's, it's incredible that more of us don't spend our time thinking about what we're learning from the matches and what these rating systems are telling us and what the impact of that is on who wins and who loses in these markets on, you know, it kind of think about like the social implications of these things. So that's something I'm pretty passionate about. I also led the review system flows for a while at Airbnb. And one of the things I'm most proud of is launching what we call double blind reviews where you don't see the other person's review until you leave your review.

1:07:27And yet we, the intention was to create more honesty and more accurate reviews. It turned out they had biggest impact. It was review rate went up because people get this email or a mesh left your review. If you want to see it, you should leave a review. And that really increased review rate, which gave us more data. And it was a really fun experiment workout. There's a great concept in the ratings in the literature on rating systems called the sound of silence, which is this idea that that there's a lot of information in in ratings that are not left. So Steve Tadella, who's a professor at Berkeley, he had a really nice paper with some folks at eBay talking about what they called effective percent positive, where rather than normalizing just by the ratings, they normalized by including ratings that weren't left.

1:08:10And what you found was this was much more predictive of kind of downstream downstream kind of performance of a seller. So there's a lot of information in that in that lack of a response. It's cool that you're able to get more of that out. So much easier just not leave a review than leave a bad review. Right? Like the downside to use just much better. Oh man, marketplaces are so fascinating. I could see why founder would want to be a marketplace founder because it's just like such an interesting space and hearing your feedback of like, no, you're not a marketplace founder. Let's think about the problem yourself.

1:08:43And it might be a marketplace might change people's minds. Also, I feel like there's like a podcast episode in every topic we touched on. I know we just kind of scratch the surface a lot of things. And now you got to run before we get to our lightning round. Is there anything else you wanted to highlight, touch on, leave people with better maybe working at marketplaces, thinking about a marketplace? I think one of the high level points I would make and like you said, there's an entire podcast in this topic is that I think people want to imagine LMs and AI driven data science, automating out large parts of what it means to do data science in industry.

1:09:19And I think that's probably the wrong perspective. In some like mundane sense that's true. It's easier for me to code than it used to be before. It's easier for me to develop visualizations that used to be. I can make dashboards faster. So like programmatically, I think it's true in some basic sense. But you know, what I believe pretty strongly and I teach data science here and my students are asked to use LMs and generative AI on a weekly basis on all their assignments. So I've got like an up close and personal beat on this. But I believe very strongly actually is what AI has done for us is it's massively expanded the frontier of things we could think about our problem.

1:09:56Hypothesis we could have. Maybe things we could test. It's just an astronomical explosion of explanations and ideas and principles. And I really think actually what that does is puts more pressure on the human, not less. I think it becomes more important for humans to be in the loop and interacting with these tools to drive the funneling down process of identifying what matters at all levels that ranges from you're carrying out a data scientific analysis. And now because you've got these tools, you can hypothesize 10 explanations, maybe 100 explanations, which of those are you going to focus attention on?

1:10:34What are you going to tell other people to focus their attention on? To your running experiments, you used to have 10 creatives you're testing for a marketing campaign. Now you've got 1000 creatives you're testing for that marketing campaign. Maybe that completely changes the game of what it means to run the experiment. What are you actually looking for now? How do you evaluate that you found something that was good enough? And I think these questions are not getting enough attention. I think people are looking for the automated tool that really cuts the human out. But what I've seen so far, and again, who knows, by 2024 I might have a totally different answer for you.

1:11:06I don't think so. But at the moment, what I see is that humans have actually become far more important to the productive data science loop, not far less. Such an important point. I feel like we need to add AI corner to this podcast where we always think about how does AI impact what we're talking about on this? Yeah, I can see that. I can totally see that. Okay, we might start doing that. We're mesh with that. We've reached a very exciting lightning round. I've got six questions for you. Let's try to knock through them so you can go teach your class. Are you ready? I am ready. All right, what are two or three books you recommended most to other people?

1:11:40When it comes to books, I have one I love that I start with always, which is how to lie with statistics. It's a tiny book for I derlhoff from 1954, which is just for anyone that likes data at any level, it's like such a fun read. It's like it's a great book. The second thing I recommend to people, and actually this is true even for people who are not, you know, not expert, is David Friedman was a statistician at Berkeley who passed away in the 2000s, early 2000s and his writing was fantastic in getting us to think hard about process. What, what, what, you know, he was especially fond of what he called shoe leather statistics where you really got your, you know, you rolled your sleeves up, you got on the ground, boots on the ground, really getting in there, really trying to understand your data.

1:12:27His writing is fantastic. His explanations are fantastic. He has a few different books at different levels. I think people love reading. Most importantly, what I like about it is he puts such emphasis on driving evidence and understanding of your processes that generate data. And, you know, I find often data scientists don't even look at examples, right? So like at Edoin S. Kimment, are you looking at actual jobs and what's actually going on in your product before you're trying to do data sites on it? So I think that's like a Friedman, a Friedman insight, Friedman mantra. And so his writing is great.

1:12:58The last one I was going to mention is nothing to a data science or anything. It's called 4 ,000 weeks by Oliver Berkman. I'm not like a huge, self -help type person, but I really like this book a lot. It's a little bit, I think it's a little bit stoic and it's approach like it's so philosophy, but it's the basic point is you're only on Earth from somewhere in the neighborhood of 4 ,000 weeks. And, you know, my wife and I have this term we called infinite queue, which is like no matter what you think you get done on a given day, more stuff is going to just keep coming in. And he basically says that recognizing that is liberating, because once you recognize it doesn't matter what you do, you're always going to have too much to do.

1:13:33There's no point in stressing out about having too much to do. And just that like small shift of mindset then puts a lot more attention on, you know, the usual thing people worry about, which is where do I want to prioritize my time. So he has a great way of writing about it, some concrete rules of thumb to help manage, you know, that way of thinking. And yeah, I think it's a great book. What is a favorite recent movie or TV show? I am a climber and one movie that I really liked was the Alphanist. I know a lot of people have seen presolo, but for anyone that kind of likes that genre, I would recommend they watch the Alphanist.

1:14:06I think climbing is interesting for it because it has a very much like a psychological aspect of it. And I think that movie is pretty good at this sort of metal level where you kind of reflect a little bit on what does it mean to make a movie about people who are obviously putting themselves into such risky situations. So I really enjoyed that. On TV, we've been watching only murders in the building, but I'm like, enough episodes behind right now that I probably won't say anything more because I'm trying to avoid any spoilers and I'm sure that people out there are trying to do the same. So great show though on Hulu.

1:14:39What's the favorite interview question that you like to ask candidates that you're hiring? I interview people probably that are a little bit different than like most of your podcast listeners. But that's there's one question I like to ask a lot. And that's if you imagine, so often in our interviews in academia, whether it's grad students or faculty, we'll ask people about their plans. And what I like to ask people is, okay, now imagine everything works out. All the challenges you're facing work out, all your plans work out. Everything hits the top end of your vision for what this could be. What do you imagine is the impact of having done that?

1:15:15Like who's being impacted by that? Why is that a big deal that that happened? And I find that's a really valuable question to ask because first of all, many people haven't thought about that. We're so short -term folks, we don't even think like boy, if everything worked out, what would be the big deal because of what I did? Startup founders tend to be better at this than most people obviously. But another reason I like it is because you'll find in that conversation that is their vision expands a little bit of additional spheres that are touched or impacted by what they're thinking about doing.

1:15:47So on both sides, it's kind of a revealing question, I think. So I find it important for my line of work, but my hunch is that might be useful for some of your listeners too. Yeah, such a unique perspective on interviewing versus at most of the guests that I interview in tech companies. Yeah, normally there's a coding question. I should say, I would never ask a coding question post November 2022 after we got AI to help us code. I think it's a superpower. AI corner. What is a favorite product you recently discovered that you really like? I also really like cycling and I'm not ashamed to admit that I think that e -bikes are the greatest thing for cycling.

1:16:29Admittedly, I'm late 40s, so maybe I'm the right target demographic too, but yeah, I love my e -road bike. It's great because it's not one of those with a throttle. You have to work, but it kicks in just when you're on your sixth hill and you don't want to go up the last hill anymore on the way home. So that's amazing. I think that's just transformative for people that like cycling but have busy lives. And I think another one that my son who's 10 wrote to me into actually is we were like in Santa Cruz browsing at a kitchen where shoppable places. And he saw an outdoor pizza oven like a tiny portable one.

1:17:03And he just did research for like two weeks and insisted we get one. So we got over the summer and after we got it, he refused to eat pizza out anymore. It's a 10 year old. So you know, that's like a maybe that's the best best thing I could say about like the quality of pizza you can get from a home outdoor portable pizza oven. Oh my god, I'm hungry. I am going to go. You have to get some pizza now. What is a favorite life motto that you like to repeat to yourself share with folks find useful in your day to day? A lot of my work involves talking to students of all stripes. And I guess these students go on to be data scientists go on to be, you know, founders and a lot of them go in the tech industry.

1:17:41So maybe in that sense, the advice is relevant. My main thing I tell people is slow down. Because I think what I've found has been happening is just we're so convinced that speed is the way you're going to find the right answer that I just don't think we slow down to develop meaningful mental models of the things we're doing. That's certainly true in the research projects I work on. It's consistently true. And you know, when I talk to people in business and I ask them about their mental, by mental model, I just mean, you know, if you're running a marketplace, what is your model of what people care about?

1:18:18What makes people stay versus believe? You know, what makes matches work versus not work? All those things shape a roadmap, right, in your mind. And I think a lot of roadmap being a lot of execution, paper writing and academia has all just become far more fast paced at the expense of kind of deeper thinking about these kind of structural features of the thing you're building. And so, yeah, with my students, but also, I think with people I interact with in industry, I think slowing down is actually more of a virtue that it's going credit for. Very similar to a model that a recent guest shared, which I think was go slow to go fast or go stay smooth to go fast.

1:18:59Yeah, I like that. Maybe I'll maybe I'll kill for that when I get to my graduate school. Yeah, final question. Your professor at Stanford University, which sounds incredibly cool. What's something about being a professor at Stanford in particular or in general, that would surprise people. Either good or bad. Yeah, I mean, we've had a rough ride as everybody probably knows. There's Stanford's been in the news for a lot of, you know, not so great reasons. I think over the last last five years, especially. So, you know, I don't know if this is the right kind of surprise, but I think one thing that I find really energizing at Stanford is people have never asked me for credentialing here.

1:19:45And what I mean by that is that I came from kind of a bunch of other good schools and obviously I've spent time in industry with a lot of great companies and a kind of cultural dynamic that can often develop this. Well, before I'm going to talk to you, I want to know something about why you're worth talking to. Like, give me your credentials, right? Like, oh, you know, where are you a grad student or like, where are you a professor? Like, tell me about yourself first. One of the things that I found very surprising when I came here is just how that never happened at any level. Grad students tell me this all time.

1:20:16Go talk to someone across campus and just launch right into a conversation about how your ex meets my why and we have something we could do together. You know, as a faculty member, it happens all the time that I just had a conversation a couple days ago with someone about, like, effectively a marketplace of experiment designs for nanofabrication here, right? Which is like totally out of left field for things I do. And yet seamless, right? Our conversation was about the substance rather than the credentialing. I really think part of the reason for that is that Stanford is sort of unique in that it doesn't have a weakness across the board.

1:20:52We have strong professional schools, law, business medicine, strong engineering school, strong humanities, social sciences, and then that and the weather is what I usually tell people, honestly, which matters a lot. People are willing to walk anywhere. I think those things combine to create a culture and an environment where you don't credential everybody. And I think that means a lot. You know, and I think that's something that I haven't found elsewhere. And you know, if people want to know something about what Stanford's like on the inside, I think that's one one aspect of it that probably isn't discussed very much.

1:21:24And that I, you know, I think that's part of what makes it really fun to be here. It's also an incredibly dreamy campus. That is a very joyful to walk around. That helps, I'm sure, or mesh. I feel like we got people's brains tingling. I think we've created new marketplace founders and also convince people maybe they aren't market place founders. So maybe we ended out at zero new marketplace founders. Two final questions. Working folks find you online if they want to reach out. And how can listeners be useful to you? I think the easiest way, you know, if someone's interested more on the industrial side is probably linked in, you know, you send me a message to connect there.

1:21:56I'm also because I'm an academic, you know, I have my own Stanford web page and it's pretty easy to figure out how to find me there as well. And how can listeners help me? I mean, you know, I, I kind of feel that most important thing that someone listening to this could do is take forward some of the messages that came out in terms of what it means to be data literate. And I think there's a lot you can do to educate yourself there. You know, maybe one final thought I'll share is that, you know, in the same way that AI generates a lot of ideas, AI also generates a lot of pros. And in data science, that can actually be deadly because you're getting more explanations that sometimes maybe are extraneous, you know, and so taking that as a little vignette, I think that what the world needs is data literacy on the part of people interacting with these tools and with each other.

1:22:42So that's the thing I care most about. The things I teach, the things I do research on, they're all connected to that thing. And so yeah, that's where I like, you know, I'm pretty excited. I do work with companies regularly. And so if there's, you know, interesting opportunities that kind of fall in the sphere of stuff we've discussed on the podcast, I always, always happy to listen. Awesome. I think we've made a dent in helping people become a little more data literate. Ramesh, thank you so much for being here. All right. Thank you so much, lady. Bye, everyone. Thank you so much for listening.

1:23:13If you found this valuable, you can subscribe to the show on Apple Podcasts. Spotify or your favorite podcast app. Also, please consider giving us a rating or a leaving review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny's Podcast dot com. See you in the next episode.