The UX research reckoning is here | Judd Antin (Airbnb, Meta)

25:16So any check the box style research is a wild example of user centered performance. I would argue every researcher has probably had to do executive listening sessions, you know, because a lot of PMs, founders, product people, but designers too, they want to get close to the customer, right? And so like, can I do some focus groups? I want to be there. I want to ask them questions. This is 97 % performance. It's well intentioned, but it isn't focused on learning. It isn't going to drive better outcomes or more impact. And then there's all these implicit ways that people engage in that kind of user performance too.

25:53A lot of it comes down to cognitive biases, confirmation bias, ego, like one of my big kind of mantras was we don't validate we falsify, right? We are looking to be wrong. That is the mindset you should use when you're approaching insights and research. I want to be wrong. I want you to do research that shows we were off base in the following ways. Tell me exactly how and why in a way that allows me to fix it quickly. But many PMs, many designers are not in that place. They do not want to be wrong. They're looking to validate. I mean, that's user centered performance. Oh, man. I think a lot of people are heinous and feeling exposed exposed.

26:34I feel like you're like this deep throat person coming from sharing these things. People don't want to talk about. There's this quote in your post. I'm going to read product managers love to ask for middle range research that they can use to justify decisions they're reluctant to make on their own. User designers love to ask for middle range research because it fits their model of a proper design process should look like executives love to ask for middle range because they don't really understand what research is for and helps them do performative user centeredness. In the end, they will decide based on their own opinions.

27:06There is an important place for intuition in product development, of course. Like the best designers, researchers, product people develop strong intuition for the product. But like you got to understand intuition is where all of those biases lie. It's where all your blind spots are. And what great insights people do, what great researchers do when you're next to them all the time is they'll expose you. I don't have to be the deep thrup because you have somebody whose professional job is keeping you honest is probably the wrong way to put it. But somebody whose capabilities are about expanding your horizons, making it so that your intuition is constantly improving.

27:50You don't have to rely on it when your intuition and evidence collide in a way that either affirms or falsifies the product decision you made. Now something really good is happening. And the other thing that isn't here in that is I had Airbnb wore many hats over the years. I was head of research two different times. I was head of design for guest products. And my last job was I was head of the design studio. So UX research, UX design, writing, localization, they all reported to me. So I've seen this from many disciplinary angles in the UX field. And researchers aren't the only ones who are guilty of this.

28:27Like I would say design, it has a ton of performance. And it comes from the fact that we have figured out user center design, this process or design thinking which which audio popularized. Like that's what we're supposed to do, right? Right. Bezos told us that we as PMs had to be customer obsessed, right? So that's what we're supposed to do. It's a really common and damaging thing when we are not genuinely like we don't genuinely have that growth learning mindset. And it's like easy to sideline researchers like we don't need them in that situation. We've got our guts. Isn't the gut where like a great PM, a great founder needs to have that gut and they do.

29:06But they need to be open to the fact that your gut is limited and biased and narrow and wrong sometimes. The two sides of this is trust your gut opinion. I don't need research. I don't need data. I have opinions in my own experience and I'm going to use the product. And let's just go with what feels right to me versus pure data driven research driven for designers that are maybe listening for product managers. Do you have any advice for just like where to fall on that spectrum and just how to best leverage research to inform that opinion? Yeah, I taught a class that you see Berkeley this semester on leadership and we talk about that a lot because great leaders develop intuition.

29:46It's the pattern matching part of experience where you develop heuristics which allow you to make good judgments even if you can't quite explain where that judgment came from. That's what the gut is. But it's also like I said where bias comes from where we're cognizant like all the cognitive biases. There's a list of 151 of them on Wikipedia. I won't name them but like all those thorny things that lead us astray, you know, the behavioral economists and social psychologists study those live in the gut. And so the advice is when you are looking to check your gut, you have to do that thing. A lot of a lot of your listeners have probably read thinking fast and slow system one system two, right?

30:29I have a here right under my laptop actually holding out my laptop screen. That's so appropriate Lenny. So the secret is not that sexy. It's system two, right? So you engage that slow methodical process in which you do analytic thinking as a means of checking your gut. Slow in the grand scheme of things, right? Slow meaning not as what second decision, not like months of analysis. That's not what I mean. The other thing you can do and there's really great research on this is you bring in the wisdom of the crowd, right? So the wisdom of the crowd is a phrase a lot of people are familiar with and it works in a specific situation.

31:07The wisdom of the crowd works when the people involved with the decision are bringing diverse sources of information and judgment to the table. Obviously if everybody has the same sources of information, then it doesn't matter how many people are out there. So if you want to check your gut, get a bunch of different guts together, get a bunch of different people in the room who can bring evidence and intuition to bear and have an open like direct and kind conversation in which we might disagree. You know who's great at that? Researchers. Leading those discussions essentially and getting a bunch of people's opinions.

31:41Yeah, this is the structural solution I'm talking about Lenny is like I never asked for research teams to have their own separate OKRs. I said two things. Number one, what's the teams? Like shouldn't the PMs, the engineers, the designers and the research? Everybody should have the same set of metrics for success, right? Because either we're doing it together or we're not. And then I said my metric for success is when they want that have that meeting without you. That's my metric for success. If they cannot have that decision -making meeting without the researcher there, that means you've developed influence, strong, real strong trusting relationships, you're an active participant in the process, not just sort of somebody who provides input into someone else's process.

32:25And that is when researchers can have huge impact. I think of the PM role in the similar way, even though people won't have these meetings with that PMs because they're off at the center a lot of the stuff, but you want to be a PM that people want on their team. There's a lot of teams that are like we don't want any PMs, we don't have any product managers, they just get in the way. And I find that that's only the case when the product manager is not great and not really good at their job because most great PMs just make everyone's life easier. They do. The Greece. I love it. You mentioned also that before we start according that the biggest challenge for these researchers is the relationship with their product manager.

33:01Can you speak to that and what you've seen there? I'm wary of over generalizing, but I can tell you that from my experience and from what I hear, the product research or product insights relationship is one of the most challenged. And I think it comes from the fact that fundamentally many researchers are just not included in the process that PMs are running. And then there, actually I kind of thought I did some asking around before this podcast. And so I thought, there are some tropes that researchers have about PMs that are worth PMs knowing. Just like four or five of them, like the things that researchers know PM say which drive us nuts because they're not true.

33:51So the first one is that research just slows us down. Research is too slow. This is bullshit. A great research team can do research in a day, a week or a month. It just depends on what you want to get out of it. How much detail do you need? How many people do we need to talk to? What is the depth or breadth? Do we need to go to seven different countries to talk about our constituencies in Latin America? Well, that's not going to happen overnight, but we don't often need that. The other way to look at that is that is it slower to get it wrong and fix it than to take a hot second to do the work to get it right the first time?

34:31You know? So that's BS. Like research, good research doesn't slow us down its pizza set. And also just along those lines, like a big part of your premises, you don't need to do as much research as people are doing like this middle research that a lot of the time is put into. Yeah. Research can go super fast. I think especially so the macro level research, I hope what it is is tied to things like annual planning processes. We did a thing at Airbnb several years that we called the we was like insights 2019 insights 2020. They were concept car projects and we spent quite a long time synthesizing the entire years worth of insights from every place we could get them and then developing with designers and engineers like a concept car for five years in the future.

35:14So that's a long process. But the micro level there's so much business value to be derived there. So much business value and it can go so fast, Lenny, right? It can go so fast. You can have you know, you can have results in 48 hours on these things. Like we did a thing at Airbnb. There's a famous story which I'll only tell in the abstract because I don't want to out anything. But it's like the we call it the multimillion dollar button. And basically we did research which revealed that people weren't going down the purchase funnel because they were afraid. The calls to actions on the button was making them afraid that it would initiate a purchase when really it was just taking the next step, right?

35:58We changed the text on the button with with help from our amazing content design or UX writing team. We basically changed seven characters and made Airbnb millions of dollars, right? Because because what we found out it was really simple. It was just like, hey, this button feels scary. The CTA and the button feels scary. So that's a great example of how micro and that happened in like 48 hours. We would discover that insight or 20 overnight basically. And we were like, maybe we should test some other CTAs. We did the conversion, like we added like 1%, which is really, really hard to do. You know?

36:34So that's a quick example of how that type of quick research can drive a huge amount of business value. This episode is brought to you by ATREFs. Many of you already know ATREFs as one of the top tools for search engine optimization. It's used by thousands of SEOs and companies like IBM, Adidas, and eBay. What you may not know is that there's a free version that was made with small website owners in mind. It's called ATREFs Webmaster Tools. It's free and it can help you bring more traffic to your website. ATREFs Webmaster Tools will show you keywords that you rank for and backlinks that you can get.

37:07It also performs automated site audits to find what issues prevent your website from ranking higher on Google. Every detected issue comes with the detailed explanation and advice on how to fix it. Visit atrefs .com slash AWT, set up a free account, connect your website, and start improving it. That's ahrefs .com slash AWT. So just to make this even clearer, I think this middle research zone is the stuff that does slow people down. I imagine it's like what are the challenges host have with payments on Airbnb? Like what you're basically saying is spend your time doing the micro stuff like usability research and then the bigger stuff that's part of overall planning.

37:49That's part of the planning cycle. It's not like every project you're working on you need to have like a whole research project on. Exactly. The micro research should be much more common. A lot of researchers think that they're that that's sort of scut work. You know that like usability is something junior researchers do, I completely disagree. Like I think we need to get back there as an industry and be like when you make a product easier to use, when you like discover problems with functionality, business metrics we care about will go up. I've seen it happen, right? But that's not just work for interns and new grats.

38:23That's for sure. And then the planning process like absolutely if we're integrated from beginning to end, we can help. And you know the thing about that middle range, I think you're right. Like that's the stuff that makes the sort of stereotype that researchers is slow. And a lot of times it's also because it's just not pointed enough. The researcher can also say in that moment, I have studied the business plan. I know exactly where like I've seen the metrics trend. I have an idea about exactly where that's going to go. We still need to do that middle range research. The question is valuable, but it's now very pointed and that the time is worth it.

38:58Okay, I want to hear the rest of these troops. Okay, so researchers to slow is the first one. The second one, I can do my own research. Why do I need researchers? And like that's true. As product people, I hope, I hope, you know, you are engaging with customers and listening well, but no offense, garbage in, garbage out. You know, like the thing, the thing is anyone can talk to a user. That does not constitute research or insights work. You know, because one user can be powerful, but one user can be idiosyncratic. And a researcher knows how to get to the heart of that really quick. They know how to take that conversation and understand and situate it in a way, which means like sure, like democratize research.

39:44That's happening. There are tools out there that will let anybody get customer feedback, voice a customer type stuff. But a researcher is there to help you turn garbage into, you know, into something that's not garbage. So and avoid the bias that can come from like you're just reaching out to your cousin's family and like doing whatever they thought you should do to the product. So that's the second trope. The third one is AB test everything, right? And like, like, AB tests are great. But one of my most painful things to do is to sit in a room full of PMs and data scientists who have just seen the results of an experiment that like flipped to statsig.

40:24And then they're like, cool, I was significantly down over this course of time for these users. And then they just start speculating about why that is because the AB test rarely tells you why it changed in the way it did. And then this endless flywheel of AB testing goes. And I'm like, hey, you don't have to guess. I know somebody who can get you an answer to the or at least evidence that addresses the question of why did we see the test result we did in a very short amount of time. Or you could use your customers as guinea pigs and throw more experiments at them over and over and spend a long time on it and get an income to the same place in the end.

41:03I think a similar critique that PMs often have is AB testing is conclusive scientifically, statistically, user research is just talking about your people. Why would I trust that? What is your best way to help PMs realize that this is actually very valuable data? And you should listen to it. It's not just this story here and there. Yeah. No, I think they're both right. AB testing is as close as we can get to causal making causal claims about products, right? Research is usually not oriented towards making causal claims or it should not be. But those causal claims rarely tell you how and why things happen.

41:43And if you want to develop, if you want to not make that mistake again in the future, you need to know how and why. If you want to build a better product in a way that doesn't just answer this narrow question that an AB test answered, you need to know how and why. And so you kind of need both. Like, beautiful partnerships between data scientists and research and insights people, like, are I think what we're going to see in that next evolution? And if you set that up, that sort of virtuous cycle lock, if you set up the engagement where those people are involved from the beginning, you don't make those mistakes.

42:13You get the causal relationship, which is valuable for one reason and the housing wise, which are valuable for other reasons. Awesome. Okay. I think there's two more tropes you had. So one of them is like a simple one, which is like everyone loves to quote that that turns out totally apocryphal Henry Ford quote, you know, about if I'd asked my users, it turns out the best of our knowledge. He did not say that. And he, yeah, I know. Isn't that sad? I know. I'd have burst your bubble, Lenny. Oh, wow. Does anyone say anything? I feel like every quote is is it valuable now? I know. What is reality?

42:47Geez. Can we? Well, let's just don't. Okay. Maybe he said that. Like, he certainly believed that's what the historians say. But the reason that makes researchers so angry is because that's not research. That's not what researchers do. A researcher who's going to ask customers what they want is a bad researcher. You need a different researcher. Like, that's not, I've never done that in my career. No one on my team has ever run a study that's like that. So that just makes researchers mad. And then the last one is about post -talk bias. We knew this already. That was obvious. And I think a lot about this book, which I would recommend to your listeners.

43:29So the author is a sociologist at U Penn named Duncan Watts. And the title is Everything is Obvious. If you already know the answer. And it's about hindsight bias. He kind of like makes the argument that we rely too much on intuition, heuristics, and pattern matching in a way that sort of is inappropriate to our experience. And it's like it leads us astray. It's kind of like a form of self -gaslighting. And it happens because we end up sort of selectively remembering things and then constructing narratives around them in a way which makes us feel like we already knew that when we infected not.

44:04And he talks about this other than one of those cognitive biases called the narrative fallacy, which is the idea that people love to make convenient simple stories about the past. If I asked you about your career, Lenny, and how you got to be this amazing podcast host, you'd be like, well, let me tell you about the series of events. And we do that. It's like part of how we make sense of our lives and the information around us. But it would probably be a lie. In the sense that we all kind of twist the evidence we have to fit the narrative we want to be true because it's simple and lovely. It makes us happy.

44:36This is going to sound self -serving, but I find them the opposite. I'm like, I have no idea how this all came about. Here's here's some things that kind of happened and somehow I ended up here. But maybe I'm being very modest and try to not give myself any credit. That's beautiful. Thank you for these troubles. By the way, this was fun. I didn't know you were going to do that. So that's a fun little collection. We got here. I wanted to ask about there's this tweet by Patrick Collison that I've brought up a couple times on this podcast that I think is really interesting. And his tweet is this.

45:06In my opinion, the best product will stem from a very strong mental model of the domain and user. User research can help you get such a model and validate it along the way. But it's important to view the syllogism of UXR as model as user research to improving your mental model of the user to what product you should build versus user research tells you what product to build. Does that resonate in any way thoughts on that way of thinking about user research? Yeah. I mean, there's a double -edged sword we talk about a lot in the research community, which is about making recommendations for design.

45:45So the best research doesn't leave it at that. Right? It tells you and it's like the what, the so what, and then the then what. But the problem with that is some researchers go too far in the other direction where they're like, we ran the study. It yielded these insights and therefore this is what we should build. And everyone else on the team is like, whoa, whoa, whoa. Right. Glad to hear your thoughts on the matter. But like there's a lot going on here. Maybe we should talk about it. So and that makes perfect sense. That's like a failure of communication. And I think that it like speaks to the the thing that Patrick is saying is like good research can sometimes tell us exactly what the problem is and exactly how to fix it.

46:28An example that is the the multi -million dollar button I told you about. But in a lot of the bigger picture questions, especially the macro ones and maybe also the really pointed middle range ones. The point isn't really this is exactly what we should do. And this is exactly what we should build. It is let us develop a framework, which is based on actual evidence. And then together as a team figure out how we want to experiment our way to a successful product. To close the loop on this specific thread, what is your advice to teams researchers to help move out of this kind of reckoning and to move forward and help the field both from a user researcher perspective and also from just like a company that may be laid off a bunch of user researchers or is trying to decide what to do with the researchers.

47:14Thank you for asking. I think I said you earlier like, you know, and I feel some pressure as maybe the first conversation that you've had specifically about research on this podcast. I think so. And I want to help. I believe so much in this discipline of research and insights. And I think there's the you know, when I said the user, the UX research discipline of the last 15 years is dying. I didn't mean that I think research is dying far from it. I think that there's a version of it, which we're now moving past and into a new version. We're going through an evolution as many do. And so the question for me is like, how can researchers and the companies and other people with with whom they work create a new version, a different version and evolution, which is hugely impactful for the business.

47:57And so the advice I give to researchers about that is develop diverse research skills, remembering like the five or five and a half tool list that I that I mentioned earlier really go deep on that business knowledge. So speaking the language of product and business and metrics and understanding exactly how to use your insights like a scalpel, building those strong relationships, which is not a thing that researchers can do by themselves. It requires two way engagements. And also in a way, which allows researchers to do fewer things better. So most researchers that I know are working on teams where they're like, I'm the only researcher and there's like, I have seven pms and 20 designers and like I'm trying to do 10 projects and no one's going to do a job, a job that way.

48:43So researchers have to learn with their partners about how to say no and focus on the most important things. But that's like only half of it, right? That's the research side. I have like two thoughts about about what companies should be doing. The first one is it's a little bit of a side, but not really like one thing I learned through the responses to to the article was everybody came out of the woodworks from the variety of insights disciplines that are out there because I come from a tradition of user experience research or user research, but there are many insights disciplines in many industries.

49:16And they all wanted to claim one type of research or another and say like, oh well, we over here in consumer insights or market research have been doing that well for years. And you know, there are many insights disciplines. And generally, I think creating silos is stupid. I actually am curious what you think because here's the number one thing I heard when I joined Airbnb and you were there. Is I, I did it like a quick listening tour where I talked to a bunch of product people and and they all kind of said the same thing. They were like, listen, we have all these different people throwing insights over the transom.

49:49And it's great. We want to hear from the data scientists, from the product specialist, from the customer service people and the voice of the customer and recent whatever, all that stuff. But they're all coming over the side and we don't know what to make of it, right? It's like too much. And that as much as anything is an argument for companies to stop siloing research disciplines. So when I joined Airbnb, I like set out to create an integrated insights function, whereas like let's do UX research. Let's talk about the market and competitors when we have to let's integrate smartly with data science functions.

50:21Let's integrate all the stuff forgetting from customer service feedback. You know, we we brought over what was then the NPS program and and sort of said like, Hey, like if we're getting customer feedback there, let's all just like use it all to fuel this one insights machine. So that's the first piece of advice like your companies. And the second one without being a broken record is to think differently about the broken cycle. So integrate researchers into a unified lean process. So if the researcher is not there from beginning to end, if there are not strong relationships between product people and design people at every level, engineering people at every level, and somebody who's their partner, their insights partner, we're going to fall back into this problem where we're just a service discipline.

51:04We're not extracting the maximum value. It comes too late. We don't know what questions to ask. We're ignorant about what research can do. And so creating that integrated lean process where a researcher is arm in arm from the beginning is the most important advice I'd give. That last piece may be the answer to this next question, but the question is how can product managers be better partners to user researchers slash take more get more leverage at a user researchers. I think that is in many ways the answer, making sure that they are creating a process for the product for their products that integrates user researchers and insights from beginning to end.

51:42Also, being willing to partner with the research on the ruthless prioritization, I used to say that a full plate for a researcher was probably three things, two big projects and a small project, like a side project. More than that, your researcher is probably not doing a very good job. A project may take 48 hours. That's okay. But so they need your help to prioritize. They need you to participate. Great PMs will take the time to be with researchers to go into the field, even to travel. Did you ever do that, Lenny? I did. I went with Luis who introduced to me, like came up with this, basically told me to chat with you about this topic.

52:23Thanks Luis. We did a whole tour to Paris. Our whole team, our whole leads of our team went to Paris to do a bunch of focus groups and a bunch of research behind like actual mirrors. They've never done that before. Yeah. The trip and it was amazing. Yeah. Can I tell you a quick story about behind the mirror? So this is back for when I was at Facebook. And it was the high times there was like 2012, 2013. And newsfeed is like really taking off. Ads are going to newsfeed. And I was working, I was a leader of a team that was working on a number of other things on how to address post quality. Like how do we think about what's a good post and how do we get feedback about it?

53:03And so there was kind of a team of engineers that thought one thing that you can do on Facebook is hide a post. Right? So they were like, this is easy. Let's look at the posts that are hidden the most and use that as the signal of what's a good post on on Facebook. Seems kind of reasonable, right? And like something tripped me on this one. And so I was like, so I did two things. So the first thing I did is I looked at the distribution of hiding by user and found out that it's power law distributed, like everything on the internet. There are a few people on Facebook who hide a tonne. And then most people don't hide at all.

53:44And so then what we did was we said, let's find, we call these super hiders. We call them super hiders. And so we said, let's find super hiders around the office. And we'll get a super hider in and we'll do like a really traditional like user interview. I just want to, we just want to see. So literally the first person who walked in, I remember because this is a person who had those fingernails that are so long you don't know how they can do cup touch screens. But they did. They were amazing at it. And it was one of those rooms with the glass. And I insisted that the end directors, the product people and they were willing, whatever.

54:20So everybody's behind the glass and I'm there with them. And the recent excellent researchers in the room and they come in and we're just doing a traditional think aloud study. And so they, hey, can you open up your Facebook app? We would just love to see you, you know, what your experience is like. And so they, they opened up Facebook and we're looking and they, they look at the first story and they hide it. And they go to the second story and they hide it. And this went on for a while. And like this, she's definitely using Facebook. But every time she'd read, finished with a story, she'd hide it.

54:49And the people in the bathroom were starting to chatter and they're like, wait, what? What is happening right now? And like good research that the good researcher that this person was, like they let it continue and they're like, oh, can you tell me what you're thinking right now? Come to find out that she was like, well, I hit that story because I'd seen it already. The model she was going for was in box zero, which was like sad because it was infinitely scrolling. She would never get there. And the reason I like that story is because the people in the back room had their minds blown. It was not that we assumed that was common behavior.

55:28Like this person could have been unique. But it was enough because those people were there experiencing the research that end of one allowed them to burst their own bubble and realize, okay, we can't think so naively about hides as a signal anymore. And we came up with a better solution. That is an awesome story and such a good example of you don't need statistical significance to get massive insights. Like one example just gives you a, wow, this might be exactly what's happening. Let's go validate that versus like we are confident. 100 % this is what happened. I love that. It reminds me actually in the mirror in the mirror study that I was talking about in Paris, there's a Facebook element to it too.

56:10We were trying to convince hosts how to feel more comfortable accepting guests to a book instantly. And whenever a theory was, if they were connected on Facebook, they would be more comfortable letting someone book instantly. Yeah. And we're just like, hey, why don't we, what if you were to connect Facebook and see if their friends and everybody in Paris was very afraid of connecting and giving Facebook any data way ahead of like what the US house to are feeling. Yeah. So it just made us, made it very clear. Nobody wants to actually give Facebook any data. So it was very anti -facebook at that point.

56:40Yeah. That's so interesting. Germany, Germany and Finn and France were always are like bellweathers for what the rest of the world would be thinking with private data privacy concerns. Oh, man. Okay. A couple more things. So a lot of this started with a lot of layoffs within user research. And I think between the lines, there's a sense of teams don't need as many researchers as they hired during the Zerp era. I think a question and everyone's mind is just like, how many researchers do we need? What is a good ratio? I imagine there's not a simple answer here, but just what's your general advice to companies of how many researchers as a right?

57:15So this is a thing I thought a lot about, especially in my role as the head of the design studio, that was like my fundamental question. It's like you have all these writers, designers, researchers, like how do you structure them? How many and where and who works on what? And the organizing principle for me was always relationships. You know, you have enough when the people who need to have a constant research partner have them. And I would much rather create pain in that situation than spread someone tooth -in -lease. So my advice was always like, don't try to create a researcher to cover this entire product space.

57:46Care a researcher up with, you know, somebody who's going to involve them in a consistent, engaged process and let them go to work and see the impact they're going to have, but protect their time. And then other people are like, wait a second, that person's doing great work. I want some of that. And creating that pain for them, like because it's a pain of loss, like is the number one way to grow head count? That's how I always approached getting more head count was not arguing abstractly for why researchers important. But by asking partners who wish they had it to do the arguing for me. And so there isn't a, you're right, there isn't a clean answer for like, hey, this is the right ratio because it really depends on the nature of the product.

58:26Like, is it a early stage product? Is it a late stage product? Are we talking about a startup or a late stage company? But, you know, I would argue there's always room for a researcher. Lenny, I'll tell you, and I use this in a keynote talk I gave lately. You published recently a list of, I think it was about 20 B2B companies and their first 10 employees. Do you remember doing that? Absolutely. Do you remember how many researchers are in anywhere on that list? I'll give you a hint. It's between zero and two. It's one. There's one researcher on that list anywhere, anywhere. And that's messed up to me.

59:06Now, look, it's just these 20 companies and like they're, you know, each is in their own space. So I'm not going to over generalize. But there is an in a researcher can drive incredible value no matter what stage a company is at. Because a good researcher makes you go faster, not slower. And they drive impact because they answer questions which are impossible to answer in any other. That's true if you're a startup. It's true if you're a late -changed company. Now, if it's your first 10 employees, you know, like one one researcher is going to go a long way. You know, as you grow, making sure that you're matching up researchers so that they have strong partners in the key parts of the business is the best way to figure out if you have enough.

59:44Interesting. So your advice is as you're starting a company, your pitch is that you will have a lot more leverage and move faster, hiring a researcher versus generally, an engineer's what you'd be trading off. Essentially, it's the most layers into being. I'm reluctant to over generalize, but I would say, and I would say I know in many founders who are in startup mode are like, I know what I need to build. The problem is that I need people who can help me execute. And I think that's right. And so everything's a trade off. But remember, imagine that you could have that Swiss Army knife at your disposal.

1:00:20As you, maybe you've got an MVP out the door and you're looking to make your first major iteration or like many startups, you need to pivot. Like this is where it's like, hey, you don't have to do that alone. We deify startup founders who pivot appropriately, but I think that is what we would might call more luck, where we deify the ones who got it right. And even though they made exactly the same decisions as the one who got them wrong. And the fact of the matter is, if you have an insights person with you, but has that Swiss Army knife of tools, like you're not in it alone, you don't have to guess.

1:00:54Like it ultimately will still come down to a tough decision that you and founders have to make. But you can have evidence that bears on that decision, which you wouldn't be able to get any other way. To close out on this and have just a couple more questions on this thread, I think one of your big messages to researchers is you can be empowered. Like it's up to you to do the right sort of research and to move your career in the right direction, not become a researcher people to need. And there's this quote that you have at the top of your post where a lot of the reaction, or I guess the way you put it is, I know what you're thinking.

1:01:29They just don't get it. We're so misunderstood. Our plight is to deliver insights that users used to drive business value while we're forgotten. Never driving the roadmap. No seat at the table consistently miscast only to be laid off in the end. And what I'm hearing from you is like, you can change that. Like you can push back on doing research that isn't actually contributing. But let me ask you what's your kind of lead lasting, I don't know, advice you would lead researchers with to be successful. Yeah, it's tough to be operating in a broken system. And so I feel that response where you're like you feel kind of powerless.

1:02:02But I think that's not likely to lead us past this moment to the next evolution of research. So that's kind of where it's like, I don't blame any researcher at all for being in the spot there. And it's been a tough go. However, crying about our lot is not going to get us anywhere. So I think the point of the article for me, and this is advice I give companies all the time when I do consulting with them, is like, hey, we can set this up in a different way, which responds to the current environment in a way which will drive a huge amount of impact. Now that takes companies making the right choices.

1:02:38It also takes researchers owning up and developing skills, right? Pushing back, understanding what research can have the most value, developing the skills and the knowledge and and language around the business, like becoming more influential, being excellent communicators. Like it's one of the things I would evaluate the most in hiring, especially research leaders, because I needed them to show and teach by example, is like, can you, like, it isn't just rigorous research. It's like a tree fell in the forest and no one was there to hear it. You know, you need to communicate it effectively. And you need to do it in a way that's appropriate to the audience, because often, if I'm talking to you, Lenny, it's different than I'm talking to Brian Chesky at Airbnb.

1:03:18And so, I got to be able to give that presentation effectively and get right to the heart of it and speak the right language. And so if you're a researcher, it's not hopeless. Actually, the discipline, the future is so bright and we can help it alone by continuing to develop these different skills as companies build a model that's more inclusive. Awesome. Okay. I have one just random tangential question about NPS. You have strong opinions about NPS. And I just wanted to hear your perspective on the value of NPS, your experience with NPS. Yeah. I do have a strong opinion about NPS. I like to say NPS is the best example of the marketing industry marketing itself.

1:03:59And I think I'm going to, the problem is this threatens many people's livelihoods, because there's an entire industry of consultants and software providers that want you to believe NPS is a useful and accurate metric. The problem is, so the consensus in the survey science community is that NPS makes all the mistakes. So it's a garbage in garbage out problem. So the likelihood to recommend question is bad for a whole variety of reasons. So it's bad because it's a zero to 11 scale. It's bad because it's usually unlabeled. So we label the polls, but that's not the gold standard for research. It's bad because it's 11 items, right?

1:04:37And there's a couple of problems with that. Number one, we find that precision goes down after five items on average, maybe seven. Number two, especially on mobile, if you're taking this survey, what percentage of those options are below the fold? We are not going to get accurate survey data. And so from a survey perspective, it's really bad. There's also this intuition, which is like, are you a person who will let, you know, like, are you, are you, like, how likely are you to recommend Windows 11 to your friends and family? And not a person who goes around recommending operating systems, the question is fundamentally flawed.

1:05:11The argument is that's the, that that question is a good indicator of loyalty, but there's a really simple solution. I need customer satisfaction. A simple C sat metric is better. It has better data properties. It is more precise. It is more correlated to business outcomes. I wanted to prove this. This is something that survey scientists know and marketers don't want you to know. And so I did, we did the work with Mike Murakami who led survey science at Airbnb and he's still there. Great researcher. And we basically re -did all that work to find out if all that stuff was true just for Airbnb. And it is.

1:05:46It's simple. Don't ask NPS, ask customer satisfaction. And the customer satisfaction question, what's the actual question for people that are there? Over all how satisfied are you with your experience with Airbnb? Or it could be some version of that, which is like, overall how satisfied are you with your experience with customer service when you have a problem? So there could be a more specific version of that question. But those questions have better properties. And a lot of people say, well, hey, everybody's using NPS. So at least it gives me a benchmark because I can compare my NPS to industry NPS.

1:06:18The problem with that is the research shows that NPS is idiosyncratic. So it goes up and down in ways that we don't understand. And there's a lot of inconsistency in how it's asked that creates variations in the data, which means it's not apples to apples. So you can't even compare your NPS meaningfully to somebody else's. I love these hot takes. I'm curious to see who comes out of the woodwork too. Why would you be so mad Lenny? I love that. I think, yeah, people, like I've heard this many times and people don't talk about it. Okay, is there anything else you want to share or leave people with before we get to our very exciting lightning round?

1:06:54Yeah, I want to add one thing if I could because this has come up on your podcast a few times recently, which is about the idea of people doing their own product walkthroughs. So should a PM just rely on their own dog fooding of the product and their own walkthrough to figure out how to fix it. And a couple of times recently, this has come up like this. And I think the consensus seems to be sort of yes, this is a good thing. And I have a contrary and opinion there too, which is that I think it is really important for everyone to dog food their own products. The problem is related to relying on your intuition about those products, which is the thing most PMs have trouble with is realizing you are nothing like the user.

1:07:43You are nothing lie them in ways that will bias the way you think about what's good and bad in your product, in ways that you can't necessarily recognize. Like some things with a product, some problems with the product, you need a pulse to recognize. And like most good PMs that I know have a pulse and so cool. But a lot of them require like context of use, priorities, constraints that you just don't have. And you can't imagine purely on the basis of your own usage. So what I think that means is that you should definitely dog food your own product. Doing product walkthroughs to identify lists of potential issues is a great thing to do.

1:08:25Prioritizing that list. Figuring out which ones are more or less a problem and for whom is an area where you should be extremely wary of relying on your own sort of opinion, expertise or intuition when you are dog fooding your own product. Thank you for sharing that. It's definitely come up a bunch on this podcast. So I think that's an important lesson for people to take away anything else before we get to a very exciting lightning route. I appreciate you, Lenny. Thanks for having me on. I appreciate you, Jed. Well, with that, we reached our very exciting lightning round. Are you ready? I am ready.

1:08:57What are two or three books that you recommended most to other people? I recently read a book by a business book by Barbara Kellerman called Bad Leadership. And what I love about it is that we spend a lot of time talking about good leaders. And she really dives into the worst leaders and what makes them bad leaders in a way that I think is really valuable for everybody. I'd also recommend I read a lot of fiction. So two recommendations there. One, recent pull surprise winner, Demon Copperhead by Barbara King -Selver. It's like an outstanding read that also is really sad and moving and illustrative, especially when you want to understand rural poverty.

1:09:36And then completely other side of fiction spectrum, if you're interested in science fiction, which I am, read the Murderbot Diaries. It's like a sarcastic robot, a sarcastic killer robot who doesn't love them. I love these fiction recommendations. I feel like we need more of these on the podcast. Yeah, everybody goes to business books. Yeah, absolutely. What is a favorite recent movie or TV show that you really enjoyed? We recently watched the last of us and it blew our mind. I watched it after I played the video game after a long last. If you are a person who plays video games and you haven't played the last of us, if you don't know, the show is based on the video game, not the other way around.

1:10:14Do you have a favorite interview question you like to ask candidates that you're interviewing? Think of a topic that you had to explain lately that was the most complex and then explain it to me like I'm five. And there are a lot of ways to vary that question. But the reason I like it is because I think and I've asked this question to VP and C -suite candidates in like multiple disciplines. And sometimes it's related to a conversation like I might ask them to explain something complicated about quantum computing or music theory or it could be a complex business decision. But I want to see if somebody can break a complex problem down in a really simple way and give me an intuition for it in a short amount of time.

1:10:58I think that is a differentiator between good and great for many people. Do you have a favorite product you recently discovered that you really like? Yeah, this is a really weird one but my whole family started indoor rock climbing recently. And there's a challenge you have when you top rope which is that you're looking up all the time. So they make these glasses which are called belay glasses and they have an angled mirror embedded in the lens so that you can look straight ahead and the view you see is up towards the person who you're belaying. And I just thought that product is like so perfect for that.

1:11:34That's a niche problem and there isn't a better way to solve it. Do you have a favorite motto that you often come back to that you share with friends either in worker and life they find useful? Yeah. So this is going to seem like pandering, Lenny, but I don't know if you remember a conversation that you and I had. It must have been eight years ago. I remember where we were sitting and it was about stoicism. Do you remember this? Anyway, we had this kind of a dance but I was in the stoicism for a while. I know you were because we talked about it and so the motto is comes from stoicism which is basically focus on the things you can control and ignore the rest.

1:12:12And you know, a lot of people think of this as you know, a lot of people think of this as the the serenity prayer or the serenity saying that is that was the 20th century invention but Epic Titus was writing about this, you know, BC and I think about it all the time. So much of the stress and pain and worry that we have in life comes from things we can't control. So I try to let those things go. Amazing. I learned that lesson from seven habits of highly effective people and just the importance of thinking about these circles. If you can control, you can influence and you have no control over and there's no reason to think about those other things.

1:12:46Absolutely. Judd, this is everything I hope it would be. We got into some really good stuff. I'm excited to hear how people react to final questions. Working folks find you if they want to learn about what you're up to. Actually, share what you're up to these days and how people can find you. And then also how can listeners be useful to you? Yeah. Thanks for asking those questions. So people can find me at juddanton .com. That's the best way to find out what I'm up to. These days I'm a consultant. I help people with UX strategy, org design and crisis management. Somehow I love dealing with other people's dumps, your fires.

1:13:17And I've found that I'm constitutionally good at it somehow. So juddanton .com is the place to find that. I also write a medium post that you can find at onebigthought .com. And you'll find a lot of the topics we talked about today, including the original post that started this at onebig .com. If there's one thing I could ask your listeners to do is to get next to your researcher. I just think if you build those relationships and involve a researcher and insights person early and often, beautiful things will happen for you and for the business. So that's the thing everyone can do for me. I love that.

1:13:56I've always done that. I love my researchers that I work with. Many of them are reporting to you. And so beautiful takeaway. Judd, thank you so much for being here. Lenny, thank you. It's been a pleasure. Bye everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts Spotify or your favorite podcast app. Also, please consider giving us a rating or a leaving review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny's podcast .com. See you in the