283. How Google and Netflix design engagement surveys that actually work, with Bill Yost

15:58So succinctly, send a survey, do something with it.

16:02Al:That's how you can ultimately be successful if you're trying to capture this really unique information that you get because there's really no other, you don't get employee sentiment through workday or any of your standard HR systems that are like capturing employee lifecycle stuff. Like you're truly capturing somebody's voice, their feelings, how they're perceiving the company. And that's really important if you're going to actually use it for something useful.

16:28Bill Yost:So we're going to come on to performative surveys in a second. But the way you were describing that, I'm imagining a bank of goodwill. and what we're doing is if we're sending a survey we're withdrawing from the account and then when we actually give the results and send the results back we're kind of like almost depositing but is there anything else we can do like any other things that would withdraw from it or any of the things that would deposit into that account if you'll forgive my flaky analogy

16:54Al:i love that analogy i wouldn't i wouldn't call it flaky at all so google geist was also very popular because there were tangible changes in the company that could point right to the survey as being inputs for why they did such a thing. There was an entire function at Google called Core that existed because of Google guys' feedback that did not come in as positively as I think a lot of the stakeholders would like. And it pointed to a need for like a pretty big restructuring and a new team to fill some of these gaps that were being identified through the survey. So they like did it. They staffed a VP, put hundreds, maybe thousands of heads behind this function, and it became one of the institutional benchmarks of what we did well with this survey.

17:42Al:It was like, we heard your feedback. We made a whole new organization to try to address this specific feedback, right? That's the top level thing you can do. I don't think every company is going to staff a thousand person organization just based on survey feedback. But you can imagine there's lots of trickle down, smaller things that could manifest, right? If you're seeing in survey feedback that parental leave is something that leaves a lot to be desired, you could work with your benefits team to actually get a more compelling or competitive parental leave package in place. And then when you launch it, make sure you say, we did this because we've heard your feedback.

18:22Al:Based on the results of this, we know we can do better, so we sought out to do so. And now here's this new program. You know, we hope that it meets your needs and we'll ask that same question next cycle. So we can actually check this with data as well. And we can have the pre post sort of observations of what happened after we launched such a program.

18:42Bill Yost:I think a lot of people who are reticent about sending surveys are exactly described there. Everyone says we want some kind of parental leave. And they're like, we can't do that. We're a company of 15 people. So what would you say to those people who don't want to do the survey because they're worried that people can ask for something they can't give?

19:02Al:That's a great question. So I would say first, don't waste time asking questions that you don't intend on doing anything with. Right. The textbook example of this is, hey, do you feel like you're paid enough? Because no one in their right mind is going to say, yes, I do. I mean, I will say I've seen this result. Some people do say that, but you would be silly to not say, no, I want you to pay me more. But if you don't actually have budget to do so, the company's not in the financial position to give everybody raises, you shouldn't go ask everybody if they want raises because you know where this is going to land exactly.

19:40Al:So focus the survey, laser focus the survey on things that are actually in your control and you're willing to change. Because there's another tradeoff that you need to consider every time you send a survey, which is called survey fatigue. means exactly what it sounds like, right? If the survey is too long, people are going to be like, this is too long. I don't want to do it. And they're just not going to do it because ultimately it's optional. Or if they get too many surveys, even if those surveys are of an appropriate length, they're still going to be like, I'm not going to do a survey. I did a survey two days ago and I'm going to get another survey next week.

20:15Al:All I have is surveys coming out my ears here. So I don't actually want to do any of this. That eventually leads to a place where you're not going to have response rates for anything. And unfortunately, it was not a favorite part of my job when I was working with this at Google, but we did have to do a lot of consultations in the same way that I tell people, you don't really need a dashboard here. I had to say, you don't really need a survey here because you're not actually trying to get to new information. You're probably trying to validate something you already know. And the meta trade-off here is yet another survey that's going to burn everybody out and it's going to lead to them probably not taking my survey, the big annual one, which we want them to take.

21:19Bill Yost:episode 124, where you're going to hear how industry giants from Canva to YouTube define leadership.

21:24Leanne:Listen to Billion Dollar Moves wherever you get your podcasts.

21:32Bill Yost:So let's invent something called the Yost coefficient. And this is a formula which is derived, which tells me how long a survey should be and how many times I should send it a year. Obviously, it depends. We both know it depends. But let's just say that I'm a small business, 25 employees. How many surveys should I be doing a year and how many questions should it have?

21:56Al:It does really depend. I will say the big survey that I sent, depending on who you were, because some functions or roles could have more questions than others, could creep into like 120, 130 questions, which is a lot. However, it didn't really impact response rates significantly when of the survey was that long because those questions were all perceived as valuable or lightweight enough that people could move through it easily and still get to the end of that submit response button, which they ultimately wanted to do so that their responses could be heard and their management chains could see the results and everything.

22:33Al:That said, I wouldn't recommend a company of 25 sending 130 person or 130 questions survey out the door because there's how much information are you really going to be able to make use of at that scale? So one of the variables in this coefficient is definitely size of the company. Another variable here is going to be your willingness to change or the number of levers you have to pull to actually make meaningful change based on the results. And another variable is how often is that survey or all surveys going out every year? Because if you think about when you start a new company, at a new company, you might get a new higher onboarding survey.

23:15Al:You might go to orientation and get a survey for how that orientation went right after. You might get a survey from the IT department being like, how was your laptop set up? You can just be inundated by surveys really, really quickly out of the gate. And then it comes time for the twice yearly or four times yearly engagement survey that's also just being like, are you proud to work with the company? If you were offered a competitive role elsewhere, would you take it? Those sorts of questions. And so it's hard for me to say how many is too many for the individual because I have personally burned out on surveys just from being in the space for so long.

23:51Al:Like if Panera Bread sends me like, how was your meal survey? I'm like, nope, not doing that. There's just no way I can at this point. But if I had to throw a rock at it, two or three per quarter is probably on the upper max of what I would be looking at. It often will hinge on – most companies will have their annual or semi-annual employee engagement survey. At Google, like I mentioned, it was once a year for the big one that I worked on. They've since changed that significantly. I'm not sure if they're still sending a big annual one, but they started shifting to a much lighter weight weekly survey to try to get less latent data because the big complaint was, well, that survey was in July and now it's December.

24:36Al:So who gives a shit? You know, the things have changed. And there is some merit to that, actually, to be totally honest with you. At Netflix, it's going the sentiment survey is going out twice a year. So you get a little more regular data, but you still get into this period of like, well, the survey went out in, let's say, February. The results came out in March. The next one is only going to come around in August. So there's quite a bit of time still between those two. So it's a very delicate balance. But I would say, yeah, maybe two to three per quarter across all the surveys that people are hitting is probably sufficient.

25:10Al:because more than that, it's just going to make me raise my eyebrow a little bit and be like, are they that valuable though?

25:18Leanne:And they very well could be, like I said.

25:20Al:So to summarize, it's number of questions, size of your company, how often you're sending surveys, throw that into the magic pot. That's the coefficient. I don't actually know what the ideal answer is. Your mileage may definitely vary.

25:33Bill Yost:What about these little pop-up surveys that come out like once a week and it goes, how are you feeling today on a scale of one to five? Do you want to throw rocks at your manager today on a scale of one to five? Like sort of five little questions that pop up at the corner of your screen. Does that work?

25:51Al:If the question was, do you want to throw rocks at your managers today? The legal implications of that aside, because I'm certain the corporate lawyers would have something to say about that, you would hook me back in. I would take surveys again if you were asking me those sorts of questions. Unfortunately, the ones that I've programmed and wrote over the years are much more boring than that particular one. But what you're describing is called a pulse survey, right? Most regularly coined as just regular, literally heartbeat pulse is what you're trying to catch when you're sending those questions out.

26:22Al:I have heard mixed success of those across across organizations. I don't know if this is still true, but famously, Amazon would have a survey question when you would log into your laptop. Like it was a requirement, like, do you want to throw rocks at your manager today? Yes or no, you click yes, and then you can go to your inbox and like start your day or whatever. Not sure what response rates are for that, or if that's successful for them. I know when such an idea was brought up as a, should we do it? It was always kind of like a, from a research perspective, you don't really want to compel people to take surveys that way.

26:59Al:You know, you want them, you want them to come to us, not necessarily force them to us because then you can potentially implement bias in the results because you're going to have people who are going to be like, strongly disagree to whatever the question is, because they just want to log into their laptop and they hate the idea of this. now taking the forcing them to ask the question to log into their laptop part laptop part aside like maybe you get a survey every week in your inbox that's like hey please take it we want you to we're not forcing you to what i do see in practice is you launch those surveys you'll start with a response rate you know that maybe you're happy with let's say 40 to 50 response response rate, simply meaning the number of people who take the survey divided by the number of people you invited to take the survey.

27:49Al:And so you get maybe a 40 or 50. Then the next week, you see it tick down a percent. And the next week, you see it tick down a half a percent. The next week, another percent. And over this arc of time, you're going to drop, drop, drop, drop, drop, because it does feel very operational at that point. And if you're asking a question every single week, to go back to your example, you know, you have this bank account, you're depositing and you're withdrawing, you're taking a lot of withdrawals from people. Like it's still, okay, one, two, three questions. It might be very, very quick, but it's still a withdrawal nonetheless.

28:20Al:And like the same way that I go buy avocado toast every day and my paycheck only comes twice a month, you know, eventually the avocado toast is going to drag me out and I'm going to be like, I need more paycheck here. But that's a crude example. But what I'm saying is specifically, like you're not doing the read back. You're not looping back with those results to your respondent base at that same cadence. So there's a real imbalance there. And so if you're going to try to explore this pulse survey type of stuff, there can certainly be merits to it. And it doesn't always have to be once a week.

28:57Al:You could say a pulse survey might be every month or something like that. You just have to make sure that you're ultimately fulfilling the other side of that transaction or eventually you're going to just make the survey be completely useless by way of not getting enough responses for it to be significant for anything i've heard a rumor

29:15Bill Yost:that you guys are clever and you have a way to to select out people like me will just go yeah click click click click run and get on with my day i don't care is that true mathematically we

29:28Al:can do anything i'll say that right we've got we've got the data we have unlimited time to figure it out, semi-limited time to figure it out. If we wanted to actually say like, Al is completely jaded and is not giving us quote unquote honest responses, maybe we could assign some sort of percent likelihood that that's the case. And then we just only look at the responses for people who are under that percentage threshold. In practice, you don't want to do that because you don't know for sure, right? I can never be 100 % certain. To spin this into a different but related field is predicting when someone leaves the company, right, is going to quit, is one of these hot button issues in my field because we have AI and machine learning and we know a bunch of stuff about employees.

30:15Al:So just like make it clear when somebody is about to leave, even if I can create a model with near absolute certainty that says Al is about to leave the company, what are you practically going to do in that situation? You're going to sit you down and be like, hey, Al, computer told me you're going to quit. You know, we want to talk about that. And in the same way, I can't be certain that you put strongly disagree. I can't be 100 % certain that you put strongly disagree the whole way down the chain because you're jaded versus because you're really, really unhappy. And if you're really, really unhappy, we don't want to dismiss your results, right?

30:51Al:You're the reason we're sending this survey. We don't send surveys to pat ourselves on the back and be like, hell yeah, everybody's happy if you remove all the unhappy people. Now the snake is eating itself from our side of the house. And we don't, as professional survey practitioners, we never want to do that. I will say there is this methodology called heartbeat analysis, though, which tries to tackle this exact problem. I'm aware of this because a good friend of mine, Chris Patton, and another analyst named Justin Pearl, they created this at Google and then it won some fancy award through maybe the PSYOP organization.

31:30Al:It was like the Wiley Award for badass survey stuff. I don't actually remember what the title was, but it actually creates a baseline of responses and then looks at where for each person and then looks at where the questions, where someone's baseline went above or below that. So if you went through and put strongly disagree for everything, your baseline is going to be strongly disagree. But if you picked one or two questions of agree, strongly agree, you know, you went the other direction, that's going to pop that as being like, hey, he went out of his way to change his responses here because he must feel really passionate about it.

32:08Al:That's where the kind of heartbeat element comes to it. So there are definitely ways to get to that answer without completely throwing your responses away. And also, we work really hard to get everybody to take the survey. So, like, I don't want to throw responses away. I never want to do that. In fact, part of the survey administration process for us operationally was people would get cold feet after they submitted the survey. So maybe they said yes to the throw rocks at the manager question. They're like, oh, my God, I just I was having a bad day. I don't want to throw rocks at my manager. Like, delete the whole response.

32:42Al:And we would do that. Of course, we would honor it. But that makes us sad because we want all the responses that we can get. high response rates validates our work. It makes it worthwhile. It actually, it also makes our responses statistically significant where it matters. So yeah, we love all survey responses, one and the same, even if you're really, really jaded and hit no to everything, Al.

33:04Bill Yost:Something you just said there, it kind of in passing, I think would terrify a couple of people listening because you went, oh, I noticed Al said X. Now I know it was just an example, but how anonymous are these surveys and how anonymous should they be? This is a fantastic question.

33:23Al:Okay. Sorry, listeners, we're going to get really nerdy for a second. Most surveys are not anonymous. In fact, it's a pretty big taboo to say that you're giving an anonymous survey because anonymous means, I mean, technically speaking, we can do it, but it means we can't connect any information about your responses back to any metadata that matters. So one of the most popular ways to review survey results is through what we call a heat map, right? You look at the top level by each question. Let's assume we're using five-point favorability questions here. Like what percentage of people at the company level said they agree or strongly agree with something?

34:03Al:We call those a favorable response. That's one set of one column in the heat map. And the other columns are variables that we care about, metadata related of the person. This could be the business function. This could be their VP. This could be their tenure at the company. This could be their location, region, country, site, you name it. And this is how you're able to get a lot more value from these survey responses because you can see pockets of the organization where sentiment differs. You can be like, whoa, new hires really freaking hate our benefits package, right? Like that's, okay, Why is that?

34:41Al:You can dig into that further and maybe you can try to decide if this is an onboarding problem or maybe the compensation packages for new hires is not as compelling as the ones for tenured employees. It gives you a lot of red meat to chew on, if you will, in terms of analytical value. And the only way to do that is by what we call a confidential survey. survey. And so confidential means a very, very limited number of people can actually tie Al's responses to Al's identity. But what we do is we collect it with your identity and then tag it to enough metadata that we publish out for end users to use, but not so much that we can make it very, very simple for Al's manager to be like, Al's unhappy.

35:33Al:Let's figure this out. Let's get him out of here or whatever it is, because privacy is still really, really important, but it becomes this trade-off where we do need to know some information about you to make those results as useful as they can be. And this is where I can nerd out about this all day is at Google, we used to do both. So there was a confidential version of the survey and there was a fully anonymous version. Now, I mentioned earlier that some of your roles or locations or functions might get slightly different or expanded questions based on who they are. You couldn't get that, obviously, on the anonymous side of the survey because we fundamentally do not know who you are.

36:12Al:So the anonymous survey was much shorter. It was really whittled down to what we called the core questions of, let's say, 20 to 25 general purpose questions about Google. And then the confidential side had way more than that because we could customize stuff. If you as a manager, Al, had seven people on your team, those seven people in the survey, if they're electing to take the confidential version, could say, hey, how do we feel about Al's vision, about Al's commitment to our career development, you know, whatever. And we could literally inject your name in there so that it becomes a more personalized experience for that.

36:48Al:But we did both because we conceded, you know, there might be people who are like, we find that creepy because we don't want ultimately people to be able to know who said what. But when it shook out in practice, we would have, let's say, 90 % response rate of the company. 89 of that 90 went to the confidential survey. It was always 1 % or less of the respondents who were going to the fully anonymous version because people wanted their responses to flow up to their management chain. They wanted their function leads to know how they felt about things. And they ultimately had enough trust in our survey team to administer it fairly and know that we're not leaking their data or exposing their identities to people.

37:31Al:And so this is where that trust, you know, that deposit withdrawal really, really takes hold because we do need to abide by that. And I was at one point one of three people in the company who could reach in and look at your responses, Al. And I'll be honest, not that interesting. I'm not in this business to try to dox people or be like, Al hates his manager, get him. You know, I'm just I'm a professional analytic person who is trying to do the best thing. And I also do have, you know, as much as I joke and laugh, have some ethics that I try to abide by, which is I want to continue to have trust from the survey respondents, but also stay gainfully employed.

38:12Al:Because if I started leaking people's responses, I would face my own kind of disciplinary measures for sure alongside that. So eventually I was very proud. We got to turn off the anonymous survey. We're just like, we're not actually, the juice is not worth the squeeze for us because we're getting very limited information from a response size basis. And also the information we get, we can't really do a lot with it because it's just like, we don't know how we can focus that information towards a specific set of people, know how pervasive or, you know, narrow the problem might actually be. And it's a ton of work to have two surveys running in parallel.

38:49Al:Like it was a real headache for me. So I was very proud when we finally turned that off. So I'll say most surveys across industry are of the confidential variety. Netflix's is as well, right? We can ultimately tie it back to the user, but we keep that under lock and key, really just the analytics team, a few survey admins who are able to do that. And when we do need to get to someone's identity specifically, it's for reasons that are typically unfortunate but good intentioned. So meaning like, unfortunately, you'll have instances where you'll have comments where people are like, hey, I was harassed at a company function or something like really awful happens, right?

39:28Al:In that case, we do want to go in, figure out who that person is so we can get it to employee relations or legal and figure out how we can remedy that situation because we never want to find a situation where somebody said something really terrible in a survey. We have no idea how to fix that. And then we're just kind of saddled with this really disturbing problem that could be addressed. So, yeah, that's my soapbox. Confidential versus anonymous. miss. Most surveys are confidential. Sorry if you've turned the podcast off at this point, listeners.

40:02Bill Yost:Well, if they have, they've missed goals and also they won't hear you say sorry. So they'll just be blaming you for the rest of the day, won't they? Honestly, that is genuinely, that's exactly the kind of answer I was looking for. Now, Bill, I don't know whether I've read this or whether we chatted about it when we did our little pre-call, but there's something I've got a note down here, something about how leaders, you can tell them the results of the survey and they just almost don't hear it and hear what they want to hear. Does that make any sense to you?

40:26Al:Yeah, I've seen that happen. I think, you know, have you ever watched a movie or a TV show with maybe a partner and something ambiguous is presented and then like the episode ends and you're both like, well, I think X happened. And the other says, well, I think Y happened. That's because even though you've both been presented the exact same information, you're still viewing it through your own lens, right? This personal lens that's really distinct to you and is quite hard to replicate across the board. We run into that a lot in the survey world when data are messy, people are messy. The survey might be messy.

41:05Al:You know, the question might not be as well written as you wanted it to be after you looked at it on the other side and you saw the results. As an analyst, it's my job, it's our team's job to frame that information in a way that removes the noise and presents it to a stakeholder in a way that hopefully can be universally resonated with. But that's not always going to be the case. A specific example of this that comes to mind is return to office, right? Hot button issue across every company that has to reconcile with if they made pivots during the pandemic. Some of their, or maybe most of their workforce went remote.

41:44Al:And now the question is, do we bring them back to office? If we have a survey that, let's say, asks nine questions about the benefits of working from home, and one question about the productivity gains that you have while working in the office, and the survey is screaming loud and clear that everybody likes the flexibility of being able to work remote or at home or whatever. and we present this to a room of, I don't know, let's say seven executives, at least one of those executives is going to have this lens of we're more productive in the office, right? And so it doesn't matter that nine of those questions are saying like screaming, blaring red, we want to work from home or we want to work flexibility.

42:26Al:They're going to look at that one that goes, well, look, we're more productive in the office and I'm actually optimizing for productivity and not employee engagement or experience. Therefore, we need to get back to the office. That's going to happen, right? And every time you think you have a concrete solution or answer to a specific problem, you're probably wrong because you're going to encounter numerous different lenses across the stakeholders, these executives, whomever you're presenting the answers to, and they're going to inject their bias, whether it's conscious or not, into these answers.

43:01Al:And sometimes you can try to push back and say, well, that's not quite what I was saying here. But sometimes you just have to accept that this is the nature of the beast when we do this kind of work. And I've heard plenty of stories from other analysts in other spaces who went into a presentation with their concrete set of recommendations and then walked out with the executives being like, well, we're going to do the exact opposite set of recommendations. And it's just like, well, all right, I'm not going to die on this hill today. So I did what I can do in terms of gathering and presenting the information, but the business is going to move as it does.

43:39Al:And this takes us back to, you know, were they planning on doing this to begin with? Was this all a performative survey? These are the problems you need to solve far, far upstream. By the time you encounter them where we are downstream, when we're presenting the results, it's definitely far too late at that point to try to make corrective action. So the best you can do to try to avoid these situations is have very, very strong survey design, well-written questions that removes ambiguity wherever possible and gets you to a pretty conclusive results across the board, but also know you can do everything right.

44:14Al:You know, you can drive the speed limit, you can buckle up, you still might get into a car crash. Like these things just happen, unfortunately.

44:21Bill Yost:I want to talk about someone who sort of falls between the, I only take what I want to take from the survey, don't care what the results are. And somebody goes, I will accept every result as like, okay, that's what we need to do. Let's talk about the human person, the real person who's in the middle. They get results, they're disappointed with them because maybe they're a founder and there's 30 people. They've built every single one of these up from the ground up. They're disappointed with the results. How should I feel when I get results that don't tally with the way I thought things were around here?

44:51Al:I saw this most often at Google because managers would get reports tied to their name, their organization. And any number of those questions could have a poor result and it can be personally hurtful, right? Nobody wants to hear that they're doing a bad job. Nobody wants to hear from 12 people on their team that they're doing a bad job in a very particular area. I would often say to them, one, you're not going to be perfect in every single area. There's just no way, there are no rock stars among us who are going to have 100 % favorable across the board. If you do, it's because you have a team of three people, and they're all really, really happy people, right?

45:36Al:Eventually, when you get into the 7, 8, 9, 10 number of people who are filling out the survey, those varying lenses that we talked about earlier for the stakeholders and the executives also applies to the employees, And people are going to view the world differently from that angle. Also, some people are just quite harsh graders, right? A neutral to them, a three on the one to five point scale might actually be pretty good. But to you as the manager, it's going to be like, that sucks. That's a shitty score, that three. I want a four or a five. You know, I want that favorable point. And so what we would tell them is focus on the people who did respond neutral.

46:15Al:You know, not literally focus on who they are. But look at that as opportunity area, right? Neutral means you're fence sitters. They can probably be swayed into becoming favorable next time around with very little change. Unfavorable, meaning they said strongly disagree or disagree to the question, that's going to be a little harder to bridge the gap on. But if you try to put the optimist lens over top of it, if your percent favorable is, let's say, 33 % and you're not thrilled with that, your percent neutral is 33 % and your percent unfavorable is 33%, you could shift just a small number of people and next cycle you have a 66%, right?

46:59Al:Because you moved that whole neutral bucket over to favorable. Now two-thirds of people are happy. like that's significantly better, a significantly better result than where you started. So look at it as an opportunity to learn, grow and do better. And then also on the survey administration side, a well-written survey is going to have resources for all of these questions. Like we should never give a question that leaves somebody completely high and dry, right? If the question is, I receive valuable feedback from my manager and that's low, right? There should ideally be some sort of from a curated guide from HR that's like how to facilitate meaningful feedback conversations, the cadence at which you should do it.

47:47Al:Like there should be a roadmap in front of you to be able to try to drive that score higher. Hopefully a research driven roadmap that says like we've seen when managers do X, Y, and Z, this score will raise by some percentage. And then that gives somebody a tangible path forward to try to address those issues. Because ultimately, if a manager is coming and saying, oh my God, these scores suck, I'm heartbroken, it means deep down they do want to do better, right? The willingness to change is absolutely there. So a well-designed survey will enable that for them, right? We're not here to just get bad scores and then say, oh, you're out of here, right?

48:27Al:Continuous improvement is the strategy. That's the mandate here. So yeah, I would say don't take it too personally. Look at your neutrals as an area where you can improve and lean on what the company is giving you to try to improve those particular scores. And if the company is not giving you anything, I think you as the manager should be empowered to say, I need more help here. You know, you can't just tell me this is bad, fix it. That's not productive for anybody. And I think most survey teams would look at you and go, you're right. That's exactly, that's totally true. We do need better guidance here.

49:03Al:Action plans are what we would typically call those in the survey jargon world.

49:07Bill Yost:There are so many platforms out there that in terms of HR that say they do engagement surveys. Some are clearly just a play for AR, MRR, AR. Some actually seem decent. Is the one out there that you think is pretty good?

49:22Al:I would say vendor assessments are very important if you're a large company and you need to, let's say, onboard a new survey provider. There are, as you say, lots of names in this space. Qualtrics, Coltramp, Perceptics, Medallia, to name a few, right, are all dabbling in the employee engagement world. The reality is you could facilitate an entire employee engagement survey through a Google form if you really saw fit. So I would say you need to meet your company's needs and objectives with the platform at hand and try to find the right fit. I definitely have my favorite and I have my least favorite.

50:04Al:I will not drop them on this pod because I don't want to throw anybody under the bus. But it's really, again, your mileage may vary. What are you trying to solve for? I know if you look at market share, I think Qualtrics has quite a bit of it. And it's because they've been around for a really long time. They have a very robust solution offering. But that's not to say that any of the 20 or 30 other vendors in the space might not be the right fit for you because there's the cost element to this. There's the number of people you're trying to administer the survey to. There's the analytical value out of the box that you need from the survey vendor.

50:41Al:because what we often did at Google and what we do somewhat at Netflix now is we take the results back from the vendor, bring them into our internal data warehouse and do our analysis and our reporting inside the box of what we own because there are lots of good reasons for us to do that. But if you don't have a mature business intelligence function that's ready to take on that data, you might want to look at something that's more sophisticated because the cost benefit there is going to be a different value proposition.

51:09Bill Yost:Bill, people are going to want to hear more from you. Out of all of our guests, you're going to be in the top five people who want to follow you. I'm guessing LinkedIn is the best place, is it?

51:18Al:LinkedIn is definitely the best place. You can search Bill Yost and it's the only one with a pirate flag in there. And also, if you're in the United States and you want some cookies, you can go to www.cookiesworthsharon.com. And what's our acronym here? I was thinking truth lies in work. Let's say code TLW will get you 15 % off cookies for the next, let's say, two months after this show release.

51:50Leanne:That was Bill Yost, people analytics lead at Netflix, former Google Geist architect, and quite possibly the only person who can make server design genuinely riveting, apart from me, of course.

52:01Bill Yost:Of course.

52:02Leanne:Three takeaways for leaders. First, only ask questions you're willing to act on. if you can't give everyone a pay rise don't ask if they think they're paid enough laser focus your survey on things actually within your control otherwise you're making withdrawals from a

52:18Bill Yost:goodwill account you can't afford to drain second close the feedback loop loudly and clearly when google survey flagged a structural problem they built an entire new function to fix it and told everyone that was why it doesn't matter if you hire 25 or 25 000 people you do need to say we heard you and here's what's changed. That's what gets you the 90 % response rate.

52:38Leanne:And third, confidential beats anonymous. Nearly every well-run survey is confidential, not truly anonymous. And that's a feature, not a flaw. It's the only way to find out that, say, your new hires hate the benefits package or that one team is quietly miserable while the rest of the company is fine.

52:57Bill Yost:You can find Bill on LinkedIn. Search for Bill Yost. That's Y-O-S-T. Look for the Pyroflag. And And if you're in the US and want 15 % off some delicious cookies, head over to cookiesworthsharon.com and use the code TLW for a discount.

53:11Leanne:How much of a discount, Al?

53:13Bill Yost:That we will find out when we order our own.

53:15Leanne:My notes say 15%.

53:17Bill Yost:Bill, sorry if it's less than 15, but now it's 15%.

53:21Leanne:This is Truth, Lies & Work. We will see you next week with cookies.