Dr. Joel Shapiro | Northwestern | The Grocery Prediction Case - It's Not Just About the Data

0:00The CMO Confidential Podcast is a proud member of the I Hear Everything Podcast Network. Looking to launch or scale your podcast? I Hear Everything delivers podcast production, growth, and monetization solutions that transform your words into profit. Ready to give your brand a voice? Then visit IHearEverything.com. Welcome to CMO Confidential, the podcast that takes you inside the drama, decisions, and choices that go with being the head of marketing. Hosted by five-time CMO Mike Linton. Welcome marketers, advertisers, and those who love them, the Chief Marketing Officer Confidential. CMO Confidential is a program that takes you inside the drama, the decisions, and the politics that go with being the head of marketing at any company in what is one of the most scrutinized jobs in the executive suite.

0:52I'm Mike Linton, the former Chief Marketing Officer of Best Buy, eBay, Farmers Insurance, and Ancestry.com, here today with my guest, Dr. Joel Shapiro. Today's topic, the Eurogrocer case. It's not just about the data. Now, Joel is a professor at the Kellogg School of Business at Northwestern, where he has taught decision science for nearly 11 years after stint as an associate dean. Prior to that, he was a doctoral fellow at Rand Corporation. And to top it off, he also has a degree in law. Welcome, Joel. We could talk to you about almost anything. Yeah, you could talk to me about pretty much anything.

1:34I don't know if I didn't have any good answers, though. And the data stuff, I'm pretty good. Well, that's good. Let's go into the data stuff. Before we dig into this case, let's talk about, ground our listeners in what you teach and why. Like, tell us about economics and decision sciences. Yeah, sure. So I'm happy to. So my background is really is like a data guy in public policy. So I did my PhD in policy analysis because I was always really interested in social systems and social science and how we can use data to make people's lives better. And that's kind of what I do, except I made a bit of a transition about 12 years ago into business.

2:16And, you know, for me, business moves fast. Businesses are accountable. When I was doing work in policy, it was too easy to write a report and it sits on a shelf and, you know, policy and government kind of moves slowly. So I found that a little frustrating. So really what I do now is I teach students how to use data and data science and analytics to make better decisions in business. And we move fast and make quick decisions and test things. And hopefully they're innovating along the way. Which is, you know, I was going to ask you about why you didn't go to politics or medicine or something. And the answer to that, which is I want to be teaching people about decisions that make things better at speed.

3:00Is that fair? Yeah. I mean, I think sometimes we get a little bit too enamored with the big home run ideas, which is fine. Like big home run ideas are good. The issue that I have is sometimes data helps us make little better decisions. And over time, those aggregate pretty quickly. The problem with home run ideas is sometimes they work and sometimes they don't, and they can take a long time. But I love the fact when data helps us make little decisions better on a daily, hourly, weekly basis, because those things, as I said, they can aggregate pretty quickly. And I like that pace. And, you know, when you actually measure things, you can see that improvement over time, which helps give creables to this whole thing.

3:40I think that, well, one, it's a lot like a 401k. You put in a little bit of money every year and suddenly there it is. And I think this is very apropos to marketing. And so, you know, when you look at the entire business landscape and you look at this use of data and decision sciences, talk to us about marketing versus other functions, just in general, writ large, about using data to drive decisions. So I think of marketing as one of two functions in a business that is pretty mature on the use of data. I think operations is another one. So, you know, when people are in operations, they tend to think, in my experience, they tend to think like an engineer.

4:24They know how to build strong systems, that sort of everything works really well together and get some reliable output out of it. And marketing to me is really good at using data for social systems, systems that are comprised of people who do measurable things and their measurable outcomes and the use of data to sort of inform what really works. Pretty mature. You know, I was the chief analytics officer at a sales software company. And when I started there, I made this terrible assumption. And I was like, well, I bet salespeople use data really well because marketers use it really well. And in my mind, I was sort of connecting.

4:59That was like the worst assumption I could have made. Marketing, really mature. Sales, not so much. And other parts of the business sort of vary as well. And as part of that, before we get to the year-grosser case, is part of that because the way the data is structured and absorbed by the Salesforce and the marketer, because if I'm sitting there, I have all this CRM data, all my search data, my digital data, and the Salesforce sometimes doesn't have that data. Is that the reason or is there another reason why sales, why you made the wrong assumption on sales? I think it has more to do with the fact that people in sales pride themselves on knowing how to work with people.

5:36And they've simply been more resistant to input from, you know, some sort of artless algorithm, you know, something like that. I think it's more that people are a little bit more defiant in sales and say, no machine is going to tell me how to sell. I know how to sell. I've been doing this for a long time. And for whatever reason, you know, marketing measured early. They came up with really good ways of measuring success and attribution and running experiments. And I wish I had I wish I had cracked a nut on why sales hasn't done as much. But I think it's just sort of this. I know how to sell. Don't let the data tell me.

6:10Don't tell me how to do it. Talk to me about those heartless, nefarious algorithms. Those are for the marketers. Keep them in the back room. All right. So let's sketch out the case for the listeners, the Eurogrocer case. And Eurogrocer is a pseudonym for a real company. Even if you're Sherlock Holmes, you probably can't take a guess. But Joel, let's start with the business problem and then the data science solution. Yeah, yeah. So I'm happy to. So Eurogrocer, as you said, it's a pseudonym for a real company. Not surprisingly, it's a European grocery store chain and super interesting case. They teach with it a lot and it sounds sort of mundane on its face, I think, but there's so many layers to it that teach us some really interesting things.

6:54So here's the summary of the case. Eurogrocer does about$8 billion a year in sales and they had about a$250 million problem. And that problem was that they were ordering the wrong amount of stuff for their shelves. Sometimes they would have too much and it would spoil and sometimes they wouldn't have enough. And both of those are problematic, right? You don't have enough, you forego profit. You have too much, you bought things that you didn't need to buy. And we also know that the grocery margin is, you know, I recall when I was in retail, you know, if you could make over a two, you were killing it.

7:30Yeah. And that's 2%. That's a 2%. So you're killing it. So 250 million on 8 billion. That is a lot, a lot of money. Tell me if I'm getting that wrong, but that's my recollection of grocery. No, that's exactly right. And it's so funny you say that because I get some people when I teach this case and I'm like, it's a$250 million problem. And they'll be like, yeah, but they're$8 billion. It's only like 3%. So why did they even care? I'm like, well, where I come from,$250 million is a pretty big number. So they care, right? So that's the problem. They're not ordering the right amount of stuff. Okay.

8:13And this is a great data problem because if you could predict demand, you could arguably order the right amount of stuff. Yeah. Right. And so what they do is they hire a data science consultancy to come in, a consultant firm to come in and, hey, can we actually predict demand? So that's sort of the the context here. They bring in this consultant to try and predict demand and they have a really high quality data science consultancy. And the very first thing that they do is they run up a little pilot test. Not everything is predictable. Like don't spend a lot of money and resources, hiring somebody to do this until you test it out first.

8:52Cause it's not predictable. You don't want to be, you know, all in with a big price tag. Turns out it's pretty predictable. Demand is pretty predictable. Not perfectly. So nobody's ever going to make the claim that you can know exactly how much to have on which shelves and blah, blah, blah. But it's also, it's also just, I want to make sure everyone's getting this right. Cause I would say it's grocery. So the market isn't like wildly expanding and contracting for something. It's kind of, you know, you got your population around your stores, you're delivering all this stuff. You probably have an awful lot of your own first party data right here.

9:26So for me is a, is a great data feed for, for somebody. Yeah, exactly. So what they do is they try and predict the demand of different product categories at each one of these stores. How much stuff are people going to buy in bakery at this one store and dairy over here and meat and seafood and so forth? And that's sort of the level at which they do the prediction. Can we predict category demand? I mean, like 25 categories per store. Can we predict category demand at each one of these stores? So they do a little pilot test, and it turns out, as I said, they couldn't do a perfect prediction. There's no way they're going to solve this whole$250 million.

10:07But the pilot test shows that they can get 42 % of the way there. That's real money. That's real money. I mean, you're sitting over a 1 % profit gain. Yeah, it's$106 million improvement. that this pilot test suggests. If you roll out this prediction more broadly, you're looking at$106 million, okay? Next thing that you want to do as a business leader, if somebody says to you, you can do something that has$106 million benefit, I'm pretty sure your next question is, how much is it going to cost me? Yes, but then generally, how much is it going to cost me under the assumption that we're not going to have to build something super new?

10:53How fast could we do it? Yeah, yeah. Yeah. So really good question, too. Capabilities were pretty much right there. They felt good about building it fast. And the price tag that they put on it was about$16 million. And I just want to pause on that for a second. So$106 million benefit for a$16 million investment. And this is annual. So$106 million is an annuity and the$16 million is a one-time cost. $16 million is going to be an annuity too. All right. So you're sitting there at a pretty big gain. Yeah. I mean, there's a couple of million dollars of upfront costs, but if you look at it yearly, it's about$16 million.

11:39Now, do the math on that. You can do the math. It's a six and a half times return. Yeah. So it's been based out in like two months. I mean, it's the biggest no-brainer ever. Now, keep in mind, this was a pilot test, and we're talking about extrapolating these results to the broader enterprise. But if I told you that you could get$6.50 back for every dollar that you spend, that's going to get your attention. You know that's going to get your attention, right? And what's interesting about this story is that we could stop right here. We could be like, data, it's amazing, and analytics, and we could all raise our hands in victory and be excited about it.

12:18But that's not the way the story goes. So the Eurogrocer board looks at this and they consider this$16 million a year investment and they say, thanks, but no thanks. We're going to pass. What? You're going to pass on it? Why would you possibly pass on an investment like that? And the reasons why they passed on it are where this turns fascinating and where there are a lot of lessons to be learned. And I can share those with you unless you want to. Oh, no. No, we want to hear because this is where the data science consulting company is tearing his hair out and saying, this is one, I can't believe we're not going to do this.

13:00Two, this was a case study we're going to share from stages all over the world about how great we are at consulting. Yeah. We're not going to do it. So tell our list why it didn't get done. Yeah. So there are really three big things here that I'll highlight for you. So one of them is a little bit of the misalignment with the results of the data model with the realities of the business. And what I mean by that is you can tell a grocery store that I can predict exactly how many avocados people are going to want to buy next week and the week after, and you should make sure to order different amounts.

13:38But if you walked into some supplier contracts, or if you've got some logistics processes that don't allow you to have that flexibility, you can't take full advantage of that, you know, predictive accuracy. Right. Accuracy. They sometimes refer to this as being a model-driven organization. You want to have your organization set up so that you can use the results of a quantitative model. That's really important to being successful with data. And in this one way, Eurogrocer couldn't really adapt to the accuracy and the flexibility and the sort of periodic changes in those predictions. So that was one big issue.

14:16You with me? Yeah. I mean, that says I know the answer, but I just can't do it. Yeah. I need to get 100 avocados for store A and 700 for store B. But since I can't actually get that done, I'm going to get 400 and send 400 to each store. Yeah. But I'm going to get 800 and send 400 to each store. Yeah. And, you know, there's a cost to that, right? It's nothing more than just additional cost. But now the board is sort of like, huh, man, we can't really do that. So maybe this isn't quite as good as it looks. The second thing that they have a problem with is they charge for the data science team and they're like, okay, you're telling us that, you know, we need this many avocados here and this many here.

14:56Why is that? Oof. So that's a challenge for people who work with data models. When we build out data models, oftentimes, not always, but oftentimes there's a trade-off between accuracy of models and transparency of models. So what they did here, the data science consultancy here built what's called a neural network. You might have heard of it. It's a really complicated, technically advanced way of building a predictive model. And it's really good in accuracy and it's really bad in transparency. And when somebody who's in a position of decision-making authority says to you, explain to me why we should do this, when your only response is, because the model tells me to, oof, that's a challenge.

15:43That is, because you'd be saying, look, why is this one store, sticking with the avocado strategy, going, or example, why are they selling seven times more avocados than this other store that's only 70 miles away? Yeah. Or 20 kilometers away since we're doing a European grocery. Yeah. Yeah, so, you know, when people ask you why and you can't answer why, sometimes that can be a problem. And this is always an issue with people in data. Are we building models to be maximally useful or transparent? And when those two things are intention, one of them has to give, and it's not always clear which one of them it should be.

16:26Okay. You said there was a third thing, I think. Did we get them all out? No, we didn't. So the third is so fascinating, and it just speaks to such a broader sort of area and concern that I have with data. When the store managers were brought in on this, they saw that little parts of the data didn't quite look right to them. It was like, oh, this shows that orange juice was on sale at this year. I've been a store manager here for 20 years and we've never put orange juice on sale. I don't think your data's right. Oh, it turns out that there were some errors in the data. So the data science consultancy team goes back to it and they're like, oh, we'll fix the errors and so forth.

17:04And they fixed the errors and they reran the analysis and it all looked about the same. The errors were very minimal. But once people lose faith or trust in the data, it's really hard to get it back. And now all of a sudden, you're in a position of, you told me the results the first time and the data was wrong. How do I know whether I can believe you now? That notion of people not trusting the data, I find fascinating and important and one of the sort of least understood or remedied things that we need to talk about in the data space. And those three things together, not being able to use the data well, the lack of transparency in the model and lack of trust in the data ended up killing it.

17:45And the board at Eurogrocer said,$16 million is still a lot of money and we're going to do other things instead now. Oh, I have to do a couple of follow-ons on this one. The first is, having been in a lot of companies, sometimes the culture, it just wants to resist this. And if it's resisting, it will find any reason, like the data or the avocado difference, and it will just keep pecking away, pecking away at this until it finds a reason to not accept it. And there's other companies that are more on the forefront of, oh, my God, we know we got to do something that will say they'll see the 100 and over 100 million dollar savings and they'll go, all right, there might be some errors in this, but we got to go get it.

18:34So let's overcome the errors. How much of this is culture and how much of this is the pure presentation of the model? I know you can't actually split this out, but the pure presentation of the model by the data people. Yeah, it's a really good question. As you know, I don't really know the answer to this. I mean, I will say that any organization that's serious enough to bring in a top flight data science consulting firm to do this, I think at some point the culture's got to be there. I mean, it's possible that someone, you know, okays it and then they're not involved in the decision making. So all apart.

19:09But I think those are the two issues. And I don't know which one is which in this case. Okay. We're going to talk about what people should do when they're in this kind of position, how to make sure it gets through the company. But when you look at AI now and the answers AI is producing, and then you want the transparency of AI, which to me looks like it's really challenging to figure out any transparency. How does this speak to companies taking all the AI information they get and then absorbing it? Because it sounds like this grocer would say, well, I can't possibly see anything in this giant LLM model.

19:48I'm not going to do anything. What do you think is going to happen on that front? Well, I mean, I think one of the nice things about LLMs is that if you review what an LLM tells you, a large language model tells you that you should do something, I think we're still at the point where a lot of people don't sort of just blindly follow it. Mike, that is something to you that, you know, this might be a good idea or you should go execute this strategy. And I think people, you know, still tend to be responsible and accountable. And I think that there's this there's this accountability that keeps people from doing too many crazy things with LLN.

20:26So I think we're in an OK shape with that. When it comes to actually taking like quantity, when we talk about AI, like predictive modeling, there's a lot of really poor understanding of how these kinds of quantitative outputs can be used. And so, you know, as things get automated, LLMs get more and more dangerous because they stop having a layer of human sort of review on them. So that concerns me. But, you know, the marketing person who uses an LLM to sort of help them come up with a strategy, the one who's like actually reading the output and being like, it's a good idea. And do I have evidence that supports that?

21:03That feels like a still very human thing to me. So I'm heartened by that. So as long as AI has some adult supervision, you would feel comfortable. Let's go back to the avocado analogy. You're a grocer. What are you teaching your students about other than these three points you made? What are some best practices our listeners can take away when they are when they're trying to push data through the company? Because I think you mentioned the difference between data leadership and data science. Give our listeners some tips. So I actually talk a lot about data leadership. I just taught a new course on data leadership at Kellogg this last year, which is really interesting.

21:48Sort of the, I have like the seven big things that we talk about, but sort of most relevant to what you're talking about here is building trust is just incredibly important. And, you know, one of the most important things about building trust is expectations up front. And so if we just talk about like something in the Eurogrocer case with the errors in the data, one of the most important things that I do with anybody at the beginning of a data project is tell them that there will be errors. Data is always imperfect. It is always imperfect. And, you know, I don't think people lose trust when mistakes are made.

22:23I think people lose trust when no one talks about the mistakes. And more than anything else, I tell people that if you're going to be successful with data, you got to set the right expectations and you got to be transparent with things when they are imperfect. And that's just that's just how to be influential, I think, generally and how to get people to trust you generally. Super important with data. Got it. Any other tips you would throw down before we move on to the next topic? No, I mean, you know, you always want to know your audience and you want to make sure that you know what they care about.

22:56And I work with a lot of data teams on improving their communication. And because they're not always the best communicators, as you could probably imagine, people who are really good with data. And the two things I always tell them are, you know, know exactly what the purpose of your communication is. Are you trying to sell somebody something? Are you trying to make yourself look like the smartest person in the room? where you're trying to influence a decision to know what you're trying to do and know what your audience cares about. And if you're really purposeful about that, a lot of communication issues can be solved.

23:27I like it. We did a show with DJ Patel, a big data scientist, and he said, you need a Spock on the bridge, but Spock needs to be able to talk to the crew and the crew needs to be able to talk to Spock before anything happens. Yeah, yeah. And so I think that's really good. I know you have another case or maybe two that deal with asymmetrical risk. So let's talk about asymmetrical risk for our listeners and then give us the thumbnail and then we'll talk about case one and maybe case two. Yeah, sure. So, you know, I told you before that my background is in public policy analysis. And that's right.

24:10Sort of earned my stripes, so to speak, as an academic around more public policy issues before I shifted to more business stuff. And so anytime I see something that's more sort of in the policy space, my, you know, my radar always goes up and I'm interested. And so there's a super interesting and totally, totally sort of different context here than the Eurogrocer case a number of years ago. So the state of Illinois, their Department of Child and Family Services had started being more data driven. And what they had done was they had started using a predictive model that was intended to help them predict kids at risk of bad outcomes, specifically around like being abused or even perhaps being abused so badly that they might even die.

24:55It was a risk tool, a risk detection tool that was supposed to help their caseworkers find kids who are more at risk than they might otherwise realize, so they could deploy resources to the kids most in need of these resources. That's sort of the overall thing. It worked really well in other jurisdictions, but in Illinois, it completely tanks. And the reason it tanks is for two things in particular. Number one, the model starts predicting that lots and lots and lots of kids, thousands of kids are in like imminent danger. Yeah. Didn't seem realistic to anybody. But at the same time, there were these two toddlers.

25:35If you live in the Chicago area, like during the like two 2010s, early 2010s, something like that. Two toddlers, tragic incidents of kids who are actually killed by their abusers. Just the worst possible case you can imagine. And when you go back to the predictive model, that model actually predicted that both of those kids were safe. It didn't have them as high risk according to their prediction. And so the head of this department, the DCFS at the Department of Child and Family Services in the state of Illinois, looks at this thing and she goes, it's predicting too much risk for some kids. It's predicting that these two kids weren't at risk and they ended up in these tragic circumstances.

26:18We're getting rid of the tool. It's not really helping us. and it's such a misread and a misopportunity of how data can be helpful to us and i'm happy to sort of wear that the asymmetry right and and the asymmetry there is look a false there's a false positive and a false negative but the consequences of one brings one terrible outcomes to the kids But also, you know, the PR and the entire questioning of the model blows this up. So that's the asymmetry. Give us lessons learned from this thing. And what should you be doing if you're dealing with a super asymmetrical risk? Yeah. So, look, I mean, when we start talking about data and doing predictions and stuff, sometimes you're right.