Kristin Wozniak | Too Much Data & Not Enough Thinking - Are You Following Goodhart's Law?

0:00Kristin Wozniak:The CMO Confidential Podcast is a proud member of the I Hear Everything Podcast Network. Looking to launch or scale your podcast? I Hear Everything delivers podcast production, growth, and monetization solutions that transform your words into profit. Ready to give your brand a voice? Then visit IHearEverything.com. Welcome to CMO Confidential, the podcast that takes you inside the drama, decisions, and choices that go with being the head of marketing. Hosted by five-time CMO Mike Linton.

0:41Welcome marketers, advertisers, and those who love them to Chief Marketing Officer Confidential. CMO Confidential is a program that takes you inside the drama, the decisions, and the politics that go with being the head of marketing at any company in what is one of the most scrutinized jobs in the executive suite. I'm Mike Linton, the former CMO of Best Buy, eBay, Farmers Insurance, and Ancestry.com here today with my guest, Kristen Wozniak. Today's topic, too much data and not enough thinking. Are you following Goodhart's law? Now, Kristen is the chief growth and data officer at Cosmo 5, a company that helps brands combine human and machine factors to win the point of selection with consumers.

1:26Previously, she served in analytic and insight roles at Cassette Media, Impact Research, and Wavemaker, and she's here today to chat about data obsession, Goodhart's Law, and the unhealthy focus on acquisition. Welcome, Kristen. Thanks for having me. All right, let's jump right in. First question, even though your career is steeped in data, You believe many leaders have an unhealthy obsession with data. Tell us about that. I think it's because my career has been steeped in data that I think people have an unhealthy relationship with it. You're like a recovering data addict. Okay. A little bit.

2:07I'm recovering. I don't know if I'm in it still. Yeah, absolutely. I mean, I think when I started my career all those many years ago, data and measurement was not nearly as embedded into the culture as it is today. I mean, partly just because it was 20 years ago and we didn't have as much. Now I feel like every conversation in every moment and every decision has to be fueled by data or it's wrong. I watch CMOs and marketers at all levels struggle, you know, wanting to make a decision and feeling like they have too much data or not enough, never feeling confident enough to make a decision without data, questioning their instincts.

2:51And I think fundamentally what the obsession looks like to me is that, you know, at its best, data should be used to empower curiosity. And I think that we've moved into territory right now. Data is powering certainty or the illusion of certainty, which are very different things. and I think lead us into tricky territory when it comes to innovation and forward thinking. And you've probably seen a lot of instances where people say, well, let's go get just a little more data. And a bunch of times when, even though the answer seems to be relatively clear, there's that one little doubt that sends people back to the well.

3:31Do you have any good examples of that where people have just dragged stuff out or made an error there? I mean it's funny I was thinking about this I'm like oh goodness what examples what examples do I bring to the table because there are so many and I think any individual example on its own never feels catastrophic it's when you put them all together that it starts to tell a complicated pattern my best example and I get it all the time and it's sort of that that moment where I want to just pull my hair out and say like do you hear do you hear what you're saying where I'll have a marketer or whomever come to me and say, it's so important that we do first to market innovation for a category.

4:12We want to be the first, first to market. And then you'll bring them a first to market opportunity and they'll say, I can't really do this though until you can prove to me that it works. And I go, but I can't though. And that to me is like the crux of the data problem is here we are like mired in so much information and data that we've lost our ability to like think about anything interesting or innovative until we can truly prove that it's got value. Or, you know, the offshoot of that is here's benchmarks or here's how it's worked for other companies. And it's, well, I'm a snowflake. That doesn't apply to me.

4:48Or that benchmark's not the same as mine. Or my company operates differently. So it's like wanting all the data and then not wanting to believe it and needing it to make a decision, but not trusting it. And we're just in this constant back and forth. Well, there is that innovation issue. I used to argue a lot, But if you wanted the return on Ben Franklin flying the kite for electricity, you would never fly the kite. Because someone would say kites are expensive. You don't know if it's going to work. What are you going to do with electricity? And you would actually never do it. And my thing is, just go fly the kite.

5:22Just try it. Try it. Especially if it's not that expensive or not that involved. Just go try it. Tell us more about when you, what are the symptoms of data obsession? And then you also have, there's friction in managing data tools, I think you said. Tell us about this. What are the symptoms of this? So I'll start there. You said it yourself. One of the biggest flags for me is when the answer to a question is we should go buy another tool, get another data set. That sort of obsession with like, I can't actually make a decision here. I know, let's add another tool to the toolkit. And that is a mistake most of the time.

6:07And we're starting to see data, ironically, data prove the fact that it is a mistake. If you think about the cognitive load that goes along with monitoring, say, 25 different dashboards or data inputs, we're moving to a place now where the cognitive load that's required to actually manage and sort of circle the square of all those data sets is becoming so heavy that you have no cognitive space left for the real human. which is judgment, strategy building, creative ideas. And now with AI coming into the fold, they're calling it AI brain fry, which is monitoring multiple AI systems is actually creating physical, negative physical side effects for marketers.

6:57So things like chronic fatigue, eye strain, headaches, mental fog, inability to make decisions, everything about it feels like you have what did someone describe it as like 12 browsers open in your mind at all times and you just can't manage it so that brain is like killing us and then if you have all these data sources inevitably some data will say this is not a great idea because if you just keep especially if it's not if it's a new or relatively unproven idea how do you decide how many data sources you need and how do you manage the ones you have? And how do you know we are overloaded? Like what are the symptoms you see here?

7:41You can always find data to prove yourself right. And this is a wildly problematic situation we built for ourselves. I see this on award applications all the time. So if you're submitting an award for, you know, a media innovation award or something like that, People will write up a case and then just before they submit it, they go, oh, I forgot to input the insight. So they go up and find a whole bunch of data to justify the thing that they already did that they felt successful about. When that starts to happen, we've gone to a bad place. You know, instinctively, you feel like this is the right thing to do.

8:20You've done it. There's good results. and until you can find the data to justify that it was the right thing to do from the very beginning you can't enter it in a submission or talk to it with any kind of confidence because people go did you really did you really have done that like did you have data to make that call in the first place i feel like we ask a lot of questions around why we should do something and that to me leads us to data fatigue or data selection bias, confirmation bias, when we should be asking, why shouldn't we do something? So if you have a great idea, ask yourself, why wouldn't I do it?

8:58And use data to prove that instead. Yeah. So I hear you say, look, if you think it's right, when in doubt, go do it. But there's also kind of the data killers, I would think, where they will find the one piece of data where you got 20 good pieces of data to say, we should do this. Someone will show up in the meeting with the one piece of data that says it might not work and gum up the whole works. Do you see that symptom as well in this? Yeah, of course. And this is where, sorry, go ahead. And when you see it, how do you combat that where you get the right balance on data you need versus data you have?

9:37I mean, this is where our conversation and curiosity is critical. And like, let's be real for a moment. Sometimes that person walks in a room with that piece of data and they are hell bent, not on finding a good decision, but defending a platform. And if you're a person who walks into a room and their mission is to defend a platform or a position, it's game over. It's really hard to navigate past that because that person will be loud and fierce and relentless. And that to me is not so much a data concern as it is a collaboration teamwork. How do you actually create an environment where different opinions can be heard?

10:16So that happens a lot. And those are those moments where you kind of have to decide whether you want to fight or you have to just sort of let it be and fight the next fight. Most of the time, though, with that data point, it's a great opportunity to have an important conversation. So if you have 19 things pointing, saying, going right, and you have one data point going left. This is a great opportunity to say, okay, is this noise or is there something here that signals a new opportunity, something we haven't missed or something we've missed, something that maybe is interesting, but not for us, like dissect that a little bit because that outlier is there for a reason.

10:56And it might just be noise, but it might signal something new. So the conversation is the key, but if that data point is a defense or an offense in the meeting room, it's done. And then how about when the person with the negative data point is your peer or your CFO or even your CEO, do you have any tips for managing that? Because really what this is, is an innovation stopper because if you don't do something, you don't get any results. And if you do the old thing, you're not going to get any new results, but some people don't want to take any risk at all? Do you have any tips for moving them? I mean, there's a million things to do.

11:36And look, I've worked at companies before where that mentality is the mentality that drives the entire company forward. And if that is the mentality from the entire C-suite team, you're going to have a hard time. It's going to be a terrible job. You're going to have a terrible time. Chances are you'll last a year. You'll do your best to try to move the needle. Eventually you'll leave because this is not for me and I can't make a difference here. And that happens. And that is the symptom of the data obsession right there. Because that group of people or that mentality is, for me, a mark of insecurity in your job and insecurity in your decision making.

12:15If you need to so fiercely defend yourself with a piece of data, it's like you're hiding behind it, right? Instead of actually thinking about what impact it could have, how you could grow the business, hearing somebody else's opinion. And that's an indicator of a whole bunch of challenges in a culture, I find. In terms of how to actually combat it in the room, sometimes I find it's simply asking questions in a different way to incite a different conversation. So like I said, instead of asking why should we do something, it's why not. So you've brought this piece of data in. Thank you for this new piece of information.

12:50it tells us that we shouldn't do this. Great. But why also should we not do your thing? Let's argue both sides. Questions not around, how do I put it? Questions more about what is that piece of data telling us? Exploratory questions, that is it. But fundamentally, that is a patience, a collaboration, and a recognizing if you're cooked. Well, and I also think there's an art in defending what you want to do where it's got to be a bigger idea than a piece of data so that you can you can still get it to market so you have some new stuff out there and um and then sometimes you can trade with the company like if in x weeks this isn't working i'll kill it but let's go and see um because i think oh sorry i totally cut you off no that's all right, this is just a chat.

13:45So you're not cutting me off. So go ahead. I was going to say that's also a great opportunity to set up the idea of an experimentation roadmap. If you're putting on their productive hat here, right? Which is great idea. Love this new piece of data. Let's try that. Let's put together a hypothesis in terms of what we think that could benefit or how that could benefit us. Let's test it. Let's document it. And let's do another thing at the same time that we test and document. That's another way around these things. It's not a no necessarily. It's okay. Let's try it. Let's go in with an idea of what we're trying it for.

14:20Let's map what works. Let's write it down and let's make sure we don't lose sight of it. Because sometimes that data points the right one. Sometimes that liars is the thing you should follow. I totally agree with this. The other thing, often by teams, we would come in with more than one idea because it's easier to block one idea than it is to block a couple. And then if they're all in the same kind of camp, you can trade them off, but you want to get something through. And you have to be ready for the blockers. I want to flip over to Goodhart's law that you talked about. What is Goodhart's law?

14:57So it's an economic principle established in the 70s. Fundamentally, what it says is the second a measure becomes a target, it's no longer a good measure. And often it's described in the context of another silly sort of concept called the cobra effect. So imagine there's a town that has a cobra problem and they say, I know how we get rid of it. We're going to put a bounty on the head of every cobra. So our KPI is you bring in a cobra, we give you$10, we reduce the cobra population. goodhart's law says but wait what actually is going to happen on the side is you're going to have a collection of people set up cobra farms exactly breed more cobras kill them bring them in and make a fortune so your kpi has actually done the exact opposite of what you intended it to do so the idea of how many cobras are brought in as bounty is no longer a good measure of your cobra population or a reduction of it um this is like marketing in a nutshell most of the metrics that we look at, I would argue, are very much mired in the challenge of Goodhart's Law.

16:03So is this an example of Goodhart's Law gone awry, would be, I pay you for traffic, not for conversion. So you will bring 10 gazillion people to the business or to the store or to the site, and none of them will buy, but you will get a super bonus because you brought a lot of traffic there? You've touched on so many things in that. So I'm going to rewind to the very first part. Let's pretend it's leads. You're paid immediately. So your goal is to bring a hundred leads in in a month. The intention of that goal or that KPI is that you then go back as a marketer and think about what do I have to change in my strategy, how we show up, how we speak to customers to bring in leads to grow the business.

16:49But in the realities of our world with short timelines, no budgets, constant sort of breath down your neck from someone wanting to see results ASAP, what marketers often end up doing is not actually thinking about how they can improve the system to grow leads, but actually either game the data or game the process. So for example, I could stand on a corner of the clipboard and have a whole lot of people write their name down and I come back and go here are your leads and I gave you leads I gamed the system like these are not good leads or I gained the data so I changed the definition of what a lead is so a lead maybe is supposed to be email plus intention plus a phone number plus a follow-up date and I've changed the definition now where it's just an email and I also hit my goal so often we gain the system or the data instead of actually addressing the systematic problems or challenges to generate growth because it's easier.

17:48And to your point, made worse by the fact that we've so often connected those goals to people's personal incentives and compensation. Yeah, one of my advices is almost the first thing I do is I look at comp because people will work to get paid. And often they are optimizing how they get paid, not the company. And that's the company's power. But that's how the company has done itself a disservice by trying to granularly pay everybody in a way it gets sales profit and retention because people only have to care about their measure. And so that's Goodhart's Law and Crazy where there's cobras everywhere.

18:33Yeah. And like, and then what's worse than that for me is then the senior teams are surprised when people just want to focus on leads. It's like, well, but what about the rest of the business? It's like, well, you're, you've decided that my compensation is exclusively this. Why are you surprised when I don't want to focus on any other part of the consumer journey or the business? um it's we do it all over the place in marketing and i said at the beginning or i said it in an earlier call today i forget which is that you know our consumers don't see your brand in these little bits and pieces like a consumer doesn't look at your brand and go wow they have a great lead strategy but i really hate their social approach like they just experience your brand and yet we compensate and measure and um compensate measure and and sort of conduct our business based on these individualized KPIs.

19:24It's a complete muck up. Well, and then the business meeting starts out, I love the cobra analogy, which is we've killed 10 gazillion cobras, but we're still losing people to cobra bites. We have to kill more cobras. And that's how the business goes sideways and you don't make any money. But that takes somebody to look outside on that and look at the big picture. One of the things we talked about pre-show was, you know, you've got Goodhart's Law, which is focused on usually one single thing. When the business really cares about customers' money, profit, and, you know, sales. One of the big things there is retention.

20:05And there's so many measures there where people pay off. NPS, they pay off this. Why doesn't retention matter more? Because it's so complicated. It does matter, I think. And we all inherently as marketers know that it does. But we are working in a system by which we optimize toward individual outputs instead of optimizing towards that bigger goal. So although we see that it matters, of course it does. But our entire decision-making infrastructure is organized around these individual inputs. So even though we know it's there, the idea of being able to take this big idea and action on it in any kind of way that, you know, looks nice in the QBR, that hits your CFO's goals, that you can defend to the board, none of those things exist as part of that system in the way it's designed right now.

21:03So of course - That drives me nuts because on the retention front, the data is overwhelming. That retention is one of the best ways to drive profitable sales. but all these like good hearts law and comp and everything are in the way in that yeah this is where i feel like people just override the data any thoughts as to why that is sure um short-termism and i mean i know we're so tired of talking about it everyone says short-termism i think part of the challenge is to the short-termism yes so if you do a great search strategy and you get a whole bunch of, you know, clicks, everyone celebrates great.

21:43Retention is a longer play. I remember, I think about this a lot and it's such a weird thing. I love the Toronto zoo. So bear with me for a second. I was talking to the tortoise and the turtle keeper at the zoo one year. And she told me that there's tons of research being done on turtles and tortoises, but it often gets abandoned because the turtles outlive the researchers. So the researchers grants disappear and then no one picks it up. So they're never actually really sure about the longevity and the lifetime and what really goes into a turtle being so healthy for 300 years. I feel like in some ways that applies here.

22:18Like a retention is not a two-year CMO term or even a five years. It's a long, and you have to be able to pass the baton to the next team who wants to then keep the same goal moving. And we don't have the infrastructure for that. So I think that's part of it. I also think, unfortunately, retention is a moving target. So one of the things I talk to folks about a lot is that we focus so much on outputs and not nearly enough on inputs. Understanding what truly drives a metric forward or success forward is where our effort needs to be. And those things can be great keyword strategies or a strong creative.

22:58It can also be economic factors out in the real world, cultural narratives, the weather, a whole bunch of things that are beyond our control. And in marketing, we do not like addressing things that are beyond our control. Well, and retention is usually the whole company has some impact on retention. And since there's no real way to fix it, I think it's also hard for the measures to permeate. But that's one where the data, the end data is overwhelming. So the diagnostic data, maybe Goodhart's, there's too many cobras and turtles, not enough turtles. This is a first time turtle researchers have ever been discussed on the show too.

23:39So if our listeners are sitting there and they're saying, okay, Kristen, tell me, I'm going to look in the mirror and I say, I'm a data obsessed company. Goodhart's law is killing us. What's the cure? Like, is there a 12-step program for people? Or what do you do if you realize I have overdated my company? Well, what's the first thing? Admitting you have a problem is the first step. So, I mean, I guess that's it. So you've already done step one. Yeah. I mean, I think there, I don't think there's a clear 12-step process. I think there's multiple individual pieces that we can all deploy, depending on what works for you, to start moving the needle towards longer or bigger term thinking.

24:26So one of them I think about a lot is, and you brought it up earlier, which is look at your KPIs. If your KPI goes up 20 % and your business stays the same or, in fact, decline, are you celebrating that KPI still? ask yourself that question because sometimes you should celebrate it because if your kpi is something like increase awareness by 20 points and you do that and your business doesn't go up right away i can see that because awareness is a long-term metric but there are some metrics where to your point you come with all these cobras and people are still being killed by cobras like you have a problem your metric is wrong so ask yourself if the metric grows and your business doesn't is that okay?

25:06Like I'd say that's one. Well, and the other question on that is, why are you killing cobras? Is it really a cobra problem or is there a bigger thing you want to achieve? And I think there's probably a bigger thing you want to achieve and cobras are just temporarily in the way. But the problem with that is then you immediately go from like, mine, I'm sort of saying that look at your KPI versus your business as like the dip your toe into course correcting versus the, hey, you've been focusing on all the wrong things because that feels insurmountable. So I'm trying to think about like the, what can you do today to ask yourself or start moving your business in a less data obsessed way?

25:49One of the other things I talk about is buddy metrics. Oh, tell us about buddy metrics. So back to the lead example. Yeah. If you're driving leads and that's your only KPI, great. I can bring you a whole bunch of leads and they can be garbage. But what makes that metric a good one? And potentially it's qualified leads, like they actually convert. Yeah, converted leads. Leads that lead to sales makes it a really good one. Exactly. So don't just KPI the lead volume, buddy it up with the extension of that metric, which makes it meaningful. doing those two things together means at least you're having a conversation about the complete picture that you're trying to achieve well i hear you saying this and this is one of the things we've talked about on the show a couple times which is if the metric doesn't in the end impact some financial metric in a direct line eventually it's probably not a really great metric it's a diagnostic.

26:50And you ought to think about what you're paying for, because in the end, if it doesn't translate to some financial metric, your investors aren't going to be that happy with you, usually. Usually. But that, I mean, therein lies the challenge too, is then how do you then have this conversation with the investors? But again, the investor timeline is usually three to five years. So they don't care about a 10-year, 15-year retention strategy. They want growth to be able to sell and reinvest. So it goes all over the place. One of the other things I suggest to folks is we've talked about incentives, which is review your team's incentives, including your own, to make sure you're not from the very beginning, setting yourself up for single-minded failure, essentially.

27:39But one of the other pieces is I think most of us probably can think about this person if we took a moment. But typically an organization has one or two people who are not the ones who are achieving all those individual KPIs. They're not the ones bringing in the leads. They're not the ones who are great at optimizing a campaign. They're not the one running out and glad handing at events or whatever to make those connections. But there's that person who sort of sits in the center of your organization, who's a great mentor, who can connect the dots between strategies, who can think creatively outside the box, who spots risk, who is willing to share and ask questions and kind of be the productive pain in the ass in the room in a lot of ways, that person is critical to getting you out of a good heart's loss cycle.

28:32Find that person, stop worrying about whether or not they're hitting their lead goals, and ask them to start evaluating your business holistically and looking for gaps, looking for opportunities. That person - Judgment and judgment, perspective, and wisdom are harder to measure the data. And that's, I think, what you're saying. What should our listeners be thinking about from a data perspective in 2027? What would you tell them to be ready to do next year? So I actually, it's such a funny thing. My husband was talking last night. He was reading an article in The Athletic about basketball analytics and basketball strategy.