Dr. Ed Dobbles | Research and Analytics in an AI World

18:02CMO Confidential:Let's talk about deterministic systems, probabilistic systems, and also synthetic panels. Just talk about all three of those things. And then we'll talk how our users should be thinking about this stuff.

18:17Dr. Ed Dobbles:So, deterministic is like a calculator. You should be able to punch in two times two and get four every time. Probabilistic is like asking your smart coworker, you know, well, what do you think is going on in the marketplace? Yeah, this. You ask them tomorrow? Yeah, it's not going to get the same answer. You're going to get something else. The best analytics people that you've ever met, that I've ever met, are a combination of both those. You can consistently get the core answers out of people, but they also can take context and continue to change your answer. The AI system of the future is going to need to do that as well.

18:58Dr. Ed Dobbles:But you have to build that logic into the system, and that's something that you're going to have to develop. So here's an example. My AI, because I don't trust my AI to consistently get the right answer, I put a sentinel question in every time. So you may not have ever heard of Malort. Malort is Chicago-based spirit. It tastes nastier than Jägermeister. It's only on menus in Chicago. It's on about 6%, 7 % menus in Chicago, and it's on 0 % anywhere else. So whenever I have my AI analyst pull data in the alcohol category, I say, check this data point. Make sure you can do it. Because if you pull that data correctly, I know you're probably on the right tab.

19:45Dr. Ed Dobbles:Or if I'm really nervous, I'll have two AIs both pull the data separately and I'll compare answers. that's using deterministic and making sure you build deterministic into your system the

19:57CMO Confidential:probabilistic side of things in deterministic if malort i guess i'm saying it right is sold yep and i'm sure malort owes us some royalties for putting them on the show but uh but that's deterministic because it's either sold or not sold that you find it or you don't on the menu

20:16Dr. Ed Dobbles:are not probabilistic no probabilistic is like your smart friend it's uh what's going to drive the business which uh which menu should malort be on chicago but probably not the fancy place in chicago probably the closest to the university uh would be a better place for you to be uh and and so you have to build that into the system as well and that's where you could get a different answer each time. And that still is where humans are better today. Your mind is better than the best AI currently at putting these data points together and saying what you should do with it. But the AIs are catching up.

20:58Dr. Ed Dobbles:They're getting better all the time. And what about synthetic panels? Synthetic panels, I got to tell you, I believe in them. But what I always say about them is, All models, this is a quote from somebody, it's not mine, all models are wrong, some are useful. Synthetic panels are definitely wrong, but they're also models and sometimes they're useful. So back to the example, if you're building a panel that is pre-fed, your custom LLM, your custom GPT with all of your information, this is the choosy mom artificial panel or the tech and entertainment enthusiast panel, you can use that to brainstorm.

21:38Dr. Ed Dobbles:You can say, okay, I know what you know about this consumer. You can put yourself in that mind of the consumer. Let's brainstorm back and forth. I do that all the time before a pitch. I did it before this podcast, How to Think About It. That's useful. When you make the leap into saying, well, instead of just using it as a way to help refine my thinking, I'm going to make it in place of deterministic data. I can just ask the panel. That's a big no for me. That's where it fails.

22:11CMO Confidential:And how do you know if you are training your panel or whatever, if you have bad data going in or you don't have complete data going in, are you by definition building bias or omissions or errors in that panel? And how do you know? Because what you have is you don't have any sound input. You think you do, but you don't. And then you get these answers. give our give give us some help for how do you know you're building this right okay the answer

22:48Dr. Ed Dobbles:is uh this is why i used to have hair and i don't have hair anymore uh data always kills you you always every place you go uh the data is always uh you know a mess it's like somebody's garage

23:01CMO Confidential:and they've never cleaned out yes never have i gone into company everyone says you know we have pristine data and we love it.

23:07Dr. Ed Dobbles:Yeah, it doesn't happen. And so there's a couple things you have to do. You have to find, just like the Mallard example, you have to find some things that are consistently correct and build those into the system. Either your analyst needs to know, these are the things. At H &R Block, for example, if I ever saw numbers that were pointing up, I was always questioning those because we're in a long term.

23:32CMO Confidential:business was going down. And a bunch of states were giving you free tax stuff under a certain income level. Yeah, for Diageo. That would be the marketplace.

23:41Dr. Ed Dobbles:If you don't see great crown royal on the menu in Texas, something's wrong with your data. And so you have to build that type of stuff into the system. And your great analysts will know that. And you can build that into an AI as well. You have to work that into the system to get there.

23:58CMO Confidential:But you are making a case that If I don't have a human in here somewhere between the decision maker and the data, and I have anything wrong, I could make a whole bunch of mistakes.

24:11Dr. Ed Dobbles:You could. Here's what I'm saying. So I know that I've tried to do this for 30 years. I know most of these fail. I am still, and here's the shocker, I am still an evangelist for this. I believe this next wave is going to do it. Now, maybe I'm like Lucy and Charlie Brown. and I'm going to get the ball pulled away from me again. But I believe this one is going to do it because it fits where people are. I want to talk to my data. I want to talk to an analyst. And I think you could build a system that starts out with, I'm going to give you the basics, and at some point I'm going to say, I'm going to have the system tell you, you should go talk to a human being at this point.

24:50Dr. Ed Dobbles:This is more than I can answer for you. And you can build those guardrails into the system.

24:54CMO Confidential:So we got a bunch of people out there that are probably under enormous pressure from their boards or their executive teams or themselves just to make huge progress here. How should they think about building their research and analytics teams for the future? Give us best practices and biggest mistakes people could be making right now. Yep.

25:18Dr. Ed Dobbles:So the pressure everyone's going to feel is the pressure that I have lived over the last year and a half. I went from having a 20-person team to having me. And I'm still getting a ton of stuff done because I'm using AI. People are going to want that, and they're going to want that in their system. uh they the people who are going to do this best are going to be a couple different people first off they're going to be people who focus on the foundation how do you clean up the data to the point where you can trust it to live on its own to and when you say foundation you say

25:52CMO Confidential:i am looking at my database and thinking it is as cleaned up as i can make it

25:58Dr. Ed Dobbles:or as reasonably cleaned up as you can make it it's never going to be perfect but how can you make it reasonably clean on the metrics that matter the most? And the reason that's important is because the biggest skill that I've had as an analyst is not statistics or storytelling. It's triage. We had a friend who we worked with. He used to say 10 times nothing was nothing. And every time a small brand or a small geography asked me a question, I was like, yeah, that might be a big deal for Fargo, but 10 times nothing is still nothing. I'm not going to spend time out. But if you could string together enough of those wins because you knew the data was good enough and you can unleash the power of Fargo or your minor brands, that's where people are going to win.

26:40Dr. Ed Dobbles:So in the short term, that's what you see in the job market right now. You see people spending a ton of time saying, how are you going to build those generative BI solutions that do that? The next thing that's going to happen about that is there's going to be a transition for analytics teams and marketers, CMOs need to know this. If your analytics team is built on a foundation of I give you access to data, those guys are about to be out of a job because the data, the AI system is going to eventually get to a level where it's just as good as you. It's just like coding because there is a right answer and you can say, did I get the right answer or not?

27:16Dr. Ed Dobbles:Eventually, you're going to trust that more than you trust having 10 or 20 or 30 people on your team. So you got to be ready for that transition, which means while you are building the generative BI solution that allows that, your analytics team, the leader that you have to be as a CMO, has to say, what are you going to be next? Are we going to give that money back to the CFO that we're going to go from a 20-person analytics team to a 10 or 5-person analytics team that just supports this function, this tool? Or are we going to invest in higher value things? So things that our AI can't do very well today.

27:54Dr. Ed Dobbles:The next level analytics, the next level of thinking that's on there, the more of those probabilistic things versus deterministic things.

28:04CMO Confidential:So it is probabilistic thing. The next level thing would be, I'm creating agents that can talk to me and brainstorm with me. Is that the next thing?

28:14Dr. Ed Dobbles:That one can do it, although we can do that right now. you know there's all sorts of things you can do to get with the next tool I think about it as if you can dream it today you can build it and you know think about all the times we've spent all the time you spent trying to educate organizations on the business what if you built an automated podcast and just showed up in people's

28:42CMO Confidential:feed that's what we're doing right now yeah we're doing right now but I can take my AI can take this

28:47Dr. Ed Dobbles:job too and replicate it. And so you can do some things like that that make things better. So, but I think where the future beyond generative BI is for analytics is that deeper research, the predictive analytics. I had a guy on my team at Diageo who was literally a rocket scientist, had a PhD in astrophysics, and he was answering questions like what was going on in Florida. That's not a good use of his time. You know what a good use of his time was? When we built a predictive analytics, a custom forecast for every liquor store, bar, restaurant in America for every one of Diageo's products. So we knew what to sell into the Morton's in FIDI versus the Mike's Grill in rural Ohio.

29:34Dr. Ed Dobbles:Those are where you're going to spend your money. So if you can take the lower value stuff and give it to AI and spend your higher value time on that, then your analytics team is going to get bigger or different or at least full ground as opposed to being eaten alive by your CFO.

29:50CMO Confidential:And when I look at my data, who should look at my data and how do I know I've cleaned up my data to an adequate level? I mean, other than outcomes where there'll be judgment on the outcomes, but like, how do I know if I'm not a data person or, you know, I'm a CMO or director or CFO, I look at the data and I say, this is cleaned up or not? Yeah.

30:15Dr. Ed Dobbles:So the way I think about it is twofold. First off, you start with a narrow set of metrics you can trust. You don't try to boil the ocean. I'm going to only look at Nielsen data for the first wave of this. And I know Nielsen data is inherently clean to begin with. We've used Nielsen data consistently. The next level of that is our sales data. And we know our sales people. So if you're using data sets that are consistent, all you're doing is slicing it. Those are the ones that you can trust. And then you pilot them with some people who flag it as, it doesn't look right. Something looks wrong about that.

30:49Dr. Ed Dobbles:Who spot something that fails the Malort test or something else.

30:53CMO Confidential:Well, this is a journey versus a single effort. And then before we get to our traditional last question, how of fame worst practices you see in the market today when you look at research and analytics?

31:11Dr. Ed Dobbles:Oh, yeah. So the thing that drives me nuts is, you know, companies that talk a great game about, I want to build analytics for the future. I want predictive analytics. And then all of their work is data polls.

31:34Dr. Ed Dobbles:There's a dissonance between I want to dream that I can do what Uber does in surge pricing and reality what I'm actually going to use the business for. As a CMO, you need to know this is what we're going to do and this is what we're not going to do. You need to set those boundaries. And that's where the biggest mistake is. And it drives me nuts. Like, don't hire somebody under false pretenses or don't have them build a function that says you're going to split the atom if all you're going to do is, you know, move things.

32:05CMO Confidential:But this says, as a hiring manager, you have to have, or a CMO or whatever position we're talking about, you have to have some decent working knowledge of how this all works. Or you will ask for stuff that you can't actually produce. Is that right? Yeah.

32:19Dr. Ed Dobbles:Or you have to have a good leader to work with to say, okay, we're going to put some boundaries on this. This is where we're going to go and where we're not going to go because you can't do it all.

32:28CMO Confidential:Well, that's a good one. That brings us to our traditional last question. Practical advice we have not yet discussed and or the funniest story you can share on the air. You can pick one or both, but you must pick at least one.

32:42Dr. Ed Dobbles:Okay, I'm going to give you two funny stories. Okay, wow.

32:46CMO Confidential:Double header, all right?

32:47Dr. Ed Dobbles:Double header. I just have to give you a funny story to attempt. Funny story is this. I don't know if you remember this, but Best Buy used to have a racquetball court. Oh, yeah. I got hit in the nose. That's exactly it. So you used to routinely kill me. Like, not hit me. You would beat me 15-2, 15-4. I would trash talk like I was winning, but I was consistently losing. And the only game that I ever came close to beating you, I still didn't, is you took a racquetball to the nose, and it took you off your game.

33:22CMO Confidential:My glasses. I had to go into the next meeting, tape glasses, and start the meeting by going, all right, get it out of your system.

33:31Dr. Ed Dobbles:Cheat to win. Injure your opponent, and you win. I don't know. That's a bad moral, but it's my favorite Mike story. The practical advice for people. So here's what I know about analytics people, and here's what I know about CMOs. The meeting that they hate is the governance meeting. Nobody says, oh, great, the governance meeting is next on my calendar. I can't wait to talk about that. I've been working with AI nonstop for a year and a half. I'm pretty good at this shit. And one Friday night when I was working on stuff and I was tired, I've been working on it long. I just wanted to end the day and I wanted it to run overnight.

34:10I gave a sloppy command to Claude and I woke up the next morning with a message in my email.

34:18Dr. Ed Dobbles:Your Claude account was funded and I had like 10 of them. And I ended up spending$1 ,500 on something that I gave bad instructions with no boundaries to Claude, and it did exactly what I told it to do and spent a decent chunk of money on it. So – It's a good thing it didn't have access to all your investments. It's a good thing it didn't have access to my 401k to do whatever it was going to do. And so this is why you actually need to think about this, because imagine if that happened on a corporate scale versus the Dobbles.ai scale. Some real cash could be spent. And so I hate the governance meeting as much as everyone else.

35:02Dr. Ed Dobbles:But there's a real practical reason why you got to pay attention to it. So the tip is actually set the boundaries, like build the things like your spending limits. So now all my AIs can spend no more than$500 on it. So build that into your system or you will get burned. All right.

35:21CMO Confidential:I think a great way to end the show. Thanks for joining us, Ed. And thanks to everyone for listening to CMO Confidential. If you're enjoying the show, please like, share, and subscribe. New episodes drop every Tuesday on Spotify, Apple, and YouTube. And our catalog gives you access to nearly 170 shows, including Colonel Mustard in the study with the job spec, how poor design shortened CMO lifespans, the AI marketing battle, a view from the front lines, dissecting compensation, a primer on understanding and negotiating pay. And what does social first mean? And is it right for you? Hey, all you marketers, stay safe out there.

36:01CMO Confidential:This is Mike Linton signing off for CMO Confidential. Thank you.