Aneesh Dhawan | The AI Trust Gap in Research and What to Do About It

0:00The CMO Confidential Podcast is a proud member of the I Hear Everything Podcast Network. Looking to launch or scale your podcast? I Hear Everything delivers podcast production, growth, and monetization solutions that transform your words into profit. Ready to give your brand a voice? Then visit IHeareEverything.com.

0:22Aneesh Dhawan:Welcome to CMO Confidential, the podcast that takes you inside the drama, decisions, and choices that go with being the head of marketing. Hosted by five-time CMO Mike Linton.

0:41Welcome marketers, advertisers, and those who love them to Chief Marketing Officer Confidential. CMO Confidential is a program that takes you inside the drama, the decisions, and the politics that go with being the head of marketing at any company in what is one of the most scrutinized jobs in the executive suite. I'm Mike Linton, the former CMO of Best Buy, eBay, Farmers Insurance, and Ancestry.com here today with my guest, Anish Dewan. Today's topic, the AI trust gap in research and what to do about it. Now, Anish is the co-founder and CEO of NIT. NIT pairs forward deployed researchers with autonomous research agents to deliver decision-ready insights at speed.

1:27Previously, he founded PurePix, a purpose-driven creator marketing company, and NIT recently fielded an AI Trust Index Survey. That was designed to measure the state of AI trust, rigor, and governance in market research. We're going to talk about that and a few other things. Welcome, Anish. Hey, Mike. Thanks for having me. Great to have you. First question. Tell us about the research business in the age of AI. What are you seeing and is research really at the front of AI or is it actually a laggard? Yeah, I think it's definitely in the front of AI here, which is honestly a little surprising to me because I think as an industry, you know, research hasn't, the way you've been running research really hasn't changed for the last decade or so.

2:17But let me, you know, as we looked into the research that we just did on research, what we found was that almost every single one of the 150 research experts we were talking to at the primarily enterprise brands in the US was already using AI. So I think it's an industry that's ready to adopt and embrace AI. But as we'll get to later in today's show, and as we got into our research study that we ran, there's a lot of concerns and considerations on how best to use it. We'll get the concerns in one minute, but I want to say, are people using, are researchers using AI to save money or to do better research or both?

2:54It's both. I want to say, you know, six, nine months ago when we really started the industry embracing it, it was really around more time efficiencies. But I think the conversation is really shifting to doing it better. And look, research is such a great use case for AI. It is so analysis heavy, synthesis heavy. It makes it a perfect, perfect kind of a ground for or use case for using AI to speed up the process. But I think the biggest thing now is like, is it actually driving impact? And that's really what we got into within this study. Well, let's tell us, tell us about the study. Why did you even run this in the first place?

3:32I mean, you're sitting around, you think, you know what we should do? We should do an AI study with 150 enterprise companies. Sounds like super fun. Tell us why you did it and what kind of were the big takeaways? Nice. So we're all researchers ourselves. So we actually love this stuff. It is so fun. But look, at the end of the day, there's a lot of questions going around the industry right now on how best to use it, how best to measure impact of AI within research. And for the stakeholders who are ultimately consuming the research, the CMOs of the world, the product leaders of the world, they have these questions top of mind as well.

4:04So we wanted to get answers. And we know no better way to do it than to run research on it. So we went out. We talked to. It was a total of 154 researchers, primarily from enterprise customers or enterprise brands doing over a billion dollars in revenue, all in the US. And like I was saying, Mike, what we found was pretty awesome. Like, I mean, not only is everyone using research, over 90 % of the people we talked to said that research is actually making them, AI is actually making them better at research. But we found some really interesting insights around their biggest fears. And it's not being replaced by AI.

4:44It's actually how AI is being used by non-researchers to make research decisions. We must talk about the greatest fears. The greatest fears of researchers on AI. All right, let's hear them. Awesome. Well, look, we were kind of surprised by this as well. When I was going to conferences a year ago, the big talking point was, how is AI going to replace the researcher's job? especially junior researchers, people just coming out of college or early in their career. That has really shifted in our industry. So we actually asked researchers, we said, what are your biggest fears? And we collected a lot of qualitative data around this.

5:23And by far, the number one fear was AI being used to run research without the research rigor, without the governance. So decisions being made directly from AI outputs. And that was actually, people were twice as more scared of that research being used or those insights being used than they actually were about them losing their job to AI. And I think that's really telling when, you know, a year ago, that was kind of the most talked about, oh my goodness, we're going to lose our job to AI to now, look, AI is a great tool and we're going to use it. But there's a lot of downsides to how it's being leveraged without the rigor behind it.

5:59Let me make sure I have this fear right. This is like I, as anybody in the company, and I ask research to do something for me, or I do my own research using AI, and I get, I think you call them unedited outputs from AI. And so my biggest fear is that management or some decision maker is going to take this and screw up. Yeah. Is that right? I think that's a great way to frame it, right? And we can talk a little bit about why I think that's actually a pretty big problem, especially if you're looking at who we talk to, right? These are research leaders at large enterprise organizations. Right, where there's a lot of discipline usually around research in a large enterprise.

6:45It's not like you're just at some startup trying to figure it out. You have usually a well-worn path for research. Exactly. So tell us how this manifests itself. Yeah, absolutely. And one thing I would add, Mike, is like you were one of the stakeholders of research, too. Like it's driving pretty big decisions. Oh, yeah. Look, it was huge. I love research. I even have a degree in research. So, yeah. OK, here we go. Yeah. So, look, what we found was it comes down to kind of three things that people wish there was more governance around. The first thing was around business context, right? one of the big reasons people were worried about how those AI insights were being leveraged was if those insights were generated without the AI actually having context on the business, it might miss the mark.

7:38I'm sure we've all experienced this. You're chatting with Claude, chatting with ChatGPT. Sometimes the output you kind of look at and you're like, kind of have to squint to see what it's saying and how to use that. It's because it's missing all that context, all that context that we have sitting in our heads, especially the research experts that have sitting in their heads. The second piece of the governance that they really were looking for was around transparency. And by transparency, we mean like, where is that data coming from? You know, a lot of this is a lot of times this is kind of brought up in the same realm of topics around hallucinations, right?

8:11Where AI is just making up an insight or coming up with an insight without actually being able to pinpoint where it's coming from. And then lastly, it was around the judgment. Like I always think about research as part art, part science. I think AI can do a really good job of executing against the more scientific aspects of research, but it's that judgment, that art that makes research really magical, that pulls out that unique insight that can really drive a decision. And AI just doesn't have that. And so this is the area where there was a desire for more governance. If we can get more context, more transparency, and more judgment in that process, researchers would feel a lot better about how those insights are being used.

8:51Two questions. I'll ask them both in order. One, is this different than researchers sharing the raw data earlier when you just, you know, people would run a study and then you'd ask for all the raw data? And then secondly, is governance and judgment saying, I want control of the outputs here? Or when they say governance, what do they actually mean? So two questions. Two questions. Okay. Yeah. So to your first question on raw data versus unedited output, that's a really good question. And the way I think about that is with the raw data, data in itself is harmless, right? You still have to do a lot of work to go from raw data into an insight that drives decision.

9:34And that's where the magic of research happens, right? Taking all this data, analyzing it, identifying the insight and going from there. What happens with AI outputs and especially unedited AI outputs is it's already taking you to that conclusion. So all that messy work where the magic is happening is now being done in sort of a black box in a lot of times. And that's where you can start getting into trouble, right? Where if it doesn't have the right context, if you don't have the right transparency, if that judgment isn't baked in, you might end up with an insight that at the surface looks pretty good and reasonable.

10:09But when you start digging into how we got there, then you start seeing that kind of fall apart. Can you give me an example, a hypothetical example of this, perhaps? Yeah. So look, there's a lot of great examples that we've seen from our customers, which I'll try to think through. You can just call them like, instead of naming the company, you can just say an automaker or something like that? Yeah. So like, okay, one thing is we do a lot of work in private equity, for example, right? So like a lot of the times, like what AI will end up doing is it will give you an insights that it kind of groups to the mean, right?

10:51So like that insight might be really interesting and it might be what most people are saying, but it's actually not what you want to do. It's kind of in the more long tail of the insights that are coming out that are more interesting that you might help that actually might drive a decision. So it averages everything down to an AI average instead of a customer segment. Exactly. And that's where the magic happens, right? So whether you're making a big investment decision or you're running a big creative campaign, and that's where the human judgment comes in as well on like, what is the true context of the business, the research objectives here that we want to move forward with?

11:27And so that's an area where we've seen this pop up a lot when you're using AI tools without some of those guardrails baked in. All right. Now, all you researchers out there, Anish has used the word magic like 100 times. So, you know, you can share this all with your research friends. Hey, we also discussed earlier the concept of budget, accountability, and AI slop in research. Tell us about that and tell us how you know you're getting slop versus real research when you're using AI. Yeah, yeah, that's a good question. So look, the way we've seen a lot of enterprise brands adopt AI, and this is not just for research, is you kind of see this.

12:16Initially, it started off with just heavy on the usage, right? You probably heard the word token maxing being thrown around. It's like use it for usage's sake. which I get the value of that, right? It kind of forces someone to, you know, change behaviors, right? If you really focus on the usage. But I think in that process, we've forgotten a lot or have missed, like not measured correctly, what's the actual input or the impact to the business, because usage for the sake of usage is not what we're trying to go for. So what we see happen in research, for example, is you use all these research tools, you bring them in, but the output they give you isn't actually better than what a human would have done.

12:56And a lot of people will be like, great, that's fine. At least it's a lot faster. But when you actually measure the true amount of time that goes into it, that's not really always the case. So an example of what we found in the research that we did was that we found that on average, a researcher was spending almost another three to five business days, you know, about 10 hours across three to five business days to then take that AI output and turn it into something that was decision ready. And when you start doing the math on that, the ROI on time savings there is not as large as you thought, right?

13:31Like where, hey, it's taking me down from three weeks down to three days. You know, you're not actually saving that much time if it's like almost a week and a half, two weeks now to get to the same point that you were with human led. And so I think that's where we're starting to see the shift from, hey, just use AI for the sake of using AI to more, hey, is this AI tool that you're actually using helping you get to a better spot and truly saving you time when you incorporate all that stuff that has to go into getting it decision ready? So this translation layer, or I just named it the translation layer between AI and now I'm giving it to decision makers.

14:09Is that a permanent thing or will AI and the researchers get a lot better at that transition layer so they can go a lot faster using AI? I think we will, which otherwise I wouldn't be doing what I'm doing today. And look, I think that if you think about it as kind of the role of the researcher, right? And I'll just use a metaphor of a chef, right? We don't want to go from the researcher being a line cook to a fully automated kitchen with robots, right? We really want to elevate that role of the researcher to the head chef. And I think that's the ultimate model that I think will work. The head chef is the one building out the recipe, tasting the food, making sure it's good before it goes out to the customer, making sure there's quality ingredients coming in, and then the AI is executing it.

15:00And I think that's the ultimate model where you get the best of both worlds. So you get the speed of AI, but you also get that output that comes out of it that's truly driving impact, both from a time, cost efficiencies, and also the new types of research it can help you unlock. So in this research world, and I agree with where you're going with it, but in essence, the chef can, you do a lot more research here at speed and the translation layer gets taken care of itself better and better. How far away are we from research utopia? I think we're really close. I think we're really close and what it requires is just a new way of thinking about what is the role of the human and what is the role of the AI agent or agents, right?

15:49Historically, like I was saying a year ago, it was, hey, is AI going to replace the role of the researcher? I think we're all starting to come to the conclusion that that's not the case, at least that complex, nuanced enterprise research. And then how do you design a system where you can drive that context, where you can drive it with that level of transparency and embed it with that judgment? And I think the technology is there. I think it's just about designing those systems and implementing those systems at the largest enterprise brands in the world. So I think it's here. We just have to go out and really diffuse it now.

16:25All right. You heard it here first, research utopia. So if I'm sitting here as a potential, you know, I want to hire a vendor, I want to become that chef. And that means I'm going to dish a lot of this out to agents or and I'm maybe going to have a vendor do that. And it's not going to be you guys. It's just whoever. What are some things our listeners should look for to get the most out of the vendor they pick? Yeah, we talk about this a lot. I think it comes down to those three things. And the questions I would ask is, is this AI platform tool, AI vendor built to understand my business context, right?

17:07They should be able to explain how it is using your business context to do this. Is this platform built with traceability, with transparency in mind? This can be at the feature level primarily, right? So you can pinpoint exactly where is this insight coming from. And ultimately - This means, this transparency thing means I can go all the way back to source material. Exactly, yeah. You should be able to go from the highest level insight that's been generated all the way down to exactly where in the transcript, where in the data do we actually see that? And that should be seamless. Our product philosophy is one click.

17:43So you can see exactly where the data is coming from. And then lastly is, where is the human judgment in this? And I think human judgment should show up in two areas. One is how is the product built, right? Like at the prompt level, right? Are there researchers working with the engineers to actually build this out in a rigorous way, in an expert way, but also at the project level, right? Every single project you run, do you still have that quality control from the head chef before the meal goes out that night? And I think those are the two different areas where I really do feel that at least for complex enterprise research, you should always have judgment involved.

18:23Okay. So you also are talking about, and there's a theme that we started on the beginning here, governance. What does good governance look like? And where is everybody now in the governance trail in your mind? Yeah, I think this is evolving a lot as well. So yeah, the way I always think about good governance is - Because governance is one of those things, a lot of people hear governance are like, everyone's going to slow me down with a bunch of rules that are going to just get in the way of us getting to market. Yeah, look, I think it goes back to that metaphor of the head chef, the line cook, or the fully automatic robotic kitchen.

19:01What governance means, for me at least, is making sure that those three things are included, the transparency, the judgment, and the context, but doing in a way that is, you know, I don't think the future of research is going to be democratization of insights. I think it's more the federation of insights. And what I mean by that is you still need to have that expert judgment, the magic that we were talking about, somehow controlling and baking in those best practices, the guidelines, how it's being used, so that ultimately you can actually put these tools in the hands of the stakeholders. And I'm still a really big believer in that.

19:42I want to get to a world where marketers, product people can run research on their own. And I have a POV on what that might look like in the future. But to do it in a way that, you know, the spine of it has been built or the foundation of it has been built with really rigorous and expert research judgment. And they're continuously updating that and refreshing that with the business as a business. How do I know if I have expert research judgment? Because there's very few people out there thinking I have terrible judgment. Almost everybody, if they have a decision-making pen, thinks I'm a good decision maker.

20:20How do I know if I have good judgment or my team has good judgment or when I look at my vendor, they have good judgment? it's a good question um and i think look it when i mean judgment there's like the business judgment which i i believe many stakeholders have that and and know that within their own field and then there's research judgment and i think the best way to actually figure it out is is to is to work with them right and like one of the biggest things is you have to kind of see it to believe it um and it and it is kind of where that art of research that magic of research comes in is like can you nail those insights based on what the business needs, what the research objective needs.

21:04There's one level of just here's the data, but then going from here's the data to here's the story and here's what it means for you is where that judgment comes in. And I think when it comes to that, I think the only way to see that or seeing is believing when it comes to that. Okay. And you said you had a point of view of where this all shakes out in the long run. You want to share that? Yeah, look, I think research is being really, really reimagined right now with what's possible with AI. I think if you go to any large enterprise, Mike, and I know we were chatting a little bit about this when we first met, it's like the majority of decisions today are still not being informed by talking to your customers.

21:45And it's not because people don't want to, it's you just don't have the time, you don't have the money. And some decisions maybe just aren't worth going and spending hundreds of thousands of dollars on. I think the way the world is going is, you know, AI makes you, it gives you the ability to run research faster and gives you the ability to run research cheaper and research better. And that's opening up a whole new type of, or a whole new world of the decisions that you can now inform with research. I think the other kind of next frontier of this is what's possible with synthetic research. And that's an area we've been spending a lot of time and effort around, which is these large enterprise brands all have a lot of data on their own synthetic panel you can make exactly they have everyone has so much data on who their customers are it's really how do you pull the right signal from that data and how do you use that to project or predict where the customer is going and so i think the the headline of that is the the time it takes to get to a decision that's informed by a customer is shrinking to almost instantaneously in a world where there is synthetic or simulated data.

22:51And that just opens up a whole new world of where you can run research and what decisions you can inform with your customers. And we're still figuring that out. And I think it's going to take a couple of years to figure out what those workflows look like and where you actually incorporate that. We actually did a show with Ed Dobbles, who was a research leader. And he said, it's always been about data and making better decisions. and it's never kind of got to where it should be, but I think it's this time it's going to. So, um, I, you know, we all hope you're right. Um, anything else from that study that where you're like, I didn't see that coming, uh, conclusion or factoid that popped out of the study.

23:34Yeah. Yeah. I just really want to emphasize how quickly the space is moving and how quickly people are kind of like, like how people's kind of point of view is being shaped. One of the things I mentioned, again, goes back to like that number one fear. I just think that's such an interesting question to ask people, what are you, what are you excited about? And what are you worried about? And seeing that flip over the last like, six to nine months, or nine to 12 months from like, a fear of replacement to now a fear of being kind of almost bad use. Yeah, yeah. Yeah, or like misuse. It's really, really interesting.

24:11And I think gives us a line of sight into where the market is at from like, what is this thing and how do we use it to, oh my goodness, AI is an incredible, incredible tool if used properly. And that's kind of, I think where the state of research is today. And are you planning to run this survey constantly? That is our plan. Yeah, we want to kind of track how these POVs and reactions to AI are evolving as people are continuing to operate. So a year from now, you can come back and we can talk about the evolution of research. There we go. Let's do it. Part two. Let's talk about the trust gap. If our listeners are out there and they're thinking, okay, do I have a trust gap?

24:56As you talked about a governance gap, how do they investigate how they're progressing on this if they're not in the research group? Yeah. Well, first of all, and especially for the folks listening to this podcast who probably are more on the stakeholder side than the researcher side, I do think there is a responsibility of setting the tone of, hey, we want speed, yes, but we also want rigor and ideally also efficiencies on cost. So I think it comes from the top on what do we actually what do we want to really hang our hat on? What do we really want to, how do we want to design the system? And I do think you can, for the first time ever, have all three, right?

25:41You can have really fast research, you can have really rigorous research and you can have it at a fraction of the cost of what it was years ago. But I think that requires you to make upfront investments, right? And I think the investments it requires are investing in identifying and documenting that business context. and pushing your partners and pushing your team internally to, you know, not make that trade-off between speed and rigor and, you know, push for traceability, push for transparency in it. And so, like, I think that to me is the biggest takeaway from this is, like, we can have it all, but I think we have to move from a world of, like, token maxing, using whatever you can just for the sake of using it to really being, you know, kind of focused on speed and rigor and cost.

26:32So is it fair to say what you just said is develop really good habits versus go for really fast answers? Exactly. Yeah. Yeah. I think we have to hold ourselves to that standard. And I think that's what's going to make the entire industry better as well. But it is a different way of thinking about it because I do think it also requires a little bit more of an upfront investment in terms of how you're approaching these. Well, and if it works, it'll be happier consumers and happier companies all around because everything should, in essence, be better decisions, right? Exactly. That's the plan. All right.