Three ways AI is reshaping marketing insights

0:09Hello and welcome to The Walk Podcast. My name is David Tiltman and we are back with another episode from the Insight to Impact event live in Denver, Colorado. This is the second in the series brought to you in partnership with Profound and we're interviewing some of the most interesting speakers across the event, including CMOs, researchers, and many more. Quick reminder, Insight to Impact is a new event the team at Lions have introduced this year. We brought together the Market Research, Get Together TMRE, and an event called Content Marketing World. Then we threw in some content from Lions brands, including Contagious, Effie, and of course, War.

0:48Now we've been recording live at the conference, and over the course of this series, we'll be bringing you exclusive interviews with CMOs, groundbreaking new research, and much more. So be sure to tap follow on your favorite podcasting platform to get new episodes as they drop. On today's episode, we're going to dig into a topic that has inevitably dominated many of the sessions and conversations across the stages at Insight to Impact. And that is how AI is reshaping marketing operations. My first guest will be Pam Forbus, Global Chief of Insights and Analytics at Mondelēz International, who, among other things, will tell us how the snacks company is implementing AI within its insights division.

1:31Then Matthew Wohl from Profound joins me to examine some of the practical ways brands can boost their AI exposure and look at the threat that AI might or might not pose to jobs in our industry. Finally, I talked to Tracy Adams from the Advertising Research Foundation and Keith Smith from the Marketing Science Institute to go deep into their new research examining synthetic data and what marketers need to watch out for in AI-generated datasets. But before we get into all that, here's a quick word from our sponsor.

2:08Today's episode is brought to you by Profound. Profound's AI marketer proactively finds the work worth doing, does it, and keeps you in the loop where needed. That's why a third of the Fortune 100 choose Profound. To win AI search, go to tryprofound.com.

2:28Welcome back. Well, today on the stage, our Insight to Impact. Pam Forbus, Chief Insights and Analytics Officer at CPG giant Mondelez, went deep into the future of the marketing industry. And of course, AI was at the center of it. So we wanted to bring her onto the podcast to talk about some of that thinking. So Pam, welcome to The Walk Podcast. Thank you. Appreciate it. Well, let's just start with, I mean, you're talking about the future of the marketing industry, but what is it? Well, it's interesting. One thing that I'm thinking a lot about is the future doesn't happen without leaders leading into the future.

3:09I mean, that's a fact. So anything that's going to happen in the future, and you know it's a lot different from where we were to where we're going to be, it's going to take leaders to lead into that future. So challenging the insights industry today in a few of my speeches around what are we building that's going to lead us into the future? Because I think it could go a couple of ways, right? What are those? Well, I think one road is we can do things a lot faster, a lot cheaper. A lot of what we do can be automated. So are we automating ourselves to save the company a bunch of money? Or are we thinking about different things?

3:47Are we just automating what we already do? Yes. Are we taking AI to help us do what we already do? Or are we going to do something new and different? And this is a sort of classic thing with all sorts of technology innovation, isn't it? You've got a sort of a version which is making existing workflows or existing processes or existing things quicker or cheaper or whatever it might be. And then there's where the bigger payoff is, which is how do you actually completely rethink the way you do things at all to deliver greater impact? Correct. And if you think about a ladder, which we all want to do, it means having a new value proposition to the company.

4:31It means being known for something different because that other thing is going to get automated. It just is. It could be folded into an IT function, for example. Or if we really stand out and stand up, that's a different talent profile even. So I would say insights people tend to be a bit more introverted, tend to be more tentative, tend to do great work and offer a recommendation. But they don't put their name on the line and say, here's what we should go do. And that's what leaders want. And we're going to have to be a lot more brave. So that's the sort of insights functions within companies like Mondelez, your own insight function.

5:15Do you feel that tension internally? Absolutely. Like we have to be more brave and not be waiting to be asked to either answer a big question that hasn't been asked of us. We got to dig in and say, hey, we can be part of that answer. And we can force our way into the forums and tables where we're not maybe at today with really good value add that all of a sudden we're becoming indispensable and we're being asked back and asked back again. so um it's a it's a journey and we got to do it pretty fast though because this is i was going to say i think like conceptually we can all understand that sort of distinction between doing doing the old thing quicker or and doing something completely new but it's how you actually get to that particularly in a in a market where growth is slow uh you know there's pressure on costs and all that sort of thing.

6:16Actually having the internal permission to take that leap may be one of the challenges. Well, we're going to have to do both. And I come from food industry, which is a slow growth, low growth industry. So finding ways to add productivity through lower costs is going to have to be one of our challenges. To me, that's called the adversity advantage, though. That means I have to do more with less and I have to fight really hard for priorities that are important to the company, make sure my team's focused on them. So it's a bit of a transform while you perform. So yes, automate and help have AI do the answers on demand as much as possible to democratize what we already know and help the company understand what we should know.

7:06But then really think about what questions aren't we addressing as an insights function. And a lot of those are not going to come from any one leader. They're the, this is my opinion. Having worked in the business so long, the biggest opportunities are cross-functional. You've got RGM working on pricing and promo and impact price size architecture. You've got an innovation team thinking about the next innovation. You've got marketing thinking about brand media, right? But what is the value of coordinating those things? Just sort of exposing that insights team maybe to a broader range of internal teams.

7:50Absolutely. So, and doing insights work that brings them together. We're the only like function that can be sort of agnostic and let the consumer or the shopper be the arbiter of truth, right? So we can come in with a point of view that might be counter to what the RGM team wants to do around raising prices or doing more promos. Or even, you know, maybe investing media isn't the right thing at this time. We actually do need to get our price size architecture properly set up before we turn on the media. So no one's going to ask us to do that. Got to take the reins. And I think it's a real opportunity.

8:28So that's what I'm thinking about. Do you see that as a vehicle to sort of making insights, giving them more profile within businesses, more stature? Yes. And it can be very scary for insights leaders who have themselves kind of grown up as a support function to marketing or a support function to sales and maybe even finance. But I think with AI, that augmented intelligence of insights, AI is going to help us do that even better. It will help us connect data. We've got to build the data foundations, build the connected AI, and have AI help us connect those dots. Because it's now about asking the right questions versus staying in our silos and surfacing maybe the things that the company's not asking for that's going to bring real value.

9:23Well, all those little questions that take up our day is automated. That's great. But then we become indispensable. And if you can just show little green shoots for that, I think you'll get a lot of traction pretty quickly. And where have you seen specific success at Mondalees? Are there sort of areas where you feel like you've really cracked it or areas where, you know, sort of work in progress? It's definitely early days, right? So we do have several pilots that we're working on called commercial analytics that cuts across those barriers. We are also playing with things like synthetic data that that'll allow us to maybe do things faster with the data, the insights we already have.

10:10Building those synthetic personas or panels on proprietary data, I think, is key. We're learning like we're not there yet, but we're trying to move pretty fast. I think job one is data foundations. We need really strong data foundations that are AI ready. And that's what we're focused on while we build generative AI capabilities to mine that data. And then it's about the talent. How do I equip the team to, it's actually freeing for them to go ask questions that they've never been able to have time for. um ai augmented you know human in the loop type analysis and then i'd say the other occupational hazard we have which i'm probably demonstrating right now is our our inability to be you know succinct and clear and logical and linear that's not necessarily how an insights brain works we're whole brain and we the best ones are whole brain and they they can see the systems and it's have a hard time really articulating, AI is great at that.

11:13I can do a mind dump into an AI tool and it can help me logically get my thinking in a better place. And I'm more equipped to walk in and say, here's how we're going to monetize our demand data. These are the three things we need to go do. I've had AI help me, equip me to speak like that, to be more declarative, to be more brave about my points of view. And you talk about synthetic audiences in there. Where are you finding them most useful? There's a lot of talk. I mean, there's been a lot of talk in TMRE this week about, you know, the rights and wrongs of synthetic. And I think it's a space where everybody is still trying to feel their way a little bit.

11:57What's your take on it? Here's where I think it's really, really good even right now. If you can build a synthetic persona on your own proprietary data, it's actually just a new way for your stakeholder to understand the insights. So many companies have done what they might call a demand space segmentation, right? How do we learn about the demand spaces? PowerPoint. Maybe someone's created a nice little video about that demand space, right? What else can we do? Oh, I've seen immersions done. You can recruit consumers and have a conversation. But think of all the data we have on a demand space, whether it's hard, hardcore, you know, structured data or all of our qualitative, even some social listening.

12:47And I build a persona and now marketing, innovation, R &D, even our general managers can have a one on one chat with a consumer in a demand space, maybe even one that shops Walmart versus Target and have a real conversation. all of a sudden they're starting to really understand the way humans understand each other through conversation it's a different way of democratizing the insights built on your proprietary data so that's about internal buy-in knowledge sharing whatever you might want to call it rather than this has given me a unique insight into the target audience that we never have got otherwise i that's why i said i think there's other use cases but i think this one's pretty pretty darn good already yeah so um and it's people like it so that i think it's going to cause us to be more consumer centric and more thoughtful before we just throw things you know out there for consumers to react to just let's go have a conversation with that synthetic persona or panel and get some feedback and then go talk to real consumers so our hit rates are better um that the real consumers remain important so can it replace the real consumers even claude will tell you we're five to seven years away from that and only if we do the side-by-side calibration work that's required and and i don't know that companies are willing to actually invest in the side-by-side they're that's they're gonna want to go right to the fast cheap solution which right now i think they're saying are about 20 accurate yeah um i mean whereas we're on the show floor there is i mean literally 50 ai companies um here so is it changing the type of vendor you work with the sort of partner you work with or or is if you manage to do a lot of the stuff either in-house or with with the sort of existing sort of partners you already had definitely new emerging partners that are coming on the scene um but most of the current partners are also evolving and adding AI capabilities.

14:48So we're, well, industry's shifting. You know, we're learning to have to do things with third parties inside our firewall. So it's sort of a hybrid. It's in-house outsourced so that we can protect and make sure the AI is doing the right things with our data. So that's what's changed. I think the biggest change for me is working with our IT function. to build these capabilities versus being able to just work with a third party and do whatever I want. Looking ahead, we're obviously deep in planning season for 2027. What are you most excited about next year? Where do you think this is going to go next?

15:29Good question. Well, we're really focused on understanding the broad portfolio of capabilities that this function is in charge of. Which ones do we prioritize to partner with our IT function to really accelerate um ai and it starts with data foundations i know that's not sexy but um you know just building like a true insights person it's got the data right if we don't have the data right it's just crap so um we're thinking very you know what is phase one phase two phase three of the data foundation we need to build i just spent an hour with a partner talking about that and then um how do we quickly parallel almost build an ai agentic capability on top of that data to, again, right now I have the remit to go find all data that we have available so that we can put AI on top of it and get smarter.

16:29So now I'm crossing functional silos. I'm getting into the data that our sales team has, that our supply chain has. How do we build that data foundation that can really help with end-to-end intelligence? And I've been given permission to go figure that out, which is great. That's not an insights role anymore. That is a much broader enterprise role. With that will come the breakdown of silos that have existed forever around functional silos, around sales planning, marketing planning, you know, FP &A planning, and we can actually have an end-to-end conversation about what's right for the consumer.

17:06I'm very excited about that. Wouldn't that be nice? That is a perfect note to end on. Pam, thank you for joining us on The Walk Podcast. My pleasure.

17:17This episode is brought to you by Profound, the AI marketing platform that comes with an AI marketer. A third of the Fortune 500 and brands like Figma, Ramp and Cursor use Profound to control how they show up in the AI search. Go to tryprofound.com.

17:42my next guest is Matthew Woe senior manager for brand and content at Profound now if you haven't heard of Profound they are an AI marketing platform that recently raised 180 million dollars in series d funding giving it an impressive valuation of 1.8 billion dollars Matthew's session here in Denver offered a lot of interesting food for thought for marketers as they adapt to the era of AI. And perhaps more importantly, there are some much needed reasons for optimism. Let's hear that interview now. Matthew, welcome to The Walk Podcast. Thank you so much for having me. So I want to start a little bit with Profound itself, because not everybody will be aware of the company.

18:27but um i mean it's it's a fast growth uh fast growth company in the sort of it's the ai marketing space recently raised 180 million dollar series d funding valuation of 1.8 billion so you know there's a there's a lot of tailwinds uh behind you just just explain very quickly what it is profound does and um and and you know for people who don't understand uh what the company is Absolutely. Yeah. So Profound is an AI marketing platform that brands use to control how they show up in AI answers. So we work with about a third of the Fortune 100 brands like Walmart or Estee Lauder or Comcast, as well as brands like Ramp, Figma, Cursor.

19:12And at a high level, our platform is able to show you what real people are asking AI about your brand and then what the models say back about your brand, the visibility, the accuracy, and the sentiment and relative to competitors as well. And then we have an AI marketer inside of Profound that gives you ideas of what can you do about it. It brings project briefs backed by data that's your brand data. And then you can deploy Profound agents to actually do work and inflect those metrics. Now, this is a hugely interesting space. There's obviously been a lot of discussion about it during the conference this week.

19:48And in particular that either how you might start to influence what shows up in the models uh when somebody uh writes a relevant prompt or um asks a question about a particular category or indeed a particular brand there's a couple of things you talked about earlier in your stage session that really stood out to me i don't know if we just sort of start with those and it was really around the sort of whatever it is the model finds about you or says about you, it can bring in a load of other information that is either incorrect or maybe just about a competitor. So just talk to me a little bit about what you found about those spaces and some of the examples you've shared.

20:33Yeah, absolutely. It's really interesting when you look at the responses on Chachapiti, Claude, Gemini, they won't always give you a direct answer. So for example, the example we use often is you could ask how much is the Nike AlphaFi 3? And instead of just saying, oh, it's $285, ChatGPT will give you a paragraph's worth of information. And in that paragraph, we ran a study actually on 50 ,000 real prompts and conversations and found that 47 % of the AI answer is actually not directly answering the question you asked. It's sort of extra information the model has decided to put in, yeah? Exactly, yeah.

21:15And it's trying to be helpful, but as a marketer, this 40%, 47%, excuse me, is actually what makes our jobs a lot harder because that is where AI will start to opine and editorialize based on information that might be out of your control. So it could be providing inaccurate information about maybe your fees or your positioning relative to competitors, or it could be bringing in negative sentiment towards your brand. And so it's really important to be able to have visibility into that and also be able to control it. You used the phrase, I'd love you to talk a little bit about this, non-commodity content.

21:53What does that mean? Yeah. So when thinking about how you can control how your brand shows up in AI, it's our belief that you have to create content that reflects how AI uses the internet. And there's kind of three pillars to this. You have to create fresh, specific, and original content. So original content is where we start talking about non-commodity content. What that means is ChatGPT only has the information that we provide it. It's already been trained on a ton of information that's on the internet. But when you release a new shoe, for example, it doesn't know how much that shoe weighs. It doesn't know how much spring it has, how durable it is.

22:30ChatGPT can't wear that shoe and run it for a marathon, but you can. So it's kind of your job to be this API between ChatGPT and the real world. And so you have to create content that ChatGPT could not find on its own. You shouldn't be writing about how the shoe is just quote unquote great. You should be saying it weighs X amount of milligrams. It has this this length of durability uh or you know people who wear the shoe and we we found in our lab test that they're able to see xyz benefits or increase some sort of metric uh that ai can grasp onto and is this content when you talk about content in that uh scenario is this content that a human will ever read or is this content that is engineered for for the algorithm that's a great question i actually got this question earlier today from someone as well where it's like sometimes the the content that our marketers are creating for ai are so boring to read you know and you're like what do we do like how could we ever show this to someone but i think this comes into the idea of in today's world you have to market to both humans and agents at the same time it's something we've called the two audience problem with we've we've seen this in in other research as well.

23:47Exactly. Yeah. And I think there's this barbell forming where you're going to have to heavily index on one and the other, and they can certainly talk to each other. But what I would say is, yes, you will be writing blogs at a certain point that are not meant to be consumed by people, but by agents. Because that third pillar, if we go back to it, about creating content that's specific, AI agents are extremely thorough, right? And so you're going to have to add details and specs that a human would never reasonably want to know. But again, it gives AI a concrete data point to start building a comparison between you and potentially competitors.

24:23If you just stick to buzzwords like best in class or highly rated, there's no real way to create a benchmark for LLMs to then say, oh yeah, I can confidently say and advocate for this brand and say that they're better. Where should that concern be? Is that on owned channels or are you trying to place this in lots of different places all over the web? Yeah. So what we say is you always want to nail your home field advantage. So on your website, you have the most control. So these kinds of details should absolutely be in your blogs and especially pages like your product description pages, if you are producing physical goods or in cosmetics or CPG.

25:02And they should also be in your pricing pages, for example, like maybe a lot of people are going to your pricing page to compare against other alternatives. Pricing pages are often one of the most cited pages on your website when it comes to AI because it has a lot of the most rich specific information. So you should definitely invest in specificity there. And then when it comes to third party publications or just third party sources, which make up a lot of influence in AI search, you can definitely offer these specifics. ultimately it's up to them if they want to include it but it's not something that i would omit how controllable is this i mean this also sounds like great advice uh but the the see you know the models themselves draw from different sources they they have different approaches um as you say like the the models do unexpected things as well so to what degree is this are you able to control the output.

26:02Is this about playing the odds and just trying to sort of get a more favorable outcome? But yeah, I'm just interested in that degree of control. Yeah, I mean, models are always going to be probabilistic. So you'll never have absolute control. But what we've seen from customers are examples like, for example, the largest global electronics provider, one of the largest global electronics providers, they use Profound and they were able to build a Profound agent that optimized 1 ,000 PDPs for them in just a day. And by their own estimate, they saved 900 hours of work. But more specifically, they saw a 78 % increase in AI citations for those PDPs.

26:41And so while maybe we can't put a direct number or concrete number on how controllable this is, we see it across every client that we work with that this kind of work does inflect these metrics. Yeah. Yeah. I'd like to talk a little bit about uh the concept of marketing engineers which profound is has sort of introduced to the market and is is pushing very hard um but i guess just take one step back before we talk about the role itself uh you talked a lot in your presentation about this idea of um marketing outputs becoming much more of a sort of system more of a system um and just breaking down internal silos.

27:26Let's talk to me a little bit about what you mean by that and how you see all these, you know, what might be social media or content or advertising or customer experience, how it all starts to come together in this kind of world that you're seeing. Absolutely. Yeah, I think it's a big question. So there's going to be a couple paths that I go down here. I think the first one is the rise of AI search is kind of the impetus for this role or the motivation behind why brands are hiring for this role. Brands like Google, Figma, Databricks, Okta are all hiring marketing engineers. And it's because in AI search, to be the answer, you have to be the answer everywhere.

28:05In just the last week, we saw that in profound data, to form its opinion of a single brand, AI looks through 350 ,000 sources. And that's gonna happen for every brand every week. And that's an impossible amount of ground for one marketer or even one marketing team to cover. and that means that to win an AI search you have to make it a team sport you have to make it something that your social team content team AEO team growth marketing team product marketing team brand team are all bought into because all of the work that they do impacts your presence in AI search and also AI search impacts them so that's the first thing but then you realize okay that still might not be enough.

28:47So the top brands that we work with are hiring this marketing engineer to supercharge the work of these teams. And the marketing engineer's role is to build marketing agents that scale the work of each of these functions. And so then once you start building agents, you start rolling these out across the org, the fear that starts to creep in is how do we prevent slop? And that's a very valid concern. Our stance on that and the reason why you need to build systems is because slop, in our opinion, is actually a skill issue. It's like messy handwriting. It's something that can be corrected if you just take the right steps.

29:22And there's kind of levels to building agents. So you can build an agent that's a one out of 100 and one that's 97 out of 100. And the difference is the system around the agent itself. So this is kind of a long-winded way to get to the point here, which is marketing engineers are not only the ones who sit at the intersection of the marketing team they're not only the ones who build the agents but to get good outputs out of agents it all starts with context context is the antidote to slop and to inject unified real-time rich context into the agents that you produce that's where you need a true system and so we're seeing just tell me what you mean by context there is that sort of um things like what I called in other situations, things like category entry points, when is it a consumer might want you and what kind of information they want?

30:18Or do you mean something different? It's the context that makes your marketing feel distinct. So I think out-of-the-box AI tools, when you use them to create marketing, they often lack the proper context. And therefore, everyone's marketing starts to sound the same. So generic context creates generic marketing. But then let's say you take it a step further. You're a little more advanced than you have, let's say, a cloud project, right? That context is isolated to your project. And if someone on the other team is using their own cloud project, now suddenly you're using different context to produce marketing for the same team.

30:51And so that creates inconsistent marketing. And so the solution is rich, unified, real-time context about how your brand sees your company, your strategy your execution your market your offering and your competitors there's kind of a codification of brand strategy into into something that is machine readable that can be used within with one of these workflows you're talking about exactly so it's what allows you know profounds ai marketer for example to create marketing that sounds like you wrote it yourself because it is one-to-one in tune with how you think about your brand and the specific points that you'd want to emphasize the details that you would need to include and would otherwise have to prompt an out-of-the-box AI tool to include.

31:37So the other hard part about this is sometimes you're fighting stale context. So if you can get all this information into the AI tool you're using, that's great. But then in a month, a lot of it's going to be outdated. So this is a big reason why actually we see marketing engineers in the field building these marketing second brains. And these are the marketing engineers that are really pushing this role forward, super technical, but at Profound we figured there's got to be an easier way to do this for everyone so we built Profound Context Manager which connects to your Slack your teams your meeting transcripts your emails and is a living synthesis of all of your brand data and also your personal data how you interact with your AI marketer in Profound how you prefer to write and because of that these these sources of truth about your brand competitors offerings etc can self-update in Profound so that all the marketing your AI marketer is creating is always in line with your messaging and positioning, even as they shift over time.

32:34So where you've seen this marketing engineer role be adopted, where does it tend to sit in the organization? Is it kind of a senior role that sits above all the different sort of individual teams, or is it to the side of the teams, more like a coordinating role? How do companies kind of fit this into an org structure? Yeah, that's a great Great question. Transparently, this role is still brand new. It's about six months old, at least since we introduced it to the market back in April at Zero Click, which is Profound's flagship event series. And so I would break it down by company size. So for example, if you're a startup, marketing engineer might be a great first hire because they can produce the output of a 10x marketer because they know how to build these systems.

33:22If you're scaling up, Maybe a marketing engineer is like a power IC that shadows and sits next to the other marketing functions to understand what systems they need to unblock the rest of the team. And then they would probably have a, they would probably report directly to like the head of marketing is what we have seen. and then for a much larger enterprise company we've actually seen marketing engineers lead entire teams and build out their own kind of org within the marketing team and that's really exciting because then you can start getting super complex and build really really robust systems that you roll out across thousands of marketers the final thing i want to just explore is um sort of linked to that which is the overall impact of of ai in general on the kind of marketing workforce and you know we've all seen that data that came out a few months ago about how marketing roles were going to be decimated and and this kind of thing but the future you're talking about sounds more positive it sounds like there's actually well bluntly still quite a lot of work to do by humans.

34:32So talk to me about how you're seeing this, this sort of impact on marketing teams. Yeah, absolutely. I think there's two parts to it. I think the marketing engineers that we see kind of emerging are a lot of folks who've actually already been doing this role, but just haven't had the title or the scope to run wild with it, to have that impact across their whole org and not just their own team. And so these are often unsung heroes, people who are already trying to build these systems and just help the team out. And so I think they're going to be incredibly, incredibly valuable. I think at a higher level, when it comes to, you know, these fears around will AI take my job as a marketer?

35:10I profoundly have a pretty strong stance here that we won't see replacement in marketing. We're going to see augmentation. And the way we break that down is in fields like math, science, physics, even engineering, there are verifiable fields where the right answer is definitive and it's very, very clear. And that's what AI is so good at getting right. That's why you see AI solving millennia math problems today. But then in the middle of this spectrum of which fields have verifiable work versus maybe more abstract work, maybe you have legal, right? Legal, you have case law, which can be a good guideline, but there's 10 ways to skin a cat there.

35:53So on the opposite end of the spectrum is where we think marketing sits, right? At its core, marketing is still very human. Yes, you'll have to start marketing to Asians as well, but what works in marketing today or even a month ago might not work six months from now. There's not a verifiable answer because you need that creativity, that judgment. And so what you need is a tool that can help get you there, but over time, marketers will start to become their key responsibility and role will become approvers, judgments of taste. And I think that's what we're already starting to see at Profound. So to sort of summarize the kind of messy, slightly unprovable nature of marketing may be what ironically saves marketers.

36:35Absolutely. Yeah. And so in a lot of ways, it's more of an art than a science, right? And so I think AI is going to be incredibly powerful at helping us create more marketing at a higher quality than ever before, but you still need someone to steer it. Steering AI in marketing tools and marketing use cases is going to be an incredibly valuable skill over time. And marketing at a macro level is going to be how brands differentiate themselves as well. If everyone can ship product at the same speed, can develop new features, develop new products at an incredibly fast rate, then the moat becomes your brand.

37:13and so marketing as a whole is only going to get more valuable. That's a great place to leave it. Matthew, thank you so much for joining us on the Walk Podcast. Of course. Thank you so much for having me. This is a lot of fun. Now, anyone familiar with Walk will know there's nothing we like more than taking a big marketing topic and, frankly, nerding out in the data and understanding what works and what doesn't. My next two guests have a reasonable claim to work for organizations that like doing that even more than we do. I'm joined by Tracey Adams, Senior Director for Research and Insights at the Advertising Research Foundation, and Keith Smith, Managing Director at the Marketing Science Institute.

37:52I spoke to them following a presentation on a highly debated topic among researchers, the role and reliability of synthetic data. Let's hear from them. Tracey, Keith, welcome to The Walk Podcast. Absolutely. Thanks for having us. We're very excited. Thank you. Lovely to be here. So firstly, just a little bit of background, ARF and MSI are sister organizations. Why don't you just explain a little bit about how the two of you work together? So Tracy, why don't you kick us off? Sure, thank you. So I am the Senior Director of Research and Insights at the ARF. The ARF is a nonprofit. We provide research services to our members.

38:31Those span research companies, media agencies, advertisers. And the idea is to be as impartial and evidence-based as possible. And my name is Keith Smith. I'm the managing director at MSI, the Marketing Science Institute. And we're just an organization to ARF. And really, our role is to bridge the gap between academia and industry. So we really bring academic research, it's new ideas, new methodologies, new techniques to an industry-practical relevant context. And then vice versa, really support and fund academic research that industry has expressed the need for or the demand for. Now, the two of you have been here talking about synthetic research.

39:12Huge, huge topic, of course, within TMRE this year is a huge topic generally in marketing insights. Now, I want to just dive basically straight into the research you shared, because I think it's really interesting to get your sort of kind of industry level input here. Your research identified three recurring risks. We've got regression to the mean, over-optimism, and instability across small input changes. So could you just walk us through those and tell us what you found? Yeah, absolutely. So I'll first just note that this research is largely based on actually three plus years of academic research for the most part that's been exploring these synthetic data issues and questions over a pretty long period of time, as it has become far more popular in the industry space.

40:03Really, we're looking at, I would say more broadly, challenges when it comes to accuracy, really variability in the data, and then stability. So really what we're looking at is, you know, we've seen across a large body of research that these types of responses can be very biased. Synthetic data has certain sort of built-in biases that exist, things like over-optimism, things like preference for technology or new ideas, and other potential sort of bias challenges, essentially. And so we're trying to call out the fact that these biases exist, and you really need to be cognizant of them and aware of them.

40:40Same thing is really true around the variability question. So if you actually start exploring this data and working with this data, what you'll discover is that the distribution is incredibly narrow compared to a normal response curve from a survey response. What you find is that the average may be extracted accurately from the synthetic data, but there's little to no actual variability in those responses. And while you might think, well, the average is kind of our focus when it comes to a lot of the survey response, in marketing, oftentimes we're trying to target those tails of the distribution, right?

41:13We want to think about segmentation or unique potential perspectives. And you're just not going to have those, essentially, when you really think about the synthetic data. So everything's sort of the middle of the bell curve. Yeah, exactly. It's pulling back just the middle and that's pretty much it. Tracy, what have you learned from this research? Let me just add that the bias that we see, it's not randomly wrong, right? The bias is very systematic. So we can identify, and we should identify, that's the whole point, exactly where that happens, when that happens, and then stress test that. That's like the whole idea behind the framework.

41:50um the variability loss exactly like keith described you know the average the center is the same but the population is hugely different if you're happy with just a broad directional thing then that's you know maybe okay it doesn't matter that much but we all know that we all care about the nuances we care about the heterogeneity we care about those unusual responses those are the the ones that really matter the most and when you say the the biases can be predicted can they therefore be corrected? Right. Well, so for instance, if we know for a fact that AI models and systems, they're super pro and enthusiastic about a wide range of things.

42:33Humans are not exactly the same. They're much messier. There's more nuance to consumers' responses. If we know that they always kind of reduce the nuance in terms of if I ask them for a product recommendation And it just goes back and it gives me the same five, six brands over and over again or the same products within those brands. You know, that's a huge issue here. Obviously, we want to have as much range as possible or at least to reflect the market in a more accurate way. And to expand on that just real briefly to your point, it may be possible to correct for this bias, right? But I think what we're starting to see now as these techniques are becoming more adopted in the field is that different models may have different biases.

43:19And so the bias correction that you may need to engage in may be very different depending upon the model that you use, the way that you approach sort of different model versions, different prompts. And that actually gets at the third place where it can fail, essentially, which we really haven't talked about yet. And that's the model stability or the prompt stability. Let's talk about that. Let's go into that. What does that mean in layman's? Yeah, yeah. What we find is that the responses we get from these types of synthetic data are very fragile, right? So you change the way that you ask the question.

43:53You change the order that you ask questions. You change the model that you use. You change the model version that you use. Each one of these things can actually generate very different responses based on those prompts or those combination of inputs. And so we really argue that if you truly want to be confident about your data, you really need to sort of test that from multiple different perspectives, change the order, change the way that you ask the question, change the models, and really make sure that you've got a stable result because these systems are really not designed to be stable in that regard.

44:27They're going to give you a different response based on the input that they get. The sort of picture you're painting is quite a sort of... uh because often i hear sort of people make claims that you know uh synthetic data is 80 percent correlated with with uh sort of human survey responses there's always a big number but it sounds like there's a lot of variation in this so there's actually as always with these things it's much more complex when you you kind of look under the surface is that is that kind of a fair summary i would say yes and i think part of the challenge that we're discovering in the field is that many of the techniques that a lot of vendors will propose are oftentimes very black box.

45:10So it's hard for us to actually be able to evaluate the methodology behind them, the reasons why we might see some of these perturbations or some of these challenges to distribution and to stability. And so I think it is a challenging marketplace right now to really try and make sense of what a lot of vendors are proposing around these techniques. Yeah. The other aspect here is that the black box of it, they themselves don't know how to understand what's going on. So when they come and they say, oh, we validated this, you know, really? Did you? What exactly did you do? What kind of questions did you ask?

45:47What kind of failures were you looking for? Right. And I think that's kind of what we're trying to get people to realize. There are different failure modes, we call them, and each one is a different diagnostic. And you really need to understand validation is not just this abstract thing. It's something that is actually critical and doable. I'm going to throw out some numbers. Please do. This is the War Podcast. We're not afraid of numbers. There we go. I knew that. The ARF did recently a survey with marketers. It was a two-wave survey. We had 203 agencies and advertisers take part in this. And we asked, you know, a bunch of questions about how do marketers use AI on a daily basis, workflows, what kind of utilizations, you know, they.

46:31So everyone is using AI, of course, that's not a big surprise. I think what is maybe of interest is that only 36, I think, percent actually use synthetic. So it's not as broad and entrenched as maybe. As maybe you would expect. certainly given the number of vendors on this show floor would uh would would uh suggest the other number that i really like to to say and i've seen this a bunch in the past two days the in general with with ai there is 92 confidence in the output which is a very very high number among marketers in the survey 92 highly think that the output is good accurate valid but when they actually come to validation 30 something i don't know the exact number actually do the validation and actually go and check to see whether the out so what a gap so so there's a huge gap there i did so you you think there's a loss of confidence being placed in these platforms that that actually isn't being tested exactly and i want to add to that and i'm sure keith is gonna want to jump in here one of the points that we're trying to make is the output of ai is so plausible It is so professional sounding and so convincing how easy it is to just say, oh, sure, that makes sense.

47:52And not take the time to actually dig in and see what exactly the output includes. The catchphrase for this research is basically plausibility is not validation. And that's what we're seeing is that a lot of companies are saying, look, this is a plausible result. You know, just because it sounds, it sounds, oh, yeah, that sounds sensible. It still needs to be tested and validated. Right. Yeah. You've got a lot of non-researchers saying, well, I can do my own research. I can, you know, use AI to actually ask the questions I think need to be asked. And while that is true, you can get plausible answers back.

48:26What's not being asked is what's not in the response you got back or are those responses stable or accurate or actually show variability. Right. And so there's just a lack of validation, I think, happening right now in the marketplace around these techniques are there um certain use cases uh for synthetic data which where there is more confidence because again people are using it for lots of different things anything from you know planning planning their marketing strategy through to testing creative or ads so there's a lot of a lot of stuff going on in this space are there specific use cases where there is more uh validation or credibility?

49:06Yeah, so we actually, in the framework, sort of lay out what we argue are three sort of decision structures or decision use cases, essentially. And it's not so much a matter of whether or not it's more validated in that context. We actually use this decision organization to say different types of decisions require different levels of validation. So sort of our lowest level of validation requirement is essentially what we're calling exploration. So this is really when you're trying to increase the state space around which you're trying to develop ideas, right? So whether that's concept testing, whether that's trying to explore different types of marketing plans, you're looking at options.

49:42You're trying to explore a different set of options that might make sense. In that case, the need for validation is still there, but it's lower, right? Because there aren't really critical decisions being made in that context. The next step up with what we call prioritization, and this is really when you're trying to take a state space that you have and shrink it down, where you might want to then explore more deeply some of those options. So you might decide, hey, I've got a list of concepts. I want to narrow down to three to start actual product development around those concepts. In that case, the validation requirement is a little bit higher because you really do need to make sure that you're getting validated support for making those decisions that are going to generate additional spending in most cases.

50:23And then our sort of final set of decisions, which we call the highest validation requirement, are really decision steps. So this is essentially, I'm about to decide to go to product launch. Or I have - Where the money starts getting spent. Exactly, right? Or I've got a critical forecast that's going to decide our marketing plan. In those cases, really you have to explore deep validation. And we actually talk about multiple different types of validation. In those kinds of cases, you're going to want to get multiple channels of validation rather than a sort of a single channel that you might for an exploration.

50:57use case. So if we're looking sort of 12 months into the future, we'll come back together at this conference next year. What do you hope will have changed? Or do you feel like what would sort of progress look like in a year's time? Well, on my end, I hope that the next time we do a survey with marketers, I hope the gap between the confidence and the validation is going to be, you know, more reasonable. There's another thing that we didn't talk about, which was about training people to work with AI. I hope that gets higher and a bit more rigorous. Is that at an industry level or within individual companies?

51:33Well, within individual companies have participated in the survey. Again, the agencies and the advertisers, there is more training than there was in the past, but still it's not at a level where you feel comfortable to know that those people are actually doing all the analysis and measurement. The other thing that I would say, I hope the a conversation about AI and especially synthetic is going to be less like oh my god this is amazing we're going to solve the world and how wonderful and it's going to be more grounded in you know what is actually being done how it is being validated and I think also how can the results of one specific analysis or measurement can then be generalizable also for other things because I think maybe that is kind of the gray area right now.

52:20Oh, I did this thing and amazing the digital twins worked or the personas worked, but hang on a minute. Let's try and see if that is actually something that we can also take from that very specific use case or case study and then apply it to a different audience, a different, you know, so I feel like that would be more grounded in something that's more substantial that we can feel comfortable to say, oh, okay, I believe this. I would say an understanding that there's multiple avenues for validation that really do need to be explored, whether it's comparing it to humans, whether it's comparing it to theoretical concepts that exist in the marketplace, whether it's comparing it to other data that the organization may have that's not necessarily survey oriented, but maybe more transactionally oriented.

53:06All those sort of different types of channels, I think, are important ways to be thinking about validation, not just does it look plausible. So let's test it against those things. And I think as well, really taking a more critical perspective or critical eye to the use of AI. And I don't mean critical in the sense of challenging it and saying it's not appropriate. But I mean, each individual person looking at their individual use of AI and pushing it to actually give it more than just plausible answers. So, you know, on an employee-by-employee basis, really recognizing that you can use AI as a partner to explore multiple different perspectives and avenues to really try and get at a better understanding of what's happening in the marketplace, rather than just taking what the AI is giving you and saying, look, this is a great solution.

53:54Let's ignore what isn't said or what's omitted from some of these discussions that we get back from the AI systems. Good. Keith, Tracy, thank you so much for joining us. Thank you so much. We appreciate it. That's all we have time for on The Walk podcast today. If you weren't with us in Denver, but today's conversation made you wish you were, it will be worth marking your calendars for Insights at Impact 2027, taking place from November 2nd to 4th in Nashville, Tennessee. And if you've enjoyed this episode, be sure to tap follow and catch the final episode in our Insight to Impact series, which is coming up next week.

54:30See you next time.

54:36Thank you.