E123: The Meta Incrementality Report: Lessons from 640 Haus Experiments

16:52Olivia Kory:And what is incrementality for all the people listening to this that are have had their, you know, head in the sand because I'm behind the whole group. And I feel like in the last 18 months, I've been learning a ton about incrementality. So maybe even explain that. All right. Let me try, Olivia, you can tell me how bad my explanation is or like how you, how usually you go about explaining it. Cause I'm sure you probably have a much better one, but I think incrementality is trying to understand which conversions would have happened or which conversions would not have happened if you didn't run an ad.

17:20Olivia Kory:and a certain number of the purchases that you get, right, you might think meta is driving all this performance or YouTube is driving all this performance. A certain amount of those conversions would have happened anyways. And so incrementality is really trying to understand which of those conversions were caused by the media you ran. And usually that's done with a holdout where you'll have like one part of, you know, the group that'll get the ads and a certain part that won't get them. And then you're able to accurately compare, which is called causal measurement, the impact that your ads actually drove, which is the incremental lift.

17:52Olivia Kory:I'm gonna put this in practice. At Ridge, I'm running North Beam, right? So I have Facebook ads that are hooked up to North Beam and I got a 2X one day click ROAS on those, right? What House will do is tell me, hey, that is your best ad. Like you still need North Beam to tell you what your best ad is. But House will tell me, hey, that ad is actually a 1.8 because there's some amount of those purchases that would happen without that, right? And when I talk to Connor, because Olivia, you so graciously pointed out all my good ideas come from Connor. I basically just, I talk to him for two hours every day and the best stuff he says, I just take it out.

18:26Olivia Kory:But it's that you're lost in the jungle. That is being an e-commerce marketer. And you need a compass. You need a topographic map. And you also need a normal map telling you how to get back to the freeway. So one of those, let's call it Northbeam, is going to be your compass. It's going to tell you directionally what's going on. You still need like a path to get back to civilization. That's when you layer on all these different tools. So did we get that wrong?

18:52Operators Podcast Hosts:I'm so glad Cody had to do the what is incrementality. No, no, it's the student. The student becomes the master. Yeah. And I mean, like, Mike, you hit on this in the beginning at Quibi. We didn't have an incrementality problem because we didn't really have organic demand. So this is it's not really a problem when you're first starting off, like even starting house, like kind of knew, you know, what would work. It's a first world problem.

19:15Olivia Kory:you're saying?

19:15Operators Podcast Hosts:It's a luxury. When I when we come like Sean, you know, you talked about your incrementality problems last year on Meta. Sometimes I say like, congratulations, you have an incrementality problem, because it means that you have organic demand. And that's when this really becomes a problem. I typically see this starting around maybe like mid eight figures, you're starting to diversify off of Meta, you're starting to do more partnerships, you're lighting up top of funnel. And it just starts to get really fuzzy, like what's actually driving the business, especially if you're omni-channel. So that, yeah, that's just my only note is that this starts to really become a problem when the business is taking off.

19:52Operators Podcast Hosts:That's around the same time all these platforms are kind of taking credit for these conversions while you're left unsure of whether these are driving new business for you. And the other thing I would say in terms of the incrementality is this idea of a holdout experiment kind of analogous to a randomized control trial in healthcare where we're understanding, you know, placebo drug, like what would have happened anyway. And then there's the geo testing part that's important. This is what allows us to understand omni channel impact of when you're moving off of the world of user level tracking and click based kind of journeys.

20:26Operators Podcast Hosts:And you just move off of that tracking and into the world of regional testing. We say, hey, instead of not delivering these ads to this set of users, instead, just turn off your marketing in these regions, maybe, you know, turn off Ridge ads on Meta in 30 % of the country, then we're able to see not just what is the impact on D2C and what is the impact of these digital actions, we can understand how users are going to Amazon and into retail as well, which has really helped like complete the picture.

20:53Olivia Kory:Yeah, that's where a lot of the anchor mentality is, right? Is that it's that digital ad didn't lead to a Ridge.com conversion. It led to a conversion in Shields or something that it's very difficult. You've got to use, I'm guessing, some statistical models to kind of parse that stuff out of, hey, when we do these holdout tests, we can see how this is popping up in different channels. And like ads work. I mean, I'm going to spoil the test we're going to do. It's like, look, you know, when you show ads to people, you show it to enough people at a large enough scale, like some action will be taken.

21:26Olivia Kory:Right. And we want those actions. If you're a D2C brand advertising on Meta, you want all those people to go to your Shopify store, make a purchase, subscribe for emails, come back via SMS and retargeting, whatever else. But that's just like the world's way messier than that. Half of all e-commerce transactions happen on Amazon. And we're in Best Buy and Nordstrom's and Shield's and all these places. So when I run an ad, I'm increasing my surface area of success by being all these new channels, right but like that also decreases the signal from the ads right yeah um so cody when did you decide to go all in on incrementality it was probably 2023 i think we're we're on our way i think we did 100 million that year but we had like a point during it was really two things kicked off we had a point during the year um where we planned actually before that we so we were spending on youtube and um youtube was kind of our attempt to get some upper funnel try to reach some new audiences, we were spending like probably 5k a day, but like we couldn't really justify spending more.

22:29Olivia Kory:We thought it was working, but obviously in North Beam, like the return on ad spend was way lower than meta. We saw signs that we thought it was working. I think at the time, like our MER wouldn't have been good if we were spending that much. Like it was enough of our budget that like we couldn't just go waste, you know, five to 10k a day. And we'd look at like post-purchase survey and as we spent more, it would go up, but we didn't quite have the confidence to say like, hey, we should be doubling this. We should really be doubling down on this. And we actually tried to run our own holdout without a data science team.

22:58Olivia Kory:I don't know if I ever told Olivia this, but without a data science team internally. And we ran it and I was just like, I don't really know. I don't know if there's a lift. I think there is. It just was terrible data and terrible math. So that was one of the things, but I just hadn't come across House before. Maybe I'd heard of another incrementality platform. I'll let you guys decide what to do with that. I had heard of another one, but never really looked that far into it. And then we also, we got into this point where like 80 % of our spend was on meta and we weren't growing as fast as we want.

23:28Olivia Kory:We hit a little bit of a plateau and we just were reaching the same people over and over. And so we needed new strategies to break out of that. The problem with that was we needed some type of upper funnel strategy that was going to reach people that was not going to look good in a North Beam or on a platform. And it was what our meta team was recommending. We just got into disruptors. But like, you know, if you run a reach campaign, you're going to get almost no attribution. So I think for both of those things, incrementality was just a tool to be able to properly measure, invest in some upper funnel brand awareness stuff without just lighting money on fire.

24:02Olivia Kory:Because I think we were we were big enough at the time where we know we knew that we needed to diversify and do some non-click based things to reach new people. But we weren't big enough where we could just do some brand spend and light money on fire. So incrementality was and is really the only tool and way to measure those properly. I think this is actually just the way you said it's really good, Cody, that to me, incrementality has been a lot about understanding what's economic and what's not for our business. I mean, obviously, like you said, Sean, ads work. And if I could just saturate everything talking about Simple Modern, then we would sell more stuff.

24:35Olivia Kory:But a lot of that would be uneconomic. You know, I'd be upside down on it and I'd go out of business. And so what I'm trying to do is find what's the maximum surface area of economic advertising that I can do. And incrementality for us, because we're in a lot of different channels, potentially really grows the universe of advertising that we can do. And then the other thing it does for us is that it helps us to understand on low incremental things how much we're willing to pay for them. So a really good example we've talked about on the pod, when you advertise your brand's keyword on Google or on Amazon.

25:10Olivia Kory:Like sometimes we've had periods where we went like, we're not giving them a dime. We're not advertising there at all. But there's obviously like some incrementality by advertising on simple modern key terms on Amazon. It's just really, really low that most of those people, 80, 90 % of those people would go ahead and buy anyway. So we just bid accordingly. We're able to kind of take the numbers and adjust them and say, well, hey, you know what? Nine out of 10 conversions Amazon's going to show isn't really incremental. So we're going to take that number. we're going to divide it by 10. And then that's what we're going to kind of base what we can bid off of.

25:43Olivia Kory:And so it's been helpful there. And even understanding in some of these low incremental places, how much are we willing to bid? Because what's economic?

25:50Operators Podcast Hosts:Well, I was just going to say something that Cody always talks about. I know it triggers him is when someone says like something is binary, like incremental or not incremental. He's been pushing this hard of like incremental at what cost? Because like brand search, you know, it can be a low incrementality. It can also be super cheap. And so you need to understand the economics of it. And I think he's been really good about educating the market on that concept.

26:14Olivia Kory:I'd even argue they're correlated, right? That the reason why it can be really, really cheap is your click-through rate goes to the moon. It's because people are so highly qualified. If somebody searches Simple Modern, there's like almost 100 % chance they're going to click on something. And because that click-through rate is so high, you can bid really low and still get the volume. But that also, I think, I bet there's a really high relationship across channels between incrementality and click-through rate. Yeah. I was just going to say, yeah, Olivia, right now we have a 47 % incremental lift on meta, but we're 30 % above our breakeven CPA.

26:50Olivia Kory:So yeah, they're definitely not perfectly correlated. Most brands are leaving money on the table with SMS, not us at Ridge. SMS is our number one revenue driver and PostScript is the reason why. Here's the deal. SMS is an email. If you treat to like email, you're doing it wrong. Postscript gets it. They help brands like ours turn SMS into a high margin, high ROI channel. And here is five reasons why Postscript prints money for us. They stole the five reasons why landing page from all the DTC operators, but here you go. Five reasons why Postscript print money for us. One, we stop wasting sends.

27:26Olivia Kory:Blast and text randomly, bad move, expensive move. Postscript dialed in our automations and subscriber LTV tracking. The results are abandoned cart recovery rate doubled. They told me that. They did the data. It doubled. More subscribers equals more cash. List growth isn't just about getting numbers up. It's about getting buyers in. PostScript's AI pop-ups and cashback incentives powered by Fondue grew our list three times faster. And those people are actually spending money. AI-powered SMS equals instant conversions. Customers don't wait around. If you're not answering instantly, you're losing sales.

28:02Olivia Kory:PostScript AI does the heavy lifting. We went from hour-long response times to closing sales in seconds. An hour-long response time is actually best in class. We have an amazing customer service team using RichPanel and other awesome tools, but it still takes hours to respond to all these people coming in. With AI-powered SMS, we're getting instant conversions. The number four reason why PostScript is the king of SMS. PostScript auto-optimizes for profit. They have this new thing called infinity testing. It consistently finds the best messages to send. No guesswork. We saw a 20 % revenue boost just by letting it run.

28:33Olivia Kory:Number five, we're not getting fined. One SMS compliance mistake equals a lawsuit. PostScript makes sure we don't mess this up. We scale without worrying about legal headaches. The bottom line is brands are winning at SMS and growing faster than anybody else because they're using PostScript. If you're serious about scaling up, go to postscript.io, tell them Sean sent you and get a free SMS audit. Your competitors are going to do this. If you sell pans, they are rushing right now. So if you're not on PostScript, you got to get here before your competitors. Get that free SMS audit. And the results.

29:06Olivia Kory:For Ridge in 2024, we had a million opt-ins. We had 500 ,000 new subscribers. Revenue was$37 million. Grew 100 % year over year just on SMS. The ROI, 108x. This is 151 % increase year over year. We had a 106 % increase in revenue, 108x ROI, and close to 500 ,000 new subs. SMS is killing it for Ridge. We think PostScript is the way to go. We think PostScript is the best name in SMS. They have five reasons why. They're not going to waste money on sends. They're not going to leave your customers out to dry. You're going to get more subscribers. They're going to instantly convert them with AI. They have auto-optimization for profit.

29:48Olivia Kory:They're going to keep you compliant. I think SMS is still the frontier of messaging to customers and PostScript's the best name in it. Ridge did$40 million in SMS last year. This year we'll probably double again,$100 million in SMS probably because of PostScript. Thank you for sponsoring this podcast. Thank you guys. Talk to you later. All right. And I want to, so we're going to have two quick points. The first is I want to unpack what Mike's talking about where he has a lot of different sales channels. So if Mike runs TV ads, okay, it's going to look horrible on his D2C business. Simple Modern TV ads, trying to get people to buy on SimpleModern.com.

30:26Olivia Kory:In North Beam, it'll look literally bad. Like maybe you won't get a sale, Mike. But if you hold an incrementality holdout test on that, you'll see that sales go through the roof at Walmart and Target and Sam's Club and all these other places you are. Which we've actually, we're actually a customer of Houzz and we've been running these exact tests. And I don't know, Olivia, if you've run into a brand that's as omni-channel as us, but it seems like that's what you guys are seeing is exactly what Sean's saying, that there's actually quite a bit of halo across these channels.

30:55Operators Podcast Hosts:Yep. And the more view-based a channel, the more we see that impact distributed to omni-channel. So a channel like TV or channel like YouTube has disproportionate impact on retail and Amazon because there is no click-through.

31:09Olivia Kory:Yeah, dude, without a doubt. And people are already going to be doing shopping at Target every week, at Walmart every week, at Amazon every week. The next thing, this is for the audience. So we talk a lot about marketing measurement and I'll use rich as an example. It's yesterday I spent half of all of my money on marketing, right? So to make the product, to pay the people, to make a profit, to ship it to America and then ship it to a customer is still less than what I pay on marketing, right? And we optimize the hell out of the rest of that value chain, right? Cogs, we have nailed that down, right?

31:45Olivia Kory:Freight, I will negotiate with my freight vendors like crazy. Those are mostly solved games where marketing is not really a solved game. Speaking of, Dweeb Connor just called me my CMO. But marketing is not a solved game, right? So if I'm going to spend$300 ,000 a day on marketing, if I can get slow improvements in that via house measurement, Northview measurement, MMA measurement, they all kind of just pay for themselves. Because if all of them make 1 % better, it's$3 ,000 a day. So that's like why we spend so much time talking about it. That's for the audience. You guys have anything you want to add before we move to the next section?

32:15Olivia Kory:Yeah, that was such a humble brag to say you do$600 ,000 a day. I mean, dude, we're coming off a prime day. We had a good time.

32:23Operators Podcast Hosts:Marketing is the paid is usually the biggest expense. It's the biggest line item on the P &L. And so sometimes when I'm wondering, is this experiment worth it? Like a step change performance improvement in meta can make a very meaningful difference in your business. And so it keeps you grounded of like, hey, yeah, this this is worth it.

32:42Olivia Kory:From my perspective, Sean, the single biggest hurdle in our business is finding out how to spend more. Because we have kind of these margin handicaps where we don't have the landed margin structure that you have and we're going across channels. And so when when somebody says they're spending a ton of their money on marketing, I actually admire that. I'm like, that's a skill, believe it or not. Being able to economically spend a bunch of money at marketing is like a cheat code if you can do it. And it's not just burning some VC money. If it's like, yeah, I can actually make this work and I can keep the wheel going.

33:15Olivia Kory:I think it's a huge skill. On the opposite of the business, it's a solved problem. Like Mike, if you want to make more money, you can cut cogs, right? You can cut cogs with better inventory management to free up more cash, better inventory orders, bigger inventory orders. That's how you cut cogs, right? Freight, you can have more nodes shipping stuff out. Those are all solved problems. I can't tell you how to spend more money on meta tomorrow, right? That's why we do all this testing. And that's why on a marketing heavy org, it's the most important thing, right? So you're basically just paying for it.

33:43Olivia Kory:It's like you're playing blackjack and you're paying for insurance or whatever when you run incrementality tests or MTAs or whatever else. Well, and it's so much more contextualized to the business. Like Cody spending more for where he's at right now probably looks very different than Ridge spending more, which looks different than Simple Modern spending more. And so like you're saying, if you're trying to like say, hey, how do I increase my, you know, landed cogs margin? Like, OK, here's the six things you do, you know, basically, no matter what type of business you're doing. But if you're trying to increase how much throughput you're getting on marketing, it's like in terms of the channels, the tactics, the techniques, like there's like a thousand different ways you could go.

34:19Olivia Kory:And I think it varies wildly based on the business.

34:22Operators Podcast Hosts:Just one more point. I think Cody Taylor said this on the Marketing Operators podcast of like, you know, he was kind of like calling the executive team to like, hey, don't spend so much time in the ad account. You need to, this is like the growth marketing, absolute smash. Like you need to focus on products and expansion and, you know, don't worry about the ad account. And I actually really agree with that. I think like maybe as an executive team, get out of the ad account and don't worry about it. But somebody on your team needs to be focused on it. Like that was like just that. Like I was like, you know what?

34:55Operators Podcast Hosts:I agree with this maybe in the C-suite of getting out of the ad account. But like if you don't have somebody who's thinking about this, you're leaving money on the table because these are the things that actually make a difference.

35:06Olivia Kory:You're not allowed to come on either of our podcasts and say you agree with Taylor.

35:10Operators Podcast Hosts:I am. Well, I am saying this to, you know, maybe Cody, maybe he was. Maybe he was lecturing you there. You're a CEO now. So maybe he was telling you to get out of the ad account.

35:20Olivia Kory:But yeah, what you said was Taylor Holiday was right. People should spend less time in the ad account. But your counterpoint is, well, someone has to. So yeah, dude, if you have a team of 50 people, 60 people, I don't spend any time in the ad account, right? I don't think Connor really spends time in the ad account anymore, right? We have three or four employees. That's their full-time job. Our job is to bring new tools into the ecosystem so the ad account gets better, right? if it's new ad channels, new ad concepts, new products, like that's what we should be bringing into it. But it is, I mean, very privileged for me to be like, yeah, I don't have to spend time there anymore.

35:51Olivia Kory:If you're listening to this and you're a seven figure brand, you're like, yeah, nice, Taylor. I have to put on the ad account. I can't pay for anybody else. Well, and you've earned your way out of the ad account is the way I would say it, Sean, because you guys created the structure of like, hey, here's what we do. Here's how we approach it. Here's the playbook. And then once you have that, it's easier to pass that down and have other people run it, right? But somebody's got to establish that. And most people like that are not at executive levels of thought, it's difficult for them to establish the playbook.

36:21Olivia Kory:It's kind of what I've learned. That's probably the executive function. Okay, so before we get into the data, last calls on thoughts about incrementality, everything going on, the past, the present, the future. Cody, you're the biggest adopter. You're the biggest bull in the whole space. Olivia, it's gotta be cool. Like obviously you guys did not invent incrementality and other tools out there and people are using it. But to see, you know, you get into the DTCX community and just like see everyone talking about it. And obviously you guys have pretty much all of like the top brands, everyone I know of as a customer, like it's got to be a really cool feeling.

36:57Operators Podcast Hosts:My biggest question when I joined Zach was like, do enough brands actually care about this? That, you know, they're willing to like really go back into their org and like change the way decisions are made. And so, Cody, I mean, you probably are, you know, very much responsible for driving the movement. But like the movement is happening. And I wasn't at the Meta Summit, but I heard it was like every other every other presentation. And so, yeah, it's been great. I think we're still like in the early innings, right? Like what I was talking to Meta about actually is that most of their brands now are running incrementality tests, but only a percentage of those brands are making decisions on those experiments and actually making changes and reallocating budgets accordingly.

37:38Operators Podcast Hosts:And that's the next job to be done is like, how do we really operationalize this and get it closer to kind of the source of truth such that you're getting value from these experiments? So that's the next job to be done.

37:49Olivia Kory:Dude, and Cody said it. So incrementality and geolift studies are like a science and you're a practicer of the science, right? Just like MMMs, there's a bunch of MMM tools out there, but like it's existed as a concept for 25 years or whatever. So, you know, we're going to go over this data. I promise the data. I'm leading with the data. But maybe you could just tell us why House is my preferred incrementality partner. Why does Connor choose to pay you guys? Not just yours. I think preferred every operator and every operator. There you go, dude. A clean sweep across the board. Like, why does the service win?

38:24Operators Podcast Hosts:Our company is filled with PhDs in science. They all kind of came over from Amazon. These are PhDs in economics. And they're like really sweating the details in a way where I can come on this podcast and like confidently talk about this data and know that it's right and know that these experiments were run correctly. I think there are a lot of decisions you can make when you set up an experiment. Cody kind of ran into this himself and in power questions and significance questions and statistical questions that really matter when you're deciding whether or not you want to bet your business on a result.

39:00Operators Podcast Hosts:And like that is our mission as a company is we want to create a platform and create this kind of suite where like we ourselves would bet on the results. And for better or for worse, you know, that makes us like a little bit more expensive. but I can come on this podcast and feel good about the data that I'm sharing. And I think that really matters in the world of measurement.

39:20Olivia Kory:You might remember that Ridge switched to RichPanel about a year ago. Since then, our orders are climbing, yet support tickets aren't. RichPanel's self-service portal, now it's 47 % of our shoppers, solve their own issues instantly. We used to answer every single ticket with a person. Now, that one change is on track to save us$500 ,000 in support salaries. just this year. Our customer satisfaction is at an all-time high. It's above 96%. And we're doing that with less people with more orders. AI is all the rage. Ecom operators are finding ways to leverage AI and improve the revenue per employee.

39:57Olivia Kory:Rage targets over$2 million per full-time employee and Rich Panel is helping us achieve that goal. Rich Panel offers one of the most straightforward ways to leverage AI and improve your bottom line. Here's how Rich RichPinnell's AI is different from anything else out there. First, RichPinnell spends the first 24 hours just learning your past conversations. Then it takes all that knowledge and creates self-guided service flows. There's no chatbots, there's no hallucinations. It is just exactly what your customers need to see and hear to solve their problems immediately after that first 24 hours.

40:29Olivia Kory:Our team saw the impact on day one. We're getting more tickets. They're being resolved faster, better, with a higher level of competency, a higher level of satisfaction, all with less people on the team. So if you're serious about boosting revenue per employee and freeing agents to do higher value tasks, book a demo. Switching is painless. They do data migration. They train the staff. Everything's handled. It's a white glove service for operators, listeners. Brands like Ridge, Jones Road, Pila, Lomi, they're all already switched and they rely within two weeks. They guarantee 30 % ticket reduction in the first 60 days or your money back.

41:04Olivia Kory:Think about that. You're going to get a bunch of customers coming to you. Q4 is just around the corner. And they guarantee 30 % reduction in human tickets in the first 60 days. If that's interesting to you, go to richpanel.com slash demo to book a demo. That's R-I-C-H panel.com slash demo. Get started today. Thank you so much, Rich Panel, for supporting the world's number one e-commerce podcast and being on the greatest show on earth. Let's get back to it. All right, dude. Well, let's get to the data then. So we are going to screen share for all video listeners. Okay. They'll be able to see it.

41:42Olivia Kory:For audio listeners, which is, you know, at this point, 50 million of you guys, we will just dictate what's happening. So me, Mike, Cody, we'll cycle through. The first thing, this is the house meta report. Lessons from 640 incrementality tests. What did we learn, guys?

42:00Operators Podcast Hosts:Yeah, this is so exciting. I mean, usually when we're all on Twitter, probably not Mike. Mike is better than that. But Cody, Sean, and I, when we're on Twitter and we're reading these posts, it's like not really backed in a whole lot of data. It's somebody maybe who ran one or two tests and they're making claims. Like, this is a very large sample. This is probably one of, if not the largest sample of incrementality data that I've seen. So we have 640 tests on Meta that we've run over the last 18 months. Average test duration is about a month, give or take. The effects of Meta are more immediate.

42:39Operators Podcast Hosts:So we only had about a week-long post-treatment window. We'll talk about that a little bit more in terms of what that means. But that's where we end the test and kind of observe the behavior of the markets for some period of time to understand latency. And then size of brands in these experiments. So folks often ask, like, what types of brands do you work with? We work with everyone from, like I mentioned, kind of low to mid eight figures all the way up to the Fortune 100. And, you know, like as Cody mentioned, this data set is probably very heavy in D2C, just given how many of the D2C brands that work with us.

43:17Operators Podcast Hosts:Average annual meta spend is about 14 million. So that's the average meta spend of the brands included in this study. And they arranged dozens of verticals, pretty much, you know, all of consumer in terms of categories.

43:34Olivia Kory:I guess what you're saying is the average brand in this study is at least spending a million dollars a month on meta, right? And is that kind of a target to work with house? Like you want to be spending across multiple channels, roughly a million dollars a month or more. Is that like that's the dream customer coming to you?

43:50Operators Podcast Hosts:I mentioned this up top, but I think if you're smaller and you're just starting out, incrementality is not your biggest problem. And that's not going to help us probably get more customers for me to say that. But, you know, if you're if you're spending primarily in meta, your digital business, I don't think this is a tool that you need. And I often I'll often tell brands that like, hey, if I were you, I would just run meta conversion of studies, get familiar with the vernacular, you know, like the the type the terminology that you see in terms of these incrementality tests, run them for free. And then when you start to see the incrementality really start to kind of deviate from what you're seeing in platform, then come back.

44:30Operators Podcast Hosts:And so, yes, this is, you know, the average size brand. They're on a few channels. There are many of them are omni channel. You know, Mike mentioned, like, are we the only or the most omni-channel brand? That's like a very big use case for coming to house. I'd say that's the that's like the exception is when a brand is smaller, but they're very omni-channel there that that picture is very fuzzy in terms of what's driving the business. So, yeah, this is a sweet spot where at that average spend, you're probably on more channels than just meta. And that growth flywheel is taking off. Like I mentioned, you've got partnerships really starting to take off and you're getting a lot of value from this approach because it's really changing the decision making in your org.

45:15Operators Podcast Hosts:It's changing the way you allocate dollars.

45:17Olivia Kory:And to give you a real life example, like we're running a bunch of podcasts right now. This will come out before, you know, we're doing a very big flight of podcasts for August. We'll be on all the top podcasts. Millions of dollars in spend. We made the decision because of house results. Right. So that's how these incrementality tests end up influencing real world spend. So 640 tests. Roughly how many brands are in this test? Can you share that? Hundreds. So this is just from the stats side of things. This is just a massive sample size.

45:51Operators Podcast Hosts:Massive.

45:51Olivia Kory:And you're really good about this, Olivia. And I know your team thinks this way. You think in terms of nuance. We talked about it earlier. It's not like, do things work or not? It's just nuance and spectrums and things like that. But like with a sample size this big, what we're probably going to talk about from the results doesn't mean it's always true in every situation. But you're going to get a really good feel about the way that the system is generally working when you get to see that much data. So it's a real, like one of the reasons why it's great to have you on the program is that from your seat, you're able to look at the whole ecosystem in a way that very few of us are able to, where you're seeing across a massive swath of brands, what are the things that are holding to be true?

46:36Olivia Kory:And so this is fantastic. I appreciate you sharing the data with us. Okay. And then Cody, tell us the language real quick. As the former CMO, what do all these words mean? So you have Advantage Plus campaigns, formerly known as Advantage Plus sales campaigns or Advantage Shopping campaigns or ASC. So we'll probably call them ASC from here on out. That's like the new type of kind of AI assisted campaign that we believe Meta is going to for all the campaigns, but it hasn't been confirmed. Olivia, I actually reached out to Meta. So let me see if I have an answer on that. I saw your tweet. Manual campaigns are old campaigns, non-advantage plus, where they had a lot more targeting options, a lot more campaign setup options.

47:19Olivia Kory:So it was a little bit less streamlined and allowed advertisers a little bit more flexibility. Incrementality factor is, I'll just read it, but the ratio between the amount of incremental sales a channel drives relative to sales reported in platform. For example, let's say like we run YouTube ads at a 0.5 row S, for example, just in the platform based on the people that click on it. well, based on the views, right, 40 % of people that see a YouTube ad are pricing it on a TV. It might actually drive a lot more incremental conversions than that. Let's say we find that it drives double the amount.

47:51Olivia Kory:That's a 2x incrementality factor. So then we can go and we can adjust our either North Beam or channel level CAC targets to reflect the incremental lift that it's having. Or for example, let's say you run a Google brand search campaign at a 4x ROAS, but really you found that only a fourth of that was actually incremental, you can haircut that and now divide that, you know, essentially by four to figure out what your target needs to be. And so that's something that I think Olivia talked about operationalizing incrementality. That's something all brands should be doing. IROAS is like return on ad spend, but it's just the incremental return on ad spend.

48:28Olivia Kory:And then post-treatment window is you can actually leave the results open. So after your holdout is gone, after you've removed your exclusions and you're targeting now the whole country or your test is off, how to actually leave the results open to see if there's a lag. So something like meta we'll talk about, maybe not a big lag, but YouTube or TV or something like that might have a longer lag and you might actually get more or better results if you leave that test open for a little bit longer. Can you tell us about how you set that window? What does that say about the duration of how long an advertising impacts a customer's behavior?

49:03Olivia Kory:Sure.

49:04Operators Podcast Hosts:Yeah. So there is a length of time where we stop feeling confident that any effects are like actually a result of the test. And so what we say is we'll measure post-treatment window for as long as the test ran. So if you ran a four-week test, we'll do a four-week post-treatment window. Beyond that four weeks, we're not sure. It's hard for us to really differentiate signal from noise. So we say make your post-treatment window at most the same length as the test. But what we see, you know, like we see more kind of view-based upper funnel channels have a significantly higher lagged impact. And we'll talk about this in the data.

49:45Operators Podcast Hosts:Like, you know, YouTube showed a lot more activity in the post-treatment window than Meta did. And so that's, you know, just likely a function of the algorithm and the fact that Meta is finding users who are further down the funnel. in market to buy?

50:01Olivia Kory:So the incrementality factor, think about it as your kicker, right? So if you get like a 2X incrementality factor, it's like a 2X kicker. So if whatever your Northpeam results or in-platform results look like, you can double them or you do times about whatever your incrementality factor is. The post-test observation window, which you guys are talking about right now, Criteo famously had a 365-day look back, right? Where post-click, they're like, yeah, every sale for the next year, that's us. Until the heat death of the universe, we get it all. Yeah. So we all know that that's ridiculous, right?

50:34Olivia Kory:But we all know that a Super Bowl ad will have more impact because people will talk about it for a week or two weeks or whatever than an Instagram reel ad, right? So it's just like trying to measure those things and actually square them into the real world. So awesome. That's the framing of the data. Now we can get into the juicy stuff. Olivia, what do you got?

50:51Operators Podcast Hosts:Cody, are you wondering why you joined tonight? Like Sean is making you explain all the hard stuff. My job has been so easy tonight. This is wonderful.

50:59Olivia Kory:It's 845 here. I thought I was just here to supervise, make sure these CEOs actually know what they're talking about with marketing. Cody has two kids, two young kids. He's like, hey, I can only join at this particular hour. It's the only time I'm like, oh, okay, awesome. Now explain how marketing works real quick, Cody. That's the podcast.

51:17Operators Podcast Hosts:Should we take bets on when we're going to lose Mike?

51:20Olivia Kory:No, he's in, dude. He's locked in. He's excited. Listen, I'll go for three hours. Let's talk. I mean, if I'm going to get free coaching from Cody, Olivia, and Sean, then I'll take all I can get.

51:30Operators Podcast Hosts:All right. Let's do it. I just switched slides to the findings. So we have three slides of findings. The first, the headline is Meta Works. Meta's impact is undeniable. Um, we have a Slack channel at house called cool tests, and this is where the scientists will kind of post, um, you know, cool tests that they've seen. And it's often meta tests because what we'll do is we have the pre-treatment where the control and the test lines are kind of moving right on top of each other. And then we launched the experiment. We turn off meta in part of the country and you just see this huge gap in sales in the regions where the, you know, the meta is still on and where we turned it off.

52:16Operators Podcast Hosts:And so, like I said, you know, meta is, it's just, it really works. It represents 77 of the 100 highest Lyft experiments ever run on house. That of course is a function of spend. You know, we talked about this a little bit, like, well, isn't Lyft a function of spend? Yes, that is true. But advertisers would not be spending that much there. if it didn't work and if they hadn't validated it. So that's key point number one. I'll run through all the points and then you guys can dig in wherever you'd like. But I'll run through. There are five points on this slide. So number one, the impact is undeniable.

52:54Operators Podcast Hosts:Number two, brands are highly dependent on meta. Meta drove on average a 19 % lift to the brand's primary KPI, whether that's revenue, new customers, whatever they care about most. What this means in terms of that 19 % is that on average, the brands in this study, if they turned off Meta, pretty much overnight, they would see their business fall by one fifth, they would lose 20 % of the business. So this is pretty, this is massive. And again, this is just the average. We've run tests where we'll see Meta's driving like 60, 70 % of a business. And I love to see those tests because it's so clear and it's cool.

53:32Operators Podcast Hosts:It's also a little bit scary, I would say, is when you're that dependent on meta, if you're a health and wellness brand who is affected by their changes, it's like, you know, you do want to be a little bit more diversified. So that's point number two. Brands are very dependent on this channel. Number three, this is where we will get into the post-treatment window, but the impact on meta is more immediate than we see on other channels. So 96 % of account-wide meta studies detected significant lift by the midpoint. And the average lift in post-treatment IROAS was just 26%. So to help you contextualize this, I believe when we launched the YouTube report, the post-treatment effects were like as big as the test.

54:14Operators Podcast Hosts:So, you know, if they were getting a one in terms of IROAS on YouTube in that post-treatment window, that would bump them up to a two. Here in meta, we're just seeing a little like we're, you know, we're seeing about 25 % more coming through in the post-treatment window. And like Cody, you've run a lot of house tests. site, you know, we've talked about this, like, for a lot of these channels, a lot of you see a lot of the effect happening at the final at the post treatment window. But with meta, it is very immediate. We see it almost instantly. So that's point number three, in terms of immediacy.

54:46Operators Podcast Hosts:Number four, this is going to be a big one, I think folks are going to be really interested in this. Meta actually under reports if you're looking at click based attribution. So we see on average, D to C IROAS exceeds that seven day click in platform by performance by about 15%. Meaning if your ROAS in platform is a one, you're probably getting a 1.15. So this is great news in terms of a lot of fear that meta is overreporting. This, of course, depends on your attribution settings. But if you're measuring on a seven day click, it's actually underreporting. And the last point before I finish up, point number five.

55:25Operators Podcast Hosts:Five, omni-channel impact. Across brands doing at least 25 % of their business outside of D2C, 32 % of Meta's impact went toward non-D2C channels. This is Amazon and retail. So we're seeing a significant impact in terms of spending on Meta, driving Lyft in Amazon and in retail. So those are the five points I'd say, you know, that really, I think, hammer home that like this channel works. You know, no matter what you do inside of the ad account, this is a significant driver to all of our business.

56:01Olivia Kory:Yeah, look, meta is a hell of a drug. You're telling me it hits harder, lasts longer, acts faster than every other ad platform out there. So also, you know, water is wet. There's a lot of meta hate. And I've been at the center of some of it, being like, oh, my God, like it's just stealing sales. It's circling the bottom of the funnel. But that's also with me spending$50 million a year inside of Meta. Like it's an amazing platform for a reason. It's a duopoly in America for a reason. So the way that I understand what you're saying here is like Meta really works. Everybody's using it, that it hits bottom of funnel.

56:39Olivia Kory:And so with some things, you don't want to draw a conclusion too early if it's working or not. But it should be working almost immediately if you're trying it on Meta. And that it's underreporting. But the reason why it's underreporting is a lot of times these sales are popping up in places like Target or Amazon, you know, that are kind of outside of the attribution, you know, the cookies and stuff that Meta is using to drive its in-platform reporting. Is that a good summary?

57:07Operators Podcast Hosts:Well, actually, this this stat that I shared doesn't include Omnichannel. So if you're just looking at D2C, it's under reporting. It's just under reporting D2C if you're on a click based attribution because.

57:21Olivia Kory:So they're way under reporting impact is what you're saying. Like whatever you have in platform, especially if you're in other channels, that it's significant, like maybe 50 percent more. Is that am I reading that right? Like maybe 50 percent of the you're under reporting or you're only getting two thirds of the performance is showing up in platform as what's happening across maybe all of your channels.

57:42Operators Podcast Hosts:Yeah, it looks like about 50 % there, right? Because there's about a third of the impact, 33 % happening on Omni and about 15 % underreporting. So it's probably close to, yeah, underreported by 50 % if you're Omni.

57:55Olivia Kory:Well, Sean, I think you've said this before, but like Facebook privately will tell anybody who will listen, if you could see all the things that we see, you would know we're driving even more impact than we can report on. And these numbers would certainly back that up. Oh, dude. And like meta has now embraced, dude, we'll measure it any way you want. Like you could tell somebody, if someone's really confident, really good where they're like, pick any game, I'll beat you at it. You know what I mean? They're like, we'll do MTA if you want. We'll do MMM. We'll do holdout. We'll do, you know, you name the test.

58:28Olivia Kory:Meta's like, yeah, we'll win at it. And it's the best ad platform for a reason, right? Like, you know, we'll talk about display networks for a second. I already on Criteo. I think we can do that because they're not a company anymore. There's so much click fraud that happens there, right? There's like, there's all of these bots out there in the world. And Meta has done a great job keeping that out of their platform, making sure you get good, honest clicks going in. This is kind of unrelated, but interestingly on, it's not unrelated, it's just a different platform. On Amazon, there's actually a report you can run now, which is basically fraudulent clicks.

59:02Olivia Kory:And it's not insignificant. It was something like 20%, 23 % of our advertising clicks, Amazon system was deeming to be fraudulent and not charging us for. You can actually go and pull this data at this point. But to your point, Sean, it's like, gosh, if one of every five clicks, one of every four clicks is fraudulent. That's going to move your numbers around a ton.

59:26Operators Podcast Hosts:Sean, Criteo is trying to make a comeback. I've been talking, they want to be tested against house. So if you want to be a part of that pilot program, let me know.

59:35Olivia Kory:Tell them we'll give them a free chance to get roasted or praised on the operators if they give us$50 ,000 ad credits to test. And 10 ,000 for all listeners. So listeners, I'm looking out for you guys. There you go. I think none of this surprised me a ton, but I think the omni-channel impact is probably larger than I would have thought. That's a pretty huge impact. And I think anyone who has an omni-channel presence, even, are we including Amazon in that, Olivia?

1:00:02Operators Podcast Hosts:Yep, Amazon in retail.

1:00:04Olivia Kory:I mean, that should fundamentally change how you set your ROAS targets. Yep, yes. Based on 32 % lift elsewhere. Well, Cody, this is exactly what we're in the middle of right now is that it, and we, we've known that people shuttle that. So do you guys remember? I don't know. Maybe I'm really dating myself, but before Alexa was kind of like, uh, Amazon Siri, Alexa was a website that you could go to and look at traffic. Oh my gosh. Alexa just started talking to me in my house. Um, but you could go to look at, uh, kind of traffic rankings and like web traffic stuff. I don't know what the go-to is now, but we used to look at this a lot in my previous job.

1:00:44Olivia Kory:Anyway, the most interesting thing that they could do is you could pull a report, which here's all the most commonly viewed websites right before somebody comes to this website. And here are the most common places they go after this website. And it was really fascinating to see. And we don't have a tool like that right now. Maybe there's one that exists and I just don't know about it. But what we know pretty much for a fact is that our customers, there's a lot of one, two, where it's like I'm on simplemodern.com and then the very next place I go is amazon.com. Or I'm on amazon.com and the very next place I go is simplemodern.com.

1:01:19Olivia Kory:And we've known it by watching sales data. But then as we started to get into the house testing, it's certainly showing up. And I think that the most interesting thing, and I don't know, Olivia, if you can share any principles here, but this would be really helpful. When does that omni-channel presence, when do you see bigger percentages and when do you see smaller percentages? Do you know anything that you can share with the group about that?

1:01:42Operators Podcast Hosts:We have data on which channels drive outsized impact to omni. And again, like I mentioned earlier, it's the view-based channels tend to have outsized impact because there's nothing to click on. So you're just going to go by and you're kind of preferred channel of choice. but in terms of of of halo effects like the more you're selling in amazon and retail the more you're going to see in terms of halo effect it's it's pretty like you kind of see it here you need to be really doing um volume in amazon or retail in order to see that impact and i would say that you know sometimes brands like get excited about this too early like they think they're going to find a bunch of impact on on amazon and retail like but they're only doing five percent of their business there.

1:02:27Operators Podcast Hosts:So it's kind of like a math thing, right? There's just like, you're not going to see that if you're only doing, you know, a fraction of your, an immaterial percentage of your business is omni.

1:02:38Olivia Kory:So to press you a little bit on that, is your perspective that what happens is a lot of this, I saw this on YouTube and then I typed in my browser, amazon.com and searched for it, or Or is it more the advertising is just kind of planting brands in front of people? And then through serendipity or whatever, I'm walking through the aisles of Target and I see that product. And I'm just more likely to pick it up off the shelf now because I happened to see that YouTube video last night.

1:03:07Operators Podcast Hosts:Your guess is good as mine. I think I think it's probably both like a little bit of both. That's what we see. And we see we see more lagged effects. So I think that kind of supports both scenarios there where it's either they're not clicking, but they're going and they're looking or they're, you know, when they shop, it's top of mind. So, yeah, I don't, I don't, your guess is as good as mine there. Yeah.

1:03:33Olivia Kory:And Olivia is not a behavior scientist. She's a data scientist. She's just bringing the data for us. So I don't know why people shop like that. It's crazy to me too, Mike. Like, also, Mike, if you want a good website that tells you where other, like, where similar websites to you and where the traffic goes, SimilarWeb basically a kind of open source set. So there's a free SimilarWeb plan that will tell you a bunch of stuff. Good tip. Do you use that very often, Sean? And if so, how do you use stuff like that? Yeah, we mostly use it for competitor tracking, right? Or if we are looking to do, like, any sort of, we don't do these anymore, really, but, like, content sponsorships on websites, right?

1:04:11Olivia Kory:there was a company I can't remember what their name it was started by this girl Lee who I used to work with at Hawk Media and it was like their whole thing was just like it was a content website for DTC brands to whitelist through

1:04:22Operators Podcast Hosts:Quality Edit

1:04:23Olivia Kory:Quality Edit that's totally it yeah so if I wanted to like see if Quality Edit it's going to be a good fit or whatever we would go on similar web and check that out but dude Olivia great poll dude you're on top of it you're in the DTC ecosystem

1:04:35Operators Podcast Hosts:yeah she's cool she's cool just the Omni piece like this is I think part of the reason Meta likes us and you hear about us a lot in disruptors is that when you complete that picture with omni-channel impact, it does change the calculus in terms of how you're thinking about investing. You think you're getting a 0.6 IRO as you're probably getting a one. And now you're like, oh, well, at this efficiency, I should maybe think about increasing my budget. So I think that this is really important and I don't want to understate it. Yeah.

1:05:04Olivia Kory:And some more data from Insider Ridge. We sell on Amazon. Our Amazon tacos, So total account ROAS, right? Total account spend on Amazon is like 8 % or something, right? Now, we have a multi-million dollar per month Amazon business. Do you think I'm that good at Amazon ads where I'm getting a 12 to 15X MER? Or do you think all those people are coming from my Facebook ads? Without a doubt, they're coming from my Facebook ads. I don't even need a holdout test to tell me that, right? But then we can actually, if we increase spend on podcasts, does that go up? That's why we actually do one of those holdout tests.

1:05:37Olivia Kory:But without a doubt, meta ads drive performance everywhere. I mean, nobody can say anything about that nowadays. Jason, what do you like about Ceres? You were gone for the past week in Italy. Were you checking Ceres dashboards? I wasn't checking anything in Italy, but I will say this. Whenever anyone on my team, like growth team or anywhere else, sends me data anymore, it's likely coming out of Ceres. And every once in a while, I'll get something from them and it looks really slick. and I'm like, is that Saris? And I'm like, yeah. And I always feel really good about our investment. I do get a daily email of some metrics out of Saris that comes through.

1:06:15Olivia Kory:But one of the things that I've been thinking about a lot lately is AI. I mean, duh, right? Everyone is, duh. But I finally woke up a few weeks ago. I finally woke up a few weeks ago. I was like, I got to get serious about this. And I was actually talking to the Saris guys and they launched the next generation AI powered stuff in their platform. which I'm excited about. It's Sarah's IQ. It's invite only right now. If you mentioned operators podcast, you can probably get access, hopefully. But the next level of all of this is, first, with AI, you need to have all your data somewhere. I mean, it's going to be really messy to just drop stuff into different AI platforms.

1:06:58Olivia Kory:If you have your data in one place, then you can leverage. You can really leverage AI. So the way I've thought about our investment in Sarah's analytics was getting all of our data in one place, understanding our customers better, understanding our marketing metrics better. But then being able to lay your AI on top of that is, I think, the really exciting next level here. You brought up bringing in Costco, Amazon, and Shopify. The ability to have all sales channels with the same data is so important. If you're listening to this and you sell on Amazon, you know how painful that platform is. You have to wait a couple extra days to pull your reports.

1:07:35Olivia Kory:The SKUs aren't the same as the rest of your SKUs. The titles are different. So to get actually clean data to compare, it's very manual. Using XRS Analytics, they matched all that up for us. So they cleaned all that data. So I could actually look at what is the true margin profile of an Amazon sale? What's Amazon return rate? What's Amazon customer frequency? All the type of data, compare it to our.com and have a comprehensive overview. And now with their new AI tool, I don't have to look at it. I could just ask it. I could just tell me cool questions. So doing that across our five Shopify stores as well.

1:08:06Olivia Kory:We have an EU Shopify store. So now I can compare EU ring customer cohorts versus Canadian ring customer cohorts versus Amazon ring customer cohorts. So now we're getting to different functions of my business across different channels, different categories, and actually looking at the margin, the repeat rate, and like, is it worth investing in these different things? So if your business starts to feel like a spider web and just like keeps going and going and August, just like this nonstop proliferation of different channels. Maybe it's time to check something out like Sarah's Analytics. I remember Sean at the very beginning, this is a very typical Sean thing.

1:08:42Olivia Kory:Sean was like, what is this? Why do I need this? Right. And I'm actually really happy that you've, you come to understand like how important having a data warehouse is and what it does for managing your business. Because you are not like a true believer from the beginning. No, I was angry when you told me I needed to pay for something new, but Jason was right. He convinced me, he got me on board, I'm using it. So a rare Jason W. So thank you, Jason. I'm on Saris Analytics. He's on Saris Analytics, proud sponsor of the Operators Podcast. So if you want a data warehouse and you want an AI-powered data warehouse, Tell Saris about the Saris IQ feature and then you crave it.

1:09:28Olivia Kory:You're demanding you get access to it. So, all right, guys, talk to you later.

1:09:34Operators Podcast Hosts:I'm going to move on to the fun stuff. This is the media slide. This is actually, I think, the most interesting part of the study around Advantage Plus. So Cody mentioned up top that when we were talking through key terms, there's Advantage Plus. This is their AI product. Give us a budget. Give us some assets. and we'll do the rest. And then there's the manual campaigns, the old way, the legacy. I don't know what we call it these days, but it's like the old way we all know meta. And just fun fact, like when I was first starting on Facebook ads at Netflix, we had, I mean, the whole value proposition for our approach was like, we're going to personalize these ads based on who is most likely to sign up for any given title.

1:10:18Operators Podcast Hosts:And we would cherry pick like the most specific audience segments for, you know, the Marvel fans are very different from the full house fans. And so we would get really quite specific with our targeting. When we followed Meta's kind of advice and started testing this in terms of, all right, we're going to remove all those audience segments. We're going to go broad, optimize to conversions. We call this like unfettered targeting. This was very controversial. Like this was probably half of my job at some point at Netflix was educating the organization on this decision that we were making. And like, one of the my one of my fondest memories is when I was working with Ashton Kutcher on his TV show, The Ranch.

1:11:00Operators Podcast Hosts:And I had to explain to him that we weren't doing any targeting for this like middle America sitcom. He's like, what are you talking about? Like, why? You know, this is not stranger things here. And so I had to kind of like give the meta pitch of like, hey, they can find audiences better than you can. But yeah, so so we We compared Advantage Plus to like, you know, these more manual campaigns where you have more levers. This is quite interesting. I think there's a lot of nuance here. Advantage Plus only beats manual, the legacy approach, 42 % of the time, which means that manual is actually winning more than Advantage Plus.

1:11:37Operators Podcast Hosts:I didn't necessarily want this to be true because I think this complicates the lives of marketers. of like, I would love to come on this podcast and just say like, just relinquish control, do other things, focus on other areas of the business. Meta's got this. They're close. This is close. It's not there yet in terms of parity, but again, early innings for them still, like they're gonna improve. And I'm curious to hear what you all think about this. 12 % lower IROAS for Advantage Plus at 18 % lower daily spend. So even at a lower spend level, iROAS is not, it's not a parity with manual campaigns.

1:12:15Operators Podcast Hosts:But again, 12 % is not much. So I'll pause there. Like, I'm curious, like how you guys think about this in terms of whether it makes you say, oh, 12 % is not that much. And our team could do other things. Or if you're like, oh, no, this is meaningful. And I'm going to resist the AI product until they get to parity.

1:12:38Olivia Kory:AFC loses to boomer buying. I'm going to call it boomer buying, right? Like the way we used to do ad buying. Quick aside, when Cambridge Analytica happened, I remember being like, what's everyone freaking out about? I'm like, when the news broke, it was like 2016 or whatever. I'm like, that's how Meta's built. You can go in there, you can click the people you want to target, right? So I think us marketers were maybe too bought into violating personal identity freedoms or whatever being like, oh yeah, I want to target that specific person. But anyway, boomer buying is beating ASC. You probably just made everyone really upset because one of the worst things about Facebook ad buying is you keep touching it because you feel like you can make an impact, right?

1:13:25Olivia Kory:Like, oh, I'm going to tweak this audience. I'm going to tweak this thing. And it's never perfect because it can't be perfect, right? You can't even approach being perfect because it's such a big, like ever moving target with all these other factors. But this kind of proves that those people are, you know, even if they're crazy, that they're kind of right. Cody, is that is that your day to day life? I don't know. I'm not I'm not in the ad accounts anymore. Take that, Taylor Holiday.

1:13:49Operators Podcast Hosts:I mean, this like what I take away is that their algorithm is so good. That's why this is a problem, right? Like it's so good at identifying intent that it becomes like a question in terms of incrementality. And that's the tension here is, you know, when you've built the best conversion optimization engine in the world, you're going to deliver to some people who are going to buy anyway. And, you know, it's an interesting dilemma. But the other points I'll just run through real quick is Advantage Plus delivers less in terms of lagged effect. So effect is more immediate here. Only 17 percent of the lift comes in the post-treatment window versus 32 percent for boomer buying, as Sean would say it, for the manual approach.

1:14:35Operators Podcast Hosts:Advantage Plus is overreporting incrementality slightly relative to manual. So they're overreporting by about 12 percentage points on average relative to manual. And then there are some points that I would say are neutral to positive for Advantage Plus, which is that it actually is delivering a higher new customer share. So slightly higher, 5 % flat in terms of retargeting. So they're not over delivering to retargeting, which is something, Cody, I think you'll probably be interested in talking about that comparable omni channel impact. And I'm going to save this last point for its own discussion, because I think this might be one of the most more interesting learnings.

1:15:16Operators Podcast Hosts:But yeah, it's like when I look at this data, I'm like, oh my gosh, their conversion optimization is so good that it actually is something you need to keep an eye on in terms of incrementality.

1:15:27Olivia Kory:Yeah, Mike, you've heard us talk about this a million times, right? Me and Matt debating back and forth if it's just circling the drain, right? If you build a money machine that's only designed to find you more money, whoever's the easiest transaction, it'll go there. What Meta is being paid to do is to go get people that click the link and buy something. Like that's like, hey, your mission, you know, should you choose to accept it is just get me as many people that click this link or and then go and buy the thing. And what you're really saying, Olivia, I mean, incrementality is especially in the omni channel incrementality.

1:16:01Olivia Kory:That's like people not doing exactly the expressed stated behavior of the advertiser. It's still good for the brand, but it's like, it's maybe not shocking that Meta's product is going to be even better at getting that in front of that particular type of buyer. The person that's ready to click and buy, you know, on your website or wherever it is, you're directing them right then. Yeah. And Olivia, you said this, this only matters until it doesn't. Meta is going to take away boomer buying. And there's a rumor that it's already gone. Cody, do you want to talk about that? That like all campaigns are just advantage plus now?

1:16:36Olivia Kory:I think that's true. Olivia, did you get any confirmation whether or not that's true? I think in our account, I feel like I've seen that.

1:16:43Operators Podcast Hosts:I don't have confirmation. The comms are a bit unclear of like they're saying you can still have the manual controls for now. There hasn't been any comms around like, oh, no.

1:16:53Olivia Kory:In new campaigns? I'm not sure. I think it's fair to say, Sean, that what you're saying, I mean, it's like at some point clue in to the way this thing goes. This is definitely going to go away. I mean, partially, I bet the technical complexity of being able to give this like Burger King, have it your way, you know, boomer buying is just a real pain in the butt for meta. and you even wonder if what's going on is they're like, sure, tell us everything you want. Okay, yep, yep, yep. And then they just like, here, we're gonna give you to the AI. It's actually like one of my favorite stories is that on a bunch of crosswalks, you know, the button you hit that's supposed to like tell it, hey, I'm waiting.

1:17:35Olivia Kory:A bunch of those and a bunch of door closed buttons on elevators, they're not actually hooked up to anything. It's like a social thing. They just, they give it to you so you feel like you're in control and you feel better, but like they don't actually do anything. And whether or not they take away the interface, I think that's pretty much what's going on here. Yeah. And to explain, a lot of that data is purchased via third parties, right? So, you know, Apple was sharing user data back to Meta and that caused a big fight. We all remember that, iOS 14. But other people sell data, right? So your credit card data gets bundled and sold like a million times, right?

1:18:11Olivia Kory:And that's how they build the manual targeting aspect. And Meta wants to go to a world where their internal AI system is predicting that for you with probably no or less data going into it, right? They're going to spend less money on data processing, less money on third-party data, and do more of that internally with AI. So that's why they don't want to support it anymore. It's expensive to support, right? Well, and I think it's probably true that what they would probably argue to kind of, you know, be their spokesperson in this situation is that like you thinking that you can outthink its AI by taking like six demographic, you know, filters and adding them in there is just laughable.

1:18:51Olivia Kory:Like these AI systems are using like, I don't even know how many dimensions, you know, they're using dimensions that we don't even know exist, right? They're like playing 8D chess. And so like their ability to go and target people is just going to be superior to anything that we could do through boomer buying is my guess. Like they'd probably say, hey, 99 % of the time when you do boomer buying versus the AI driven model, you're going to get worse performance. So why even offer that to people if the normative outcome is that they're going to do worse? But Olivia has the data over the last 18 months.

1:19:26Olivia Kory:Us, you know, hardworking media buyers were John Henry. We were taking down the locomotive. We were killing it. But I just think it's like that advantage goes smaller and smaller every single month. Then eventually it's like PMAX. You can't do manual stuff anymore.

1:19:42Operators Podcast Hosts:Yeah, Mike, on the point about the elevator button and just, you know, giving marketers the feeling of control, we see this often in terms of MMM. We're launching an MMM kind of built on the foundation of experiment data. And our customers want to send us so many inputs and so many covariates that we can use to kind of build their model. And what we find is that like that additional data doesn't really help much, but it does give the feeling of like, hey, like I am making this model better. And I imagine that's how meta feels as well as like, hey, you can give me these, you know, these signals from all these different places.

1:20:18Operators Podcast Hosts:But, you know, we know what works.

1:20:21Olivia Kory:Well, I'll tell you what, I would love it if meta did, and they never will. And so it doesn't matter. But I'm going to say it anyway, which is, I think it would be incredibly helpful to marketers to understand what are the dimensions that the AI is looking at, because that can inform a lot of other things that we do in other areas of the business from everything from product development to how we run our social accounts and messaging, you know, the type of creative that you're building. And so that's one thing that's been unfortunate about the move to AI is that it's become just this black box where it's like, you even ask some of the people inside these companies and they're like, yeah, we don't know how it works.

1:21:00Olivia Kory:And I think that there needs to be an increased level of data to flow back there. I'm for sure on board with the idea that I'm not going to be able to outsmart the AI in terms of telling it how to target people. But if it's never telling me what it's doing at all, that's unhelpful as well. What we need is we need a system or part of the value add that Meta and Google, other people are offering is like, yeah, you just tell us what to do. We're going to go do it. And then we're going to come back to you and give you a report on what was really effective, how we went and found those people or the types of things that we looked for that you should be, you know, aware of as a brand.

1:21:34Olivia Kory:Dude, I would love to see that. And I wish, I wish that, and we know they're listening. This gets passed around meta internally. So please. Well, my question is, I don't understand why they wouldn't. I mean, it's a value add. And usually when you're adding value add consultative type services, you can pad your margins with it. So maybe, there you go. There's your business idea of how to make the next, you know, whatever trillion dollars meta. If you can give us data about what your systems are doing and why they're so good, There's obviously proprietary pieces to that, but like there's a bunch of non-proprietary stuff that would really help us as a brand.

1:22:07Olivia Kory:Dude, if we're just pitching meta business ideas, I've pitched them this and they didn't want to hear about it. Charge every ad account$5 per user they add. Okay. Call it an insurance fee, like security fee, whatever. Dude, I did the math one time and it's like$800 million in incremental revenue. That's so interesting because like when we've worked with club channel, like Sam's Club and Costco, like one of their big things is like, you have to pay for the membership. And I kind of pitched one of the clubs on like, why don't you give the people the ability to come in and shop the deals? Because a lot of people don't want to buy the membership until they see, you know, the prices.

1:22:43Olivia Kory:And it's just like a sacred cow. It's like, we can't give up the membership fee. And then what you're pitching is the opposite, that it's like a membership fee. And it like, it's a pretty good business model. Ask Costco. Yeah. Yeah. But, you know, selling, selling air, selling ads, the best model on earth. So, all right, Olivia, what's this next point?

1:23:00Operators Podcast Hosts:Well, they used to, Mike, they used to give us a lot more audience insights pre-Cambridge Analytica. Like, I don't know if you guys remember, like, there used to be an audience insights tab. And we used to use that in a lot of our planning in terms of, like, marketing strategy. And now it's all gone. So I would like to see that, too. I would like to see them be more transparent with this model and what's feeding it. But, you know, we can dream. Okay.

1:23:28Olivia Kory:One of my hottest takes is that as a marketer working in 2016, I was like, Hamish or Lidica just wasn't that big of a deal. Now, if you don't work in marketing, I get it. You freaked out. But as a marketer, I'm like, yeah, that's how I was selling wallets. I think everybody's accepted that that was just a tempest in a tea kettle kind of moment and that people were just pissed off and wanted to be able to, I mean, I just think what it is, is a lot of people saw an outcome that just did not compute for them and they wanted to be able to blame it on something. And so that was like the thing. Oh, it's like, oh, somehow you just manipulated a bunch of people into thinking and behaving in a way that they didn't really want to behave.

1:24:04Olivia Kory:And there's really no evidence at all that that was what happened there. There's so much spent on political ads. Okay. Literally a hundred billion dollars or whatever. It's way less than that, but like$5 billion a presidential cycle. But no, Facebook was the bad guy. It's like, come on guys, what the f*** you talking about? Let us sell wallets.

1:24:24Operators Podcast Hosts:That was the moment I learned that like I was an expert at this thing and I was reading the story in the New York Times and I was like, what else might they be wrong about if they are like getting the story this wrong? So the moment I started questioning things, honestly, but not to get political. I want to hear Cody's thoughts. I'll give this last this last point. I think this one's fascinating. And then I kind of want to hear what Cody reacts to in terms of the slide and the Advantage Plus learnings. But I've had this hypothesis. I've obviously, you know, I used to kind of know every test that every customer was running for a period of time.

1:25:01Operators Podcast Hosts:And I noticed this happening where when a customer was either all in on ASC or all in on manual, they were getting better results. and I was like, I have this hypothesis that like the hedging of like doing a little bit of ASC and a little bit of manual is not actually working. And we saw that play out in the data here. So a heavy up strategy where you're like either all in on one or heavier on one than 50-50 outperformed. So this also complicates our lives a bit because it's like, well, if you're not sure about ASC and manual, just split the difference and go 50-50. That's not working as well.

1:25:39Operators Podcast Hosts:It is. And this is where I think the learning is you need to test this for your business and figure out what's right for you.

1:25:44Olivia Kory:I think overall, it's a little bit concerning because obviously, whether it's now or in the future, we will have no choice. So it's not great that the thing that we're being forced into is less effective than the old version that we had. There might be other factors or variables, or maybe they will crack it. I think your take on it being less incremental because it's so good at finding intent is a very interesting one and very, very, very generous to meta. We'll see if that's what it is or not. But yeah, the word out on the street is that, you know, ASC just like Sean says, like circles a wagon and just doesn't do a great job finding new people.

1:26:26Olivia Kory:And something has either broken or just changed about the algorithm, how it will go after the same people. And it doesn't mean always existing customers. It might just mean prospects and it's harder to get reached. It's kind of bundled this like retargeting and and prospecting strategy. And it's definitely just what most people are noticing anecdotally or, you know, from the data in their brands. so I do think it is something that will have to be improved and meta will have to that's not to get ahead and and and I guess spoil it but some of the strategies that you're going to talk about later on I believe that's partly why where there are now like new best practices of of strategies to help try to find some new people it's it's absolutely the the thing that we are struggling with right now we are spending more money year over year and we're reaching less people and that's not the goal it's not what you want to do and I think just having ASC is is a challenge that But I do wonder if that's why like IA, like incremental attribution that they're testing and talking about and have rolled into GA now is something that is an attempt to solve this because maybe they know that, that that is a negative case of their algorithm being so good.

1:27:34Olivia Kory:We'll see. I do think they're going to learn this and hopefully they're listening. Like Sean said, I'm sure they are. But find ways to help us explore some new audiences rather than just going after the same amount of them. Olivia, I got a question for you. So you have the data for 18 months. Did it get better over time or did it get worse over time?

1:27:51Operators Podcast Hosts:I expected that question and I don't have an answer to that. We haven't cut it in that way yet, but that'll have to be a follow-up, right? Because Sean, I know your theory is like 2024 meta sucked. I kind of want to see if incrementality has gotten better since then. I don't have it.

1:28:08Olivia Kory:Okay. Yeah. Because it's like the whole rollout of Andromeda really happened in January of this year. And what I could see is that ASC was totally like 50 % worse than manual bidding up until this year. And now it's beating it. That'd be my guess. So don't touch your ad account until we get the second episode out, guys. Okay, it's no secret that margins are getting squeezed and profitability is a challenge for brands in 2025. Not a secret. That is where Northbeam's profitability benchmarks come. It's a new feature. You could check it out. This is a tool from Northbeam that tells you the exact targets that you need to hit to get to profit inside of Northbeam.

1:28:45Olivia Kory:So no more guessing, no more month-end financial surprises. Hitting these targets helps you ensure that your ad campaigns are actually driving the right behaviors to make your business profitable. That is wonderful. The brands that track this stuff probably already do in Google Sheets. I know I do. I'm going to be checking out this Northbeam new feature. And Northbeam is kind of a big level up from this. And here's what sets kind of what I have seen. Here's what sets profitability benchmarks apart. So instantly you get to quantify your goals into clear benchmarks. Every ad gets measured against those targets right inside of Northbeam.

1:29:17Olivia Kory:That's kind of cool, actually. And you see and you can kind of at a glance see which ads are doing well, which needs to go against profitability benchmarks, not just ROAS or whatever else you're using right now. You scale up the winners, you cut the losers, you know the drill. It is hard to argue that this is not great for brands. Profitability benchmarks from Northbeam is like having a speedometer in all your channels, campaigns, ads. You can kind of see everything at a glance against profitability, not something else. So stop leaving money on the table. Fuel your creative performance. Make every ad dollar count in every channel.

1:29:48Olivia Kory:Hold them all to the same standard. If you want to reach out to Northbeam, just let them know the operator sent you and they will take good care of you. Okay, next slide. Olivia, what are we talking about?

1:30:00Operators Podcast Hosts:Okay, Cody kind of previewed this, but we dug into this idea of, all right, if we, if you know, the hypothesis here is that ASC is really good at identifying intent. How do we actually use meta as a prospecting tool? How do we introduce our brand to new audiences who might not be in market to buy? Like, you know, that's the stated goal from Cody is I need to introduce this brand and new people. We are seeing a huge rise in terms of mid funnel testing. We'll talk a little bit about mid funnel, a little bit about upper funnel. Mid funnel, When I say mid funnel, we define it as a conversion objective that is not a purchase.

1:30:39Operators Podcast Hosts:So this is like an event on site, like, you know, view content or PDP visit or add to cart quiz complete. We're seeing a huge rise in this test, I think, to solve the point on the previous slide, which is around how do we how do we how do we go a bit a bit shallower in terms of the funnel, but not totally upper funnel. So mid-funnel testing has increased 121 % in frequency across the customer base. And it's not far off in terms of performance. So mid-funnel drives just 14 % lower D2C IROAS than purchase optimization. This is the catch, 85 % less spend. So they're taking, basically brands are taking like 15 % of what they're spending on purchase objectives.

1:31:25Operators Podcast Hosts:They're optimizing to a shallower KPI and it's doing really well. Now, the question, of course, is like, can that scale? And I don't think we should ever hold this media to the same standard in terms of IROs. Like, I'm fine with 14 % lower. Of course, if it's like, if it's 4x less efficient, you know, that doesn't seem like that's something you could make work, even if you assume that there are lagged effects. But 14 % is not bad.

1:31:55Olivia Kory:Can I make a really nuanced point there?

1:31:57Operators Podcast Hosts:Yeah.

1:31:57Olivia Kory:The upper funnel spend is probably a scale strategy. So it's probably on top of business as usual purchase spend. So it's not like apples to apples where like when you're when you're testing meta like purchase, you're testing like a probably like a my guess is a account wide holdout where you're testing almost like the zero to let's call them 50k, whatever they're spending. And then when you're testing a mid funnel, you're probably testing like that spend or maybe most of that spend as purchase with like 10 percent or whatever is on top of that most likely. And so that I'm guessing there's no true holdout.

1:32:31Olivia Kory:Like everybody is getting meta conversion spend. And then that as part of the holdout. Am I correct on that for the most common setup?

1:32:39Operators Podcast Hosts:I need to check on that. I think, you know, some brands actually just add it on top of everything they're doing and others will like restructure the account in the way where this is like it's just its own cell. And they account for the fact that this is, you know, it's not just on top, but they want to see it like alongside their purchase optimization campaigns in two different cells. OK, cool.

1:32:58Olivia Kory:Yeah. Anytime like we've tested something like that, like there was never any like true sandbox or true meta holdout. So like comparing like the customer incremental was like slightly elevated because they were getting the regular meta spend as well.

1:33:11Operators Podcast Hosts:Yeah, that's a good point. So so yeah, so Cody is getting into like a test design thing where if you're just adding this on top of everything, you can't expect it to be super efficient because the marginal efficiency of those like last dollars into meta are, you know, you're holding them to a very high standard. So a lot of brands will like the question here is, should you take should you carve out some of your purchase optimization optimized media and optimize that to mid or upper funnel? And we're seeing that that's actually been working pretty well.

1:33:42Olivia Kory:The question is, you know, how much you can't have wine unless you smash a couple of grapes and you can't have grapes unless you grow them. Right. So what I'm trying to say by that is. a lot of us, most accounts, 70 % of listeners are just running ASE campaigns right now. Right. And it's very hard to scale that and keep incrementality high. Right. If you're spending a million dollars today, you're going to spend$2 million next month and you want more incrementality on that on ASE, it's very hard to do. It's just going to circle the bottom of the drain. You might have to grow some more grapes, right?

1:34:15Olivia Kory:So that's moving to a mid funnel, moving to an upper funnel. And those people will move down into ASE, right? It is a funnel. They will move down and that'll help increase overall account incrementality. So when we're saying that mid funnel is 14 % worse, it is worse, but think about it in three dimensions that like, it is what makes the second layer actually as incremental as possible. So that's the nuance I read from this. Yeah, there's not anything to harvest at the bottom of the funnel if you're not doing some cultivation at the higher levels of the funnel. And then all of a sudden your lower funnel stuff isn't producing the volume or the kind of ROAS that you want.

1:34:59Olivia Kory:You're wondering why, and it's because you're not doing any cultivating further up the funnel. Dude, yeah. If you want that line, you have to grow those grapes.

1:35:06Operators Podcast Hosts:So, yeah, I often push back on brands who say, you know, I want to go top of funnel. I want to create demand. And then they go straight to TV. I like I kind of I remind them that, hey, Meta is, you know, reach can reach more people on Meta than any other platform out there. So, you know, it's just it's kind of an afterthought in terms of this idea of going up a funnel. And we're seeing that the I see I'd say like the mindset start to shift.

1:35:33Olivia Kory:Yeah, dude. And we all know that a TV North Beam ROAS will be 0.1 or whatever. But then you see that same 0.1 on meta reach campaigns, and you're like, this is the worst thing ever, right? So I think a lot of it is just bias built in. So you can reach anybody anywhere if you try hard enough.

1:35:49Operators Podcast Hosts:And Sean, you guys have had a lot of success with these mid-funnel PDP optimization, right? They're holding up. They're hanging in in terms of even North Beam performance, too.

1:35:59Olivia Kory:Oh, yeah. You said you've seen a 121 % increase, and that's because Ridge said we like doing it. So you've seen about 8 million people copy us on that. We're spending, I mean, some months 30, some months 40 % of our budgets are going to these mid-funnel objectives. And when we measure it, it's the most incremental thing in our account, and they look just as good on NorthBeam. So you're actually seeing like the same because this is the most interesting part of this is that you're saying, Olivia, that even though it's not built for purchase optimization, it's mid funnel that you're you're seeing it perform pretty comparably to purchase optimization.

1:36:38Olivia Kory:But then on the kind of incrementality and omnichannel effects, they're way bigger because you're kind of getting outside of the very, very bottom of the funnel. Is that an accurate way of saying it?

1:36:48Operators Podcast Hosts:Yep. This is on the slide here. I'll just voice over that mid-funnel is driving better omni-channel halo effects. So 70 % impact on omni-channel versus just 46 % for lower funnel and delayed effects. So 44 % of that lift is happening in the post-treatment window for mid-funnel versus 26 % for lower funnel. So it is hanging in there. The incrementality is comparable. and if you account for for omni-channel and in post-treatment window it's it's very close um actually code cody was like the originator of the mid-funnel test he had i i often say like you know you can run amazing tests if and answer really big business questions if you're patient and you run a long experiment and cody ran like it must have been like a four-month test that ran where he was just doing mid-funnel and he ran it through black friday cyber monday to see how it was like three months before Black Friday, Cyber Monday, he started spending on this and actually had the patience to see that through, like through November to see how all those folks converted when the time came for the sale.

1:37:53Olivia Kory:We're also launching this week. We're launching 20 % of our account on MidFunnel. We had what I believe was the best episode of Marketing Operators. It's going to drop tomorrow. Me and Marketing Dweeb Connor talking all about meta testing. So great timing. But yeah, I know Ridge has seen success with it, but we're struggling with this ASC reach thing. And so definitely very excited to test that again. Dude, Connor's going to shoot me in the face if that's the nickname that sticks. So that's awesome. Yeah. Well, look, Cody's a patient guy. That's why he's the youngest, most handsome CEO of a beauty brand of all time.

1:38:30Olivia Kory:I mean, this guy's crushing the game. He's so patient.

1:38:33Operators Podcast Hosts:There's patience. And one of my favorite test is when we ask people when they're coming in, like, how much of your business is driven by paid? Nobody really knows. And I'm like, you know, you can answer that question. Like, if you put all of your media in an all up test and run a holdout, like a 5 % holdout on everything you're doing, we can answer that question for you. And nobody wants to do it. I don't know, they don't have a, it's not very actionable, but I think it's like such a cool test. We've done it a few times.

1:39:02Olivia Kory:Yeah. So Connor has pitched it on me and it's like, I sound like we have the guts. It's like to choose. Do you really want to know the answer? That's probably the problem, Olivia, is that for one reason or another, people don't want to actually know that number. So, Olivia, what do you see on upper funnel versus mid funnel? And when would you advise somebody to think about one versus the other? Or is there a stance on that?

1:39:25Operators Podcast Hosts:Yep. So for listeners, we just covered mid funnel. That's the non-purchase conversion objectives. Then there's upper funnel. This is like true upper funnel reach and awareness objectives on Meta. These are underreported significantly by Meta. 4.35x incrementality factor. So what Meta is reporting in terms of ROAS, we're seeing about 4x that in terms of incrementality. So significant underreporting happening here. Much better omni-channel effects, as you might expect, much better latency in terms of post-treatment window lift. I need to pull up the stat. It's like if mid-funnel is 14 % worse than purchase, it's like 26%, I believe, in terms of...

1:40:13Operators Podcast Hosts:So mid-funnel is a little bit worse in terms of efficiency, and upper funnel is a little bit worse than that in terms of performance. But you have to remember why you're doing this. You know, like you you shouldn't expect it to be this like it parity in terms of efficiency with with lower funnel. And Cody, you've been pushing us there in terms of like then what should our goal be? And I don't have a great answer to that, but you do need to you need to be patient and you need to be willing to kind of carve off some of the budget that you can't hold to the exact same IRO as goals as your lower funnel.

1:40:49Olivia Kory:So like when, and honestly, I've been asking myself the same thing, but like, as we are trying to, you know, improve our meta performance and reach new people, we're probably, we have a roadmap where we're going to test these mid funnel. So we're going view content with 10 % of our spend at the same time, quiz complete 10. That's what we landed on today. But we're also probably going to test depending on how that goes, like back to like a reach campaign, which we ran three tests on and we've had some success with, but like, when should brands think about this versus reach? Or do you think if you're just on purchase, like start lower and don't go straight to the top if you haven't tested a mid funnel event?

1:41:21Operators Podcast Hosts:I would first test your lower funnel. Like a lot of brands don't even need to go here yet. So I would say like this is, you know, this this is not a problem unless it's a problem. Like test your lower funnel, you know, and then if you, you know, as Sean mentioned, like they were running into incrementality problems. You guys have probably been spending on meta for a decade. you know hard to reach new people if you're not happy with your rolling reach like you mentioned Cody that's worth a test and then this is a journey I would say like the mid funnel upper funnel is not like just light it up and you'll see immediate success it's kind of like this thing where you have to try and tinker with it in terms of the right events that work for for your business and whether you you know like you is it actually not that much better to optimize to an add to cart Are you not doing the thing you're trying to do by introducing the brand to new audiences by just going like a little bit shallower?

1:42:17Operators Podcast Hosts:Like, do you need to go all the way up funnel in terms of reach? This is where I'd say like you kind of need to test that for you in your own business.

1:42:24Olivia Kory:Okay. I've spent a quarter billion dollars on Meta. Okay. $250 million in a decade. I still don't run reach campaigns. So, you know, some quick math for everybody. zero times 4.35 is still zero. Okay. That's the incrementality factor. Mid funnel, I'm having great results. So if you've spent a quarter billion dollars on meta ads and you're still just running ASE, it's probably time to test mid funnel. If you're probably around a hundred million in lifetime spend, maybe just get on house and just test if ASE is working and then you can work your way up. But I think once you cross, I don't know. I don't even think Jason is running upper funnel reach campaigns yet.

1:43:07Olivia Kory:So I think you've got a long way to go. That's because it's all being run by corporations that don't know what's going on. Yeah, dude, McDonald's has the snack wraps they want you to know about. They're going to run reach campaigns.

1:43:18Operators Podcast Hosts:There's a reason I put on this slide that mid funnel is seeing like a boom. We're seeing 121 % increase. I think demand for upper funnel is waning. But it's not, it's honestly, Sean, it's not as bad as you say. But yes, it's probably a big leap to go there.

1:43:34Olivia Kory:Okay, so 0.001 times 4.35. You can do that math yourself. Sean, I've spent up to$250 ,000 on Meta. Am I ready for upper funnel? Yeah. Yeah, Mike. I think you should just let ASC rip for you, bud. That's actually my marketing strategy. Guys, here's how we're going to fix this. We're going to start at the top of the funnel and just work our way down. Yeah.

1:43:58Operators Podcast Hosts:So we have to go to the, this is, I've made you all wait for probably the most, sought after stat in the industry right now on incremental attribution. So Meta rolled this out of beta a couple, a few months ago, I believe. And this is where you can say, you can tell Meta, I want to optimize for incremental customers. And they've got to give them credit here. They've trained an algorithm to go find incrementality because they were playing chess a few years ago and they have a bunch of conversion lift studies that brands have run. They have enough data to actually train a model on incrementality, which no other platform can say, because no other platform has been really pushing incrementality.

1:44:40Operators Podcast Hosts:So they're many steps ahead just in terms of even being able to offer this. That said, we still have a ways to go. We're seeing very limited sample, we only have a handful of tests, this is a brand new feature. But just a 43 % success rate in terms of incremental attribution, being more incremental than BAU or standard attribution. So I'm optimistic this will get better. I think this is just amazing that they had the foresight to run the test that got them the data that enabled them to do this, but not there yet.

1:45:14Olivia Kory:I hear there's a new model. We've tested it once, like the old one. I hear there's a new model. Don't know if anything's going to get better, but just throwing that out there. Yeah. And like we said this earlier, it's like you have to have so much confidence to be like, you can measure me by any way and I'll still win. So yeah, hats off to Meta for embracing your mentality, embracing all this stuff. Also, Olivia, hats off to you for coming on the Operators Podcast and trying to get us to learn something because this is an hour and a half where I'm like, dude, I really, I really hope people listen to that episode.

1:45:44Olivia Kory:Listen, the 18 people that are still with us, congratulations. We're sending you a merit badge.

1:45:50Operators Podcast Hosts:They're going to be so mad that they had to wait till an hour and a half in to get that stat. I'm sorry to the listeners.

1:45:58Olivia Kory:Olivia, five hours in with all the fulfill ads so this is this is going to be five hours into the content okay hit the summary slide what hold on one can i say one thing on incremental attribution so the one one thing and i'm gonna get in my soapbox here i saw like people tweeting about this and they're like this is crazy update like you can now see which amount of your conversions like wouldn't have happened anyways and i'm like that's that's not what this is like that's just incrementality in general like this is actually being able to optimize towards that and i think even at the meta summit a lot of people that just kind of missed that aspect.

1:46:32Olivia Kory:So Sean, I agree with you completely that it's actually so cool how Meta is embracing incrementality and leaning into it and say like, hey, come audit us and no other channels are doing that. But this is beyond that. And again, I'm not saying it works. We haven't tested it in a while, but we're going to again. But this is actually being able to say, hey, take that data and then optimize my ads towards that and actually only spend on the impressions that you believe are going to be incremental. So don't know if it's going to work.

1:46:59Operators Podcast Hosts:Yeah. Well, I was just going to say, this is their answer to the ASC issues, right? Like, okay, if they're really good at identifying intent, incremental attribution comes in and kind of plucks out like, okay, who are the people who are going to come in anyway? And it's training a model on that. Yeah.

1:47:14Olivia Kory:Cause that's the worst possible. Like if you, if you assume bad motives on Metta's part, it's like, oh, you're going to use the power that you have to go and find all the people that already planned on coming and buy my stuff later today and then just show them a bunch of ads so you can jump in front of that and get them to click their way into my website or something and you can take credit for it. And so it is helpful to have a sense that that's not what they're trying to do. They're not trying to use their learning to actually just go and put their stamp on all these conversions that you'd already earned, but to actually grow the number of conversions you're getting.

1:47:50Olivia Kory:Yeah, dude. I mean, I love the big blue beast. There's a reason why they're worth two trillion. Well, here's my craziest thought. I know this is like probably totally out of place in everything we've talked about. And I'm probably too far out in the future by saying this. But, you know, Sean and I recently had a conversation. I think I asked him. He got to see a top secret product that does not involve a screen or clicking. and I am of the view that we're at peak screen right now and that we're at peak click and that we're going to funnel people towards purchasing behaviors differently in what's coming, the age to come.

1:48:26Olivia Kory:Now, I don't know how quickly we get there, but it could be quicker than we think. I mean, we're already seeing disintermediation by like the AI models, but imagine a world that's functioning more on vision or other things, on voice, and that takes out the click. and this stuff matters way more in that world than it does today. Like I think we might right now at this point have the cleanest lines from intent, from like an action to purchase and being able to track it and attribute it that we're gonna get going forward. I think it's gonna get much messier going forward. So being able to measure like how something spiderwebs out a lot of the different channels and over time is gonna become more important, not less.

1:49:08Olivia Kory:There you go. There's future Mike and his prediction.

1:49:10Operators Podcast Hosts:That was deep. That was deep for 9.45 at night.

1:49:14Olivia Kory:Yeah, for 9.45, you know, listen, if we go another hour, I'll start really getting into the bag. Okay, Olivia, hit the summary slide.

1:49:24Operators Podcast Hosts:Summary. I mean, this is, I'm curious what you guys take away from this, but here are my takeaways, right? Like, oh, and oh, man, I waited for an hour and a half in to also shout out Tyler Horner, who actually did this analysis on the house team. He's amazing. He's the one who took all these research questions and actually dug into them. It's been months of work. So shout out to Tyler. These are our takeaways. Meta is delivering, but you need to test the tradeoffs here between automation and incrementality. And if you're investing a lot into meta, this is what I was mentioning, it's worth it to understand what ASC is doing versus your, you know, your BAU, because this could potentially be millions of dollars.

1:50:07Operators Podcast Hosts:So need to need, you know, they're delivering, but you need to test this for yourself and make sure you understand what's working for your own business. Learn to think of Advantage Plus, the AI product, as the intent identifier. So this is probably most beneficial for brands with shorter conversion cycles, but we need to monitor. We need to understand, OK, how is this? How does how does this need to be augmented with upper funnel, mid funnel strategies that actually help you create demand and not just collect demand for for folks in market? So it's my takeaways is, you know, like understand the role of each of these products in the mix and start testing.

1:50:49Olivia Kory:Yeah, look, I mean, my takeaways are Meta works. I would have zero money if Meta didn't work. OK, like Meta saved me from poverty. When Meta came out in 2014 or whatever with an ad product, it gave me everything that I have. I am living proof that it worked. It's cool that it continues to work and that you've tested 300 brands and whatever, a billion dollars in spend to prove that it works. But so, yeah, that makes me feel good. But it is changing, right? It's going to change every single week, every single month. And Cody's talking about testing stuff that he's already tested, right? And that's because this isn't a static organization, right?

1:51:25Olivia Kory:There's thousands of engineers whose full-time job is it to try to sell one or two more incremental wallets every single day, right? So they're changing the algorithm. You have to continue to test and change on top of it. And it is being okay with the rules always changing, right? Manual bidding is going to go away. Maybe you've built your career on being a manual bidding expert. The robots are coming for that job. Polish off the resume if that's what you're staking your career to. But then once again, always be testing that. And then I'm an upper funnel believer, but I am not a top of funnel believer.

1:51:59Olivia Kory:So I think mid funnel is definitely good. Like don't be running those reach campaigns, video view campaigns, if you can help it. But here's my prediction. I've specifically asked Simon, who's a VP at Meta for this product, but it's advantage plus, plus demand generation. So I call it demand plus. And so just this Advantage Plus campaign where 10 or 20 % automatically goes to top of funnel stuff and your performance does look worse, but it'll be the only thing in your ad account. You will own nothing and you'll be happy. Advantage Plus Plus trademark. Sean Frank, he actually has already bought the domain.

1:52:36Olivia Kory:So Meta, if you do this, you're going to have to go through him. The one thought that I would add is that five years ago, seven years ago, we've talked about this. You didn't necessarily have to be a PhD to run Facebook ads and do pretty well. because you could hit ROASs that were much higher than today. I think that that's pretty widely accepted, that ROAS has gone down over time as competition and demand for this ad space has gone up. And that actually incrementally understanding more about your business and the returns allows you to dial in your bidding. So when you can hit a five ROAS without doing anything, you don't really need to think much about this stuff because you're not changing your bids a whole lot.

1:53:19Olivia Kory:But when for us, like the difference between can we spend$10 ,000 or$20 ,000 depends on the way that we set our ROAS targets. And that depends heavily on incrementality, what's going on in other channels. This is actually really material. So that's my main takeaway is that knowing your numbers and knowing the impact of your marketing can enable you to spend more in a highly competitive market. And that's a competitive advantage. Hell yeah. Okay, Cody, final word on the pod. January 24th, 2025, Sean Frank tweets, is meta incremental anymore? Well, Sean, now you have your answer. Yes, it is. You only had to wait.

1:53:58Olivia Kory:You had to wait six months from your lips to Olivia's ears. I think you owe meta an apology. Oh, dude, I owe meta a lot, but they owe me too. Cody, thank you for being here. Olivia, thank you for being here. Mike, thank you for staying late. If you made it this far, you're a true friend of the operators. an hour and 45 minutes in. Awesome data. House.io slash meta to get this report sent directly to you. Thanks for Phil, Northbeam, PostScript, Saracenalytics, Rich Panel. Thank you, EFC, for making an awesome community. Ecommerce fuel. Go ahead and join. Guys, it's an amazing episode. 145 minutes into it or however long.

1:54:38Olivia Kory:Great job being here. That was fun. Awesome. Thanks, guys. Thanks, Olivia. Great stuff. Good hanging, guys.

1:54:50phots Riley