Bonus: How to Prove What’s Actually Working in Your Ad Spend — The Incrementality Mentality with Stella's Brenden Delarua

0:00Your ROAS does not mean ROI. What a lot of brands don't realize is most of the time with channels that they're on, they might already be at a point of saturation, meaning if they spend more, they're going to start getting diminishing returns.

0:16Brenden Delarua:In the incrementality industry, I just think it's so confusing. It's over a lot of marketers' heads, and that's why I think a lot of companies are charging so much for it, because they're saying this will achieve more incremental revenue or profit for you. It's like a black box for how it works. Like, don't worry about the data models. Me and my co-founder, Vinny, built Stella to kind of democratize this space and make incrementality testing much more accessible and easier to understand and much more affordable for D2C brands.

0:52Brenden Delarua:Brendan De La Rua, welcome to the D2C podcast. You are a, at least a triple threat. You are a media buyer, you are a SaaS builder, and you are an entertaining and educational TikTok creator. I've been enjoying your channel, sir. Thank you, Eric. That means a lot coming from you. Well, welcome to the D2C podcast here. Let's start with why. Why did you build Stella? Oh my God. I mean, that could go forever, but just to make sure I'm keeping my answers short and sweet here. I mean, I was a media buyer. In general, I'm just an anxious person. So I love to get good results for my clients. But like weekly client sinks, whether results were good or bad, I was nervous, just because I could never fully feel confident in explaining why performance was the way it is.

1:38Like performance is good. Now, I don't make a change next week, it's bad. And I'm going to go on this like, weekly sink with this like, well respected, like marketing leader, just gave me a lot of anxiety. And I think what it was, was sometimes I would smash all my ROAS goals as a media buyer. And I'd be in a board meeting or meeting with the business leaders, talking about how profit from ads is like the highest it's ever been. And then I'm getting checked on like, well, our actual profit in our P &L is like the lowest it's ever been. So something's not adding up here. And like, what could it be?

2:11And those types of conversations have always like, taken me aback and being up all night researching how to fix this, how to not do this again. And eventually, the answer I stumbled upon was incrementality of finding out what's actually causing growth to happen. And it's become such a game changer from becoming a media buyer and knowing about incrementality to explain the causal relationship between ad spend and business outcomes. Because right now, they are just not correlated. your ROAS does not mean ROI, right? So finding incrementality and understanding how to do these like causal analysis, these lift analysis or holdout studies, inverse holdouts has really helped overall media buying.

2:56Anyways, so I always gravitated to it as a media buyer. And then I noticed the current tools in the market are just super expensive, super hard to use. Even if you're spending 12K a month on them, you might still need a data scientist on your team to actually run them and interpret them properly. So me and my co-founder Vinny built Stella to kind of democratize this space and make incrementality testing much more accessible and easier to understand and much more affordable for D2C brands of pretty much all sizes, but mainly mid-market.

3:29Brenden Delarua:You mentioned you can't trust your on-platform ROAS. That's been a trope as a media buyer for years, pretty much every platform you're on. Where do you see media buyers may be going wrong the most right now when it comes to, I guess, overspending or underspending? Yeah, I mean, I think there's like a inverse relationship between like the traditional marketing funnel and causality. For example, like top of funnel campaigns are typically the most incremental, where the bottom of funnel campaigns are the least incremental. So like the most common use case you'll see or people talking about on LinkedIn when they think they're finally understanding incrementality is like branded search is just not incremental.

4:06Even though it has the highest ROAS, it has a 5x ROAS on platform, right? Yeah, exactly. And if you wanted to optimize for ROI, if you're a media buyer like I was, and you're getting pressured by the CFO of a large publicly traded company to get better returns, you see this 50x ROAS of branded search. You're like, I'm just going to start investing more in branded search. Because it makes sense. But those are non-incremental. I mean, there's some cases where branded search makes sense. Like when you're in a really competitive industry, if your competitors are on your branded terms, it makes sense to keep it live.

4:39But most of those conversions would have happened anyways, because people already have the intent of searching for your brand. They're probably going to just scroll past the ads and get to the organic listing anyways. And then on that, it's the same thing with retargeting, right? Like retargeting is typically the highest ROAS, but usually the least incremental. But the inverse of that is like CTV and YouTube are typically very incremental, what we're seeing. And when I was a media buyer, those were the channels that would get cut first, or what I would call like a spillover channel. After we're spending too much on Google and meta, like what's our channels that we're going to spill into?

5:14And usually it's like CTV, maybe YouTube. But now if I were to be a media buyer again, I'd probably be very bullish on CTV, OLV, linear, and YouTube based off all the incrementality studies we've been doing.

5:25Brenden Delarua:Okay, walk me through how a brand who maybe is spending on meta and Google, and they've just launched their streaming or CTV campaigns. Walk me through using Stella, how someone determines that, oh my God, this thing that looks like low ROAS is actually driving people further down funnel. Yeah. So to set up a study with Stella, we tried to make it as easy as possible. We still run into some use cases where users are reaching out to us, asking us to kind of explain certain things. But the most important part of a holdout study is selecting the right locations, which a lot of people kind of skip over.

5:59A very common scenario we see is in the US, people will pick like 25 states on 25 states off. And that's just arbitrarily picking locations with no correlation. So the first thing Stella recommends is like you upload your data to Stella and Stella will look at the last 120 days of your data and tell you the best regions to activate CTV in if you're activating CTV ad. So whether we're doing a state level or DMA level or city level or zip code level, Stella will tell you the best regions to compare to have a high correlation from the start, even before we add in like a what's called a weighted synthetic control.

6:34So we have very high correlation. So basically, you would activate CTV in only certain regions, what like whatever Stella has told you to to activate in, let's say for like example, it could just be like California, Oklahoma, and I don't know, maybe Tennessee, it would say activate CTV, we'd activate in those three regions. And we would look at basically total revenue. We're not looking at in platform metrics. I don't care what the CTV vendors claiming that they've driven. I want to look at like Shopify, or WooCommerce or whatever you're using, I want to look at your source of truth in those states to see if there is a statistically significant lift that is caused by nothing other than that intervention, which is activating CTV.

7:17And that's what we'll see. And typically, you also see conversion rates increase on meta and Google. But regardless of how the sale happened, whether they clicked a meta ad in Google ad, if they're influenced by CTV, Stella will be able to pick it up in the final analysis. Interesting. And then what role do you place on post-purchase surveys? Oh, I love post-purchase surveys. I feel like sometimes I have to write like anyone in the D2C space, I have to write like Twitter posts or LinkedIn posts. And I feel like it's easy, like writing posts thinking of like an enemy, right? So I've had a lot of people ask me if I like believe in MTA still.

7:51And I'm like, I love MTA, obviously, coming from that background, I don't think incrementality should replace like multi touch attribution or post purchase survey. I think what they all all three of them, I think, fill a role in just like showing the full picture. I think MTA, it shows like 40 % of the picture. And for a lot of brands, that's enough. But obviously the issue comes when you're making 100 % of your decisions on 40 % of the picture. I think post-purchase surveys extend that picture to about 60%. They add an additional 20 % of the picture. So if you're making your ad buying decisions or business decisions based off MTA and post-purchase surveys, you're going to be right more than half the time, in my opinion.

8:35I think where causal analysis like incrementality or media mix model comes in, shows that additional 40 % of the picture that gets really muddy when you're a larger brand. When you're a smaller brand, you probably don't need it. But when you are making 10 million to 100 million in annual revenue from your D2C shop, that's when incrementality really becomes powerful.

8:57Brenden Delarua:What do you mean when you say always on incrementality? To me, that makes me think of like when you're investing in an ETF or whatever, how any dividends are just automatically rolled back into the fund. So you're always getting more efficient. So me and my co-founder are still on every sales call for Stella. And we get asked all the time from marketing leaders, not data scientists or CTOs or even CEOs, like they want a dashboard to log into and see like how incremental they're doing right now. And incrementality studies don't really work like that. How we describe them, they're like a snapshot in time, right?

9:30Like right now, as we're filming, this is October, right? So if we're doing a holdout study of YouTube in October, whatever incrementality factor we give back is going to be YouTube's incrementality of October of 2025 with the bid strategy that you have, with the creatives that you have, with the competitive market that you have right now. It's going to change next month in November when Black Friday, December, Monday comes. It keeps changing over time. But we kept getting requests for like a daily dashboard where people can log into and see how incremental their campaigns are. So my co-founder and I put our heads together like, how could we do that?

10:08Is that something that's even possible? So we've built a tool called Always On Incrementality as like a response to those questions or requests we were getting. What it does is it ingests your platform data from Shopify, Google, Meta, TikTok. We're adding in connectors right now. And it ingests campaign level data and it runs daily causal analysis on every single campaign that you have live. And when you first install the app, it pulls the last two years of data. So like if you ran a study, you know, a couple months ago and you don't know if it worked or not, Stella will be able to look at it and run an analysis.

10:46However, the caveat there is since everything's automated, we're very strict on what we allow to pass our criteria for validating a study. So some tests might come back invalid, but we show every test so you can see if there wasn't enough spend or there was other nuanced external factors that just muddied up the results so we don't want to show it. Anyways, it's a pretty cool tool. I'm excited about it. And I think the value in it is like Stella's AI. We have like a custom AI built into it that can really be your collaborator on like, Stella, what do I do? Like, what do I do, Stella? I don't understand any of this.

11:24Stella can walk you through exactly how to use these incremental insights to drive net new profit for your business, which is, I think, is such a win.

11:31Brenden Delarua:Was that always the case? Because, yeah, just sort of on Stella, heystella.com, looking right now, it really draws attention to the prompt as the main interface. Was that something you always wanted to do with this? Or did that just kind of come about as the best way? We try to keep our ear down to the floor when it comes to like the market of like what's happening and then like how we can pivot. And we're a pretty lean team and we pivot pretty quickly. But one of the things we were noticing where we would hop on calls with like agencies or marketing leaders who are like, why pay for an incrementality tool?

12:04We're already doing it ourselves. And then when we see what they're doing is they're just uploading CSVs to chat GPT and saying, these are our test locations. These are our control locations. How incremental was it? And then they're taking that insight and putting it into their decks. And we're like, that is not the right way of going about doing that at all. But we noticed it. We noticed it a lot. And we still do. So our idea was like, what if we made Stella to mirror like a GPT or a cloud, at least in the UI, to feel native to these marketers? Because ultimately, we're building a data science tool for marketers, not for data scientists.

12:40So if they are telling us that this feels native to use a GPT and they feel comfortable using a GPT, let's build it to make it feel similar. But with data models that are actually very accurate and have fallback models to make sure that we're providing real insights. Because ultimately, it doesn't matter how much you're paying for like whatever tool. You just need to be right, especially if you're going to move a million dollars of budget from one channel to another channel. And you're doing that off of bad data because that could obviously cost your client his job, his or her job, if you are allocating budget wrong based off wrong insights.

13:17So that's kind of the idea of why we've like focused on the AI component of Stella.

13:21Brenden Delarua:And then how do you, so the second big tool you guys offer is media mix modeling. Talk to me a little bit about your methodology there, maybe what you've updated from other players in the space that are doing the same thing. When we were doing media mix models at the agencies I've worked at, there's like open source tools, right? Like there's the two big ones are PyMC and Meridian, Meridian owned by Google. There's also one called Robin from Facebook. It's like a frequentist model. There's like frequentists in Beigeian models. Okay. Familiar with Beige? Oh, are you? Well, a little bit. But I had a startup called Tap for Tap that was like a tap exchange where it was like a click exchange for apps.

13:58Brenden Delarua:It was way back in like 2013. And we were kind of using Bayesian models to determine how to match apps up, essentially. Very cool. Yeah. So we've leaned towards Bayesian models. One of the reasons being is we can run an incrementality study. Let's say YouTube is like a 2x iROAS. We can put that IRO as what's called a Bayesian prior into our MMM to help calibrate the MMM, be a little bit more accurate over time with consistent testing and MMMs. But we looked at PyMC, we looked at Meridian, and we kind of saw good parts about both of them. A big common issue we hear a lot from Meridian is the baseline revenue.

14:36So if you turned off all media spend, how much money would you still make? A lot of people are getting negative baselines, which don't make any sense. It should never be negative. It might be zero, meaning if you turn off all ads, you'd make no money. Usually there should be some baseline there. But we really liked Meridian's budget optimizer and how they're forecasting. So we took inspiration from a lot of these open source models to build out a proprietary model. We also consulted with some pretty big names in the measurement space who've built large custom models for other companies. And we just did this big collaboration of building a model that is repeatable, accurate, is the main thing that we're going for.

15:17But also just very easy to use, especially for a generalized self-service platform like Stella.

15:24Brenden Delarua:So you talked about people just using ChatGPT with their data. Throw that in there. I'm sure there's probably lots of people doing that. That's the low budget version. Then you also mentioned that you guys are coming in at quite a bit of a lower cost than like what's available out there. Talk to me a little bit about what kind of investments people are making in these kind of tools and tell me about what's Telecost. I think what it is is like, so my co-founder and I used to kind of, you know, do these types of analysis for clients at an agency we worked at. or if they wanted, we would be the, or mainly I would be the one sitting on vendor calls with our now competitors, they who shall not be named.

16:01Um, but, and I always love the tools. I'm like, I never say like their tools are bad, like their tools are amazing. And this, the, what they're building at our competitors is like top notch. But what I would be tasked with is going back to the client with like, Hey, you know, they want 12 K a month for a six month contract minimum. or a year minimum. And that was always hard for the client to stomach. And sometimes it'd be like, yeah, maybe we don't need it at that price. And let's move forward. And then I just think the value prop of like, you know, use this incrementality tool to uncover like wasted spend and become more profitable.

16:38But it's ironic, the minute you sign for those platforms, you're immediately less profitable, like in by a lot. So there's that that side of like the the gap we saw in the market. The other side was I've done years of marketing for B2B companies that don't want to show their pricing on their website, that make you sit through like four rounds of like almost interviews. I was going to say interviews, but like sales calls to see if you're qualified to even know the price. And then by the time you know the price, you're like, oh, that's out of budget. If you would have just told me that we wouldn't have wasted any of this time.

17:08Or I've marketed products that like in the space, there was just a better product that was cheaper and better. and it was so hard to figure out angles to market products like that. So when we were starting to build Stella before we actually had a product, I was like, I want to make this thing easy to sell, like the easiest product to sell. I want to have like the highest accuracy, but I also want it to be the most affordable tool by far. Like I don't want anyone to compete with us. And that was like a very strategic decision. Like we did that intentionally. And I'm not sure if maybe there's like a perceived quality thing we're running into.

17:44But I think on demo calls, when we show people the tool or when they do a free trial, because we also offer free trials, they've realized like how powerful Stella really is. But yeah, so like Stella runs at$2 ,000 per month, just month to month. You can roll off any time. We just rolled out an annual plan that's just$1 ,000 a month paid annually. So$12 ,000 a year compared to$12 ,000 a month for Stella. and you get unlimited uses of the incrementality testing tool. Or for the MMM tool, it would be an additional 2K or 1K for the whole year if you want. But we did that very intentionally. So Art's been working for us.

18:20Brenden Delarua:Can you walk me through an example? Because I know you've used the tool a lot in your own media buying. Can you walk me through an example where either you or a client has basically ran one of these ongoing studies or one of these studies and made a pivot that really aided their business? Yeah. Yeah, I mean, the biggest one that comes to mind is like kind of what I explained earlier, like investing more in YouTube or CTV types of tactics, because the idea is the first step with an incrementality study is like a holdout. So you're activating CTV or you're turning off CTV in certain regions to monitor the eyebrow at your current spend level with your current creative.

18:54And then we start measuring, we do what's called a scale test. So it's like a three cell experiment where cell A, you keep spend the same in certain regions. cell b you increase spend by like 250 percent in in other regions and then cell c you would increase spend by like 500 so like a substantial increase and then you run those and look for instead of i roas or incremental roas you're looking for marginal roas but what this test allows you to do is allows you to start mapping out this like marginal growth curve so you can see how far you can scale before you hit a point of diminishing returns and what a lot of brands don't realize is most of the time with channels that they're on, they might already be at a point of saturation, meaning if they spend more, they're going to start getting diminishing returns.

19:42And they don't know it because the wall is kind of like invisible. So a scale test shows like, okay, CTV is very incremental, but we can really only increase spend by about like 50 % rather than 250 because we're starting to get these like diminishing marginal returns. So we've seen that play out very well for clients that have run the scale test and seen how far they can scale to maximize incremental revenue, where now they're just running CTV and they're super bullish on CTV because it is showing this like insane lift in their shop. Like, I wish I could show you a Shopify account, but you can just see kind of where they started investing a lot in CTV.

20:17It

20:17Brenden Delarua:kind of like spikes up like right there and kind of gains for a while. What about AppLovin? I'm always curious about what AppLovin is showing in the incrementality world. Yeah. So AppLovin obviously is like a newer platform. I've never run ads on AppLovin when I was a media buyer. We've done, honestly, a handful of AppLovin studies, maybe five or six AppLovin studies. And every single one, so an incrementality factor is a very simple equation of like iROAS found in Stella divided by platform ROAS. So for example, if your platform ROAS is a one in the iROAS is a one, one divided by one is one, right?

20:55So that's 100 % incrementality factor, meaning from our holdout study, it looks like the platform is reporting exactly the incremental contribution of that channel. With AppLovin, we have seen that like pretty much every single time we've run a study. So like we'll run a study for a client, we might not have access to AppLovin like the dashboard, we just get the spend data to run the analysis, we'll pull in some confounders to isolate certain variables. And then we'll send back, okay, AppLovin was like a 1.2. Like, what are you seeing in platform? And they're like, a 1.2. And I was like, oh, that's interesting.