Bonus: How to Build a Testing Roadmap That Drives Real Profit—Not Just Revenue with Drew Marconi

0:00Drew Marconi:You spend all this money and time driving traffic to your site, and we help you find ways to get the maximum profit and yield from that traffic. In the last four and a half years, I have seen a lot more openness to treating pricing as a business lever. Pricing needs to be broader than just the list price, and this is a drum I've been beating for a while now. The reason you test is to learn more about your customers so you can find opportunities for incremental profit. The way that you learn is through statistical sampling. Your visitors, your traffic is the limit on how much learning you can do.

0:36Drew Marconi:It's your budget. If a visitor comes through who was not tested on, you just kind of like wasted some of your budget.

0:50Drew, welcome to the D2C podcast. We're going to do an amazing deep dive on the state of A-B testing. But first, I was wondering if you could give me a bit of your hero's journey. What brought you into Build IntelliGems?

1:05Drew Marconi:Yeah, kind of a funny journey. Not where I expected to be. I started my career in consulting. So I was at McKinsey for four years. I spent a year as chief of staff to the CEO there, did a bunch of education technology work, a little bit of pricing thrown in, followed a colleague to this company via transportation that was doing consumer ride sharing. They're actually about to go public selling that sort of technology to cities. The time we were a consumer, we were doing flat prices and they're like, hey, go figure out how we should price. I was on the growth team, so we were running a lot of promotions and then got tasked with this large problem of determine how we dynamically price.

1:46Drew Marconi:And that's where I actually got paired up with my co-founder who took it on from the technical side. So we spent four years there building out things like surge algorithms and dynamic discount competitive price tracking we decide we want to start our own thing we're like it's going to be dynamic pricing for mobile games spent three months on that pretty quickly realized it was not for us and we had been like well e-commerce people people must know how to dynamically price and discount like this must be a thing already and then we actually started having conversations and we realized that people you know were barely ab testing anything let alone the dollars and sense of their site.

2:24Drew Marconi:So that was like early 2021. We started having those conversations in e-com. IntelliGEM started with price testing. Then we added shipping and offers. And then, hey, we're going to let you content test and do the classic CRO stuff. And now we're adding on personalizations and AI. So it's been this very iterative journey to get here. And so at this point, how do you describe the core problem that IntelliGEM solves? we help take the traffic that you send to the site you spend all this money and time driving traffic to your site and we help you find ways to get the maximum profit and yield from that traffic so we help you run tests that figure out what's going to drive more profit per visitor we help you build personalizations that discover ways to get more profit from different types of visitors We help you look at that entire journey, not just, hey, let me change the button color.

3:22Drew Marconi:Let me change around the placement of things, but widen the lens and say, how does pricing impact this? How do my offers and discounts and promotions impact this? How does my shipping impact this? How does the visual layer, like sometimes, you know, moving around the buttons does matter a lot. How does my copywriting matter? my checkout page and give brands kind of the tooling to iterate, modulate, and measure what they do there. Pricing, I think, is such a... It was interesting that you kind of bit that off to start because I think it's something that a lot of brands, they kind of put their finger in the wind to begin with.

3:55They kind of set their price. And now there's pressures, obviously, with tariffs where brands are considering raising their prices. So I think pricing is like such a highly relevant thing. We talked in the pre-interview about Delta 2, who announced their dynamic pricing, which is not really what you're talking about. But I think that's an interesting avenue as well, where there was a bit of a backlash for them sort of proposing that they're going to be charging people based on external data or something. What do you think, like, what's your statement, maybe on the state of pricing in the DTC space right now?

4:26Big question.

4:28Drew Marconi:Number one, in the last four and a half years, I have seen a lot more openness to treating pricing as a business lever. People realize it has been a very volatile few years. Your margins are under attack. Pricing needs to be a tool in your toolkit to like navigate choppy waters and like be excellent at operating your business. 2021, 2020, when we're having initial conversations, people are like, I haven't changed prices in five years and I'm not going to because why would I? Now everyone's like, how can I get more on top of this? So I think, number one, people are seeing pricing as a lever that they need to actively manage, or they should be if they're not already.

5:12Drew Marconi:Piece two, I think this is what I wish to be true. I don't know how caught up it is, but pricing needs to be broader than just the list price. And this is a drum I've been beating for a while now, but your discounts that you are giving are part of your pricing strategy. Your shipping rates that you are charging are part of your pricing strategy. Your return policy is part of your pricing strategy. And, you know, I think people hear pricing and they think about the list price and all of that stuff needs to be in conversation with one another. And then third, yeah, I think tariffs were like the biggest shock to the system.

5:49Drew Marconi:I've heard more people ask, how should I price this year in the last six months? And I had like the previous three years. Interestingly, I mean, we've seen a lot of price raises. It hasn't resulted in raises for everyone, but it's been a forcing function to say, hey, I'm going to take this seriously. I'm going to go get data instead of vibes about how to change this. And so we're starting to see, yeah, a lot more teams like approach it analytically with clear ownership. Do you think some brands are afraid of that like Delta backlash about that idea that if a customer sees a price that's higher and then goes back and maybe sees another price or something that you're losing brand equity or pissing customers off, do you think brands are afraid of that?

6:27And should they be?

6:28Drew Marconi:Yeah, I think dynamic pricing sounds like a dangerous term. And I think there's a lot of fear around changing your prices. As a business owner, you're like, hey, if I change my price, my customers are going to be mad at me and my business is going to go away. It's very anxiety inducing. I feel like being a pricing expert is at times like being a therapist. What do you think it's worth? What do you think it's worth? Yeah, like, let's just go gather a bunch of opinions. We hear concerns from people of, oh, if I change my prices, my customers are going to be mad. If I test prices, what happens? Will people notice?

7:09Drew Marconi:And I think it's usually the fear is not well founded. It's other things coming. It's fear. you know we've had I think 600 million shoppers go through price tests very minimal issues where customers have you know reached out or there's been kerfuffle and what I tell the entrepreneurs I work with is like okay you can do one of two things you can test it and get data and understand how people will react to these price changes and make yourself equipped for these decisions or you can YOLO it and be subject to the same risks. So would you rather be smarter going into this risky territory or have less information?

7:50Drew Marconi:And like that often kind of gets conversation through. Another differentiation I think I spent a lot of time talking about with customers is like there's dynamic pricing. There's, hey, we're going to be changing the list price constantly. And I think that's where like Delta, you know, got some blowback or Wendy's last year. But then there's like dynamic offers and discounting. And like, you know, my wife and I can be sitting on the couch and both get Uber Eats offers. We're just going to use the better one. We're like, oh, great. Yeah, it's a good deal. So I think there's like dynamic pricing is this big, scary turn.

8:20Drew Marconi:But when you start breaking it down into, hey, are you open to running a test to measure elasticity to make better pricing decisions? Yes. Are you open to segmenting your audience and deciding who gets different offers? Yes. Are you open to having a variety of shipping rates and thresholds depending on what that customer has in their cart or where they are? Yes. And that decomposing of this big idea often makes it like people are quite comfortable with the bite-sized components underneath. Maybe go into a bit more detail there and imagine either use a brand as an example or an imaginary belt brand that I'm creating to walk through what that process looks like.

8:57What do people bite off first? Is it simply a segmenting their traffic and sending them to two different offer pages with two different prices? How does that and then how would it unfold from there to test beyond just the list price?

9:08Drew Marconi:Yeah, yeah. So let's take your belt brands, Eric's belts. You're starting with belts. Maybe you have a few other accessories. Maybe you're selling Crocs type gibbets. that stick on to those belt buckles. I think my first push would be what's the strategy? Like what's your most important thing that you're going to accomplish this year? Is are we trying to scale and be break even on the first purchase, but get as much new volume as possible? Are we trying to just, hey, this is a side business. You want to make as much profit any given year as possible. It's like, yep, I want low maintenance. I want that.

9:42Are you trying to do a subscription gibbets program?

9:46Drew Marconi:and that's actually the most important thing to get off the ground. Or, hey, you're going to launch hats later this year and we want to prepare customers for that. So a good testing roadmap starts with a good strategy. Like, what do we think is important for driving this? From there, we're going to decide what makes sense to take off first. And I think in a lot of cases, like the levers we're going to look across are pricing, shipping, which is going to impact every order and AOV and conversion rate, the offer, And that could be the mechanism of the offer. Is this free gift? Is this percent off?

10:20Drew Marconi:Is this dollar off? The messaging of the offer. This is typically for new customers. What are you telling them? How is that landing page happen? And then the investment amount, like what percent back are you giving? So it can be, hey, I need a great hook that gets new customers in the door. Or it can be the UX. People don't understand how special these belts are. People don't realize how valuable they can be. People don't know why you'd put gibbets on a belt. we're going to go into that. And so we'd start to prioritize. In some cases, we're going to do sequentially, maybe we have the ability to like, go across, let's say we pick pricing, what I'd probably do to start with you is say, Hey, let's, let's start with the core belt collection, you've got a long tail of, you know, NFL team branded belts, but you got this core selection that is totally in your control worth doing, we're going to take that and we're going to run a straddle price test.

11:11Drew Marconi:We're going to do 8 % lower prices, 8 % higher prices, and we are going to measure the elasticity. When we lower it 8%, how much does conversion change? What does that mean for AOV? What does that mean for your profit per visitor? And similarly, when you raise it, what happens? And so we're looking for profit per visitor, you send 1000 visitors to the site from your ad campaigns, at these different price points, when you take into account the conversion rate, the AOV and the margin percentage, how much profit are you making from those thousand customers? And like, that's going to be where we start.

11:46Drew Marconi:It impacts everyone. It gets big things from there. It gets iterative. Hey, higher pricing is working. Let's test even higher. Oh, let's go to that NFL collection. As we've changed this price, our shipping threshold needs to change. And you can often be doing content tests alongside, like change your copy, change the layout as you're doing these more commercial tests but um i mean there's there's that's one example um that we totally made up but like different strategy different catalog profile can mean a very different um starting approach this belt with gibbets idea is really starting to grow on me you've got that you've got that bracelet that has all those bangles that moms end up getting with different yeah the pandora bracelet the pandora bracelet we need a pandora a belt for dads.

12:32Drew Marconi:It's like, yeah, like people do love belt buckles. So what if you could just get like eight belt buckles on your belt? I think we got to start this. This is, we may have a brand. Yeah. This is why we might've just done it. I, and I love this idea. Everyone's looking for incrementality, true incrementality. Right. And so you can, you know, when it comes to your actual contribution margin on a product by product basis. So talk to me a little bit about like the kind of like, when, when you're talking about doing testing here? Are you talking about doing it sort of across the full funnel? Are you selecting traffic sources to say, okay, we're going to do this with our meta traffic.

13:07And then on the, you know, Google side, we're going to do something different. Are you, how do you, how do you think about breaking down the segments that you're doing the tests on?

13:14Drew Marconi:Yeah, it's a great question. And that's like, you know, how do we get to personalizations? You know, different customers from different channels and different sources are probably going to respond to different things. We have a whole more targeting possibilities and rules than anyone could ever use. I like to recommend people start broad. You know, like let's say you're running acquisition for this belt brand. Take all of your paid traffic. Like what you are trying to do is stay break even and acquire as many customers as possible. So isolate to pay traffic, the people that you are driving to the site, run the test on everyone.

13:52Drew Marconi:And maybe we're playing around with the presentation of the offer. Like it kind of ends up the same amount, but we're messing with that landing page. Run that for two weeks. We're then gonna look in the dashboard and see not just what one overall, oh, variation C was the best, what one per source. Was there a different winner for Google or for Meta? was there a different winner actually for your, you know, like retargeting campaigns versus your true prospecting campaigns? And as we find those sub segments of customers where they actually behaved differently, maybe the retargeting folks responded super well to a free shipping offer, whereas the folks coming from Google just needed like education and some free informational material, like a free gift was what they responded to.

14:46Drew Marconi:you can find that in your data and then say hey i'm going to roll this out as a personalization so people coming from this source i'm going to make sure they have this offer this experience on the page that's congruent with what they saw in the ad or in the campaign and this other group over here is going to have these and building like a kind of a rule-based personalization system that it could be as simple as just changing the image or changing the hero copy or it could be as complex is like they get an entirely different offer structure and discount price but it starts taking the broad data and then looking at your sub segments it fits into your three ease framework which uh jogged your memory on last time explore experiment and extend so you start you could you just sort of maybe start with a segment and then extend it out yeah yeah you go explore the data come up with your hypotheses don't just yolo a test because you saw a screenshot on twitter explore your own data, come up with hypotheses, experiment with it broadly, and then extend your learnings to different segments, push out these different versions to the people who respond well to them.

15:53So what are the tests that people are doing right now? I don't know if you have any specific examples, but for really dialing in their Black Friday offers that they're going to be making this year, because I think that's on everyone's mind right now is how to optimize and make sure that you've got the pinnacle offer and set up for Black Friday, Cyber Monday.

16:11Drew Marconi:yeah uh well we just had a lot of people use labor day as a dry run and you know if you missed that then think about it for next year um or howloween i think too right how howl isn't really a shopping holiday for some brands um but yeah it's it's a time when people are expecting it to deal it's a good way to run one of these like do take the experiment approach of i'm going to try a few different mechanisms. Let me see about a free gift. Let me see about a straight percent off. Let me try a volume, like buy more, get more, spend X, get Y, spend 200, get this much back. So we saw a few people run straight up mechanism tests.

16:52Drew Marconi:Like, do people value a free gift more or do they value percent back? And there, you know, it's looking at, I think we had someone do free gift over 150 bucks or 10 % off over 150 bucks. And, um, you know, the free gift cost about$10 to provide. What we saw was the free gift actually like conversion rate ended up about the same in the two groups. The AOV was$10 higher for the free gift. People were more excited about getting that extra gift. It seemed like a higher perceived value, even though the cost of goods sold was 10 bucks or 15 bucks so conversion rates the same the aov went up people were more incentivized therefore the profit per visitor and total total gross profit went up so that was one case they're like oh great actually we should like go get a little more supply of this gift that we used um it was like a frother for a beverage company and make that a big part of our holiday plans another really interesting one i just saw and was debriefing on today this was in the apparel space, they have kind of three categories of their catalog.

18:01Drew Marconi:There's the best sellers drive a third of the traffic or third of the sales, pretty small selection. Core, you know, maybe another 20 SKUs also drive a third of sales. Then everything else, long tail also drive a third of sales, but it's like hundreds of SKUs. And they were kind of dialing in. They started with a couple different offer levels of let's try not to discount the best sellers, but get 10 % off tier two, 20 % off tier three versus 5 % on the best sellers and then tier two 20 % off everything else and as the weekend went they watched these results and actually turned to the dials to say oh shift more traffic towards the 5 % 5 % variation um and that was actually going at like the catalog level what do customers respond best to like which products do I need to get the discount on so But they were in there toggling the knobs as the weekend went based off what they were seeing.

18:57Drew Marconi:And that's something that we've seen people do for Black Friday. So, yeah, I don't know. Those were two I got debriefed on the team by today. I'm sure there were a bunch of other tests out there as well. Everyone wants to be able to test consistently, but I feel like people go in cycles. There's also different people in the organization who maybe have purview over different parts of whether it's you're testing the front-end offers or you're testing the back-end conversion funnel. What are the best brands sort of handling, creating an ongoing culture of testing and who looks after that in an organization generally?

19:31Drew Marconi:Yeah, often, I mean, the organizations that we see do it best, the VP of Ecom, like whoever owns the site, has made testing a priority. And they have found the tool and they've tasked someone to be in charge of that. Now, that may be a combination of people, but there's a senior advocate, VP of Ecom, or the founder, and then clear owners for these things. We talk about widening the lens of what's testable. It's not just the UX. It's also the price and the offer and the shipping rate. I keep saying this list over and over again. But in that Wednesday evening, maybe ops team is setting the shipping rates.

20:06Drew Marconi:Maybe your acquisition marketer is doing some offers and your retention marketer is doing others. So they need to bring that group together and say, hey, we're going to have a testing approach. I want each of you guys to be doing this. And we're going to like meet weekly and talk about new ideas like that weekly experimentation meeting usually is what new ideas do we have? Let's put them on the board. what tests do we have active do they seem good are they ready for analysis if it is let's analyze tests that are done decide what to do and then let's groom and decide what to test test next let's like groom that backlog all these ideas and they're typically looking at an ice framework which is what how easy is it to implement how confident are we that this is going to be a winner and what's the impact if it wins and using that to prioritize the tests.

21:00Drew Marconi:So you have different people running the tests, but you're talking about it as a group. You're making sure that it's a cohesive strategy and then you're looking at the same metrics. One of the reasons I really like profit per site visitor as a metric is it puts all of these different initiatives on the same footing. Like the person going and setting prices is looking for incremental profit per visitor in the same way that the person building a new in-cart upsell experience on the development side is looking for incremental profit per site visitor in the same way that maybe you're adding a site performance tool that speeds up the site, we can test it and look at the impact on profit per site visitor.

21:42Drew Marconi:So everyone is held accountable to incrementality and looking at that same metric. And then, yeah, I mean, I know immediately when I pull up someone's dashboard, whether they're one of these teams because they have clear naming conventions in there. They have like dates and good data about what's being tested. And I see multiple users logging in. When I see that in a dashboard, I'm like, this is a high performing team. Like we don't even need to talk to them. They're going to go get a ton of value through their testing program. And approximately how are they thinking about like the number of tests and maybe the frequency?

22:13Like are they, how often are they testing pricing? Is it a continually thing or do they test it quarterly? like what does the testing regimen look like for one of these successful teams?

22:23Drew Marconi:A test at minimum is always life. Like the reason you test is to learn more about your customers so you can find opportunities for incremental profit. The way that you learn is through statistical sampling and information. That means that your visitors, your like traffic is the limit on how much learning you can do. It's your budget. And so you need to be spending that entire budget. If a visitor comes through who was not tested on, you just kind of like wasted some of your budget. So the best teams almost always have at least one thing going. At a certain scale, you can actually have multiple things in parallel if you have enough traffic and run things mutually exclusively.