Why Machine Learning Sucks for Predicting Ad Success & What Big Media Layoffs Mean for Marketing

0:00All right. So we're going to kick this episode off by talking about how and why machine learning isn't what you might think it might be in terms of predicting ad campaign. success. So let me explain what that means first. And then we're going to go through a paper here. And then Neil and I, we're going to jump through a couple of different ideas that we have for today's topics. But basically let's say you're Facebook or meta. Okay. Let's say you're meta. Let's say you're Google. You have a huge corpus of data and you see a lot of people running billions of ads out there, right? So you have a sense for what tends to work and what doesn't.

0:34Now you would assume that these machine learning algorithms would have, would do a good job of predicting the next campaign success. Right. And like, I would assume the same thing too, because you work off of data, but here's an interesting thing. This paper, and I read part of this paper. I won't say I read the whole thing here, but Neil, can you see my screen? Okay. So here's what it says over here for, for those of you that haven't subscribed on YouTube, go subscribe on YouTube. Cause we're just going to get better and better there. But this tweet says fascinating paper. It finds that even with access to rich data sets of user level features, sophisticated ad engagement, propensity scoring techniques, fail to approximate the causal lift results of RCTs.

1:16RCTs just call it a test, right? So RCT just means like a test. That's all it means at the end of the day. Just think of it as an experiment. And in some cases, dramatically overestimate them. So this graph over here that those of you that are watching right now, this just shows like in the RCTs in red. So that's the original test, right? That's the control version basically and it shows these are the other like machine learning models and this is like another model over here but the wider the range is that the larger the discrepancy so you can see that the more these the the discrepancy starts to get pretty big um as you know as more and more let's call it um the the lift deciles like i'm not a scientist right but um these are the the the projected lift estimate.

2:03So it could be anywhere from look 3 % here to 20%. And if you have such a wide range, you're basically not predicting anything at the end of the day. That's what this thing is calling out here. I'm trying to really simplify it for people. And this is predicting it for upper funnel, mid funnel and lower funnel. And you can read this entire paper. I'm not going to read the whole thing here, but if you, I'm going to move to sharing this tab instead, you can download this PDF. Can you see this Neil? Yeah, I can see it. Okay, so I'm not going to read it, but I'm going to tell you who actually helped with this paper.

2:33So it was done by the Kellogg School of Management at Northwestern University. It was also done by someone on the meta ads research team. And then this is also done by somebody else at the Kellogg School of Management team. And so basically, the conclusion is, hey, even with all this sophisticated tooling and sophisticated machine learning models, at the end of the day, creativity cannot be predicted. That is the takeaway here. The other big takeaway that I would have to say is from all the years I've run ads, and I've been doing this for 23-ish years now, I've learned one thing that has been consistently true.

3:10And what most people do with advertising is they look to see what worked for the competitors or worked in the past, and they keep redoing the same thing over and over again. Similar to the concept of banner blindness, we typically see the best performing ads are ones that are unique and are doing something new and fresh that others haven't really tried. Or if they tried it, it hasn't gone mainstream yet where everyone's just doing it. So these new type of ads, these new type of strategies, they're the ones that tend to do well. And it's hard to use machine learning data to predict that. And I'll give you a prime example of this.

3:45so back in the day if you were selling a product like a bar of soap you would typically just put it in the grocery store or run ads on google saying check out this bar of soap doesn't leave your skin dry um you know has lotion in there has antibacterial stuff so you don't have to worry about getting sick as much yada yada yada then the way they started selling bars of soaps for new companies was hey let's type try to do those squatty potty type of ads so you guys may have seen like the loom ads or the squatty potty ads where they'll create a funny type of commercial that is very humorous like with squatty potty the unicorn uh uh pooed uh sprinkles rainbow ice cream rainbow ice cream there you go and that converted extremely well they generated millions and millions of dollars in revenue how do you use machine learning and ai and i'm not saying it's impossible i'm just you know i'm not that much of an expert in machine learning i'm not actually any expert in machine learning or AI.

4:45So I don't know how you actually input data and get it to predict what's going to work in the future when what works in the future is usually going to be something drastically different from everything that we've seen continually over the last 23 years in marketing. So here's the interesting thing. So, and then we can, we can move on to the next topic, whatever you have queued up next, Neil. But so it says, here's, here's the abstract over here. So at the top of the paper, it says, and by the way, the title of this, this, not this experiment, but I guess you could call this an experiment, is close enough, a large, close enough with a question mark, close enough, a large scale exploration of non-experimental approaches to advertising management.

5:24So you can just go Google that. But here's what it says. So with access to over 5 ,000 user level features, these data are richer than what most advertisers or their measurement partners can access. Overall, despite having access to large scale experiments and rich user level data, we are unable to reliably estimate and add campaigns causal effects. So same thing to what Neil's saying, same thing with what I've seen with our clients and our experimenting in the past. Like, sure, you can try to draw inspiration from other people. We always recommend creating a swipe file, looking at what other people are doing, but don't try to copy it word for word.

5:56Don't try to do the same thing. You have to add your own spin to it and that's how it's going to be successful at the end of the day. That's where creativity comes from. Creativity can build on what other people are doing, but if you try to copy it exactly, you're going to end up just chasing other people at the end of the day. And that is by default, not being creative. Yeah. And speaking of advertising, dude, is it me? Or do you feel like the whole world is turning into one big ad? I feel like, yes, the whole, you know, you, you explain more what that means, but I get what you're saying. Dude.

6:24Okay. So I was looking at the rocks profile, all right. And this is on Instagram that I was specifically looking at his profile. And when I was looking at The Rock's profile, I was like, okay, you know, I was like, what portion of his post actually promoting a company he's involved with or brand he's involved with or movie he's doing or anything like that, right? And 10 of the last 12 posts, no joke, were promoting a company. In essence, it's pretty much an ad. Now, I'm not saying he's getting paid for it because some of these companies he owns. But in essence, he's still promoting something. And it's not just The Rock.

7:03If you look at what you and I do on social media, we may put up mainly educational content that's not promoting anything, but we're hoping that it helps us build a following, which it has over the years, nowhere near the size of The Rock. And a portion of those people will be like, wow, people like Neil and Eric put out decent stuff. They're smart. What company do they work for? Oh, they have their own marketing agency. Oh, can I hire Neil or can I hire Eric? Right? And it's created business for you and I. and even though we're not directly saying hey sign up for our ad agency we're still indirectly promoting it and it's not just with that it's just like everything you go to airport now and and this has been like this for years now but you know it's becoming more and more popular in most terminals because i fly so much internationally you see hsbc on the ramp from the airport to the airplane because you walk on that bridge thing and you see hsbc ads everywhere like everything is literally turning into some sort of ad and then people are promoting stuff and you now even have ai influencers like lil mcquilla who's like made money from prada bmw i think gucci as well the uber app you know itself has ads in it if you take an uber some of them have those old school taxi cab, like triangle boxes on the top of their roof.

8:25And they're putting ads on there as well, or in the backseat of the Uber, you see a little screen with ads. Like everything is turning into advertisement. And I don't necessarily think it's a bad thing. I just think it's reality. But what's funny is, is I was talking to someone the other day, they're like, man, this is getting overwhelming. It's too much. I'm like, dude, instead of worrying about it or complaining about it because you can't really do much. Just jump on the bandwagon. Are you going to make more sales by doing what everyone else is doing? Or are you just going to skip out on the sales and the revenue?

8:57You can't do much by complaining. You can only do much by doing. So look, at the end of the day, Neil, do you know who Peter Drucker is? Yes. Remember that name? Peter Drucker. I have a book over here, by the way. Managing for Results. I have this over here. This is sitting right next to me. But anyway, Peter Drucker, he's like a godfather. old school management guy, right? And he's wrote some really good books. How to Manage One's Self is a really good one. It's only 50 pages. Anybody listening to that should read it. It's pretty simple. But my point of this is Peter Drucker said there's only two main things that matter with business.

9:31Neil, do you know what they are? No. Okay. So one is marketing. Huh? I was just saying, I'm assuming you're going to tell me. Yeah, I'm going to tell you. Here it is. So one is marketing, two is innovation. Those are the two main things that matter in a business. And this guy's done like a lot of stuff, right? Written a lot of books and all that. And so what we're saying at the end of the day is everything is marketing. And like, we've talked about how we believe there's going to be a rise in creator operators and Neil and myself, we're very much creator operators. So we operate our business, like that's 90 % of our time, but we spend maybe 10, 20 % of our time creating content.

10:08There's going to be a lot more of that because marketing, yes, marketing is getting harder, but still everything is marketing at the end of the day. So what do people need to do? People need to adapt or they're going to die as a business. So that is that. Are you good on that one before we move on? Yeah, let's move on. When I first started my business, the overwhelm was real. I didn't have the tools to help me scale. If only I had Shopify from the start to handle all the behind the scenes work. Shopify is the platform behind millions of businesses globally from huge names like Mattel and Gymshark to the smallest brands just getting started.

10:40The best part about it, it has everything you could need. Say you're ready to launch your own design studio. Shopify has hundreds of templates to create a beautiful store that matches your brand style. Or if you need help with content creation, Shopify has got AI tools that write your product descriptions, craft page headlines, and even enhance your product photos. Shopify lets you easily create email and social media campaigns to reach customers wherever they are, whether they're scrolling or strolling. If you're ready to sell, you're ready for Shopify. Turn your big business idea into with Shopify on your side.

11:11Sign up for your$1 per month trial and start selling today at shopify.com slash marketing school. Go to shopify.com slash marketing school, shopify.com slash marketing school. If you've ever built a website, you know how tough it can be to keep a strong design while ensuring site performance is fast. That's where Framer comes in and it totally changes the game. Framer is the design-first, no-code website builder that lets anyone ship a production-ready site in minutes. I recently built a custom landing page in just a few hours. Animations, fast load times, responsive layouts, all without writing a single line of code.

11:47It was easy to use, and the end result was polished enough to look like a developer spent days on it. One of my favorite things is that Framer's AI handles translations with a single click, so your site can be up and running in multiple languages almost instantly. And with real-time collaboration, your whole team can jump in and work together, no issues with versions or miscommunication. Whether you're starting from scratch or using one of their templates, Framer lets you focus on design while handling the technical side for you. Animations, responsive layouts, search optimization, and all the things that matter.

12:17Ready to build a site that looks hand-coded without hiring a developer? Launch your site for free at framer.com and use code MS to get your first month of pro on the house. That's framer.com, promo code MS. Framer.com, promo code MS. Rules and restrictions may apply. Okay. So next one. So Neil, have you read about this one? So there are, there've been some big media layoffs in 2023. So when you think about vice, when you think about Buzzfeed, when you think about like, there's just a lot, a lot of media layoffs, right? And I think it's been like the, the, the amount of layoffs increased by like 78 % or something like that year on year.

12:55And we want to talk about what that means for marketing in general. So I can start it off first and then we can go back and forth on this. But here's the thing, right? It's one, media companies, they're like the old media companies, especially the ones that haven't adapted, they're getting crushed because the new way of doing media, it might be like on TikTok or Instagram or on other mediums, YouTube, for example, right? Or maybe people are just starting their own sub stacks, whatever it is exactly. It's just, it's changing, right? The other thing is with the advent of AI, the cost per writer or the cost for writing in general, it's getting a lot cheaper, right?

13:31And so people are really sticking around with the best people that they have. And maybe they're letting go of the people that are maybe B and C players because they only want to focus on the top tier. So those are just two things that are leading to this. What else you got, Neil? Dude, with BuzzFeed, have you seen their stock? It went from$10 when they came out, somewhere around$10, nine and change, or 10, whatever you want to call it, to now 25 cents as of today. Oh my God. you know and to give you idea oh my god dude down 97 percent in five years uh-huh they have a 35 million dollar market cap it used to be where they had a 1.4 billion dollar market cap this is like it's ridiculous on how much it dropped but the reality has kicked in one a lot of these media companies are really relying on one or two channels to one of the algorithms, you know, well, this is actually one still when some of the algorithms change, they get crushed.

14:31And two, they were growing by spending tons of money and they weren't really profitable. And at the end of the day, people were like, you can have all this revenue, but if you can't eat profit, who the heck cares? It's just a lot of overvalued companies and they've taken beatings and continually will take beatings over time. Dude, check this out. So look, media companies have slashed over 20 ,000 jobs in 2023. Okay. And then here's another one I'm reading from Poitner. I guess this is Dave Poitner, maybe, maybe not. But 2023 was the worst year for news business since the pandemic. It's a sharp reversal from the last two years when layoffs fell and some news outlets even expanded.

15:08So CNN laid off hundreds of employees. Gannett cut its division by 6%. Washington Post, NBCUniversal, ABC News all announced layoffs. and look dude hold on let me just share my screen this is crazy here hold on hold on hold on so do you see this yes okay check this out so ad week cut 10 10 or so alabama media group more than 100 bar still sports nearly 25 so 100 people or so um cbs owned station 17 is probably more in that. And then 16 % at Coindesk, CNBC, Conde Nast, about 270 people. I think this number is going to continue to rise. And this just goes to reinforce for everyone that if you don't adapt your marketing, especially in this rapidly changing market, you're going to be left behind, right?

15:56And nobody wants to have layoffs happen at the end of the day because it's really painful for your people. It's also really painful for your business. the funny thing with cnbc is it when the economy gets bad people still watch the stock market channels like cnbc and bloomberg you know a lot just because they want to know what's happening when it's both good and bad so they're still getting tons of viewership yeah but this is just a lot gant has been crushed over the years that company has been struggling for a long time they haven't figured out how to get that yeah whatever you want to call it i know they bought uh word stream.

16:32But they've really struggled to figure out what they're going to end up doing over the years. Yeah. Look, I mean, we're not trying to be doomers here. We're just saying, hey, we're calling things out as they happen, and we're trying to help you get ahead of it. And that's all at the end of the day. So look, if you enjoyed this episode, please don't forget to rate, view, subscribe. Subscribe to us on YouTube as well. It helps us a lot. We're planning to push a lot harder into YouTube. And yeah, that is it. See you tomorrow.