The ultimate guide to Martech | Austin Hay (Reforge, Ramp, Runway)

23:12Can we build a financial model to show how much it's going to cost us both in terms of operational efficiency and actual real fixed and variable costs to like end up in that state? And then how do I create a graceful argument to persuade people that we should spend engineering time and resources? And usually that's how it's pretty clear. It's like, you know, if it's less than a certain amount, how do you justify spending any engineering time on it? You have to wait for the problem to become big enough. But then back to your other point around like, how do I give growth managers out there something useful?

23:42I would say like the big thing that people forget in an early stage of a company's lifetime is that the company will live out last to you, hopefully. Like, you will not be the last growth manager unless the company fails. So I tend to take a little bit of a different approach on most, which is like, I think you should always be thinking about the future. That doesn't necessarily mean you should make design choices that over index towards the future so much that you misproduct market fit or you make poor product decisions. But when you set up tools and you pick tools and you implement them, you should be thinking, what's going to happen a year from now if I don't change anything?

24:20And is this going to be a catastrophic situation or not? And then try to take actions to mitigate that risk. Some examples are like, you know, if it's $2 ,000 to get SSO and two days to set it up and that prevents you from having a security problem where somebody downloads all your users, it seems like a great investment. And guess what? Over time, if you don't do that, you're going to eventually have to hire an IT person to go and set a SSO for all your tools. So some of this is more of just like being a good steward about managing first with third -party tools with an eye towards the future.

24:49It's always a trade -off, right? Because the more time you spend when you're building product or really in a company's lifetime, that time could be spent on other things. So if you waste it, you know, managing third -party tools or sending up correctly, then maybe you miss out on a key product feature. So I think it is a tough balance to strike. Coming back to the different kind of roles within the growth umbrella. If someone has someone leading paid growth, let's say, and they're just like a paid growth person, do you also find a more tech person to work alongside this person? How connected would you be to someone that's just like responsible for paid growth?

25:24Maybe a key difference, too, because we didn't talk about this in the beginning, but there's like marketing technology and marketing operations. So in my mind, this is just my own kind of mental framework is, marketing technology has tech in it. So it's usually an engineer or somebody with an engineering background doing that function. Marketing operations is usually not always technical. It's maybe a system analyst or a business analyst could be somebody really, really smart, but they might not have an engineering background. So I think that's a key distinction, too. And you typically see that in B2B, where you'll have a MAR ops function, which is like setting up campaigns, sending email blasts, debugging, doing analytics work, SQL queries, all semi -technical work, but not engineering -based.

26:06So in my mind, when we talk about marketing technology, I'm really thinking of it as an engineering -based role. And even by background, I'm not a software engineer, but I was a civil engineer, and I learned how to program, and I went through a bunch of coding to get there. So that's my way into the engineering world. And you typically find that a lot with marketing technologists, in particular, is there are software engineers, where they've got enough experience to kind of moonlight as software engineers. And so we get to this problem set of like a user acquisition person. How would they rely on a marketing technologist?

26:40Well, I think like the most superhuman user acquisition people out there are engineers. And they just like, they don't need a marketing technologist because they set up the tool themselves. They know how the paid campaign runs, and they just do it all. And you'll typically find these superhuman's at small startups, where like, you know, the engineer is just told by the co -founder, hey, go figure out how Facebook ads work. And like, you know, superhuman is born. More often, though, that doesn't happen, or those people, once they do, once they never want to do it again. So you'll typically find the role split, right?

27:11And that's the natural thing that happens as you scale. You divide responsibility. And you'll see how the person who's responsible for bidding and acquiring users, and paying down those campaign costs, and you have the person who's in charge of, how does it all work? How do we get this thing to actually run? And that's very similar. We have a ramp. You know, we have an amazing user acquisition team. I know Shree bot she was on here a while back. He hired a guy named Cody Morgan at ramp, who has a user acquisition team. And the way to think of it is, like, my job is to help support them and running all their campaign needs.

27:40And when they have a directive from the CEO that says we need to improve CAC, or change any of our metrics, it's my job to partner with them to help them do that. And actually, one of the coolest and most fun projects that we worked on early when I joined Ramp is we were optimizing, we're trying to get top of funnel data all the way down to the bottom of the funnel and tie it with opportunity data. So we could send that back to the ad network. So I rather than like, you know, optimizing your campaign off of when a user clicks the button on the website, you're actually optimizing it off of, did the opportunity occur?

28:11And what was the kind of the ideal value for that opportunity? And you're sending that data as a synthetic event back to Facebook and all those guys. So it can be really cool and super advanced stuff, depending how deep on the funnel you get and how complex your business is. So you're generally not running campaigns every year on on say Facebook or adwords. You're mostly as a martyck person supporting people who are doing that. Yeah. Awesome. How have been them use tools and technologies to do it? Great. Do you find that to people give you goals, are you responsible for growth goals of your own?

28:45And in general, our martyck people should they have goals and growth goals on their plate or are they just there to support people who do? Oh, that's a great question. And I would like, maybe this is like at the end of the podcast, we ask people about this because I would love to know what is a better version of going. So there's two ways that I've thought of it. One is, my goals are directly tied to the people I'm serving. So if user acquisition has, we have, I mean, we do. We have a growth goal and we have a cat goal at ramp, right? So like, my goals are tied to them. So I'm going to help make sure that that is achieved.

29:18But then there's also like a cost and efficiency goal that I internally think is valuable. Whether or not the business thinks is valuable, it doesn't really matter. I just, I come from a sales background and I like to run lean and efficient teams. And so I'm always thinking myself, how much were the tools when I came in? How much are they now? Have I set us up for success that as we grow or cost per user or cost per seat comes down? And how much more efficient are we because of that? The ideal world is that you actually are growing as a business, making more money, hiring more people, acquiring more users, and your total cost of tooling per person goes down.

29:52That's like the dream. And there's lots of ways you can build that financial model. But I mean, that's kind of, that's what I think most marketing technology leaders should strive for is to make sure that they're controlling costs over time, because most businesses don't. There can be some goals that are discrete in nature that are not cost -efficient but more like net capability related. So it's like, hey, we want to design a first -party system that's world -class that achieves these three goals, right? Maybe we want to incorporate artificial intelligence into some part of our product platform and incorporate their party tools.

30:25And those are more like discrete protocols. And the same way that a business might launch an external product goal to like launch a feature. They sometimes also might have internal protocols, clean up our revenue operation systems, make our email marketing system better. In particular, email marketing is one I see come often a lot with small businesses and even medium -sized businesses where they'll have picked a tool at the start of the company's life cycle. And as the company has grown, they've outgrown that tool. They need to move to like a braze or a marketo. And so there'll be a big six -month initiative to say, we just got to switch.

30:58Like, that's the goal. We have to safely get off this small tool to a much bigger, more complex tool that's going to cost us more, it is a lot more complex, but we need to do it without losing money. And so that's usually the job of a MarTech person and like some type of change transformation effort. Perfect segue to where I wanted to go, which is tooling and your recommendations and favorite tools. And so maybe we start with just like, what do you find as a good starting tool stack for people starting to think about MarTech and basically growth? And then what does it end up being generally? In terms of stack again, we think about B2B and B2C, right?

31:35B2C, I would say the stack was largely solved from 2017 to 2020. We have like a renaissance of the data architecture. So what I'm going to do is I'm going to take your B2C then and now, and then we can go B2B then and now. Great. Okay. So B2C, like if you back up to 2016, 2017, you have segment and M -Porticle on the rise of the CDP, consumer -based businesses have to collect a user. And tie a bunch of data to them and then track their actions to send about out to performance ad networks and email marketing tools and product analytics tools. And so you would see this like very commoditized stack.

32:15It would be like CDP in the middle, a bunch of tools connected. The promise of the CDP was you integrate one SDK. Your engineers don't hate you. You send all the data to the other tools. You create audiences. Great. Last but for a long time, the thing about it though that I think really changed around 2020 is that the cost of ownership of warehousing became much cheaper. And so 2021, you start getting the place where like, yeah, it actually makes a lot of sense and is really easy to store all your data in a warehouse, model all your data in the warehouse, and to do it without needing a vast data team.

32:47Because I would say, every MB was probably doing all this well before or anybody else was, but they had the main advantage of a lot of money and a lot of resources. So now, come 2020, it's cost -efficient to have a data team with your own warehouse, and to manage data centrally in something like snowflake. So now this question is like, okay, well, we've got to get data into the warehouse, but how we move data around is totally different. Right? And that's what really led to the rise of reverse ETLs. So now you can actually build your own CDP. And lots of businesses already have a consulting with a well -known financial trading platform a couple months back, and like, they have a CDP.

33:25They have all this internal data in their warehouse. But they have not been able to activate it because it's pretty old architecture, everything's batch -based, and at the day, what they need is a reverse ETL. They would need to take that data and just get it out into the world. So they need the reverse ETL component or the transformation component of a CDP. And so I'd say now, today, when we think about BDC businesses, you can either go into the traditional route, buy CDP, hook up all your tools, third party. I think that's a great move if you do not have a lot of engineer resources. Because you're not spending a ton of time and energy on a warehouse and all the modeling that comes with it.

34:02You're just spending time to implement one SDK. So I think like if simplicity is the name of the game for business, CDP, centralized stack, Craig move. If you are an advanced engineering culture and you are cutting edge and you're going to do a bunch of modeling in DBT and you already have snowflake, you should move towards the model of using a reverse ETL. What it means is that there's a way to get your data into the warehouse and then how you activate it is completely independent from the CDP. And so what that means is actually you can have lots of different variations of the stack. You could use amplitude as your CDP.

34:34Collect all your data, stream it into snowflake. They actually now have an integration with snowflake that lets you feed data directly out of snowflake. And then you could use a reverse ETL to just pipe that data wherever you want. There's a really good section though, sorry to self -aggrandize, but there's a really good section in the reforge module this fall that talks about what happens when you have multiple ways to move data. You buy amplitude for your CDP and you're moving data to your warehouse, amplitude as much integrations, but you also have a reverse ETL and you can move data out of your warehouse.

35:04Where do you choose? And I would say a lot of businesses get in trouble when they don't have a methodology or a system for how and when to move data from one place to the other. So they just do it have hazardly. And the key in systems management is you want to design a process for doing it. Sometimes a waterfall or mental model for when it makes sense to move data directly from amplitude, which is like the ingestion point of your data stream or from the warehouse where you can model and make it better. I think the key is just having like a philosophy and approach. There's not really like one answer.

35:36But that's all B2C, right? So B2B I would say, you've got to move on to that one. You mentioned reverse ETLs. What are some examples of products that are reverse ETLs so that people can look them up? Yeah. So I personally think like the reverse ETL is a capability. It's the ability to move data from a warehouse to a tool. So technically speaking, you'll find reverse ETLs in CDPs and a standalone products. Segment has a reverse ETL function. They just launched and particle has a reverse ETL function. They just launched. Rutter stack, which is a CDP, has always had a reverse ETL function where you can take warehouse data and move it to different cloud infrastructure.

36:13Then there are distinct standalone products, census, which was back base 16z, and high touch are the two standalone reverse ETLs. And like I said, I'm an investor and high touch. Love their work. We use them at ramp. At the end of the day, you should pick tools because they help solve problems, not because of like anything else so we can come back to that. Oh, one. Great. Yeah. That was perfect. Keep going. Okay. Yeah. So we talked about B2B, or sorry, B2C. B2B. I probably don't have as much history as say like people who survived the .com crash in 2008. I started really my career in B2B in 2014.

36:51So I'll share a little bit of my experience. And I'm just hopefully just saying this because like listeners may chime in and be like, oh man, this guy doesn't know what the hell he's talking about, which is like totally fair game. So 2014 though, I'm working at branch. I was working for our COO Mike Molne, who's now at this really cool company called Dina. But at the time, I was working for Mike. And as we talked about before, oftentimes growth stacks just appear because you're given a challenge. And I remember like sitting in this tiny room with Mike, we were over in Palo Alto, like right off the fills in Palo Alto in this tiny room.

37:26It was boiling in the room like so hot we were sweating. And we were like mapping out on a whiteboard how we would design our first version of our like system, like how we capture leads, how we get into Salesforce, how we would email them with a little tool called outreach at the time that was like still a startup. And you know, I'll send it to you after this if you want to show it with viewers, but it's like so MVP, but still models what a lot of people have today. There's like some adjustment point for your data, there's Salesforce, there's some type of outbounding tool, there's an enrichment tool, and then a lot of other Jerry rig stuff hooked up to Salesforce.

38:00And for the most part, that's how B2B still exists today. As you have Salesforce and then the whole world in the universe revolves around Salesforce. You just have more advanced tools, you have like gone and stuff like that. I think the big change though and what is like really fascinating and has been fun to watch is in the last two, three years, you now have this whole rise of B2B to see, which takes all the complexity of the top of funnel user acquisition system and stuff set right alongside your CRM. And how you build an elegant system there in that space, I think is one of the most complicated and intricate pieces of being a Marktech person today.

38:37And some of it just has to do with the data language, like all these B2C tools were designed with two objects, a user and an event. And so if you're not a technologist, it's like object orientation is how you kind of think about the world. There's only two concepts for the world in a user acquisition based system. A user who's a person either anonymous or known coming in your website, and the things that they do on your website are application and you kind of use all that data to acquire them or model them. In a B2B business, you have all that complexity, but at the end of the day, you might not really need it if all the person is doing is just the company is signing the contract and then you don't really care what happens afterwards.

39:14You might track users and events inside your application, but it's not for the acquisition. It's for the retention of the user. B2B to see is fascinating because you have all the complexity at the top, but then like how and when do you tie a user to a company or some type of entity object? And what tools do you need to do that? And where do they live in the system? And do those tools actually have competing priorities? Let me give you the greatest example of this that happened at Notion when I was consulting them, at Chris when I was consulting them, and at ramp is having both HubSpot and Salesforce.

39:45Both RCRMs both have the ability to track users and companies, neither RCDPs and how you actually mapped the data from HubSpot to Salesforce kind of determines how much hell you're in. And there's really no good solution. It's just like you have to figure out for yourself, like how do you want to acquire use at the top of a funnel? How do you merge them in the bottom of the form of the Salesforce? And again, there are lots of options or versions of the world. You could use Amplitude only and collect all your user and event data and then merge that into Salesforce directly. You could collect all your data to Amplitude or Segment and then post that to HubSpot, which then post that to Salesforce.

40:22But of course, as you make these decisions, your system to come is more complicated and more than one person can manage. So there's this trade -off between complexity and resources that you always have to juggle. Today's episode is brought to you by Brave Search and their newest product, the Brave Search API. An independent global search index you can use to power your search or AI apps. If your work involves AI, then you know how important new data is to train your LLMs and to power your AI applications. You might be building an incredible AI product, but if you're using the same data sets as your competitors to train your models, you don't have much of an advantage.

40:58Brave Search is the fastest growing search engine since being, and it's 100 % independent from the big tech companies. Its index features billions of pages of high -quality data from real humans, and it's constantly updated thanks to being the default search engine in a Brave browser. If you're building products with search capabilities, you're probably experiencing soaring API costs or lack of viable global alternatives to being or Google. It's only going to become harder to afford these challenges. The Brave Search API gives you access to its novel web -scale data with competitive features, intuitive structuring, and affordable costs.

41:34AI devs will particularly benefit from data containing thorough coverage of recent events. Lenny's podcast listeners can get started testing the API for free at brave .com slash Lenny.

41:49There's this big question within B2B and B2C around how to do attribution. Well, it's a never -ending struggle. I'm curious if you have any pro tips or breast practices or tools that you use to improve the way attribution happens at a company. Actually, I listened to your pod on multi -touch attribution. I'm forgetting who you were with at this point, but it was like I was just I was loving it because I talked about MMM and MTA specifically. Yeah, there's a newsletter post actually and that even a podcast. Yes, so back to our conversation and division of responsibility, I'm not always the person you should talk to to create an MMM model.

42:31I'm not a data scientist. I know how to make MMM models and I know what they are. Can you explain MMM briefly? Mixed media modeling and MTA stands for multi -touch attribution and it's these two ways of kind of like measuring the world and marketing to understand how you should allocate resources to campaign spend. MTA and MMM, though, are both underpinned by how you collect data. They're both informed by the user object and the event objects that you collect on your website or application that then lead to the data the data scientists use for MTA and MMM. That's the connection between data and Mark Tech is like often the tools and systems that we build and stand up and manage are what are used for these very complicated growth experimentation and attribution results at the end of it.

43:14And one of the most discrete things you can do for MTA because I get this question all the time around, hey, like do we need MTA? What should I do first touch or last touch? Should I do both? And there's actually really like I can send you this guide, but there's like six or seven things you can do to basically future proof yourself from neither to needing either one. Right? Because most businesses either start with first touch or last touch and then eventually they want to move to a multi -touch attribution model. And for those who don't know what that is, like first touch is where you know you kind of collect the data about where from somebody first came from.

43:47Last touch is where you collect the data about where the person last came from. So an example of this be like if I went to Lenny's newsletter from a Google ad and that's all he has that we my first touch and my last touch. If I first came from a Google ad to Lenny's podcast, but then later I came from a Facebook ad or I don't know direct then that would be my last touch. And so it's this question of like does the the Google original first Google channel get credit or does the second one the Facebook or direct credit. And you know in the first touch attribution model, 100 % goes to the first channel and the last touch attribution model, 100 % credit goes to the last touch and a mixed attribution model or multi -touch attribution model, you're trying to figure out how to split the difference, right?

44:27And usually the evolution for businesses is they start with first touch or last touch then they go to splitting it, literally 50 -50 and then somebody gets angry because they're not getting enough credit and they say we're going to go to NTA and there are both first party solutions for that and third party solutions for NTA. But back to the main thing, the main point is if you think about what you're collecting, this is for website businesses, you're collecting the referral, like they're in the URL where the person's coming from and you need any UTMs associated with that person and you also need any parameters from the advertising network that might give them the ability to counter conversion.

45:05Every ad network out there has little things they stuff into your URL that tell you that you came from them. Facebook has FBP, FBID, these sometimes encode it. Google has this thing called Google Click ID, which is just a really long string of characters that don't matter unless you know how to decode it. But all advertisers and for the longest time, advertising worked by putting parameters in URLs, pushing somebody through your web to your website, collecting those parameters and then passing it back to the ad network so they could get credit for it. And so in my mind, the best practice that everyone should stand up from day one is to basically design the system for NTA and then use whatever makes sense as you grow.

45:45And so the way that I typically recommend it, people is like, imagine when a user comes to your website, you collect the URL, collect their referring URL, collect all the additional marketing parameters you might want, G -Clayer, Tick -Tock ID, Microsoft ID, you should just make a list of them and if you don't have that list, I can give them to you. And then you should collect all UTMs. So in the URL, you're going to have UTM campaign, UTM medium. Most marketers use this to note like what the campaign type was. Now, the thing is, is that UTM is only going to be specific to the moment and time that the person came to your website.

46:20So like, back to our example about Lenny's podcast, if I come to Lenny's podcast and I came from Google ad, then my UTM is only for that Google ad. So I have a Google click ID and I have a UTM. So what you're going to do is you've got to store those parameters locally on the device, if it was a browser or whatever, you've got to store it as UTM first, campaign, UTM last campaign. And what you do is every single time that a person comes, you kind of replace the last campaign or the last value with the one that's there. So say, you know, the last one was Facebook and then I come later from a direct mail ad, you replace the UTM last medium with the new one.

46:59Now what's happening, if you're using third party tools, is that you're collecting this this this user information when the person's on the website, you're going to collect it both as a user attribute and as an event. That way, what's going to happen on the back end for your data warehouse team is they're going to see a user profile that has both the first attribution information and the last attribution information. And then for all the stuff in the middle, you're firing off a page view event with first and last, where the last might deviate if there were multiple steps in the middle. So what they can do is they can just coalesce over all the last UTMs they've seen on all your events by user to get both their first one, all the ones in the middle and the last.

47:40And so it's doesn't actually that complicated to set up. Most people just like don't do the work early on and then when they want to go back later and have MTA results, they don't have the data to do it. So one of the things I tell people who are debating this is like, let's just get the infrastructure right from the beginning. Let's set up so that you have users, you have user attributes, you're collecting first and last UTM on users, you're firing events with all those. There's some other more like complex things you can do too. Like you can set them in first party cookies and you can also set them in your third party cookies for your tooling vendors.

48:12At the end of the day, though, what matters is like you just are collecting this information from the beginning. That way when you actually want to progress your attribution model, you don't have to wait a really long time to start gathering that data. Amazing. I love the details that you're sharing. Where else people can find this sort of advice. It sounds like a core part of this is one just having a data warehouse we just throw all this data into and two having a taxonomy that you can rely on and do multiple things with down the road. Is that roughly right? Yeah, I think that's right. The taxonomy, though, I think what's interesting is it's very much guided by your third party tools.

48:50Again, that's the reason why I think companies often miss the mark here is because they're not thinking about what can my tool actually allow me to put into it in the first place. Just a make sure I understand what you're saying there. You're saying generally, maybe third party tools limit what you can do, which set you up for hardship later. Maybe what you're saying is do that yourself, the tracking piece. Is that roughly what you're saying? Yeah, I think that's right. The way to think of it is like if you build your own data warehouse, your schema is unlimited. You can do whatever you want. You can design product schema, you can design user schema and event schema.

49:29But most third party B2C tools don't allow you to control the schema. There's only one CDP I know that does that. That's Snowplow. The rest are there's a user object and an event object. So you can either stuff data as a user property onto the user object or you can stuff data into the event and fire it off as an event. But that's what you're working with. So what I'm saying is most people just don't think about the object orientation of the third party tools they think about and they don't design their website traffic or their app traffic. We didn't talk about app which is a whole different slew because doing attribution with I was 14 is much more difficult.

50:07But even in the website version of the world, people will often just collect UTMs and think that their job is done. It's actually more complex. You have to think about first and last.

50:20Think and so this goes back to the main thing that we were talking about earlier where is like the job of a marketing technologist is to think often want to use down the road about what we're going to need to solve for and design systems in an elegant way not to break the bank but to at least be the minimum vital product to actually get there. It's a lot of a model of my job and I think the job of marketing technologist is trying to preserve that future state in the most minimally invasive engineering and resource way possible. You've talked a bit about thinking ahead and a bunch of tools and platforms and I'm wondering are there any new and emerging tools platforms or even growth channels that you're keeping an eye on or excited about or finding more and more useful?

51:02I'd be remiss if we didn't talk about threads right? Threads is super interesting. The question will be like how quickly can they stand up an API for advertising and like what does that look like? Or did they just blend it in with the existing meta and Facebook architecture? One of the caveats that I'm sure a lot of performance marketers out there will agree with is like Facebook has a conflict of interest in reporting, right? They want you to spend money, so obviously they want to report the best results and that's the reason why attribution parties like branch and affsliar exist is to somewhat curtail that conflict of interest.

51:37I'll be really interested just to see how attribution works, especially when you're moving from Instagram to threads from Facebook to threads will it be the same architecture or with the same advertising platform? Will they try to do something new? So I'm keeping my eyes on that. Reddit is also a very interesting place to convert now. They're opening up their conversions API and I'm seeing a lot more investment in Reddit just because you can have embedded ads now that almost look like they can be posts that you can comment on. And I think it just speaks to kind of like the maturity of the advertising business.

52:09You know, what's happening in the background of all this is like ad attribution from affs has become a lot more difficult and mostly aggregate. From 2010 to 2020, we had the golden years of deterministic matching where you know, it was very easy to run an ad and understand with precision who installed the app. Maybe to know their name, but you actually would know their IDFA and you could tie that to their PII. You can't do that anymore. It's very challenging. Even when you can do it, the results that you would get are pretty low because nobody's going to be opting into giving you their IDFA. So what that means is like these ad networks are becoming more complex, sophisticated and interesting right at the same time that it's harder for marketers to really understand how they're spending money.

52:52And so I'm paying a lot of attention to how marketers make decisions with probabilistic data because most of the work that I'm doing now is actually saying, well, given that we don't have deterministic data about a person, a certain audience or where somebody came from, how can I find other information that will create a model for 30 % of the population? And we can use that to extrapolate to 100. So probabilistic matching and probabilistic attribution, I feel like is a skill set that more marketing technologists and marketers should just like get familiar with because that's the way that we make decisions today.

53:27Well, I haven't had heard of this concept before and that's how people are starting, or at least you're suggesting that's how people should start thinking about growth results and impact is less. Here's how much this ad drove. It's the likelihood that this ad did this sort of, had this sort of impact. Yeah, and it's not the case with all channels, but it's specific for apps that have mobile apps like they're going to be impacted by it because they just aren't going to be able to like, discreetly identify one to one the person that came from a campaign. They'll know that a group of people came from a campaign, but they won't be able to make measurement with those people alongside other attributes.

54:04For website, it's not the same, but there are lots of things that are making it more challenging. What is browsers now are stripping out those URLs we talked about? So you're just seeing a bigger and bigger percentage of people being counted as organic that actually came from a paid advertisement because when they got redirected to your website, they just, the browser truncated all those URL parameters. The second thing is cookie blockers, we talked about all these third parties before. The way that third parties often collect information is they drop a cookie in your browser that tracks you. If you've heard of segment, which is one of those well -known CDPs of the last few years, is they implant a little third party cookie on the site that contains an anonymous user ID and all of your attributes as you're navigating the site.

54:48And then once you log in, they convert that to a known or non -anonymous user ID. User that's tied to some type of entity ID or a user record. And at that moment in time, if you come back and they see your cookie, they kind of know who you are. Now, if you're blocking cookies, that means you're basically remaining anonymous throughout the entire user journey until you log in. Not to mention, like, a lot of people have lead funnels where you need that information to actually understand what the user is doing before they convert. So if you're blocking third party cookies before they even get a chance to convert, you have no information about where the person came from.

55:23You just saw that they signed up. And so it might as well be organic. So you talked about how many people are trying to get used to this new world of ATT and much harder to measure attribution and all that. Is there anything you've learned that has worked well to help you kind of recover from that a little bit in terms of measuring what's happening? Is there any tips you can share or anything you've seen work? Yeah. I mean, I think a lot of people are just like gravitating towards MMM now without really understanding when MMM is used for or not. You know, I don't know if like there's a company called ReCAST.

55:55I think you're an investor. I'm busy. And that's who wrote that article that you mentioned actually. That's right. It was Michael and it was somebody else I can't remember the name. It was another Mike Taylor. I'm not an expert at MMM. So like, I'm not going to be able to comment to quite the degree that they have. But when I spoke with Michael and when I think about MMM, a lot of my conversation is like, is this actually really realistic for our business right now? Do we have the data to run an MMM model? And how is it going to change the or kind of chart the course of our performance ad marketing business in light of having this information?

56:31And when I think about it through those lens, most of the time, businesses are not ready for MMM. They actually just need MTA and they need better probabilistic modeling. And I know that's like not a super spicy take, but I just say like at ramp and what I'm seeing at other businesses right now that are operating, it's much more of like, we're going back to the days where we understand and broad strokes how much each of our campaigns is driving and advertising revenue. We're not able to tie that to screen with the user journey and we know that some percentage of this user base might have been lost to organic.

57:07So in light of those, how do we make spend? And then also you can be pretty smart. Like you can do, for example, geo -based testing on billboards, try to isolate that as a factor if you kind of like withhold all other compounding factors. So you can be smart coordinating these types of campaigns that was really challenging, especially if you're a really big business that say runs, you know, online advertising throughout the US and you're trying to do targeted billboard tests and an isolated number of cities across the states, you know, coordinating to like turn off demographics, make sure there's not isolating factors, it can be really challenging.

57:39So there's not a silver bullet right now, I don't think. Awesome. Just a few more questions and then kind of a broader question I want to ask. So say you want to start hiring the next Austin. First of all, what do you look for in the person? What are signs that they're probably going to be worth chatting with? And then what are some interview questions you like to ask to get a sense of how strong they are? So the first thing that I always gravitate towards is just like intellectual curiosity. And I know that's like very maybe a little bit overrated, but I think you can tell pretty quickly if somebody's just interested in the world in learning things.

58:15And the thing about third party tools is you are constantly learning. You'll never be an expert in everything because there's way too many tools being expert on. I've forgot what publication. I think it's Martek Adder and Chi for something there's a publication that I subscribe to and like everything is classified as Martek. And the diagram is like huge, like cover a wall. Now I don't believe everything like that is Martek, but even if a fraction is, there are way too many tools in technology to ever be an expert. So you have to be both very interested in learning and very willing to quickly learn if you want to be in the space.

58:49And so I just generally look for intellectual curiosity as the first sign. The second thing that I think helps people a lot who have intellectual curiosity is they're scrappy and engineering. They might not be the best engineer possible, but they know how to get around. You know, they know JavaScript, they know Python, they can read API documentation and make an API request. They have enough base knowledge to basically like understand how to solve a problem that an engineer might do, even if they themselves are not an engineer. Now obviously like you can get lucky sometimes and you'll find the engineer who you know never wants to be an engineer again and decides to move into something less technical.

59:30And in those cases they're super powerful, right? But those I haven't met a lot of those people in my life. And also there's just some like business dynamics to it, you know, like you can probably make more as a backend engineer than a Martek guy. So like you probably just pursue the pathway that makes more money. It's like well, I'll put over like utility function. So you know, look for your talk to curiosity. I look for you know basic engineering scrappiness. And you know, as a side note, I would say lots of people out there, the advice that I give them is you don't have to go get a software engineering degree.

1:00:01You can teach yourself. I am self -taught. You can take a coding academy online. I think you get enough knowledge through being able to do web programming or some type of backend programming. So I would say it's not more than a six -month investment for anybody to really get the skill set that's needed. Obviously like once you get the skill set, you can build upon it with years of experience afterwards. But if you're new to the space and you're like in marketing ops and you want to get more technical or if like your user acquisition manager who did pay performance, but you're like, I really want to do things end to end.

1:00:33You can just go pick up some software skills and you probably are going to be pretty dangerous from that. And so you know those are the kind of like the two things I gravitate towards. There's obviously many more, but those are the first two. The questions that I like to ask is what is I like to ask people how they prepared for their review? This is not, I can't take credit for this. Somebody my wife told me about gave me this idea and I loved it. I think it was an A16Z partner. But I love the question because when you ask like, hey, how did you prepare, you're really asking like, how does the person think?

1:01:05How do they plan? How do they take things seriously or not? What did they read? What did they do? And if you have to prompt them to tell you all the things they did, then like they're just not a systems thinker. But if they're like, hey, actually, I read these things. I did this. I woke up. I went for a run. You know, like the more interesting complex the answer, the more interesting complex the candidate. And so I love the question because it just gives you a really good understanding of the person on a whole like right out the gate. And then the other question I like to ask is I like to ask like, so you know, you're coming in tomorrow to our marketing tech system.

1:01:43And by Friday, you have to like write up a report on all the things we should change. Like, what do you do? And I like to ask that question because it pretty much signals out people who are biased versus not. People who have tooling biases will like immediately just like we should implement this tool because I used it before. And I really like to hire people who are not tool specific or more tool like Nostec. And they think about tools as being things to solve problems as opposed to like tools being things that you just solve because you've already solved it one way. This isn't a gripe and certainly not intended to like slice at PMs.

1:02:16But one of my observations of a lot of PMs is they just like pick the tools they've already used before because it's easy. And it's a shortcut for them, which I understand. But you know, problems are not always the same. So tools shouldn't always be the same. So I like to pick people who kind of like think about the problem set and the solution space more. And they ask questions about what problems you're trying to solve, which I think is much more of an actual PM mindset of like trying to work back with from the problem as opposed to just taking the problem and regurgitating stuff that you already know.

1:02:45Are there any flags you look for that tell you maybe this person isn't someone you want to be working with? I answer that question on two spectrums. One is if I'm hiring like as an I see or as somebody who's hiring an I see versus I'm getting hired. So like one of the red flags whenever I'm approaching a company to work for them is I'll ask for their company financials. And like a company that's not willing to divulge their financials to like a director level or above person, I don't want to do business with because that means like they're hiding something. Or they have a culture where like they don't trust the most senior leaders of the organization.

1:03:19Like either is a bad choice in my perspective. So that's one of the questions I always ask when I'm going up for a job. When I'm hiring somebody red flags, you know, I feel like one of the one of the false flags, not a red flag is more like when there's a like a gap in somebody's job resume. Like I already gravitates towards that. And it's often like really explainable. A good example is I was hiring somebody once who had like a two year gap in their resume. We didn't end up hiring the person. But they went through all the stages and we didn't hire them not because of them, but because the job got removed.

1:03:51And like this person took two years to like get a philosophy degree or maybe was a poachy degree. And then also like top himself to program. So like it was like a really enriching two years. And there were lots of ways that I could see them bring their past experience and the way that they took time off together to be really well -rounded candidate. And so I would say I look less for red flags and more for like false identifiers on the resume application that may shortcut me towards the decision. Another one is just like school. You know, like people just look at your school where you went to undergrad or grad and they kind of make a decision from where the other.

1:04:30And I feel like that's also can be a really bad shortcut because there's some like amazing founders for example who went to schools you maybe never heard of. So I know I know that's not a good way to answer the question, but I don't have a good way of like looking for red flags, but I do tend to spend a lot of time on netting out of false flags. That was a great way to answer the question. I wouldn't move on to something totally different. And this isn't something I've been asking people, but I'm curious if there's something here and then maybe if there is, I'll start asking this more regularly.

1:04:59I'm curious if there's just any frameworks that you've found especially useful in your work or even life. Does anything come to mind? One thing that I want to build, so if I ever build this maybe it'll be a newsletter for you, is just a one -page doc of the most useful life frameworks. And they're just the words. And so you obviously have to know them, but I feel like I come across really good frameworks all the time and then I forget them. So I just want a one -pageer of Lenny's life frameworks. We're starting this right now. Okay, great. Number one. I like this. Okay, so I've already said this and this is that you've promised to put this at the top of the list.

1:05:38So I'm really excited. It's just tools are not meant to sell, tools are meant to sell problems. And I tell that to every person I hire, I repeat it consistently at ramp and all of my consulting gigs. And it's not just the words, it's the spirit of it. You know, like tools are just meant to sell problems. You don't have to buy a tool to solve a problem. You also don't have to buy a specific tool to solve a problem. And I think it embodies so much of like what marketing technology is trying to do. It's trying to help people understand their problems and then actually take action on them using tools and technology that are most first party and third party.

1:06:11So and most people just like focus on the tool part and focus on the buying and integration part. And so I think like if you consistently remind yourself that tools are just meant to sell problems, then you really get into a space where you as a systems person can be an advocate for your marketer or your product people. I think sometimes there is a little bit of a tendency for people to think that people that who manage this enough tools are just interested in managing and setting up tools. But really at the end of the day, we're trying to help people actually do stuff. Then there's this PPS framework that I talk about a lot, which is problem people and system.

1:06:43So whenever there's a challenge that comes up like at ramp or in a consulting gig, I like to first say like what's the problem where the people involved and what system does it impact. Usually because people just jump straight to the system. They're like, Hey, there's this problem. I just need to solve it with the tool. Hey, like I'm trying to do x, y, and z. Can you just give me admin permission straight to the system? So if you back up though, first you understand the problem like, Hey, what is this person trying to solve? What is their discrete issue? A great example is like, I'm a sales manager and I want to make it so that every time I hire somebody, I don't have to go through this really tough process of like onboarding my staff.

1:07:22All right. So that's the problem. Who are the people that involves? Does the sales manager need permission from the CRO? Did the sellers need to be trained? Is there some other confounding factor that we're not aware of? Why we don't want to just automate this thing? Once you have an understanding of the people and the problems that you're trying to solve, then it's really, really easy to design the system to solve that. And so that's like my number one framework for technologists in particular is like, don't just jump to the system, think backwards, start with the people and the problem and then move to the system solution.

1:07:52And then another one that I've already mentioned to you is it's B and B as opposed to B VB. So build and buy as opposed to build versus buy. People all the time just think like the second that you're talking about implementing a tool or procuring a solution, it's, hey, I want to I want to build this thing or I want to buy this really expensive thing. Build versus buy is a very narrowly constricting decision tree, right? If it's only build versus buy, then you've already made the decision that you can only do one or the other, which means you're already fighting somebody of your organization. Build and buy means that both of you can win and you can actually create a solution that is not only unique, but saves the company time and resources and makes everybody happy.

1:08:41It's more of a consensus driven approach. So I just whenever I hear it a meeting or a call or some discussion about how we have a tool and it's really expensive and we want to build in ourselves, I try to just use the build and buy framework to tee people up and say what about the problem can we buy? What about the problem can we build? And where does it make sense to invest our resources and our people accordingly to get the optimal outcome? A great example is a company that I was consulting for was thinking about building their own A .B. testing tool and actually we had the same problem at ramp recently.

1:09:18They're like, well, we just think we should build ourselves. This is core to our technology. We have the engineering resources to do it and they were evaluating it to build the entire system themselves or buy a third party. I think it was split .io or something like that. And the entire engagement was basically designing a financial model to show them that they could make a lot more money, save money, move faster if they just bought the third party tool at the lowest possible cost and spent all of their resources that they were going to spend building it, building around it and making it their own.

1:09:49And there's lots of hit the word synergy because it's just like so yucky. I don't mind it. I think it communicates what you want to communicate. And I feel like people don't say it as often anymore. So maybe it's okay. Yeah, because they're afraid. There are mutual benefits is a better way of saying it. If you build a tool custom to yourself when you bought a tool because the vendor at that point is committed to you and they want you to be successful. So you often can get like kind of accelerated outcomes if you build on top of a third party than if you just build it yourself. A great example is like, say you buy one of these A .B.

1:10:23testing tools and you build around it and you're a large customer of them, but you've invested a lot of your own engineering resources to make this solution your own. If they know that and they care about you, they're going to be willing to actually like make you happy in the moments where you need to change from them. Say some SDK change or a new feature or something like that. Framework that I talk a lot about in my Reforged course is like about building a stack. Everybody asks like, Hey, how do I build my stack? What should I do? What tool should I use? You and you earlier were like, what's the golden stack?

1:10:55Tell me, what five tools should I get? Just tell me. Yeah, yeah, I'll tell you. They'll be out at the very end. I got to hold out. Otherwise, you'll ditch this podcast. That's true. Yeah, we'll tell the end. Is there anything else you want to share before we get to our very exciting lightning round? The only thing I wanted to fit in, which I feel like is maybe a framework or maybe just like a really good decision -making philosophy is this concept of thinking gray. Have you heard of it before? No, go on. Okay. So Steven B. Sample is a professor at USC. He wrote a book called The Contrarian's Guide to Leadership.

1:11:37Really great book. It's one of his principles. There's actually a lot of great principles in the book, but this is the one that I think I stuck with me the most in my career. And the concept of thinking gray is like, you know, so often in life and in our jobs, we are forced to make decisions very quickly. We have to like think black or white about a problem set or a solution and then decide. And one of his tactics is this concept of thinking gray, which is actually to not decide for as long as you possibly can before you have to decide. And it's really challenging because it involves this little thing called patience, which I do not have a lot of much the most of the time.

1:12:13And I know most people don't as well. But it's particularly really relevant in systems thinking and product because so often we believe that we have to make a decision because our boss is telling us because there's an okay are because we feel the pain because somebody's complaining to me. But actually in reality, you don't have to make decision all. You can just let it sit for a while. And this also applies to I think how you move through the world and view people. Like a lot of times we will meet somebody in a company setting or in a business setting. And we are quick to make decisions about them.

1:12:48Like we even ask me questions about how I hire people very quickly. We're looking for shortcuts to make decisions about evaluating people. One of the best piece of advice though about thinking gray is it gives you the grace to not decide about people until you have to decide. So obviously for an interview decision you have to decide you have to decide yes or no. But so often you'll come across people and you'll meet them once or twice. And you know, like I feel like there's this tendency in the back of everybody's brain to be like, do I like this person? Do I want to work with them? And the question often is like not that it's do you have to even make a decision right now?

1:13:23And by leaving yourself to have space to decide you actually open up the possibility that in the future you'll make a better decision. So I think that's like a really good lesson for systems. And it's obviously like a lesson that you can apply to the rest of your life too. Austin that was awesome. And with that we've reached our very exciting lightning round. I've got six questions for you. Are you ready? I'm so ready Lenny. What are two or three books that you recommended most to other people? So first book I already mentioned but say it again it's the Contrarian's Guide to Leadership. Awesome book.

1:13:55Second book that would be really good is The Art and Adventure of Leadership by going to Warren Bennes. And these are more like philosophically leadership books. They're less about like technical specs on how to run a business. So you have to be into that. Favorite recent movie or TV show? I'm currently watching for the first time suits which I'd never seen before. Which I think is like pretty good because the story arc of every suits episode is that there's a problem. Then they solve the problem and then the problem is solved at the end. So it's like very gratifying for anybody out there who's like a high anxiety person who just like wants to have the story arc resolved at the end of the episode.

1:14:32But if that's not your jam and you like excitement. Also watching Silo which are there's a for comic relief. There's our flag means death which is hilarious. Have you seen that show? No. Now you need to go watch it. It's about Blackbeard and gay pirate captains so strongly recommend that. And then for just like really dumb comedy what we do in the shadows is hilarious. What was that? What we do in the shadows? Yeah. Okay. Wow. A lot of recommendations. Thank you for that. What is a favorite interview question you like to ask candidates? So I talked about what you did prepare but the other one that I think is really good because it forces people to get vulnerable is telling me about the most difficult or challenging thing you've overcome in the last year in your life.

1:15:22It doesn't have to be work related. It can be personal. And I think it's like a great way to just like reset the atmosphere, make people dig a little bit deeper into who they are and be more vulnerable. And I find usually it also helps calm them down because if they share them one of the most like challenging difficult and hard parts of their life then like all the other questions just are like pretty easy. So that's one of my favorites. What is a favorite product you've recently discovered that you really like? The sound super dumb. It's called Cal .com and I'll tell you the story first. I like a bit of big Cowanly user for a long time.

1:15:54Cowanly is pretty expensive. If any of you for Cowanly is listening you want to give me a promo cool but it's very expensive and then I also just found that like it is not always graceful at syncing multiple calendars from both businesses and consulting gigs and personal. And I had trouble remembering my Cowanly link. There's like I don't know. The interface is like circa 2016. So like really looking for something a little bit more notion like with like a Command K interface and just like integrations that work. And Cow .com has not failed me. It has been awesome. So if people are looking for new Cowanly tools strongly recommend.

1:16:30Wow. They're in this. What a great domain. Cal .com. I know. Right. Killer. What is a favorite life motto that you often repeat to yourself or share with other people either in work or in life? I just think a lot about the power of appreciation. And you know like one thing that I've just been thinking a lot about recently is the challenge that people might be facing in their daily lives. I actually was recently listening to another podcast by Adam Fishman and he had Brian Balfour on. And Adam is basically just interviewing a bunch of dads, which is cool. But the nice thing about being a little bit older in life being a dad is that you maybe have seen hardship before.

1:17:11And this podcast is created just exploring the stories of people I really admire and have and go through their hardship. And in that it's been a very profound experience understanding the type of challenges that people have gone through in their lives. You know people who have lost mothers and fathers early in life. People who have lost children I myself, you know, my wife and I lost her dad last year. We lost two grandparents to COVID. We lost our dog. So I think that you know the way that it ties to appreciation is like if you can understand what people are going through and you start to view them a little bit more as a human and understand what's beneath the surface of work.

1:17:50Who are they? What do they care about? What are these you so much more appreciative for what you have in the good moments when they're actually there? And this just doesn't just apply to life. It's also like business too, you know. It makes winning a lot more fun when you know the hell that people have gone through. So that's just like something I like to talk a lot about with people, especially folks who are younger in their careers who like maybe have only seen wins, like describing what the losses look like so they can kind of picture in their mind and then have some experience when they go through it.

1:18:20It's a big part of my my stick. Would an excellent answer. I am definitely going to keep asking this question. For people who are still listening, here is the promised golden stack. Okay, so golden stack. If I was a B2C business, I'd buy Amplitude for my CDP, I'd buy customer IO, and maybe I'd upgrade to Brazon the future. I put everything in snowflake, I'd buy high -touch to reverse ETL that data out to my ad networks. For attribution, probably app slayer from the mobile app, if not branch, but it'd probably be app slayer first. So that gets you got app slayer, Amplitude as your CDP and product analytics, customer IO for email, snowflake for your data warehouse, high -touch for streaming all the data to tools.

1:19:05That's like golden stack today if I were implementing it for B2C business. For B2B, roughly the same, Amplitude. If you need an attribution tool, if it's a B2B, actually, if it's a web -only business, probably we use branch because branch is better for web. So you have branch, Amplitude, connect all the data to Salesforce, hopefully, at some point in time, somebody builds a better Salesforce. That'll be for our next podcast, though, Lenny, can't cover that today. And then reverse ETL is high -touch. So very similar, except the only difference is, what do you do for an email tool? You know, a lot of people use HubSpot.

1:19:41I would try to go to a way with customer IO as long as I could. And then I'd move to Braze afterwards. So a big difference is just Braze versus customer IO for B2B. Final question for you. I heard that you're a drone pilot. And I'm curious, what is the coolest place you've flown your drone or the coolest thing that's happened with your drone? So this actually gets back to our intellectual curiosity thing. Maybe I just search for weird people when I hire, because I just love when people do interesting things that are unrelated to work. And the story is, during COVID, I didn't really want to just better myself online through a bunch of educational platforms.

1:20:16It just felt like it would be a little bit soul -crushing to center front of my computer screen and learn how to do statistics or whatever. So I was trying to look for things that were like interesting niche outside of my domain, knew nothing about. And the three that I came up with were I learned to fly drone. I became a CFP certified financial planner. And I became a notary. And it's just because they seem like really useful things that I'd nothing to do with my work and would learn about something completely interesting and different. And so those are the three that I chose. The drone stuff, it actually was funny.

1:20:50I started flying here in DC. I live in in Virginia, but maybe a mile outside of Washington, DC. And around DC, there's a restricted air zone. So after I did my FAA drone pilot license and I became a certified drone pilot, I really went down the rabbit hole of trying to figure out how to fly a drone in DC because I've seen them around. But obviously it's like a national security. And I'm probably like dramatically mutilating the exact experience. But as I went to do this, it was a very complicated archaic, but also like funny because it was all online. You have to go fall to form. You get a letter from a local representative who says you're in good standing.

1:21:33So I found like a councilman that I just knew. And then you have to like fill out all the stuff. This paperwork, this site looks like it's from 1994. There's an office literally in DC where the person approves you. Then you have to like go and get a police officer to if it's actually babysit you while you fly the drone. And so I did all that. And I got to the point where I was going to get babysat. And I called our local police department. And I was like, yeah, so like I talked in the office. I didn't even officer to come out and meet at this time. And they just like laughed me off the phone. They're like, we're not going to send a police officer to watch you fly a drone.

1:22:08And I just thought the thing was really funny because yeah, it like makes sense. Like why would they waste taxpayer dollars to like have me fly a drone. But it was a requirement. So I ended up not being able to fly this drone in DC. But if anybody's listening and they know how to fly a drone and they want to fly, I would be totally down. I have two drones. I have a Mavic Air 2 and a Skydio Enterprise, which Skydio is a really cool company as well if people are looking for crowds. Okay. So you're saying if people have an awesome drone and they live in Virginia, they should come contact you and fly some drones together.

1:22:38Yeah, exactly. Exactly. But just not in DC. All right. Well, Austin, I think we delivered on the promise of making this extremely nerdy in the weeds. And I think we've solved a lot of people's problems. I feel a lot of gratitude for you. And I feel like we taught a lot of people about Marctec, which was my goal. Thank you again for being here. Two final questions. Working folks find you online if they want to ask you more questions and how can listeners be useful to you other than coming to fly drones with you. So first, find me online LinkedIn threads. I actually have threads. I don't have Twitter.

1:23:09Like hot take. I think Twitter like ruins people's careers. I've already seen multiple careers run by Twitter. Some people just don't know how to shut their mouths. So like I just, I'm not on Twitter. But I am on threads. And I'm trying to figure it out. So if you want to talk to me on threads, you can. I'm on LinkedIn. And then I have my email address on LinkedIn too. And I'm always willing to talk to people. Amazing. And is there any way listeners can be useful to you? Yes. I have a marketing technology course coming out with reforge in the fall. If you're a practitioner of Marctec, where you're interested in Marctec, would totally love you to take the course, would love you your feedback in particular.

1:23:43I love to stand on this podcast and act like I know a lot about Marctec, but I'm still learning. And so I think like it would be awesome to just get feedback from the community about what was interesting and helpful, what wasn't there. And we can improve upon. And then the other thing is if you're ever looking for Marctec help and you want to reach out, that's great. Amazing. Austin, thank you again so much for being here. Thank you for having me, Lenny. It was a pleasure. The pleasure was all mine. Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts Spotify or your favorite podcast app.

1:24:19Also, please consider giving us a rating or a leaving review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny's Podcast dot com. See you in the next episode.