Neal Mann | AI Agents as Teammates - Are You Really Ready to Work Together?

0:00Neal Mann:The CMO Confidential Podcast is a proud member of the I Hear Everything Podcast Network. Looking to launch or scale your podcast? I Hear Everything delivers podcast production, growth, and monetization solutions that transform your words into profit. Ready to give your brand a voice? Then visit IHeareEverything.com. Welcome to CMO Confidential, the podcast that takes you inside the drama, decisions, and choices that go with being the head of marketing hosted by five-time CMO Mike Linton. Welcome marketers, advertisers, and those who love them to Chief Marketing Officer Confidential. CMO Confidential is a program that takes you inside the drama, the decisions, and the politics that go with being the head of marketing at any company in what is one of the most scrutinized jobs in the executive suite.

0:54I'm Mike Linton, the former CMO of Best Buy, eBay, Farmers Insurance, and Ancestry.com, here today with my guest, Neil Mann. Today's topic, AI agents as teammates. Are you really ready to work together? Now, Neil is the CEO of Known, that's N-O-A-N, a company that creates an AI-native platform that helps businesses organize their data into a fact layer and an API for both humans and agentic agents. Previously, he was head of global transformation at Anomaly and an editor at the News Corp of Australia, the Wall Street Journal, and Sky News. So he's been an editor of all kinds of news stuff. Welcome to the show, Neil.

1:43Thanks for having me, Mike. Looking forward to chatting. All right. Let's set the stage for our listeners. What exactly is an AI agent and how do I know the difference between an agent and just a series of prompts? Yeah, it's a great question. And you'll find in the market right now, people are confusing language, even the likes of Microsoft using the word agent when it's not necessarily an agent. A good way to think about it is to break it down into two areas, really. One is assistance. which respond to you. So you make a request and it comes back and responds to you. And then an agent actually takes actions on its own.

2:22And you may have set it a predetermined goal, but it will actually take actions, use what's called tooling to execute things for you, and then come back in a feedback loop and work out whether that worked or not, and then do it again. And that's a good way to kind of separate them in your mind. And they're best used together. You might use an assistant to actually set up and launch your agents, but you need to think of them as kind of two different things. All right. So tell us when you watch teams build and use agents, how do they decide what is going to be the agent? And then how do you put the agent into a workflow if it's not just like Neil's agent or Mike's agent.

3:06It's an agent for a group of people. I think the first thing to think about is actually to take a step back because the majority of people jump to execution first. They're like, we want something to do this. And as a result, they'll try and jump to a short term solution that might be a little bolt on. Whereas actually, if you take a step back and go, what are the most important processes that we could try and automate? And then then how do we close the gap and do that as easily as possible? Because one of the things that happens with AI when you've got the right structure is you can really rethink things from a first principles perspective.

3:44What are we actually trying to achieve? What's the shortest pathway to get there? And then build your agents on top of that. And typically, you know, in marketing, you're going to see people creating things that create content quickly, emails, maybe an agent that goes and finds their dashboard and gives them their data. But if you take a first step back and say, hey, what are all the processes we do? And then clearly think through the ones that you could really streamline and then build your agent for that. Hey, Neil, can you give us a good example of a company that has, or even a hypothetical, if you have to blot out the company, where they have built agents and they've incorporated it into the workflow?

4:26well? Yeah, I think on LinkedIn, a really good person to look for is Jason Lemkin, right? So Jason Lemkin is an investor, he runs Sasta. He's constantly out there talking about the agents that they've built. And there's a couple of interesting lessons for me around how he's done it. So they have one agent they call 10k, which is their marketing agent, it pulls all their marketing data daily, it comes up with ideas for them. And he's very open about talking about how he's using that. Now, the one thing you see, though, is he also talks about the challenges he's seeing. And one of those is that they've built multiple agents.

4:57And as a result, they're constantly updating each one. And that right there is the problem that we solve for because a lot of people right now, they're making ad hoc agents all over the place. And actually, what they really need to do is rethink their business so that they have a single source of truth that every agent runs from. And that's actually, for me, it's really interesting watching him do it openly. And if you haven't seen it, we'll put the LinkedIn in the notes. But it's well worth looking at how he's openly talking about building it and then also seeing some of the holes in the way that they're thinking about it.

5:29So I think this is really good. Plus, you keep reading stories about how agents sell all your stocks when you're not looking or erase all your code because they're like, this code sucks. So I'm going to erase all the code and start again. So when you're a leader, if you're out there, you want to do this. You've got a team that by definition is probably in various stages of I love AI or I'm nervous about it. How you even set up the collective team to work with agents and design outcomes, measures and collaboration. You can't just say, all right, we're going to build a bunch of agents and you guys figure it out.

6:08I guess you probably could. And some companies have done it, but that's probably wrong. Tell us what's right. That is going to be messy. The most important thing, if you're a marketer, there's two key things, actually. The most important one is understanding structure. So marketing in particular is often a very scattered part of the business. Things are being made all over the place. There's lots of creativity happening. The challenge with that is that when you're working or want to work with AI, you actually need real structure for AI to reference. Hey, Neil, and when you say structure, there's data structure, IT structure, like the organizational design structure, decision-making structure.

6:50Do you mean all of that or just pieces of it? Let's pick off the one that most marketeers understand, which I would say is like semantic structures. There's a structure of the language and the things that you use. If you think about marketing in particular, everybody knows and understands the mission and vision. It's a concept that's clear to everybody. Everybody knows and understands brand positioning or brand experience principles. These are concepts that are clear to team members. But the problem in most marketing departments, and I guarantee this is the case with pretty much everyone listening, is that if you quickly searched within the Google Drive, there'd be 30 different versions of your mission and vision.

7:26There might be 40 different versions of your brand positioning. You can't guarantee that Dave in the email department knows which one we're actually working from. Now, the really important thing to understand here is that those concepts are things that AI also understands. It understands what mission and vision is. It understands the impact a mission and vision can have on content or things that are delivered in the marketplace. So as a result, the first thing you need to do as a marketer is get a clear structure and a single source of truth around all the key concepts to your area of the business.

7:57And the second part that for me is really exciting, if you're actually in the marketing space, is that traditionally, you know, I worked in transformation for the best part of nearly 15 years. and it would always start in product and engineering right whereas the really interesting thing about this transformation is that it's driven by language because who if you own the messaging you can then implement ai within the business so marketers all of a sudden are actually now in the driving seat it's one of the reasons you see a lot of the uptake yes you see it in coding but you also see it in content creation marketing services of ai and so those two things of what you need to understand one get your get your context in order the areas of the business that you own so that it's referencing just one of each of the core things and then two like actually realize that you're in the driving seat when it comes to implementing it within the business and this says i have to collaborate with all my all my partners across all the business to make sure this is pretty clear right because this is yeah when you think about big companies where this stuff could be all over the place in multiple databases.

9:04This collaboration thing is a big deal, right? It's everything. That's literally everything. That's what it comes down to is you need, you know, we call it a business brain. You need a business brain built on a single source of truth that are like Lego bricks. Each of those is only one. There can only be one brand positioning, one mission and vision, one pricing structure. You need that organization so that when you're then working with AI, it's actually referencing the right pieces of information and all of the team when they're using it referencing the right piece of information the biggest complaint i've heard i've been talking to a lot of fortune 500s the biggest complaint i've heard is team members making random stuff using chat gbt or claude directly that's off-brand that's referencing something old now this gets worse when you consider that marketing often has to work with sales in a b2b and sales is making all kinds of stuff up all kind of stuff up from old pfs from five years ago you know and and that right there is the problem you need to solve for and if you're a cmo right now you need to say hey where is our structured source of truth and let's organize it now your version one for that is making sure you've got one google drive with the right um elements in it version two people use markdown files um uh which you're starting to see and then we've obviously built a platform where we just extract that and it's all on an API that everybody can build from.

10:24And that's your kind of holy grail that you want to move towards. So in some companies, this is like looking at Mount Everest when you think of all the different databases. And you just wrote an article, I think for Harvard Business Review, where you referenced AI slop. Oh, that's not me. I sent that to you. So I wish. All right. Well, somebody wrote this article and you sent it to me. So I take it back that you wrote it, but you actually, I want you to talk about AI slop. And because I think it goes back to what we're just talking about in the infrastructure and what you get if you don't do this right.

11:02So the problem if you don't do this right is LLMs by their nature are going to hallucinate because of what they are. They're ultimately probabilistic maths machines. They're going to hallucinate and make things up. So as a result, the problem you have is if you're a business where let's say you've got a team of five people and two of them are working from a single source of truth they're going to both get pretty similar output and it will be accurate if it's a real source of truth if you then have three people who aren't they're going to start to create things that are drifting or off-brand that are different now add another hundred people into that and then a thousand if you're like a global company and you've got ultimately like slop being made everywhere and then the problem is a lot of people aren't necessarily reading it checking it that finds its way back into your your single source of truth your reference and then it gets worse and it just goes from there and before you know it you're a complete mess so the plural of slop is super slop so the bigger you are the more slop you can have um hey so how do i know if i have slop like because i think i've if you're doing the right thing you're you're training people on ai before you give it to them.

12:14You're having use cases, then you're rolling those out through the company. You've done all of this right. But when you look at things, how do you know it's slop or it's not slop? So I think the challenge here is if you've actually done it right and if you've really approached it in the same way we think about it, where every single one of those Lego bricks is on an API that's referenceable by AI and humans, as a result, you can then check against it. So you can run checks, check it against my brand facts to make sure that it's on brand. You should have built a system that allows you to cross-check and reference to make sure that something is right and accurate.

12:51We've put a lot of time into building that internally. There's ways to kind of hack it together within your departments, but you want to have consistency checks so that you're not letting things get out there that are sloppy ultimately. And do you need a little audit team to do this or a QA team? What do you put on this? well so this is the it has to be objective because if i'm judging my own thing i'm going to call it not slop i'm going to call it beautiful well in the in the ai first world so for a company like ours or companies that use our platform you can automate that and run it because all those lego bricks are there to check it now this is the kind of job that you could build an agent around you could say i'm going to build an agent that checks output every single time against these facts to make sure that it's accurate and not sloppy.

13:39And then you may want a check yourself to check it. And I always do. I think humans in the loop is a really important thing when working with AI. But if you're working at a big company and you don't have that kind of structure, number one, it's going to be hard. But number two, you need to think about it like an editor, right? Like I came from that space originally in my career. Editors were there to check. And I think what you're seeing a lot now is content creation has just become so easy for people who couldn't traditionally do it. And so as a result, everybody across the organization is starting to create content.

14:12If you have some air traffic control, everybody is doing everything. Everybody's doing everything. Multiply that by everyone's got a LinkedIn profile that they want to post on. All of a sudden it gets out of control and you need some sort of editorial checks against that. You may have to actually, if you've not built an agentic business yet, you may have to put those in place in a more human way, but it's really worth it because I think one of the things we're seeing now is that particularly as CMOs, like we talked about, they own the language, you own the output and your brand is represented.

14:43And people are very good at spotting AI slop these days. And that can look pretty badly on your brand. Hey, so one of the things we talked about pre-show was the documentation required to actually set this up. That leads to clarity, that leads to all this other stuff. Tell us about the difference between good documentation and crappy documentation. So I think a really important lesson for everybody is the structure is everything. People have been told that AI is good with unstructured data. You know, you can throw it at a document and it'll make sense of it, which it can do, but it thrives on structure.

15:25So what you want to think through is imagine your business like the brand positioning, mission and vision, all of these different elements they're just building blocks that ai can then put together and leverage so you want to actually think through how you how you structure that um in an almost file-like way which for a lot of people i think is a big difference from the way that they traditionally had decks and pdfs you know coming from the marketing space how much of your life is spent making decks and now you need to think about breaking that out so it can take time we we have a model that extracts those facts for you and then you just verify them and it's dead easy and you can get set up in minutes but what's fascinating actually is when we look at the usage data the amount of time people spend managing those and tweaking the language and i think this is really interesting as a marketer because back in the day you couldn't guarantee that your brand positioning would come to life in your content for example right the massive disconnect did dave in the email department or the blog department like this dave guy don't you you've ever liked dave right um could could he actually execute whereas now the ai will reference that and imbue it in the content so for the first time you can actually guarantee that the language actually manifests in the world in the right way so people spend a lot of time managing and tweaking those facts and so you don't want to rush it um but you don't need to spend you know months getting every fact right to start with you need the core things that you can execute and then i always recommend to people to think through it like transformation because it's corporate transformation ultimately it's the biggest transformation your business will go on start in marketing then maybe bring in sales then hr and then all of a sudden you can start to spread across the business okay so now let's say i have these agents my team i have this structure things are working how do i think about pay plans and communicating with a team that includes agents?

17:26And how do I know if I have a large team? How do I really know if my team is collaborating well with the agentic agents in the structure? So I think a really important thing to consider is to not let everybody just bang away on AI themselves, right? Right now, you've got people all over the place making things across businesses, often doing it. There was a report out this week that people were doing it behind the scenes, didn't want to openly acknowledge they were using AI. The first thing is you need to be transparent with people that this is our approach. This is our strategy to leveraging agents in the workplace.

18:01The second is then if you've built your company as an API or you have that single source of truth, you can start to measure what's actually happening and look at ROI. So on our platform, for example, we have a new metric called knowledge work replaced, where I know when you do a certain task or you create a certain asset, how long that would take you in the real world, you can then as a result, show people how much time they saved. And obviously, there's a monetary value that can be attached to that. So you want to make sure that you're tracking it. Because, you know, one of the things that I think a lot of marketers are probably going to get a shout from the CEO about is, hey, we want to make this amount of savings, for example, or make us X amount.

18:39You know, a bunch of our guests have said, the reckoning of ROI is coming, You can't just spend all this money. You're going to have to produce results. While a lot of that may be cost savings, some of that hopefully is growth. And I didn't mean to interrupt you, but yeah, go on with this whole story about, all right, how do you check on it? Yeah. So I've heard that multiple times. I've heard people being, you know, CMOs have said to me, they've been told they've got to save 20 million, 50 million to your point, drive growth with AI. So what you want to make sure is everything is trackable. You need to know what was deployed, when it was deployed.

19:13think of it like a system not one-off integrations that's the way you need to think about your business rethink what your business is ultimately because if you think about it as a brain that everything runs from then look at how do we track what assets are being created how do we track how many times people were using it you want to actually be able to track that in a large-scale company because otherwise you're not gonna be able to measure any roi can you give me any example of how you might set like like it could be a hypothetical company or real client or anything how you might set this up and say okay here is my center brain and how how it works so so for us on our platform the first thing we do is we run an extraction model that takes the facts from your documents and starts to set you up then you can start to manage those yourself and add them and what you're doing there is you're actually coming out of the document space and you're building those lego bricks in your brain and then you're they're ultimately on the api so so give me i pick any hypothetical like a snack company or a bank or a credit card company and say all right i'm gonna go get all these documents yeah so if you took like let me think you know we're working with a fortune 500 that i'm off actually off to after this we're working with a fortune 500 um in the kind of legal space first thing you want to do is get the centralized documents together around let's say with brand right because your business is ultimately broken up into key key sectors key sections so let's get the brand documents together let's then extract all those key elements of that and the key facts from it let's verify the ones that are true today because there's ultimately going to be things in there that are actually old and then then let's get those set up on the api now as soon as you've done that you can then deploy that api and run agents from it and you can start doing content creation um etc in in minutes but the first thing you need to do is just get that structure um the second thing is also just making sure that those facts themselves are well structured for ai um and so i always tend to start with brand and then you want to go into how the product works product features for example these are key things that once you've got them stored ai can then reference to do things like customer support um but you need that actually within a platform that's actually referenceable by AI and humans.

21:31Got it. So let's say I look at all my stuff and I have to fire an AI agent. Someone's probably really attached to that agent. Give us some tips on firing the agent and managing the humans around it. It's a really interesting point because people will get attached to them and they will think that something is doing a good job for them. Well, and if they built it, they're going to love that thing, even if it's no good. And the reality is like agents can be expensive, right? So we're seeing this right now. We saw Uber pull back on its coding, AI coding. So I think the most important thing, and it's funny because it's almost like humans, right?

22:13The most important thing is what's the ROI to the business of this agent? In the same way you think about an employee, are they delivering for the company? The second thing you're then going to go is what's the cost of this agent? just in the same way with humans you would look at like that cost benefit analysis and again this is why tracking is so important otherwise it's just arbitrary that you're deciding something's not worth using and then you just you can nuke it and the beauty is you can easily turn it on and off um you know that is one of the the great parts of it but i think the one thing people need to be aware of is if you do not and this is such an important conversation to have right now the majority of token usage so ai agents assistants whether you're using chat gbt or you you're using agent platforms like paperclip etc they're running on tokens and the reality is the majority of token usage right now is wastage and so a good way to understand this is if you think about you've had a long conversation with chat gpt and then you find yourself scrolling back through trying to find the one piece of information everything you had in that conversation was wasted tokens so it's really important to think through like if you're going to deploy ai agents in your business, the fact layer is the most important part because they won't waste tokens because they're going to use the right facts at the right time.

23:28So as part of that ROI conversation, you need to look at like, how have we structured this and set this up? And if we set it up for the success of an agent, or have we actually set it up for them to fail? In the same way with humans, if somebody joins the company and you brain dump on them a terrible load of documents that are a mess and expect them to do their job well, they'll fail. Yeah. the job spec is whatever you think it should be. It's probably not a good job spec. So you can't give that to an agent. So tell me, so we have all our listeners out there, some who probably are just getting started in this.

24:00Other than hiring your company, give them tips for how they should get started with their first agent and onboarding the agent and everything else. And, you know, if I'm the first step for my whole company, give me some tips. So again, it's funny, you know, as we're talking about it, the more I think this is so much, it's so close to just onboarding a human, right? Like the first thing is think about how you're going to onboard the agent that you want to use. Never ask for extra vacation time though. No, and what you're going to find is they're probably going to overwhelm you with responses, right?

24:37So like that's part of it. It's like, how much do I get back? I was talking to somebody yesterday who said that their agents are constantly firing, firing messages at them in Slack. And it was just overwhelming and they don't they don't need it. So I think the first thing is actually think of them a bit like the way you would think of a human. Right. And you'd say, OK, I would I need to give them solid information about what I want them to do. I need to give them solid information about the company that's well structured. And then you need to go through thinking through your use cases. What am I actually going to try and use this for?

25:08and i think for most people one thing to understand is that because an agent can take multiple actions it can gather context in different ways it can be multi-step um now those can be overwhelming to start so you might want to just do something that's just an example of a multi-step uh so i for example built one that would cert when it found a company uh we would if we were finding a company we wanted to work for it would work with sorry when if we had a company that we wanted to work with it would search the internet look at the company's house records in the uk or the company records in the us try and find who the core founders of that company were then go to linkedin look through linkedin find those founders name match them then find their email then draft them an email then send them that email so it's about like probably 15 steps with multiple steps broken out and then it could just run um and so it can be quite a complex workflow and the challenge is if your instructions aren't good it can break down so one of the things to be aware of is as you think through working in this space is again just like with humans clarity of instruction is the most important thing and this but these will never push back on you they will just do more work right so you have to you have to really like probably manage them differently that way it's it's been really interesting so i've had users say to me you know people who've been in business for 20 years they've said wow i've realized that my communication is terrible because i'm vague i use wishy-washy language when i'm trying to instruct an assistant or an agent and as a result the response isn't great and the default for people is to blame the agent or the assistant in the same way in the office space we've all seen it somebody gives terrible instructions everyone leaves the room and is like did the cmo say that like what we're supposed to do um so it's very similar but a lot of people said wow this has made me be clear with my instructions to both humans and agents so it's really important that you think through the approach and and a good thing to do is to ask an assistant and this is how you can use assistance in this how should i best structure this um instruction for an agent and it will break it out into the relevant step so use ai as your partner in that space super interesting tell me what this means for like all the added media and other pr agencies out there is this how is this a threat to them what is this it's a really interesting question i think it's you know if we if we look back and just to rewind if we look back there was a fundamental shift maybe 12 to 15 years ago in the marketing space where content marketing started to become a big thing right and all of a sudden people started to see that you know they needed to create content and what you saw is the agency started to create content marketing arms and they would create content and then all of a sudden people started taking that in-house and you've got the big brands bringing the content creation in-house which is where it sits for many now in the same way i think what you need to consider here is that what's your role as an agency and how is AI helping you when it's working with clients, maybe streamlining your operations.

28:24You know, we've got agencies using our platform, for example, because there's a fully integrated network, so they don't need a CRM. They're using our platform to run all of their business development. And then they have every client in the platform and all of the strategy of the clients in the platform. And to give you an idea of how powerful this can be, you know, just before Before this call, I integrated the known API into Lovable, the website and app builder, and asked it to build a website for a brand that we've got in our platform, a demo brand. And with one request, build me a website based on this API, and it built the website in five minutes.

28:57now why that's so exciting for agencies is that you could be doing a pitch and you could have all of the team working from the same source of truth as you're pitching and as you evolve the strategy which happens during pitches right you you're changing and tweaking the strategy you could bring to life 20 different apps 20 different websites in minutes all from the same and limitless creative too limitless creative all from the same fact layer and when i when i saw that happen i I thought, wow, this for agencies is potentially huge. So I think it's about using AI in the right way and working with your clients in the right way with it.

29:37Use it for your business development and use it for your creative, your pitching, your ideas and getting things to market faster. Excellent. Thank you for that. Hey, so before we get to our traditional last question, I have to ask you, most common mistakes you see in the marketplace now? you've already seen best practices but let's go over the mistakes the most common mistake i see honestly it's the same mistake that you see from cmos when you're in the room with them they're too focused on the execution they are you know a lot of people i think in in the marketing space focus on the execution um and so you know you're pitching and it's it's very early and you do a pitch and they jump on one idea and they don't like it they're focused on the execution You see the same thing in the marketplace.

30:26People are too focused on trying to look at AI execution and output rather than focusing on the issues within the business that AI is going to reference. That's number one. Number two is using some of these app based like advertising creative that just spews out lots of ads. I had one the other day that was from Notion that my wife actually sent me when she saw it, and it was a garbled mess. And it automated and pushed on meta. And I think just one thing that people need to be aware of is their brand is a system, right? We know this, your brand is an ecosystem. The challenge with AI is that it can now become just so expansive so quickly that you as a marketer can completely lose control of it.

31:14And so one of the reasons why we're building our platform as we are is that if I change one fact, everything that runs from that API will update instantly. And so what you want to know is that if you're putting out more creative into the world, more ideas, more websites, whatever you're building, you need to have control over that brand because it's always one touch point can turn off consumers. And in the age of AI, it can get sloppy. Excellent. We just had a discussion with Michael Treff on this of Code and Theory about how you got to have control over all these planes or they will fly everywhere and you need an air traffic control system.

31:52So I think that really is a good way to bring us to our traditional last question. Funniest story you can share on the air and or practical advice we haven't talked about yet. You can pick one or both, but you must pick at least one. oh god can i take a minute to think about this i should have come no no you can't have a minute funny story i can share on the air or practical advice we haven't talked about yet i'm going to go with the practical advice one because i i think one of the important things that people need to take away is that ai is not a bolt-on this is the greatest transformation that your business will ever go on in our lifetime.

32:37My company is just an API. I do not use documents. That's it. It's a business brain that I can talk to. That is one of the greatest transformations in business that's ever been. Now, the important thing as a marketer is you're sat at the center of this transformation, as I mentioned. As a result, you really need to understand how transformation within companies works and how to actually help execute that. and the best way in my mind to do that is to watch the mind hunter netflix series about serial killers i kid you it is the greatest corporate transformation story you will ever watch and when you watch it with that mindset you'll understand why because it was two guys in the fbi who realized that serial killers existed and that they realized they had to find a new way to track these guys down and they went through everything you go through in corporate transformation transformation they pushed out in front they trailblazed they then realized that they had to get c-suite sponsorship which you're going to need they then realized they had to productize this then what happened is what's called the frozen middle in transformation everybody who's just at the fbi was doing their day job they were like that's not how we do it we've always done it a different way and they had to galvanize the workforce to bring them on to this new approach to hunting serial killers and it honestly i've given i've told so many of my former clients to watch that series with that mindset, because when you implement AI in your business, it's not a bolt on, it can't be a bolt on, or your company will die.

34:06You have to make it central. And you're going to go through transformation to do that. And Mindhunter is the way to know how to do it. Never before have serial killers been referenced on CMO Confidential. So perhaps Netflix will send us some swag. So I think that is a great way to end the show. Thank you, Neil. And thanks to everyone for listening to CMO Confidential. If you're enjoying the show, please like, share, and subscribe. You can find all of our more than 170 episodes on Apple, YouTube, and Spotify, which include an update from the front lines of AI, a Spock on the bridge perspective, Colonel Mustard in the study with the job spec, how poor design shortens CMO lifespans.