The 7 Biggest Mistakes Companies Are Making with AI and Agent Adoption

0:00Today on the AI Daily Brief, we are joined by Nufar Gaspar to talk about seven common mistakes in AI and agent adoption. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show notes.

0:18Hello, friends. As you know, it's Spring Break Week, and we are doing a series of interviews with really interesting guests. And today we are joined once again by Nufar Gaspar. Nufar has led AI efforts at Intel. She consults with numerous companies of all sizes, including some of the biggest in the world on their AI adoption strategy. And she works with me at Superintelligent on our agent readiness audits. In this episode, we're going to explore some common mistakes that we're seeing when it comes to AI and agent adoption with some recommendations for how to avoid them. All right, Nufar, welcome back to the AI Daily Brief.

0:52How's it going? Good. Great to be here. Yeah. So we have a lot more to talk about because we've had a chance to do a lot more stuff since last year we're here. We've been doing these agent readiness audits in the super context where we're learning directly from companies, you know, what they're doing with AI and agents, what their challenges are, what's working well, what's working, what's not. You've also been working independently of that with a number of companies. So we have sort of a big, big, interesting base of material, I think, to pull from today. Right. And we've been seeing companies of all sizes, right, from like a handful of people all the way to the biggest companies out there.

1:27Yeah, I think that that's one of the things that we'll talk about and will be interesting to sort of keep in mind is how different or similar these challenges are kind of across different sides of company as well as across things like industry vertical. But so what we wanted to talk about today is the sort of challenges or common mistakes that companies are making, basically things that are holding them back from where they could be and where they want to be and potentially some specific real-world strategies to actually help them. So we've got seven of those mistakes. What we're going to do is we'll go through them kind of one by one.

2:00I'll sort of set up and have you explain a little bit about it, and then we'll talk through each one. Is there anything else that we should have in mind, though, as background before we dive into each of these in terms of just setting the stage for where companies are, broadly speaking? Yeah, so I think there are two things that also come across a lot when you do your weekly long reads and other more opinionated stuff is that some people are starting to get a little bit frustrated with the big gap between what the media, like yourself, is promising that we should be able to do with AI, we as in companies, versus what actually happens on the ground.

2:36So there is like a very big gap there. And while what we're going to discuss here is not based on any very detailed survey of thousands of participants, it's very fresh. and anecdotally comes from many different people and sources. So that's why I feel very comfortable with what we're sharing. From what I'm seeing recently, I'm seeing different responses of leaders and managers to the current gap between AI promise and deliverable, basically. So some of the managers are starting to get a little bit more, perhaps, I don't know if skeptic is the right word, but a little bit more careful with how they're willing to invest in AI.

3:12Even in one of the largest companies out there, I'm hearing that they're not willing to introduce new tools as easily as they used to do like one or two years ago. And they're much more worried about the ROI and any guarantees about the new tools and agents and AI, whatever, that will be introduced. So that's one, like a persona of behavior that we're seeing. The other are people that are still very excited, but they're expecting agents to be like the silver bullets. and in some cases even do like a leapfrog beyond having basically nothing now and expecting agents to help them bridge the gap. In some cases, it might help them bridge the gaps.

3:50In others, we all know that agents are not silver bullets and we will discuss some strategies and things that they can do in order to get much more from agents. And in general, we're still seeing mistakes across the board and we believe that there is so much more that we can do from what is being done now. in order to really get the value and not stay in this frustrated period of promise versus deliverable. Yeah. One thing that I'm noticing as well that I think is putting sort of or is about to put even more pressure on this is there has been obviously a big increase in general global market volatility and instability over the last few weeks.

4:33And already I'm seeing it translate to new urgency and pressure around can AI and agents specifically actually deliver and more pressure to try to figure that out, right? The more tense companies are about cost, the more that they are hoping that these things can be sort of great saviors and the challenges take on even a bigger urgency. You know, AI has never really been a field where people could take three or six months off and just see how it develops. But I think that that's even less so now. And so even as perhaps some of that skepticism or frustration is increasing, I don't see leadership taking their foot off the gas pedal at all.

5:14So, you know, it's an interesting time. But let's dive into some of these common mistakes and see how much we can kind of help people move through them. So the first that you shared is this idea of a single direction approach to how agent and AI innovation should happen. So what do you mean by single direction? So I've seen two things. One is organizations kind of expecting that AI will only happen in a very organic way, kind of a grassroots. Their internal employees will be psyched about AI. They will bring ideas. Perhaps they will work about their innovation in the nighttime and in the daytime will continue to do the regular job.

5:49and through that approach, everything will happen and all the promise will be fulfilled. And from what we've seen, it's not really the case. Like even if those grassroots and these employees are very, very talented, bringing great ideas, the best that they get by doing that as a side gig is to have a very good demo or a very good pilot. But then comes the time that you have to productize. And then they say, but that's not my role to support whatever high capability that I will create. This incredible lag system will require some maintenance. Someone will need to open a support ticket and close them and so on.

6:26And they're saying, that's not my job. So maybe someone else will take it to production. And then we're seeing many very great initiatives kind of hold in that phase. And in other cases, employees don't even do much beyond perhaps using the tools for their day-to-day job because they're just too busy to get themselves less busy. they just don't have the bandwidth allocation from their managers. So that's one direction where it only will get you thus far. In the other direction, we're sometimes seeing organizations where the managers are so psyched that they're kind of taking a top-down approach to AI adoption.

7:02As an example, one CEO was saying, hey, we're going to be an AI-first company and we're going to do everything with AI and we're going to only hire agents from now on and so on. And then you interview as part of our audits the employees and you realize that there is a very clear undercurrent of fear of job loss, that there is an undercurrent of distrust in AI. So it's very clear when you read those two responses that that's not going to yield a good enough result. And even in cases where managers are more invested, creating plans and ideation and stuff like that, if they are not working in tandem with their employees, it just won't hold because it's top-down And often they overlook some of what really bothers employees, how they're actually using what are the day to day challenges.

7:48And they're only seeing from a very like a bird's eye view. Everything looks very clear. And that's not necessarily the case. Once you go into the details of the agent implementation and the use cases themselves. Yeah, it's interesting. I mean, it sounds like even though we're talking about there being sort of this challenge from two directions, Like people are either missing sort of the support for bottoms up or they're missing the need for bottoms up, you know, involvement with top down decision making. they're both fundamentally questions of leadership, right? Leadership deficits, just in very different ways.

8:20Either leadership that hasn't sort of helped people figure out how to take their experiments and turn them into something broader and connected them with sort of the resources to design a full system around them, or leadership that is sort of visionary, but not practical and hasn't got buy-in. I mean, is that a fair kind of characterization? organization um the second one perhaps i will like caveat a little bit that often they do have the buy-in but they they believe that they know enough to make the decision for the organization and then they're not the ones that's gonna go and execute and the one that's gonna go and use so they're missing critical details and thereby it will not succeed yeah i think that this is this is definitely something that is emergent from these audits because we do these audits both on sort of a leadership level where it's, you know, a handful of leaders across the organization, but we also often get down to the individual contributor or employee level as well.

9:15And a common thing is managers, even who were individual contributors themselves a few years ago, there is a certain atrophy of what the day-to-day actual job looks like, how processes have changed, right? We live in a very, very dynamic work environment where even outside of AI, the way that you get things done, the context in which you do things is evolving so quickly that there can be, you know, a meaningful gap between managers understanding of how work happens and, and employees who are doing the works understanding of how work happens, which when you're trying to make decisions around, you know, work augmentation or task replacement, that those can be pretty critical differences in perspective.

9:57Yeah, yeah, I agree. And when you look from from a very, very bird's eye view, I always laugh like it's when you fly high enough, all the lines seem straight. And if you're too senior, that always happens to you. And it's not their fault. It's just the granularity in which they can operate. So again, the risk of being reductive every time, because I think it's very challenging to paint with a broad brush the prescription, but it does feel like there are a couple common sort of ways to counteract this, or perhaps maybe even better way to look at it is organizations who aren't struggling with this particular challenge?

10:32What is it that they're doing that makes them not succumb to this particular set of issues? Yeah, so it will probably not surprise you and the audience that we're talking about the two-direction approach, to kind of merge the top-down and the bottom-up, because we have to have the leadership. Like you said, it is a challenge first and foremost, but we also have to have the experts and the employees that will be the super users and users of the system in the loop early on, whether it's to eradicate their fear, to make sure that they're on board and they understand that it's going to empower them or augment them or whatever word you're going to use.

11:08You will probably say in some case replace them, but they still have to be involved in order to define the system properly. But also managers need to be involved in the details so that when it comes the time to allocate the resources, the funds, the bandwidth of employees to build the system right or to pay for a vendor, they understand the complexity and why perhaps something costs as much as it costs or it takes as much time as people are quoting. So it needs to be kind of a bidirectional thing. And even if it's a grassroot, I think it's very important for leaders to say, okay, this is a direction that we're encouraging innovation versus this is something that was already handled elsewhere or not a strategic direction that we want to go to.

11:53So perhaps not to stifle innovation because that's not what we want to do, but also not to make it so open-ended that people are basically wasting companies' resources because they have an idea and it's not going to go anywhere. Yeah, I think that it's worth noting, too, I see agents as exacerbating rather than helping this problem. I think that agents are inherently more replacing than augmenting, at least in terms of how people think about them currently. You know, agent, the ROI that companies are looking for from agents is, can they do a thing more cheaply, efficiently, more quickly than our people do it?

12:33Now, what that doesn't say is how companies are going to choose to use those new efficiency gains. Are they going to just slash headcount or are they going to reinvest people's time that's now freed up in further growth? Like, you know, each company has to make those decisions. I think that making those decisions and articulating those things is going to be absolutely integral to solving some of the challenges of employee concern. Basically, I think that agents are ratcheting up the pressure to get this balance more right than it is currently. Maybe let's move to number two, characterize it as a haphazard approach to AI adoption.

13:08So it feels kind of related to number one, but speak more to what the common mistake here is. so here often it's it's yeah it's a little bit of a continuation but often what you will see is like a best effort let's let people and again i don't care now about the top down or bottoms up but let's let people or organizations do whatever they have in mind everyone can work on ai and should work on ai because it's the most critical technological change in our generation and so on but then when you you ask everyone to work about the eye it's literally like asking no one to work on AI because it's very chaotic.

13:42We see a lot of duplicated efforts. Even when we interviewed companies that are not very large and you ask people that are basically peers what's permissive to use, what's already been done, they don't know. And let alone when you talk about large organizations, there is a huge difference in what they know. And that creates a lot of frustration, a lot of waste in these companies. And in many cases, that's not something that will get you to good results. Another funny anecdote is that in a company where literally they had zero or near zero AI usage, we were asking them, like, who is responsible for AI adoption and are you happy with that?

14:23And because they didn't know, they said something like, well, the CEO is responsible. And because the CEO is responsible, of course, I'm happy with the AI policy. Because, like, what else can they say? So if we're talking about one title, it's very little governance, very little policy, in many cases not articulated. Like you said before, even if in some cases you sit down and have one pager that articulates what are your do's and don'ts and what is your vision with the adoption, Even something as simple as that that takes an average CEO or leader not too many hours to craft will create huge difference versus companies that are just rock and rolling on whatever out there.

15:05So, yeah. So let's go into the prescription a little bit. Like, what's the template for how to get this right from what we're seeing right now? Because obviously you've got to imagine that this is fairly evolutionary and the best way to manage this is, you know, could be different from different organizations. But what are common best practices that we're starting to see? So one thing that I'm seeing repeatedly, and I don't know if it's a popular opinion, but what we're seeing is that in companies where there are dedicated AI teams, and it can be a team that only does AI internally or people that have enough bandwidth, and enough bandwidth is more than 10 % weekly, which what many companies are trying to allocate to their employees, and that's not enough.

15:44But some kind of a dedicated group and a council that oversees AI adoption across the company, It can be cross-functional. It can be a team that is dotted to the CTO, chief AI officer or chief information officer. It doesn't matter as long as there is a very clear team that is the owner of AI. Typically, this is where you see much more mature results, better results. So that's one thing that is very important. The other is having a clear AI policy, and it should be proportional to the size, the age, and the risk of the company. So, of course, the mega corporates, they have their lawyers and they have information security officers and everyone on board to define AI policy.

16:26And they have to get it right because it's way too risky. But even for them, they often can overdo or underdo that. And even if you're a very small startup, just getting started, have someone in charge. And this person or these persons should define what is the company policy. and they should and can revise it over time, but at least have something that the employees can refer to and can work off from, even if what they want to do is argue with the policy. So, yeah, I mean, it's super interesting. My sense too is once again, that agents are complicating rather than simplifying this in the sense that when the decisions around AI were things like, you know, are we going to use Gemini or Copilot across the whole organization?

17:11There's a natural centralization there. Also, when you're first exploring, getting a committee together to do that exploring. A lot of organizations sort of defaulted to that fairly centralized approach. Agents introduce a sort of a re-Balkanization of software where the platforms that are building agents that are good for the marketing group might be different than those that are good for the sales organization, might be different than those who are thinking about customer success and might be different than those who are thinking about core product. And they also might all have different constraints when it comes to regulatory and compliance.

17:44And so, you know, I think that we're certainly noticing a reassertion of decision making from line of business leads, functional heads, things like that. And I think that, you know, perhaps counterintuitively, it sort of increases rather than decreases the need for some coordination apparatus, because I don't think that you're going to be able to make all your sort of agent buying decisions or agent hiring decisions centrally, right? Because you're too far from the actual surface area of the business that those agents are going to interact with. But they have to be coherent with one another. They have to all plug into the same set of policies.

18:21They probably all have to have a similar relationship with data. And so the haphazard approach becomes even more tricky when there is a natural impulse for many different parts of the organization to run in different directions because they're trying to just make the right decision for whatever it is that they're doing. Yes, and they will also, like everything that was defined up to date, might not hold with regards to information security and other governance elements. So let's not open this Pandora box, but over there, there is also a world of risks that they, even if you feel like you have a very good AI policy to date, you should probably revisit that and make sure that it's agent ready.

19:02Okay, number three, unrealistic expectations. I think many organizations will relate to this, but talk a little bit more about the types of unrealistic expectations that we're seeing. Yeah, so there are many unrealistic expectations that we're seeing from different people. Some of them goes back to what we talked about in the intro, that media creates such a hype that everything is seamless and ready, and the technology is already AGI and so on. So then we're seeing different things. For example, there is a huge gap and a huge time lag between how long it takes you to create a demo and what AI enabled.

19:38And in recent times, it becomes even more and more prominent with all the lavables and V0 and all the other tools for prototyping. it so easy to create a working demo in a few days. But even with the best of tools to date, to bring something, an agent or an AI capability to production in an enterprise grade with everything that it entails, access to data, good quality over time, monitoring, and so on and so on. In some cases, it can take even 10x more time than how long it took you to work on the demo or sometimes even more than that. And that's both employees and managers neglect to understand that there is so much investment in putting something in production.

20:21And that's even before I'm accounting for the support costs that are quite high as well. So that's one type of gap between expectations and reality. The other thing is that while people acknowledge that GPT is not perfect and they know that AI is not perfect, When they are contemplating an agent or something that is much more autonomous, they expect it to be near perfect. In many cases, much better than what a human will be able to accomplish. So they expect the AI to be way, way, way better than the human. And this is not where the technology is currently at. We know that there will be a lot of probabilistic effects and that even the best of capabilities that we will put out there, they will have some faults.

21:03So as of now, the best known methods are to have some guardrails and to guard against those probable mistakes. But just as an example, the other day, I was talking to a company about a potential agent and I was asking them, like, if it has a single mistake each week, will that be enough? Assuming that I'm stating the obvious and they winced so clearly that even a single mistake is intolerable, this is not something that the current technology can work with. Yeah, it is absolutely the case. No, the surveys and research is finding over and over and over again, as well as anecdotal evidence, that people's threshold for how much they're willing to let AI be wrong or how much, you know, incorrectness they're willing to bear from AI is much lower than the human equivalent for whatever set of reasons, right?

21:51It's that, especially because since customer success is such a common use case where, you know, AI and agents are already production ready, they're deployed, we're seeing evidence, people's expectations are absolutely that the computer should be more right than the people. Right. Otherwise, they want to leave it for the people. To the people. Okay. So two additional things. Of course, there are many more, but just wanted to highlight a few. So there is an expectation, and perhaps we should be at fault for that, that creating an agent is very easy. And while some agents are easy to create, it's still a technology that is new, that if you want to deploy it for complex flows and complex business cases, there is a lot of preparation work.

22:34you need to have your technological stack ready, you need to have the right skills in place and the data and so on. And we even saw in one of the interviews that CTO was saying something like, I want to introduce a new agent each week. And that's perfectly fine. And we commend them for their enthusiasm. But in reality, it's probably not going to be something feasible even for the best of companies because there is still a lot of effort for introducing agents. And maybe later this year or early next year, we will be at a state that even smaller companies can introduce a new agent each week. But as of now, it will probably take a little bit longer than that.

23:13Today's episode is brought to you by Vanta. Vanta is a trust management platform that helps businesses automate security and compliance, enabling them to demonstrate strong security practices and scale. In today's business landscape, businesses can't just claim security. They have to prove it. Achieving compliance with a framework like SOC 2, ISO 27001, HIPAA, GDPR, and more is how businesses can demonstrate strong security practices. And we see how much this matters every time we connect enterprises with agent services providers at Superintelligent. Many of these compliance frameworks are simply not negotiable for enterprises.

23:47The problem is that navigating security and compliance is time-consuming and complicated. It can take months of work and use up valuable time and resources. Vanta makes it easy and faster by automating compliance across 35-plus frameworks. It gets you audit-ready in weeks instead of months and saves you up to 85 % of associated costs. In fact, a recent IDC white paper found that Vanta customers achieve$535 ,000 per year in benefits, and the platform pays for itself in just three months. The proof is in the numbers. More than 10 ,000 global companies trust Vanta, including Atlassian, Quora, and more.

24:18For a limited time, listeners get$1 ,000 off at vanta.com slash nlw. That's V-A-N-T-A dot com slash NLW for$1 ,000 off.

24:53your prompts and workflow IP. Make your life easier, your clients happier, and your business thrive with Plum. Sign up today at useplum.com. That's Plum with a B forward slash NLW. All right, AI Daily Brief listeners, today I'm excited to tell you about the Disruption Incubator. One of the things that our team sees all the time is a lot of frustration from enterprises. There's a fatigue around small incremental solutions, a concern around not thinking big enough, tons of bureaucratic challenges, of course, inside big companies. And frankly, we just hear all the time from CEOs, CTOs, other types of leaders that they want to ship some groundbreaking AI agent or product or feature.