Introducing Maturity Maps — A New Way to Measure AI Adoption

0:00Today, I am discussing a new way to think about AI and agent readiness inside your company, and it is called Maturity Maps. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

0:17All right, friends, quick announcements before we dive in. First of all, thank you to today's sponsors, KPMG, Robots and Pencils and Blitzy. To get an ad-free version of the show, go to patreon.com slash AI Daily Brief, or you can subscribe directly on Apple Podcasts. To learn more about sponsoring the show or really find out anything else about the show, like where we are with our Agent Madness Bracket, which is going on right now, head on over to aidailybrief.ai. Today is the second day in our build week, which is happening while I'm traveling with my family. Yesterday, we did a very high-level overview around everything that had happened last quarter and what it meant for this quarter.

0:51And if we put this in the framework of this being build week, yesterday was sort of the context setting, the environment in which your building is happening. Today's episode takes that down a level to discuss the benchmarks that show where others like you are. Now, one of the things that I've been thinking about a lot over the last six months or so is just how much we need a totally different set of data and benchmarks for this new AI era. Everyone is adapting incredibly quickly right now, or at least they're trying to. It's new processes, new workflows, new tooling, new everything. And by and large, we're doing all that exploration without a map.

1:28Let me give you a practical example of where I think our lack of benchmarks could actually very significantly and meaningfully negatively impact a company when it comes to their AI adoption. Let's say that you are an early adopter company. Across your different functions, you've had really strong hands-on efforts to get your AI up and running. In the absence of knowing exactly what to measure, you're just trying to measure whatever you can, and early results are pretty positive. For example, in the marketing function, you have increased your content output 30 % year-over-year without any sort of proportional increase in the resources it takes to produce that content.

2:02Now that 30 % year-over-year growth sounds great, but what if I told you that all of your competitors had actually grown their content output by 50 %? In AI world, this is actually not a far-fetched scenario, and it shows how the need for better benchmarks and numbers is not just a vanity exercise. When we don't know how we're doing relative to peers and competitors, it makes it really hard for us to judge what we need to change, what we need to shift, and what we need to do next. Now, at AIDB and at Superintelligent, which is my enterprise AI planning and strategy company, we started exploring some of this with our AI ROI benchmarking survey at the end of last year.

2:38We had people submit and share their real use cases and share with us the impact those use cases were having across an array of eight different impact dimensions, things like time savings, cost savings, new capabilities, increased output and a handful of others. We asked them to rate impact, from negative to transformational, and found that by and large, at least when it comes to people self-reporting, they were already seeing strong and positive impact from their AI initiatives. But there are a couple of clear and obvious limitations with that study. First of all, while self-reporting is better than nothing, it's always going to be somewhat imprecise.

3:12Second, like pretty much everything that we do with this audience, you have to calibrate it to a more advanced individual and organizational user than if you just surveyed a broad cross-section of businesses in the world. Third, while it did give us some great information around individual use case impact, it didn't tell us all that much about other dimensions of AI readiness and adoption outside of just the use cases themselves. Anyone who's felt the sting of the capability overhang, in other words, the gap between what AI can do and what we're actually using it for, knows that raw capability isn't really the question.

3:44It's the systems we put around it to get value from it. And unfortunately, the research and information apparatus just has not adapted to this new reality. Not to pick on Gartner specifically, but they're the biggest in the space, and so sort of present an easy target. Tried and true benchmarks and information products like Gartner's Magic Quadrant have literally never been less useful than they are right now. Now, the idea that success in something like AI application development was going to be even a little bit dictated by choosing the right AI application development platform vendor is just so far outside of the reality of these tools as to be almost actively harmful if that's where you're putting your time and effort when it comes to trying to figure out how to adopt AI.

4:27Now, Gartner is more than the Magic Quadrant, and they are doing lots to try to catch up to the AI world, so it's not to single them out. It's more to make the point that we are in desperate need of some new frameworks, some new benchmarks, and some new tools. And so at both AIDB and at Superintelligent, we've been thinking about this a lot over the last, call it three to four months. We've experimented with a couple things that you'll probably see some version of come out at some point in the near future. One of them I call AI opportunity radars, which are basically a way of organizing use cases by function, but then also by applicability, depending on where your organization is in its development cycle.

5:02Simply put, it's a radar or bullseye type of visual where use cases are organized into one of three categories. Primetime means that most organizations, as they are, are well-suited to get value from that use case right now. Emerging means that while there are a lot of organizations that can get value out of them, there is some amount of setup cost or rights out of circumstances or infrastructure that is going to be needed to get value from that, and not all organizations are going to be there just yet. Finally, Frontier is exactly what it sounds like. where if your organization is well set up with the right infrastructure, you can be getting a lot of value from those use cases.

5:37But at this point, most organizations aren't there yet. So over the last quarter, we built an agentic system that is basically constantly seeking out every new resource it can get its hands on, assessing what those resources tell us about the use cases in different functions, and keeping these radars continuously updated, both with new use cases as well as changes in where the existing use cases are placed. But as we were working on radars, it was clear again that there was something even more fundamental, and that overly or only focusing on use cases was leaving out so much of what actual AI readiness means.

6:10When we're doing AI readiness and planning assessments at Superintelligent, we're not just thinking about what use cases a company should do, but what's the full set of change management and infrastructure development and new policy and investment in people and all this other stuff needs to go around it to actually get value from those use cases. And that led to the development of the framework which I'm going to be sharing today, which we call for simplicity AI maturity maps. Now, the concept of maturity is certainly not some proprietary thing that we invented. Maturity is just a heuristic and a framework to look at where different organizations are around some key areas relative to one another and where they should be.

6:46So the way that maturity maps work is that they organize AI and agent maturity into six different categories. Those categories are first deployment depth, which is sort of an expanded notion of use cases. Deployment depth in the context of AI maturity not only thinks about how many use cases you have in play, but how much those use cases are assistance versus full workflow automations versus actual applied agentic systems that are doing work with some meaningful degree of autonomy. The second category is systems integration. This is a measure of how deeply integrated the AI solutions and workflows that you're deploying are integrated with the existing systems that run your enterprise?

7:25Is everyone using ChatGPT independently, or does your CRM system have an agent running through it, automatically extracting insights, making recommendations, and even setting up new outreach campaigns? Systems integration is in some ways one part of the measure of how good the context that an enterprise's AI has to work with. Now, the other piece that relates to context is, of course, data. How much, what quality, and how well managed is your company's AI's access to your company's data? Does it require are people dropping in PDFs? Do you have company knowledge all set up on MCP servers? How does the AI that your company is looking to transform your company have access to the information it needs to know what that transformation should look like?

8:04Outcomes is almost a measure of measurement. Are all of your deployments pilots and experiments? Or do you have a track record of actual demonstrable and measured outcomes? Outcomes in some ways are the information you need to know what you should do next across all these other dimensions. The fifth dimension of AI maturity maps is people. And this is an admittedly broad category. A big part of this refers to upskilling and capabilities, but another piece has to do with attitudes. Given that one of the major barriers to adoption in many companies is not just going to be skills using AI, but attitudes towards AI, people is an extremely important and unfortunately, as we'll see, often neglected piece of the AI maturity pie.

8:43Lastly, of course, is governance. How clear, how established, how communicable, how known are the rules and guidelines and access provisioning around your AI systems? Do people know where to go to get the permissions they need? Do they know what expectations are? When issues come up, are there mechanisms for resolving those issues? So those are the six areas across which we look at AI maturity. Now, for the purposes of developing these maps, we've started with 10 functional maps split across some of the most common, very broad brush categories of knowledge work. That includes customer service, engineering, IT, which by the way, the difference between those two for our purposes is effectively that engineering is all the stuff that's external facing and IT is all the technology stuff that's internal facing, sales, marketing, HR, operations, finance, legal, and product.

9:30So at the end of last year, we started to put together a process for actually assessing and visualizing AI maturity across all these dimensions. What came out of that is the chart that you see here, which plots each of these six categories within a specific function on a five-point scale. Number three, the center of the chart, is the on-track line. In other words, where an average organization should be. And the word should, as you'll see, is doing a lot of heavy lifting there. Now, if on-track is a three, four is ahead and five is significantly ahead, while two is behind, and one is significantly behind, is the idea is that when you look at a maturity map, without having to read a lot of words, you can instantly see the gaps between where organizations should be and where the average organization actually is.

10:13And when you compare your organization to it, also see where you are relative to both the general on-track line and the average. So clarifying this a little bit more, a quarterly's designation of on-track is not where the average organization is. It is a subjective measure of where we think the average organization should be. As you'll see when you dig into this quarter's numbers, in the vast majority of cases, we believe that the average organization is behind that on-track line across pretty much all of these dimensions. To use a term that comes up a lot on this show, the fact that organizations tend to be behind this on-track line is effectively a visualization of the capability overhang.

10:54Now, at this point, you might be wondering, well, what gives you authority to determine what the on-track line is? It's a totally reasonable question, and believe it or not, it is a little bit more, at least, than just my opinion. We have a few different places to pull from. The first is the sort of proprietary research and surveying that we do as part of AIDB Intel, which gives us some pretty good insight into where particularly leading organizations are. Second, it's super intelligent, given that we are doing thousands and thousands of voice agent interviews every month to help organizations assess their AI maturity and plan their AI strategy.

11:24That's another pretty unique source of frontline data. And then combined with that, we built a system to go out and effectively aggregate pretty much every new survey or study that comes out that even vaguely touches AI. You might have heard me mention before that my most useful open clause are my research open clause, and this is one of the main things that they do. They are in a never-ending 24-hour-a-day, constantly hunting loop to both surface new sources, to assess those sources in terms of their legitimacy, credibility, and bias, and then to integrate that information into our larger assessment system.

11:57There are more than 480 studies and surveys from the last quarter that went into these Q2 maturity maps. Among the sources that have explicit sample sizes, the combined survey respondent base exceeds 150 ,000 professionals across more than 50 countries. The types of source categories that we have are one, big four and top tier consulting firm research. There's over 20 of those sources in that mix, major platform earnings and public market statements, analyst firm predictions and research from companies like Gardner, Forrester, and IDC, function-specific regular or annual surveys, such as Stack Overflow's engineering study, or other similar things for areas like marketing, legal, and IT, academic and government research, behavioral data sources, where companies that have access to some unique user behavior data aggregate, analyze, and share that.

12:44A good example of that is Jellyfish's AI coding benchmark, which used behavioral data from more than 200 ,000 engineers across 700 companies with 20 million PRs. Finally, there are, of course, practitioner reports and vendor case studies, although the system is careful to rate them with some amount of skepticism given that they are, of course, selling something.

13:07All right, folks, quick pause. Here's the uncomfortable truth. If your enterprise AI strategy is we bought some tools, you don't actually have a strategy. KPMG took the harder route and became their own client zero. They embedded AI and agents across the enterprise, how work gets done, how teams collaborate, how decisions move, not as a tech initiative, but as a total operating model shift. And here's the real unlock. That shift raised the ceiling on what people could do. Humans stayed firmly at the center, while AI reduced friction, surfaced insight, and accelerated momentum. The outcome was a more capable, more empowered workforce.

13:41If you want to understand what that actually looks like in the real world, go to www.kpmg.us slash AI. That's www.kpmg.us slash AI. Today's episode is brought to you by Robots and Pencils, a company that is growing fast. Their work as a high-growth AWS and Databricks partner means that they're looking for elite talent ready to create real impact at velocity. Their teams are made up of AI-native engineers, strategists, and designers who love solving hard problems and pushing how AI shows up in real products. They move quickly using RoboWorks, their agentic acceleration platform, so teams can deliver meaningful outcomes in weeks, not months.

14:21They don't build big teams. They build high-impact, nimble ones. The people there are wicked smart with patents, published research, and work that's helped shape entire categories. They work in velocity pods and studios that stay focused and move with intent. If you're ready for career-defining work with peers who challenge you and have your back, Robots & Pencils is the place. Explore open roles at robotsandpencils.com slash careers. That's robotsandpencils.com slash careers. You've tried in-IDE co-pilots. They're fast, but they only see local silos of your code. Leverage these tools across a large enterprise codebase and they quickly become less effective.

14:56The fundamental constraint? Context. Blitzy solves this with infinite code context, understanding your codebase down to the line-level dependency across millions of lines of code. While co-pilots help developers write code faster, Blitzy orchestrates thousands of agents that reason across your full codebase. Allow Blitzy to do the heavy lifting, delivering over 80 % of every sprint autonomously with rigorously validated code. Blitzy provides a granular list of the remaining work for humans to complete with their co-pilots. tackle feature additions, large-scale refactors, legacy modernization, greenfield initiatives, all 5x faster.

15:27See the Blitzy difference at Blitzy.com. That's B-L-I-T-Z-Y dot com.

15:36So in Q2, what are some of the patterns that we saw? The first you might call the adoption embedding gap. Basically, every single function-specific survey reports the same pattern. High claimed adoption, but at fairly low depth in utilization. This is maybe the most dominant finding across all these sources, that the story of Q2 when it comes to enterprise AI adoption is not just the capability overhang in general, but the applied capability overhang even when it comes to adoption inside an individual organization. A second very common finding across a huge array of these sources, there tends to be a fairly big gap between worker-level data and leader-level data.

16:14For example, one study found that in the area of customer service, 72 % of leaders said that their AI training was adequate, with 55 % of their employees disagreeing. In HR, a huge percentage of leaders report that AI is a priority, but more than two-thirds of HR staff say that their organizations are not proactive in upskilling. In fact, one can argue that people are the bottleneck that is not getting nearly enough investment. We gave seven of the ten functions a score of one significantly behind when it came to that people category. The irony is that one could argue that the single largest barrier to converting AI adoption into AI value is on the human side, and it's the thing organizations are spending the least on.

16:56To use one dramatic example, Deloitte found 93 % of AI spend going to infrastructure, with only 7 % going to anything related to people. Now, outside of people, another finding across all of these sources is that data is kind of the ceiling on everything else. Eight of the 10 functions score a 1 or a 1.5 on data. Now, I don't need to beat this drum anymore for this audience, but obviously without proprietary context feeding AI, things like your code base, your customer history, your deal data, you really are not going to get past basic assisted usage no matter how good the assistant tools get.

17:28One could argue that data is not one pillar among six, but the floor constraint that caps all the others. Another area with universal challenge is around outcome measurement. And this is not surprising. Given how much pressure there has been to adopt AI as fast as possible, one of the consequences of that is that no one slowed down, paused their adoption while they went out and figured out how to actually measure the ROI of all those investments. Can you imagine right now someone in the C-suite suggesting with a straight face that you take six months off of adoption to figure out better ways to measure the ROI first to ensure that you weren't spending too much?

18:02That, my friends, is a recipe for an early retirement. The byproduct of that, however, is that the actual evidence for AI ROI is pretty thin. Now, I will say that if I had to make a prediction about one area where you're going to see the biggest glow up this year, I think there are tons and tons of efforts around ROI measurement, and I would expect that to jump significantly in the quarters to come. Across all of these functions, the vast majority do not have any scores that are actually on track. Customer service we rated on track in terms of deployment depth and systems. It makes sense given how much focus there has been on customer service-related solutions, vertical and otherwise.

18:40Engineering as well, we have as on track for deployment depth, systems, and people. Again, given that software engineering organizations are effectively the harbingers and guinea pigs for everything else that's going to happen to all the other knowledge workers, I think that this will be intuitive. And then relatedly in IT, we also have deployment depth, systems, and people as on track. Now, of course, all of these areas share structural advantages, mature tooling, technical practitioners, and measurable workflows. Unfortunately though, even if it is the case that there are bigger challenges in some ways when it comes to the technical capabilities of practitioners and the measurability of what you do, those other areas of the enterprise are going to have to catch up.

19:16Now a few observations looking across the different functions. While customer service did have a couple areas that we rated as on track, it also I think reveals something that could be a harbinger for other areas. Remember, when it came to CS, we heard 72 % of leaders say that training is adequate, but 55 % of people actually working in CS say it's not. 87 % of customer service workers report high stress, and 75 % of leaders acknowledge that AI may be increasing stress. So you've got a situation where AI is absorbing routine cases, humans get the harder, more emotional ones, many people might not be trained for that shift, and that, plus the fact of just increased questions about the long-term sustainability of your job, the result is stress, anxiety, and burnout.

19:56Basically, CS could be the canary in the coal mine for what happens when you deploy AI without investing simultaneously in the humans who work alongside it. One of the areas that I think is interesting to point out is the two rating or behind when it comes to governance. In most organizations, IT owns AI governance for a big part of, if not the entire organization. And yet only 54 % have centralized frameworks. 50 % of AI agents are unmonitored. 88 % have had security incidents. And the question becomes, if the governance function is behind ungovernance, what does that tell us about the rest of the organization?

20:29One of the most interesting findings when it comes to sales is that it might be the cleanest example of the adoption mirage. 88 % of sales teams say they use AI, but only 24 % have it in their actual revenue workflows. A fair bit of adoption then, in other words. This is why we rate them behind in deployment depth. Much of the quote-unquote adoption is reps using ChatGPT in a separate browser tab for email drafts and call prep. Which is not a bad thing at all. It's just not the level of automation and autonomy that I think sales organizations are hoping for. The autonomous SDR dream has not fully come to fruition yet.

21:01And I don't think that most sales organizations have figured out the right integration and balance between humans and agents in the new sales working system. The deployment depth score on operations I think is another really interesting one. In some ways, operations has had automatable functions longer than any other function in the enterprise. Think statistical forecasting, rules-based inventory management, predictive maintenance. That's all stuff that predates this latest wave of Gen AI by a decade. What that means is that when 90 % of operations teams say they're investing in AI, it sounds impressive.

21:31But when you actually look at what that is, a lot of it is legacy optimization infrastructure that's been running since, I don't know, 2015. The Gen AI layer on top is often very thin, mostly asking AI questions about operational data and generating reports. In fact, one study found that only 23 % of operations groups even have a formal AI strategy. Operations is the function where the distinction between old automation and new AI maturity is showing up as a real distinct challenge. Lastly, one more interesting stat, outside of the technical areas of engineering and IT and the long-duration area of customer service, which has had so much emphasis on automation, finance is the only other function to hit on-track on any pillar, and it does so on governance.

22:11Now, why do you think this might be? It's because, of course, finance is operating in an area where governance is not optional. 69 % of CFOs report advanced or established AI risk governance frameworks. Why? SOX compliance, audit trails, fiduciary duty, decades of regulatory muscle memory. Basically, finance already knew how to govern risky tools even before AI existed. Now, don't look at the rest of finance, where we rated it significantly behind in every other category. Basically, they know how to control AI, but haven't figured out how to use it. What's interesting will be whether over the next few quarters, we see this turn into a tortoise and a hare thing.

22:48In other words, when finance does figure out how to deploy, will they do it more safely and more effectively than functions that deployed first and governed later? And will that actually allow them to catapult at some point and jump out ahead of other functions who at the moment feel like they are farther when it comes to deployment depth? So this is the idea of maturity maps. They are, of course, very nascent. This is the first quarter where we've actually fully trotted them out. And in the spirit of getting feedback and seeing how useful these things can be, we're putting up the ability on the super intelligent website, besuper.ai, to actually go not only review all of these maturity maps, but do a short quiz that actually shows you where your organization is relative to both the on -track line and what we think is the average.