A Framework for Choosing Winning AI Use Cases [Agent Readiness Part 3]

0:00Welcome back to the AI Daily Brief. Today we have part three of our agent readiness series featuring Nufar our guest bar, and we are digging into use cases. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

0:18All right, quick announcements before we dive in. First of all, thank you to today's sponsors, Blitzy, KPMG, Rovo, and Robots and Pencils. To get an ad-free version of the show, go to patreon.com slash AI Daily Brief, or you can subscribe on Apple Podcasts. For information about sponsoring the show, we are rapidly running out of inventory for Q1 now. So if you are interested and want to hear more about what's available, send us a note at sponsors at aidailybrief.ai. Lastly, a reminder as always about our AI ROI benchmarking study. Thanks to all of you who have contributed now hundreds of use cases.

0:49If you would be willing to take one or two minutes to do the same, it is at roisurvey.ai. In the first episode, we talked about the cultural dimensions of agent readiness with Nufar introducing her change framework. In part two, we talked about data and technical readiness, where we have found at Superintelligent that data issues are by far the biggest blocker for agent adoption across all the different challenges that enterprises face. Today, we are talking about what makes for a good use case and where to invest your resources and time. With this, you're going to get a little bit of an inner working in the way that we do use case recommendations, and it should provide, once again, a nice actionable framework for how you can think about which use cases might benefit your organization most.

1:29All right, Nufar, welcome back. Part three of three. This is technically we're calling this use cases, but I think this is like the make it practical, like go do interesting things kind of section of the conversation. I'll let you kick it off and frame it for us. All right. Yeah, that's a good take. Like you said, three out of three, final part of the agent readiness series, and we did save the best for last, and that is the use case readiness. And we don't just refer to the, are there enough use cases in production, but rather we are talking about whether there are enough opportunities in the company.

2:01And often these are not opportunities that the company is able to articulate, but rather ones that we're able to identify for them. It can be a question on whether the business process in the company could be augmented or replaced by agents and whether they have the right mindset in order to do proper use case discovery and execution. So that's what agent readiness from a use case perspective is. And in terms of the process, and because we are all such big fans of frameworks, I wanted to cover this topic following the following steps. So we will start with identify and then select, manage, and track our use cases.

2:39And I want to give you enough practical tools, not only to identify the next agent use case, but also to manage it almost like an investment portfolio, such that you can continuously improve your use case readiness. So let's start with the identify phase. And first, I want to talk about the sources of ideas for agents. In many cases, what I'm hearing is that like a CEO or a company leader will go into a room and they will say, we need to build an agent. Let's build an agent. And this is probably one of the worst way to source a use case idea because you'll probably build the wrong agent. Good source for you to build these agent use cases will be probably to get it from either bottoms up or mid-level up because they often are the ones that know best what is feasible and also they are the one that needs to be bought in in order to execute.

3:32The question still remains, what are these good use cases? And what I wanted to do here is to provide you with some hints for what good use cases are and sometimes what good use cases aren't. And I want you to look first for use cases that involves a very complex and highly changing decision-making. For example, to resolve a customer issue will require a different action each time there is a new customer issue being raised. On the flip side, and that's my very aggressive opinion, that in any situation where you can describe a fixed process or a decision tree with limited amount of branches, you shouldn't build an agent.

4:12you should just go for the simpler technology because the agent will probably not be worth it. The next thing that I want you to consider are places where humans are in the loop, but they are being the bottlenecks. This is often where you will find a goldmine of ROI for your use case because you need a professional judgment and you just don't have enough professionals. Think, for example, a legal contracts review. Another place where you should look into are places where you need to have 24-7 human response. It can be for employee support, for customer support, and so on. And also, I want you to think about cases where you want to achieve a high level of personalization.

4:50Think, for example, where you want to issue a highly personalized email, not just one where all the text is the same and you say, hi, Nufar, this is not the one. I'm talking about cases where you will issue a personalized outreach at the right time with the right offer and with the right text to get me hooked into your product. Next thing that I want you to consider is that I want you to only focus on use cases where you do have some tolerance for errors. For example, with one of the companies that I worked with, Payroll said, we want an agent to replace some of the processes for calculating the employee's salaries.

5:24And after I stopped catching my breath from how intimidated I was with this notion, I said, guys, you don't do an agent here. You don't even include AI in this process. You do a simple automation because with salary, you have to get it 100 % or 200 % right. So agent will not be the right way to go about this process. And until now, we didn't talk about kind of the elephant in the room. But because around agents, there is so much discussion and sometimes fear around job loss, a good place to start will be to focus on the use cases that employees want to offload. Those will be the repetitive or the tedious type of works.

6:01You should focus there and then you will create a better momentum for more sensitive use cases. And also in cases where the only way to understand how a job is being done is to say, hey, Sarah, can you explain how the job is being done? Because Sarah is the only one who knows. This is not a good place for an agent. We need to have a process that is well-defined and well-documented for an agent to be able to interject. And lastly, we want to have a use case that is extremely measurable, not just to measure the ROI, but because these agents are goal-driven entities. And if you cannot measure whether you are closer or not to your goal, you cannot implement an agent.

6:40So if you use all of these hints, one thing that you can do at your company in order to initiate the creation of many, many such ideas is to have an ideation sprint. And this will help you harvest more agent ideas in the company. And what you do there, typically you will teach the employees using slides like that or others, what agents are and aren't and what they can and cannot do. Then you harvest many, many, many ideas across the entire company. And then with this very large inventory, you do a very crude prioritization of your use case to keep it simple. And here I want you to be very aggressive and nip in the bud any use case that you can execute in a different manner, automation or otherwise.

7:22All right? So that's the first step. And with so many ideas, the next step will be to select and plan an agent roadmap. The selection should always be driven by the holy tree, the feasibility, the investment, and the value. And this means that you will need to score each of our use cases accordingly. And if you've heard the previous episode where we talked about the intentional opportunism, you by now know that I'm very, very much into having low-hanging fruits and focusing on them first. And these should be use cases with high feasibility and low investment, or ones that are highly critical and start with them.

8:02And then you will create the momentum of learning and doing in order to benefit future higher stakes use cases. So let's make it even more concrete and talk about the list of use cases that almost all companies should consider. And these are ones that we often find ourselves uncover or recommend companies when we audit them for agent readiness. And I think none of them will surprise you. But the first one will be the FAQ or the policy bots. These are either internal or external. And I'm not talking like old-fashioned bots that only know to answer from a predefined set of questions, but rather agents that are able to answer complex questions with many nuances in the information sources.

8:42The other thing that everyone needs and are asking for will be the company knowledge retrieval. I think this comes across all interviews. Everyone needs good access to their data. And even though many companies are already utilizing the Microsoft and the Glean and other solutions, often these are not enough and they need to create either an additional layer or looking for additional ways to get access to their very specific data that is fragmented across systems and so on. The next one will be operational workflow automation. So those will be drudge work at the team level, like automated status reporting and many other things that people spend unnecessary brainpower and time to do and agents can take from their plate.

9:24And lastly, I'm calling them like market watchers. These are everything related to keeping a close eye on your competition, on your regulation or everything that you need to know in order to do your business well. And you never have enough time. So these are the top most prevalent use cases, but we do find ourselves recommending many other use cases as part of the readiness audits. And I don't believe that anything on this list will surprise you. And of course, it starts with the top two most common use cases. And those will be vertical use cases around customer support and software engineering.

9:58They come across very often. But we also see in many cases content generation for marketing or other purposes, as well as many sales related use cases. They come across very, very often in the audits and with many companies that we've been working with. And this is also something that I mentioned in a previous session. Agents can often help with things related to contract and regulations and other things, and with the process of cleaning your data, which is where there is a lot of unlock with the data access issues and many things that we mentioned before. And lastly, there are many industry or even company specific, often ones that will create opportunities for growth in the company that come across as highly relevant in some of these audits.

10:44So this is a very rich selection of use cases. And what I want to encourage everyone to do is basically to manage the inventory and the choices of which agents and which use cases you want to pursue, like you would manage an investment portfolio, as I said at the beginning. And I want first to have you balance between two main elements, and those will be the efficiency or the cost-focused use cases, and also those that are more focused on the growth. And when I talk efficiency use cases, I'm talking about all the do the work with the fewer resources type of use cases, while growth use cases in my book are everything that basically has an impact on the top line.

11:26And I want you to balance the two. Often we're seeing companies highly biased towards the first one of efficiency and not thinking enough about the growth opportunities. And often the biggest value is on the right-hand side of the growth. So make sure that you pay a closer attention to those as well. And then to continue on balancing your agent portfolio, I want you to look at this proposal. And here I'm paraphrasing on the 1970s Boston Consulting Group gross share metrics. It's a kind of a classical. And I want you to look at the identified use cases in the lens of the complexity versus the value.

12:04And of course, where there is high complexity and low value. And often we see such use cases just don't go there at all. There are, of course, the low-hanging fruits or low-hangers. These are an awesome place to start. And over time, though, those should become the thing that people self-serve. So these should be catered by agent-building platforms or other capabilities rather than having a company focus And eventually, you should aim to have a handful of what I refer to as moonshots. These are kind of the high risk, high reward use cases. Often, by the way, they correspond to either a radical shift in how you do the work or they are a growth use case.

12:47So that's often where the moonshot is. And they should be led and executed by a professional and centralized AI team rather than just the best effort in the business units. And they require a significant investment and a specialized knowledge. So these are not for the faint of heart. And lastly, most of your area of focus should be in this like high or high-ish value and decent complexity because this is where there is enough value, but still you will be able to get agents out the door. And by the way, for companies that are getting low readiness scores from us in the audit, We will never offer ideas for stuff that are more on the moonshot.

13:31We will always focus them either on the low hangers or the focus areas. And over time, we'll encourage them to go after broader, bigger things. So there are a few other dimensions that you should probably consider. One is to balance between vertical versus horizontal agents. Don't just do one or the other. And also build versus buy. So keep that in mind as you balance your portfolio and make sure that you have diverse and well-balanced portfolio of all the complexity combinations, as well as vertical, horizontal, and build-verse. Okay, so this portfolio has to be centrally managed and periodically updated in order to support the constant learning and the growth.

14:15This episode is brought to you by Blitzy, the enterprise autonomous software development platform with infinite code context. Blitzy uses thousands of specialized AI agents that think for hours to understand enterprise-scale code bases with millions of lines of code. Enterprise engineering leaders start every development sprint with the Blitzy platform, bringing in their development requirements. The Blitzy platform provides a plan, then generates and precompiles code for each task. Blitzy delivers 80 % plus of the development work autonomously, while providing a guide for the final 20 % of human development work required to complete the sprint.

14:46Public companies are achieving a 5x engineering velocity increase when incorporating Blitzy as their pre-IDE development tool, pairing it with their coding pilot of choice to bring an AI-native SDLC into their org. Visit blitzy.com and press get a demo to learn how Blitzy transforms your SDLC from AI-assisted to AI-native. What if AI wasn't just a buzzword, but a business imperative? On You Can With AI, we take you inside the boardrooms and strategy sessions of the world's most forward-thinking enterprises. Hosted by me, Nathaniel Whittemore, and powered by KPMG, this seven-part series delivers real-world insights from leaders who are scaling AI with purpose.

15:23From aligning culture and leadership to building trust, data readiness, and deploying AI agents. Whether you're a C-suite executive, strategist, or innovator, this podcast is your front-row seat to the future of enterprise AI. So go check it out at www.kpmg.us slash AI podcasts, or search you can with AI on Spotify, Apple Podcasts, or wherever you get your podcasts. Meet Rovo, your AI-powered teammate. Rovo unleashes the potential of your team with AI-powered search, chat, and agents, or build your own agent with Studio. Rovo is powered by your organization's knowledge and lives on Atlassian's trusted and secure platform, so it's always working in the context of your work.

16:05Connect Rovo to your favorite SaaS app so no knowledge gets left behind. Robo runs on the Teamwork Graph, Atlassian's intelligence layer that unifies data across all of your apps and delivers personalized AI insights from day one. Robo is already built into Jira, Confluence, and Jira Service Management Standard, Premium, and Enterprise subscriptions. Know the feeling when AI turns from tool to teammate? If you Robo, you know. Discover Robo, your new AI teammate powered by Atlassian. Get started at ROV, as in victory, O.com. Today's episode is brought to you by Robots and Pencils. When competitive advantage lasts mere moments, speed to value wins the AI race.

16:45While big consultancies bury progress under layers of process, Robots and Pencils builds impact at AI speed. They partner with clients to enhance human potential through AI, modernizing apps, strengthening data pipelines, and accelerating cloud transformation. With AWS-certified teams across US, Canada, Europe, and Latin America, clients get local expertise and global scale. And with a laser focus on real outcomes, their solutions help organizers work smarter and serve customers better. They're your nimble, high-service alternative to big integrators. Turn your AI vision into value fast. Stay ahead with a partner built for progress.

17:18Partner with Robots & Pencils at robotsandpencils.com slash AI Daily Brief.

17:28If you're talking about growth, let's focus on this one, on the tracking. and the topic of measurement is one that is very important for us. I know that I completely disregard the build phase, not for lack of importance, but since it's like an entire episode. So I'm skipping over the build and moving on to the track side of things. And I want you to continuously monitor everything that you put in production and that means everything. And you shouldn't assume that just because you had a very good result in the pilot, it will be so in production. We often see that once something is deployed, the hands of actual users, not just alpha or testers, things go haywire and yield a completely different value than expected.

18:12And we've seen that more companies that are being underwhelmed by what they put in production, rather than being pleasantly surprised, the performance and quality and cost, and especially if they didn't measure enough in pilot and won't measure enough in production. So that's why it's so critical for us. To make it very concrete, I provided you here with a formula of how you measure the agent ROI. So it's going to be like a return versus an investment. So return should be measured as the benefits from the agent. And you should make sure to include the usage and the impact. And you should continuously measure.

18:52It's not easy. and whenever there is a judgment call, I encourage you to be conservative rather than anything else so you can get a realistic view of what your agent actually accomplishes and you should discount the uncertainty of the benefits. So in many cases, I'm seeing companies not accounting for the cost and the impact of the errors. If your agent causes you to have to have a refund of something, then that's a cost of managing your agent and it needs to be discounted from your return basically and you should subtract it from the investment. An investment will include the resources and it should be all the resources, what it costs you to build or to buy, what it costs you to use and what it costs you to maintain.

19:32And often we disregard these resources that will get the agent to be sustainable in production. So it will be the owner, the user's time, and of course the tool and model use. And here, especially when you do a projection, take sufficiently large buffers because in most cases, we completely underestimate the amount of investment we will need to do. And of course, you need to take into consideration the cost of your resources. And you can assume cost variability because we are seeing that models cost going down. However, with agents, because they become more sophisticated, often they will require more tokens and thereby balancing things out.

20:08So just reevaluate periodically. But there are many hidden costs and you should try to account for that. So that's the most technical slide. And with that, I want to summarize this session and give you a concrete to-do list. So first, I want you to identify only agent relevant and worthy use cases, not vibes. And then I want you to select the ones that have enough ROI and enough visibility and manage them as if you are managing your investment portfolio such that you have a diverse, well-balanced portfolio. and then rigorously track the impact and adapt over time. So you don't just get it right in theory, but you get it right in practice.

20:46Awesome. So first of all, I have to mention this. For the first two parts of this series, when we were talking about the culture and leadership changes that were required to really do this well and the data and technical readiness, we can help in ways like this, by providing information, best practices, comparative things that we're seeing, but our ability to help, and this is I'm speaking in terms of super intelligent now, is limited. But with this one, this is exactly what super intelligent does is help create systems for figuring out what use cases you should pursue. And it follows a lot of this thinking, especially this sort of bottoms up idea of actually discovering what's going to be useful from the ground level from these sort of actual work level perspectives of people.

21:33So if you are interested and need help with this, that is my shill for super intelligent for this episode. This is what we do pretty precisely. A couple other things that I wanted to double click on. The first one is really, really small, but I think I talk about it a lot because it's surprising to people. In that portfolio balance, the low hanging, but sort of like low nudging into high value, it would appear low value, but I actually think it's higher value than people think, is really that company knowledge retrieval, internal information sharing type of use case. We see this so frequently as a gateway drug for people.

22:06And what's interesting about it, and I think what pushes it from a low value to high value, or low medium value to more high value, is that the value is actually double. The first part of the value is, of course, actually getting people the information they need, what you've set out to do with that agent or that AI. But the other part is, it is like a light bulb moment for a lot of folks when they use those tools, that sort of starts their own individual thinking about how AI can be useful for them, right? When they have a problem that gets solved much more quickly than it otherwise would have, as opposed to, for example, just like my email is a little bit better than it was before because I use ChatGPT or something like that.

22:43I think it really is, we very often see that being a surprisingly high value starting use case. Yeah, it's the magic of context, basically. Yeah. Okay. So then another thing that I want to mention is on the ROI front. I think it's easy to talk about ROI, it's much harder to figure out how to do it. This is something that we're spending a lot of time thinking about better systems for helping people with. I think even having some framework like you provided at least gets people thinking more comprehensively about it. It is not as simple as performance analytics from traditional SaaS software, unfortunately.

23:18It is not the same as just what percentage of people use the tool. You have to add on some layer of exploration and insight to really understand how it all came together. However, it is absolutely the case that broad expectations of when ROI is going to show up are being pulled forward fairly dramatically right now. So KPMG recently came out with their annual CEO study. And among these CEOs they surveyed, it's something like 1300 CEOs, all$500 million companies or bigger, something like 65 % of them thought that it would take three to five years to realize ROI from their AI efforts. So this was just last year.

23:59Only about 20 % said one to three years. This year, they just released these results. It's now 69%, I think 67 % or 69 % say that it's going to be one to three years. And 19 % said six months to one year in terms of how fast they think they're going to actually see ROI, right? Like the investment has paid for itself. Moreover, I just recently saw actually just before we were recording that Morgan Stanley has said publicly that they believe that the cost that they have put into AI has now been made back in terms of value so that they are actually ROI positive. ROI is something that has been lurking as a thing that would be sort of for some time in the future.

24:40I think that even with all of this happening, it's still complex to understand what ROI is, but you're absolutely going to see way more companies showing up and saying, no, we actually have figured out systems that we're comfortable with and confident in and are seeing that happen now. This is not some future far off thing, I think. Yeah, I agree. I think 2026 is going to be in many cases, the show me the money year for the companies that have been doing the usage metrics and the let's play with the tools and encourage usage type of situations. However, in many cases, ROI measurement is very involving because you need to do like A-B testing.