Are 95% of Businesses Really Getting No Return on AI Investment? — With Aaron Levie

0:00Why are the headlines telling us that businesses are getting no return on AI investment? and our AI agents finally ready to get to work. We'll cover it all with Box CEO Aaron Levy right after this. Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. Well, today we're going to talk about AI and its application in business, whether it's actually making a difference and whether AI agents are a real thing. We have the perfect guest to do it today because we have Aaron Levy back with us, fresh off the BoxWorks AI event. And Aaron, it's great to see you as always.

0:40Thank you. Thank you, Alex. Good to be here. So did I add BoxWorks AI event or is it just called BoxWorks? And I'm just jumping around here. I like you calling it an AI event. It is just called BoxWorks, but anytime you want to jam an AI in there, we're good. Okay, sounds good. You had a lot of AI news. We'll get into that in a moment. But since you are talking with a lot of folks about AI applications in business, I want to run this MIT study by you and get your perspective on what's real and what's not. So this is from Axios a couple of weeks ago. MIT study on AI profits rattles tech investors.

1:16Wall Street's biggest fear was validated by a recent MIT study indicating that 95 percent of organizations studied get zero return on their AI investment. They studied 300 public AI initiatives trying to suss out the no-hype reality on AI's impact on business. 95 % of organizations said they found zero return despite enterprise investment of$30 billion to$40 billion into generative AI. This has been a study that everybody in the business world is talking about. Do you think there's any validity to it? You're already shaking your head. I'm shaking my head on actually like seven dimensions. We can parse each one.

2:26So it's this very kind of bipolar state of where are we in AI adoption versus AI is going to be so powerful that there's not even going to be software business models because everything will just be delivered by AI. And as with most things that have these kind of extreme polarization elements, I think the reality is just way more nuanced. We are still early in the adoption curve of AI. In the early curve of all of these types of technologies, you have lots and lots of proof of concepts. You have lots of trials of different technologies. People are trying to figure out which tool works for which use case.

3:03So by definition, you're kind of in the Wild West where there's lots of attempts at trying these technologies with various vendors and technology stacks. And many of those projects and pilots will absolutely fail because by definition, they're pilots. and we're still in the early phases. One interesting thing about this study was they saw a significant delta between companies that tried to effectively DIY their AI stack versus going with really kind of applied solutions and use cases. And this is what we tend to find in our customer base. So I think there was maybe an initial theory of, well, AI will be relatively easy to kind of get our arms around.

3:43We could build our own AI application. We'll do all of the vector embeddings of our data ourselves. We'll put it into a vector database. We'll manage the security and permissions of data access ourselves. And before you know it, a company that wanted to deploy AI in a particular workflow in their enterprise, they might have 10 or 15 different pieces of software that they have to run and manage just before a single user could actually interact with AI within that organization. So that's probably an architecture that's not going to work. You need to have purpose-built solutions that solve sort of tailored use cases.

4:17Those can be very big use cases like all of AI coding, but you probably don't want to be in a position where you have to kind of bootstrap this or build it all out yourselves. And that was one of the kind of recognitions in the survey. But I obviously wholeheartedly disagree with any of the maybe conclusions other than just you have to get your use cases right. You have to target the most effective areas for AI, and you probably shouldn't be building this technology yourselves. And so, but it's sort of empirical on our end. We get to talk to customers every single day that are seeing the immediate gains.

4:54We've talked to customers where they have had colleagues that can't actually, they can't present the actual ROI savings to their board. The actual kind of expected ROI savings to the board because the board won't believe how, they won't believe the numbers based on how good they are. So they actually have to water them down. So it's actually more pragmatic and believable based on what they're seeing. So that a terrible board. I mean, if the board can't hear the truth, well, the board, but the truth is so good that it doesn't sound credible. So so that that is the like when the ROI is so good that you actually don't you aren't going to be believed when you actually explain how this thing is going to work.

5:43So we're seeing examples all across the board, at least for our customers. We have the benefit of a very applied use case, which is we take documents and unstructured data, and then we have AI agents that can operate on that data to do things like extract structured data from your documents. So give us 100 ,000 contracts, and we'll pull out the structured data fields in those contracts. Or give us invoices, and we'll pull out the key details in an invoice so we can help automate a workflow. Those use cases tend to be very high ROI because either you weren't getting that data before or it used to be very expensive to do so.

6:18And AI is getting increasingly good at being able to execute that kind of task. And so there's immediate benefit to customers. You can automate workflows much more easily as a result. You can lower the cost of operations in some areas. So we tend to see a different set of outcomes based on the AI adoption within our customer base. But, you know, if you zoom out and you kind of think about all projects across, you know, the past couple of years, I do think you're going to get a mixed bag just as a reality of how early we are in the space. Yeah. And it says internal builds fail at double the rate of external partnerships.

6:49So spot on there. People trying to parse this together on their own versus doing it externally or having a tough time, which sort of flies in the face of like some of the conventional wisdom. I think the conventional wisdom was you wanted to be able to build internally, maybe with open source so you could customize to your use case. but it turns out some of the off-the-shelf stuff is actually working quite well. Yeah, I think you have to, you know, a lot of the challenge with either these types of surveys or even talking about architectures is you have to kind of separate the tech industry from the non-tech industry.

7:22The non-tech industry being the kind of consumers of these types of technologies and the tech industry being the builders. So open source is insanely valuable, but not in the sense where a law firm should go off and build their own AI project using an open source model. Like that is just a recipe for disaster if we think that every single company on the planet is going to go build their own technology to go automate their workflows. And that has been actually the case for a lot of pilots because we've been early in the technology and you haven't had applied solutions that you could go deploy. But open source is actually extremely valuable for a company like Box because we're powering technology for 120 ,000 customers.

8:03And so we actually do have the expertise internally to leverage those kinds of capabilities. And so I would say the conclusion from the dimension of open source, as an example, is just you probably shouldn't expect that every company on the planet is going to DIY their own AI strategy. And that's a recipe for not getting the returns and gains from an AI adoption standpoint. And then maybe the final thing I'd kind of point out is just there really is a decent amount of change management required to getting real gains from AI. This is not a panacea type of solution where you could take an existing workflow, drop AI directly into it, and then all of a sudden that workflow will be 3x better.

8:45You usually do have to re-engineer the work to take advantage of AI. And the conclusion I've recently come to more and more is, you know, I think we had this feeling maybe two or three years ago where AI was going to learn everything about how we work. It would be able to adapt to our workflows and then bring automation to our workflows. And I think realistically, increasingly, we probably will have to modify our work, hopefully incrementally, but in some cases meaningfully, to fully take advantage of AI. And that sounds maybe hard on one hand, but for the companies that do that, the ROI is going to be fairly massive.

9:20So if you think about AI coding as maybe the most obvious example right now where you're seeing productivity gains, the way that AI kind of first engineers tend to work is pretty different than how you engineered two or three years ago. The engineer really becomes more of a manager. You're deploying agents to go off and work on large parts of the code base, and then it's coming back with a bunch of work that you go and review. So if you don't change your workflow as an engineer to take advantage of background agents and how you give them the right kinds of prompts to actually execute on their task, and the new ways you should effectively think about your code base and handling the specifications and rules of what the AI agent should do, If you don't do all of that work, you're probably not going to get a 2x or 5x gain from AI.

10:08And so we will actually have to re-engineer some of our business processes to make agents effective as opposed to thinking agents will just drop into our processes and automate everything that we're doing. By the way, you've brought up pilots a couple of times. And I think it's important to talk about because this study was not just pilots. It was 95 % of organizations get zero return on AI investment. So I think the pilot thing is interesting because it's natural that pilots are going to fail. And in fact, we've had some listeners who've given me some feedback that said, because I talk often about how like only 20 % of AI pilots or 10 to 20 % of AI pilots get out the door into production.

10:45And that might be a good number because you're going to obviously have some trial and error in the early days. Yeah. And to be clear, I'm using pilots colloquially in the sense that we're just so early in the technology that we talk to customers. What a lot of times they have so far deployed is the equivalent of a pilot just because of literally how organization wide. Yes. Well, organization wide is is, you know, it's hard for one centralized survey taker to represent an organization wide. It is like, that's why, again, that's why I don't want to like, the survey is great. It's an interesting, you know, kind of conversation starter.

11:22But like, if you actually tried to go assess how is the answer, you know, answering this question and what is their way of measuring that productivity and have they actually surveyed all of the end users that are just using ChatGPT in an unsanctioned way and what they're doing. It's like, it's not possible to capture all of that. So it tends to more represent the kind of the centralized, you know, heavily sort of, you know, again, kind of, I think more likely pilot-oriented type projects because of just, again, how early we are. You know, the word agents just came onto the scene less than a year ago.

11:54So we're just early in a lot of these spaces. But again, I think it's a fantastic survey because it gets a conversation going. But I think if the takeaway was to slow down, you know, using AI or to do anything other than kind of realize what you should mitigate from a risk standpoint, I mean, then actually the failure would just be or the problem with that would just be all it's going to do is cause some companies to move even more slowly. And then you'll have other companies just outrun them. So it's kind of up to the, you know, it's sort of, you know, at the risk of, you know, the risk is now on the listener to decide what they want to do about that survey.

12:32Yeah. And I can tell you one more thing that I found super interesting about this study, which has sort of been underappreciated. So it says official LLM purchases cover only 40 % of firms, yet 90 % of employees use personal AI daily, at least those surveyed. Which just is so interesting because it means that, yeah, there's more personal use and more interest among individuals than companies to get this stuff into production. Yeah, you obviously have a reaction here, so let's hear it. Well, I know. I just think that's like empirical revealed preference. So like you don't have to – like you don't even have to survey once you know that.

13:08Why are people going off and using AI in a personal productivity sense at that rate? It's because they're getting value from it. So you almost like that is sort of now in the baseline of how people are working. It's unquestionable that if you just sort of eliminated AI just today, let's just say, you would just notice, wow, okay, I actually have to go and do that three hours of research that I used to be able to go and kick off as a deep research project and go and check back in on it after five minutes. And so it's empirical that we're choosing to use these technologies on a daily basis because they're adding that productivity.

13:45And I would argue that what we've seen with AI thus far is barely scratching the surface of what is going to start to happen as you start to deploy these technologies. But do you think the use in business, could it potentially be just individuals using, let's say, ChatGPT on their own versus scaled enterprise use of large language models? Or do you think it will be some blend? In the future? You're obviously watching in the future because you're obviously watching this happen on the other side of things. No, the future is, I think that we are in the earliest phases of just even the diffusion of the technology itself, of the basic use cases of, hey, when you're going to go research a customer, you know, why don't you get a full account plan, you know, instead of just saying, okay, this person works at this company and they're interested in these things and these are the trends in that industry.

14:40Why not ask an AI system to generate the full plan? That's super powerful, but also relatively basic if you think about how people work and the full scope of workflows that people do. One really interesting example of, again, how early we are, Claude this week announced a new capability that will generate files for you. and even though we're two and a half years, nearly three years into the ChatGPT moment, it's the first time where an AI system can, I believe, generate reliably a kind of high quality document in the form of a Word document or a PowerPoint presentation. So we're nearly three years in and it's the first time ever that you could generate something that you would sort of look at and say, oh, that looks like a good presentation.

15:27So we are only at the very, very beginning stages. Now imagine, it'll still take a couple of years, now imagine a technology like that begins to ripple through corporations. And in the future, before you go and present whatever product you're selling to a customer, instead of spending one or two hours of doing a bunch of research and making your PowerPoint file that's your presentation, you go to an AI agent, you say, I'm about to go sell to this customer, generate this presentation for me. You kick that off, and again, three minutes later, it's sort of done for you. this is going to just show up in all of our workflows every single day in almost everything that we're doing.

16:04So coders are getting the first lens into what the future looks like earliest because they're sort of wired to take advantage of these tools. And AI coding has been the kind of first breakout use case. But that same dynamic of you're going to go to an interface, you're going to talk to an agent, it's going to go and execute kind of multiple steps of work for you. that will start to emerge within all of knowledge work over the coming years. I actually am probably a pragmatist on this sense that it will not be like this instant overnight transformation of work. It will take years of change management.

16:37We just hosted our conference this week, as you noted, and it happens to be a crowd, obviously, by definition, that is sort of forward-leaning and kind of early adopters of technology. But that represents a small fraction of the total economy. It will take years before, again, all of the banks, all of the pharma companies, all of the law firms start to get wired up in this AI-first way. But, I mean, unequivocally, it's going to happen. And there's nothing that will kind of slow that train down. All right. Let's talk a little bit more about this using Claw to generate documents use case. I mean, I would imagine.

17:13So the example that you gave was using one of these to go in and sell into a client. Now, I would imagine most organizations, they have like their PowerPoint templates and the data baked in. So even if I were to go into Claude and like upload my pricing spreadsheet, my inventory spreadsheet, a document about positioning and say, make a PowerPoint based off of this, I'm sure it would do a good job. But how practical is it to then say this is going to be a way that people do their work versus something that might look like a party trick where you're going to use the other documents that you have already when you actually are going to go out into market?

17:49Oh, yeah. No, the way that this will actually show up, and I can't represent the exact date that this will happen, but Box, you'll just go to Box, and you'll say, here's my sales presentation template. Here's the new client information. please generate a PowerPoint presentation with that. And then you'll just do that with your existing data. This is not sort of some kind of one-off vibe coded document. You will use your existing assets as the source material for the next document that you'll generate. And you'll go and review its work, and that'll take you three minutes. But it will have saved you an hour or two hours of all of the time that it took to do the customer research and move around all the graphics and put the relevant information in place, that will just be done for you.

18:35And that will multiply that over a million people that do that per day in some sector of the economy, and you'll just see, that's how you'll get tens of millions of hours of productivity gained within the economy. And how are you feeling about the trustworthiness of these models? Because you've talked a couple times now about how you could use deep research to prepare you for something, or you could use these models to generate a PowerPoint and then spend a couple minutes checking them over. Are you at the point now where you think the outputs of these models are trustworthy enough that that's all it takes?

19:06I think as long as, and this is where I get very excited about now, obviously what's in the zeitgeist is context engineering. As long as you are really good about what context you're giving the AI and how you are effectively grounding the AI in trustworthy data with the right kinds of prompts and a high enough quality model, you can nearly eradicate all of, if not the vast majority of hallucinations or accuracy issues. So in our case, everything that we do at Box is we think about your existing data as the source material for the AI agent. So it's the source context for the AI agent to be effective.

19:46And so if I take an existing PowerPoint document that's our sales presentation, and I say modify this for a new customer, and you do that with a frontier model that is a reasoning model with some degree of kind of thinking mode, I would posit that 99 % of the time it's going to make infinitesimally small kind of errors or failures on that. That's just like a solved problem at this point. And it is still easily worth the kind of five-minute tradeoff for the couple hours you save to go and review its work. And we actually have this incredible front row seat in watching what the future looks like with coding.

20:26So if you talk to the brand new startups, and I don't know if you do this, but I know that you get to spend your time with the demnices of the world and whatnot. But go talk to a five-person startup that's brand new. And what's exciting is they're working in the craziest ways that I've ever seen in my entire life. I was talking to a nine-person startup the other day that estimates that they're, at a minimum, executing at the size of about a 100-person company. And that was, again, kind of conservative probably when you do the underlying math. And it's because each of their engineers has the capacity output now of 5 or 10 or 20 engineers worth of work.

21:07But they are working in a completely different way. They are managers of AI agents. They spend their time on writing really good specs for what they want to build. They spend a really good time on the design architecture of their software. And then they spend a lot of time on reviewing the output of the agent. So not every area of knowledge work will look exactly like that. But if you imagine in sales, if you imagine in marketing, if you imagine in legal work, and your role is to manage agents that are doing a lot of the underlying data preparation, research, creation type of work, and then your job is to go review that work and put it together in a broader business process, that will actually be what a lot of work looks like in the future.

21:52And this idea of hallucinations or errors will be no different than the fact that I have to sometimes review other people's work and other people review my work. And I have errors in the presentations that I create that somebody catches and they see a misspelling or they see that I changed the name of a customer in the wrong way and they change that. We will be doing that for AI agents. So it's this flip of the model where we thought AI agents were going to review our work and kind of incrementally make us more productive. We will be the reviewers of the AI agents' work. We will be the editors.

22:24We will be the managers. We'll be the orchestrators. And that's actually how you then get the productivity gains. So I'd say watch the AI coding space. Watch what startups are doing to get leverage. And then think about that against the broader economy. You know, it's really interesting, Aaron, because the last time we spoke, you told me about this person that you knew who was basically building a company on their own using AI coding tools. And so I was in the process of writing this profile of Dario at Anthropic, which you're quoted in. And I went out and found a developer doing something quite similar using Cloud Code to build on their own.

22:57So this is clearly, I mean, to the point where like Anthropic now has to put some rate limits on. But this is clearly a thing that's happening. And this is the thing that, again, I'm still, I love the MIT survey. I think it's great. It's a fun conversation topic. But the one travesty would be if people miss that what you just said is actually happening on the ground and then not starting to pay attention to what that's going to mean as that ripples through corporations and how people should probably start to think about reengineering workflows for a world of AI agents. And, you know, this happens in every single technology wave, which is actually why you have early adopters and early innovators and why you have laggards is because the early adopters and innovators are going to read, you know, your anthropic piece and see, oh, this actually is a real trend.

23:46And the laggards are going to read the MIT piece saying, oh, I've been vindicated. And some companies will then get those early returns at a much faster rate and other companies can wait. And, you know, sometimes that means that your company gets disrupted and sometimes it doesn't because you actually have, you know, some proprietary, you know, capability as an organization. Like if Pfizer or Eli Lilly took a little bit longer to adopt AI as a result of, you know, wanting to be more pragmatic, that'll be totally fine. They're not going to get disrupted. Like they have enough market position. They have enough distribution.

24:19They can afford to kind of wait for this technology to be more baked. But if I'm a startup right now, I'm probably going to use that as my advantage as much as possible to try and run circles around maybe a larger incumbent. And this is what kind of creates this nice tension in the market that creates creative destruction in every kind of wave of technological change. Okay. I definitely want to speak a little bit more about what the definition of an agent is and how you're rolling them out at box and also get your reaction about GPT-5. So let's do that right after this. Did you know your credit card points and miles can lose value to inflation?

24:54Credit card companies often reduce the redemption value of your points and miles. Now, imagine a credit card with rewards that can grow in value. With the Gemini credit card, you can earn Bitcoin or one of over 50 other cryptos instantly with no annual fee. Every swipe at the store or gas pump earns you instant rewards deposited straight to your account. Plus, sign up now for a$200 Bitcoin bonus to kickstart your rewards. Visit Gemini.com slash card today. Check out the link in the description for more information on rates. Again, if you're looking to invest in Bitcoin but don't know where to start, the Gemini credit card makes it easy.

25:33The Gemini credit card is issued by WebBank. In order to qualify for the$200 crypto intro bonus, you must spend$3 ,000 in your first 90 days. Some exclusions apply to instant rewards in which rewards are deposited when the transaction posts. This content is not investment advice and trading crypto involves risk. The Gemini credit card cannot be used to make gambling related purchases. Shape the future of enterprise AI with Agency. A-G-N-T-C-Y. Now an open source Linux Foundation project, Agency is leading the way in establishing trusted identity and access management for the Internet of Agents.