AI is a money pit — here’s why investors don’t mind

0:01Hello, and welcome to Decoder. This is Alex Heath, not Neelai Patel. I'm Deputy Editor at The Verge, and I'm guest hosting a couple of episodes of Decoder this month as we head into the holidays. and I'm very excited for what we have coming. I'll be back later this month with an exclusive interview with the CEO of Chipmaker Arm, so stay tuned for that. And if you want a sneak peek, be sure to check out Command Line, my weekly newsletter about the tech industry's inside conversation. But today we're talking about a topic I've been focusing on a lot lately. It's at the heart of the most ambitious and most costly gamble the tech industry has arguably ever made.

0:37That is, of course, artificial intelligence, and specifically the idea that one day soon, big companies across all kinds of industries will be spending hundreds of billions of dollars on AI products. Today, though, that's just not happening. Meanwhile, AI companies have taken in tens of billions of investment this year alone. OpenAI raised a staggering$6.7 billion, surpassing XAI's$6 billion fundraise five months earlier. And Anthropic just raised another$4 billion from Amazon. The list goes on. We've heard endless hype from CEOs on Decoder about what this technology is supposed to do and why this investment is justified.

1:14But so far, the actual spending on AI products nowhere near matches the level of investment being made into the models themselves. According to a recent report from VC firm Menlo Ventures, AI spending is growing fast but hit only$13.6 billion in 2024. That barely covers the year's two largest AI fundraising rounds from OpenAI and XAI. So how is this money actually being spent on AI? What are companies actually buying and what are they doing with it? And why do investors think the return on investment here will be worth it? To find out, I caught up with two AI investors, Tim Tully, a VC at Menlo Ventures, who co-authored that report on AI spending, and Nathan Banesh, author of the State of AI Report and founder of Air Street Capital.

1:58We dove into the data, the big trends they're seeing with AI in the enterprise, and where we think all this is going next, including when these AI companies are going to start generating the kinds of profits they'll need to justify the money they're spending already. But first, to put everything in perspective, I wanted to start with Tim, the co-author of Menlo's major AI enterprise report, and what he thinks the data tells us about the state of the industry today. Tim Tolley, welcome to Decoder. Hey, Alex, how's it going? Thanks for having me on. Your firm, Menlo Ventures, puts out this report every year about the state of AI in the enterprise.

2:33And it just came out up top. I want to disclose that your firm is a pretty large investor in Anthropic, which is a very key player in the model race and where spending is going in the enterprise. But it's a really interesting report. You talk to a lot of people to gather a lot of unique data. I want to get into all that. But first, can you just give us a little bit of background about yourself and Menlo as a firm and what you guys do, especially as it relates to AI? Yeah, Menlo is a 47-year-old firm in Silicon Valley. We've been investing throughout enterprise and consumer for a number of decades now.

3:10We've done investments in companies like Uber, Carta, going all the way back to the old school, Hotmail, things like that. More modern companies are like Benchling, Chime, obviously Anthropic. And then we're in some pretty interesting AI companies like Pinecone, Neon, and Unstructured. The big headline out of your all's report about the state of spending in AI this year is that spending surged 500%. So it's hit about$13.8 billion, up from just$2.3 billion in 2023. When you drill down past that headline number, what are you guys seeing? What is changing, especially as it relates to last year? Last year in 2023, a lot of the dollars were going into simply infrastructure, sort of picks and shovels, you know, places where VCs like myself like to invest early on when they feel like there's a big, big inflection point in what's happening in technology, like the mobile phone.

4:04And we see it being similar to that. You see a lot of these infrastructure platform level investments. Now it's turned to the application side. Like you saw in the report, there's a lot more application spend this year. It increased pretty significantly. And it went to places that would really surprise you, like legal, fintech, healthcare. I mean, that was really, really surprising to us. I mean, we heard it anecdotally, but to see it in the data was pretty grounding for us. And when you say applications, you mean like hyper-verticalized AI applications for certain sectors? We talk about it as departmental spend.

4:40There's departmental spend where IT departments, R &D departments, go-to-market departments are buying. And then there's more verticalized spend in healthcare, legal, financial services. So people are buying legal software that's heavy on AI. People are buying clinical documentation software for healthcare that's heavy in AI. And then accounting software that's heavy in AI. Who are the biggest spenders and what buckets of spend are growing the most? I think you guys called out coding. That's something I just hear and see anecdotally. Coding assistants are growing pretty rapidly. but I guess aside from coding, what did you guys see in your research?

5:18Yeah, from a use case perspective, it was obviously coding. I mean, this is probably making me 70 to 80 % more productive as an engineer because I'm not really just an investor. I write code every day and the productivity for me is off the charts from the code generation standpoint. So it's very helpful. Second after that would be what we call support chat bot and then enterprise search and retrieval after that. So there'd be companies like Glean, for example, where they're indexing documents in the various departments throughout the enterprise. So obviously, that 500 % surge is a big surge. At the same time, when I see$13.8 billion spend in 2024, that's big.

6:00But it's really when you just combine OpenAI and XAI's funding rounds this year together, it's that number. In terms of the actual money being plowed into these companies, especially at the model layer, we're still not really seeing anywhere near the level of ROI as what's going in. And I'm curious as an investor and as an engineer, someone who's been in the AI space for a while, how do you look at that? Well, I think it's just the tip of the iceberg in terms of what that revenue is going to look like. You hit the number correctly, it was$6.5 billion for foundation models. But I think myself and others, including who invest in the space, strongly believe that these are the APIs of the future upon which the next generation of enterprise software is going to be built.

6:43So that 6.5 number, what if it multiplies by 5 or 6x again next year and becomes$36,$40 billion, right? Then you're starting to approach valuations that you mentioned in your point. So I think it's still early days. And when you see growth like this, you get excited and you want to invest. And this is the result. And you guys don't see a ceiling in the near term in terms of appetite for budgets to shift more towards these AI applications as they get better and better. You think that we will see another, you know, maybe not 500 percent, but triple digit percentage jump potentially next year? I think so, because again, the architectures are still settling, right?

7:24How you build the software is still evolving and it's going to get better and better. And I think as that software gets better and we learn how to interact with foundation models in even better ways, you're going to see better results in the software and you'll see more software sales. And on the foundational model layer, you have an interesting point here, which is that OpenAI, You say seeded market share in enterprise AI usage declining from 50 % to 34%, while Anthropic, a company that you guys backed, doubled from 12 to 24. Why is that happening? We didn't ask questions like, why did you switch necessarily?

7:59But what I hear anecdotally is that people are using Anthropic heavily for cogeneration. It's that use case that we talked about earlier. You talk to folks, and I talk to the same folks that you probably do, and you ask what model you're using and why. and you hear that there are certain use cases that people prefer Cloud for, right? Code generation, some content generation as well. Structured data it tends to do really well with. So I think people just are starting to prefer the output that they get from Cloud and they see it as a better model for certain sets of use cases. And that's why you saw it double this year.

8:32Are the sectors of startups that are already doing really well, making a lot of money kind of the same as where big enterprises are spending? Does it map directly? So like coding or are the coding startups that are supplying that tech, the ones really benefiting right now at the earliest stages? I think everyone's benefiting, frankly, I mean, across the board. I mean, you're seeing every department, every vertical sort of thriving, to be honest, on what they're doing with the AI. It's pretty breathtaking, to be honest. So it's not just, you know, obviously the coding guys are doing quite well.

9:07It's also transcends into other departments also, right? You see legal tech companies doing quite well. What specific vertical is growing faster right now? Is there a category that you're seeing really starting to uptick? Definitely legal tech is growing very, very fast. Customer support as well. And really what that is, is the early movers who deal primarily in text, right? Because the foundation models do text-to-text translations effectively is what they're doing, right? And so what you saw is legal, fintech, digital marketing. These are the ones that were the early adopters of these things.

9:42And so because of that, obviously, their revenue profiles are going to be a little bit higher because they were at the gates earlier. So it's really those. But I think growing very, very quickly is obviously anyone who's doing code generation as well. I think photo and video generation has been largely held back from being at scale by compute bottlenecks. And we've seen that with OpenAI and Sora. and the fact that they still haven't released it. And I'm curious how much you think about the multimodal aspect of AI and how much that will affect the enterprise when that gets scaled out. And we think about it a lot.

10:18I mean, we have some investments in companies that are doing, you know, diffusion-type models as well, not just in the video and image side, but also on the biotech side as well. So, you know, we're definitely investing, but I think you hit the nail on the head, right? The compute cost on those is a bit higher. And so it's a little bit more challenging to have some of the success that maybe the text-to-text folks have had. We need to take a quick break. We'll be right back.

10:54Support for this show comes from LinkedIn. Imagine if any of the movies that included the line, I need the right person for the job, settled for, I'll just take about anyone. How many heists would have failed? How many deals would have fallen through? How many secret spy missions would have ended in disaster? So why would you accept just anyone when hiring for your business? When you need the right person for the job, you can turn to LinkedIn Jobs. And now LinkedIn Jobs is stepping things up with their new AI assistant so you can feel confident you're finding top talent that you can't find anywhere else.

11:28With LinkedIn Jobs AI Assistant, you can skip the confusing steps and recruiting jargon. It filters through applicants based on criteria you've set for your role and surfaces only the best matches so you're not stuck sorting through a mountain of resumes. Hire right the first time. Post your job for free at linkedin.com slash partner. Then promote it to use LinkedIn Jobs new AI Assistant, making it easier and faster to find top candidates. That's linkedin.com slash partner to post your job for free. Terms and conditions apply.

12:10We're back discussing AI spending in the enterprise and how it's supposed to turn into big profits in the future. Before the break, you heard Tim discussing Menlo Ventures' latest report and how one of the big findings was that companies are still mostly spending on access to foundational models like OpenAI's GPT series and Anthropics Cloud. Spending on these kinds of models jumped from$1 billion last year to$6.5 billion this year, accounting for close to half of all AI enterprise spending. And that spending is growing more diverse, with Claude eating into OpenAI's market share because of its capacity for code generation.

12:46But most of these AI model companies are not yet profitable and far from it. OpenAI is on track to lose roughly$5 billion this year, in fact. So when I caught up with Nathan Benesh from AirStreet Capital, I wanted to know what it might take for these AI firms to flip from burning billions a year to being profitable. Nathan authors a massive deep dive into the industry every year called the State of AI Report. And one of his key findings was that while AI revenues are rising across the board, there are some serious questions about the long-term sustainability of what's happening. So in your report, you write, many of these companies currently have no identifiable path to profitability.

13:25However, this isn't true for everyone as the biggest model providers see revenue begin to ramp up. So who are the haves and have nots right now? Well, there's certainly the big model companies which are not profitable and by far, those are kind of like the have nots with regards to profit margins. And then the haves would be companies that probably provide much cheaper outputs, like whether that's images, Midjourney being famously quite profitable business. Audio generation is much cheaper than it is to run massive models because the models are smaller and the data is easier to work with. These companies are pretty profitable.

14:02In some ways, you're just in the early days of this technology. So you're just trading off profit margins and profitability for growth. And I think that makes sense for so long as the market's willing to provide you those tools. I will say though that a year or two years ago, the path to profitability was a lot fuzzier than what it is today, in part because there weren't that many optimizations that came into how the models were built, how they're run, and all the data infrastructure and cloud infrastructure around it. And to some degree, it's because the people contributing to progress in AI were pretty much AI research people, not people who worked for decades on cloud infrastructure and optimizing.

14:41Like this software runs really fast and is cheap on all sorts of devices. And all those people have started to move en masse into AI. And so I think you can bet just by the sheer capability of individuals who have done optimizations in other software settings and other technical settings moving into AI will actually bring a lot of cost improvement, which will eventually help margins. So that's one of the reasons why I think you see an order or two orders of magnitude reduction in cost per million tokens, which is the unit economics of these models from one or two years ago to models that exist today for systems that are of similar intelligence, if you will.

15:15Do you think there's going to be a moment where OpenAI, which famously loses billions of dollars a year, Anthropic, etc., these big leading players, the model providers, become very profitable? And have you thought about what that scenario would be, how they get there? This is tough. I think you really need to see hundreds of billions of dollars of revenue, I think, for that to be true. in aggregate or each of those companies making hundreds of billions? Probably say each of those companies, if you assume that the race for bigger and bigger models continues. Like if we stop now, I think we could have fairly profitable companies.

15:56And if there were more price discrimination with the services, these companies sell one license to somebody in a hedge fund for the same price that they sell a license to a student doing their master's degree. So that doesn't really make a ton of sense. So if there's better price discrimination, maybe some more vertical products, and then we stop on this race of making ginormous systems that require a huge compute, then I think there's a path. If we don't, then you're going to need like hundreds of billions based on the compute spend. Do you think it's likely that they get to hundreds of billions?

16:32That seems like a tall order. It really just would require huge parts of actual labor getting translated into model jobs. There's early signs that you see like Klarna say they've let go of hundreds of customer support roles because of AI. There's signs that's starting to happen. But yeah, when Klarna cuts hundreds of jobs, that doesn't make open AI more profitable. There's no direct correlation. Maybe it's indirect, but you're suggesting that that Klarna example needs to happen a lot for the margin profiles of these model providers to really change. Yeah. And in that use case too, it's heavily competed.

17:15If it's not OpenAI, it might be some other company that is providing customer support. Dare I ask Nathan, are we in a bubble? i think there i think i think there isn't a wall you're you're of the so we're talking about this is the scaling wall conversation that's happening yeah uh where everyone you know having questions about can we keep just plowing more money into these new models and continue to get better results and you are of the there is no wall camp yeah i think we'll keep going for a while i don't know if it's five years, but I think for sure, like next year and probably the year after that.

17:52So even if the revenue is not growing at the same pace, you see companies continuing to up their investments in new models and chip orders and the race is going kind of unabated. Yeah. I think the revenue key is still growing. Like OpenAI is still really growing. Anthropic, according to reports, is like eating into enterprise quite well. I mean, Nvidia had another blowout quarter. I mean, the fact that it does or doesn't meet Wall Street estimates is kind of like neither here nor there, I think, because Wall Street has a tough time estimating that company anyway. And then Big Cloud is still printing money on these services.

18:31So I think it's still real for sure. And we're going to see way more of this like sovereign investment in AI, Like every nation state is increasingly getting convinced that they need to be investing in this. And, you know, I think big finance hasn't really come to play very much yet with regards to financing new data centers and really providing the capital for this kind of scaling. So I think it's still pretty early. And the CEOs driving this, do you think they see a pot of gold at the end of the rainbow that we all can't? Like they see, oh, you know, I get to this point and the profit is going to be just so immense that we'll have all been worth it.

19:11Or is this more FOMO driven and hype driven and kind of wisdom of the crowd driven? I think they're more motivated by pushing the boundaries of technology capability and product experiences than they are seeking gap profitability. Yeah. It's just like the nature of what drives these kinds of individuals. They want to be there when the AGI happens, whatever that means. Like technology history teaches you that you can sustain lack of profits for a very, very long time as long as you give investors in the market other reasons to believe why they should suspend that disbelief for another year, another year, another year.

19:51And for the moment, I think that's still the case. I asked him about this too, especially considering Menlo Ventures invested in at least two separate financing rounds for Anthropic over the last several years. He sees the ROI on AI arriving alongside more superior chips and models, even if it's difficult to see exactly how it will all play out. When you think about the return on investment that a firm like yours needs to have, where do you think it's going to come from? Do you think it's going to come from the models or from the applications that are built on top of the models or something else?

20:24Well, I think it's going to come from both for sure. I mean, both the foundation model layer and the application layers, not for every single company, obviously, but for, for certain ones are doing quite well. Right. And, and they're, they're selling a lot of software and seeing a lot of usage and obviously a lot of API usage and they're, they're generating real revenue. I mean, this is, This is not a sort of blip in time or a blip on the radar where we're going to get some false positive results. This is real. And you can see it in the board meetings that I attend or my partners attend and the results of their companies are returning.

21:03So this is very real. And I think you'll see it from both layers. And everyone broadly knows that companies like OpenAI and Anthropic are, yes, they're making money. They're growing their revenues. But they're also burning way more than they're bringing in. And maybe that will invert at some point. It's really hard to put a pin on when that's going to happen. I feel extremely bullish on the idea that the cost to train models over time is going to go down. There'll be better silicon. There'll be new ways to train models. There'll be optimizations that I'm probably not even aware of that are being cooked up right now and being produced in literature that will come out soon.

21:42So I'm confident that that will find a path to have cheaper inference and training over time. On the application layer, I think it's already there. Most of these folks are just calling in using API calls into the foundation models anyways. And so the cogs and opics for them to produce these software solutions is pretty small. We need to take another quick break. We'll be right back.

22:19We're back talking about AI spending and what it means for companies to start pouring money into these products to transform how they do work. Before the break, we talked about what it would look like for all these companies to start showing the kind of growth and profitability to justify these massive AI investments. But how exactly does the AI industry go from a collective total of about$15 billion in revenue to each major AI company doing tens or even hundreds of billions in revenue a year. Investors like Tim and Nathan see a major opportunity in AI agents. Right now, companies are mostly paying for access to LLMs attached to a text box.

22:55You type in a question or a command, maybe you feed it some data or code to work with, and it spits something out in return. But what if these LLMs could go off on their own and complete tasks for you. That's the dream being pursued by Anthropic, OpenAI, and scores of their competitors. So far, though, only Anthropic has publicly put out something trending in that direction, with the launch in October of a beta version of its cloud model that can move a digital mouse cursor, click buttons, and type text all on its own. It's too early to tell right now, but companies might pay a whole lot more for these kinds of agents than they do for the AI models of today.

23:30The software is just getting quite good. And the move to agentic workloads is getting quite strong. And the capabilities of what an agent can do, again, I'm not going to say that it replaces what a human being can do. And frankly, people should not be writing agentic workloads and architecting them in the way that a human does. It turns out that that doesn't really work well, is my understanding. but when you do deploy these things they're quite productive and it certainly accelerates workload like i said yeah you all are very bullish on agents in the report and i feel like it's the topic du jour it's the thing i hear constantly agents agents agents i don't think people have seen a lot of them practically in the world yet but anthropic is an early mover here through their api you can already do some kind of basic agent-like stuff on your desktop what are agents going to do, especially as it relates to the enterprise?

24:25I feel like we'll see some big consumer swings from Google and OpenAI in the coming months. But when they get deployed into companies, what happens? Well, I think it's important to understand what an agent really is, not to sound overly reductive or anything, but what it really is an orchestrated series of steps to execute. We talk about it a lot. Really, what that is, is an architectural shift, more than some kind of revolution, I would say, necessarily, that's going to change the way software is written or replace humans. Yeah, we talk about it a lot, but that's because we spend a lot of time meeting new companies that are pitching to us and even our existing companies and disarticulating their architectures and really understanding how they're built.

25:07What we're seeing is that, I talked about in the report a little bit, some of the RAG architectures that were pretty extensive in 2023 have been replaced by agentic style architectures. And it's in the report as well. You can actually see how the primary architectural approach and application construction has changed in 2024. A lot of it went away from prompt engineering and almost, I think, more than half, I want to say, and went into agents. So really what it is, is a mindset shift in terms of how people are building software more than anything else. Does it mean that a company's backend infrastructure becomes increasingly automated, that fewer people are needed to maintain it?