Monologue: Everyone Suddenly Cares About AI ROI

0:00This is an iHeart Podcast. Guaranteed human. Amazon Health AI presents Painful Thoughts. I, um, I can't stop scratching my downtown. Yeah, but I'm not itching to go downtown and tell a receptionist I'm here to talk about my downtown. Some things you'd rather type than say out loud. There's no question too embarrassing for Amazon Health AI. Chat your symptoms and get virtual care 24-7. Healthcare just got less painful.

0:57stay you. Visit vitalproteins.com to learn more and where to buy. These statements have not been evaluated by the Food and Drug Administration. This product is not intended to diagnose, treat, cure, or prevent any disease. When you own your own business, you own every decision. Catch the red eye or take the 6 a.m. Make a new hire or promote internally. Celebrate a win with the toast at the gate or unwind at the lounge. Big props to this team. Some decisions are a win-win. Like earning 8 times points on Chase Travel. Chase Sapphire Reserve for Business. The business card that gives back all you put in.

1:28Visit chase.com slash reserve business to learn more. Cards issued by JPMorgan Chase Bank. Any member of DIC. Subject to credit approval. Terms apply. Hey everyone, it's Cal Penn. I'm inviting you to join the best sounding book club you've ever heard with my podcast, Earsay, the Audible and iHeart Audiobook Club. Every episode, I nerd out with amazing guests and dive into the best new audiobooks available on Audible. It's the book club for your ears. Listen to Earsay, the Audible and iHeart Audiobook Club, on the iHeartRadio app or wherever you get your podcasts.

2:09Hi, Mimo Moed Zitron, and welcome to this week's Better Offline Monologue.

2:18Better Offline. It's a triumphant week here at the Zitron household. As I mentioned earlier in the week, Uber COO Andrew McDonald said, and I quote, that its AI costs were becoming harder to justify, and that the link was not there between spending money on AI tokens and creating more useful features. In the days that have followed, the AI industry has been embroiled in a conversation that mostly comes down to one thing. Hey, does anyone have any way to measure the ROI of AI? didn't fucking think of that one did you fellas i don't know i want to come up with that before i spent all the fucking money but i don't know i don't run a fucking business that's because up until recently most companies have been able to pay for a subscription with subsidized token rates meaning that for every dollar of their subscription they could burn anywhere from three to thirteen dollars worth of tokens they just had rate limits whatever they did just ate into those rate limits and they didn't really think about the token cost.

3:16In fact, every one of these companies deliberately obfuscates that information. You can find it through CC usage, which you need to the command line interface, and there's also slash usage with Anthropic, regardless. As a result of all of this, nobody was really measuring the actual positive outcomes of AI services or even how much a particular task would burn in tokens, which means that with the advent of token-based billing for enterprises, is everyone is just guessing about how much their annual budget should be based on virtually no useful information. Imagine you're doing something for years, years and years, and you're claiming it's the future and it's going to change everything.

3:52It's going to change your business and other businesses. It's the future of your revenue and their revenue. Oh my God, it's so amazing. But not once did your clever MBA ass say, why don't we make sure it actually is? no one did that not one of them not one of these fuckers i've been i've been checking all week talking to people all week in truth though measuring roi is pretty difficult for a lot of knowledge work because it requires you to actually know what a person does for a living and what doing a job is like and what the outcomes of said job are and what the actual like good results at the end may be which is a lot to ask for middle managers and ceos who mostly go to and from lunch and explain why other people's work is actually theirs and i want to say if you're a middle manager hearing this more than likely i'm completely correct about you most middle managers are fucking scum they exist only to steal work from people and nag them and harass them if you're not one of those great you shouldn't be offended by this this is not about you but if you are you're probably one of the people who's going to email me in which case we're we're got cried as someone else i don't give a shit but the lack of measurement of both task cost and roi as executives already crying for mercy.

5:03An engineer at a popular payment processor recently told me that their organization has, in the last week, burned over$1.5 million in Anthropic tokens, with one user, one person, burning over$100 ,000 in that time period. Over at Zillow, engineers have burned through 95 % of their annual cursor budget in less than five months, much like Uber's CTO revealed to Laura Bratton over at The Information that it burned its entire annual token budget by the middle of april axios's maddie mills reported that a consultant advising a cfo at an unnamed company said that an organization had spent 500 million dollars in the space of a month on anthropics tokens jesus fucking christ jesus christ jesus christ i'm sorry i know i'm ranting i guess that's what i do on these monologues but you spent half a billion dollars you fuck nuts have no idea why you did that did you none of you do none of these companies do they don't they don't have any idea why they're doing this executives only do this when they're very sick we gotta we gotta give them some cat milk or maybe take them to the vet to finally deal with the problem ourselves but the ai industry doesn't really have a response to all of this other than to say that somebody could theoretically make a way to measure ROI in the future based on some metrics that nobody can can explain I've read thousands of words now of random Twitter posts mostly AI generated of course by people that don't realize that it's obvious say that these huge posts and they always like yeah you know the next thing in AI it's going to be tokenomics measurement and extrapolation of ROI measurements from there they're just saying yeah at some point maybe we need to know why we're doing this not just the because we all love ai it's like the end of it sorry the beginning of death of stalin when they're all standing around being like he looks great everyone's just like yeah of course we all love ai of course we all know this and we we all know that ai is the most special and beautiful thing and it's changing my life and you're of course and my life as well yes it's so amazing.

7:08But, like, do we have a way of measuring if it's actually doing that? And no one does. Not a single one of them does. You see, the problem is they don't want to face that everybody should have measured ROI from the beginning, and only avoided doing so because subsidized subscriptions allowed everybody to ignore the problem right up until it was too late. Anthropics' supposed explosion of growth appears to have come entirely from experimental revenue, organizations that have no way of measuring costs or return on investment run by business idiots who don't know what real work is directly incentivizing workers to use more ai right up until their cfo says hey uh just saw we spent uh tens of millions of dollars on this um is it producing anything of value like can you can you just can one of you and what's insane is i would expect and never in my wildest dreams did i think that it would be everyone i thought that there'd be a couple of organizations who could be like they have a messy metric they've hammered out they've really they bone smashed it like clavicular but with metrics oh we're gonna make this look good and then we'll be able to say this number went up i thought they'd have a few i thought they'd have some sort of weird metric that they came up with they don't i've to be talking to this fuckers all week I've been asking them and no one does in fact at one company they were literally saying yeah the way we measure someone's AI use and whether it's good or not is how many pull requests they've done via AI and pull requests being someone's going to email and be mad about this calm down the thing that you do when you say hey here is the change I want to make to this code or this thing this is the plan do you agree it there they're measuring the percentage of those written by AI that is a positive metric that's what they're measuring i didn't think no one would have any idea though i truly didn't it's so funny it's so fun it's so fucking funny it's the funniest thing i'm right i've been right about this for years i've been saying for years that when organizations pay the actual cost of ai not subsidized subscriptions that allow you to burn effectively as much as you want they fall and start asking questions all of these people who've been talking about how amazing AI is, are now going, well, you know, we need to have very difficult decisions, we need to measure it somehow.

9:30And the value of AI has been entirely framed around unsustainable, unprofitable, impossible to measure tools that beguile imbeciles and those that don't want to think about reality. To be clear, OpenAI also appears to be moving most businesses towards token-based billing by giving them$1 ,000 in credits to use on codecs to soften the blow. fucking sam no nobody does it better nobody does it better than clammy sam altman clammy clammy sammy we love clammy one thousand dollars just for you honey you can use it whatever you want please pay me and that's the thing open ai broken business as well but there's no real solution here there's none and i think the next three months are going to be illuminating and probably feature pullback across a ton of organizations as they ask questions they should have been asking since fucking 2023 it's insane i was mad i would i as i sound completely insane myself as i sound completely crazed but people were calling me crazy in 2024 being like yeah you know when businesses pay the real cost they're not gonna people were acting like i was a fucking lunatic well who's crazy now would a crazy person laugh like this anyway there are no real solutions here folks there are no real solutions while somebody theoretically could move their workloads to DeepSeek or a cheaper model, it's unlikely that there are inference providers that can support that much traffic or even provide that much stability.

10:56You have to remember a big organization is putting a lot more pressure on DeepSeek than, say, a casual coder, someone who's just running their own GPU. And you have to do far more than just turn on GPUs. You actually have to build, for an organization that would be theoretically spending millions, a massive inference stack that cannot crap out. It's one of the reasons people pay Anthropic and open AI, even though Anthropic is barely stable. I also don't have any substantive proof that DeepSeek, V4, or other models can replace an Opus, I was about to say Opeth there, not editing that, or Codex model, even if they are dramatically cheaper.

11:34We have no proof of this, and despite the conversation, no one's doing it. Not heard it, not heard anyone say this. Also, in the end, if nobody can measure the ROI of AI in general, why would they spend anything? now ai boosters are currently either completely ignoring this subject which is what's happening with the obvious ones or they're going to answer by saying that we're in the early days and that there are weekly breakthroughs that will solve these problems been a lot of fucking weeks so far been a lot of them well like what like 400 of them 450 coming up in 500 weeks but this is a problem that needed to be solved yesterday and neither anthropic nor open ai has an answer for bringing down costs, let alone those of their customers.

12:19Even if they were magically profitable overnight, even if they were, they're not. They're not profitable in inference. Their customers are spending too much money. Their customers do not have cost control. That makes it very hard to budget for how much you'll spend. Basic business stuff. Basic stuff. I'm not innovative in my thinking here. You know, it's like that tweet that was like, yeah, walking around a college campus looking into a business school class and it's like profit is revenue minus costs it really is that except these people have been ignoring that for years and making people feel stupid for asking those questions i also must be clear that these companies cannot slow down anthropic and open ai project to have over 350 billion dollars in combined annual revenue by 2030 and they're gonna need it because between them they've made over 1.1 trillion dollars in compute commitments.

13:14If their revenue growth is entirely based on dimwitted executives allowing a no IT loads refused compute dump for no apparent return on investment, that is not a stable, sustainable, or even a growth business. It's a grift targeting a society-wide executive ignorance of production itself. I look forward to telling you more about it next week. I've had a lot of fun this week. We've got Paul Kudrosky coming back on the show on Tuesday, actually Wednesday, sorry, 12 a.m. ET Wednesday. I should really know that by now. It's been years. And then, yeah, I'll have another monologue for you. It's been a lot of fun.

13:46I hope you're enjoying it. Zitron out.

13:54If your best finance people are doing expense reports, chasing receipts, or spending time on month-end close, it's time to get Brex AF, a gentic finance that eliminates that work before it starts. Learn more at brex.com slash AF. Aging doesn't stop, and neither should you. with Vital Proteins Collagen and Protein Shakes. Because around the age of 30, your body needs more support for movement and recovery. On workout and rest days, reach for a 30-gram total protein shake or go with our classic collagen peptides. Help support healthy hair, skin, nails, bones, and joints so you can stay vital, stay you.

14:31Visit vitalproteins.com to learn more and where to buy. These statements have not been evaluated by the Food and Drug Administration. This product is not intended to diagnose, treat, cure, or prevent any disease. Go on. Get a little out there into the big heart of Nevada, where you can go off-road and off the map on two lakes or on horseback. Dip into hot springs and dive into deserts. Climb a mountain or make your best effort. See thousands of stars in some of the darkest skies. Stake out haunted hotels. Can you make it to sunrise? There's always something new to see because we've got plenty of space to just be.

15:07Plan your trip at TravelNevada.com. Ryan Reynolds here from Mint Mobile, with a message for everyone paying big wireless way too much. Please, for the love of everything good in this world, stop. With Mint, you can get premium wireless for just$15 a month. Of course, if you enjoy overpaying, no judgments, but that's weird. Okay, one judgment. Anyway, give it a try at mintmobile.com slash switch. Upfront payment of$45 for three-month plan, equivalent to$15 per month required. Intro rate first three months only, then full price plan options available. Taxes and fees extra. See full terms at MintMobile.com.

15:40This is an iHeart Podcast. Guaranteed human.