How To Argue With An AI Booster, Part Two

0:00This is an iHeart Podcast.

0:30Thomson Reuters and specialized bikes have since they upgraded to the next generation of the cloud. Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high availability, consistently high performance environment, and spend less than you would with other clouds. How is it faster? OCI's block storage gives you more operations per second. Cheaper? Better? OCI costs up to 50 % less for computing, 70 % less for storage, and 80 % less for networking. Better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds.

1:12This is the cloud built for AI and all your biggest workloads. Right now, with zero commitment, try OCI for free. Head to oracle.com slash strategic. That's oracle.com slash strategic. So I love Square. They make paying for stuff at a store really easy. I just use them at Court Street Grocers to buy a sandwich here in Brooklyn, New York. Their stuff's reliable, easy to use, and it just works for both businesses and their customers. Square keeps up so you don't have to slow down. Get everything you need to run and grow your business without any long-term commitments. And why wait? Right now, you can get up to$200 off Square hardware at square.com slash go slash better offline.

1:50That's S-Q-U-A-R-E dot com slash G-O slash better offline. Run your business smarter with Square. Get started today. Mint is still$15 a month for premium wireless. And if you haven't made the switch yet, here are 15 reasons why you should. One, it's$15 a month. Two, seriously, it's$15 a month. Three, no big contracts. Four, I use it. Five, my mom uses it. Are you playing me off? That's what's happening, right? Okay. Give it a try at mintmobile.com slash switch. Upfront payment of$45 for three month plan, $15 per month equivalent required. New customer offer first three months only, then full price plan options available.

2:27Taxes and fees extra. See MintMobile.com Call Zone Media Hello and welcome to Better Offline. I'm your host, Ed Zitron.

2:45Better Offline This is part two of our three-part series on how to argue with an AI booster. When we last left off, I'd started talking about some of the most common and vacuous talking points used by those who defend the generative AI industry, and why a lot of them are wholly without merit. These are the booster quips, assertions that, if you don't know much, sound convincing, but are easily disproven with the right information. And in that last episode, we addressed the quips that say were in the early days of AI, and that people doubted smartphones and the internet, things they didn't do, just like they did generative AI, which they should do.

3:20in the cycle of grief that's the denial stage now we're going to move on to bargaining this is just like the dot-com boom even if all of this collapses the overcapacity will be practical for the market like the fiber boom was all right folks time for a little history you know me i love me some history the fiber boom began after the telecommunications act of 1996 deregulated large parts of america's communications infrastructure creating a massive boom a 500 billion dollar one, to be precise, primarily funded with debt. Obviously, we're still using the infrastructure bought during that boom, and this fact is used as a defense of the insane capex spending surrounding generative AI.

4:02High-speed internet is useful, right? Sure, but the fiber-optic boom period was also defined by a gluttony of overinvestment, ridiculous valuations, and genuine outright fraud. In any case, this is not remotely the same thing, and anyone making this point needs to learn the very fucking basics of technology, Let's get going. Now, the fiber optic cable of this era is mostly owned by a few companies. 42 % of NVIDIA's revenue is from the Magnificent 7, and the companies buying these GPUs are, for the most part, not going to go bust once the AI bubble bursts. You can also already get the cheap fiber of this era, too.

4:37Cheap AI GPUs are already here. GPUs are depreciating assets, meaning that the good deals are already happening. I found an NVIDIA A100 for$2 ,000 or$3 ,000 multiple times on eBay, and you can get the H100s, which are more powerful for, what, I think$30 ,000, and those things go$45 ,000 retail, so not brilliant. AI GPUs also do not have a variety of use cases and are limited by CUDA, NVIDIA's programming libraries and APIs. AI GPUs are integrated into applications using this language, CUDA, and this is specifically NVIDIA's programming language. While there are other use cases, scientific simulations, image and video processing, data science and analytics, medical imaging, and so on, CUDA is not a one-size-fits-all digital panacea.

5:21While fiber-optic cable was, it was also put everywhere. It truly did set up the future. What are these GPUs setting up exactly? Also, widespread access to cheaper GPUs has already happened. And what new use cases are there? What are the new innovative things we can do? As a result of the AI bubble, there are now many, many, many, many, many different vendors to get access to GPUs. You can pay at an hourly rate. Who knows if it's profitable, but you can do it. And sometimes you can get them for as little as$1 an hour, which is really not good. It definitely isn't making them money. But putting the financial collapse aside, while they might be cheaper when the AI bubble bursts, does cheaper actually enable people to do new stuff?

6:04Is cost the problem? Because I think the costs are going to go up. but even if they weren't going up, what are the things that you could do that are new? What is the prohibitive cost? No one can actually answer this question because the answer isn't fun. GPUs are built to shove massive amounts of compute into one specific function again and again and again, like generating the output of a model, which, remember, mostly boils down to complex maths. Unlike CPUs, a GPU can't easily change tasks or handle many little distinct operations, meaning that these things aren't going to be adopted for another mass-market use case because there probably isn't one.

6:38In simpler terms, this was not an infrastructure build-out. The GPU boom is a heavily centralized, capital-expanditure-funded asset bubble, where a bunch of chips will sit in warehouses or kind of fallow data centers, waiting for somebody to make up a use case for them. And if an endearing one existed, we'd already have it, because we already have all the fucking GPUs. Now, here's a really big boost equip, and I have been looking forward to. I get a lot of people asking you about this. Um, Ed, Ed, you're so stupid. Why am I stupid exactly? Well, five really smart guys got together and wrote AI 2027, which is a very real sounding extrapolation that shut the fuck up.

7:18Shut up. Shut up. AI 2027 is fan fiction. If you were scared by this and you're not a booster, you shouldn't feel bad, by the way. This was written to scare you. And by the way, if you don't know what it is I'm talking about, you should consider yourself lucky. It's essentially a piece of speculative fiction that describes where Gen AI companies get fatter models that get exponentially better, and the US and China are embroiled in an AI arms race. It's really silly. It's so very silly. And I call it fan fiction because it is. If we're thinking about this in purely intellectual terms, it's up there with My Immortal.

7:51And no, I'm not explaining that. You can Google that one for yourselves. It doesn't matter if all the people writing the fan fiction are scientists, or that they have the right credentials. They themselves say that AI 2027 is a guess, an extrapolation, which means guess with expert feedback, which means someone editing your fanfiction, and involves experience at OpenAI. There are people that worked on the shows they write fanfiction about. I'm not even insulting fanfiction, by the way. Go nuts. You are a hundred times more ethically positive than these people. At least you admit it's fanfiction.

8:23Could Knuckles get pregnant? I'm sure somebody's found out. I'm not going to go line by line and cut this apart any more than I'm going to go and do a lengthy takedown of someone's erotic Banjo-Kazooie story, because both are fictional. The entire premise of this nonsense is that at one point someone invents a self-learning agent that teaches itself stuff, and it does a bunch of other stuff requiring a bazillion compute points with different agents with different numbers after them. There is no proof that this is possible, nobody has done it, and nobody will do it. AI 2027 was written specifically to fool people that want to be fooled, with big charts and the right technical terms used to lull the credulous into a wet dream in a New York Times column where one of the writers folds their hands and looks worried.

9:04It was also written to scare people that are already scared. It makes big, scary proclamations, with tons of links to stuff that looks really legitimate, but when you piece it all together is literally just fan fiction. Except really not that endearing. My personal favorite part is mid-2026, China wakes up, which involves China's intelligence agencies trying to steal Open Brain's agent. No idea who this company could be referring to. Please email me if you can work it out to idontcareatbusiness.org. Before the headline of AI takes some jobs after Open Brain releases a model. Oh God, I'm so bored even fucking talking about this.

9:40Now, Sarah Lyons puts this well, arguing that AI 2027 and AI in general is no different from the spurious spectral evidence used to accuse someone of being a witch during the Salem witch trials. And I quote, and the evidence is spectral. What is the real evidence in AI 2027 beyond trust us and vibes? People who wrote it cite themselves in the piece. Do not demand I take this seriously. This is so clearly a marketing device to scare people into buying your product before this imaginary window closes. Don't call me stupid for not falling for your spectral evidence. My whole life, people have been saying artificial intelligence is around the corner and it never arrives.

10:17I simply do not believe a chatbot will ever be more than a chatbot, and until you show me it doing that, I will not believe it. Anyway, AI 2027 is fan fiction, nothing more, and just because it's full of fancy words and has five different grifters on its byline doesn't mean a goddamn thing.

10:40So I've shopped with Quince before they were an advertiser and after they became one, and then again before I had to record this ad, I really like them. My green over shirt in particular looks great. I use it like a jacket. It's breathable and comfortable and hangs on my body nicely. I get a lot of compliments. I liked it so much I got it in all the different colors, along with one of their corduroy ones, which I think I pull off, and really that's the only person that matters. I also really love their linen shirts too. They're comfortable, they're breathable, and they look nice. Get a lot of compliments there too.

11:07I have a few of them, love their rust-colored ones as well. And in general, I really like Quince. The shirts fit nicely and the rest of their clothes do too. They ship quickly, they look good, they're high quality, and they partner directly with ethical factories and skip the middlemen. So you get top-tier fabrics and craftsmanship at half the price of similar brands. And I'm probably going to buy more from them very, very soon. Keep it classic and cool this fall. With long-lasting staples from Quince. Go to quince.com slash better for free shipping on your order and 365-day returns. That's Q-U-I-N-C-E dot com slash better.

11:41Free shipping and 365 day returns. Quince dot com slash better. Parking shouldn't slow you down. ParkWiz gives every driver a shortcut. Book ahead, save up to 50 % and skip the hassle of circling the block. Park smarter, park faster. ParkWiz. Download the ParkWiz app today and save every time you park. So today's show is brought to you with support from Square, who I really love. They offer quick payment processing, really easy to use terminals. Everything spins up really quickly. Every store I use them in, I don't really think about the experience, which is exactly what I want from any kind of tech.

12:17The loyalty programs work really easily. They're over at Court Street Grocers, S &P, Van Leeuwen Ice Cream. They're just a very easy to use and straightforward piece of tech. I can't recommend them enough. I've heard great things from businesses using them. And as a customer, never upset to see them in a store. Square keeps up so you don't have to slow down. Get everything you need to run and grow your business without any long-term commitments. And why wait? Right now, you can get up to$200 off Square hardware at square.com slash go slash better offline. That's S-Q-U-A-R-E dot com slash G-O slash better offline.

12:56Run your business smarter with Square. Get started today. Hey, it's Jonas Knox, and Toyota is moving toward a more sustainable future by giving you the freedom to choose from a full lineup of hybrids, plug-in hybrids, regular gas, and 100 % electric vehicles, including stylish cars like the Camry and Corolla, as well as efficient and spacious SUVs like the RAV4 and Grand Highlander, all backed by Toyota's reputation for reliability. No pressure, just possibilities. Visit buyatoyota.com for a great deal on an efficient Toyota today. Toyota, let's go places.

13:33now now now now now folks we've all been waiting for this moment and here's the ultimate boost equip the cost of inference is coming down this proves that things are getting cheaper there's a bonus trick for you before i get to my bet here we go ask them to explain whether things have actually got cheaper and if they say they have ask them why there are no profitable AI companies. If they say they're in the growth stage, ask them why there are no profitable AI companies. Again, I'd say it's been several years and not got one. At this point, they should try and kill you. But really, I'm about to be petty.

14:11I'm about to be petty for a fucking reason, though. In an interview on a podcast from earlier this year that I will not even quote, because the journalist in question did not back me up, and it pisses me off. Journalist Casey Newton said the following about my work. You don't think that that kind of flies in the face of Sam Altman saying that we need billions of dollars for years? No, not at all. And I think that's why it's so important when you're reading about AI to read people who actually interview people who work at these companies and understand how the technology works. Because the entire industry has been on this curve where they are trying to find micro innovations that reduce the cost of training the models and to reduce the cost of what they call inference, which is when you actually enter a query into chat GBT.

14:55And if you plotted the curve of how the cost has been falling over time, DeepSeek is on that curve, right? So everything that DeepSeek did, it was expected by the AI labs that someone would be able to do. The novelty was just that a Chinese company did it. So to say that it upends expectations of how AI would be built is just purely false and is the opinion of somebody who does not know what he's talking about. Newton then says, several octaves higher, which shows you exactly how mad he isn't, that he thought what he said was very civil, and that there are things that are true, and there are things that are false.

15:31Like, you can choose which ones you want to believe. I'm not going to be so civil. Other than the fact that Casey refers to micro-innovations, what the fuck are you talking about, and Deep Seek being on a curve that was expected, he makes, as many do, two very big mistakes. And personally, if I was doing this, I personally would not have said these things in a sentence that began with me suggesting that I, being Casey Newton in this example, knew how the technology works. Now, here's the Casey Newton quip. Inference, which is when you actually enter a query into ChatGPT, this statement is false.

16:06It's not what inference means. Inference, and I've gotten this wrong in the past too, I'm being accountable, is everything that happens when you put in a prompt to generate an output. It's when an AI based on your prompt infers meaning. To be more specific and quoting Google, machine learning inference is the process of running data points into a machine learning model to calculate an output such as a single numerical score. Except that's what these things are bad at. But nevertheless, Casey will try and weasel out of this one and say this is what he meant. It wasn't. He also said if he plotted the curve of how the cost of inference has been falling over time, well, that's wrong, Casey.

16:41That's wrong, my man. The cost of inference has gone up over time. Now Casey, like many people who talk about stuff without learning about it first, is likely referring to the fact that the price of tokens for some models has gone down in some cases. But you know what, folks? Let's establish some facts about inference. I'm doing the train. I'm pulling the big horn on the invisible train. I'm cooking. Now inference is a thing that costs money is entirely different to the price of tokens. And conflating the two is journalistic malpractice. The cost of inference would be the price of running the GPU and the associated architecture, a cost we do not, at this point, have any real insight into.

17:14Token prices are set by the people who sell access to the tokens, such as OpenAI and Anthropic. For example, OpenAI dropped the price of its O3 model's token costs almost immediately after the launch of Claude Opus 4. Do you think it did that because the price of serving the models got cheaper? If you do, I don't know how you possibly put your trousers on every morning without cutting yourself in half. Now, the cost of inference conversation comes from articles that say that we now have models that are cheaper, that can now hit higher benchmark scores. Though the article I'm referring to, which will be in the show notes, is from November 2024, and the comparison it makes is between GPT-3, which is from November 2021, and LAMA 3.23b, September 2024.

17:57Now, the suggestion is, in any case, that the cost of inference is going down 10x year over year. The problem is, however, that these are raw token costs, not actual expressions of evaluations of token burn in a practical setting. and to really, I realized that was a bit technical. These are just what it costs to do something. It doesn't actually tell you how many tokens will be burned, at what volume they will be burned, because that would change things. And well, wouldn't you know it, the cost of inference actually went up as a result. In an excellent blog from Killer Code, and I did not get the chance to find out the pronunciation of this second name, so I'm just going to call her it's E-W-A-S-Y-Z-S-Z-K-A.

18:38I am so sorry. I would rather spell it out than actually mispronounce it. I hate when people say Zitron wrong. Great blog. Anyway, let me quote. Application inference costs increased for two reasons. The frontier models cost per token stayed constant and the token consumption per application grew a lot. Token consumption per application grew a lot because models allowed for longer context windows and bigger suggestions from the models. The combination of a steady price per token and more token consumption caused app inference costs to grow about 10 times over the past two years. To explain that in really simple terms, while the costs of old models may have decreased, new models, which you need to do most things, cost about the same and the reasoning that these new models use do actually burn way, way more tokens.

19:24When these new models reason, they break a user's input down and break it into component parts, then run inference on each of those parts. When you plug an LLM into an AI coding environment, it will naturally burn an absolute shit ton of tokens, in part because of the large amount of information you have to load into the prompt and the context window or the amount of information you can load in at once, and in part because generating code is inference intensive, and also breaking down all those coding tasks at each of those tasks requiring a coding tool and taking a bunch of inference themselves.

19:53It's really bad. In fact, the inference costs are so severe that Killer code says that a combination of a steady price per token and more token consumption caused app inference costs to grow about 10x over the last two years. I'm repeating myself, I realize, but I really need you to get one thing, which is that the cost of inference went up. But I'm not done. I refuse to let this point go because people love to say the cost of inference is going down when the cost of inference has increased. And they do so to a national audience, all while suggesting I'm wrong somehow and acting superior. I don't like being made to feel this way.

20:26I don't think it's nice to do this to people. And if you're gonna do it, if you have the temerity to call someone out directly, at least be fucking right. I'm not wrong. You're wrong. In fact, software developer influencer Theo Brown recently put out a video called I Was Wrong About AI Costs, They Keep Going Up, which he breaks down as follows. Reasoning models are significantly increasing the amount of output tokens being generated. These tokens are also more expensive. In one example, Brown finds that Grok4's reasoning mode uses 603 tokens to generate two words. This was a problem across every single reasoning model, as even cheap reasoning models would do the same thing.

21:06As a result, tasks are taking longer and burning more tokens. Another writer called Ethan Ding noted a few months ago that reasoning models burn so many tokens that there is no flat subscription price that works in this new world, as the number of tokens they consume to win absolutely nuclear. The price drops have also, for the most part, stopped. You cannot at this point fairly evaluate whether a model is cheaper just based on its cost per tokens, because reasoning models inherently burn, and are built to inherently burn, more tokens to create an output. Reasoning models are also the only way that model developers have been able to improve the efficacy of new models, using something called test time compute to burn extra tokens to complete a task.

21:45And in basically anything you're using today, there's going to be some sort of reasoning model, especially if you're coding. The cost of inference has gone up. Statements otherwise are purely false and are the opinion of somebody who does not know what he's talking about. But you ask, could the cost of inference go down? Maybe? It sure isn't trending that way, nor has it gone down yet. I also predict that there's going to be some sort of sudden realization in the media that inference is going up, which has kind of already started. The information had a piece on it in late August Where they note that Intuit paid$20 million to Azure last year Primarily to access OpenAI's models And is on track to spend$30 million this year Which outpaces the company's revenue growth in the same period Raising questions about how sustainable the spending is And how much of the cost it can pass along to customers Christopher Mims in the Wall Street Journal also had a piece about the costs going up Do not be mad at Chris Chris and I chatted before he submitted that piece Like he literally on Blue Sky called me out It fucking rocks by the way big up to Chris Mims, because it's nice to see the mainstream media actually engaging with these things, even though it's dangerous to the bubble.

22:50But you know what? The truth must win out. And the problem here is that the architecture underlying large language models is inherently unreliable. I imagine OpenAI's introduction of the router to ChatGPT5 is an attempt to moderate both the costs of the model chosen and reduce the amount of exposure to reasoning models for simple queries. Though Sam Altman was boasting on August 10th about the significant increase in both free and paid users' exposure to reasoning models. They don't teach you this in business school. Worse still, a study written up by VentureBeat found that open-weight models burn between 1.5 to 4 times more tokens, in part due to a lack of token efficiency, and in part thanks to, you guessed it, reasoning models.

23:27I quote, The findings challenge a prevailing assumption in the AI industry that open-source models offer clear economic advantages over proprietary alternatives. While open-source models typically cost less per token to run, the study suggests that this advantage could be, and I quote the study, easily offset if they require more tokens to reason about a given problem. And models keep getting bigger and more expensive too. So why did this happen? Well, it's because model developers hit a wall of diminishing returns, and the only way to make models do more was to make them burn more tokens to generate a more accurate response, which is a very simple way of describing reasoning, a thing that OpenAI launched in September 2024 and others followed.

24:07As a result, all the gains from powerful new models come from burning more and more tokens. The cost per million token number is no longer an accurate measure of the actual cost of generative AI because it's much, much, much, much harder to tell how many tokens a reasoning model may burn. And it varies as Theo Boing, I'm keeping that, alright? You get the real cuts, as Theo Brown noted, from model to model. In any case, there really is no changing this path. These companies are out of ideas. Now, another, another one of my favorite ultimate boost equips. This is a classic, and I still get this on social media.