The AI Enshittification Bubble | Cory Doctorow

23:40Demetri Kofinas:Preach, Corey. Now I know why you've written over 30 books because you have a lot to say. So we were just, I don't remember which episode this was. It was a recent episode. We were wondering out loud what exactly happened to the metaverse. It kind of just disappeared. $61 billion later, all that's left for it is some extremely embarrassing Gartner predictions that we'd be spending$13 billion a year by now in the metaverse. Also, I do find it interesting because your specific point about users abandoning the platform, because I've seen that certainly happen on Twitter. I abandoned Facebook a long time ago.

24:12Demetri Kofinas:I'm curious, what do the numbers actually tell us? And are they being padded somewhat to make it seem like people are using the platforms more than they actually are? Because I know a lot of people that are using social media way less today than they used to. Yeah. Yeah. I mean, there's so much, and maybe this starts to get into some of the AI discussion. These guys get to just be Texas marksmen. So they fire a shotgun into the side of a barn and then draw a target around the place where the pellets went in. So Facebook can look at their monthly numbers and pick the one that looks great and then announce that that's their metric.

24:44right? You see this with AI where they say, well, you know, one of two things, the stories that we get from AI, one is that, oh, we're getting better at solving the same benchmark, which is great. So what you're telling me is that this benchmark that didn't exist when you started testing now exists. So you can write bots that, you know, are tuned to achieve the benchmark. That doesn't tell you that the bots are getting smarter. That tells me that you're getting better at tuning bots to whatever, take the LSAT. And then when you say that sometimes AI boosters will say, oh, well, we're actually measuring it against new benchmarks.

25:16In which case you say, well, then that's not getting better. Right. I mean, you know, I could run 50 miles with one shoe on and then, you know, next week it's, I can run 25 miles with, you know, a bucket on my head, right? These are not the same skill and announcing that you've managed one. And then the other doesn't mean you're getting better. It just means you're doing something different. And, you know, the story, the investor story for AI involves the story that you're always getting better and much better very quickly to justify, well, more than a trillion dollars spent in the last year. You see this with all the tech companies, though.

25:49You know, it happens in their accounting. So this is a speciality of a company called SoftBank. They're a venture capital firm behind, you know, WeWork and Uber. They put a ton of money into OpenAI. And their specialty is this business that loses billions of dollars is profitable, but according to a set of accounting standards that we just made up that are not the generally accepted accounting practices, is the GAAP, which are like, there's a reason there's a GAAP. It's because if you're allowed to make up your own accounting standards, anything can be said to be profitable that doesn't actually make it profitable.

26:21And there is no such thing as a kind of profitable that's so cool you can't express it with ordinary math. That's called losing money. And anytime someone tells you we're profitable but not according to GAAP, you should reach for your wallet.

26:34Demetri Kofinas:So the web had really great unit economics. In other words, the cost associated with servicing each incremental web session fell, even as the hardware and software costs related with getting on the web were also falling, this is not what the cost curve looks like for the AI build out, where each subsequent foundation model becomes more expensive to train and to run inference on. How do the unit economics of AI influence how and where it fits within this three-stage platform cycle that you talk about? Yeah. So unit economics, another term for economics, it's very useful to know a little economics.

27:12Don't get an econ degree that inflicts a certain kind of neurological injury on you. But by all means, study some economics. So unit economics are the term that is used to describe what happens to the fortunes of a business as it signs up more customers, improves its technology, and sells more of its products or services. So as you say, the web had great unit economics. And obviously, there was a lot of money lost in the early years of the web. But as a sector, every new user who signed up to use the web made the sector more profitable. And every time that user went back and used the web again, the sector got more profitable.

27:49And every generation of the web was more profitable than the previous one. Now, it could not be more different with AI. Every new user that an AI company acquires costs them money. The more that user uses AI, the more money they lose. Basically, AI companies are selling$100 bills for a dollar a piece. That's the subsidy they're supplying to their customers. And not only that, but every generation of AI technology loses more money than the previous one. And, you know, we can even decompose that into things like capital and operating expenditures because, you know, if you have a business that has high capital expenditures, You have to put a lot of fiber in the ground, say, before you can light up your phone company.

28:29Well, if you go bankrupt and someone buys that fiber at 10 cents on the dollar, the operating expenditure for keeping the fiber going, that's just like electricity for the routers and technicians to operate them and some customer service people in marketing. So it may be that once you erase the capital expenditure, the operating expenditure is low enough that things can go on. And one of the things that apologists for bubbles will sometimes say is, well, the bubble is necessary to get dumb money to finance the capital expenditure, to bootstrap the sector. They lose their shirts, but then someone smarter comes along and operates the capital that's left behind.

29:05But that's not really how it works with AI. The operating expenditure for the frontier models, the foundation models, are extremely high. And again, we go back to these opaque accounting metrics. So one of the claims that the AI companies keep making is that inference, which is like what happens when you actually go to a model and say, like, make me Sonic the Hedgehog with giant boobs or whatever, that the inference is getting cheaper. And so, you know, they've trained the models and that was expensive, but using the models is going to get cheaper and cheaper. It doesn't seem to be the case. For one thing, well, we can't even look at their books, but someone leaked a set of financials to Ed Zitron, who's a very good reporter on this beat.

29:45And Ed analyzed them and saw that while they are spending a remarkably small amount of money on inference, according to their own books, they are spending as much, if not more on marketing than Coca-Cola. So think about all the marketing you've seen for Coca-Cola this month and ask yourself, have you seen that much marketing for OpenAI? Like where is the army? Like Coca-Cola's got like four ad agencies that just handle its account. Where's the four ad agencies handling OpenAI's account? The hundreds and hundreds of people who must be doing this. And Ed's theory, which I think is probably right, is that what they've done is they've hidden inference in their balance sheet as marketing.

30:24They're saying, oh, well, we give away$100 bills for a dollar a piece. That's marketing. That's not inference, even though the marketing takes the form of inference as just being hidden off their balance sheet. So you asked how this applies to inshittification. Now, one of the things about Facebook, Google, Apple, Microsoft, Oracle, these companies that have been so ardently inshittifying is that they had a good business to begin with. right? Google made a lot of money selling ads. So did Facebook. Apple makes a lot of money selling hardware. The squeeze that they put on was not to become like operationally sustainable.

30:59It was to extract more profit after saturating the market. So I want to say that there's like a big difference between a company that squeezes you in order to extract more revenue and a company that squeezes you because they're doomed. And they're basically like grabbing onto you and pushing you under the water to kind of buoy themselves up to get a few more sips of air before they sink to. OpenAI, you know, and the other AI companies, I think money losing as companies in the history of the human race. No one has ever lost as much money as quickly as they have. And it's not like they're losing less money, they're losing more.

31:37And I'll say one more thing about their unit economics. Even if you take it at their word, that inference is going to get cheaper. One of the things that we've seen recently is that when someone comes out with a better model, like Claude is currently better than ChatGPT, the switching costs are really low. Everyone just switches, right? Everyone gave up on ChatGPT and went to Claude, which means that ChatGPT, OpenAI, now have to spend hundreds of billions of dollars trading the next model or they're going to go under. And when they do, well, then Anthropic is going to have to make a new version of Claude.

32:09The idea that you just make the one good AI and then stop because no one else is going to catch up with you has been belied by the entire performance of the industry to date. So even if you take them at their word, which I don't think you should, I think they are lying. But even if you take them at their word, that doesn't mean that they have a business. They don't.

32:28Demetri Kofinas:So let's dig into this a bit more. As you stated in previous investment cycles, classic case is a company like Uber, the subsidy phase was used to track new users, lock them into the platform, and then begin a process of extraction. What is the logic here? Is it just to create KPIs and fuel the growth narrative that then sort of assists this flywheel of share price appreciation, but that eventually has to lead somewhere. So just walk me through the logic here. Yeah, I don't know that they know where it's supposed to end. I think that they might have a little irrational exuberance. By them, I mean the people who are pumping the stock swindle.

33:03But as you just alluded to, when companies are growing, they enjoy very high valuation, what's called the price to earnings ratio. So, you know, a company that is making a million dollars a year that has a price to earnings ratio of 10 has a valuation of 10 million. Like you add up all their stock and it comes to$10 million, sort of what it would cost you to buy the company. Companies that are growing have much higher P to E ratios than companies that are static. And that's because a share in a company is a claim on its future earnings. And so a company that's growing, you know, a company that made a million dollars this year, but is anticipated to make$2 million next year, that company is worth more than a company that made a million dollars this year and a company that's also going to make a million dollars next year.

33:45I think, you know, kind of political progressives like me have done the world a disservice by repeating this idea that endless growth is the without limit is the ideology of a cancer cell. I don't think, you know, the people who are running these companies are trying to grow endlessly for ideological reasons. I think they're trying to do it for material reasons, because the corollary of the idea that a growth firm is worth a lot more than a mature firm is that when a growth firm matures, right, when you Google and you have 90 % search market share, well then you're grossly overvalued and that's again why you see these flash crashes when these companies announce that their growth is slowing down now the thing is when you have a high value stock it's not just that your own stock portfolio if you run the company and have been paid in stock or if you're a shareholder of the company that it's worth more it's also that it's easier to keep growing because when you have a growth stock it's highly liquid which means that you can use that stock in place of cash to engage in growth-friendly activity like hiring key personnel and acquiring other firms.

34:50So if you can buy another company by using stock, then all you have to do is type some zeros into a spreadsheet and you own another company. Whereas if you can only buy another company using money, you better not try and make that money on the premises. Manufacturing your own money is very strongly frowned upon by the treasury department. you have to go get it from a customer or creditor or an investor, which means you lose those auctions, which means that once you start to go into a nosedive because you've reached saturation and the market's selling you off, the most obvious ways to pull out of the nosedive, like hiring people and buying other companies, they get exponentially harder.

35:25And all the people that you were paying in stock suddenly want to be paid in cash. And if you can't, they go to somewhere else. And so you have this kind of self-accelerating collapse. And so firms have to come up with growth narratives. And, you know, early in the saturation phase of dot coms, their growth narratives were about eating each other's lunch. You know, Facebook was going to become YouTube because they did the pivot to video. Google was going to become Facebook because they had Google Plus. The problem is that when you explain that you're about to become Facebook, Facebook makes a very strong case that you're not.

35:56And so there's a lot of tedious, he said, she said, nah, uh-huh. Whereas if you claim that you're going to conquer a market that doesn't exist, like the metaverse or NFTs or blockchain or Web3 or what have you, sure, you might be straining credulity, but who is anyone to argue with you about the size of that market opportunity? You literally just made it up. So if Gartner says there's$13 trillion in the metaverse, no one can offer facts to rebut them. It's all just vibes at that point. So there's been a lot of narrative work in the last 20 years, and it's been accelerating. And now we're at the peak of narrative, which I think is AI and maybe its successor super intelligence.

36:35And I don't think they know what they do next, except maybe do it again. The problem is that we are now in a totally different scale. You know, metaverse was a$61 billion loss for Facebook. They spent$150 billion of the last three years. They lost another$150 billion this year on AI. And they're saying that they're going to lose more next year, more than$150 billion. $150 billion here,$150 billion there, pretty soon you're talking about real money. Yeah.

37:03Demetri Kofinas:Also, the last big bubble we saw was in crypto, and that wasn't a high CapEx bubble. You didn't need to expend a ton of capital, in other words, to sell a narrative. In this case, the narrative is tied to the CapEx, which makes it a more expensive story to sell. While you were talking earlier, I remembered which conversation it was in which Meta came up, and it was my recent episode with Patrick Boyle, where we also discussed SpaceX. And I think Patrick made the point that SpaceX's stock was priced at a hundred times sales at IPO, which means you'd need a hundred years of current revenues to make your money back, which of course means that SpaceX is even more dependent on the AI growth narrative to justify its valuation.

37:45Demetri Kofinas:But when you look at what constitutes AI or AI related revenue at the company, most of it is just selling ads on Twitter. So that's ironically one of the things that this boom has in common with previous booms, which is that whatever the shining new thing is, everyone wants to be that because being an AI company immediately confers upon your stock a certain halo effect, irrespective of the cost curve dynamics that we've been talking about. Another distinction, however, that you seem to draw with Web 1.0 and Web 2.0 is that the AI growth story is being sold on the back of widespread unemployment, and that the investment argument only makes sense if the businesses employing it can also use AI to cut labor costs.

38:33Demetri Kofinas:And you seem to think that the promise of AI is that it either replaces humans with machines or turns them into machines outright by relying on them to do more of the execution and less of the executive work. And this relates directly to the title of your latest book, The Reverse Centaur's Guide to Life After AI. So why don't you take this opportunity to explain what the title of the book refers to and then elaborate on the sort of - Well, it won't surprise you to know that this isn't the first time in human history where automation and labor came into conflict. This is a fairly standard conflict for us to undergo these days and for the last several centuries.

39:11And so there's a pretty good literature about automation and labor. And one of its bedrocks is that when capital drives automation adoption, it tends to do so in service to increased throughput. Because if you're buying a subscription or if you've got a tool that you've acquired that's depreciating off your balance sheet, you want to maximize the use of it. Whereas when labor is in charge of the adoption, It tends to do so in order to improve quality. And of course, where you have a cartel or a duopoly or a monopoly, you can reduce quality and increase throughput, right? You make the prices lower or you make the costs lower and you pocket the difference and you foist a worse product on the world.

39:51You mentioned customer service before. I think that's the canonical example here that there used to be these great fake AT &T ads that Lily Tomlin would do on Saturday Night Live where she'd pretend to be a Bell operator. and they would always end with her saying, we don't care. We don't have to. We're the phone company, right? So, you know, that's a great way to lower your costs. If you don't have to give a damn about solving other people's problems, right? Then you can like lose their luggage, rip them off, overbuild them, whatever. And like, they're just screwed. And you know, the alternative is two cans and a string.

40:24So we see with automation adoption driven by capital, that things tend to get worse, yet capital tends to get richer because they reduce their costs. Maybe sometimes they reduce prices too. So this brings us to centaurs and reverse centaurs. In automation theory, a centaur is a person assisted by a machine. So the person is the head of the horse-human hybrid, and the machine is the body. The person has all the discernment, has the intellect, has the judgment, and the body, the machine is tireless, it's strong, it's dexterous, it's fast, but it doesn't tell you what to do. It goes in the other direction.

41:03A human on a bicycle is a centaur. You practically look like a centaur when you're riding a bicycle, but like so is a human with a spell checker or a calculator. Now the inverse of this is the reverse centaur, and it's pretty obvious I think what it means. It's the horse's head on the human body. it's the machine driving the person. It's when a person is conscripted to serve the machine. And because you're the weakest link, you're the slowest part of the machine, the machine's going to be faster than you and stronger than you and won't get tired as quickly as you, the machine is going to work you at the absolute limit of your endurance and capability.

41:37And so you don't just get used by the machine, you get used up by the machine. And we see this in just like the tropes of automation and labor. Think of Lucy and Ethel on I Love Lucy trying to get the chocolates into the chocolate box at the end of the conveyor belt. And the chocolates are coming so fast that they lose control and they start flying everywhere. Or we all know that scene of Charlie Chaplin getting sucked into the big machine and chewed up by the gears in modern times. But you see this everywhere. Amazon has built the most automated warehouses the world has ever seen. And they are also the warehouses with the highest rate of injury.

42:09And there's a good reason for that. It's not a coincidence. It's a consequence. You spend seven or eight figures on warehouse automation, you don't run those machines slower than they need to be run. You run them at maximum speed, which means that the humans who are the slowest part of the system are running at maximum speed. And you run a human at maximum speed for as long as they can go and they will eventually slip up. And when you slip in a warehouse, sometimes you end up impaled on a forklift.

42:33Demetri Kofinas:Are you familiar with Mark C. Taylor and his book, Speed Limits? No. He was on the show in the early days. I think he was episode three. He's a theologian and a philosopher. And the book very much, it focuses very much on automation and speed and that the world is speeding up faster than we can keep up with it. Really? There's a good book like that. Do you know James Glick who wrote Chaos, Making of the New? Absolutely. My favorite book ever is The Information, one of my favorites. Oh yeah, me too. What a great book that is. So he wrote a book, a very weird, really good book called Faster. So you know this book?

43:08Demetri Kofinas:I haven't. I thought you were going to say a different book, but I'm not familiar with this. He made all the money writing chaos because chaos was this giga seller. He got into weird rich guy hobbies, including flying ultralights, and he crashed an ultralight and lost his leg and killed his son. And he was in a hospital for years. And it was at the start of the web. And he just was out of the world in this incredibly traumatized state for years. and he came out and the world had just gotten three times as fast. And he wrote this book faster that is a memoir of the accident, grappling with what happened to him and his son, and also a meditation on speed and society.

43:51It is one of the most profound and thought-provoking books I have ever read. I can't recommend it highly enough. I mean, the information I think might be his masterpiece, but that book is so good.

44:02Demetri Kofinas:Wow, man, that's heartbreaking. I had no I have no idea that that happened to him. Jim is a wonderful guy. I'm a father of an 18-year-old. And every now and again, whatever we're doing, when we're doing something fun, but maybe a little dangerous, even just riding bikes around. I raised her in London and then we moved to LA riding bikes around. I think, God damn, what would it be like to be doing something with your kid and have that kind of lethal accident? How would you ever come back from it? he's a remarkably well-adjusted man for someone who's lived through such an incredible trauma. And his writing just gets better.

44:41But that book is special. In some ways, it's like a first novel or first book. The first book you write has got everything you ever thought of writing in a book in it. And then everyone afterwards is different. And it's kind of like he was reborn, I think, after the accident. There's the new Jim. And it's kind of like a first novel for the new Jim Glick. It's very good.

45:02Demetri Kofinas:Wow. I mean, it sounds excellent. I don't know if I have the heart or the stomach to read that at this stage in my life as a father of a young boy, but it sounds excellent. I thought the book that you were going to suggest was Time Travel, which I haven't read - Oh, another great book. ... but also just looks like more of a time machine actually. Much more cerebral, much less of a gut punch. Yeah. So one more question and then I'm going to move to the second hour, Corey. So there's a lot that I agree with in your book and then there's some other stuff that I feel like doesn't track with my own personal experience.

45:29Demetri Kofinas:And this centaur and reverse centaur doesn't fully track. So for example, for me, although I don't know that this is inconsistent with your argument, for me, I feel like actually AI is making me more human because I can offload more and more of the rote and perfunctory work to AI. Is that just because of the nature of my work, that there are certain people who are going to be able to remain centaurs and become more empowered. But also, do those opportunities become smaller and smaller in the world we're heading into and that we all become entrapped in a model that eventually turns us all into minotaurs?

46:08Demetri Kofinas:I mean, help me sort of understand exactly the thinking there. Well, you remember the first question you asked is like, how should people think about my work? And I said, well, as a science fiction writer, I'm both excited about the possibilities and frightened of the dark potential. And so I'm not one of those people who thinks there's people who say like AI is going to make you stupid or AI is going to be addictive or what have you. I think those narratives are very overblown, but the people who say your boss bought AI because he wants to either fire you or fire the person who works next to you and make you do the job of two people and then blame you if anything goes wrong.

46:43I think those people are a hundred percent right. and so you know one of the things about the center reverse center framework is it resolves a seeming paradox because i know very skilled programmers who are extremely reliable narrators of their own experience at least historically who say i use ai and it makes my life better and it makes me a better programmer than i've ever been i understand architecture in a way that i never did because i'm thinking through architectural decisions all the time and harnessing this fleet of AI agents that are working within the architecture I'm designing. And I'm having to review that code and I'm finding the inconsistencies in the way I conceived of software because the AI is just the sorcerer's apprentice doing exactly what you tell it.

47:29And even if that makes no sense at all, in the same way that just like programming makes you better at thinking through what a problem is. Because once you decompose a problem into a set of discrete steps And you have to explain how each one works to the sort of idiot prodigy that is the computer, which can do anything you ask of it, but only what you've asked of it and nothing else, including making any inferences about what you really meant. So there are lots of programmers who had that experience, and I believe them when they say it. And then there are other programmers I know who are just as smart, who are just as accomplished, who have just as good a track record of reliably narrating their experience.

48:06And they say, I have never made so much tech debt in my life. I work in avionics, never get on a plane again. I think the whole civilian aviation industry is now doomed. We are shoveling tech debt into airplanes at a rate never seen, right? And those people, I think, are not lying either. And I think the difference is who gets to decide, right? The people in the second group have seen nine-tenths of their colleagues fired and are being asked to make up the difference and are being recruited to be what Dan Davies calls the accountability sink, the person whose name is signed at the bottom of the work and who takes the blame, even if they have been set up to fail by being ordered to review code at a pace that no one could possibly review it, and to review code for periods over which most people code review is among the more boring parts of programming, the least interesting, the least rewarding.

48:58And it's the part that the reverse centaur does. You just audit the code that the AI writes.

49:02Demetri Kofinas:I feel like actually there's an interesting intersection between your work and Shoshana Zuboff's Surveillance Capitalism, and it comes up right now when you talk about who decides. I mean, one of her arguments, central arguments is that it isn't the technology, it's the logic that animates the technology that we should focus on. And in your work, you talk about the thing that you say is who is it for, on whose behalf and who are the people in charge. Not what it does, but who it does it for and who it does it to. Yeah. Exactly. I think that's such a powerful thing to understand when trying to grapple with the subject today, which is that as technologized as the world has become and as much of what this conversation is about is technology.

49:41Demetri Kofinas:Behind it is people and power structures. And I think that's the critique that folks like you make. And I sort of throw you in a bundle of folks who are on the progressive side, who are technologically literate. There's a whole group of you over the years that have grown up either in media theory or have come from the tech side or worked in journalism. And so I think, look, we're going to move into the second hour, Corey. One of the questions I want to revisit is the tech bubble itself. And I want to understand really your thinking here in terms of when and how it pops. Is it popped, for example, as a result of investor capital just running out because of how much we need to fuel this narrative that we talked about earlier?

50:17Demetri Kofinas:I'd also like to talk about the distinction between open and closed because you aren't bearish on AI as an industry. You're just bearish on, I think, some of these large foundation models and large big companies that are at the forefront of this huge capital expenditure boom, but not necessarily some of these smaller models. So I'd be curious to get your opinion on where you think the value is going to accrue. Also, how does the economy change? I feel like there's a huge opportunity for entrepreneurship here. Again, I've seen it in my own life and I could see how this technology could really be like rocket fuel for a lot of independent creators.

50:50Demetri Kofinas:But that also brings me to a question about what is the future of content creation? I mean, even the term content creation is kind of like a really awful way of describing what used to be writing or speaking. But the internet, Google in particular, I mean, the internet was a really amazing place to spend your time in the 90s. But I compared it back then, or I compare it now to what it was back then to a kind of Amazonian jungle where you had hundreds or thousands of independent tribes and cultures speaking their own distinct languages that would be literally right next door to you, didn't even know they existed.

51:25Demetri Kofinas:And then Google search engine changed that. It made that entire jungle findable and you could find everyone. But eventually Google built a business model around their search engine and the internet itself, the web so to speak, changed and it became harder as an independent quote content creator or whatever you want to call it to be as successful as you were during that golden age. And I wonder how does AI change this? How do these large language models change this if you're spending more and more time on platform and you're not getting diverted out. So how does search and discovery change all of this?

51:57Demetri Kofinas:Cybersecurity and a number of other questions that I want to ask you about in the second hour. For anyone new to the program, Hidden Forces is listener supported. We don't accept advertisers or commercial sponsors. The entire show is funded from top to bottom by listeners like you. If you want to access the second hour of today's conversation with Corey, head over to hiddenforces.io slash subscribe and sign up to one of our three content tiers. All subscribers gain access to our premium feed, which you can use to listen to the rest of today's conversation on your mobile device using your favorite podcast app, just like you're listening to this episode right now.

52:31Demetri Kofinas:Corey, stick around. We're going to move the second hour of our conversation onto the premium feed. If you want to listen in on the rest of today's conversation, head over to hiddenforces.io slash subscribe and join our premium feed. If you want to join in on the conversation and become a member of the Hidden Forces Genius Community, You can also do that through our subscriber page. Today's episode was produced by me and edited by Stylianos Nicolaou. For more episodes, you can check out our website at hiddenforces.io. You can follow me on Twitter at Kofinas, and you can email me at info at hiddenforces.io.

53:10Demetri Kofinas:As always, thanks for listening. We'll see you next time.