Cory Doctorow Says Be Skeptical of the AI Sales Pitch

0:00Some people treat ChatGPT like some kind of smart search engine, and some use it to get work done. ChatGPT Work is a new way of working in ChatGPT that can take action across your apps and files, stay with a project for hours if needed, and turn a goal into finished work. It's designed to help you move from a chaotic starting point to a reviewable first version. So all the source materials, briefs, and scattered information that you have to grind through to turn into something useful can just become something useful. Put ChatGPT to work on your most ambitious ideas and projects. Get started at ChatGPT.com by selecting Work Mode, available on Plus and Pro plans.

0:42At Venture Global, we think about what can be done, not what's usually done. Through innovation, Venture Global is not only building some of the largest energy facilities in the world right here in the United States, but delivering American energy at a fraction of the cost and a fraction of the time. So while others are busy talking, we're busy building. That's Venture Global. That's unstoppable energy. If you listen to financial news, you know a lot of time to spend thinking about what's next. The next opportunity. The next investment. The next move. But sometimes what matters most is being ready for what you never saw coming.

1:26For more than 75 years, Cincinnati Insurance has worked with independent agents to help protect businesses, homes, valuables, and more. Because planning for the future isn't only about knowing what's next. It's about making sure you're ready for what you can't predict. Let Cincinnati Insurance make your bad day better. Find an independent agent at CINFIN.com. Bloomberg Audio Studios. Podcasts. Radio. News. Let's just have enough object permanence to win a game of peekaboo and not assume that when these people who are barreling towards a flotation, who are losing money at a rate never seen before, who are reliant on normal investors bailing out their insiders, as a way of walking away with their pocketbooks intact, let alone having the technology continue, that their statements just shouldn't be taken at face value.

2:21There should be a certain degree of healthy skepticism when they talk about this stuff.

2:35Stephanie Flanders:I'm Stephanie Flanders, Head of Government and Economics at Bloomberg, and this is Trumponomics, the podcast that looks at pretty much everything in the economic world of Donald Trump. This week, we have a summer conversation with Corey Doctorow, blogger, journalist, science fiction writer, activist in support of online rights. He was the author, you might remember, of In Shittification. But we're talking today about his latest book, The Reverse Centaur's Guide to Life After AI, How to Think About Artificial Intelligence Before It's Too Late. I read this, actually, just on a few days' vacation and immediately thought of bringing Corey on to Trumponomics because although he takes seriously some of the concerns and debates about life after AI that we often discuss on the show, and he offers this reverse centaur idea that we'll get into as a way of thinking about it, but he also thinks fundamentally and fervently that the AI companies are full of it.

3:34Stephanie Flanders:Most of what they say about AI's capabilities is either hugely optimistic or just not possible in any world any of us is likely to live in. And with all that, the book is also well written, funny and short. Hooray! Corrie Doctorow, welcome to Trumponomics. Well, thank you very much. Pleasure to be on.

3:57Stephanie Flanders:I was thinking about it because although it is quite a short book, which is always welcome on holiday and when you're reading an awful lot about AI and other contexts, there is many sides to it. On the one hand, it is a guide to thinking about AI. And in some ways, I would say to listeners, this is a kind of companion discussion to the one we had with Sarah O 'Connor about her book, We Are Not Machines, back in June. But you're also calling bull on many of the grander claims about AI that animate a lot of that discussion and concern. So I want to get into all of that. But first, I guess we should explain your reverse centaur idea.

4:36Stephanie Flanders:I don't know if it's going to quite catch on in the same way that incentivification did, but, you know, we could do our best. Sure. Well, you know, it won't surprise you or your listeners to know that this isn't the first time we've had a conflict between automation and capital and labor. And so there's pretty good literature describing what happens when automation is taken up either by workers or by their bosses. And broadly, we find that when workers are in charge of automation adoption, they tend to use new tools in order to improve the quality of their outputs. Whereas capital, because they want to maximize a resource that's depreciating off their balance sheet or a subscription they're paying as a fixed cost monthly, wants to improve throughput, realize a good profit from that at the expense of your suppliers and your customers.

5:17And so this brings us to AI and centaurs and reverse centaurs. In automation theory, a centaur is someone who's assisted by a machine. So whether you're using a spell checker or riding a bicycle, you are the human head, you have the discernment, you have the judgment, you exercise will, and then there's a machine below you, you know, the human head on the horse's body. And it's stronger than you and it's faster than you. It's able to go longer than you can, but it's not bossing you around. And the corollary, of course, is that a reverse centaur is the reverse. It's the machine head on the human body, the person who's being directed by the machine.

5:52And, you know, where you have a system where you're trying to increase throughput because you've automated, you are always going to be running the machine at the limit of the weakest, slowest part of the system, which is the person. We see this with Lucy and Ethel at the end of the conveyor belt trying to get as many bonbons as possible into the chocolate box or Charlie Chaplin being sucked into the gears of the machine in modern times. the reason that the motif of automation is the speed up is that when you're a peripheral for a machine doing the part of the job the machine can't do when when you're only there because your boss put seven figures into the amazon warehouse and and wants to realize the maximum dividend from it that machine's going to run as fast as it can which is really as fast as you can and as long as you can go and we all know if you work as fast and as long as you can you will make a mistake.

6:38Stephanie Flanders:We actually did have a discussion around the latest round of Amazon automation in the UK a few weeks ago. And I think some of these absolutely sort of what they would call the cutting edge ones have less injury, but the jobs themselves have become, if anything, even more soulless than they were and even more kind of demotivating than in the past. People aren't running around these warehouses anymore. All these robots are coming to them, but they are definitely still probably suffering more from the psychological problems of having a completely soulless job, which, as you say, is defined by what the machines can't do.

7:16Stephanie Flanders:The way you started reminded me that you have what some people would say was a kind of Marxist tinge to the way you think about the world. And particularly when you focus on the sort of power relations in AI, I, you say we should look at not what the AI tool does, but who to and who for. Yeah, I don't know if that's quite Marxist, but certainly I do think that doing a neoclassical econ degree inflicts a kind of very specific neurological injury that makes you incapable of perceiving or reasoning about power and leads you to engage in fantasies about things like reveal preferences, where, you know, you observe someone who's flogged their kidney to make the rent and go, look at that person with the reveal preference for one kidney, rather than thinking at all about power.

8:00Stephanie Flanders:Well, as someone who studied Marxism alongside neoclassical economics, I don't know whether that's just kind of termed, termed, termed, termed, mitigated. You're speaking to something that a lot of people say in terms of having this desire to have a more human centered approach to AI, to have AI be supporting humans in their work, as opposed to humans kind of running around trying to fill the holes left over by AI. But that doesn't feel like the path we're on when you look at the way these tech companies are structured and how much money they have invested on mass automation. Well, I would say that that human centricity is not just about being squishy and big hearted and wanting people to have a good working life.

8:43It's also about the kinds of outputs that AIs can produce and the limitations on producing high quality output with AI. So one of the things about the Centaur versus Centaur framing is that it resolves a seeming paradox, which is that you encounter, especially like skilled white collar workers who are quite senior and are historically good, reliable narrators of their own experience who say, well, I've just started using AI to write software. I write software. I'm good at it. And I can't tell you how great the software I'm writing is today. It's so wonderful. I've become just like an incredible, like Louis Pasteur of app making.

9:19And then on the other hand, you have equally reliable narrators of their experience who are equally senior who say, like, I work in aviation systems. No one should get on an airplane ever again because we're shoveling tech debt into the sky with the help of AI at a rate never seen. And that will never be mitigated. Right. And so the question is, how do we resolve the seeming paradox of two groups of people who are very similar, seem to be doing the same thing, but having a different experience? Well, once you realize that one group are the centaurs and the other group are the reverse centaurs, you realize they're not doing the same thing at all, right?

9:51There's just some superficial commonalities that when a skilled worker can direct their adoption of a tool broadly, I think that that is good for the quality of their outputs, because that's what it means to be skilled and have discernment. But specifically in the case of AI, these defects that are intrinsic to AI systems that they whimsically call hallucinations, which I think sounds better than we have a defective product, these defects can only be mitigated by people who can distinguish a valid output from an invalid output. When we talk about like using chatbots to teach kids, by definition, kids don't understand the subject that a chatbot is trying to teach them.

10:28They are incapable of knowing when the chatbot is quote unquote hallucinating, right? Experience a defect, which means that you really can only get reliable outputs out of an AI if you operate it at the speed that corresponds to the ability of a skilled practitioner to carefully examine its outputs. And when you increase that speed, even if you have a very skilled worker in the human in the loop, they develop what is called automation blindness. And anyone who's ever gone through an airport checkpoint understands this. You cannot remain vigilant for things that occur very infrequently, which is how we've produced in the form of the TSA, the world's water bottle spotting as mother's history has ever seen, who 95 % of the time failed to spot the guns that red teams smuggle through the checkpoints, right?

11:18And so we see this with white shoe law firms that have these skilled lawyers who get the chatbots to help them with their briefs, and they run down the slippery slope. And the next thing you know, there's a federal judge tearing three strips out of them and saying, I catch you with one more hallucinated citation. I'm referring you to the bar and I'm going to make your client pay your opposing counsel's legal bills for reviewing your defective draft.

11:42Stephanie Flanders:The classic things in medicine, people say, well, you'll be able to use the AI for scanning all these radiology and other things. And you point out, again, this kind of automation blindness actually makes the doctors less good at spotting things with that. Well, I guess it really depends on the configuration. I actually do think AI can produce higher quality outputs. But the difference is, I think that those higher quality outputs don't realize a cost savings. I think that if you want to fire 99 of your programmers and have the remaining one mark the AI's homework, you're going to get a lot of bad software.

12:13If you keep all the programmers on the strength and add an AI that they use or don't use as they see fit with the token budget to go with it, some of them will use it to produce better software, right? But that's not the same as a wage savings. And while I think there is a market for tools that improve the quality of outputs, no one who's investing in Sam Altman's money furnace is saying, let's see how many sales calls you can have where your salespeople go out to a manager and say, how can I increase your cost today? Right? That's just not the return that they're anticipating. So you mentioned radiology.

12:45I've got a personal experience with this. About seven weeks ago now, a radiologist at the Kaiser Hospital in Los Angeles told me that I'm cancer-free. It's very nice. It was always considered a very treatable form of cancer, but you worry. And I've spent a lot of time talking to radiologists during my course of treatment and also reading the literature on radiology and AI. And it seems very clear that there are some solid mass tumors that some chatbots can spot that some radiologists miss. And that Kaiser Hospital, it's right next to the big church of Scientology, because that's how we roll in Los Angeles.

13:18And if there was like an AI salesman, like deking out a Scientologist who wanted to give him a personality test to meet with the CEO of the Kaiser Hospital and say, look, here's the plan. You've got the 10 radiologists you employ at$300 ,000 a year each. That's a$3 million wage bill. Each of them runs 100 x-rays a day. Give me another million dollars. And we're going to stick a chatbot behind them that a couple of times a day taps them on the shoulder and says, take another look at that one. So in addition to the extra million, you're probably going to have to hire another part-time radiologist to take up the slack.

13:50But fewer people are going to die of cancer. As a cancer survivor, I'm totally all over that. That's not the sales call, right? The sales call is you go past the Scientologist, avoid this personality test, get up to the CEO's office, say fire nine of your 10 radiologists. That's$2.7 million a year in wage savings. Split half of it between you and Sam Altman. Have the remaining radiologist mark the AI's homework. turn them into what Dan Davies calls the accountability sink. They're the one who takes the blame when someone dies of cancer. And I think that's like the sales pitch everyone's counting on.

14:21That's why we see this, I think, frankly, very irrational exuberance to invest in AI companies. And it's why I think this sector does not have a future as a sector dominated by extremely money-hungry firms that replace every trivial task with heavily subsidized AI. It's just not going to happen.

14:44Stephanie Flanders:I guess they may not be very good at saving costs for companies, but they could be very good at selling to your employer that they can save costs. So the risk is, and you identify this in the book, you could still get all the disruption that people are worried about, even if ultimately the savings aren't there, or certainly the quality of the work is not there when you replace most people with AI. There's one piece of this future that's been very prominent. There was the Citrini kind of analysis, you know, probably on the far end of, you know, this is going to happen much faster than people think.

15:21Stephanie Flanders:And you're probably on the other end of the spectrum. There was a recent Economist magazine cover that was basically suggesting that agents were going to cause total chaos for government departments all over the world because everyone was suddenly going to be able to put in every claim and resist every parking ticket, whatever it is, use all their rights to the absolute fullest without needing to pay for lawyers. But that world where also we're all sort of able to do enormous amounts of comparison shopping for our car hire or any of these other things, you claim relies on companies and websites all over the world agreeing to consistently label things and organize themselves in a way that agents can easily navigate.

16:06Stephanie Flanders:And as you point out, they don't necessarily have that much interest in doing that. So you basically say that's not going to happen, or at least not to the degree that people are saying. And that just seemed to me, if true, that just seemed to be a massive flaw in this whole view of the world. Yes. The first generation of agentic AI really failed badly. So this was where you tried to create a single agent and say to it, you know, figure out how to use a search engine, then find the travel websites, then figure out how to book a flight, then go all the way through the workflow, then tell me how much the cheapest flight is that satisfies the following requirements, right?

16:43And that was just catastrophic. Those tasks are so specialized that no one could get an agent to really do them. So now what the agentic workflow looks like is decomposing the task to a bunch of specialist bots. So you have the web searching bot, and you have the thing that knows what airport codes are, and you have the thing that can parse out the junk fees and whatever. And so you're breaking this down to a series of specialist agents, each of which hands off to the next one. And the claim of the agentic firms, so let's just sort of steel man their claims here, right? The claim of the agentic firms is that they can get something like 95 % accuracy out of these highly specialized bots.

17:21Well, the thing is that if you decompose a project into like 20, 95 % accurate steps, you get error rates, right? They're multiplicative, multiplicative error rates, a 95 % accurate task performance multiplied by another 95 % task performance. Once you get about 14 steps in, you're under 4 % reliability. So that's looking like it's not going to be hugely reliable either, except for some pretty trivial instances. And not only that, but it's very expensive because each of these chatbots is burning its own set of tokens, right? So decomposing the task at the expense of doing more compute. And so then you have the big idea that everyone's got now, which is agentic workflow, which is that you get everyone in the world who has a website that might be addressed by an agent to redesign that website for machine to machine communication, where the agent says, what is the true price of a flight from New York to London next Wednesday?

18:15and it gives a faithful response. Now, websites go to enormous lengths to hide this ball, right? To stop you from actually knowing what these prices are. This is how we get surveillance pricing, right? The entire direction travel runs against this right now. Now, I know I'm saying that's good.

18:34Stephanie Flanders:You should explain what it is. It's sort of an element of basic economics segmenting your market, but it's sort of on steroids. Yeah. So the idea here is that whatever AI is bad or good at, one thing it's really good at is multivariate statistical analysis and automatic experimentation. So you have these massive surveillance dossiers on everyone who's buying and selling in the world, whether that's workers selling their labor or consumers buying the products of labor. Largely, that's because the US hasn't updated its privacy law since 1988, when it became illegal for video store clerks to disclose your VHS rental history.

19:08And every other form of consumer privacy invasion remains legal in the United States because they just can't pass a privacy law. And in Europe, they do have a privacy law, but all the tech companies pretend they're Irish, so it doesn't get enforced. So you have this kind of privacy vacuum in global law. So you've got these very detailed dossiers on everyone. And so you ask the AI not to do what they were doing in the Don Draper era, which is like dreaming up demographic categories and going, oh, you're a depression haunted can stalker, you're a high school dropout from the Midwest, you're, you know, whatever, and then like running focus groups to find out about you.

19:42Instead, you say naively, take all these people based on their behavioral data and just naively put them in different buckets and now start experimenting on them. So when a consumer comes to you before eight in the morning, that's a variable you can control. You might discover without ever planning to that parents before eight in the morning, when they're trying to get the kids at the door, will take a 15 % charge increase without ever even having to plan it. Same thing happens with wages. AI is actually remarkably good at it. This is just basic multivariate analysis with a lot of automated A-B testing.

20:14And it's the kind of thing AI is great at. And so what we see is more and more labor and consumer surplus being harvested by capital. This is a thing that cuts against the Hayekian idea of markets as accumulators of information, right? Because you're not getting these demand and supply signals. It's effectively a command economy being run by people at the center of the economy who are acting as economic planners on the basis of giant amounts of surveillance data.

20:46This is the Bloomberg Tech Minute brought to you by ChatGPT. Now with ChatGPT work. I'm Carol Masser. Has the AI moment arrived at corporate offices worldwide? Not yet, and it's perhaps several years away, according to one survey of corporate recruiters who circle the world's business schools each fall and spring. Bloomberg's Rob Mandelbaum notes that, according to the Graduate Management Admission Council, the skills recruiters sought most among 2026 graduates of business school master's programs, including MBAs, were communication, problem-solving, and adaptability, which changed little from last year.

21:22Training in artificial intelligence tools ranked in the bottom third of the skills list, but recruiters did say AI-specific training will be more important in five years. On that, recruiters rated current graduates as only somewhat prepared or less to work with AI tools and said business schools are not doing a good job of teaching AI skills. Bottom line, according to the survey, what most companies want when it comes to AI skills is for candidates to be able to automate routine work. That's the Bloomberg Tech Minute brought to you by ChatGPT. Put ChatGPT to work on your most ambitious ideas and projects.

21:58Get started at ChatGPT.com today by selecting Work Mode. Available on Plus and Pro Plans. This is Alexis Christophorus for Bloomberg Surveillance. AI is everywhere. Outcomes are not. Not because AI doesn't work, but because AI hasn't reached the workflows yet. The campaigns, the launches, the quarterly planning. Those workflows are still run by humans alone. AI is making individuals faster, but it's not making businesses more productive. That's the gap Asana is built to close. Asana is the operating system for human agent teams, your easy button for AI productivity across every team. Ready-to-go AI teammates pre-built for marketing, ops, and IT.

22:41No prompt engineering, no setup. They show up where the work is happening, already onboarded in your workflows, ready to deliver. And the more they work with your team, the smarter they get. With Asana, your whole company can work on the same plan towards the same goal, whether you're a team of 10 or 10 ,000. Asana, where humans and agents workflow together. Try it at asana.com. That's A-S-A-N-A dot com. From game day crowds to memorable meals, Genius by Global Payments keeps your kitchen and floor perfectly in sync. Real-time menus, seamless updates, big league reliability for any business. That's genius.

23:29Stephanie Flanders:You mentioned the cost of the credits, the tokens that are being used by the AI. I just wonder whether that will end up putting some constraint on this, because it goes to the sort of economics, what you would say was the bad economics of AI, that every model has cost more to produce, which is exact opposite of what we've seen with many other technologies. And we see that also in terms of the users as well. You know, famously, people having been madly encouraged by their employers to use AI are now having this desperate reverse ferret, as we would say in the UK, where it's like, no, no, you have to be limiting your tokens, you have to show the value.

24:04Are costs going to be a constraint sooner than some might think?