The Theory That Explains Why Everyone Went Crazy

0:00You're being sold an AI future where you're obsolete or irrelevant. That vision is wrong. At Palantir, they're building AI that helps workers and unlocks their full potential. American workers are our nation's greatest strength. AI shouldn't eliminate them. It should elevate them. Palantir is here to tell their stories. From factories to hospitals, AI is freeing people from drudgery, letting them do what humans do best. Create. Solve. Build. Palantir, making Americans irreplaceable.

1:01Acrobat Studio. Learn more at adobe.com slash do that with Acrobat. Bloomberg Audio Studios. Podcasts, radio, news.

1:26Hello and welcome to another episode of the Odd Lots podcast. I'm Joe Weidenthal. And I'm Tracy Alloway. Tracy, you know, there's a lot of like concerns about AI, obviously, these days. And if we anyone who's like reasonably intelligent can like list tons, like maybe like it's going to go rogue and be smarter than us or maybe whatever. But I still think like what or maybe there's just going to be this like flood of disinformation and deep fakes or maybe it's going to put all journalists out of business, which is certainly plausible. But I think, you know, like something I think a lot about is just this idea that regardless of what happens, like we're going to be increasingly sort of trusting like a black box for answers that we really have no idea where those answers, however you want to describe that, come from.

2:15Yes, absolutely. And this is something that's come up on the podcast a number of times now. I'm thinking way back to an episode we did that was basically about the black box of algorithms and how difficult it was to understand what goes into them and then what comes out. And then, of course, we recently did that episode on pricing and the idea of algorithmic pricing, the idea of building proxy consumer profiles. And you're right. The issue is we know that there's this new technology. We know that there's all this data floating around, but we don't entirely know how it is coming to the conclusions or creating the output that it actually is.

2:56You know, the pricing thing is interesting because, you know, in a market economy, you know, you could argue it's like, oh, at any given moment, you know, you're being served up to like the optimal price. Right. And in theory, even with the most advanced algorithms and stuff like maybe there's some like this price is happening at the best price for both the seller and the buyer, etc. But I think people just have a sort of deep, intuitive distrust about the fact that you can't go there and touch it and verify it and see this is why this exists in the state that it is. And I think it's going to create a lot of, I don't know, cultural apprehensions as more and more decisions and more and more things that affect our lives just seem to emerge spontaneously out of the box.

3:45Absolutely. I'm thinking of all the people working at Chipotle who are going to have to answer questions about not just portion sizes now, but also questions from customers about are they getting the best price? Have you seen those awful videos that people are taking of the Chipotle workers? Yeah, I've seen some of them. It's so vile. I can only imagine. It's so vile. Anyway, that's a separate thing. But yes, like all of these things. And, you know, it's it's not just with AI, obviously. And like this sort of world exists in increasingly black boxes. You put in a support ticket. You try to like talk to someone in an embassy or consulate or anything you do.

4:22and you sort of like send out some requirement or some request to some bureaucracy or some company. And then it moves around. I was on a, I had a flight recently that was delayed for nine hours. And there's like this palpable frustration that everyone feels that the person, you know, standing at the gate, like can't answer their questions and they can't get anyone to answer their questions. And it just sort of like, you know, everyone explodes and everyone knows it's not the gate agent's fault, but still like, you know, there's just this frustration. Like, where is the answer to what's going on?

4:53Right. And you can't ask a single person because that single person, like the gate agent, doesn't have the answers. I think what's happening is like society has organized itself in such a way as to devolve responsibility. And the creation of all this new technology is basically going to, I guess, ramp all of that up. Right. Like so it might not even be the gate agent that you're asking in the future. It might be you trying to like, I don't know, ask the algorithm like why it decided to bump you versus someone else. Or actually that already happens. Right. There is an algo that dictates who gets bumped from the plane and who doesn't.

5:35So, yeah. The other big one, of course, is health insurance and why some claims are suddenly denied and you never get this answer. Anyway, the world is already filled with systems in which we have some question and no one actually can sort of, you know, give you the answer. Yes, absolutely. And I think we might have the perfect guest. We do have the perfect guest. It's someone we've talked to multiple times on the podcast. One of the smartest guys around, always interesting, always worth paying attention to. We're going to be speaking with Dan Davies. He is the author of the new book, The Unaccountability Machine.

6:11Why Big Systems Make Terrible Decisions and How the World Lost Its Mind. So this should be really fun. Dan, thank you so much for coming back on the podcast. Oh, thanks very much for inviting me. You know, before we get into the meat of your argument, the unaccountability machine, I'm actually curious about the second half of the title, How the World Lost Its Mind, because I certainly feel like the world lost its mind. But I'm like a middle-aged boomer. And I feel like, you know, anytime you get to my age, you're like, oh, the world's gone crazy. the world's gone mad. Why is everything so nuts these days?

6:46Do we actually know that the world's gone mad or is it just because we're all sort of old and cranky now, everything seems like the world's gone mad? Well, from your own perspective, you can never be sure. But I think there's actually reasonable, objective ways that you can check up on this just by noticing that the world gets more complicated as it gets bigger and it gets exponentially more complicated and I mean that in the literal mathematical sense because the number of connections grows faster than the number of things whereas our capability to understand the world manage it and make decisions doesn't necessarily grow exponentially so this is the story I would argue of economics it's the story of any management book that's worth reading, because the central problem of management is the world is getting faster, more complicated, faster than you can process that complexity.

7:42What are you going to do about it? How are you going to reorganize? And, you know, we've just been through a global financial crisis. We've just been through a political, what Adam Tews calls, poly crisis. I think there's decent reason to believe it's not just because we're getting older, And it is actually a crisis of the ability to make decisions matched up against the speed and complexity of the decisions that we're having to take. So when we talk about the lack of accountability and making bad decisions, give us some concrete examples of things that you have spoken about or written about in your book.

8:20What are you thinking about here? Well, I mean, there are kind of, there's little trivial funny examples and there's big, huge, serious examples. So, for example, and I apologize in advance because this is quite disgusting. Oh, is it the squirrels? Would you rather I didn't talk about squirrels? No, you could, this is bad though. Tell us the squirrels. If there are children listening, maybe get them out of the room. Children don't listen to our lives. At the start of this century, there was a craze for squirrels as pets in Europe. And squirrels were being imported from North America and China to be pets, and they had to have the right paperwork.

9:02And so one day, a load of 400 of the poor little things showed up in Schiphol Airport in Amsterdam without any paperwork and without a return address to send them back to. and it's difficult to know what the airline should have done but you can't help thinking that there must have been a better solution than what they actually did do which was that they threw all 400 of them except for one or two that escaped into an industrial shredder and this this caused an outrage there were questions asked in the Dutch parliament and people immediately started asking, how did this happen? Who is responsible?

9:46And in fact, the press release from the airline apologising for this is studied as a masterpiece of crisis PR in business schools. But when they went back to inquire, they ended up realising that no one had ever really made the decision that that was what they were going to do. The government's biosecurity ministry had set some standards for the importation of small animals. The airline had set some standards for compliance with that policy. The only people who were expected to make a decision about whether this was grotesque and couldn't be done or not were some low-level employees in a shed at Chipporl Airport.

10:30And frankly, people who work in sheds aren't usually going to be thinking that they're meant to be second-guessing the government. And so what happened is that you had this phenomenon that turns up a lot of the time at all levels of organisation, which is that something happened which nobody wanted, but was the predictable output of the system that they had created.

11:10Silicon Valley is selling you a future where you're obsolete, or worse, identical. At Palantir, they're witnessing something different and revolutionary. From re-industrializing the nation's defense base, to shipyard workers building faster, and frontline workers boosting productivity, AI is transforming work across the nation. AI is not replacing American workers or flattening them into conformity. It's unleashing what makes each one irreplaceable. Their judgment, their craft, their creativity. When American workers become more powerfully themselves, they own the future. Palantir, making Americans irreplaceable.

11:52Running a business is hard enough, so why make it harder with a dozen different apps that don't talk to each other? One for sales, another for inventory, a separate one for accounting. Before you know it, you are drowning in software instead of growing your business. This is where Odoo comes in. Odoo is the only business software you'll ever need. It's an all-in-one, fully integrated platform that handles everything. CRM, accounting, inventory, e-commerce, HR, and more. No more app overload. No more juggling logins. Just one seamless system that makes work easier. And the best part? Odoo replaces multiple expensive platforms for a fraction of the cost.

12:30It's built to grow with your business, whether you are just starting out or already scaling up. Plus, it's easy to use, customizable, and designed to streamline every process so you can focus on what really matters, running your business. Thousands of businesses have made the switch, so why not you? Try Odoo for free at odoo.com. That's O-D-O-O dot com. Yeah, this is, first of all, that's grim. But also, like, when you describe it that way, you can see it because ultimately, like, all right, here's this awful thing that happened to 398 squirrels. I guess in theory, someone had to, I don't want to get, I don't know, dump the bag into the shredder.

13:12I have not researched the details intentionally. That's not a very satisfying answer. I mean, yes, okay, maybe there was someone who did the physical thing, and I guess the entire operations of the airline and the port and the customs bureau could just blame that one person. But that's not a very satisfying conclusion, I guess, in terms of how this actually happened. It's funny, I brought this documentary up. I was thinking about this too, like with the destruction of the old Penn Station in New York, which is like this like extraordinary, like, I guess like Roman or Greek building. And they just tore it down to like build the pretty awesome hideous.

13:52Yeah, something hideous in its place. And now the new Penn Station is terrible, but it feels like the same thing. It's like, how did no one stop and say like, wait, does this make any sense in the long time, in the big picture? Yeah. And the thing is, they'd created a system on the assumption that squirrels would show up in ones and twos and they could be dealt with as individuals and that therefore you would never get into this sort of situation. because when you build a system, you're always building a model of the world. And if something happens which doesn't fit into your model in the world, your system might do something awful.

14:26And there's a sort of symmetry and a kind of resemblance here with much bigger and more grim things like the Boeing 737 MAX, like the LIBOR scandal in financial markets. It's not so much that anyone sat down and said, let's form a conspiracy to manipulate interest rates or let's build a plane that crashes under certain circumstances. It's just that no one set things up so that that wouldn't happen. Do you think, I'm afraid going forward, the squirrels are probably going to be our archetypal example of this. But do you think with the squirrels, for instance, the hyper specificity of the goals or the job roles of everyone involved contributed to the outcome.

15:14So in the sense that you have, you know, I guess the Dutch Wildlife Department who is trying to protect Dutch wildlife. Then you have like the guys at the airport who are charged with actually carrying out these orders. And then you have the airline, which is charged with like looking at the paperwork. Do those tend to lead to, when taken all together, do those tend to lead to worse outcomes? Yeah, absolutely. It's this kind of fragmentation of the decision, which is a result of the industrialization of the decision. And it's industrialization literally in the Adam Smith sense that you don't have anyone in the pin factory building an entire pin.

15:54You have 13 guys all performing one simple operation, and that's a much more productive way to do things. But then when you apply it to decision making, you have the problem that everyone assumes that everyone else is going to react when something unplanned happens. So this is, like I say, it's the central problem of every good management textbook. How do you deal with information? How do you get a drink from a fire hose? How do you stop yourself from being overwhelmed? The answer is always in some way or another, you build a system to make the decisions for you. But once you've built that system to make the decisions for you, you no longer feel ownership of that decision psychologically.

16:37You no longer feel like you're accountable for the decision, because if you were accountable, you might be able to change it. But if you're going to be held accountable for this thing, you haven't really moved that thing on. You haven't really delegated it to the system. You mentioned Boeing. And speaking of Boeing, there was a great blog post back in April from Steve Randy Waldman, Interfluidity, who I consider another one kind of in your category of people who have basically been writing interesting things on the Internet for a very long time. And he has this title is seeing like a CEO and this idea that, you know, when Boeing merged with McDonnell Douglas, the McDonnell Douglas CEO became an outside hire.

17:19And he had to essentially gain some legibility into this new organization that he inherited. And that was the cause for some of this like streamlining and offshore or, you know, offshoring things like that. Talk to us like about, you know, you mentioned Boeing and it's more, you know, the crises there that's been going on several years. It's a more severe, serious issue than the squirrels. But how do you think about what happened there? I think about it in very similar ways to Steve-Randy Walporn because the – and you see it in Boeing, but it's visible to a greater or lesser extent in very, very many companies, probably the majority of companies today, that the C-suite has an information environment which is almost completely composed of financial numbers.

18:08because the financial numbers are taken by them as objective facts. We can talk long and hard with accountants about how objective those financial numbers are and how many assumptions go into them, but they arrive on a spreadsheet looking very much like objective facts about the world. Things like engineering level, in principle, are objective facts, but you have to do a lot more work to find them out and to know what's relevant. and then you have issues like culture and kind of the social environment which aren't even capable of being quantified yeah so there's always this tendency if you're trying to do that thing of manage your own capacity to manage your information flow that you're going to concentrate on the things that look finite and look manageable and that's always going to be the financial numbers which can be a big problem because financial numbers can mislead.

19:09You can create illusions in an accounting system the same way that you can with anything else. How did we end up here? And I'm thinking specifically to one particular development in the world of business, which is the creation of the limited liability company. And I guess the clue is kind of in the name there. But what were the decisions or the trends that sort of came about in creating the current system? Well, I mean, the limited liability company is certainly a big kind of change if you think about these things in feedback terms and information terms. And one of the big arguments of the book is that we should be looking to the mathematics of information theory rather than the mathematics of optimization to explain and model some of these things but a limited liability company is an information filter it tells you that outcomes below a certain amount aren't going to affect you anymore and that changes your information world it changes what you care about but then what i think really started doing the damage so to speak was the development of the leverage buyout in the 1970s and the shareholder value movement as it was kind of really kicked off by Milton Friedman's essay on the social responsibility of a business to increase its profits, New York Times 1970, but then built on by just the entire kind of two decades of business school research that followed from that.

20:46Because again, thinking about it in information terms, a leveraged buyout is a massive screaming signal. The requirement to make the payments on debt becomes a signal that swamps anything else you might be thinking of. Because if you are a CEO and you've got LBO levels of debt, that's your priority. You can't think about anything that isn't related to servicing that debt. Let's go back and talk about the sort of big picture ideas in your book. You talk a lot about this field called cybernetics. And And cybernetics as a name sounds like something that they would have like come up with in the early 90s, like the Wired magazine people like would get into like cybernetics in 91.

21:32But actually, this field has been around at least since the 1940s. I'm surprised they even had a word like cybernetics in the 1940s. But what is cybernetics? And talk to us about the sort of general framework you use as a sort of to start talking about the stories in your book. Sure. I mean, you're right. It's Second World War kind of talk. It's originally from a word meaning the man who steers the boat. So it's cybernetics in that sense. And the first guy to use it was a scientist working on creating an automated gun site for the United States Air Force. and the idea here is that there is some quantity that is preserved in an automated gun site between the operator the radar the server motors and that and all the components of that system and that component is information and so at the time this was uh norbert vena i'm talking about the uh scientist in the automated gun sites another guy working in the same field you might have heard of was Claude Shannon at Bell Labs, who was inventing information theory at Bell Labs and has some fundamental theorems there.

22:48And in many ways, science of cybernetics is information theory applied to control. So you might have an information theory kind of piece of maths that tells you how much bandwidth you need to transmit a given signal. And the cybernetic interpretation of that math would be that it's telling you how much capacity you need to manage a system that's of similar kind of noise. So this was all made huge use of in controlling things where you have access to the whole information environment. So a lot of that early maths from the first cyberneticians has just kind of stayed with us through the invention of the electronic computer.

23:38And a lot of it is actually at work in really modern artificial intelligence. I met a guy called Ben Rechts who works on recommendation algorithms. And he's quite upfront that a lot of his fundamental mathematical techniques are from the 40s and 50s just being applied in the context of massively more computing power. What I'm interested in is where you take those kinds of theorems and apply them in a slightly more unrigorous, slightly more metaphorical sense to situations of management and organisation where you don't have access to the full information environment and you just have to say, we're going to think about this not in an optimising economics kind of neoclassical economic sense.

24:29We're going to think about this as a system that has to be kept under control. And we're going to say, well, how much resource do we need in order to stabilise this system as a system? Which is a kind of abstract way of looking at it, but it's the same fundamental problem of management. How do you get a drink from a firehouse? How do you match your own capacity to manage to the complexity of the thing that you're in charge of. Wait, can you give us a practical example of the application of cybernetics, I guess, because it does sound, to your point earlier, it sounds a little bit abstract in my mind.

25:05Well, I mean, practical example, I think, is the history of the developments of the corporation. So from the first days of the American railroads, which were probably the first really big corporate structures the world ever saw you have this problem that as the network builds out it gets more complicated and the ability of the head office to manage it doesn't grow faster and you can try and solve that by adding more people you can get a great improvement by adding wireless telegraphs but fundamentally at some point this railroad is going to grow big enough that you have to devolve you have to split it into branches and you have to give autonomy to some of the subsidiaries because that's the only way that you can match the bandwidth of the management to the bandwidth of the control problem so i'd say that what what we always see in any big organization is that it grows it gets more complicated it tries to deal with that by adding more resources at head office, it ends up not being able to keep up, and then it reorganizes.

26:23And the reorganizations almost always either involve pushing responsibility down to the shop floor or down to the branches, or they involve spinning off parts of the business into a separate organization and giving up the task of controlling it at all.

26:55Running a business is hard enough, so why make it harder with a dozen different apps that don't talk to each other? One for sales, another for inventory, a separate one for accounting. Before you know it, you are drowning in software instead of growing your business. This is where Odoo comes in. Odoo is the only business software you'll ever need. It's an all-in-one, fully integrated platform that handles everything. CRM, accounting, inventory, e-commerce, HR, and more. No more app overload. No more juggling logins. Just one seamless system that makes work easier. And the best part? Odoo replaces multiple expensive platforms for a fraction of the cost.

27:33It's built to grow with your business, whether you are just starting out or already scaling up. Plus, it's easy to use, customizable, and designed to streamline every process. so you can focus on what really matters, running your business. Thousands of businesses have made the switch, so why not you? Try Odoo for free at odoo.com. That's O-D-O-O dot com. Hey, Ryan Reynolds here. Wishing you a very happy half-off holiday because right now Mint Mobile is offering you the gift of 50 % off unlimited. To be clear, that's half the price, not half the service. Mint is still premium unlimited wireless for a great price.

28:12So, that means a half day. Give it a try at mintmobile.com slash switch. A front payment of$45 for three-month plan equivalent to$15 per month required. New customer offer for first three months only. Speed slow after 35 gigabytes if network's busy. Taxes and fees extra. See mintmobile.com. Wells Fargo announced a new$20 million program in 2025, teaming up with nonprofits to support small business owners. That's how Wells Fargo is helping strengthen small businesses and communities. Wells Fargo, the bank of doing. Learn more at wellsfargo.com slash say do. What are accountability sinks? The accountability sink is just it's a name for a particular move of cybernetics that I notice a lot of these days, which is when you consciously break the feedback links from the subjects of a particular decision to yourself or to the unit that's kind of meant to be making it so your gate agent joe is just a classic example of the accountability sink because they talk to you with the voice of a corporation and they say that this is the policy, there's nothing I can do to change it.

29:27And then you only able to talk back to them as a human being like yourself. So you can't get mad at them because it's not their decision. Right. People do get mad. Just to be clear, I did not get mad at my nine hour delay. There's a video of Joe out there somewhere. There's no video. I just sat there and I closed my eyes and I did stand around because I was sort of curious the gate banter, but I did not get mad. I just want to establish that. Yeah, but then you might ask them, and I will confess I've done this, I've asked someone politely for the phone number of someone who I can call up who is responsible for that decision.

30:03And, you know, it's not the policy. You can't get that phone number. The whole point of this was to create a sink into which unpleasant feedback can be poured and dissipated harmlessly. And when you start thinking about these things in terms of accountability sinks, you start seeing them everywhere. Because everywhere that there's a policy that can't be broken and no feedback to the person who could get the policy changed, that's an accountability sink. That's a way that someone has protected themselves from the consequences of their decisions, possibly at huge cost to the organization that they're working for, but possibly not.