CrowdStruck

0:00This is an iHeart Podcast.

0:30Thomson Reuters and specialized bikes have since they upgraded to the next generation of the cloud. Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high availability, consistently high performance environment, and spend less than you would with other clouds. How is it faster? OCI's block storage gives you more operations per second. Cheaper? Better? OCI costs up to 50 % less for computing, 70 % less for storage, and 80 % less for networking. Better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds.

1:12This is the cloud built for AI and all your biggest workloads. Right now, with zero commitment, try OCI for free. Head to oracle.com slash strategic. That's oracle.com slash strategic. Wells Fargo has awarded$138 million in grants to nonprofits, supporting military and veterans with housing, small business, career transition, and more over the last 10 years. It's one of the many ways Wells Fargo seeks broad impact in communities. Wells Fargo, the bank of doing. Learn more at wellsfargo.com slash say do. Support includes contributions from Wells Fargo and Company and the Wells Fargo Foundation.

1:54call zone media hello and welcome to a very special emergency episode of better offline i'm ed zitron i'm your host and i'm recording this from inside a closet in a hotel in san francisco you're very important to me

2:18on friday afternoon i sat at my desk and just started writing without any clear aim or objective, other than a desire to wrap my head around probably the most cataclysmic technological meltdown that I've seen in my career. And of course, I'm referring to the CrowdStrike situation. How was it that a piece of software, one that few people understood, made by a company that people really didn't know, was able to shut down our banking system, air travel, TV, logistics chains, those weird screens that you see around, and of course, hospitals? And as I wrote this script, I found myself returning to some of the themes that I wrote about in The Rot Economy and The Shareholder Supremacy and many other pieces that speak to a larger problem in the tech industry.

2:59A complete misalignment in the incentives of most major tech companies, which has become less about building new technologies and maintaining them and then selling them to people who would then use them over time, and more about capturing monopolies and gearing organizations to extract value from the things around them. Every problem you see is a result of the tech industry, from the people funding the earliest startups to the trillion-dollar juggernauts that dominate our lives, and the fact that it's no longer focused on the creation of technology with a purpose and organizations driven towards said purpose.

3:35Everything's about expressing growth, and about showing how you will dominate an industry rather than serve it, and providing metrics that speak to the paradoxical notion that you'll grow forever without any consideration of how you'll actually live that long. Legacies are now subordinate to monopolies. Current customers are subordinate to new customers and products. Well, they're considered a means to introduce a customer to a form of parasite designed to punish the user for even thinking about moving to a competitor. The key difference between what happened on Friday with CrowdStrike, and by the way, it's still being fixed and as I'll explain later, will really take some time to be fully resolved, and my criticisms of other companies like Facebook and Google, is the sheer violent nature of this failure.

4:21The decline of search and social tools we use in it is kind of a gradual incremental kind of rot. CrowdStrike, meanwhile, was a demonstration of what happens when the rod fully consumes the timber holding up the building. What's happened with CrowdStrike is completely unprecedented, and I'll get to why shortly, and on the scale of the much-feared Y2K bug that threatened to ground the entirety of the world's computer-based infrastructure once the year 2000 began. You'll note that I'm not saying that Y2K was overhyped or dismissing the scale, because Y2K was a huge society-threatening calamity waiting to happen and said calamity was averted not through any kind of magical thinking but through a remarkable half-trillion-dollar industrial effort that took a decade to manifest because the seriousness of such a significant single point of failure would have likely crippled governments, banks and airlines.

5:17People laughed when nothing happened on January 1st, 2000, assuming that all that money and time had been wasted, all of the media was just being hysterical, rather than being grateful that an infrastructural weakness was identified, taken seriously, and that a single point of failure was dealt with, and that a crisis was averted by investing in stopping bad stuff happening before it does. Crazy goddamn idea, huh? But as we speak, millions, or even hundreds of millions, of different Windows-based computers are now stuck in a doom loop, repeatedly showing users the famed blue screen of death thanks to a single point of failure in a company called CrowdStrike, the developer of a globally adopted cybersecurity product designed, ironically, to prevent the kinds of disruption that we witnessed on Friday and we're still witnessing today.

6:02And for reasons we'll get into shortly, this nightmare is going to drag on for several days, if not weeks to come. The product, called CrowdStrike Falcon Sensor, is an EDR system, which stands for Endpoint Detection and Response. If you aren't a security professional and your eyes are glazing over, I'll keep it brief. An EDR system is designed to identify hacking attempts, remediate them, prevent them. They're big, sophisticated, and complicated products, and they do a lot of things that's quite hard to build with the standard tools available to Windows developers. But, as I'll get to later, not Microsoft.

6:35And so, to make Falcon's sensor work, CrowdStrike had to build its own internal kernel driver. Now, kernel drivers operate at the lowest level in the computer. They have the highest possible permissions. but they operate with the fewest amount of guardrails because massive control and they're very important to the system. Very technical people are going to hear that and be like, that's not the right way to put it. Get out, not your podcast. But if you've ever built your own computer or you remember what computers were like in the dark days of Windows 98, you know that a single faulty kernel driver can wreak havoc on the stability of your system.

7:08The problem here is that CrowdStrike pushed out an evidently broken kernel driver that locked whatever system that installed it in a permanent boot loop, meaning that it just started, blue screen of death, restarted, kept doing it. The system would start loading windows, encounter a fatal error, and reboot, and then reboot, and then reboot again, and again, and again, in essence, rendering the machine useless. It's convenient to blame CrowdStrike here, and perhaps that's fair, and I intend to do so several times. This should not have happened. On a basic level, whenever you write or update a kernel driver, you need to know it's actually robust and won't shit the bed immediately.

7:47Regrettably, CrowdStrike seemed to borrow Boeing's approach to quality control, except instead of building planes where the doors fly off and Boeing is the noise it makes when they fly off at the most inopportune times, it released a piece of software that blew up the transportation and banking sectors, to name just a few. It created a global IT outage that has grounded flights and broken banking services. It took down the BBC's flagship TV channel for kids, infuriating parents across the British Isles, as well as Sky News, which, when it was able to resume live broadcasts, was forced to do so without graphics.

8:21In essence, it was forced back to the 1950s, giving it an aesthetic that matches the politics of its founder and former owner, Rupert Murdoch. By no means is this an exhaustive list of those affected either. The scale and disruption caused by this incident is unlike anything we've ever seen before. Previous instances like this, particularly rival ransomware outbreaks like WannaCry, simply can't compare, especially when we're looking at the disruption at the sheer scale of this problem. Still, if your day has been ruined by this outage, at least spare a thought for those who'll have to actually fix it.

8:55Because those machines affected are now locked in this boot loop, it's not like CrowdStrike can just release a new software patch and call it a day. Undoing this update requires some users to have to individually go to each computer, loading up safe mode, a limited version of Windows with most non-essential software and drivers disabled, and manually remove the faulty code. And if you have encrypted your computer, that process gets a lot harder. Servers running on cloud services like Amazon Web Services and Microsoft Azure, you know, the way that most of the internet infrastructure works, requires an entirely different and much more annoying separate series of actions.

9:33If you're on a small IT team and you're supporting hundreds of workstations across several far-flung locations, which really isn't unusual these days, especially in sectors like retail and social care, you're especially fucked. Say goodbye to your weekend, your evenings. Say goodbye to your spouse, your kids. You won't be seeing them for a while, and I'm really sorry. I'll buy you a drink sometime. Your life will be driving from site to site, applying the fix and moving on. Forget about sleeping in your own bed or eating a meal that wasn't brought to you by DoorDash. Good luck, Godspeed, God bless.

10:04I do not envy you. I'm so grateful I have a fake job. But you know who I do envy? Those buying the products that follow this utterly seamless ad break which will likely echo my exact sentiments on literally every issue ever.

10:24Parking shouldn't slow you down. ParkWiz gives every driver a shortcut. Book ahead, save up to 50 % and skip the hassle of circling the block. Park smarter, park faster. ParkWiz. Download the ParkWiz app today and save every time you park. So I've shopped with quints before they were an advertiser and after they became one. And then again, before I had to record this ad, I really like them. My green over shirt in particular looks great. I use it like a jacket. It's breathable and comfortable and hangs in my body nicely. I get a lot of compliments. I liked it so much. I got it in all the different colors.

10:55along with one of their corduroy ones, which I think I pull off. And really, that's the only person that matters. I also really love their linen shirts, too. They're comfortable, they're breathable, and they look nice. Get a lot of compliments there, too. I have a few of them. Love their rust-colored ones as well. And in general, I really like quints. The shirts fit nicely, and the rest of their clothes do, too. They ship quickly. They look good. They're high quality. And they partner directly with ethical factories and skip the middlemen. So you get top-tier fabrics and craftsmanship at half the price of similar brands.

11:22and I'm probably going to buy more from them very, very soon. Keep it classic and cool this fall with long-lasting staples from Quince. Go to quince.com slash better for free shipping on your order and 365-day returns. That's Q-U-I-N-C-E dot com slash better. Free shipping and 365-day returns. Quince.com slash better. Hey, it's Ryan Reynolds here from Mint Mobile. Now, I was looking for fun ways to tell you that Mint's offer of unlimited premium wireless for$15 a month is back. So I thought it would be fun if we made$15 bills. But it turns out that's very illegal. So there goes my big idea for the commercial.

12:04Give it a try at mintmobile.com slash switch. A front payment of$45 for a three-month plan equivalent to$15 per month required. New customer offer for first three months only. Speed slow after 35 gigabytes if network's busy. Taxes and fees extra. See mintmobile.com. And we're back. The significance of this failure, which isn't a breach by the way, and in many respects is far worse, at least with the disruption it caused, is not its damage to individual users, but to the amount of technical infrastructure that runs on Windows, and that so much of our global infrastructure relies on automated enterprise software that, when it goes wrong, breaks everything.

12:40It isn't about the number of computers, but the amount of them that underpin things like security checkpoints, or systems that run airlines, or banks or hospitals, all running as much automated software as possible so that the cost can be kept down. Hey, remember the rot economy? Jesus fucking Christ. The problem here is systemic. That there's a company that the majority of people affected by the outage had no idea existed until, well, a day or two ago. That Microsoft trusted to the extent that they were able to push an update that broke the back of a chunk of the world's digital infrastructure.

13:12Microsoft, a company, instead of building the kind of rigorous security protocols that would say, I don't know, rigorously test something that connects to what seems to be a huge portion of Windows computers, well, they just chose to do something else. They just screwed the fuck up. As pointed out by Wired, the company vets and cryptographically signs all kernel drivers, which is sensible and good because kernel drivers have an incredible amount of access and thus can inflict serious harm, with this testing process usually taking several weeks. What happened, Microsoft? How did this slip through Microsoft's fingers?

13:48Well, for this to have happened, two companies needed to screw up epically, and boy fucking howdy, did they. What we're seeing isn't just one major fuck-up, but the first of what will be many systemic failures, some small, some potentially larger, that are the natural byproduct of the growth at all costs ecosystem where any attempt to save money by outsourcing major systems is one that must simply be taken to please the beautiful sexy shareholder that they all love so much. And this is a problem with the digitization of society, or more specifically the automation of once manual tasks. It introduces a single point of failure, or rather several of them, all clustered together like a rat king or a katamari.

14:31Our world, our lifestyle, Our and our economy is dependent on automation and computerization, with these systems in turn dependent on other systems to work. And if one of those systems breaks, the effects ricochet outwards, like ripples when you cast a rock in a lake or throw a body in for some listeners. Friday's CrowdStrike cock-up is just the latest example of this, but it isn't the only one. Some of you might remember the SolarWinds hack back in 2020, when Russian state-linked hackers gained access to an estimated 18 ,000 companies and public sector organizations including NATO, the European Parliament, the US Treasury Department, and the UK's National Health Service by compromising just one service.

15:11So the winds are Ryan. Remember when Okta, some of you might know Okta, it's a company that makes software that handles authentication for a bunch of websites, governments, and businesses. Well, when they got hacked in 2023, they then lied about the scale of the breach. Hey, do you remember when those hackers leapfrogged from Okta to a bunch of other companies like Cloudflare. Yeah, they provide the content delivery services and the services that protect websites from being, well, brought down by a bunch of bots for much the entire internet. Everything feels like it's being held up by like twigs and chewing gum.

15:44So you probably know the quote, no man is an island. And it's especially true when we're talking about tech, because when you scratch beneath the surface, every system that looks like it's independent is actually heavily, heavily dependent on services and software provided by a very small number of companies, many of whom are not particularly good. And this is as much a cultural failing as it is a technological one. The result of a management culture geared towards value extraction, building systems that build monopolies by attaching themselves to other monopolies. CrowdStrike went public in 2019 and immediately popped on its first day of trading thanks to Wall Street's appreciation of them moving away from a focused approach to serving large enterprise clients, building products now for small and medium sized businesses by selling through channel partners.

16:28In effect, outsourcing both product sales and the relationship with the client that would tailor a business's solution to said client, especially when something is so serious like this. I want you to really think about this and think about this problem, because the problem isn't so much selling to small businesses or medium businesses. It's the fact that CrowdStrike made its money selling to the enterprise and specializing in that. And that's the thing. when you broaden out, when you must grow in all directions at all times, in all ways to please the horny beasts of Wall Street, you lose your focus.

17:03But that isn't the only problem, because CrowdStrike's culture appears to also fucking suck. A recent Glassdoor entry referred to CrowdStrike as great tech with terrible culture, with no work-life balance, with leadership that does not care about employee well-being. Another from June 2024 claimed that CrowdStrike was changing its culture for the street with KPIs, as in metrics related to your success at the company, driving behavior more than building relationships, with a serious lack of experience in the public sector and senior management. So glad that this company is selling into, like, government.

17:36Anyway, moving on. Others complained of micromanagement, with one claiming that management is the biggest issue, with managers asking way too much of you. And it doesn't matter if you do what they ask, since they're not even around to check on you. And another saying that management is arrogant and needed to stop lying to the market on product capability. That's what I love to see. We all love to see that. I'm very happy to read that. And while I can't say for sure, I'd imagine an organization with such powerful signs of growth at all costs thinking, a place where you, and I quote, have to get used to the pressure, that's a clique that you're not in, likely isn't giving its quality assurance teams the time and the space to make sure that there aren't any Kaiju-level security threats baked into an update.

18:23And that assumes it actually has a significant QA team in-house and hasn't just, as with many companies, outsourced the work to a body shop like YPro or Infosys or Tata Consultancy. But for a moment, I'm going to change gears a little to try and explain what actually happened and why it suggests that the issue is likely the product of cost-cutting an institutional failure within CrowdStrike. In the aftermath of Friday's incident, we've seen some analyses about what actually went down with them. First, some throat clearing. I haven't verified this stuff independently. From what I've read though, and from speaking to developers, this all seems relatively plausible, but maybe worth googling this a little yourself.

19:03But I'm gonna give it a go. So the kernel driver at fault was written with a programming language called C++. This language was developed in the 1980s and it's very good for writing high performance applications. Anything where you're concerned about speed, like the internals of an operating system or a video game. It's pretty popular for that. And it's also pretty dangerous too. So dangerous in fact that it's often referred to as an unsafe language. Without getting too into the weeds, C++ makes it incredibly easy to shoot yourself in the foot, the arse and the dick at the same time. It's big, complex, and has few safeguards, while providing many opportunities for developers to screw up very badly.

19:42Like the languages derived from C, it forces developers to deal with a lot of low-level stuff, like handling memory allocation, that you don't really have to deal with in many popular languages like Python, Java, Rust, Swift, or C-sharp. And this matters because if you screw this up, your code will break, or I don't know, it might introduce some kind of potentially disastrous security vulnerability. In 2019, Microsoft researchers said that 70 % of all security vulnerabilities were the result of memory management issues. And I doubt that figure has changed much since then. And earlier this year, the White House Office of the National Cyber Director urged developers to stop using unsafe languages like C and C++ and start using modern and safer alternatives like Rust.

20:27With me so far? Alright. So, from what I've read, the CrowdStrike Falcon sensor kernel driver crashed because it had something called a null pointer error. Essentially, the developer wrote some code that told the program to look for a memory location that didn't exist, and didn't write any safeguards to protect against it. When this happened, the driver, and so the operating system, crashed. This is a rookie mistake, and I've talked to multiple developers that have backed this up. If you take an introductory C++ programming class at university, they'll cover this in the first year. Kind of boggles the mind how trivial a mistake this is, and how it made it into production code, which is the code that goes out into the real world, and how it wasn't caught either by CrowdStrike or by Microsoft, who were supposedly obligated to vet this driver.

21:11And if the reports are true, someone really, really, really screwed up. Really badly. But if you don't want to screw up, if you want to really do well in life, I advise you to buy one of the following products or services, which I, of course, fully understand, know all about, and won't be embarrassed by.

21:37Parking shouldn't slow you down. ParkWiz gives every driver a shortcut. Book ahead, save up to 50 % and skip the hassle of circling the block. Park smarter, park faster. ParkWiz. Download the ParkWiz app today and save every time you park. This is Jason Timp from Hoops Tonight. Toyota is moving forward to a more sustainable future by giving you the freedom to choose from an incredible lineup of trucks. now available in both gas and hybrid models, including the legendary Tacoma, the powerful Tundra, both available with the iForce Max hybrid powertrain and backed by Toyota's legendary reputation for reliability.

22:15More choices, less compromise. Visit buyatoyota.com for a great deal on an efficient Toyota today. Toyota, let's go places. Wells Fargo has awarded$138 million in grants to nonprofits,

22:53And we're back. And to be clear, I don't want you to think that I'm letting Microsoft off the hook either. assuming the kernel driver testing roles are still being done in-house do you think that these testers who have likely seen their friends laid off at a time when microsoft was highly profitable and denied raises when their well-fed ceo probably took home over a hundred million dollars in salary for a job he's eminently bad at you think these people are doing their best work do you think they go into work jazzed full of piss and vinegar ready to save the world or do you think they hate their job and they're being forced to do too much and they're miserable and the people that knew what the fuck was going on haven't been fired and the people who managed those people and the people that wrote the code that they're edited you think anyone knows what the hell is going on no they don't and this is the culture that's poisoned almost the entirety of Silicon Valley.

23:49What we're seeing now is the societal cost of moving fast and breaking things, of people like Marc Andreessen considering risk management the enemy, of hiring and firing thousands of people, tens of thousands in some case, to please Wall Street, of seeking as many new possible ways to make as much money as possible to show shareholders that you'll grow, even if doing so means growing at a pace that makes it impossible to sustain organizational and cultural stability. When you aren't intentional in the people you hire and retain, the people you fire, the things that you build, the way that they're deployed, maintaining your systems, understanding how and why things were written, the decisions that were made 5, 10, and 15 years ago, you're going to lose the people to understand the problems they're solving and thus lack the organizational ability to understand the ways that problems might be solved in the future or disasters might be averted.

24:44This is dangerous, and it's also a dark warning for the future. Do you think that Facebook or Microsoft or Google, all of whom have laid off over 10 ,000 people in the last year, have done so in a conscientious way, in a knowledgeable way, a people-focused way, an organizationally rigorous way that means that the people are left who understand how their systems run and the inherent issues built into them? Do you think that management types obsessed with unsustainable AI bullshit are investing heavily in making sure that their organizations are rigorously protected against, say, one bad line of code or one dipshit error?