CrowdStrike: All Systems Down | Digital Dominos | 2

0:11It's early morning on July 19th, 2024, in London. Inside a hospital room, a nurse preps an elderly man for a heart procedure. Before they begin, she wants to take one last look at his medical history, so she grabs the Microsoft Surface tablet off the mobile cart near his bed. But when she taps on the tablet's screen, it turns solid blue and shows an error message. It looks like the dreaded blue screen of death, but she's never seen it on one of the hospital's tablets before. The nurse exhales, frustrated. and heads to the reception desk to see if she can find a functioning tablet. But when she reaches the hallway, she stops short.

0:53It's total chaos. A dozen nurses are hurrying back and forth, many of them carrying their own malfunctioning tablets. The nurse walks up to the reception desk to ask for help. Hey, could you pull up a patient's records for me? My tablet's just gone haywire. Everybody's been asking me the same thing, but my computer has the same blue screen as yours. Oh, did you call IT? I've been on hold with them for 10 minutes. I think the whole network is down, and I just heard from a friend at another hospital. They're having the same exact problem. Oh, no. Do you think it's a cyber attack? I hope not. They're already talking about postponing this morning's surgeries until they can figure out what's going on.

1:37I'm worried for some of the patients. They can't afford to wait. What the nurse doesn't know yet is that this isn't a cyber attack. It's something else entirely. And a company called CrowdStrike is the source. CrowdStrike makes the cybersecurity software that protects this hospital's Microsoft-based systems. And when the company pushed out their latest update overnight, there was a catastrophic error. In the hours that follow, the damage will spread across the globe. It will affect airports, hospitals, banks, emergency services, and more. And CrowdStrike will come under intense scrutiny as people everywhere ask the same question.

2:22How could a single error by a single company bring so much of the world to a standstill?

2:41From Wondery, I'm David Brown, and this is Business Wars.

2:52CrowdStrike

3:08CrowdStrike was founded in 2011 by cybersecurity analysts George Kurtz and Dmitry Alperovitch. Their belief was simple. Modern threats required modern defenses. And this meant moving security to the cloud. The company gained early notoriety by investigating several high-profile hacks, including North Korea's attack on Sony Pictures in 2014. But when CrowdStrike was hired by the Democratic National Committee ahead of the 2016 election, it became a political lightning rod targeted by President Trump and a flood of conspiracy theorists. Still, the business kept growing. In 2019, CrowdStrike went public with a wildly successful IPO, nearly doubling its valuation overnight.

3:55And by June of 2024, they had earned a place in the S &P 500. But its most defining moment would come just a month later, when a routine software update triggered what would become the largest IT outage in history. This is Episode 2, Digital Dominoes.

4:24Let's take a moment to talk about the product that took CrowdStrike from startup to multi-billion dollar business. It's a platform called Falcon. Unlike traditional cybersecurity tools, Falcon is entirely cloud-based. It runs on CrowdStrike servers, not on the machines it protects. Once it's installed, Falcon is given a very high level of access, enough to monitor a client's systems in real time, detect malware, stop intrusions by hackers, and neutralize threats before they spread. As part of this monitoring, CrowdStrike provides automatic updates to Falcon as many as a dozen times a day. For customers, this is a huge advantage.

5:06They save time and money that would otherwise have been spent on manual updates. By the mid-2020s, more than half of Fortune 500 companies relied on Falcon, along with countless governments, hospitals, airlines, and transit systems. By the way, what we're talking about here is a bit like the holy grail of enterprise software, the sort of thing that only companies like Microsoft thought they could do. Becoming so embedded that getting rid of you at some point is almost unthinkable, but this gets at something larger that CrowdStrike is doing here. They're not just selling cybersecurity. It's more like peace of mind at scale.

5:49But the thing is, this scale tilts both ways. When you're everywhere, even a tiny mistake doesn't stay tiny for long. When you combine Falcon's deep system access with its continuous flow of automatic updates and global reach, it's not a question of if something goes wrong. It's when. And on the night of July 19th, 2024, one of Falcon's updates contains a logic error. This error is pushed to millions of Microsoft Windows-based systems around the world. Within minutes, these systems begin to crash. It is a massive computer outage affecting all corners of the globe. But it wasn't just airlines and airports.

6:35Hospitals were also knocked offline. Nurses switching to paper orders. Many hospitals forced to cancel procedures. Also affected banks, 911 systems, trains, buses, and subways, retailers, FedEx and UPS, court systems and driver's license offices. The list of affected systems is staggering. It also includes TV stations in Europe and the U.S., including Sky News, MTV and ESPN, and even the Paras-Olympics organizing committee just days before the opening ceremony. At first, many assume the worst, a massive cyber attack. Within 90 minutes, CrowdStrike identifies the error and pushes out a fix, reverting the changes that caused the problem.

7:24But for millions of systems, it's too late. They've already crashed and are now unable to connect to the cloud to receive the new update. These systems will require someone to manually reboot them and delete the faulty file. It's impossible to say how long it will take for this to happen. And to make matters worse, it's now the middle of the night in the U.S., so most IT workers are offline.

7:54As CrowdStrike scrambles to help its clients implement the new update, CEO George Kurtz goes public. At 2.45 a.m. on the West Coast, he posts a message on X that reads, quote,

8:20Now,

8:26technically, this statement is correct. But it's missing something. There's no apology, no acknowledgement of the harm done, and no empathy for the millions of affected people. The Post is widely criticized. Well, there's certainly a lesson here for anyone who runs a business. It's about where technical truth and human truth collide. Kurtz's message was accurate. There was nothing wrong with his words. What was wrong was what wasn't said. In a crisis, people don't just want to know what happened. They want to know that you understand what your problem costs them, the empathy factor. When empathy is missing, even the best explanation sounds like deflection.

9:16It's now around 3 a.m. in Portland, Oregon. Mayor Ted Wheeler is huddling with Shad Ahmed, the director of Portland's Bureau of Emergency Management. Many of Portland's most critical systems run on Microsoft software that uses CrowdStrike's Falcon. Mayor Wheeler is growing increasingly anxious as both men receive a stream of emails and texts with updates on the damage. Okay, Shat, what are you seeing? Good news and bad news. CrowdStrike just tweeted that it's not a cyber attack. Okay, well, that's a relief. What's the bad news? Ahmed shifts in his seat, his expression darkening. Well, to start, Portland International Airport's going to cancel flights.

10:00It's 3 a.m., so hopefully that limits the impact. Well, that's assuming this gets fixed quickly. Our IT people are trying to reach CrowdStrike, but they've been on hold for a while. Wheeler's phone buzzes on the table between them. He glances down. It's a text from the head of TriMet, the city's public transportation system. His jaw tightens. Yeah, that was TriMet. They're unable to display arrival times or service alerts, and rush hour starts in just a few hours here. And I just checked with Providence Hospital. They can't access patients' digital records. They may have to cancel surgeries. Wheeler stands abruptly and starts pacing.

10:40My God, this is such a mess. It might as well be a cyber attack. The results would be the same. So what's your ballpark estimate for when this will all get fixed? Well, honestly, it could take days. City government has nearly 500 different systems, and I guess half are affected. They all have to be manually rebooted in safe mode and then put in whatever fix CrowdStrike is offering. Wheeler stops pacing and looks back at Ahmed, his face pale in the fluorescent light. What a nightmare. I'm considering declaring a state of emergency. What do you think? Oh, that's a big step. and we risk causing panic that only makes things worse.

11:23Just then, Wheeler gets another alert on his phone. Well, that's 911. Their call tracking system's down. All right, that settles it. Grab me the emergency declaration template. I'm going to issue it now. As Wheeler signs the order, similar conversations are unfolding in cities across the U.S. and far beyond it. Airport directors are grounding planes, hospital administrators are canceling procedures, and transit officials are warning of shutdowns. All because of the same blue screen. And in each place, the same realization is setting in. This isn't a quick reset. Every affected computer has to be fixed by hand.

12:10Many by people who are still asleep or already overwhelmed. With every passing minute, the ripple effects grow. What began as a software update is now a global logistics problem, a public safety problem, an economic problem. But for CrowdStrike, it's something bigger. The company built its reputation on stopping catastrophes before they start. Now, it's at the very center of one. And as the sun rises across Europe and prepares to rise over North America, the fallout is only just beginning.

13:13On the morning of July 19, 2024, the U.S. wakes up and learns that countless critical systems have crashed overnight, and that CrowdStrike is the source. When the markets open, CrowdStrike's stock drops sharply, declining by as much as 13 % during the trading day. Online, anger explodes. CEO George Kurtz becomes a punching bag, taking heat not just for the massive outage, but also for his initial response, which did not include an apology. Around 9 a.m., Elon Musk weighs in. He tweets that the outage has seized up parts of the automotive supply chain and that Tesla has removed CrowdStrike software from all of its servers.

14:00The question now being asked everywhere is the same. How can a single software update bring so much of the world grinding to a halt? Critics argue CrowdStrike should never have pushed their update to millions of machines all at once. A staged rollout might have caught the problem early and spared the damage. Others go further, questioning whether it was a good idea to put so much of the world's digital infrastructure in the hands of a single company. And CrowdStrike's old political baggage comes back. For years, some supporters of former President Donald Trump have claimed, without evidence, that CrowdStrike helped falsely blame Russia for the 2016 DNC hack.

14:46And now, new conspiracy theories begin popping up online. Just one day earlier, Trump officially accepted the GOP nomination for president at the Republican National Convention. Some supporters suggest that the timing of the outage isn't a coincidence. While the conspiracy theories churn, CrowdStrike focuses on helping clients install the new update. They're able to get many systems back online within a few hours. But for others, the damage will take days or longer to unwind.

15:25It's Friday morning, July 19th, about 12 hours since the first outages began. Inside Atlanta's Hartsfield-Jackson International Airport, it's pure misery for both travelers and airport employees. A Delta Airlines ticketing agent is standing behind the check-in counter. She sighs as she looks out over the line of customers queued up in front of her. The line stretches endlessly, snaking through the terminal and spilling out toward the curb. And the customers waiting in it do not look happy. The good news is that her computer seems to be running again. The blue screen of death is finally gone. But the bad news is that her monitor now shows hundreds of canceled flights and hundreds more delayed.

16:14Clearing the backlog of flights could take days. And meanwhile, more travelers keep arriving. The agent exhales and reminds herself that all she can do is take it one customer at a time. She calls for the next traveler in line to step forward. I can take who's next. A 30-something-year-old woman steps forward, holding a baby to her chest while a toddler clings to her leg. Hi. The Delta app shows that my flight to Cincinnati's been canceled. Is there another flight I can get on today? I need to get home. Let me take a look. I'm really sorry, but there's nothing today. The next available flight is Sunday at noon.

16:59Sunday? That's two days from now. I wish there was something sooner, but we're working through a huge backlog of cancelled flights. Honestly, your best option might be renting a car. I've already been to the rental place. It's no better than here. Just long lines, and they don't even have any cars. I understand. I can book you on Sunday's flight, and you're welcome to stay in the airport until then. That's the best I can do right now. I'm sorry. You're telling me to sleep here with a baby and a two-year-old? The agent doesn't know what to say. There's no good solution. And there are hundreds of people just like this customer waiting in line.

17:44It's going to be a long day. Delta later estimates that the outage costs it over$500 million in lost revenue. And they are far from the only airline to face disruptions. Flights are canceled or delayed in cities across the world, from London to Hong Kong to Mexico City. In total, it's estimated that more than 16 ,000 flights are canceled worldwide. As CrowdStrike's engineers work overtime to help their clients recover, the company's stock price continues to fall. And as the public face of the company, CEO George Kurtz remains under intense scrutiny. That morning, he appears on NBC's The Today Show via remote video.

18:29On screen, Kurtz looks defeated and exhausted, with noticeable bags under his eyes, like a man who hasn't slept all night. Anchor Hoda Kotb asks him what happened. I want to start with saying we're deeply sorry for the impact that we've caused to customers, to travelers, to anyone affected by this, including our companies. So we know what the issue is. We're resolving and have resolved the issue now. The system was sent an update, and that update had a software bug in it and caused an issue with the Microsoft operating system. And we identified this very quickly and remediated the issue. And as systems come back online, as they're rebooted, they're coming up and they're working.

19:14and now we are working with each and every customer to make sure that we can bring them back online. But that was the extent of an issue, the issue in terms of a bug that was related to our update. Then co-host Savannah Guthrie gets right to the point and asks him how one software bug can shut down so many systems across the globe. Kurtz begins to look even more tired and rattled, and his voice starts to fail him. Well, when you look at the complexity of cybersecurity, you're always trying to stay one step ahead of the adversaries. Excuse me. Just one second, please. Kurtz takes a long drink from a bottle of water.

19:56Oh, yeah. Take a drink of water. Yeah, sorry. Sure, it's been a long night. It's been a long night. We're always trying to stay one step ahead of the adversaries. Despite his shaky performance, Kurtz continues making the media rounds. He also posts several more times on X, providing updates on the progress CrowdStrike is making in getting systems back up and running. But by now, something has shifted. This is no longer just a bad day for CrowdStrike. The outage has exposed just how fragile the systems underneath modern life really are. And it's raised uncomfortable questions, not just about a single update or a single company, but about the risks we accept when everything everywhere depends on software working as it should.

20:45But while systems around the world are slowly coming back online, the reckoning for CrowdStrike is just beginning.

21:13In the days that follow, the damage keeps mounting. CrowdStrike's stock price continues to slide. When the markets open on Monday, July 22nd, the next trading day, shares fall by another 13%. And less than two weeks later, the stock is down more than 30%. For many customers, the disruption doesn't end quickly. Because the error was caught by CrowdStrike but crashed Microsoft Windows machines, the two companies have to work together to help clients recover. And fixing the problem isn't easy. Each affected computer has to be booted into safe mode. Then, technicians must manually locate and delete the faulty CrowdStrike file.

21:56For organizations with thousands or tens of thousands of machines, This is a lengthy, labor-intensive task that can take days or even weeks to complete. And as teams scramble to repair their systems, opportunists move in. Hackers take advantage of the confusion, posing as CrowdStrike employees in phishing emails and phone calls, trying to gain access to various systems. Others create hundreds of phony websites with CrowdStrike in the domain name, hoping to trick desperate users into handing over access. It's an ironic twist. A company built to stop hackers has, unintentionally, created a new wave of scammers.

22:40In response, CrowdStrike warns its customers to be cautious, urging them to verify every request and deal only with confirmed CrowdStrike employees. By July 29th, 10 days after the initial crash, CrowdStrike reports that 99 % of users are back online, a full recovery they admit could take months for some companies. But even after systems stabilize, the scrutiny doesn't fade. It intensifies.

23:16It's September 24th, 2024, two months after the outage. On Capitol Hill, the House Homeland Security Subcommittee on Cybersecurity and Infrastructure comes to order.

23:30CrowdStrike has been called to testify. The company sends Adam Myers, their senior vice president of counter-adversary operations, as a representative. Congressman Mark Green opens the hearing by laying out what's at stake. On July 19th, Americans woke up to a shock. Their flight home, grounded. Their scheduled medical procedure, canceled. Their call to 911 wouldn't go through. Green goes on to emphasize the scale and reach of this failure. A global IT outage that impacts every sector of the economy is a catastrophe that we would expect to see in a movie. It's something that we would expect to be carefully executed by malicious and sophisticated nation-state actors.

24:21But this wasn't an attack. It was a mistake. A single software update that slipped past CrowdStrike's internal checks. Green continues. Mistakes happen. However, we cannot allow a mistake of this magnitude to happen again. As the July 19th outage has demonstrated yet again, our networks are increasingly interconnected. Our nation's security depends on a strong public-private partnership for protecting our networks. Ensuring our partnership is strong is important because our adversaries always watch how we respond to these type incidences, just like the July 19th outage. And you can bet they're watching us right now.

25:03Soon after, Myers makes his own opening statement, in which he explains what happened and apologizes for the disaster. But he's not out of the hot seat yet. Members of Congress grill him with questions, starting with Representative Mark Green, who asks Myers, who was responsible for the update, AI or a human being? Thank you for your question and your comments. AI was not responsible for making any decision in that process. It is part of a standard process. We release 10 to 12 of these updates, content updates, every single day. And so that was part of our standard operating procedure. Then, Representative Green gets to one of the core criticisms of CrowdStrike in the wake of the outage.

25:49The faulty update was pushed to millions of machines simultaneously around the world, magnifying the impact of the disaster. And these updates are automatic globally. They go global all at once when you send an update out. Myers pauses and stumbles a bit before answering. The updates were distributed to all customers in one session. We've since revised that. In the full testimony, I've included a graphic that depicts what that now looks like, and that is no longer the case. Okay, so your CrowdStrike's no longer fielding your updates like that simultaneous, universally, if I understood the answer to your question.