How To Break the Internet with Chris Stokel-Walker

0:00This is an iHeart Podcast.

0:30Thomson Reuters and specialized bikes have since they upgraded to the next generation of the cloud. Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high availability, consistently high performance environment, and spend less than you would with other clouds. How is it faster? OCI's block storage gives you more operations per second. Cheaper? OCI costs up to 50 % less for computing, 70 % less for storage, and 80 % less for networking. Better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds.

1:12This is the cloud built for AI and all your biggest workloads. Right now, with zero commitment, try OCI for free. Head to oracle.com slash strategic. That's oracle.com slash strategic. Wells Fargo has awarded$138 million in grants to nonprofits, supporting military and veterans with housing, small business, career transition, and more over the last 10 years. It's one of the many ways Wells Fargo seeks broad impact in communities. Wells Fargo, the bank of doing. Learn more at wellsfargo.com slash say do. Support includes contributions from Wells Fargo and Company and the Wells Fargo Foundation.

1:53So I love Square. They make paying for stuff at a store really easy. I just use them at Court Street Grocers to buy a sandwich here in Brooklyn, New York. Their stuff's reliable, easy to use, and it just works for both businesses and their customers. Square keeps up so you don't have to slow down. Get everything you need to run and grow your business without any long-term commitments. And why wait? Right now, you can get up to$200 off Square hardware at square.com slash go slash betteroffline. That's S-Q-U-A-R-E dot com slash G-O slash betteroffline. Run your business smarter with Square. Get started today.

2:26This week on a very special episode of Health Discovered, we're taking a closer look at a condition that affects hundreds of thousands of men each year, prostate cancer. I first found out about my cancer at the age of 45. Anything with cancer, you just think death sentence. In this episode, we'll explore the science behind detection along with the practical steps men can take to protect their health. Listen to Health Discovered on America's number one podcast network, iHeart. Open your free iHeart app, search Health Discovered, and start listening. Cool Zone Media. Hello and welcome to Better Offline.

3:08I'm your host, Ed Zitron.

3:18Better Offline. Now, you may have forgotten, but about a month or two ago, we had a huge, complete meltdown of the computer systems of the world when CrowdStrike failed. I did an episode on it. And well, we've all just kind of forgotten about it. And today I'm joined by Chris Stokel Walker, who's an author, journalist, lecturer, and starting new column at The Guardian as well, aren't you? Yeah, for a whole week, Ed, not a whole week, a whole month, I'm taking over tech space, which will be very exciting for the month of September. But the reason I brought you on is you wrote a great article for The Independent back at the end of July about how CrowdStrike isn't the only cyber company that could trigger a global meltdown the second they fail.

4:03And this is a subject that fascinates me because I love disaster movies, and also this article was terrifying. So why don't you walk me through it? yeah so basically back in july we had that odd outage that people might remember uh basically people woke up in asia australia and then eventually the uk uh and encountered blue screens of death on microsoft windows which is something amazing that you know i'm 35 now and i remember that happening when i was a kid and then it's never happened since like for all that you people make fun of Microsoft. Actually, their PCs are decent. But then suddenly everything went to hell in the handbasket.

4:45So it turns out that CrowdStrike, which is one of the big service providers for kind of antivirus tools and software, had misconfigured basically the thing that protects us, so that it actually harmed us, which is just the ultimate in fantastic arities. And it turned out that it was actually, within CrowdStrike, Like the thing that was bugged failed because the bug checking thing had a bug in it, which is so good. It's so good that we have everything built on. Yeah. And this is the thing, right? And I guess this is something that a lot of your listeners and others will chime with because you share a similar sensibility to me, which is that we have built a huge thing, which is kind of like on a house of cards that is actually hiding the fact that we have built a huge thing.

5:36that humans are involved in this and humans screw up frequently. Yet we think that actually, because this is whiz-bang tech, you don't have to worry about it. It will also sell up. The glory of automation. Yeah, precisely. And yet it's not. It's some guy who's overworked, overtired, and can't type properly and has fat fingers like you and me. Yeah, and a combination of private software companies like CrowdStrike and then, as you'll get into, open source solutions that are a lot of people doing them for the love of the game, which is wonderful but at the same time we've got this patchwork system that holds up the internet and a lot of the tech we rely on and we don't really know and the fact that so much of it is automated is also terrifying yeah and this is the thing that that update was sent out to millions of pcs over the course of an evening uh while people were sleeping and people didn't realize that there was a massive issue with it until they started to wake up and at that point it is essentially too late and the best part of this story i think is that to fix the issue and we have kind of fixed the issue a couple of months on for a lot of them you had to actually get individual people to go out to either a computer or a terminal somewhere and put in the actual update to unscrew up the problem that had existed initially but you highlighted there ed to kind of the issue of how much of this uh is seen as kind of public utilities nowadays like the plumbing of our our kind of entire world and yet it is kind of really rickety and held together with scotch tape and a little bit of chewing gum i did a thing a story a decade ago about how we had a similar issue with a thing called heart bleed which was another code what happened there so basically it was it was another similar thing where um there was an update to a thing called open ssl which is the the software tool that encrypts all the data that is sent through uh payment systems and passwords and so on and so forth.

7:32So whenever you see that kind of yellow lock on your web browser, that is usually running OpenSSL. And there was an issue with it, which essentially meant that all of the information that people thought was being shared in encrypted form was actually being shown in plain text. So someone could, in theory, come along and snoop on everything that you're putting in there from bank account details to passwords and so on and so forth. The reason why that happened is essentially the internet at the time and to a certain extent now or less so was being run by two guys named steve the whole thing was developed by a guy called steve marquez and his friend who was also called steve they were kind of this weird transatlantic bromance where they kept going open ssl it was this initial volunteer run project that kind of became a key part of the internet and that frankly these multi-billion dollar companies use day in day out but they decided that they didn't really want to pay a penny for the upkeep.

8:30And OpenSSL is one of the named things that could break the entire internet from your article as well. Yeah, this is the key part of the internet's plumbing. And there are loads of these, right? This is the thing that we don't realize. Until things go wrong, and generally in IT, people want to have 99.999 % uptime until you have that kind of miniature final element where actually something does screw up and actually you start to recognize that this thing is held together pretty precariously. We don't realize it, but there is a sort of cabal of a half a dozen or so companies whose job is essentially to keep this stuff running.

9:13And sometimes they do screw up. and open ssl how was that actually funded is it donations or yeah so initially at the time back in 2014 when heartbleed was heartbleed was uh kind of an issue they were essentially relying on donations uh this was open source software and that is kind of the basic principle of the web we often forget about this actually it is hobbyists that set this thing up alongside a huge military industrial complex the xkcd comic which is uh everything's held up by a guy called runk yeah Yeah, yeah. Turns out it was actually Steve, but basically the same sort of thing. And that is the problem, is they weren't fully funded.

9:52I did a follow-up story back in 2014 because Heartbleed kind of drew the attention to this. And it comes back to that idea of how you started our podcast here, where you said, this thing happened, this huge chaotic thing, and you've probably forgotten about it because it's been two months and we moved on. Same thing with Heartbleed. This happened. There was this huge outcry. they got a bit of funding around about a million dollars that was meant to kind of make them back on an even keel and so that they could be sustainable reality is kind of didn't work what do you mean it didn't work well in the sense of they still had this issue happen again and again so what was meant to be kind of a fronting up i suppose of big tech companies saying actually you know what we recognize that this hobbyist service is a vital part of our internet's running, we will fund it so it is sustainable, didn't necessarily happen.

10:48Not to the extent that we had another heart lead from OpenSSL, but they do struggle still to kind of keep things going. And that is, I think, the big problem here, which is the news agenda moves on, people forget very quickly. And because there is then not a problem for a little while longer, we kind of lurch from one catastrophic near-miss to another. So OpenSSL, what does it actually do though? I know it's the little padlock on browsers, but what is its foundational point? Yeah, so basically it shepherds across data from a user to kind of a service provider. So if you think about it as kind of you input text on your laptop your phone wherever you are it will then encrypt it it will transfer it over to a payment provider to uh your bank to frankly also pretty much anywhere that you put a password in and it will ensure that that is encrypted all the way but there was an issue with the coding of it which meant that actually um again comes back to fat fingers uh some elements of what was being transferred went into kind of excess memory, which basically meant that bits of it were encrypted, but then large parts of it weren't.

12:17So if you were unlucky, then the bits that weren't encrypted could be your credit card details, your sort code, and your account number, and the CBC number. And that's why there was kind of this big red flashing light back in a decade or so ago where people thought, you know what, actually, this exploit, if left unchecked, could become a massive issue and a real boon for cybercriminals. I think actually, had it happened in 2024, we would have seen much more of a significant issue in terms of second, third, or more to ramifications, because cybercriminals would have been all over that stuff. And if OpenSSL breaks again, that just means that the internet is not really encrypted, that every place that uses it is kind of at risk?

13:05Every transaction on every place? yeah and that is that is the issue that is the high wire act that is the internet and you know we've seen outages like this come and go time and time again we you know we've had uh the crowd strike incident where you couldn't actually get onto your your desktops or your laptops we've had outages on major payment platforms we've had banks going offline we've had social networks kind of disappearing for hours at a time. And invariably, this is just a very simple error that snowballs repeatedly, and we're kind of doomed to repeat it. And I guess the challenge is like, how do we put the web on a firmer footing that prevents this from happening again and again?

13:53It feels like funding the OpenSSL Foundation would probably be a good start. But But I imagine that's not happening. We need to build the computer that makes pictures of Garfield with a gun. Exactly. This is the thing. Commercial interests always come into this. And the reality is, as you and I both know, and as many of our listeners will know, companies, tech companies in particular, will take action when they realize that the spotlight is on them and that there is this kind of intrinsic demand for them to do that. As soon as it's off, they're back to doing the same old thing. They are happy with the status quo as it is.

14:29Which is crazy as well, because what worries me about OpenSSL is that nobody will really be to blame, and thus nobody will really feel responsible. They might kick in some money here and there, Google especially, very reliant on them, but I just don't see them doing it. yeah and this is the thing that the only people that i really noticed when i was reporting out that story a decade ago who felt any kind of like guilt or personal alarm or kind of just like even responsibility for it were actually those two steves like they were they were gutted the story i did for you yeah the story i did for buzzfeed back then was um it was very difficult to report out because they had been taken out of like not to overly stereotype but they were like super techie nerdy people like they right they were very happy being in the background tinkering with this thing they realized that it was important and they took their jobs very seriously but they had never been put in the spotlight and and they were initially very wary was speaking to me because suddenly this thing happened and it's over the course of like hours that they got kind of thrust into the limelight they had the daily mail knocking at their door which was one of the reasons why they were the super wary of talking to me so it took actually a few days of winning them over and saying you know what like this isn't going to be a hit job like i'm not looking to kind of you know hold you up and say this is the person responsible for this happening.

16:07It's more I wanted to tell the story of why this has happened, why it's an issue, and why we shouldn't have the ability for kind of slight errors in upkeep to cause catastrophic effects.

16:29Parking shouldn't slow you down. ParkWiz gives every driver a shortcut. Book ahead, save up to 50 % and skip the hassle of circling the block. Park smarter, park faster, ParkWiz. Download the ParkWiz app today and save every time you park. So I've shopped with quints before they were an advertiser and after they became one. And then again, before I had to record this ad, I really like them. My green over shirt in particular looks great. I use it like a jacket. It's breathable and comfortable and hangs in my body nicely. I get a lot of compliments. I liked it so much. I got it in all the different colors, along with one of their corduroy ones, which I think I pull off, and really that's the only person that matters.

17:05I also really love their linen shirts too. They're comfortable, they're breathable, and they look nice. Get a lot of compliments there too. I have a few of them. Love their rust-colored ones as well. And in general, I really like quints. The shirts fit nicely and the rest of their clothes do too. They ship quickly, they look good, they're high quality, and they partner directly with ethical factories and skip the middlemen. So you get top-tier fabrics and craftsmanship at half the price of similar brands. And I'm probably going to buy more from them very, very soon. Keep it classic and cool this fall with long-lasting staples from Quince.

17:35Go to quince.com slash better for free shipping on your order and 365-day returns. That's Q-U-I-N-C-E dot com slash better. Free shipping and 365-day returns. Quince.com slash better. This week on a very special episode of Health Discovered, We're taking a closer look at a condition that affects hundreds of thousands of men each year, prostate cancer. I first found out about my cancer on my birthday at the age of 45. Found out my cancer has spread to my pelvic bone. And from there, life just changed. About one in eight men will be diagnosed with prostate cancer during their lifetime, and the risk increases with age.

18:23Anything with cancer, you just think death sentence. And the only thing I could think about was, who's going to take care of my family? You have to go out there and build your support system. You got to build your team. In this episode, we'll explore the science behind detection, along with the practical steps men can take to protect their health. Listen to Health Discovered on America's number one podcast network, iHeart. Open your free iHeart app, search Health Discovered, and start listening.

19:18Upfront payment of$45 per three-month plan,$15 per month equivalent required. New customer offer first three months only. Then full price plan options available. Taxes and fees extra. See MintMobile.com. This is Jacob Goldstein from What's Your Problem? Business software is expensive. And when you buy software from lots of different companies, it's not only expensive, it gets confusing, slow to use, hard to integrate. Odoo solves that because all Odoo software is connected on a single affordable platform. Save money without missing out on the features you need. hidden costs and no limit on features or data.

19:51Odoo has over 60 apps available for any needs your business might have, all at no additional charge. Everything from websites to sales to inventory to accounting, all linked and talking to each other. Check out Odoo at odoo.com. That's odoo.com.

20:13it almost feels like people people getting mad at the homeless to some extent where it's like oh this person is on the street and there there are problems that are happening around them and they they are ups and you blame the person who is the victim here you blame the fact that when you look at the internet right now and its instability you're like oh well these open source people are doing it for free it's their fault because they should have fucking they should have been better at doing this thing for free that holds up the entire internet versus the fact that the problem is that the entire internet relies on this underfunded group of people.

20:49And it really is, I just want to be clear for listeners, and Chris, you of course know this, when I say this holds up the entire internet, I do actually mean that. It's very easy to fall foul to hyperbole sometimes, but this is genuinely that level. When Heartbleed happened, it sounds like it could have been truly catastrophic. it could it really could and this is the thing that we overlook is either there are kind of um not amateurs because these people are super professional in terms of what they do and they take their jobs very seriously but they are either not paid or they're paid a pittance particularly in comparison to the total compensation packages that you see watching around silicon valley and And yet there is this kind of super extractive approach from big tech companies of we will kind of roll in whatever it is that you provided us, often open source.

21:43And this is the kind of big secret, right, of a large part of big tech's success is they rely on these open source developments that have kind of underpinned key parts of their tech. And if things go wrong, they can always shift the blame onto those open source things and say, well, this isn't actually our fault. This is the fault of our supplier, the thing that came second or third order down the line. And you've got this big movement in cloud as well towards like composable architecture, which involves a lot of slotting in open source solutions as well. It's just, it almost feels like we need a big tech mutual aid thing for open source.

22:24I wish that I don't think there's any way we could get a government to do this but I think they should force big tech to put like a percentage of revenues not profits into open source and have very defined layers for them because otherwise you get situations where I don't know the entire internet's underpinned by two steves yeah exactly and I think this is this is the unfortunate thing is that should be the lesson that we learned it should have been a lesson that we learned from Heartbleed. It should have been the lesson that we learned from XZ Utils, which was another issue that we encountered relatively recently where there was, it turned out, we believe, a bad actor kind of deliberately inserting malicious code into another thing that kind of underpins large parts of our digital lives.

23:11Turns out that the volunteers that were running that couldn't keep track of it. One of them literally talked about their burnout and how they'd kind of taken a step away from the project and yet we always see these things kind of passing by in the rearview window we say oh you know what isn't that such a shame we ought to do something about that and then we move on to the next thing and we don't pay attention taking a step back what was xz utils for the american listeners xz for british and canadian um what happened there yeah So this is kind of, again, another bit of free software. This was back in sort of spring of 2024.

23:53A kind of malicious hacker had, we think, basically socially engineered their way into the upkeep of this bit of open source utility, which is essentially designed to kind of compress data. So the idea was that it would kind of take a big file, chunk it up, make it smaller, get rid of the bits that you don't need. And it was kind of, in the same way as you talked about cloud architecture, slotting in lots of really useful open source tools, this is a similar thing where you could slot in exeutils, exeutils into whatever you were building and it would be fantastic. This guy had kind of offered to volunteer here at a time when the original developers, the custodians of this tool were feeling very burned out, said, I will help.

24:49The original person then took their eyes off the ball. This malicious actor started putting in backdoors intentionally, their ways of accessing kind of the data within. And it was only spotted basically by a Microsoft developer who happened to come across this um and just to be clear though xed utils looks like it's a big part of linux which is a which people who use consumer software may not realize is basically underpinning most server architect like a ton of server architecture a ton of web architecture right yeah so servers web servers uh cloud hosting tools uh lovely webcams basically anything that's connected probably your fridge if you have a an internet of things fridge if you are that frivolous then that will be connected in some way to linux i hope not i hope that your diet coke is not being kept cold by an iot fridge and there's someone who knows me so but so this microsoft developer found it and so it turns out that just the corruption of open source like it can happen as well with these open source projects particularly i imagine when they're underfunded and the people get burned out.

25:58Yeah, and this is the thing. Again, it's another example of a kind of hobby project that turned into something bigger. Nobody who has the money either realized or decided that it was important enough to fund directly. The people involved who are often, again, I don't want to stereotype and I don't want to kind of make them seem too much like a victim here, But these folks are often super humble, super helpful, just trying to keep their heads above water, essentially, because they've made something that has proved very, very useful, and they don't want to trouble people by shouting for help. in this case same thing happened single person in charge of this tool didn't want to shout too loudly about the problems that it would cause him in his life he kind of took a step away decided to get help from someone turned out they were bad and nobody decided to shout about it so again this one was this one was more deliberate rather than a fuck up i would also say they are victims these people are doing some of the most important work in the world while sundar peshai gets 200 million dollars a year it that i see these people as heroes and victims at the same time yeah and i agree i think i think what i mean by that is they would not want to be seen as either the hero or the victim in the piece.

27:27They don't, they don't, they have a, I think to be involved in open source software more generally, you have to have a very kind of, it's almost quaint, right? You have to be very, very community minded, very kind of, I am doing this for the good of everybody, but I also don't want the praise for it. and so either kind of being presented as like this hero defending us against all of the bad stuff or being presented as the victim who is you need pity i think that's the thing they don't want they don't want right but they just want money and i think we get back to the systemic problem then because i don't see them as like any kind of pathetic thing or indeed i mean that i think there's something heroic about what they're doing but i think what they're doing is cool I think what the problem is the systemic lack of support for them.

28:23We blame these things, we don't, but people may blame these projects for breaking. Oh, it didn't work as well as it should. But it turns out that it's just, we put all of this pressure and these requirements on these people and on these projects and then don't give them the support at all. So naturally, I'm going to say to my listeners, please go and fund your open source movements. brought on molly with wikipedia molly molly white of course about wikipedia very early on in the show fund these projects because they deserve it but the funny thing is is the some of the sometimes i've seen very stupid idiots say things like well if they were fun if they were private and corporate entities they'd be fine and nothing would go wrong except we look at crowd strike and it's the complete opposite yeah and the worst thing is with crowd strike is it's they thought they knew better right like this is this is the key thing they thought that they were doing everything perfectly they kind of crowed about how good their tools were how well they could protect people and then they didn't and then and this is kind of the endemic thing is you you can't introduce profit to the equation because if you do, you're looking to cut corners.

29:43Now, we still don't really fully know why this happened. CrowdStrike is still taking a hit to its business. It still has the threat of legal action from those customers who were affected and there were an awful lot. Airlines were knocked off for basically an entire week in the United States. you couldn't get anywhere on some airlines because their systems were just so completely broken. But the idea that you can just kind of throw money at it through a big tech lens doesn't really work because you need that idea, that kind of ethos of I'm doing this not for profit, not for myself and not for the company that I work for, but I'm doing it for kind of the greater are good.

30:30And I think the problem, if you brought this into a Google or even a CrowdStrike or whatever, is that you end up looking at the bottom line and realizing, actually, I need to acquire customers. I need to keep them. That's going to be my focus, not just making good stuff and making it work. So onto profit-seeking entities, one of the others you mentioned is Fastly. So Fastly, why don't you walk me through Fastly? Because I know there are other companies in this realm too. Yeah, so Fastly is kind of what you would call an edge cloud provider. So that is basically an attempt to try and bring the internet speeds up a little bit, make them a bit quicker.

31:14So it's the idea of bringing files that are commonly used or websites that are commonly used closer to where the users want to request them. The thing that people often overlook is that the web is essentially still a data transmission system. And so you have to, if I was to pull up a YouTube video from my home in the UK, it would be very silly for me to put that request through YouTube servers in the United States. Because I would have to send the request to the United States. The request would have to be fulfilled. YouTube would have to go looking for the video. It would then have to send the video back to me and then it would have to be played.

31:53Now, we're really talking about kind of a fraction of a second there, but it can be done quicker by serving it closer to me physically. By a content delivery network like Fastly. Exactly. So that is what Fastly does. The problem is that it went wrong around about three years ago. Again, like CrowdStrike, a misconfigured file got pushed out of the company's systems because Fastly is used by Amazon, by Reddit, by Twitch, by the UK government, by PayPal. All of those platforms were affected, which is kind of a big issue. This guy who runs Fastly, hugely wealthy man, and has done an awful lot of good, makes the internet faster, but the problem is, again, it is a private company, It is a single point of failure for many, many platforms and many websites that we use day in, day out.