What Caused A Third of the Internet to Go Down - DTNS 5128

0:03Can you hear us? This is the Daily Tech News for Monday, October 20th, 2025. We try to tell you what you need to know, follow up on context. and today we're really going to try to help understand. That's right, Tom, because today Amazon once again took out a chunk of the internet, and what we mean by chunk is all the stuff people like to use early in the morning. Watch how you drive that thing, AWS. I'm Tom Merritt. And I'm Rob Dunwood. Let's start with what you need to know with that big story. Now, we do not cover internet outages, I say, every time we cover an internet outage, because a lot of times there's internet outages that if they affect you, they feel huge, but they're gone by the time we actually talk about them.

0:43And so it's talking about something, it's like talking about your dreams, you know? People who are unaffected are like, nah, I don't know, why do I care? However, if they're extremely big or extremely interesting, then we will talk about them. And we definitely have the first one with the AWS outage that took out large parts of the internet early Monday morning. Starting at 3.11 a.m. Eastern time, Amazon Web Services reported increased error rates and latencies for multiple services in, for those who follow this sort of thing, the very familiar U.S. East 1 region in Northern Virginia. I suspected something was up because our echo couldn't respond to turn off a light at about that time last night.

1:27By 5.01 a.m. Eastern, Amazon, I'm just going to give all the times in Eastern, by the way, by 5.01 a.m. Eastern, and Amazon identified the problem was related to DNS resolution of the DynamDB API endpoint. That's a database. And to oversimplify, that meant that a website relying on AWS in a lot of cases couldn't get the domain name resolved. So let's say Reddit, which was affected, wanted to query some data. The location that needed the data, say reddit.com slash page that needs the data.html, couldn't be resolved basically because it couldn't figure out where reddit.com was. But the data was fine.

2:06You just couldn't get to it. While this was just in the U.S. East 1 region, enough relied on data stored there that it affected hundreds of websites on multiple continents. By 6.35 a.m. Eastern, AWS said it had mitigated the DNS issue, but some AWS services, especially EC2, were now having what they were calling at the time knock-on issues. EC2 operates virtual machines for companies, which is why you may have seen intermittent issues even after they said they got it done. By 8.48 a.m. Eastern, AWS recommended that clients avoid certain availability zones. That's basically you can pick which data centers in your region you want to use.

2:52They're like, avoid certain of those. And at 9.42 a.m., AWS started rate limiting new launches of those virtual machines, those EC2 machines. At 11.43, Amazon said it identified a problem that started in the EC2 internal network and what it called, and I'll quote them here, an underlying internal subsystem responsible for monitoring the health of network load balancers. Now, that could mean the system might have inaccurately marked nodes as healthy or unhealthy. So if they mark a healthy node unhealthy a bunch of times, that could cause a delay because suddenly these nodes are unavailable because they're unhealthy.

3:33But the load balancer's routing also depends on DNS updates. It might also have failed to propagate routing changes, which would explain why they thought it was a DNS problem early on. at 12 13 p.m amazon started talking about pulling back on the throttling indicating it seemed to have mitigated the problem in the subsystem and could start putting everything back together and at 103 p.m amazon announced similar problems in its lambda system which is why i couldn't turn on my studio lights by asking my echo to do it right before we started to record uh aws power is a huge chunk of the internet around 30 percent uh of cloud market share uh by some estimates and its us east one region is particularly crucial problems at that region have caused big outages in 2020 in 2021 in 2023 and now in 2025 so you know it's kind of like haley's comment rob just Every once in a while, we get a U.S.

4:34East 1 AWS problem. Yeah, so here's the thing. It is, DNS is a mess if there's any issues with it. When we hear these outages, it is very common for it to be a DNS-related issue. And, Tom, you and I are both old enough to remember using not just pay phones, but having to use the telephone book at the pay phone. Yeah, right. And have you ever gone to a pay phone and you want to make a call? It's like a really important time to make a call. and all the pages that you need to call the numbers that you need are ripped out. And there's just nothing you can do. So the phone is available, but you can't call anybody because you don't know what number to dial.

5:11And that's essentially what's happening here. It's like the internet is not down. The internet is very, very robust. But a big chunk of the internet runs through Amazon AWS. And it's not down. It's just it can't route stuff right now. So it's affecting literally Earth. There's so many people who are having problems with this right now. And it's because of that pesky DNS that always, not always, but often seems to be one of the culprits that there's a problem with it and it causes these massive outages. Yeah, I'm always struggling with that metaphor because fewer and fewer people every day remember what a phone book is.

5:48But I guess it's like if your contacts app disappeared, right? And now suddenly you have to be like, I don't know anybody's phone number. I just look them up in my contacts. Same sort of thing, you know, for kids out there to understand. Yeah, but that is exactly it. That is the symptom of the problem, though. It seems like there was something in the load balancing, and it seems like it wasn't just EC2, since they're now having a Lambda system problem, that there is a load balancing code problem that is causing routing not to propagate. And so when it looks up the DNS in that database, the load balancer is not providing updated routing information or the routing information is bad because people change this stuff frequently on the fly.

6:34That's one of the things that's so valuable about AWS and why so many people use it is that it can do that kind of stuff on the fly for your website. So I'll be very curious to get a deeper look once they define what the actual problem underlying it was. Right now, it seems like they're still trying to scramble to fix it though. They're definitely still trying to figure it out. We You will hear of a postmortem probably later today, if not tomorrow or whenever they get this figured out. My gut tells me that that meeting is going to start off like, see what had happened was because because it's just like, yeah, this is it's just cascading and it's getting it seems like it's getting bigger and bigger and bigger as it goes as it goes along, which is what these type of problems do do this.

7:19like this goes down and because that's down this goes down because that's down this has been marked you know marked erroneous uh you know enough times that well we need to take it down to check it there's just so many things that happen because of this thing that could have been initially little at the beginning and it's just cascading and getting bigger and bigger yeah uh and it's tempting to start to look for conspiracy theories like oh is it a hacker you know has somebody gotten into it because it's oh it's in northern virginia isn't that right by dc uh us east one is just a big region it's in northern virginia because that's where the tech industry grew up around aol uh i don't know that there's much conspiracy to be found there usually these things end up and thankfully amazon is actually pretty good about explaining what happened uh once they've you know gone through and figured it out uh usually it ends up like there was a code push that had an unintended effect or uh had a mistake in the code uh it's usually one of those two and then like rob was saying it just cascades out from there yeah so it's it's here's the other interesting thing we just probably in the last two or three weeks we had a comment that came in that was saying what happens to my movies if aws goes down we both kind of said well how often does that happen well it seems like it happens more often than we know it's about every year lately at least because the new operating system that's on the new fire stick uh loads your apps from the cloud uh But apparently, let's see, 2020, 2021, 2023, 2025, four out of every five years.

8:54So almost once a year, your AWS go down and you won't be able to use your fire stick to get your app. So now we know the answer to that. DTNest is made possible by you, the listener. Thanks to Larry Bailey, Michelle Sergiu, Kirk Stephenson, and a brand new patron, A Beck. Welcome A back. Good to have you. Welcome to the family.

9:22All right, there's more than outages to talk about. Let's get to the briefs. Samsung is beginning to test its Exynos 2600 chips, its first two nanometer gate all around, or GAA SOC. Korea Economic Daily compiled reports of internal tests that showed the Exynos 2600 was faster than the A19 and the Snapdragon 8 Elite Gen 5, both on CPU and GPU. These are internal tests under local conditions, but they do bode well for the chip even when it meets real-world conditions. This also makes it believable that all Galaxy S26 models, not just the base model, will use the Exynos 2600 chip, at least in some regions.

10:00Yeah, they tend to sell it with Qualcomm chips in the U.S., but the Exynos, when they use it, is often used in a lot of places outside, and it looks like they may go back to using it more widespread. They barely used it at all in the last generation. And this kind of stuff leaks out to Korea Economic Daily because Samsung is excited about its internal numbers and it wants people to hear them. It leaked very carefully. If you notice, if you go look at the actual data that was leaked, GSM Arena has a good breakdown of it. They didn't leak any compute numbers for the Snapdragon 8 Elite Gen 5. So the compute numbers for that probably weren't that impressive, but they had very impressive results in other places like GPU and performance against the A19 and the Snapdragon.

10:48So that's good news, and Samsung needs some good news right now. Yeah, if Samsung can get to where they only have to make one phone for Earth, that solves a lot of problems for them because they are building multiple devices of the same device just based off of where you get that device. And part of it is because on the higher end phones, we really want to have that processor that's doing everything that it can do at the highest rates. But if they can get one that will do this and they are a good partner with Snapdragon right now that they do tend to work well together. But that chip is more expensive.

11:25And if Samsung can get to where they're using a lesser expensive chip, that creates a lot of opportunity for them and definitely gets to where some of the issues that they're having, they just won't have. Yeah, it's tempting to want to compare this to Apple Silicon, where Apple controls the whole stack. And so they can do a lot of things because of that. Samsung's not exactly in the same position, though, because it has so many more models, first of all. So there's never going to be one ship that works in every single one of its models. Across its flagship, across the Galaxy S model, sure. And that's probably what's important.

11:58But they also sell the Exodos chips to other makers. Apple doesn't sell Apple Silicon to anybody else. so Samsung also wants to modify that design to be useful for other people whereas Apple all they care about is if it works in an iPhone or a Mac or whatever they're making it for. Well it's been a month since the launch of the iPhone 17 speaking of Apple and sales estimates are beginning to come in. Counterpoint estimates that the iPhone 17 sold 14 percent better in its first 10 days than the iPhone 16 did both in the U.S. and China. In addition to sales numbers we're also getting reviews based on a month of testing.

12:34Nothing surprising in the flagship models, in the base model, but CNET and Apple Insider are among the reviewers that were surprised to find that battery life and camera quality were both sufficient in the iPhone Air. Yeah, the one that everybody was like, ooh, that camera, I don't know, ooh, that lower battery, I don't know. Now, they weren't equivalent to buying a Pro or even a base model, but CNET noted that durability wasn't a problem either. And most of the reviews are like, actually, it's adequate. I could go most of a day without having to charge. Not that big of a deal. This is really good news for Apple, and it definitely is good news for Apple's market share, at least to their market cap, at least this morning.

13:18Their stock price is running on this news. So, you know, we'll see where it ends up. But this is not a bad thing for Apple. These phones are selling pretty well. And there was a bit of consternation. They didn't know if they were going to sell that well because they didn't have the Apple intelligence fully baked in yet. But they're selling well just on the hardware. That's a really good thing for Apple. Yeah. X is testing a new way of handling links that lets you open a link while still seeing the like, repost, and other buttons. The idea is to boost engagement numbers. X is also launching a marketplace to let paying users get inactive usernames for free or, in some cases, by paying a fee.

13:57Priority usernames like full names, multi-word phrases, or alphanumeric combinations will be transferred for free. However, some rare handles could cost you from$25 ,000 to a few million dollars. Rare handles are short, generic, or culturally significant names. If multiple people want to buy a handle, X will select who gets it based on their use and engagement of the platform and intend to use the handle. Rare handles like usernames, for example, like at pizza or at Tom or at one are the ones that will cost you a lot. So, Tom, I just want to know, are you putting your bid in for Brad Tom? Yeah, I don't think so.

14:32I don't think I'll be dropping a cool million to get at Tom. I'll stick with my ridiculous Ace Detect username that I've had forever. this is something that would have been even bigger news a few years ago i think right now it's big news for the folks who are still using x and there's still a lot of people still using x and i think there's a lot of abandoned usernames out there you should be careful if you're somebody who got rid of your username on x because it could be taken up by someone for free if it's not you know, one of these short handles, uh, and you may not like that if you kept your X account and just deleted all your posts, which I know a lot of people did.

15:14You also might want to look in and make sure that your account hasn't been labeled inactive, uh, because inactive accounts can be recovered and those usernames could go into this marketplace as well. I, um, I'm, I'm kind of lucky. My name is spelled weird. Um, you know, I use two B's and Rob, and then my name has an E in it that probably shouldn't be in there, uh, in my last name. and I use that handle everywhere. So for me, this is not a big issue. But one of the things that I do is I just make sure that even if I'm not using a platform anymore, I log into it every three or four months just to make sure that I keep the handle so that I don't have a problem with somebody grabbing my very unique name some point later in time.

15:52Yeah. And I don't know how much money they're going to make on this, but if X has shown me anything over the past couple of years, it's that their power users, their super fans will drop some money. So I imagine they're going to bring in a few dollars here and there. A few noteworthy items about technology that could help make data centers more power efficient to talk about. First of all, Alibaba says it has created a new pooling system that let 213 GPUs perform like 1192 and was able to let it cut its use of NVIDIA GPUs by 82%. It's called the Aegean system. They used it for several months in their model studio in Beta, and they will present a peer-reviewed paper on the system at the 2025 ACM Symposium on Operating Systems, or SOSP, in Seoul this month.

16:48Also, and I think this one's even more fascinating, scientists from Xscape Photonics, publishing in the October 7th edition of the journal Nature Photonics, described how they have created a highly efficient photonics chip. The scientists were trying to improve lasers in LiDAR, you know, the LiDAR sensors that are used often in autonomous vehicles. They found that by accident, they had created a natural frequency comb. A frequency comb usually needs bulky, expensive lasers to split light into multiple colors, and then each color can transmit a separate data stream. Well, a small frequency comb they created can fit on a thumbnail-sized chip, meaning they can replace racks of individual lasers, and even, in some cases, replace fiber that can only send one single wavelength, because each color in that frequency comb can have its own data stream.

17:44Xscape photonics principal engineer and former Columbia engineering researcher Andres Gil Molina said if you can make them powerful, efficient and small enough, you can put them almost anywhere. That's going to be a big deal for saving power in data centers if if they can make it work at scale, which is always the deal with these things. That's pretty impressive. It's amazing to me how accidents are often what changes technology that we use for the better. I mean, there's so many things. It's like we accidentally created microwaves by melting the chocolate bar. We accidentally do these things that ultimately 10, 15, 20, 30, 50, 100 years later, we can't even imagine life without that accident that happened.

18:28And that might be something that's happening here. Yeah, that is one of the arguments for basic research is you don't know what you're going to find until you go poking around, right? It's scientific exploration. and this is quite promising. A lot of people picking this one up. Finally, some quick headlines. These are the kinds of things that are just good to know. We don't have to explain them a lot to you, but you might want to file them away if you want to understand the news in the future. Aura is redesigning its app, including a new feature called Cumulative Stress, which combines data from sleep, heart stress, temperature, and activity.

19:02Aura is that ring that you put on that gives you some biometrics on what's going on. An antitrust complaint has been filed in China on behalf of 55 iPhone and iPad users. I bet you can figure out what the complaint is. It accuses Apple of abusing its control over App Store distributions and payments. So not even in China can they avoid this one. Adobe is offering something it calls AI Foundry that uses copyright safe models to train on a company's own intellectual property to create personalized tools. There's a pretty nasty bug in the October Patch Tuesday update for Windows 11. It only will affect you if you go into Windows recovery mode, but it stops USB devices from working, meaning you don't get to use your mouse or your keyboard in recovery mode, which can be a problem, and has caused people to search out non-USB options for mouse and keyboard, or even just roll back that patch.

20:00an opening eye employee who goes by runo on x denied a report on cnbc that gpt6 will ship by the end of the year and eric mitchagovsky posted on x that the pebble core apps have been accepted on google play and apple app stores that is ahead of the revived pebble watches shipping later this year folks what do you want to hear us talk about on the show uh one way to let us know is our subreddit great stuff over there you can submit stories and vote on them at reddit.com slash r slash daily tech news show we in every episode of dt nestle some shared wisdom and today curtis reminds us of an old friend yeah curtis wrote ever since the show started covering talking to co-pilot on windows it reminded me of way back when windows 95 included dragon naturally speaking and I was telling my computer to open and close windows and apps.

20:57Lo and behold, Jen mentioned the same app today and brought it all together. Great show. Thank you, Curtis. So true story, Tom. I actually, Dragon, naturally speaking, I had to use it. I broke my hand and had to have surgery on it and couldn't type. So that had just come out and it was like, okay, let me just see if I can talk to my computer. And I don't think I ever stopped talking to computers. I just get to my microphones and just kind of go all day now. Yeah, I remember, I'm trying to remember what year it was, because I think it was before I got Windows 95. There was a piece of software that I worked at a bookstore.

21:35And so I was able to take software home because we sold software as well. And I took home this system that had a microphone included with it and would let you navigate Windows 3.1 by voice. It was all pre-trained stuff. You had to say it just right. but I remember blowing people's minds when I would like talk to my computer and things would happen on the screen and here we are 2025 just now finally getting to the point where that's useful 30 years later but yeah it was it was it was definitely something because like I could not type I had a big old cast on my hand it made it listen it didn't make it as easy as it is today to talk to stuff and get the stuff printed out but it did make it easy to be able to you know use my PC and not have to take off.

22:19Yeah. Well, it made it so you could actually do stuff, right? That you probably wouldn't have been able to do otherwise. That's a big deal. We want to know what you're thinking about. So if you've got some insight into a story, share it with us at feedback at daily tech news show.com. Yeah. Big thanks to Curtis for the trip down memory lane there. Thank you for being along for daily tech news show. You keep us in business folks. Uh, if you like the analysis we give here and you get a little value from what we do, All we ask is you give a little of that value back at patreon.com slash DTNS. Talk to you soon.

22:56The DTNS family of podcasts. Helping each other understand. Diamond Club hopes you have enjoyed this program.