Breaking the internet

0:00President Donald Trump has been back in office for one month and what a year it's been.

0:10We've covered a lot of Trump, but today explained this past month from pardons to executive orders to Greenland to Guantanamo to tariffs to Maha to Elon and Elon and even more Elon. But today we're going to talk about the websites. DEI would have ruined our country and now it's dead. I think DEI is dead. So if they want to scrub the websites, that's OK with me. Government web pages are disappearing. Sometimes they come back. Sometimes they don't. And it's part of a greater problem we have online. Some call it digital decay. Others call it link rot. Whatever you call it, our internet is disappearing.

0:45And we're going to help you understand why it matters and what we can do about it on the show today.

1:01This is gay. Today it's fine. This is dead. You're listening to today's flame. Today's flame. Sean Ramos from here with Addie Robertson, senior editor at The Verge, here to tell us about the websites. What is going on with the government's websites? So Trump signed a couple of executive orders, one of which defined officially the idea that there are only two genders, male and female. And another one that ends, quote unquote, diversity, equity and inclusion in the government. We will forge a society that is colorblind and merit based. And so the result here has been that more or less across the government, in addition to the kind of thing that we saw in the first Trump administration, which included purging information about climate change and some other general climate related issues.

1:59We've seen just a massive cut of anything that involves racial equity or transgender people or really anything that is sort of a subject of Republican culture wars. The CDC is currently scrubbing information from their website right now to be in compliance with a recent executive order. Here are some of the pages that have gone down. The Trump administration has taken away reproductive rights dot gov from the federal website. They also have scrubbed federal websites for any search of abortion. Within hours of President Trump's inauguration, the Spanish version of the official White House website disappeared.

2:37The website now gives users an error 404 message. A lot of the stuff initially happened very quietly.

2:47Reporters noticed it. People who used the information on these sites, which included data on the CDC or even transportation statistics, they have ended up uncovering a lot of this. And from there, the way that the Trump administration has mostly addressed it is in response to lawsuits that there were claims that they deleted this data improperly. There was a court order that required them to put it back up. And they have responded by putting it back up with a big banner that says we reject this information. We were forced to keep it online. But it violates something like, say, our dictate that there are only two sexes.

3:25So we find it unscientific or we find it against our policies. Any information on this page promoting gender ideology is extremely inaccurate and disconnected from the immutable biological reality that there are two sexes, male and female. Is there presidential precedent for something like this happening? Or is Donald Trump and Doge and Elon Musk and the gang like the first administration to come in and just start ripping apart websites? First of all, just for context, every time there is a new presidential administration, there is data that changes their priorities. There are new programs or old programs that get retired.

4:03So it's not necessarily surprising that some things have changed. But we have, as part of this, seen just a massive and really unprecedented deletion of information, including information that is required for people to do their jobs outside of the White House. And so it's a really huge issue right now.

4:28I don't think we've ever seen this kind of scale of data purging, especially of records. and scientific research. Obviously, the first Trump administration deleted some data in ways that seemed very ideological, aimed at suppressing information about climate change. The White House and other federal agencies are also revamping their websites, for instance, scrubbing mentions of climate change. And Trump is blasting... And obviously, there have been pages that just disappeared at the end of terms, but that tended to be more about oversight. It tended to be more that there was a changing of the guard and they didn't really know where everything was.

5:10So some websites are disappearing. Sub-websites are disappearing and coming back. Some websites are still up. Is there anyone who has like a full grasp of what exactly is gone forever? There are nonprofit groups and journalists that are working to preserve this information. There were already groups before Trump took office, like the Environmental Data and Governance Initiative, that we saw a little of this in Trump's last term. And so there was this effort preemptively to preserve information, which includes not just web pages, but also just collections of data from groups like the CDC. So there are all of these, not necessarily fragmented, but individual and private efforts.

5:55And also one of the really big load-bearing institutions here is the Internet Archive and the Wayback Machine, which has always maintained this project that archive data at the end of every term, but now has become a place where you can go and check and see what's disappeared and has become part of this process of identifying and trying to recover data. Beyond the American people perhaps needing access to some of this information, beyond any number of institutions needing access to this information, it points at a bigger problem we have on our internet right now, right? Something called link rot?

6:34Link rot or digital decay. Which is a general phenomenon where webpages either disappear or they move in a way that makes them more difficult to find. And so the internet, which is a series of links that point to information, ends up with all of these little dangling ends and dead links and places where you can no longer find information that someone has referred to, or when you can simply no longer find a record of it at all. A 2013 Harvard study, for example, found that half the hyperlinks in Supreme Court cases, today's equivalent of footnotes, are broken, a phenomenon known as link rot. Why do webpages disappear?

7:19The most obvious case is when a page is just taken down, maybe sometimes because the entire website went under, maybe sometimes because they think that page is no longer valuable. Government agencies remove documents and companies fail and with them the sites they host. Think of Geocities, Yahoo Video, and more recently the news site Gawker. There are also incidents where just the URL of it, the link that points to that information changes And so it's harder to find. So if you previously linked to it from another web page, then that's just not going to go there anymore. The wonder of it is it's very, very simple.

7:58Anybody could go and set up a web server on their computer and make it available to the world. Unfortunately, it's too simple. It's fragile. That if something happens to that piece of equipment, that website, just blink, is gone. So you've been covering this issue, Addy, for more than 10 years. is link rot getting worse online or is it sort of you know continuing a pace link rot has been an issue that people have been identifying in some ways since really the beginning of the internet but for definitely at least a decade a really significant proportion of web pages and links have no longer functioned i think the latest research was something like 38 of web pages that existed in 2013 are no longer available.

8:48This is, I think, not necessarily an issue that has suddenly snowballed, but I think we're seeing some unique circumstances now that have added to it. One of them is something like search engine optimization, where Google rewards pages, or at least people think it rewards pages that regularly refresh or that seem like they are providing new information. And so, for instance, CNET, which is a really venerable tech publication, removed a bunch of its older articles because it wanted to appear in Google search results more highly. And so there was this sense that, OK, it makes people more likely to find current articles, but also just this trove of information disappears.

9:34Right. I mean, I think we can all, you know, mourn the loss of like our Geocities homepage from 2003. Yahoo! But it's a lot rougher when, like, I don't know, some billionaire buys out an alternative newspaper and just decides one day to shut down its website. Sometimes it's a billionaire that buys something and shuts it down. There are also just more insidious phenomena that I think really kind of speak to the commercialization of the internet and the sort of cannibalization of anything that can be turned toward profit. So you have old websites that, say, have a name people recognize, and then they get resurrected, but they no longer have the old information.

10:20They've been filled with AI-generated new articles that can sort of capitalize on this old name as this zombie site. Or you have issues where there's a link that goes dead and somebody tries to kind of hijack that link and they either they contact the website administrator or they find some other way to get that to point to a new page that will then build their own credibility but doesn't provide the original information. So there are all these cases where archival gives way to profit. That information was useful sometimes because it provided, say, statistics or it provided evidence. If you're, say, looking at Wikipedia and there's a dead link that no longer provides the information it used to.

11:08And sometimes just because these things are a valuable record of what the Internet used to be and of how people lived, there are a lot of things that at one point would have been written down on paper or in some other medium that's just a hard document and people can look back on it. But at this point, a huge amount of our culture takes place on the internet and the internet is a very fragile place.

11:35Addie Robertson, reader at TheVerge.com. When Today Explained returns, we're heading into the Wayback Machine to hear from the people trying to archive the entire internet one webpage at a time.

12:04What turns unrest into a revolution and where could it lead? The big open question is whether this set of protests that are currently underway is the end or the beginning of the end of this third phase of sort of modern governance when it comes to Iran, the end of clerical rule. I'm Jake Sullivan. And I'm John Feiner. And we're the hosts of The Long Game, a weekly national security podcast. This week, we cover the massive nationwide protests in Iran and the U.S. response. The episode's out now. Search for and follow The Long Game wherever you get your podcasts.

12:45This is Today Explained. So let's just have you start by saying your name and what it is you do. Sure, yeah. Hi, my name is Mark Graham, and I am the director of the Wayback Machine at the Internet Archive. Which is a not-for-profit that has been preserving the web since 1996. 2006, journalists use it all the time. But for the uninitiated, I asked Mark to show us around the Internet Archive. Where do I begin? It's like walking into a very large library and say, show me your favorite book. Well, for example, last year, it was a big news story that MTV News was shut down. And the founding editor of MTV News wrote about it on LinkedIn.

13:29And there was a lot of other editors talking about it. It's like, oh, my God, all of our articles are gone. They're missing. And I just casually, you know, waded into the conversation and go, hi, check here, Wayback Machine. And they were like, oh, my God, you guys, like, got it all, pretty much, yeah. And they said, well, you know, people say, well, what did you do? What did you do when it went down? You must have – I say, we didn't do anything when it went down because we've been doing our job all along. We've been working to archive the public web as it's published on an ongoing and continuous basis.

14:08So if we have to start paying attention to something after it's gone down, that means we screwed up. So with that example, with MTV News, give us a sense of what you guys were doing in advance of that website going down to make sure that people could find out, you know, I don't know, what Everlast was singing about in 2004. Hello, Jancy Dunn here, and joining me now is former House of Pain member Everlast. Welcome, Everlast. Thank you.

14:39So for any one of a number of thousands of reasons, we set our web crawlers and archiving software out on a mission every day to identify and to download web pages and related web-based resources. We bring in millions and millions of URLs every day that are signals to us, signals of where new material is being published on the web. And we make sure that we archive all of those URLs, all the web pages associated with those URLs. And then we look at those pages and we identify links to other pages. And then we go to those pages and we archive them, et cetera, et cetera, et cetera. That's where you get this metaphor of crawling like a spider throughout this web.

15:30And the net result of it is that we add more than a billion archived URLs to the Wayback Machine every day. And this material, as it's added to the Wayback Machine, is indexed and is immediately available to people who go to web.archive.org, enter in a URL, and then are able to see a history of archives that we have of the web page that was available from the URL at any given time.

16:05I want to talk about government websites now because that's sort of the reason we're having this conversation today. I think most people probably think the government will take care of archiving government websites, But here we are in a new administration and websites are disappearing, coming back online, and people are worried. When you, a, you know, an archivist of the internet, see government websites disappearing, coming back online, becoming unreliable, how do you react to that? Is that like better or worse than regular websites that are non-governmental going offline? Well, as an American, my tax dollars help pay for some of this stuff.

16:49And that much of it is maybe a benefit to people. Certainly, my first reaction is, hmm, that might not be such a good thing. I do want to underscore that there is the National Archives and Records Administration that does do archiving as well. But for whatever reason, we seem to be like one of the main players in the space of trying to archive much of the public web, including, and right now especially, U.S. government websites, and making those archives available in near real time. Were you caught off guard when you saw the new administration removing web pages, removing websites? This is pretty normal in some respects.

17:35It's normal and expected, and it's what's happened, frankly, for each administration in the time that we've been working on this effort. I mean, look, it's under new management, right? For example, you wouldn't expect the whitehouse.gov website under any new presidential administration to be the same as it was before. So we go out of our way to try to anticipate the frequency in which web pages should be archived so that we got a pretty good shot at getting those changes. You're saying, you know, the whitehouse.gov site obviously changes administration to administration. I think to some degree people understand that, that Joe Biden's administration probably wouldn't have been posting trolly valentines about immigration, you know, a year ago this time to their Instagram account.

18:30But what we're seeing here is websites that people need, websites that record public health information going offline, briefly, permanently, what have you. No, that's true. Is that a different degree of sort of erasing the historical record or messing with the historical record than we've seen? It's different. It's certainly different in terms of the number, seemingly. I mean, we're still in the early stages of this administration. But, yeah, I'd say on the face of it, you're right. the historically we haven't seen major u.s government websites taken offline like we did say for example you with regard to usa id but um and i'm gonna i'm gonna leave that kind of analysis to others and uh and really just focus on trying to uh archive the material the way back machine, the internet archive, mostly funded through donations, the generosity of people, institutions, even governments.

19:37Is that going to be enough to archive the internet to the extent that, you know, future generations will want to see and need? Enough is a very subjective term. Well, as an archivist, for me, it's never enough. Because you don't know. No one knows what is going to be of use, value, importance in the future. Maybe even the near future of tomorrow, much less like the very far off future. And since millions of people use our site on a daily basis, we get a lot of feedback from them. It motivates us, but it also helps direct us and inspires us to continuously try to do a better job at being the best library that we can be.

20:29Godspeed. There you have it. Let me ask you one last question, Mark. You guys have been at this for nearly three decades. Certainly you've saved a lot of stuff, and certainly a lot of stuff has fallen through the cracks. I wonder, is there something that slipped through the cracks that you could tell us about that might suggest to our audience what is lost when we can't archive to the extent we want to or need to? Okay, so it kind of caught me up with that question. I'll just say, I don't know right now. I can't say that thing. Gosh, I wish. Okay, I got one. I mean, this is just in recent history.

21:12Apparently, there was a page up on the CDC website about bird flu last week. It apparently was only up for a few minutes, and no one got it. Huh. And by losing that fleeting webpage, that one, you know, maybe minor, maybe major webpage about bird flu on the CDC's website, what are we losing? Well, we're losing part of the story, right? We're losing part of our understanding of the evolution of arguably a significant health issue, concern. We don't know where this is going to go. I don't know. I guess that's the other point, right? I mean, you don't know necessarily now that which is going to be very important in the near or longer term.

22:02In the time of Martin Luther, there was raging, raging debates. And much of that debate took the form of things that were written on pamphlets. The pamphlets at the time were considered of little value. I mean, people read them and they shared them, but they didn't necessarily save them. So today, a scholar of that time or someone like me who's just kind of strangely curious, what I would give for a collection of those pamphlets. Yeah. I mean, and you are comparing, in a way, a CDC website to the Protestant Reformation. But I think you mean it, don't you? I do, because I don't know. And one really can't know without the benefit of the long historical view.

22:55And that's not something that we have access to today. Why? Because we don't have a real time machine.

23:10Mark Graham, known exclusively to Amanda Llewellyn as WebMG. Check out the Wayback Machine at web.archive.org. Amanda produced the show today. Laura Bullard helped and wore the hat. Jolie Myers edited. Andrea Christensdottir and Patrick Boyd mixed. And Andrea even made some original music. Thanks to the Free State of Aftonia for the Wi-Fi. Oh, and it's Today Explains' seventh birthday today. What did you get us? Maybe show some love in the comments, in the ratings, in the reviews. They say it helps. And thank you for listening for however long you've been listening. If you're new to the show, feel free to browse the archive.

24:07Thank you.