Can we preserve knowledge … forever?

0:00This message comes from Schwab. Self-directed investing, trading, full-service wealth management, automated investing, financial planning, thematic investing, retirement planning. And to think, that's just a small taste of what Schwab offers. Because Schwab knows that when it comes to your finances, choice matters. No matter your goals, investing style, life stage, or experience, Schwab has everything you need all in one place, so you can invest your way. Visit schwab.com to learn more.

0:58feel like I'm a different person. Yes. Do you feel that way? Ideas worth spreading. From TED and NPR, I'm Manoush Zomorodi, and we are going to take you back to 1989 and to the first gay video game ever. Yeah, it was a big deal. The sounds had to be kept very small. Everything had to fit on that 800 megabyte diskette. It was very limited.

1:37The basic premise of the game is that you assume the role of a lesbian detective named Tracker McDyke. And you are searching for your friend who is a drag queen and her name is Tessie LaFemme. This is CM Ralph, the creator of Caper in the Castro. She's got some inside information about something that's going on in this little neighborhood of the game, which is the metaphor for Castro Street in San Francisco. But before she can get the information to you, the phone suddenly hangs up. And this is where you're left. You're left, oh my God, where's my friend? In the late 80s, CM was working in Silicon Valley by day and at night making Caper in the Castro on their Mac Plus.

2:27Tessie, thankfully, if you finish the game, you free her. She's trapped down in a cellar with all these biocontaminants that the secret plot is they're going to release this into the air or water or something. A lot of it's not clear to me still, but that's the threat, the annihilation, basically, of LGBTQ community. And I think that directly reflects how it felt back then. During that time, it was the height of the AIDS epidemic, and I had seen so many of my gay male friends just disappear. It felt at times like we were under annihilation, and nobody cared. No one was helping us. We had to help each other.

3:17So when Caper and the Castro was complete, CM shared it with friends. Someone posted it on an online LGBTQ bulletin board, and the game took off. Yeah, you know, it just kind of spread. So at its high point, about how many copies do you think were downloaded? The estimate I had been given, and this was after it had been out for about five years, somebody had estimated it was about 250 ,000 copies. Wow! Yeah. But we have no way of knowing. So did you make any other games, or was that it? This was really the only game I made. CM put the original diskette of Caper in the Castro in their bottom desk drawer.

4:01And for years, they didn't really think about it. They figured that that part of their life was over. Until someone got in touch in 2017. I'm Adrienne Shaw. I'm a professor in media studies and production at Temple University. Adrienne was researching old LGBTQ video games, and she'd heard about Caper in the Castro. And I wanted to know more about the person making the game. Like, why did you even start doing this? Like, how did you do it? So I reached out to CM and just said, hey, I'd love to talk to you more about this game. I was stunned. I'll be honest with you. My first reaction was, what?

4:39No, that can't be right. CM pulled out that old diskette from their bottom drawer, but... The operating system for that game had died a long time ago. It no longer existed, so there was no way to actually play the game. And CM said, like, do you think there's a way to get these games playable again? And I was like, I don't know, but let me ask. Adrienne ended up finding someone with the technology to extract Caper and the Castro off the diskette. And she got it uploaded to the Internet Archive, the online library of the Internet. And that's where it is now. You can play the game online. How does that feel, like, that your game is available?

5:21Does it? No, I don't think I'll ever be used to it. I do get mail from people, 10 to 15 emails a year. People just wanting to thank me, and I get a little teary. What do they say? Can you read us one? Yeah. Dear CM, I had a wonderful time this evening playing Caper in the Castro. Thank you for this priceless contribution to both the history of computer gaming and the LGBTQIA plus community. And thank you for helping pave the way for the rest of us. And that, you know, when I get stuff like that, I'm like, oh, man. Yeah, because it's more than saving your game from being forgotten. It's really saving a piece of history.

6:09Yeah, it's my love letter to my community. From filing cabinets to floppy disks, the way we store our information is always in danger of going obsolete, taking along with it entire chunks of our past. And it's not just our stories and data that are in jeopardy. Thanks to climate change, the world around us is morphing faster than ever, too, quickly making maps outdated. So how can we keep our records up to date and safe? Is it even possible to truly save anything for all eternity? On this episode, we search the past, present, and future for answers. Boy, everything digital is just completely ephemeral.

7:00Whether it's formats that go out of date, like those floppies, just try to run one. Or a CD, find somebody that has a DVD player. I mean, it's just, it's starting to get such that things that are really recent are just going away. This is Brewster Kale. The average life of a webpage before it's either changed or deleted is 100 days. That's it. I think it was kind of a cruel joke to call webpages pages. because you think of them as lasting a long time. You know, Gutenberg Bibles and all of that kind of thing. And nope. Brewster knew this was going to be a problem. Disappearing websites, missing chapters of the Internet.

7:39He knew this way back in 1996. And that is why he created the Internet Archive. As we heard, it's where Caper and the Castro can still be found. And over the years, the Archive's mission has expanded, saving old books and movies, TV shows, and music. The idea is to try to build the library of everything, the Library of Alexandria for the digital age. We can make all the books, music, video, webpages, software, everything ever published by people available to anybody curious enough to want to have access to it. That was the dream of the internet, and the Internet Archive is a part of making that dream come true.

8:20Which is a lofty goal and a huge endeavor. Because how does someone even begin to back up the internet? Brewster started by building something he called the Wayback Machine. Yes, probably the most used and important part of the internet archive right now is the Wayback Machine, where we have collected webpages by going and basically clicking every web link on every web page every two months. So if you go to archive.org and type in a URL, we'll show you different versions of that URL over time. We collect about a billion URLs every day. And we're finding that it's really important to journalists that are trying to find what actually happened.

9:09Lawyers love it because they can use it to say, hey, you said this before, and now you're not saying that it's the only record often. Yeah, and you can dig up information that someone's deleted. Yes. Take Donald Trump's tweets. A large part of the policy of the impact on our country while Donald Trump was first president was through his Twitter feed. And then they turned it off. So it all just kind of disappeared. So we have a copy that we have made available through the Wayback Machine that makes it so we can see what it is. Or when a company goes under GeoCities and just everybody's sites go away.

9:50They're endless of these sites that go away or make different business decisions. And so people say, gosh, I'm glad that I can get a hold of that. Right. But we run into things like locked files, databases you can't get through to. Some of those make things real challenges. We're working back and forth with the different websites to try to make things available. And the web also has parts that go obsolete so that the old websites, you can't replay them anymore. So there's challenges every day. Do you ever worry about things being lost to the past? I mean, I can imagine that this would make you neurotic.

10:30Oh. Like, oh, we missed something. Oh, yes. We missed Napster. Oh, really? So Napster was maybe the best, biggest music library ever built by people, and it was shut down. We didn't get it. And if you just take the libraries in Ukraine that are being purposefully targeted, just the same way the Nazis targeted the library in Belgrade, it's a way of erasing a culture. You go after their libraries. So, yes, we're worried about this all the time. Right. So what do you do? Do you try to go back and fix things that you missed? Do you have an example maybe? Well, on Wikipedia, we've tried to take all of the footnotes, all the citations, and turn them blue, turn them into little links.

11:16So we went and worked to fix the broken links in Wikipedia. We've now fixed over 15 million broken links. We've prioritized the books that are referenced in Wikipedia and acquired those books, bought them, or got them donated. and we digitize them and then put them back in such that if there's a page number, you can click and turn right to that right page. We did a big project on Ukrainian Wikipedia to try to collect all of the books that are referenced and make those clickable. So how much harder is it to collect everything that's on the web now compared to, say, a decade ago or 15 years ago simply because it is behind paywalls or you can't access it without a login.

12:03Yes. So we have robots that are going around and collecting a billion URLs. And fortunately, there are over 100 people that work for the Internet Archive that are trying to work on keeping it all alive. We don't collect every YouTube video. It's just too big. But we try to collect ones that are referenced a lot or the linked to from Twitter pages, say. So we can't collect everything, but we collect a lot. And if we're not collecting the right things, go to archive.org. There's a save page now feature, and you can put in a URL, and people do this all the time. It's used at about 80 times a second.

12:46So even anybody can go and participate in making things permanently available. I just did this for the obituary for my aunt. I went to the webpage, made sure that that obituary from that funeral home service was archived. So I did that this morning. So you too can go and participate in building the web archives. In a minute, a legal battle between the Internet Archive and some of the biggest book publishers. At issue? Whether archiving e-books is digital piracy or preserving the best of humanity for everyone to enjoy. I'm Manoush Zomorodi, and you're listening to the TED Radio Hour from NPR. Stay with us.

13:43support for this podcast and the following message come from wayfair it's way day at wayfair where you can score the best deals in home and upgrade your space that work within your budget and the best part, everything ships fast and free during Wayday. Get up to 80 % off with fast and free shipping on everything. Head to Wayfair.com April 25th through the 27th to shop Wayday. That's W-A-Y-F-A-I-R.com. Wayfair. Every style. Every home. This message comes from Schwab. At Schwab, you can get everything from self-directed investing to full-service wealth management, all in one place. No matter your investing goal, life stage, amount to invest, or know-how, you can invest your way with Schwab.

14:33This message comes from Whole Foods Market. Make Mother's Day special with brunch or dinner from Whole Foods Market. Start with quality cheese and charcuterie with no synthetic nitrates. Then choose sustainable, wild-caught, or responsibly farmed seafood on sale. Finish with seasonal bakery treats like strawberry pretzel cream pie Or keep it easy with ready-to-heat quiche Lorraine, deviled eggs, and fresh-cut fruit. Celebrate mom with Whole Foods Market. It's the TED Radio Hour from NPR. I'm Manoush Zomorodi. Today on the show, for all eternity. And we were talking to Brewster Kale, the founder of the Internet Archive, a nonprofit that is trying to digitize everything that we humans create, from websites to music, old movies, and of course, books.

15:23The Library of Congress is about 28 million books. We've digitized maybe six or seven million. We physically own probably on that order. So we still have a long way to go. And it's getting harder to proceed because e-books present very particular problems. For example, that e-book you downloaded, you don't actually own it. So it turns out the big publishers don't sell e-books. They license them. So your e-book that's on your Kindle or whatever, you don't actually have that. Not in the same sense you had a physical book. You can't pass it down to your kid. And any time they want to change it, they can change it at any time or make it go away.

16:05It's a licensing issue. And Brewster and the Internet Archive started trying to get around it by buying physical copies of books, scanning them, and making their own e-books to lend out. So we started that in 2011, and in the beginning of the pandemic, four of the largest publishers decided to sue the Internet Archive to say that you aren't allowed to digitize and lend. What the Archive calls equal access, those publishers say is digital piracy. And that suit is ongoing. We'll hear probably next year from the district court, and it'll probably be appealed. But we'll see. The big concept that I never really would have imagined would be at play is digital ownership.

16:48When you buy a digital file, do you own it in the same sense that you owned a physical thing? You can't just go and post it and give it to everybody. That's understood. Fine. But do you get to keep it? And what the big publishers are saying is no, there's no such thing as digital ownership anymore ever. So that's the absolute opposite of what we were doing with the internet in the earliest days when we were trying to democratize access, democratize creation. I actually went back into the TED archives and watched your talk from 2007 where you laid out your vision. And you knew then that there would be conflicts, even if you didn't know what they were.

17:29There's a political and social question out of this. All of this, as we go digital, is it going to be public or private? There are some large companies that have seen this vision that are doing large-scale digitization, but they're locking up the public domain. The question is, is that the world that we really want to live in? What's the role of the public versus the private as things go forward? How do we go and have a world where we both have libraries and publishing in the future, just as we basically benefited as we were growing up? So universal access to all knowledge. I think it can be one of the greatest achievements of humankind.

18:02like the man on the moon or the Gutenberg or the library of Alexandria. It could be something that we're remembered for, for millennia, for having achieved. People have no idea of the heroics that not only the staff of the Internet Archive, but now a thousand other organizations we work with on the web collection, about 500 libraries and the book collections, of how much goes on to try to make it so that the web that we sort of take for granted works. I guarantee if you've never heard of the Internet Archive, you've used it because it's just woven into everything. That brings me to a final sort of existential question, Brewster.

18:42If everything digital eventually becomes obsolete, how do you archive the archive so it doesn't become obsolete too? Boy, libraries are, you know, they're destroyed all the time. And it's often governments or large, powerful entities that seek to destroy them. So you want more than one copy in more than one place. Then you also want to make it so that it's still used, so that it's cared for. Our collections are almost completely on spinning disk. We have to replace those every five to ten years. So we need people to want it to stay around. Fortunately, there are many, many, many people that are seeing this as a path forward.

19:25Not an easy path. No. Building a library of everything is a challenge, but it starts one web page at a time, one book at a time. And if we see ourselves as preserving history collectively, we'll all make it come true. That was Brewster Kale. He is the founder of the Internet Archive. You can see his full talk at TED.com. Thanks also to CM Ralph, artist and maker of the video game Caper in the Castro, and Adrian Shaw, professor of media studies and production at Temple University. Since this episode first aired, the Internet Archive has had numerous challenges. It had to restrict access to half a million books on its site after losing legal battles with publishers.

20:13More recently, media outlets have been blocking the site from archiving their articles because of concerns that AI platforms use that material to train their models. Today on the show, preserving our past, present, and future for all eternity. Which begs the question, where are we going to keep all that information? And how will we keep it safe? Absolutely. One of the biggest challenges is the sheer amount of data that we're generating. And I don't think we really realize, most of us, where all of that data is being stored. OK, maybe I'll pay a few more bucks a month so that I can store my data in the cloud.

20:57But what is that cloud? You know, it's this huge server facility. This is molecular biologist Dina Zielinski. So it's projected that by the year 2025, the global data sphere, as in like all of the digital data that is out there, is projected to exceed more than 175 zettabytes. And one zettabyte is equal to a trillion gigabytes. So it's really become an urgent problem. You know, we've come a long way in digital data storage and computing, but we're running out of storage devices, quite literally. Dina says there is a possible solution to our data storage problem, a microscopic solution that's been around for billions of years.

21:44DNA. I actually have some in my fridge here. You can store, you know, that was a collaboration I had with a local artist, but we stored a digital museum on DNA. And so it's currently in my refrigerator. Wait, so you have a museum on DNA in your refrigerator right now? Yes. I love the image of you going to get a scoop of strawberry ice cream and seeing this little vial of DNA holding all this data right next to it. I forget it's there sometimes, and then I move my mustard jar over and there it is. It's so great. The idea of DNA data storage is actually not new. And we can actually say that DNA is the original storage device.

22:38It stores all of the data that makes up living beings. And so it's been optimized over literally billions of years. Dina says you can use DNA just like a hard drive or a floppy disk. But it does an even better job. It's true. I mean, it's absolutely true. It can beat any man-made device in terms of longevity, density. It has a very small ecological footprint. I mean, it's biocompatible. It's something that we'll always be interested in. and we can think of DNA as just another storage device. So instead of a compact disc or a floppy disc or a hard drive, we simply use the molecule of DNA. Yeah, and when it comes to DNA's durability, I have heard you use the example of Oetzi the Iceman.

23:36This is a 5 ,000-year-old man whose body was found frozen by hikers in the Alps in the 90s. Ah, yes. See the Iceman. So he was up in the Alps and it was nice and cold and dry. And these are sort of the ideal storage conditions. And all that is to show that, you know, even after thousands of years, we're still able to extract meaningful information from the DNA. So DNA in itself is very fragile, actually. If I just have DNA in a tube, for example, it's very susceptible to UV radiation. It is really only stable at room temperature for a relatively short period of time. And then it starts to sort of degrade and accumulate errors.

24:20But if you store it very cold and very dry, we could theoretically store data that's critical to humanity in such a place, in a naturally existing cold and dry place like Utsi the Iceman. Here's Dina Zielinski on the TED stage. Storing data on DNA is not new. Nature's been doing it for several billion years. In fact, every living thing is a DNA storage device. But how do we store data on DNA? We first learn to sequence or read DNA, and very soon after, how to write it or synthesize it. This is much like how we learn a new language. And now we have the ability to read, write, and copy DNA. We do it in the lab all the time.

25:08So anything, really anything, that can be stored as zeros and ones can be stored in DNA. To store something digitally, we convert it to bits or binary digits. Each pixel in a black and white photo is simply a zero or a one. And we can write DNA, much like an inkjet printer can print letters on a page. We just have to convert our data, all of those zeros and ones, to As, Ts, Cs, and Gs. And then we send this to a synthesis company. So we write it, we can store it, and when we want to recover our data, we just sequence it. Okay, so is there a machine that does this that says, right, there's a zero and a one, and so I'm going to turn that into an A or a C or whatever else it would be and transpose it or write it onto this synthetic material.

26:02Right, so that is actually the easiest part of this whole process. So we can theoretically store up to two bits or two zeros or ones of data per DNA letter, right? So you have a possibility of a zero or a one stored in A, T, C, or G. And so the encoding scheme is actually very simple. So if you have a 00, that becomes an A, 01 becomes C, 10 becomes G, and 11 becomes T. And that's it. It's just a text file, right? So you have a text file at the end of this whole process with just sentences of A's, T's, C's, and G's, and we send that text file to a synthesis company. And in a matter of days, depending on the complexity of your DNA sequences, they'll sends you back a tube with a sort of dried powder containing your DNA.

26:58With a dried powder? Yeah, so DNA is quite stable if you remove the moisture from it. And so usually it comes in a tube with just sort of this thin film on the bottom. Okay, so you've got the tube with the dried powder. How long can you leave it in there for? So if you store it, for example, in your refrigerator in your kitchen. I mean, it can easily last years, even tens of years. Ideally, you would freeze it and that would further extend the life of the DNA. Even for thousands of years, like Ötzi? Yeah. So, I mean, when we synthesize DNA, it's still fragile on its own. But if you keep it protected so we can store DNA in silica beads, so glass beads.

27:48And some researchers at ETH in Zurich, Robert Grass's team, have shown that the DNA is stable for thousands of years, and they've replicated sort of a European climate. And they showed that by inducing temperature and other stress, they were still able to recover the data from the DNA. And let's say you were like, today's the day. I'm going to get the data out of the DNA. How would you go about doing that? So just add water. So you just take your tube of DNA, you add water to it, and then you load it onto the sequencer. And then you simply convert your A's, T's, C's, and G's back to zeros and ones and decode the data.

28:35Okay, so this is something that you need highly specialized computers and machines to do right now, right? Yes. Even though I have DNA in my fridge, I cannot do much with it. Okay, got it. So just being at home, it's not like you're going to store your data in the fridge like you store data on your desktop, on your laptop or something. What we're talking about is really putting critical pieces of humanity's collective knowledge in a safe place. Yeah, so DNA storage is mostly useful for archival or cold data. One of my favorite examples is here in France. They store the rights of man in DNA, and they actually keep it in the French National Archives here.

29:25Wow. And you can make copies very easily and very cheaply. But it is currently stored safely in a vacuum tube in the archives here. So why aren't more people doing this now? I would say in the next decade or so, the synthesis costs will drop down sufficiently in order to make this a scalable and affordable alternative. But for the time being, it's simply too expensive. But if we can store critical archival data, that's really the ideal solution for DNA, at least in the next decade or two. How expensive is it right now? I mean, it's insanely expensive. It's kind of a moving target, but we're talking on the order of thousands to millions or billions of dollars just to store even text documents.

30:19To give you a concrete example, in our study back in 2017, we stored two megabytes of data. We basically compressed a few files, including an Amazon gift card, one of the first movies ever made, and an operating system. All that totaled to two megabytes, and it cost about$7 ,000 just to synthesize that. So it's still way too expensive. But this is where scientists are once again turning to nature. So instead of using this very costly chemical synthesis process, there are actually enzymes that exist in nature that synthesize DNA. And so there are quite a few groups and companies that are specifically working on improving that method of synthesis.