How AI is decoding ancient scrolls | Julian Schilliger and Youssef Nader

0:06You're listening to TED Talks Daily, where we bring you new ideas to spark your curiosity every day. I'm your host, Elise Hu. Today's speakers did something rather remarkable. They used advanced AI and computer vision to virtually unroll ancient scrolls buried long ago by the eruption of Mount Vesuvius. Machine learning researcher Yusuf Mohamed Nader and digital archaeologist Julian Schilliger shed light on their approach and explain how AI can open up new possibilities for understanding ancient knowledge.

0:46We always think about the potential of AI changing the future. But what about the potential of AI changing the past? My name is Yusuf Nader. I'm an Egyptian AI researcher and a PhD student at the Free University in Berlin. And last year, I led the Vesuvius grand prize winning team on exploring this very question. You see, the story starts almost 2 ,000 years ago. A Greek philosopher that we believe was Philodemus of Gadara sat in one of the many rooms of the Villa de Papiri. He talked about music. He talked about pleasure. he talked about what makes things enjoyable, questions that still plague us until today.

1:32One of his scribes wrote down his thoughts on sheets of papyrus. The sheets were rolled and stowed away for later generations. Fast forward 150 years, not even more. Mount Vesuvius erupts, burying Herculaneum, the villa, and the words of the philosopher, under a sea of hot mud and ashes. Now fast forward again to the 17th century. People are excavating around the area. They found beautiful statues, breathtaking frescoes, and some weird-looking pieces of charcoal. This is when the first scrolls were discovered, and people were racing to excavate more of these. What knowledge is included that is not known to us now?

2:23What things should we know about these scrolls? My name is Julian, and I am a digital archaeologist. When the pyroclastic flow hit the scrolls, it had a destructive effect. It tore into them, shred off pieces, and it charred them badly. People 250-something years ago were curious what's lying inside those scrolls, hidden and not accessible anymore. Because of a lack of technology, they had to resort to physically unrolling and thereby destroying most of the scrolls. To this day, only the most damaged and deformed scrolls remain in their initial rolled-up configuration. Fast-forwarding a little bit, the computer age arrives.

3:26Yousef and I are born. We are going on and getting our education. And, well, at the same time, Brent Seals, a researcher and professor, had the idea to use CT scan technology to actually digitalize the scrolls with the hope of one day digitally unrolling them. It's a difficult question how to unroll this digitally. Nat Friedman, a Silicon Valley investor, also saw this research, and he wanted to help. That was in 2022. He reached out and together with Brent Seals, they created the Vesuvius Challenge with the goal to motivate nerds all over the world to solve this problem. They created a grand prize, promising eternal glory and monetary incentives to anyone who could do that.

4:28I myself, I saw that in the internet while writing my master thesis at ETH Zurich in robotics and I was instantly happy to solve it. At least try, you know. And I went on, joined the Discord community where all the people that were also contestants and playing with the scroll data were exchanging ideas. I joined there and started working on it. Also, there on Discord, I met Yousef and Luke, who would become my teammates and with whom I would actually win the grand prize. Surprisingly, it went on and made global headline news. It even got into the British tabloids.

5:13So when we started, there were two main problems still remaining. One, you had to unroll the scroll, and two, you then had to make the ink visible. Yousef will tell you more about that part. For me, the most exciting thing was the computer vision problem of unrolling those scrolls virtually. I decided to iterate on a tool that was created by the Kentucky researchers and make it faster, less prone to errors, and just iterate on it and make it better. The Vesuvius Challenge team saw that and also implemented a team of 10 people that would use my tool. They would annotate scroll data where they created a red line where the surface would lay.

5:58The algorithm then would take it into 3D space, creating a three-dimensional representation of the surface. Computer algorithms would then flatten it and create a segment. This all would be called segmentation in the space of the scrolling and unrolling community.

6:25so I created open source commits to this tool and implemented new algorithms from my studies like optical flow to better track the sheets during the through the volume first off those were really small segments and I added improvement made the code faster and had lots of feedback from the community. They were really happy, and I was happy getting lots of feedback. It was a really positive environment. So in the end, I could track the performance of the algorithms, how the segmentation team performed, and I could see that my improvements from start to finish would be around about a 10 ,000-fold improvement over the initial version.

7:12This algorithm was then also used to unroll all the area that you can see in our submission, all the sheets were generated with these methods. In December, I was looking for teammates. I made a blog post and I showcased my newest algorithms, reaching out to anyone that was willing to team up. Yousef and Luke got into contact with me. They were happy to team up and I was happy as well. So, After the virtual unwrapping, the words still are not visible. The main problem is that the ink that was used at the time was a carbon-based ink. And carbon-based ink on carbon-based papyrus in a CT scan isn't visible, or at least to the naked eye.

8:03So the same team at the University of Kentucky decided to test whether the ink is present at all in the CT scans. For this, they took some of the pieces that people broke off the scrolls, and they fed them into the same pipeline of the X-ray CT scanning, and this gives us the 3D data that we're working with. You can also, because you can see the ink, and it's like an exposed surface, you can even improve it with infrared imaging, and this gives you a ground truth of what letters you're actually trying to find. And then from there, you can train a machine learning model to try to find these letters.

8:39The way this works is that the model looks at very small cubes at a single time and tries to decide whether there is ink present in this area or not. And then when you keep moving this cube all around, the model gets to see different data samples and then tries to understand what ink actually is. So this is how it looks while the model is training. The model is starting to see the letters perfectly. So the data is there, the ink is there, but it's just very hard to find and see. There's a different way that you can find this ink, one that actually scales very, very well. So this is where my journey begins with the Vesuvius Challenge.

9:21There is this neat idea in computer vision literature where if you don't actually have labels, if you don't have the goal that you want your AI model to reach, you can pick an intermediary goal along the way. So one idea to just let it know about the structures and familiarize it with the data is to show it different views of the same image and tell it that these are the same images. And after that, you take this model and you train it like the previous models that the University of Kentucky did. And while the approach doesn't fully work, it also doesn't fully not work. And the first image that was produced by the model, there were some very faint signal in there.

10:02It seemed like the model is catching on something, but it wasn't clear exactly what the model is catching on. So I decided to take these predictions and create a new ground truth, asking the model, hey, I think these might be letters, I think there's something in there, try to find more of this. And my ground truth actually has four correct letters and four other delusions, but that was okay. So training a new model with this data, the model started to find more ink, find more letters, and the lines even looked complete. So I thought, what are the chances that if I do this again, the models keep improving?

10:42And this was the core behind our grand prize winning solution. Repeating this process over and over, the models kept improving. The main trick was you needed to prevent the models from memorizing what the previous models have learned. You're essentially asking the model to learn what the other model has learned. So overfitting was a serious problem that required a lot of experiments. But in the end, getting the recipe right, we were able to predict all of these letters without the models ever seeing them. These were the first 10 letters. This was the first coherent word read from an unopened papyrus sheet.

11:24From there, scaling the process, within weeks, we had now columns of text. even special characters that the paperologist found very interesting that the model was able to find. The approach was open-sourced, and the data and the code was out there, and the race for the grand prize was on, recovering four paragraphs at an 85 % clarity. And the key to our success was perfecting the data and the model with so many iterations and so many experiments. In the end, we were able to recover more than 14 columns of text and 2 ,000 letters.

12:102 ,000 characters safely stored away two millennia ago. In just nine months, we discovered them again. AI helped us in large portions, writing better code and even being part in our algorithms. It opened a window into the past. What's next? Let's open this window more. AI will help us access information that was so far safely locked away. In the words of the author, we do not refrain from questioning nor understanding. And may it be evident to say true things. As they appear.

13:28Martha Estefanos, Oliver Friedman, Brian Green, Autumn Thompson, and Alejandra Salazar. It was mixed by Christopher Faze Bogan. Additional support from Emma Taubner and Daniela Balarezo. I'm Elise Hugh. I'll be back tomorrow with a fresh idea for your feet. Thanks for listening.