AGI on the Horizon: DeepMind's Revelation in Lossless Compression Algorithms

0:00So the headline here is that DeepMind's chinchilla is a 70b model and it has crushed traditional compression algorithms and opening it's I think really kind of opening new avenues for AI. utility. So data compression has long been a kind of cornerstone of computing, enabling the efficient storage and transfer of huge, you know, sets of data. I mean, if you've watched the TV show Silicon Valley, you'll know it's all about, you know, their fake company was all about like this company that could do this really impressive compression stuff. But pretty much getting files to be as small as possible makes a big difference, right?

0:35Like if you're able to watch a YouTube video and let's say the YouTube video is, you know, a tenth of the size because it was super compressed, but the quality was just as good. Everything would load faster. There'd be no buffering. There'd be, it just solves a whole lot of the problems on the internet today. If we were able to compress things more. So apparently AI is quite good at this, alarmingly good. Some people say, but this new study from deep mind, which is of course the subsidiary that Google purchased, um, the AI studio that Google owns now, um, the study by DeepMind suggests that machine learning models previously renowned for, of course, text-based tasks may actually outperform traditional algorithms in compressing data, and in some cases, significantly so.

1:16So the study published on ARXIV, this is kind of like, how do I explain it? If you don't know, ARXIV is like where everyone's publishing studies now. They don't have to be super peer-reviewed, and you just kind of get them out quick and dirty there. and so some people kind of question the the validity of things that are published there so for example if you remember a while back when we had like the whole um the whole like what was it like that magnet um oh yeah it was room temperature um superconductor material or whatever that was published the paper on that was published on arxiv i think now it's been disproven i don't think it was actually truly what they said it was but that's where it came amount.

1:58So just so you know, in any case, this study they published is titled language modeling in compression. And it asserts that DeepMind's large language model known as Chinchilla 70b can perform lossless compression on various data types more effectively than established algorithms. So specifically when tested on image patches from the ImageNet image database, Chinchilla compressed the data to 43 % of its original size. And this was outperforming the PNG algorithm. So PNG, as you may know, is kind of a file type that a lot of images are made in. I think like maybe the most standard and common that you'll see an image is JPEG, right?

2:40And then PNG is kind of famous because it can do transparent backgrounds. But yeah, PNG is a more compressed algorithm for these images. But even more than PNG, this chinchilla is compressing it like 43 % more than an image's original size, and it's beating the PNG algorithm right now, which achieved, I think, 58%, only achieved 58%. So even more startlingly for audio samples from the Libra speech dataset, set chinchilla managed to compress them to a mere 16 percent of their raw size which was outshining flax 30 percent so chinchilla and using ai to create an algorithm to compress is literally beating some of these famous algorithms that we've had so in this this is what they said quote in this case lower numbers is the results in the result means more compression is taking place and lossless compression means that no data is lost during the compression process.

3:42So that's what the paper was kind of clarifying. I think this finding stands in contrast to lossy techniques like JPEG, which trade data fidelity for file size reduction. So you can get essentially smaller file size in JPEG, but as it gets smaller, you're, of course, losing fidelity. The image is getting worse. So what's really interesting here is they're able to compress this smaller, the algorithms they use compress the images smaller, but there is no loss in data, meaning the image is just as high quality as kind of the raw original file. So I think what makes these findings really fascinating is that Chinchilla's 70B was primarily trained for text-based tasks.

4:24However, its capability to outperform algorithms, you know, specifically designed for image and audio compression underscore its versatility and opens up some new conversations about machine learning models being more than just hex prediction tools, right? So when we talk about chat GPT or GPT-4, we're like, yeah, this thing's great. It's just, you know, it's not that like intelligent. It's just kind of predicting, it's just predicting the next word in a sequence. Okay, well, you know, how come these things are able to create really complex algorithms that are smarter than any human created algorithms we have so far, right?

4:59So it's definitely bringing up this conversation of, you know, what more can it do? Why is it capable of writing these algorithms? I think this revelation kind of aligns with broader perspectives within computer science that associate data compression efficacy with general intelligence. So the thesis is that compressing data effectively requires identifying patterns and making sense of complexity, much like how a kind of form of understanding or representation of the world operates. So let's talk a little bit about the Hutter Prize and the intelligence compression nexus. You're like, what the heck is that?

5:35So the concept of linking compression ability to general intelligence isn't new. So this has actually received some scholarly attention through initiatives like the Hutter Prize. This was named after Marcus Hutter, who is a researcher in the field of AI and one of the authors of the DeepMind paper. However, this prize aims to incentivize the most efficient compression of a fixed set of English text. So the rationale is that efficient text compression would necessitate an understanding of semantic and syntactic patterns in language. so this is not dissimilar from kind of human comprehension right like people really do say like this is complex stuff and some people say like humans really have to understand what's going on this isn't just like text prediction this model would really have to understand so um i think the paper doesn't stop at compression deep mind researchers argue that the relationship between data prediction and compression is bi-directional um so that is to say you know a robust comp compression algorithm like GZIP could theoretically be reverse engineered to generate new original data based on what it has essentially learned.

6:48So in one experiment, both GZIP and Chinchilla were tested to predict what comes next in a sequence of data. And not surprisingly, while GZIP generated, you know, completely nonsensical output, Chinchilla displayed far better generative capabilities. So though DeepMind's research has not yet undergone peer review, it has raised I think some really intriguing questions and opens the door to possible new applications for machine learning models in fields beyond text. I think given the ongoing debate and studies examining the relationship between compression and intelligence, it's likely that this won't be the last we hear of this topic.

7:25And I think that as data continues to grow exponentially in both volume and complexity, efficient compression is going to become increasingly essential, right? You think every time the new iPhone comes out and it gets higher and higher quality, we get, you know, more megapixel cameras like all of our data and everything we create all of our content our image photos everything is just increasing in size and I think with that increasing in compression is really important so if DeepMind's chinchillas 70b is any indicator machine learning models just might be a game changer that we have been waiting for in this field so definitely something and will continue to follow in the future.