1. Podcasts
  2. Best AI papers explained
  3. Deep Learning is Not So Mysterious or Different
Best AI papers explained

25 Jun 2025 · 22m · 13 chapters

Episode artwork

Deep Learning is Not So Mysterious or Different

In short

Explains major “mysteries” of deep learning generalization—benign overfitting, over-parameterization, and double descent—using unifying ideas like simplicity/compression bias and “soft inductive biases,” then contrasts this with what’s genuinely distinctive about deep neural nets (representation learning, universal learning, and mode connectivity).

Guests

No guests are named; the episode is a discussion between two hosts (“Deep Dive” hosts).

Key claims

Deep nets can fit noisy labels yet generalize because among zero-training-error solutions they prefer simpler/compressible ones; similar effects occur in linear models and Gaussian processes (e.g., GP fitting noisy CIFAR-10 like ResNets). Over-parameterization works via flexibility plus compression bias (lower effective dimensionality/Kolmogorov complexity). Double descent arises when effective dimensionality decreases after interpolation due to a growing volume of flat, compressible solutions. PAC-Bayes bounds relate expected test error to empirical risk plus a compressibility term (description length/file size).

Notable examples

CIFAR-10 noisy-label fitting; figure references to GP vs ResNet, double descent curve, and mode connectivity; transformers becoming translation-equivariant; context learning (LLMs doing forecasting from prompt examples).

Written by AI. May contain mistakes.

Chapters

0:45

Unpacking Deep Learning Mysteries

Exploring how deep learning challenges traditional concepts in machine learning.

“Or even that mysterious at all when you look closer.”
3:00

Understanding Benign Overfitting

Discussion about benign overfitting and its implications for deep networks.

“And this idea of model preference leads nicely into the next puzzle.”
4:55

Over-Parameterization Explained

How over-parameterization can lead to better generalization in models.

“So bigger models find simpler solutions in a way.”
7:18

Double Descent Phenomenon

Explaining the double descent phenomenon and its implications for model complexity.

“Random forests, linear models, the source shows examples in figure one.”
8:57

From Hard Constraints to Soft Inductive Biases

Transition from hard constraints in traditional learning to soft biases in deep learning.

“So traditional machine learning often relied on restriction biases.”
10:14

Frameworks for Explaining Generalization

Introducing PAC-Bayes and its relevance to understanding model generalization.

“We already mentioned order-dependent regularization in polynomials canalizing higher-order terms more heavily.”
12:56

Deep Learning's Unique Capabilities

Discussion on what makes deep learning models distinct despite generalization parallels.

“Formally, it's often expressed using Kale divergence between a prior and posterior distribution over hypotheses, but intuitively it measures how much information is needed to specify the learned model.”
14:00

Understanding Deep Learning Generalization

Explore whether deep learning is truly unique or just a rehash of old ideas.

“Because the compressibility term gets smaller, even as the empirical risk stays near zero.”
14:37

The Power of Representation Learning

Learn how deep learning's ability to create representations sets it apart.

“But does this mean deep learning isn't special at all?”
16:15

Universal Learning in AI Models

Discover the trend of universal learning in AI and its implications.

“Now, is representation learning totally unique?”
Show all 13 chapters
18:09

Mode Connectivity and Its Significance

Examine mode connectivity in deep learning and its impact on model training.

“Representation learning, universal learning, anything else truly distinctive.”
19:15

Bridging Communities in AI Research

Understand the call for collaboration across different fields in AI research.

“It changes how we think about the loss landscape, why different solutions generalize similarly, and it's inspired practical techniques like stochastic weight averaging, SWA.”
20:39

Future Directions in Deep Learning

Explore future work needed to understand the mystery of neural networks.

“A call for intellectual humility and cross-pollination.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Best AI papers explained, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00Welcome to the Deep Dive, your shortcut to being truly well informed. Great to be here. Today, we're taking a deep dive into, well, deep learning, specifically a recent paper that opens with a real bombshell. The textbooks must be rewritten. It's quite an opening, isn't it? Really grabs you. It does. And it kind of perfectly captures how deep learning has, you know, turned a lot of our standard machine learning ideas upside down. Yeah. For a long time, things like overfitting, generalization, they felt like settled concepts. Exactly. Like gospel. But then deep neural networks came along and seemed to just ignore the rules.

0:38Right. So our mission today in this deep dive is to unpack this argument that maybe these mysteries aren't actually unique to deep nets. Or even that mysterious at all when you look closer. Precisely. We're going to dig into the specific things people found confusing and look at the unifying ideas that actually explain them using the sources you shared. Okay, let's chip in. First up, one of the big head scratchers. Benign overfitting. Ah, yes. Benign overfitting. So this is the idea, right? That deep networks can perfectly fit noisy data, like even random labels. Zero training error. Which sounds like it should be terrible for performance on new data.

1:14Absolute memorization. Totally. Recipe for disaster. Yeah. But somehow these networks still generalize pretty well on, you know, actual structured real world data. It really did seem like a contradiction for a while. How can you memorize garbage but still understand the real world? But here's the twist and what the source material highlights. This isn't some unique magic of neural networks. Not at all. You can actually get the same behavior from, believe it or not, simple linear models. Simple linear models? Really? Yeah, or even high-order polynomials if they have a certain kind of bias towards simplicity baked in.

1:48Okay, hold on. A basic polynomial doing what we thought was a superpower of complex deep nets. It surprised me too, initially. But the source shows it clearly, like with a Gaussian process, a GP. Oh, right. I remember figure one and the source showing that. The GP fitting noisy CIFAR-10 labels just like a ResNet. Exactly. Both fitting the noise perfectly, getting zero training error, but the GP still generalizes reasonably well. So what's the core idea then? What allows this seemingly paradoxical behavior? It boils down to basically three things. First, you need a really flexible model, one that can fit anything, including noise.

2:26Okay, a flexible hypothesis space. Makes sense. Second, a loss function that really pushes the model to fit the training data perfectly. Zero tolerance for error. Right, driving that training error down to zero. And third, this is the crucial part. Among all the solutions that fit perfectly, the model has to have a preference for simpler ones. A simplicity bias. Ah, so it's that preference that saves it from just outputting the memorized noise. It finds the simplest explanation that still fits everything. You got it. It's actively seeking simplicity within the set of perfect fits. That makes a lot more sense.

3:00And this idea of model preference leads nicely into the next puzzle. Over-parentization. Right. The fact that models with way, way more parameters than data points can work incredibly well. Which just feels wrong based on classical stats, right? More parameters, more complexity. Surely that means more overfitting. That's the classical intuition, yeah. But again, this isn't actually a brand new idea exclusive to deep learning. Oh, how so? Well, people in Bayesian statistics and non-parametrics, they've been dealing with models that effectively have more parameters than data points for decades. Like Gaussian processes again.

3:35Exactly. GPs theoretically have infinite parameters. And there's a known connection between GPs and certain kinds of infinite-width neural networks. So the core idea has been around. Okay, history lesson noted. But why does it work so well in deep learning? Why does adding parameters often seem to help generalization, not hurt it? So the source points to two main reasons. First is just flexibility. More parameters simply give the model more power to capture complex patterns in the data. Okay, that's intuitive enough. More knobs to tune. But the second reason is a really interesting one, something called compression bias.

4:10Compression bias, tell me more. It turns out that larger models, perhaps counterintuitively, often develop a stronger bias towards solutions that are more compressible. Meaning they can be stored more efficiently. Essentially, yes. After training, even though they have billions of parameters on paper, the actual learned solution can be described or stored using less information. It has lower Kolmogorov complexity. So the effective complexity is lower even if the parameter count is huge. Exactly. They might have fewer active parameters, or the parameters they do have are arranged in a way that's simpler to describe.

4:48And this compression, this simplicity, is linked directly to better generalization. Figure sets in the source gives a nice visual for this. Wow. So bigger models find simpler solutions in a way. That's wild. And there's empirical evidence backing this up too, like findings that larger models often have smaller effective dimensionality. Right. I saw that reference. Maddox and others in 2020. And also a stronger bias towards low Komagorov complexity solutions, which Goldblum and colleagues showed more recently. Okay, that helps explain over-parameterization. And it also touches on another point, the role of optimizers.

5:21There was this idea that stochastic gradient descent, SGD, with its randomness, was essential for generalization. Yeah, that was a popular narrative for a while, the implicit bias of the optimizer itself finding good solutions. But the source kind of throws cold water on that being the only explanation, right? It does. It points out that you can get competitive generalization even with full batch gradient descent, which is totally deterministic, no randomness. Or even apparently with just guess and check methods. Yeah, which is pretty surprising. It's just the magic isn't solely in the stochasticity of the optimizer, although that can certainly play a role.

5:58Other factors like this compression bias are also key. Okay, another layer peeled back. Let's move to double descent. Yeah. This one has a striking grass. Oh yeah, the U shape that turns into a W shape, kind of. Describe the curve for us. What happens? So as you increase model complexity or parameters, the test error first goes down, which is good. That's the classical learning curve. Less bias, better fit, standard stuff. Then error starts to go up. This is the classical overfitting regime. Too much complexity, fitting noise. The peak of the U, where we used to think you should stop. Exactly.

6:33But then, as you keep adding even more complexity, past the point where the model perfectly fits the training data, the interpolation threshold, the test error starts to go down again. Down again, after overfitting. That's the modern interpolating regime. Right, and often the training error is basically zero during this whole second descent phase. And that felt like black magic. How can adding more power possibly fix overfitting after it already happened? It seemed to break the bias-variance tradeoff. It really challenged conventional wisdom, but guess what? Let me guess. It's not actually new or unique to deep learning.

7:07Bingo! Double descent phenomena were observed way back, like, late 1980s. Seriously? Decades ago. Yep. And just, like, benign overfitting, you can see it in other models, too. Random forests, linear models, the source shows examples in figure one. Okay, so if it's old news and happens elsewhere, what's the explanation? Why does the error go down again? The source uses this concept of effective dimensionality, which we touched on. Right. The idea that the model's actual complexity isn't just the parameter count. Exactly. So initially, as you add parameters, the effective dimensionality goes up.

7:41The model gets better at fitting complex stuff. Error goes down, then up. Classical regime. But past the interpolation point, something else happens. The space of possible solutions that fit the data perfectly becomes huge. And within that huge space, the volume of flat solutions, solutions that are smoother, more compressible, and generalized better, starts to grow significantly. Ah, so there are just more good solutions available in that over-parameterized regime. And they become easier for the optimization process to find. So even though the raw parameter count is still going up, the effective dimensionality of the solutions being found actually starts to decrease.

8:18The model finds these simpler, more compressible fits within the vast space. So it's not just getting bigger. It's finding more efficient ways to use that size. Finding flatter, simpler, minima in the lost landscape. Precisely. Which leads to that second descent in error. It's less about parameter counting and more about the characteristics of the solutions the model converges to. That's a really elegant explanation. Yeah. And it feels like all these threads, benign, overfitting, overparameterization, double descent, are pointing towards a common theme. They are. And the source frames this unifying principle as a shift from hard constraints to soft inductive biases.

9:01Okay, break that down. What's the difference? So traditional machine learning often relied on restriction biases. Think about a convolutional neural network, a CNN. Right. You build in assumptions like locality and translation equivariance through the architecture itself. Hard constraints. Exactly. You're telling the model, you must process information this way. These are the only kinds of patterns you can look for. You're restricting the hypothesis space up front. Okay, so that's the old way, the hard restriction. What's the soft inductive bias there? Instead of rigidly limiting what the model can represent, a soft bias specifies a preference.

9:35The model could represent many complex things, but it's gently guided or nudged towards certain types of solutions, usually simpler, smoother, or more compressible ones. Even if other, more complex solutions fit the training data just as well. Exactly. It's like there's a path of least resistance during training that leads towards these preferred solutions. Figure 3 and the source contrast this nicely. A hard wall versus a gentle slope. Less, you can't go there, more, this way is easier. You got it. It's a learned preference or an emergent property of the training dynamics, the architecture, or maybe even the data itself.

10:11And how is this soft bias actually implemented, or how does it arise? The sources give several examples. We already mentioned order-dependent regularization in polynomials canalizing higher-order terms more heavily. That's a soft bias for lower-order fits. Makes sense. What else? There are things like residual pathway priors, RPPs. They take hard architectural constraints like equivariants and turn them into soft preferences. And the interesting thing there is that sometimes these soft biases actually work better than hard ones. Yes, especially if the symmetry or property in the data isn't perfect.

10:44A soft bias can adapt better than a rigid constraint. Fascinating. And even overparameterization itself, as we discussed, can implicitly create a soft bias towards these flatter, more compressible solutions. So the bigness itself helps to find simplicity. In a way, yes. And maybe the most striking example is vision transformers, VTs. Ah, right. The source mentioned this. VTs have fewer built-in constraints than CNNs. Much fewer. They're more general purpose. But after training, they can actually end up being more translation equivariant than CNNs. Which is totally backwards from what you'd expect based on architecture alone.

11:21Exactly. And the explanation seems to be this emergent soft bias towards compression, which, in the context of vision, often leads to discovering things like translation equivariants because it's an efficient way to represent visual data. Wow. OK, so this shift to thinking about soft biases really changes how we evaluate models and generalization. It does. It means older frameworks like VC dimension or radomarker complexity, which focus on the size of the hypothesis space. They just can't explain these phenomena. They're tied to that restriction bias mindset. They predict that bigger models should always generalize worse, basically.

11:55Right. So if those don't work, what does explain deep learning generalization? And the answer, again, perhaps surprisingly, comes from frameworks that have also been around for a while. PAC bays and related countable hypothesis bounds. Decades, yeah. Not brand new ideas invented just for deep learning. So what's the key insight in PAC bays that makes it work here? Instead of just penalizing the size of the hypothesis space, these frameworks focus on the distribution over hypotheses. They consider which solutions the learning algorithm is likely to find. Okay, so it's not just what can be represented, but what's favored by the learning process.

12:31That sounds much more relevant. It is, and the core intuition can be distilled, roughly, into an inequality. Your expected error on new data, expected risk, is less than or equal to your error on the training data, empirical risk, plus a term related to model compressibility. Empirical risk plus compressibility. And compressibility here is related to that Kolmogorov complexity idea. Exactly. Formally, it's often expressed using Kale divergence between a prior and posterior distribution over hypotheses, but intuitively it measures how much information is needed to specify the learned model. And crucially, this can be upper bounded by the actual file size of the stored model parameters.

13:14So even a giant model, like an LLM with billions of parameters, can generalize well if? If it fixed the training data well, low empirical risk, and the learned solution is highly compressible, can be stored in a relatively small file size. Figure 2 in the source illustrates this connection. That's a profound shift. It means complexity isn't just parameter count, it's description length. Precisely. And this framework handles many of the mysteries. It works for deterministically trained models, not just stochastic ones. Okay, that's important. It can give meaningful, non-vacuous guarantees even for these huge models.

13:45Unlike VC dimension, which would just give uselessly loose bounds. Right. And it handles continuous parameters because on a computer they are ultimately represented with finite precision. Good point. Plus, counterintuitively, the bounds can sometimes get tighter for larger models precisely because of that stronger compression bias we talked about. Because the compressibility term gets smaller, even as the empirical risk stays near zero. Exactly. And finally, it's computable, at least in principle, using file size as an upper bound on Kolmogorov complexity. It's a powerful, practical framework that aligns much better with what we see in deep learning.

14:23Okay, so we've spent a lot of time showing what isn't necessarily unique or mysterious about deep learning generalization, over-parameterization, benign overfitting, double descent explained by soft biases, and pack base. Right, demystifying some of the headline phenomena. But does this mean deep learning isn't special at all? Just old wine and new bottles? Oh, definitely not. While those specific generalization puzzles might be explainable by broader principles, deep neural networks do have some truly distinct and, frankly, still quite amazing capabilities. Okay, let's pivot to that. What does set deep learning apart beyond just getting great results?

15:01I think the first major one is representation learning. Learning the features themselves. Exactly. Most classical models work with fixed basis functions. Think linear regression, SVMs with fixed kernels, polynomials. The way you represent the input X is decided beforehand. You engineer the features or choose a kernel. Right. But neural networks learn their own representation. The source puts it as learning X, where V are learned parameters within the basis functions themselves. They learn how to look at the data. They learn a similarity metric tailored to the task. Precisely. And this is incredibly powerful for high-dimensional complex data like images, audio, text, where defining good features or similarity metrics by hand is almost impossible.

15:45Because standard distance metrics just break down in high dimensions. They do. And this ability to learn representations enables something more profound than just interpolation between seen data points. It allows for extrapolation. Making predictions far outside the range of the training data. In a sense, yes. The source gives the example of recognizing a face rotated 60 degrees, even if the model only saw rotations up to, say, 20 degrees. Because it's learned something fundamental about faceness and rotation, not just memorize specific angles. Exactly. Now, is representation learning totally unique?

16:18No. Kernel learning methods try to do something similar, but Ennins seem remarkably efficient and effective at learning these representations, especially complex, non-stationary ones. directly from data in high dimensions. Okay, so adaptive representation learning is one key differentiator. What else? Another huge one is the trend towards universal learning. Challenging the no-free-lunch theorems. Pretty much. The old wisdom stemming from no-free-lunch was that you need specialized learners for specialized tasks. A model great at image recognition wouldn't be great at text, and vice versa. You needed strong inductive biases tailored to the domain.

16:55But that's not what we're seeing now, is it? Not really. we see models like transformers achieving state-of-the-art results across incredibly diverse domains, language, vision, reinforcement learning, even scientific discovery. One architecture seems capable of tackling almost anything. How is that possible? How does it avoid the no-free-lunch trap? The hypothesis, again linking back to Kolmogorov complexity, is that both neural networks, especially large ones, and natural data distributions share a fundamental bias towards low complexity, towards compressible structures. So the models are intrinsically suited to finding the kinds of patterns that exist in the real world.

17:37That seems to be a big part of it. And the most stunning example of this universality is probably in context learning in large language models. Right, where you just give the model a few examples in the prompt and it figures out the task on the fly without any weight updates. Exactly. An LLM trained only on text can perform competitively on tasks like time series forecasting. just from examples in the prompt. It's like they've learned how to learn general inductive principles during pre-training. They become these incredibly flexible meta-learners. That feels genuinely new. It really does. It's a level of universality and adaptation that we haven't seen before.

18:10Okay. Representation learning, universal learning, anything else truly distinctive. There's one more phenomenon mentioned in the source that's quite unique and still a bit mysterious. Mode connectivity. Ah, yes. This one is about the lost landscape. All right. All right. The traditional view of training deep networks was that you'd run the training multiple times with different random initializations, and you'd end up in different local minima, different solutions separated by high-loss barriers, like isolated valleys. Made sense for non-convex optimization, lots of places to get stuck. But then researchers discovered something surprising.

18:46If you take two different solutions found by independent training runs, you can often find a simple path between them in the weight space, along which the loss remains remarkably low, near zero. So the seemingly separate valleys are actually connected by pathways at the bottom. Figure 7 in the source shows this visually. Exactly. It suggests the low-loss regions are not isolated points, but part of a large connected manifold or basin. What are the implications of that? It's profound. It changes how we think about the loss landscape, why different solutions generalize similarly, and it's inspired practical techniques like stochastic weight averaging, SWA.

19:25Where you average weights along these paths. And also model merging, where you can actually average the weights of independently trained models to often get even better performance. That wouldn't work if they were in truly isolated basins. Precisely. So while maybe not entirely unique in all of optimization history, this specific kind of high dimensional mode connectivity seems largely characteristic of deep learning's complex lost landscapes. It's a distinct feature. Okay, so let's try and wrap this up. The main takeaway from this deep dive seems to be... That while phenomena like overparameterization, benign overfitting, and double descent looked really mysterious at first and challenged old intuitions.

20:05They aren't actually magic tricks exclusive to deep learning. Right. They can largely be understood through the lens of longstanding ideas like soft inductive biases and generalization frameworks like PAC bays that focus on solution complexity or compressibility rather than just model size. It's about appreciating that these principles apply more broadly. Exactly. And the paper really calls for bridging communities' deep learning, statistics, information theory, and recognizing that progress has often been shared, even if the language was different. We need to connect these dots. A call for intellectual humility and cross-pollination.

20:42I like it. Me too. So, as we finish, maybe a final provocative thought for you, our listener, to chew on. We've demystified some things today. But the journey isn't over. The source suggests future work should focus more on analyzing the properties of the actual solutions that neural networks find. Not just the training process, but the destination. Right. And investigating specific priors or biases, like this compression bias, that might lead to even tighter, more informative generalization bounds. So maybe the real mystery isn't that deep networks generalize despite their size, but understanding the intricate details of how they find such incredibly effective, often compressible solutions.

21:21And perhaps how much further can we push these principles? Can we design systems that are even better at universal adaptive learning, that truly learn how to learn across any domain? The potential for truly general artificial intelligence built on these foundations. That's the next frontier. Indeed, it feels like we're just scratching the surface of what these universal learning machines might be capable of.

From the publisher

This position paper, "Deep Learning is Not So Mysterious or Different" by Andrew Gordon Wilson, argues against the notion that deep neural networks exhibit unique or mysterious generalization behaviors like benign overfitting, double descent, and overparametrization. The author contends that these phenomena are not exclusive to deep learning and can be understood and formally characterized by long-standing generalization frameworks, such as PAC-Bayes and countable hypothesis bounds, rather than requiring a re-evaluation of established generalization theories. A central unifying principle proposed is soft inductive biases, which embrace flexible hypothesis spaces with a preference for simpler solutions consistent with data, as opposed to restrictive biases. While highlighting these commonalities, the text acknowledges that deep learning possesses distinct characteristics such as representation learning, universal learning, and mode connectivity, which still warrant further investigation. Ultimately, the piece seeks to bridge understanding across different machine learning communities by demonstrating that many perceived "mysteries" of deep learning are explainable through existing theoretical frameworks and are reproducible with simpler model classes.

More from Best AI papers explained

All episodes