Grokking, Generalization Collapse, and the Dynamics of Training Deep Neural Networks with Charles Martin - #734

26:42That's what's going on inside the brain. That's a theory. It's called the critical brain hypothesis. And so we can see these signatures of criticality, I call them signatures of emergence of AI, inside actual neurons. And it turns out our experimental work shows that the closer the quality, I have this layer quality metric, there's a sweet spot at two. There's a value of two. And when all the layers reach two, we think that the model is perfectly optimized. And the goal of the theory is to try to get your model so that all the layers reach two. And that's what we're trying to do, try to develop technology to do that.

27:20But, you know, here you can, and that's sort of, And we see it in, you know, just really basic experiments on things like grokking and double descent, where you can really flush it all out. But, you know, you have to, you know, we try to study small models and then apply the results to larger models. And then we go back and try to figure out what's going on. And that's what we've been doing. And it's been very successful. There's a lot of open questions. But, you know, the idea of it is, I give people, so that's what the spiking neurons are talking about, is that you can measure these signatures, and you can see them in the actual models you're training.

27:54And you can use them. You can use them, make better models. So you've mentioned grokking a couple of times. What's grokking? Grokking is a phenomenon where if you take a model and you train it for, you know, some small data set, it tends to reproduce the training, it has perfect training accuracy, zero error. And people think that it's somehow memorizing the training set. And then it has almost no test. And the test error is like as big as it could be. It's horrible, right? And then all of a sudden, it very quickly learns how to generalize. And so the training accuracy stays high, but then the test accuracy gets high.

28:36So my test accuracy might get like 85%, you know, 90%. Not super high, but it gets very high for a small data set. That's crocking. And it's something that people, it theoretically is very odd because it's odd that you would get a model that can describe the training data perfectly. You think, well, it must be overfit. It must have memorized the data. And then all of a sudden it's able to generalize. So the kind of non-linearity in the learning is what's interesting about it. Yeah, a phase transition really just goes from not being able to generalize at all to being able to generalize extremely well.

29:13And it happens just sort of suddenly. And that's called, the grok is to understand something. So they call it grokking. Yeah. And then generalization collapse. Generalization collapse, we detect it. What we define is if you continue training, all of a sudden it stops generalizing. It starts going down again. So it still has memorized the training data in some sense that the training accuracy is perfect. But it starts to learn, it generalizes, and all of a sudden it stops generalizing. And it starts going down. And it might go down to like, you know, instead of like, you know, 10 % accuracy would be random.

29:47It might go down to like 50 % accuracy. So it doesn't get it doesn't get it does a little bit of learning, but it's confused. It's like in this state of confusion, deep confusion. Got it. And these terms come from the title of the paper that we've referred to here. Grokking and Generalization Collapse Insights from HTSR Theory. And HTSR Theory. Heavy tailed self-regularization. You got to have an acronym in science. Every theory needs an acronym. Right. So HTSR. That's the theory. Heavy-tailed self-reportation, work I published with Michael Mahoney back in 2021. So, wow, it's been five years. Wow, it's hard to believe it's that long.

30:26So this is the theory behind Weight Watcher. This is why Weight Watcher works. Part of the theory. So what we discovered is that when people think about a model memorizing, they think, oh, the training accuracy is perfect. It must be memorizing. No, no. We actually, and it turns out that, you know, we discovered this other phase of memorizing, which is more like confusion. And so there's a state of memorization and there's a state of confusion and they're very different. And you can detect them using Weight Watcher. Both states appear to reproduce the training accuracy perfectly. Why would confusion reproduce the training data perfectly?

31:06Okay. So what's happening is, what's happening in these models? It turns out that in the state before grok, and we call it the pre-groking phase, training accuracy is perfect. Why can't it generalize? It turns out some of the layers have good quality, but some of the layers have very bad quality. So what's happened is you've learned the training data, but the layer that learned the training data, the layer that needs to learn how to, the layer that's most important for generalizing hasn't converged. So the idea is that it's not that you are memorizing before and then you've switched from like this thing called memorizing to this thing called generalization.

31:56It's more that part of your network is sufficiently memorized or not sufficiently memorized, but sufficiently learned, right? It's converged. And other critical parts have not yet. And that's why you don't. And that's required for the generalization. Yes. Now, there's kind of a, somebody, someone has suggested, I saw a paper recently that said the reason it doesn't learn, there's like a numerical instability in the softmax. And so it just gets numerically unstable. and it just kind of, it's trying to learn, but it just can't find its way. And you have to train it for a long period of time until it eventually jumps out of this, jumps over the hill and learns.

32:42Confusion or what we call, you know, we call overfitting in Weight Watchers. Like a confusion is that now the layers learn too much information. So they're over-converged. They're over-baked, right? They're burned. You know, they're cooked. They're overcooked. So they've learned too much information and now they're stuck. And all they know is what, and they can't, they get confused as to what's going on because they've learned so much that now they can't generalize. So it's a different thing. So in one case, we see the layers, some of the layers converge and going down and some of them are going to stay up here.

33:18And in the other case, they're all way down here. Like, they're all way down. They're all way too low. And what you want is you want them out here in the safe range. And that's what we're seeing. And that's, and the thing that that could happen in a real model in the real world, if you're fine tuning, you know, you're training that you'll see some of the layers are, say they're down here, they're good, but some of the layers are up here. They haven't learned anything yet. They're stuck. And if you, if you go too long, they all fall down. You, you burn the system, you burn it. So I, to me, I like the analogy of baking a cake because it's like, if you turn the oven up or you leave it in the oven too long, it, the whole thing will burn.

33:53Right. It just, it'll cook, it'll over cake. And so something has gone wrong. There's some numerical instability in the solver. Maybe the soft max is off. Maybe there's some goofiness in the training data. Something went wrong that's preventing the model from converging. And we can see that. And that's a different phenomenon than overfitting. And the goal is to try to, how do you fix it? Yeah, yeah. So a couple of things jump out at me as interesting here. One is this idea of like, I don't know, really just kind of thinking about this as layers, I think, and really kind of locking in on that, I think is interesting.

34:31And then that each of these layers can be independently underfit, overfit, or, you know, in the target zone. That's interesting. but then that kind of leads me to and I'm this is kind of a lead into like to what degree have you looked at these things or is this like you know next steps here but like you know when I think about what I think about like data centric AI or like the focus on data curation as a way to get models to perform better, more efficiently. You could also think of it in this context as like, can I construct an incremental training data set that targets the deficiencies of a particular layer or set of layers?

35:24Yeah, yeah, I think so. I mean, that's the kind, we haven't looked deeply at changing the data set as much as changing the learning rates on the layers or making a new kind of regularizer. But I think absolutely that there's something, and because there's numerical instabilities, data is, right. Right. My take on data centric AI is that these guys went into industry and tried to apply their their stuff. And then they realize that it's just a mess. Right. Like, I've been doing this for 25 years. I tell you right now, you know, you're I can't get an SVM working in production. You think you're getting an AI model working?

35:58You have no idea what's going on in these companies. Right. So that to me is like, you know, that's just they didn't understand like their business model, what they were selling. And, you know, it's that kind of thing where trying to figure out, you know, how to get, you know, how to target the data correctly. That's exactly one of the things we'd like to know, for example, is, you know, this is stuff we'd like to get into more if we could, you know, is can we figure out which parts of the data are being learned and which parts are not? Right. And that you should be able to see. You pass the data through and you see which neurons light up and which ones don't.

36:35Those are the kind of things we'd like to do. you know, doing an ablation study on a model to figure out where in your data set is it insufficient, where, what part of the data do you need to fill in? Maybe like part of your data set is not fully filled in. And if you added more data there, maybe even fake data, you know, auto-generated data, you might be able to improve training, stuff like that. Those are things we'd like to understand better. Um, and you know, we're trying to, you know, and of course I'm trying to raise funding actually right now to do that, to develop some of this technology.

37:02Um, and, and I I think it's just, you know, these are things that require just a lot of experimentation to figure out. But we definitely see it now. Like there's a paper that came out just this week from Stanford by Chris Manning about how if you look at some of these large models like Lama or Quinn, that a bunch of the layers are not learning. They're looking at how the residuals flow through the layers, and they can see that some layers are not really converging. Some layers are not learning anything like, yeah, we told you that like three, four years ago. Like it's on the website. Yeah, I wish they'd used the tool.

37:35But, you know, it's a validation of what we've been saying for some time, is that our technology can detect this without seeing the data. And is there a one-to-one relationship between the characteristic that they were talking about and your characteristic? I'm not sure yet. We're starting to just dig into it now. We've seen cases, like we have a paper that came out, where I guess the broader point is like, This sounds like a really interesting way to characterize individual layers of models, but there are probably a ton of different ways to characterize it. Well, yeah, the main thing that we can do that no one else can do is we know the cutoffs.

38:15We know the bottom, we know the lower bound is two and the upper bound is like six. So you can use any model. You can measure the entropy, the distance. You can do whatever you want. You can see, you know, if you take two models. Are those practical bounds or theoretical bounds? Yeah, those are practical bounds, yeah. They work in production. Yeah. It really does work. Are they empirical or are they based on some theoretical analysis, like mathematical analysis? Combination of both. They're based on a combination. There's some new theory coming out. It uses techniques from theoretical physics called renormalization group.

38:49And so it's a combination of theory and experiment going together. You know, we have theories that show there's a boundary at two. We actually have two different metrics for the lower bound. And they have to line up. And when they line up, then we know. So there's some deep theory, like I got a hundred page theoretical physics paper on this thing to really justify it. But you see it empirically as well. And this crocking paper in particular is an important paper for us. We wrote this nice 10 page paper to really show, look, it works perfectly. And for me, the other thing is that I had somebody else do it with the tool.

39:20So it's independently verified. Right. You know, so these are real things you can use in production. I mean, they're real. So the challenge is, as he says, you have to analyze the data, you know, and that's, it's always tough. I mean, from a consulting perspective, look, I did a large project with Walmart. I tried to help them, you know, fix their search engine, you know, and you find out, oh, you know, you need to deploy your product on our Jupyter Notebook serving system. Okay. But you're a consultant. You're not allowed to have access to our Jupyter Notebook serving because we put all our customer data there, including all their credit card numbers.

39:56and you can't have access to that. Okay. You know, what do you want me to do here? You know, that's very common. The same thing, I did a project years ago, almost 20 years ago for France Telecom. And at the end of the, and we had to only use fake data. I'm not an EU citizen. Since I'm not an EU citizen, I can't access personal data. And so what happens in these big companies is that, you know, they'll stuff all the data on Hadoop into one system. And then it turns out they have compliance issues. GDPR, CCPA. You know, you, you, you can't access things. So it's a huge problem. And so part of Weight Watcher is that I wanted to build a tool that I didn't need access to any data because nobody ever lets me look at it anyway.

40:37So. So it's a problem and these are real production problems, you know, uh, and so part of, you know, trying now to convince people, you know, it's, it's an easier sell to give someone a tool and analyze the model. Cause you know, I mean, in a sense, I don't have the tokenizer. So I don't really mean I could kind of reverse engineer the model. It's kind of hard, but getting customer data and trying to look at customer data is much harder. You know, it's a different ask. It's a different ask in the org. There are compliance issues around it. So these are things that, you know, we'd like to try to do as a product level, but it takes some time to do.

41:13Yeah, this is a Herculean effort to try to get around a very pragmatic organizational problem. It's very hard, you know. I always tell people, you know, if you want to do something, you've got to, I have a friend who's, he's figured out a way to like automate all of the marketing in this company. And he's just, he's totally obsessed with prompt engineering. He's got this, and I, and he says, because the last, and they won't let him do a pull, he can't do a pull request. Cause he, you have to get, you have to get someone to sign off on the pull request. And so he built the whole thing in Google sheets, you know, in Google drive.

41:44And it's just this, so this, you know, something an accountant would build, you know? And I said, the last thing you want to do is be able to do a pull request because you'll get into the engineering org. And once you're in the engineering org, you have to follow their best practices. And those best practices may not work for what you're trying to do. They're best practices, they're just not best for you. And not a lot of that, we see a lot of this here is these kinds of tools to run in an org. Part of like I said, why fine tuning is hard. Well, I've had people come to me, we want you to build a model for us, but, and we want the model and we want you to design this model for us, but you're not allowed to ever look at the data for any reason.

42:23Cause it's a compliant, I'm not going to do, how can I constantly get it to work? I don't know how it would work, but you know, so there's those kinds of things are real and they exist in big orgs. And so, you know, looking at data and peaking, we certainly would like to look at and peek at what's going on with the tool. And this is something that we're trying to figure out how to, how to do that in a compliant way. In the paper, you benchmark the results against a few different approaches, activation sparsity, absolute weight entropy, absolute local circuit complexity. Talk a little bit about the prior work.

43:01So in this paper, there are people at Google DeepMind and people at Anthropic, which are trying to find out metrics to figure out what's going on in Grokett. and yeah we picked sort of the four top ones that people have in this in the theory and what we find is that none of them can detect this third phase of anti-grocking none of these metrics they have i mean they can kind of detect grocking like they can see the phase transition but because there's a threshold they don't know what the thresholds are they don't know that anti-grocking or this sort of confusion phase exists the generalization collapse and their technique Like, even if they could detect it, it's not clear what, you know, when it's happening and when it isn't.

43:42So that's sort of the part of the paper is not only can we detect this new phase, but none of the existing proposed metrics can detect it. Only our tool can detect it. So we can just, you know, we pick, you know, the top ones, deep mind and anthropic, right? Those are the big ones, right? And so that, and that's sort of the point is that whatever, you know, even like they have this thing called circuit complexity, where you're trying, you know, this is a big thing that's come out of anthropic recently, the circuits. How do you know the circuit is overfit? Can't tell. Can't tell. And it may see that it has higher complexity.

44:12Or like people do compression. Another thing is, oh, if you compress, the quality of a model is correlated to how much each layer can be compressed. Okay, that's true. No question about it. Question is, what about if you over compress? If you over compress, you overfit. That's the third phase. They can't detect that. So that's what we're able to do with the theory is that we can detect this overfitting phase. And, you know, this is something that I'll give you an example of where it might. And it's not always bad, but let me give you an example where you might actually want to do this. We've looked at models that are like segment anything models, the SAM models from Facebook that allow you to do zero shot vision learning.

44:54In other words, it turns out if you look at those, a lot of the early layers in the SAM models are overfit, according to our theory. So here's my— Early as in base. Yes, closer to—well, they're closer to the data. So you think of the early being that, you know, the closer the data, the earlier the model is closer to the label, the later the layer is. So what I think is happening is that there are these primitive features in all of vision. you know, lines, line segments and little circles and things like that, that our visual system picks up. I think what's happening in the segment anything models is that they look at a large number of natural images and they memorize these abstract features, primitive features in the data.

45:38And that's why they're able to perform good zero shot, zero shot learning, because most natural images, you know, in the natural environment, things are pretty much all the same. You know, a tree is a tree is a tree. And, you know, it could detect what a tree is because it can see, oh, that looks like a tree because it has the same primitive features. You know, a tree in China looks like a tree in the U.S. Although a pine tree is not the same as, you know, whatever, maybe a cherry blossom in Japan is not the same as a pine tree, but its primitive features are close enough that it can segment them and detect it as a tree.

46:10And that's what I think is going on. So it's not necessarily that overfitting is always bad. It might be something good. And so if you're trying to build a zero-shot learning model, you might want to optimize for this kind of overfitting in the early layers. I'll give you an example where it might be bad. We've looked at models like Lama Guard, these guardrail models. Guardrails show the opposite behavior. The layers near the labels seem to be overfit. And so what I think is happening, again, this is all conjectural. I haven't gone and done ablation studies. This is all conjectural. But I think what's happening in the guard models is that it's overfitting to, you know, whatever the examples you're giving to try to build the guardrail.

46:51And which is why they can be, you know, you can get around them. If you can get deeper into the model and find the more abstract thing to tell the model to think about, it's able to go around the guardrail. So it may be that if you're trying to build guardrails for models, you may need to have, and these are the instruction fine-tuned components on top of Llama. So if you're trying to instruction fine tune a model to give it a guardrail, you may want to have the overfitting go very, very deep into the back layers near the data to prevent that kind of backdooring. So we have cases where overfitting is not necessarily bad, but knowing what it is and how to detect it is what we're trying to do with the tool.

47:31So these are examples we've worked out and sort of the goal is, look, we have an open source tool. That's 200 ,000 downloads. I'll give it to you, try it out, and then talk to me about what you're doing. We'll see if we can figure out what's going on, how to make it work for you. That's essentially the Weight Watcher project. And when you compare against these other methods, are you strictly comparing against the ability to predict this thing that you made up? Or are there more like objective metrics that the other labs have published. I'll give you one that was very surprising. Okay. A couple of years ago, I took a look at all the existing base models, you know, Lama and, you know, whatever was back, the CUNY was hot back then, you know, this stuff.

48:20And I compared the average alpha quality metric to the hallucination metric. Okay. And it turns out, according to our theory, the closer you are to optimality, the more the model hallucinates. Interesting. Yeah. That's counterintuitive. Like what? Well, you would think that the thing that hallucinates more is not optimal in some way. No, it's more creative, right? Right, right. People are like, oh, we don't want them all. But other people, hallucination is a feature. In fact, it was so amazing. I gave a TED Talk on it. I mean, it was incredible. Like you think about talking to a child like my three-year-old niece, right?

48:57She'll just make stuff up. It sounds good. I'll make up a little story. It sounds good. I'll tell you, right? That's what they do. That's what children do. They're creative, right? They're exploring still. So these models, they seem to have this hallucination ability seems to be related to the optimal convergence properties. And so that's an example of where we looked at that. I was like, wow, that's really cool. And, you know, maybe it means that certain layers are contributing more to the hallucinations than others, right? And there's a trade-off between, you know, thinking inside the box, thinking outside the box, right?

49:30And I always say, you know, people want models to not hallucinate. In other words, you don't want them to be so creative, right? You don't want them going off into the tangent. Maybe you want them really to memorize the data and not be so good at generalizing. And that tells you something about what the layer should look like. And that's the kind of thing we've been able to figure out just using the theory and comparing to, you know, some things other people are doing. That hopefully, I don't think people are, I don't think they're gaming, maybe they're gaming the hallucination metric, but it'd be the other way, you know, trying to convince people that it's not hallucinating.

50:04But we definitely, that's the kind of stuff we're seeing. And by the way, I think ours is the, that was the only metric of all the metrics. That was the only one that actually correlated. Like the other stuff didn't really, it was just sort of random, like just random stuff, all these other, yeah, the other evals are not really correlated. Like these other evals people come up with, they don't seem to, you know, whatever they're doing is not as correlated with the quality metrics you have as the hallucination. I think hallucination is actually testing something fundamental about the model and how it's trained.

50:37And the other evals seem to be just maybe overfit to the data in some way, you know, very specific to the data you're using. So you test it on one data set, you see one thing, you get different data set, you get something else. Mm-hmm. Mm-hmm. Do you think of this as a mechinterp or mechanistic interpretability, or is that a tool that you're using, or is that an academic thing and you're trying to solve problems? Yeah, yeah, it's an academic. I don't actually read any. There's some people ask me stuff like that. I don't know. What are they doing? I don't look deeply at what they're doing because I know what I want.

51:15I know theoretical physics. I know what I want to do. So like that, that's what we had. That's why it's nice to have someone come and collaborate with me, you know, or, you know, we wrote this crocking paper and, you know, someone else, well, I wouldn't, you know, go out and find all these other metrics and see what they can do. The mechanistic interpretability stuff. I think I, you know, my impression is that it hasn't gone anywhere, right? Like there's like, you know, half a dozen things they've done. And what practical impact has it had on what anybody's doing? I don't know. I maybe internally in the throat, it gets useful, right?

51:50Cause they're doing things specifically for their model. But a lot of it seemed very specific to specific data sets and specific models and not something you can really apply. You know, if I'm fine tuning a model for, you know, Home Depot or someone like that, that's, that's an Atlanta company, Home Depot. How do I, how do I use it? I don't know. So it hasn't, it hasn't been something that we've looked at in any depth. I'm happy to collaborate with anyone doing it. I'm happy to kind of compare what we're doing to them. But we have so many things on our plate that we just haven't looked deeply at it.

52:25So it's the same thing. Sometimes people ask me, well, how is your theory related to like reproducing Colonel Hilbert's basis? Some academic. I don't know. I've never read any of the papers. Colonel is something from physics. What do you guys, I just don't know. I mean, it's, you know, if you're doing it, you tell me, I'll explain to you what I'm doing. I'll explain to you what I'm doing and you tell me how it's similar to what you're doing. I don't know what you're doing. The problem is there's just so much going on. Every day you wake up and, you know, there are a thousand new papers that have been, you know, I think a hundred ML papers being published every day.

52:58So how do you keep up with everything? Right. How do you keep up with everything? I rely on other people to come to me with interesting stuff. You know, I go on, I, if you, you know, that's it. You know, I go on Twitter. I spend a lot of time on Twitter trying to see what are people talking about on Twitter and try to follow good people and listen to podcasts like this and try to keep up. But, you know, it's everything's moving a thousand miles. You know, you're moving at a couple hundred miles an hour. So you're just doing the best you can. I sort of lucked in that, you know, I did this stuff in the 90s.

53:32and all the guys I worked with, they, they went off and the guys who were really sharp went off and became quants and are now running like, you know, the investment arm of Dubai or something like this. Um, uh, you know, or they, they went off and, you know, they, they went off and started companies, but most of the people in the, in the, the physicists, they sort of, people sort of forgot about how you could apply theoretical physics to AI. So I kind of snuck in, you know, through the old, like an, it's like an old back door in the lab that people forgot was there. And I was able to sneak in and do stuff.

54:03If people, I think if enough people remembered all this theoretical physics stuff that the opportunity wouldn't be there because they would have known about it. But I, I sort of, you know, I got kind of lucked out that I'm old enough to remember this stuff. You talked about the analogy in a quant for all of this stuff. Like what's the physics analogy for all of this stuff? Well, you know, physics and quant are very close, Right. So in physics, this whole idea of we talk about self-organized criticality and the emergence, signatures of emergence, there's a technique in physics called renormalization group.

54:38My undergraduate advisor, Ken Wilson, won the Nobel Prize for developing renormalization group. And it's it's actually a really fundamental thing in theoretical physics to understand the properties of the universe. Why do electrons have mass? Why do quarks have mass? Things like this. And how do you describe it? And it turns out that it can be used to describe things like when water boils. Okay. When you have water and you boil it and you see all the bubbles. Phase changes? Yeah. It's a phase change. Yes. It describes the phase change. It's the phase boundary. So renormalization group is the mathematical theory used to describe phase boundaries between, to describe phase changes.

55:14That's the theory you apply for Weight Watcher. And so it turns out that you can think of this. It's kind of convenient that everything's a matrix. Yes. Well, you know, I grew up in the Cold War. I learned all this math. It turned out to be useful. So it turns out like if you think about boiling water, when you watch water boil, and you look at the size of the bubbles, there are all sorts of bubbles. There are little bubbles, medium bubbles, big bubbles, all sorts of bubbles in the water, right? There's not like one size of bubble. That idea is those are the correlations in the system. There's little tiny correlations and there's medium-sized correlations of really big fluctuations.

55:54That's analogous to the information in the layer. So when a layer is learning, it learns little bits of correlations between the training data. It'll learn sort of medium-sized correlations and it learns long correlations between the whole data set, across the entire data set, right? There's correlations across all the data. and they're sort of equally distributed. That's the analogy from physics. The fact that the bubbles, when you boil water at the phase transition between water and a gas, that all those bubbles are basically the same, are different sizes and shapes. They're all circulars. They're all different sizes.

56:34It's the same idea that when a layer is learning the information in the training data, it has to learn all the correlations of all the different sizes. and if it doesn't learn all the correlations, then it can't generalize that well. And if it learns, and that's the idea, and if you cross the boundary, you know, you might be, you might overdo it. Or so you're like, you're going from water to ice, you freeze out. And if you freeze out, you get stuck. And if you freeze out, you're overfit, right? So there's sort of this boundary between ice, water, and gas. And, you know, water is sort of, it can adapt to any situation.

57:16You remember like Bruce Lee said, be like the water. If I pour the water in the vase, it becomes the vase. I pour it in the cup, it becomes the cup. I pour it in the glass, becomes the glass. Be like the water. Being able to generalize is like being like water. If you don't learn enough information, it's like you've overboiled and you're a gas, and there's just no structure. There's nothing there. Just random. And if you learn too much, you freeze and you're like ice and you're frozen and now you can't generalize. That is exactly the analogy. That's a great analogy. Yeah. Because it comes from, that's the physics.

57:50And it's the exact same mathematics and physical theory you use. You know, there's a, I'll give you, like there's a paper that came out today. It was, if you take a reinforcement learning system and you train these models on random reinforcements, like 25 % of the feedback is just random, they still get better, right? How could that be? How could you fine-tune the model on random stuff? That's like RL dropout. Yeah, right. It's because it's like the system is frozen and you heated it up and cooled it down again. That's so much of like all these training recipes is like how do we introduce just enough noise?

58:31Just enough random. Yeah, yeah. Yeah, that's right out of that's right out of physics. The idea is what we call spin glass theory or glass theory that when systems become too brittle, basically they're brittle, right? They're brittle and brittle systems are easy to bake. You think about a metal. If you want to make a metal, that's not probably have to heat it up, cool it down, heat it up, cool it down, heat it up, cool it down, becomes strong. If you freeze it really quickly, it becomes brittle and it will crack. It's the same thing. And in the math and the physics, all the same physics and math.

58:59like all the math used to describe that in the physics is the same stuff I'm using to describe neural networks it just turns out that uh everybody forgot it yeah that's all because you know we all or they all died you know they're all you know they're all right they're all retired right I shouldn't say that should be they're all retired right all the guys are retired didn't have to work anymore but you know there's a saying in science that science progresses when old scientists pass away. Right. Because they, they take, they, they, they stop, they, the old guys stop the new guys from doing anything new.

59:33They don't want anything new. So when they pass away, now you can start publishing, you know, they're no longer there interfering in what you're trying to do. Yeah. But it, it, it turned out that, um, it turns out that this stuff does work. If you spend the physics theories are useful for some things. And, and we're trying to basically, and, and the other thing is it's, to me, it's, it's really important that you have an open source tool. and the work be 100 % reproducible. I need to be able to give the tool to somebody else and they need to be able to run it and try it. Sometimes this stuff works out.

1:00:04Sometimes we don't get the right result. We don't know why. I don't know. What do I know? It doesn't describe, you know, 80 % of the time we know what's going on. But 20 % of the time we see things we don't understand and we're still, you know, picking at it to try to figure it out. But hopefully the tool is still useful to people. And that's the goal of this. So how does the renormalization group stuff and the physics, you know, basis of this lead to HTSR and this whole power law stuff? So in renormalization group, there's this idea of a volume preserving a scale invariant transformation. And that's the idea.

1:00:42There's a scale invariant transformation. Things operate, you have different scales. The physics, if you change the scale of the system, the physics remains the same. just some of the numbers change like the the mass of the electron or the mass of a quark might change depending on the scale see this thing you know the scale okay now is this related to like gauge invariance and that kind of stuff do you know that kind of i do know kind of uh to be technical because i'm using a hard cutoff technically i'd probably my system is probably not gauge invariance so gauge invariance a different kind of invariance this is a scale invariance and if you do renormalization group like i probably need like loop corrections to do the gauge invariance but But yeah, but it turns out that, so when I was formulating the theory, I was trying to understand, I have this HGSR theory, we have this alpha metric, and it seems to be this universal metric.

1:01:29Like every model seems to like to be at two, right? Two seems to be universal. There's this random matrix theory tells you at the bottom of the universality class, there's alpha equals two. There's a little tiny universality class in between. For some reason, two is special. So, you know, I got to figure out a way to derive this from first principles. And in physics, that would be called a critical exponent, a universal critical exponent. Whenever you measure a system near a phase transition, simple systems, simple systems, they exhibit critical exponents. However, the change in, say, the heat capacity is governed by some power law.

1:02:07And it's a critical exponent. And it's the same thing you see in the neuroscience, the self-organized criticality, that these neurons seem to have some sort of critical exponent. they all approach the same, all the data approaches the same exponents. Maybe there's some fluctuations because it's, you know, these are small systems compared to like, you know, boiling a pot of water, you know, with, you know, 10 to the 23rd atoms in it. So this alpha is a critical exponent. So I knew that I need to be able to figure out a way to derive this. And as I'm going through the derivation, sort of in the back of my mind, you know, there's got to be some, you know, critical exponents are typically associated with renormalization group.

1:02:43So when the renormalization group theory applies, you typically expect to see a critical exponent. So we would expect that alpha equal two is that. So in the course of trying to derive the HCSR theory, I realized I have to do this renormalization. In order to make the theory, the math easier, I have to make this assumption about a scale invariant transformation. And then I realized, oh, that's renormalization group transformation. And then I tested it empirically. And it turns out that when you measure alpha equals two, that funny parallel, it turns out you can also measure the scale invariance.

1:03:17You can test whether the system is scale invariant by looking at the trace log of the eigenvalues or what's called the log determinant. So it's a log determinant relation that arises or the trace log. So it's simple. You just compute the eigenvalues using SVD or whatever eigensolver you want. You simply sum up, you take the logarithm of them and you sum them up and you start at the tail and you work backwards. And when, as soon as you get close to zero, boom, that's, that's where the information concentrates. That's the scale invariance. And it turns out in that, and that's the connection. Now I haven't been able to prove that the alpha equals two is in fact the critical exponent for this transformation.

1:03:55I think I know how to do it, but you know, I, I do stuff like that. I'm going to be living out of my car. You know, I need to get away to fund this operation. Right. Right. So, but I'm fairly certain that the alpha equals two is related to this. You can measure them both. And if you, and it turns out you can measure the scale, you can use, it's called the dead X condition because the determinant of the, of the exponent should be one. So in the Weight Watcher, you can measure it and you can see that if you violate the dead X condition, the scale invariant or normalization group condition, if it violates it, you seem to be overfitting.

1:04:28And it turns out like we didn't publish this in the grokking paper because it was too much because it'd be like 50, you know, but it turns out it also works in the grokking paper we published that you can see the dead x condition change. So it turns out that these two things are very related and there's something you can actually measure that the idea, I'll tell you where the idea came from. If you're curious is that we had this sort of side project at BlackRock that was sort of like a side project. We were trying to figure out, can we detect when the market's going to crash? And there was this, you know, cause yeah, cause the market crashed, you know, and we're like, what if we get detected?

1:04:59is there's a theory by a guy named Dieter Sornay, who has this theory about why markets crash. He actually has a book called Why Stock Markets Crash. They published like 20, 25 years ago. And the theory says you can measure the signatures of renormalization group in the stock market. And I was spending our spare time. I was saying we could see them measure. And it's a different technique than what I'm using for Weight Watcher, but it starts by looking for parallel signatures and looking for something called long periodic fluctuations around the parallel signature. I have a blog post on it for Bitcoin.

1:05:30Can you detect when Bitcoin's going to crash? You know, that kind of stuff. So it was like this hot project. Well, you know, it's, I'm not going to trade it. I'll let you trade it. You know, you can protect it, but can you trade it? And it was like a hobby project we were doing. Because I work with, you know, with one of Dieter's classmates at BlackRock. And we were doing this. I knew about the work. And we were sort of doing this in my spare time. Because I knew about this renormalization. It was like one of these like crazy applications of renormalization group. And it turned out, but it was this idea that I could measure the signatures of scale and variance in a physical system.

1:06:04And so I had this idea, there's a way to measure scale and variance in a physical system. And they use it to do things like, can you predict when, and when an avalanche, like when you predict, if you have a crack in a material, can you predict where the material is going to like, is a bridge going to collapse? And so you can look for cracks, the best, basically the distribution of the cracks. and as the distribution of the cracks start following a power law, now you've got a problem. The bridge is probably going to collapse. And then that's sort of like the practical aspect of it. I said, gee, I wonder if I could apply this to neural networks.

1:06:34And it turns out you can. It turns out it works. And we can predict when the crash, in this case, being the overfitting, you know, the generalization collapse. That's the crash. And so it turns out it works. And so that was sort of where the idea came from, just sort of, you know, doing sort of these, you know, When I was at BlackRock, my job was to come up with crazy ideas to predict the stock market. You know, just nutty things, you know. And the point being that only the nutty things are going to work because everyone's tried everything else. So you got to try something no one else has ever tried.

1:07:06Otherwise, you can't predict anything. So I would just come up with this sort of this nutty stuff all the time. This is one of the nutty ideas that we had. And it turns out you apply it to neural networks, it actually does work. So these old theoretical physicists, you know, they were doing, you know, they're smart guys, right? You know, they made the bomb. So, you know, they know what they're doing. It does work. It's just somewhat remarkable to me that, and so that's what's going on. And I'm happy to go through all the math and be a nerd out on as much as you want. But that's sort of the, you know, if you derive all the, I have this long paper.

1:07:38It's about 120 pages long. It's in draft form. Every week, Mike and I, Mahoney, I ask him, try to find some typos in it. Because, you know, it's got like 500, I think it's got like 300, 400 equations. Is it up on archive or something? Is it something that folks can access? Yeah, because I don't want to put it in the archive until we find all the typos, but I have it on a GitHub repo. And it just says draft. And I'm happy to share it. If you find, like last week we found I was missing a trace operator on one of the, you know, an appendix A3. There was no trace. So until we get all the typos out, it's just, we don't have the last time to read this thing.

1:08:11You know, it took me a year probably to write it. But it's something we'll eventually get on the archive, maybe another year. Like the HGSR theory paper, it took us three years to get it published. So that work was done, like it was done in 2018 and we didn't get it published until 2021. It just took that long. So maybe when I retire, I'll get the CETL paper published. But, you know, I'm happy to have anyone read it. And, you know, it's, you know, if you want to, if you like theoretical physics and you want to nerd out and spend, you know, a month, you know, on vacation doing this, uh, probably you'll get divorced if you do that, but, you know, maybe you are divorced to give you something to do.

1:08:52In the, not the HGSR paper, the Gronking paper, one of the noted limitations is that you validated all this on a three-layer MLP. And MNIST is the data set. That, in some ways, is pretty far from how you'd like to use these ideas. Well, you know, it's a trade-off, right? If you think about doing development of the Schrodinger equation, okay? I used to do quantum chemistry. So we run quantum chemistry on big systems, right? But you got to start with the hydrogen atom, right? You got to start, you have to start, if you have to make a theory, you at least be able to do small things. I see it more like the Bohr atom, like what I'm doing.

1:09:33You know, it's, Mike always says that's so arrogant of you. I go, well, the Bohr atom's wrong. The Bohr model is wrong. What do you mean? You know, but it's more like the Bohr model of the atom. You know, we're trying to come up with the Schrodinger equation. Once we come up with it, you know, like when I was in grad school, we would study, you know, like hydrogen diamer or, you know, or nitrogen diamer. And you'd see these interesting properties. And then you have to go and apply it to big systems. So I just don't have the capital. I'm non-anthropic. I don't have any funding for this at all. This has all been a hobby project, right?

1:10:00This is all my spare time. I'm trying not to get divorced, you know, if I keep working on it. But, you know, we only have so much compute resources. And we need to understand these very fundamental things. So my approach to this is we try to study small problems and understand them analytically. And, you know, do analytics here. We have like deep, I mean, the way we watched this idea of the renormalization group, I mean, this really is 300 pages to derive the equations. And in the end, you get one metric, which is one little 10 line subroutine you can put inside the code and you test it. And so my idea is, look, I make an open source tool.

1:10:34I give it to people. You can test it on bigger systems and see if it's useful. And that's the idea. You know, as we tell you, like, well, like we don't understand things like make the Gronk and we'd like to study bigger problems. but you know if you look at an attention model there are a couple things going on there are you know like they have the attention block and yes well does it apply to llms okay well it turns out it seems to work really really well for the internal parts of the attention block the k and q matrices it seems like like even in something like llama where you know half the model seems to be overfit not the k and q's the k and q's line up really really nicely with the theory for some reason, the V matrix, it'll go like this and it just blows up and comes back.

1:11:18Like what happened there? Like, so we know that like, we know it works inside the attention block. Does it work for all the layers of the attention block? And if not, why not? So those are things we're trying to understand better. And it's just really hard. You know, it's, and those models are hard to train, right? So we're trying to look at like what model, maybe BERT is, that's a, that people are fine tuning BERT in production, for example. That, that's still a common thing people do, right? for building classifiers. And so that would be like the level of model we might look at next. All of the theory has been developed on like, this is hydrogen model stuff, right?

1:11:52It, you know, it's the Bohr model, right? MLP unminsed. And I said, and we give the code away, try it on fashion mints, try, you know, we did some experience on CIFAR-10, CIFAR-100. You know, there is a point where, you know, it's, you know, I would love to do it on, you know, big systems, right? But, you know, it's compute, right? It computes, it's expensive. So being able to do observational studies, you know, instead of having to run big experiments, instead of doing like running experiments on small models, which compute is expensive, my approach is sort of like doing meta experiments by doing observational studies on hundreds of models, right?

1:12:33So we have a paper in Nature where we looked at the time, we looked at like 500. At that time, there weren't that, even then, you know, 2020, I think we did it, 2019, 2020, look at 100 open source models. I think it was 500. We looked at 500 open source models and compared how the average metric compares to those 500 models. Today, I have a website on the web. Wait, what's your website? I just have, you know, different models, Llama, Quinn, DeepSeek, you know, Falcon. You know, we just try to look at the big models and write up reports and show them. And that's basically the best we can. So trying to do observational studies.

1:13:09You know, I'd like to study like, you know, you know, 100 ,000 models on Hugging Face. But, you know, that would probably bankrupt me and my VC and I'd probably be sued. But that's what we're trying to do. That's awesome. Awesome. Maybe an interesting question, since you are kind of out in the field working with customers that are trying to put Gen.AI to use any beyond the stuff that we've discussed thus far, you know, hot takes, hard-fought lessons in terms of, you know, making this stuff work? Look, I think that the magic has been the prompt engineering. And that has opened the door for being able to do things when you don't have data.

1:13:54Because data is so hard to get. And training models is hard. And it's hard from a personnel perspective. Like you have to pay a lot of money to hire people to know what they're doing. And if you don't know what you're doing, you can really spin your wheels and not make any progress. and the gen AI stuff has been phenomenal. I don't think anyone really expected in context learning to work the way it does. Now, I see a lot of people trying to do, I've worked a lot in cert. So I see a lot of people trying to do RAG and then I realized RAG is old. RAG is from the nineties. That's called latent semantic analysis.

1:14:29That was invented in Chicago. I've been trying to do RAG for years. It never really works. And that has been, you know, people think, I'll just stick stuff in a vector database and it'll be fine. And you just get all sorts of nonsense, right? I mean, it's not going to, there's no guarantee. So that I think is something people are, and I've done a lot of work in search, most of my, a lot of industry work in search. And you sort of see there's a bit of a naivety about it, that there's a lot of emphasis on getting the search engine to work, the plumbing. So right, getting the vector space, getting the, getting the documents translated into an embedding?

1:15:11And how fast can you do that? Because that's slow and it takes compute. And so you have to pay for that. And what size embedding should you use? And getting the vector database up and maintaining the database. And should you buy a vector database or use an off-the-shelf one? I mean, the original one was developed by Spotify. Something called Annoy. There was an Annoy package over years ago. And then there was the one from Facebook and then one from Microsoft. And now there are all these commercial versions. And what I've seen working with search people is that 90 % of the resources are more go into these, getting the thing operational.

1:15:44And then you get to the relevance and you ask, oh, I can just do RAG and I'll get relevance. It's terrible. You know, it just doesn't really work. And the reason is because it doesn't learn from the clickstream. You want to learn if you're doing a system where you're trying, you're either, you're trying to learn from the clickstream, You got to learn from the clickstream. You just have to train a model on the clickstream. And, you know, that means you either have to put some model on top of RAG. To put that in other words, you talk to enough folks that are doing search and retrieval, like the thing that they're focused on is relevant as opposed to plumbing.

1:16:23And RAG, the thing that they're focused on is, you know, that retrieval step as opposed to the generation. and uh you know what you're highlighting is this idea that um and also you know in traditional search like that was what they were like that was their job and their job was to make sure that when the user searched they get the results that they're looking for and it's not like a one and done it's like you know i've worked with i've worked with guys from google ebay walmart you know the big the big the big engines yeah yeah yeah and i tell you 90 of the you know it's just not relevance is hard and this is the point that i'm getting at yeah and people work on it for years they work over years and and people who try to get into it are very naive and you know they i mean i'm not sure i've ever seen an a b test that i believe right i mean i've been like even like i've seen clients who don't understand like they'll put all this effort in the engineering you have to run an AA test just to measure the variance.

1:17:28They don't do it. Like, okay. I mean, I've worked on internal search. Yeah, I've worked on internal search, semantic search. I mean, all the different variants of search. I invented technology for search. We worked on eHow, first billion dollar represents Google. And, you know, it's, you know, aardvark acquired by Google. Search relevance is widely, widely underestimated. It's overestimated how hard it is. I mean, excuse me, underestimated how hard it is. It's misappreciated for how difficult. And there's very little good academic information. In some sense, search relevance is like trading on the stock market.

1:18:03The people who really know how to do it aren't going to tell you what to do. Because that's where the gold is. And I think that thinking that you can just do rag and expect that to just give you good results is, you know, are you actually, you know, if you have good results? Do you even know what your bounce rates are? I mean, you know, I've seen, I mean, I've seen cases where we, you have search. And so part of the machine learning of Weight Watcher is like, yeah, I'd worked in search so much. I was thinking about, you know, I want to redeploy the search engine every day. I want to retrain the model and I want to make sure it doesn't go bananas.

1:18:34And sort of the motivation was how do I model the thing? I can't do A-B experiments all the time. They're expensive. A-B experiments are expensive. They're hard. They're difficult to interpret. And a lot of data. And a lot of data you have. and people do all sorts of, they'll do things in production for their A-B tests that you really shouldn't be doing. They're running multiple tests at the same time and there's leakage between the experiments. Stuff like this. I'm like, it's useless. Like the noise is, I was trained as a physical scientist, man. I mean, I did quantum physics. I mean, I understand how an interpretive experiment.

1:19:08You know, you can't have, you can't have error bars this big, you know, if you're only looking at that much. And it's sort of like, What are you doing? And, you know, it's just well, and that part of the problem with the rag stuff is that it doesn't take into account the clickstream. And so you have to somehow do that. Meaning, unless you've got either an implicit or explicit feedback mechanism from the user, then you're flying blind. And you see now with AI, this is part of the problem with fine tuning. is really what you'd like to do is fine-tune some sort of adapter on top of the rag to adapt it so it will work well.

1:19:48That thing has to work. Think about a fine... You can't have... You're running inference. If you're in e-commerce, you've got a 200-millisecond, 250-millisecond SLA. You can't... How are you going to run inference on that? I run SVM. The SVM has a... It runs in 30 milliseconds. I could, you know, you know, it just, all the, all the fact, all the, in fact, I can run, you can run XGBoost in 30 milliseconds. You can run the SVM in under 10 milliseconds. All the overhead is based, but you know, it's just latency from the network, but you're trying to run, you're trying to do a model and you're trying to do production search and you're trying to run inference on this huge thing.

1:20:27You know, you've got to boil it down. Even Facebook only uses like simple embedding models. Here's one for you. You know, Facebook doesn't use PyTorch production. They use cafe too. Like, and they have simple, you know, simple models. So I think a lot of what I see sort of is what I'm seeing is that there's just, you know, search is still very, very hard. And, you know, and in rag is not a magic box. Um, and you're seeing now what's really interesting are integrate, you know, people trying to do integrated LLMs that learn how to do search on the fly, right? The search is integrated into the LLM somehow.

1:21:03And I think it's just, it's really, it's just, that's probably the, in terms of being in the field, like trying to get this stuff to work. And then the RAC stuff itself, you know, you have prompt engineering issues. You have to ask the right thing, you have to get the right documents. So I think a lot of that is, you know, there's sort of a bit of a naivety about it. And so if you're trying to train in, And that still is probably one of the biggest challenges I see. And again, and also because you have to, you have to track the click stream all the way through, right? You have to track from the user, know what the user is doing.

1:21:34So I think there are a lot of people wanting to do this kind of stuff. And it, there, you know, there's always a lot of, just a lot of money left on the table, right? To, to get it right. And, and whether this technology can help you, you know, part of the idea is, you know, can you fine tune a model that you can put like a little tiny model, an adapter model you can put on top of the RAG system. You know, if you get sort of the, if you get the prefetching right, right? You get the broad spectrum of it. You get the document set. You can do relevance in real time with this stuff. And that's sort of, you know, some of the motivation to doing this.

1:22:08And I think from the real world, you see, and of course now we're seeing that Google for the first time since eHow really is starting to lose traffic, right? They're losing traffic. And I don't use Google for anything anymore. I mean, I use Gmail, but, you know, maybe Google Docs. But what are you using instead? I use OpenAI. I have O3. If I'm paying for it, I'm going to use it. I use it for everything. If I don't use O3, I'll use Grok. Maybe I'll go to Gemini. But, you know, Grok is, you know, O3 is probably O3 is just slow. It's just slow. Yeah. But I'll go to Grok if I need something quick.

1:22:43You know, Grok is quick. And then I'll go to O3 maybe for, you know, and I started using Codex. Codex is pretty cool. You know, I'm not a cursor guy yet. But I, you know, I try to, I don't know, I like this thing. It's my code. You know, you tell it to fix a problem and it just erased. Codex did this this morning. It erased one of my unit tests. I fixed it. It runs now. Where'd it go? It's completely gone, you know? I like Codex because you can check the pull request. Like, you know, you make sure it doesn't delete everything. But I think that's, you know, but search to me is still like, that's one of the big ones.

1:23:17and one of the main uses for this technology because it really gives you a natural language interface to search. I was almost suggested the other day, I wanted to remake Aardvark. Aardvark was a product that was sold to Google. I was a scientist at Aardvark where you would ask it a question and we'd go out and find someone on the internet to answer the question for you. And I wanted to integrate this into the LLM so when it lies to you or screw something up and you don't know how to fix it because you're way out of your, you know, you're punching way out of your weight class. You don't go find someone to fix the problem that the LLM created.

1:23:52Like find a real person, like find an expert to fix whatever the LLM broke. I thought that would be a good product, right? But, you know, the technology is amazing. I'm very bullish on it, but, you know, it's not a magic, you know, I'm a silver bullet, right? You still have to pay attention to what you're doing, but it is an amazing technology. And that's why I'm sort of all in on it, right? Sort of like, you know, Sergey Brin said, he came out of retirement just to do this. So I'm sort of like, that's what it is. It's just all day long. It's all we do now. And, you know, having worked in AI and machine learning since the 90s, it's unbelievably exciting.

1:24:30But it's also a little overwhelming sometimes, too. Which is why you have these great podcasts. You can try to figure out what's actually going on. Absolutely. Absolutely. Well, Charles, it has been great catching up and digging into what you've been working on. Thanks so much. Hey, thanks for the time. I really appreciate it, Sam. All right. Thank you. All right.