Information Theory for Language Models: Jack Morris

28:37Sometimes you think you should have figured it out earlier. And then sometimes you do stuff and you kind of realize it's really complicated and probably not worth it. So I was testing different decoding algorithms for embeddings that are closer or text that's closer to the text that's in embeddings. And I was testing these kind of like inference time adaptation models for samplers. I think we tried a lot of architecture and like kind of training tweaks. We should have tried RL. I think that would work. But finally, we found something that ended up working. And I guess I'm just saying this all because I thought it was like so rewarding.

29:15Like we were just banging our heads against the wall. I'd have biweekly meetings with my advisor who kind of suggests things. Sometimes we would agree we were mutually stuck. Sometimes I would get feedback one way or another and try something new or try a couple things. And we had this idea that it was possible from the information theory arguments. And this other thing where we would kind of like take our best guess at what the text was and re-embed it and see that it was kind of far from the true embedding. So we had this proof that like a better method could like leverage this kind of information.

29:50And then when we finally solved it, it was awesome. We had this number that was like 30 for months. I think at one point, I got it to 35. And actually, I think I was like, oh, I'm done. I got it to 35. And my advisor told me, you can't really just propose a new problem and show you push a metric from 30 to 35. That's confusing and probably not that meaningful to people. And I think that was kind of like a local minimum for me where I was like bummed. But then we ended up getting the number to like 97 or something which neither of us knew were possible we were all just we're just kind of staring at this graph like oh my god like who knew you could get this much information from an embedding and that was like so great like um just sort of this it was so rewarding and so it was invigorating honestly like that research process of like we picked a good problem and then we spent so long trying stuff that didn't work which i'm probably forgetting how frustrating that was i'm sure it was terrible but then like actually solving or at least like coming up with a much better way of solving the problem i don't know if i'd say we solved it but we definitely learned a lot from where we started was like was great and it completely solidified for me the fact that i should have gone to grad school to have this like life experience and like makes me want to do research forever you're clearly clearly um sort of in love with the the journey uh which i think is important because this is what keeps you going through the tough parts.

31:17Is this a good time to talk about the universal geometry side then? Yeah, yeah, yeah. Let's do that next. I think that's a good idea. So we have this more recent follow-up. So the first part I was talking about ended up in this paper called Text Embeddings Reveal Almost As Much As Text, which was published in 2023. And then we recently had a paper come out on Archive, which will hopefully be published at some point and it's called Harnessing the Universal Geometry of Embeddings, which was also that was probably like the only other time I felt like we've made like, maybe there have been two more times, but that was probably the second of three times where I felt like we made like a real discovery about like the unknown.

31:59And it was like very rewarding just for its own intrinsic kind of elusiveness. And I'll start from explaining it in terms of the prior paper. So we built a system that can do embeddings to text and it works very well and we're very pleased with ourselves. And then we went to a conference, we talked to people about it, we talked to the vector databases. I think some of them changed our privacy policies, which was somewhat gratifying. And then we kept getting this perpetual question, which is like, well, you're just assuming we use the OpenAI model or you're just assuming we use the most popular text embedding model if they fine-tune their own model or if they use a model that you're not training an adversary for, then you can't solve the problem, which is like true.

32:44Like none of the vector detect stuff works unless you have this assumption of like knowing the encoder and also being able to make a lot of queries to it. But we had this kind of underlying theory that all of the models learn very similar things. Like we have some preliminary evidence for that, like certain models that are fine-tuned from the same base, you can kind of swap their representations without doing much. or if you look at the nearest neighbors a lot of the models will give you the exact same nearest neighbors even though they have completely different training bases and then there's this paper that came out last year called the platonic representation hypothesis from some folks at MIT which is really really compelling and I think just like great intersection of philosophy representation learning deep learning research like I love this paper and it's such a beautiful idea, which is something like all models are trained on data from the world, and there's only one world.

33:40And so as the models get better by scaling data and scaling model size, they're sort of converging to learn the exact same thing. And in this paper, they have evidence based on correlations for doing this with vision and language models. It's very neat. And so we saw this. So basically think about, you know, you're us, you see this platonic representation hypothesis paper, a lot of people have this shared idea, like, you know, Claude and GBT4 probably do a lot of very similar internal computation, because both of them are trained on trillions of tokens of human written text, even if they have different architectures, like, maybe, you know, the actual basis or like the numbers, if you look at them look different.

34:24but in some way they're like kind of computing the same thing and i think it's even more true with these like embedding models which have like really only one objective that works and they're probably all trained on like ms marco which is a really popular data set and pre-trained maybe on wikipedia but we wanted to basically combine this platonic representation hypothesis idea with the vectatex thing and produce a system that can like align models so that we can do embedding inversion. But you know, it's valuable for more than just embedding inversion. You can use this to kind of glue together models.

35:00That's what actually got me super excited. And by the way, I think there's a few related threads. I think we did an episode with Nicholas Carlini where he had an extraction attack on one of the GPT models and they got it fixed. The other thing I want to... I just really want to spell it out for people just in case they're not thinking it through. Being able to invert embeddings also means that you can back out secret prompts or contexts that might leak customer information. That's potentially harmful and obviously an attack vector issue. I think one of the things I had a question about was whether or not position embedding does affect it and extension of position embedding is affected because obviously contexts are going to get longer and longer.

35:41Your ability to invert will obviously decrease with longer context. What now? Maybe not that important. No, no, no, no, you're totally right. So we're operating in this space in our work where the sequences are relatively short and the embeddings are relatively large. Like, I think we're kind of at a great advantage from that perspective. And you're definitely right. Like, if you embed an entire book to a 500 dimensional vector, there's just no way you could get the entire book back. like there must be this, these kinds of collisions, like, you know, in information theory, like if you have lossy compression, two different inputs mapped to the same code, which means that you can never determine which input formed the code.

36:29And I think that's probably what will start to happen. Like if you have two books, and you swap just one word, and you embed them, I don't know, someone can try this, you'll probably get like a perfect collision. And in that case, inversion is impossible. And even like when you don't take it to the limit, it probably just gets very, very hard. Like things get super compressed. So I don't know how well this work scales. Like it's a great question, like exactly how much information you can sort of cram into one of these vectors. And I don't have a sense of where the boundary is. It'd be interesting to talk to some, one of the like linear algebra people from like the math department on like, how literally can we take inversion?

37:06Like, you know, how, like what measures of a matrix do they have where we can like kind of run that and like try to get some meaningful information out of that. This is like where information theory starts to collide with linear algebra and all the other stuff. Totally, yeah. There's always this detail where we're running these on computers and so we don't actually have like real decimal numbers or real numbers. We have like floating point representations of numbers, which are like very, it kind of like throws a wrench into the mix. Do you have any consideration of like superposition when like sort of non-linearity, like you keep like stuff information in the lower bits, but I don't know if that matters.

37:47I really don't. It's just like a nice thing to think about. Yeah, yeah, it is a great question. And I get a sense that like a lot of the less important bits are more useful for computation and maybe the higher order bits are more important for like storing data or something like that. But I'm not sure. These are the kinds of questions I'm actually hoping to explore over the next few years. like, I'll skip ahead for a second. So we have this result that's like, maybe the third sort of like discovery I was alluding to, which is like, a way to measure the exact capacity of a language model. And we get this number, if you train a language model on a ton of random data, and you measure its rate of memorization, yeah, can you open the right curve?

38:30This is sort of the discovery I'm talking about, like, no matter how you scale the training size, you hit this like perfect, perfect-ish plateau in total memorization, which we call the model capacity. And the question I've been stuck on in the back of my mind for a while is like, how is that actually implemented? So this is a transformer that is trained for many, many data points and many, many training steps. And so it's almost like if you have the 10 to the 6 point on the x-axis, the capacity we don't have to actually say the numbers, but it's basically perfectly dividing its computation between all of the data points.

39:10Like every one of the 10 to the 6 data points gets like a tiny sliver of the model parameters because they're completely independent random strings. So I don't really know if superposition is occurring here. Like it seems possible to me that the model would learn like these completely independent columns of computation one per data point. But it's also possible it's learning some kind of like combined thing where it's maybe it learns like a load and a store and it's like sort of like loading and storing bits using these generic operations and then in the end it reconstructs the random string so even though like the data is completely independent the kind of like compute is very similar in terms of like predicting random strings but yeah I guess this is all to say like about superposition and everything I have no idea how the mechanisms are actually implemented inside the models and that's like one thing I'm hoping to learn about in the next couple of years.

40:05It's a reasonable question whether it's meaningful to learn. I think there's a lot of things that is like nice to know, but maybe not that useful. Latent space alignment is very, very useful. Dataset efficiency in theory, cool, but like practically people are just going to go for the biggest dataset they can. Like the scaling laws that kind of worked out insofar as like the relationship of computer data amount of memorization i don't know i think maybe this is a good point to maybe also bring in the idea that andre has been pushing for the last like i think year and a bit of the cognitive core like what is the dumbest possible model that knows nothing but is you is smart enough for tool use to do everything else right so you can run it on device and fast inference it's open source whatever uh so gemma 3n is like a really good candidate right now because it's like a 4b model that is like claimed to be better than lama 4 and gpc 4.1 according to you know certain arenas that shall not be named this is where things get complicated like it feels like language models kind of implement things and know things almost in the same way and it's like really difficult to disentangle like whether they're memorizing fact from whether they're like learning useful ways to generalize about new stuff.

41:28But I agree this would be really nice. I don't think we have a lot of evidence that we can build a system like this that like is really, really good at reasoning, but really dumb about the world. Like, I don't know if we have the tools. Yeah, maybe, maybe not. I think the existence proof is humans, right? People always lean on humans as like the existence proof. It's not a great existence proof because I think if you've talked to people about the number of neurons that we have and you make a neuron roughly equivalent to a parameter we have something like 100 trillion in our brains so like and like we consume like 20 watts of energy like it's nothing like we're so much better than language models it's not even funny and then the last feature of us is that we're self-prooting which is uh not something that language models do as well oh like we were forget stuff no like we are not deeply densely connected like we like connections will drop therefore we're more efficient you know i see i'm like a language model where everything is always connected all the time yeah or you like you preset the skip layers or whatever and that's it you know like it's not it's not really actually anything that involves with learning it's just like something you do based on ablations and like guesstimates even if we did want that i'm not sure if we have like the right frameworks or methods for actually building like what you're talking about yet?

42:46I think the world is much closer to where you're at, the way Andre is at. Andre is like kind of wishing for an optimistic world. Our conversation with Noam Brown was like, yeah, reasoning is emergent. If you gave the O1 harness on top of GPT-2, you would get nothing because GPT-2 didn't know enough. You need a GPT-3 and GPT-4 in order to then get O1, like as GPT-4 is the base model. Which is like, yeah, that's, I mean, that's, that's reasonable. The way I put it is like, in order to use tools, you need to like, in order to search Google, you need to know at least search terms in order to like, then search Google and then learn what you need.

43:23And if you don't know what to search, then like, you might just be too dumb. i like the kind of uh ethos like maybe you could do some kind of free training or whenever the model doesn't know something it can just google for it and that way you try to encourage it to learn words without or like to guess words correctly without actually storing the information into its weights yeah it seems like a nice like goal at least yeah you need some kind of online learning probably or memory and some combination of that yeah it's exciting you know like i think like if that is the the direction of of where this all ends up that's great but like people aren't are not doing that instead we're building you know 500 billion dollar data centers in the middle of texas and like you know all hail the the god cluster uh that just will you know eventually wrap around the sun and consume solar energy because that's what we need.

44:20Do we finish out the universal geometry thing? Let me finish the kind of methodological description. So we had this goal. So yeah, back to the embedding universality. We started with going from embeddings to text. We know about this platonic representation hypothesis. And maybe I'll skip over the details, but basically we had total inspiration from computer vision in this model from 2017 called CycleGAN, which is, among other things, it's a way to map between two different distributions without any underlying notion of which thing should be mapped where. It's just based on some kind of idea of closeness.

45:04So the cool thing about this, if you look at the top left, so I guess the top left is Monet, so Impressionist paintings, and this picture on the right is a photograph. So like, it's learning this kind of like semantic notion of what content goes where, just by mapping a distribution of Monet pictures to a distribution of photographs without actually telling it which Monet picture should map to which photograph. It's kind of a subtle point I'm making. It takes a little bit of time to wrap your head around or maybe like go to the middle one if you don't mind the zebras and the horses. so like it's clearly learning like what an animal is and what legs are and sort of like more abstract stuff like what uh the camera position should be and what grass is and stuff like that and it's learning like what a horse that looks like a zebra is which is actually like a complicated semantic concept like we don't have a a data set that has a horse and then that horse as a zebra we just have separate horses and separate zebras but somehow this this gan system is able to like elicit this sort of mapping property.

46:12It's like kind of a magical connection that it learns. And I'm still like in awe that it's possible at all. But we more or less like repurpose this system and like we built our own, but like this idea, we took it and we applied it to model embeddings where instead of zebras and horses, we have like BERT embeddings and GPT embeddings or like two completely different models with different architectures. so I think these are GTR which is a T5 based retrieval model and GTE which is based on BERT so they have different training data different architectures different downstream objectives different embeddings but yet when we do this cycle GAN in the embedding space they just perfectly sort of snap to the same place which is amazing and has some pretty deep implications of like the platonic stuff like maybe the models actually are learning a lot of the same functions or something and in some semantic way they're like very close and yeah this is a diagram of how our system looks it's weird to me how profound it seems uh you you seem you seem like deeply impressed by it and then the other thing is like uh when we talked to the uh the to Emmanuel from Anthropic who did the circuit tracing and mechanistic interpretability work they were like excited that like the same thing in different languages maps to the same circuits and I'm like what you would expect.

47:36Yeah, yeah. I don't know why. I don't know. I feel like this feels more profound to you than it does to me. I'm like, yeah, obviously. No, that's so fair. Maybe it's just self-congratulatory and we're happy that we're the people that got it to work. Yeah, exactly. Yeah, it does seem obvious in retrospect. And I think that's constant feedback I've gotten from research. People tell you that this seems obvious to them. But you have to realize that you came from a perspective of no one ever having done this before, and they're coming from a perspective of you telling them it's true. And if someone had told you that this was true, it would be maybe obvious to you too, if that makes sense.

48:17The way I put it is that we have the intuition, but not the proof. You did the work, and you have at least some evidence that it's true, whereas we just have intuitions, right? So part of research is just confirming intuitions. The applied part comes from like, okay, now that you know this for a fact, what do you do with it? Yeah, right. I think the details can be really interesting, like the details of the proof, like which models are most similar to one another and to what degree can you get them to align and on which distributions does this property actually emerge? And like, that's why reading papers can be fun sometimes is because they kind of answer all those little questions.

48:55Yeah, I would say, okay, I pulled out something very current, which is Gemma 3N, which launched, which sort of was generally available yesterday. I would say that for me, and you can correct me if I'm wrong, the most immediate implication is mapping adapters to language models. so the dream is that you have a language model backbone let's say this one is like a 2b language model backbone and then you offload your vision so you only you only load in the vision and parameters or the vision adapter when you need vision you only load in audio you only only load in speech text-to-speech whenever you need it because these are all separately trained you're just sort of aligning latent spaces and you can sort of train them separately and i think Like this helps to make us more confident in, one, it's more efficient.

49:43That's a given. Two, it helps to make us confident that we can just add capabilities without taking away or catastrophically forgetting others. So they're just sort of like stacking more parameters. Just stackable. So that's very cool, yeah. Swappable, stackable. It's like a fatter version of Laura's that is not really that model specific. I would say Apple and Google are pursuing this for their on-device stuff, is my sense. Is this open source? Gemma? Yeah. For a given definition of open source, which is like, we released the waste of Hugging Face. Here you go. Oh, that sounds like open source to me.

50:19Oh, yeah. I guess it's open waste, but not the data. Not the data, not the code. Not the code. Yeah. Right. Yeah. I would say that this is quite soda in terms of efficient models. maybe a small LLM also from Hugging Face would be also in that in that category it's not that many people working on very good very efficient models yeah this is a very deeply related question and something that really interests me which is like what is the limit of like a 100 million parameter model like if you imagine you know 100 years from now when we have maybe our computers are gelatinous blobs and we all communicate through telepathy, will we have 100 million parameter models that are at the level of today's O3 Pro or whatever?

51:06And if so, how would that even be the case based on scaling laws? Do we have special data? Do we come up with a brilliant new training scheme or some type of magical architecture? I really don't know. Or maybe we really are at the plateau already. I don't know. It seems like when we are doing things like calling a small model like a 27B model as small, that's what Ms. Charles is doing. You know, we've plateaued a little bit in terms of what we can do to compress things. I have a fun theory that this is where we mix quantum computing with models. We have to change what a parameter means. We have to search through very high dimensional space and resolve them much quicker than we can with conventional compute.

51:51That would be my pie in the sky thing. I said 100 years. That's very reasonable to me. Throw quantum at it. Yeah, I'd probably have to get a second PhD to know what's going on there. I think that we should establish the definition of small model as being a model that a grad student can inference at reasonable time on a single GPU, which is probably like 7B, maybe. I don't think 27 is small under any reasonable. Is it MOE? Mistraw? I don't think so. I think their stuff is default dense. Don't quote me on that. This is coming off of just a lot of pre-trained data that is potentially collided. Okay, there's two more papers that we wanted to cover and then we can sort of wrap it.

52:44you had an approximating, you had a language model training data. I think this is a little bit also newer. How does this rank in terms of your overall work? Yeah, let's return to the kind of information theory question. So yeah, maybe we'll skip over the contextual embeddings in the case of time, but we'll group those papers. Great paper. Hopefully people start training with that technique. It's kind of a free lunch. Those questions are all about information and model activations. Like, how much can we recover from this given vector? Or, like, what data does this vector represent? Or what computation does this vector represent?

53:23And there's really two types of... Like, if you want to taxonomize, there's two types of whatever you call it, dense information storage mechanisms. One of them is activations or embeddings, which we were discussing already. And then the other is weights, which are the things that are used to perform the computation, but not the computation itself. And so we have now two papers in this direction of what is stored in the weights. The first one is about language model capacity, which is called how much can language models memorize or how much do language models memorize? I never remember which one we settled on.

54:01And then the other one is called approximating language model training data from weights. The first one is like, I think has a lot of deep messages about how language models store information and how they work in general. The second thing is like a proof of concept of maybe like a longer term research project. Let's start with the capacity stuff, if that's good with you. Do I have the paper for that? I don't know. You know, we can return to the question you asked me, which is something like, why do we care? Or like, what is this useful for? And I don't know if I have a good answer for this. I think this is somewhat profound.

54:36Like, it's kind of like in physics, you know, when they try to measure these constants like gravity. People tried to measure the rate of acceleration of gravity for a long time or like those Greek guys like back in the BC era when they were trying to approximate the radius of the earth based on shadows. We're trying to take the GPT architecture like the main one and just measure how much information it can store. And we did this through the lens of memorization, which I think we can skip over for the podcast and we'll just talk about like information storage and weights. Like these curves to me are pretty crazy.

55:16Again, maybe it's like the sort of discoverers folly or something where I'm like, oh, this didn't exist before. So it seems so cool. But then you're saying like, it seems somewhat obvious. No, no, no. Don't let me take that away from you. Yeah, no. Again, like I independently was asking how come there's not enough people exploring L11s from information theory. and then like you come along and your embeddings works become like an information theory exploration. And I'm like suddenly, like I'm very aligned to like exploring this, promoting this and encouraging more people to figure it out. Because like that's ultimately how we figure out this whole compression issue and what Andre wants, which is like the cognitive core, right?

55:54The most efficient model for the most capability. Like that is an information theory question. Totally agree with that. We could start here. Like so transformers that are trained in 32-bit precision, we approximate, can store about 3.6 bits of information to maybe 3.9 bits, somewhere in there per parameter. And like, why is this? I mean, for some perspective, this is quite bad. Like, if you have 32 bits available and you can only use three to four of them, like you're... Just store 32, bro. yeah yeah then you'll like you know you could build your own ai lab if you can make these models that much more efficient i don't know how they're implementing this mechanism or where the kind of bottlenecks come from or even now that we know this what it's necessarily useful for i guess the tools that would be interesting to me are knowing like given a data set if you could predetermine the exact model size and maybe architectural properties required to get a certain level of performance, that would be really neat.

57:03And like, we don't even know how to do that. We don't even know what the difference is between doing low retraining, which trains less than 1 % of the parameters, and full fine tuning, which trains all the parameters. We don't even really understand the difference there. So I think this is like maybe like a baby step sort of in that direction, but there's a lot of unknown ahead of us. Okay. Do you think this is a hard limit? Do you think someone can come up with a better algorithm, but better architecture and then sort of just change the slope? There are two axes here. One is the ability of the model to store data.

57:33And I think we can definitely improve that. I think like maybe even if we tested this with the LLAMA architecture, like there's sort of like a GPT++ architecture, like I would guess that can store better data just because the kind of numerical flow is a little bit better. The nonlinearities are maybe like a little bit more suitable to training, like that will probably raise the bound a little bit. And then the second axis is that our measurement tools are just not that good. Like this is, you know, me, I'm a grad student, I'm running all these hyperparameter sweeps and sort of like, we draw conclusions from them.

58:04But even that being said, like, there are probably ways to measure this better. And but all that would do is push the number up. So it's possible like, there is a way to store five bits per parameter if you have like, a better optimization technique, or if you were a super genius and you could just perfectly set the weights to store the data, then maybe you can do better. And this is just sort of like what we can reach through optimization is this 3.6 bits per parameter. But I would be happy if someone came along with a much better measurement tool. Like this is just sort of like the first measurement.

58:40I mean, I would guess in the future, like, you know, people will look back and say like, this is like somewhat, oh, in one direction or another for whatever reason. And that's just how science goes. I have no problem with it. What we do is we call this the Morse constant 3.6, right? I would never. And then we set a challenge, like a leaderboard of like, beat this, right? And let people go. Yeah, that assumes that we know the true constant ahead of time and we can measure the error rate. It's doable. You laid it out here. Yeah, yeah, yeah, yeah. That makes sense. one minor doubt i have is like the goal actually is a memorization is generalization right the best memorizer model may not be the best generalizer model and you like this incentivizing people to to max this number might actually just be fruitless in terms of actual intelligence like you just get the best actual compressor like you're just going to get gzip that's totally true and there's this pattern in research, you know, time after time.

59:44It's like someone poses a question and then people answer it over and over and over again, but it's often much more fruitful to just ask a new question. Maybe it just doesn't matter how much GPT models can store and you should just like work on something else. We'll figure that out. Did you want to dwell on this side at all? Yeah, let's just talk about it real quick. Definitely not the algorithm itself. By the way, what are your tools for doing these kinds of charts and these kinds of diagrams? Like, I just come kind of curious behind the scenes on the tools. I think like visualization has definitely been a fun hobby of mine during grad school.

1:00:20This one, actually, Oscar, my co-author made this one. Maybe I gave some like prompting, but he made it. I think most the last few papers have all been in diagrams, Google diagrams. I was using Figma for a while and Illustrator. I think Illustrator actually is the best tool. Oh, did you know the Transformers diagram was in Adobe Illustrator? Oh, yeah, yeah. I did know that, actually. Yeah, because that's the only way you can get arrows that sort of like curve like that. And they have good shadows. Diagrams is like the least robust, but it's the most accessible. And honestly, if you're good, you can make pretty good stuff.

1:01:02Excalibur is nice, too, if it's not going in a paper. Yeah, it's just too rough for a paper, but you need something professional looking, you know? It helps, like, if you're going to publish your work, you need to make it look nice and professional and, like, official, right? So this is what it is. Yeah, yeah, and I think there's something worthwhile about saying, like, okay, if I'm going to put my name behind this, I want to spend time making all the references perfect and all the diagrams professional, all the captions are correct, and I think it's, like, important to put that level of detail into your work.

1:01:33That's a little choogy, but let's finish this off. So, okay, we're talking about bits, information theory, what information sort of embeddings. We were talking about language model capacity. I think a much more practical question is, maybe this is more analogous to the vector database hacking embedding threat model we discussed, is like, if you have access to a set of model weights, what can you learn about the data? So like you were just mentioning, Gemma 3B came out yesterday, and you can download it and it takes up a certain amount of space on disk and it was trained on some data, but we have no insight into what the data was.

1:02:11I mean, it's probably English. It's probably some distribution of web text. I would guess there's a lot of code and we seem to have a lot of information about the model, right? You have this file and there's like many ones and zeros, which means something, but it's kind of like a very highly compressed version of the training data. But I would be extremely surprised if they do any type of like private training. Like there are these mechanisms for doing like differentially private language model training or even just anonymization in the pre-training pipeline. I bet they don't do any of that. They just sort of like train on the data and then they kind of know that we don't have the right tools to decrypt the model weights.

1:02:50And so that's like my dream is we can come up with some way of translating model weights back into text data sets. and so in the most recent kind of drop, paper drop is that paper approximating language model training data from weights and it turns out to be a really hard problem. Like trying to go from model weights to text is really hard and we do something a lot simpler which is like, well, there's two ways we make it simpler. The first thing is we assume access to two checkpoints which I think is probably not the case in Gemma but in the case of DeepSeek, If you download the 400 billion parameter model weights, it's this giant file.

1:03:32And you can actually get two of them. You can get the base model weights and the fine-tuned model weights. So the way we put this, you have this kind of like difference in parameter space telling you what DeepSeek fine-tuned on. And it's very controversial. I mean, they're sort of like geopolitical, definitely at the corporation level, they're really interested in the implications of like what did DeepSeq train on. And they've released this kind of treasure trove of information of what they trained on, which is the actual model weights. But we have no tool for like interpreting or kind of decrypting this weight difference.

1:04:07And so we started with something really simple, which is instead of even just trying to like regenerate the training data, we take just a web corpus and try to do selection of training data that kind of like looks like the true training data and gives us performance that's as close as possible to the true training data. So there's this complicated method, but it's something like you just sort of like look at the data point gradient and see if it points in the direction and weight space of the fine tune. And then you take like the top data set. There's some tricks to it, but it's basically just like gradient based selection based on this weight difference.

1:04:47And it seems to be okay. Like it can get us pretty good training data. So I guess if you actually wanted to use this, it would be like your competitor releases a base model and a fine tune. and you're trying to recreate their data set. So you can take this weight difference and take a giant web data set. Like if I was doing this at a company, I'd probably try to scale it up to trillions of tokens and then select the exact data points that try to produce the model. And it turns out you can train a pretty good model with that. We don't get to quite the performance of the original model, but it does seem to be like trending in that direction.

1:05:24This is like very creative. I don't know what the use of it exactly is. Yeah, like when would you be in this exact situation? Decently often for the open model labs. Even DeepSeek R1 has released an update. Mistral does it pretty frequently. Lama does it frequently. It's not impossible, but I think that I really like the creativity in using quote-unquote synthetic checkpoints to do this, which is, I don't think I've heard less from any other place. So I don't know if you came up with the idea. It's like linear interpolation in weight space. Okay. That's a bunch of the recent work. I wanted to sort of cap things off with the datasets question.

1:06:06Is that a good? You can ask me whatever you want. Well, it's not a good. It's not an ask. It's just like, I think this is a very good thesis. I think it's a hot take. I almost invited you to speak based on just this alone, but it was a little bit late. Oh, for the conference. Yes. when I look for conference keynotes, I look for something that has a broad overview that can put the last few years in perspective or it's an insight that you can reasonably rely on to last for a while so you can get some mileage out of it. I think a lot of ideas in AI come and go, but things that are scaling laws, things that are trend lines, things that are like, there's no new ideas in AI, that I pay attention to.

1:06:49So maybe you want to recap, what's the backstory if there was one? Yeah, yeah, sure. So the meta backstory is I've sort of started writing on Substack. And this is a post that I wrote a few months ago. The highest art form of humanity. Yeah, yeah. Yeah, publishing papers wasn't doing it for me anymore. And I moved to Substack. And this is the name of the post. There are no new ideas in AI, only new data sets. One guy pledged me. But then I found out he was like my former student from a class I was teaching. so i don't think it really counts it counts he's a friend he's your first supporter a pledge is a pledge man i'll take whatever i can get so so the underlying thesis is that whenever maybe i'll lay out this framework first so there's this uh this book called the structure of scientific revolutions by thomas kuhn that i read near the beginning of my phd which suggests that science kind of moves in these cycles where not very often there's something he calls a paradigm shift, which is like, you could think of it as a zero to one innovation where everything changes.

1:08:00And then it's followed by a rapid period of small innovations, a lot of like reapplication of previous techniques, pre-paradigm shift techniques to the new era. And then things sort of slow down as we wait for a new paradigm shift. And I was kind of asking myself what's unique to the paradigm shifts that we've seen in in AI. And by the way, to me, AI and language models are somewhat synonymous at this point, like at least for the foreseeable future. I'm certain that will change, but basically everything that's pushed the boundary to whatever we have now that resembles like intelligence has come from language models.

1:08:38And so those breakthroughs came in a few steps. So I think the idea is also like a meta commentary on the research community because what everyone wants as a researcher is some kind of like cute new method that no one has thought of before that just works on the existing data better than the previous methods that's like for whatever reason like the kind of most glamorous thing people think you can do as a researcher like mamba it's like it's like a transformer but it's like more efficient and works better so that's what a good idea looks like and I think everyone wants to find something like that.

1:09:15But if you look at what's actually born out in practice, it's never been like that. I think all of the things that I would consider paradigm shifts in the Kuhnian sense came from a new technique, but trained on new data. And I think the new data is super, super important. So I wrote it as a series of four paradigm shifts. The first is the emergence of deep neural networks with AlexNet, which I think was like 2010 to 2012 era where we just started training on ImageNet, which is like a scale no one had ever seen before of millions of images. And then the second thing was Transformers and BERT and this Attention is All You Need paper 2017, the first GPT 2018, which is web scale pre-training.

1:10:00Like no one had ever done that before. No one had ever tried to scrape all the text off the internet and then tokenize it and feed it into models. Like, it's a crazy idea. And I think, like, we should be honest. I mean, transformers are incredible and, like, their staying power is never going to cease to amaze me. They're, like, much more optimal than I think anyone ever knew. And I don't know if we'll ever beat them. But the real innovation is web-scale pre-training. And I think, like, we honestly probably could have gotten this with RNNs. I know, like, the scaling laws paper shows that RNNs have worse curves for scaling, but probably people would have been, like, I bet you could have built ChatGPT with a very sophisticated RNN.

1:10:42Like, you didn't even need transformers. What you need is web scale pre-training and the third innovation, which is instruction tuning. And we thought it came with, like, reinforcement learning, but I think the big innovation of instruction tuning is actually the human preference data, which is, like, gathering positive and negative pairs of what looks good, in terms of a chatbot interface. And actually, it turns out you can do supervised learning on that too. You can do DPO, which is a form of supervised learning. You don't even need the InstructGPT techniques. You just need the data. So I'm sort of playing devil's advocate here, but I actually think this is true that if we had the right data sets, we almost could have scaled 2015 era techniques and gotten something that looks like at least InstructGPT.

1:11:28Reasoning models are a little different. like there i'm not sure if we could have that with rnns or not like i don't know if i'm in a position to comment on that with certainty but they do fall into this framework which is they really did emerge from a new data source in this case it's something like a little different it's like verification with symbolic systems like math calculators coding environments unit tests like things where we can provide numerical feedback to language model outputs but we we built a way to to learn that and leverage it to get more intelligent systems. And so whatever the fifth thing is, whether it's video or embodied AI or some kind of crazy innovation on reasoning models, whatever comes next will probably be some type of new data source that we're not using yet.

1:12:17That's a really good thesis. I would say that the researchers I talked to would somewhat disagree. Obviously, this is like a hot take type of thing. and like you already acknowledge that RNNs don't scale to the same extent. Like they operate on the slope of the curve. Whereas, you know, I guess like the amount of data or the type of data or the core insight just changes the order of magnitude of the x-axis, right? That we are mostly working on. But like both are important. The way that I think someone put it to me was an improvement on compute or data efficiency is the equivalent of having a whole bunch more data that otherwise would be a lot more expensive to collect.

1:12:59It's likely that the frontier models right now are just a collection of hundreds of these small little experiments that just stack up. You mentioned Muon in your post, which seems to be the atom killer. Curiously enough, still none of the big models use Muon, but vibes are good. Yeah, yeah. And the value of building better optimizers is really incredible. Like it's just a free launch. You can just sort of like plug in a slightly better training mechanism and then you save like a ton of compute and a ton of training time. That's like hugely valuable. I think this is cool because I think like it puts us in a mode of like if you were ever to ask what comes after reasoning, it has to be something on the order of this.

1:13:44And most ideas are not. Most ideas are not. and so this is cool in a sense of like it just jolts you out of incremental thinking into like what really is missing for the next paradigm and i don't have an answer do you have one do you have candidates oh i i really haven't even considered that too much i guess like scaling reasoning you gotta do the autocomplete for step five i mean you got us all the way there and you're like you know you gotta show us the way now we can say it's an exercise left to the reader but i mean the reality is like predicting the future is too damn hard you know like i i don't maybe it'll be obvious to me in hindsight in five years but sitting here today i i really can't derive from first principles what the next wave of innovation will come from yeah i think we have a few years left like each of these uh phases lasted for a few years reasoning just started last year kind of we got some uh some juice on this one cool i think that is uh a broad overview we've went way over time but like i really enjoyed this i guess my parting question for you is kind of a meta one so i'm not an academic i'm like kind of self-taught i just read a bunch of papers and like i talk to people all day as part of the podcast how do i rate in terms of like my my questions as though like could i pass as a grad student like or like what's my distribution like maybe i was maybe more industry-oriented than academics?

1:15:09I think you got to realize that like the only person that's an expert in your area as a grad student is you. And even like eventually your advisor defers to you for a small set of questions that fall within your very niche expertise. So like I think you're clearly like a very good generalist and have like a huge amount of background on these topics and to the point where I would say you're passing the grad student Turing test. And I think if you went to a talk, people would just assume you have some weird research area of your own that they don't understand. My research area is AI engineering.

1:15:47I'm trying to making it up as I go. But no, this is super helpful. Okay, well, that's about all we prepared. All the best in your search, all the best in your PhD. I assume, apparently the current PhD meta is you do a bunch of small papers, You staple them together and find an overall theme. You do a defense and that's it. That's the journey, which is kind of cool. I would love to do that. I'm too old to do it, but it's cool. Yeah, yeah. It's a great thing to do at any age. Well, it's better to do a sub stack, right? And then you have people subscribing and pledging along the way and getting validation.

1:16:25And yeah, that's better than a PhD. Sub stack. That's the title of the episode. like Substack better than PhD. But no, thanks. Yeah, thanks for your time. This is really great. Where can people find you? What are you looking for? Really? I'm online. You know, you can follow my Substack and Twitter. I tweet pretty consistently. And you're putting papers out. I guess like the most meaningful thing, to be honest, is to engage with the research and send me an email if you really care. That's amazing. And like, I love having those kinds of discussions. And you mean like what I'm looking for in a job or out of life?

1:17:04Your research direction, like what interests you over anything else? If there's someone out there looking who has a problem and is looking for someone to help them on it, like you are the guy for underscore. Oh, yeah. Hopefully, if you listen this long, like I think like my research is a lot more well connected than some people's PhD research and that it all falls into like a very small manifold of like all possible problems. And so if you want to work on anything within that space or that's sort of like adjacent to the problems that we discussed in terms of like language model, maybe not even language model, but model, weight and activation information, I think anything that can be described as that is very interesting to me and I would love to talk.

1:17:54Awesome. Well, we'll put your contact info in the show notes. And thanks for your time. Thank you.