#326 Zuzanna Stamirowska: Inside Pathway's Post-Transformer Architecture Designed for Memory and On-the-Fly Learning

29:26Craig Smith:You cared enough to pass over to your friends and and etc some people didn't care or like a bit like epidemics um so this is this is the logic then when you bring it to gpu and i get get questions about this because of course you know was being seen uh like with the paper and online and github is just a small portion of uh of our work um what do you what do you see uh what do you see in the code and in the very GPU-friendly implementation that we have there are ultimately, we have to talk about matrices. And it looks very much like a transformer. But there are important differences that happen.

30:11Craig Smith:Attention is linear. So when you look at sizes of, let's say, our matrices, the traditional transformer has this large square. right? And there will be squares and they're dense. Whereas for us, it looks, the only dimension that we care about is the N. That's the number of neurons effectively. And the objects we work with look like very kind of narrow, but very long snakes. Okay. So we have those huge squares in transformers that grow quadratically. And we have this very long snail, which effectively we only care about the length, which is n. And this is the number of our neurons. And then to make it sparse, because of course, we had to, as I said, we had to apply some makeup to make it work on GPU that really likes dense matrices.

31:08Craig Smith:We actually multiply it by a positive sparse vector of activations. And this, once you actually multiply one for the other, you get back your graph.

31:25Zuzanna Stamirowska:Yes. Just.

31:28Craig Smith:So this is for the listeners who might be asking, okay, where is the sparsity hidden in the implementation that is GPU-friendly? Actually, mathematically, it is there. So you get it back, and hence you can actually do other things like spot the synapses responsible for specific concepts, et cetera. I mean, sparsity is very much there, even though if you just look at the code like this, you may not spot it because it's hidden in one specific multiplication. And the fact that we have sparse positive vectors of activations, this is a very big difference that our CSO loves to explain as well. I mean, I think once he used a tower of Macaron to explain the kind of the space of those factors.

32:17Zuzanna Stamirowska:And you talked about you need a certain amount of signal to activate a neuron. That sounds like spiking neurons is...

32:30Craig Smith:It's an edit. This is not... I mean, intuitively, I mean, not in all the implementation details, but intuitively, it's not too far. So we're getting to, so what actually response to our paper was especially good from the neuroscience community, because indeed we kind of started to show the way of bridging very non-organic AI with the models and the thinking of how neuroscientists see brain activity of course we're not getting to the levels of chemical interactions right like reactions that happen in the brain um but we we are offering like a plausible brain-like explanation for for

33:21Zuzanna Stamirowska:for how reasoning may appear and then the sparse uh uh the the activation of when the network is working. I understand that it's very localized. You're not using the entire network. It's whatever the edges connect. That sounds a little bit like mixture of experts. Am I wrong there?

33:54Craig Smith:This isn't just at a very, at infinitely small, I mean, not infinitely, but really tiny, granular scale. So if you were, because mixture of experts, you know, is more kind of defined from top down. Here, imagine that your expert can be even just, you know, one, potentially could be just one edge, right? And you don't decide expert groups that are being decided on their own. kind of organically while training. This is important. This is also maybe coming really deeply from the complex systems intuitions. We really don't impose any structure on the system. The system does what the system wants to do.

34:42Craig Smith:And that's the point. That's the magic of it. This is also why we believe this kind of, ultimately the space that's created it can be very interesting for reasoning. Because at the same time, the concepts appear, the more important ones are strengthened. And it gives you a topology, potentially with shortcuts, to explore. And if you believe that reasoning is a way, some sort of search in the space, then this gives you an interesting topology to explore. And it's aligned with the method of learning itself.

35:17Zuzanna Stamirowska:And then as the system learns, edges become stronger or connections become stronger or new connections are formed. So the graph becomes denser, not in the number of nodes, but in the number of connections.

35:45Craig Smith:And yes, plus some may fade over time.

35:51Zuzanna Stamirowska:Yeah.

35:52Craig Smith:So yeah, some could be if you don't use them. I mean, you have only positive activations, but you may have fading as well. So in a way empirically what we find is that we get like almost a scale-free type of distribution of degrees of nodes, which gives it. This is a type of structure that we find in the real-world organic networks quite a lot. Because this is one that's known to be pretty resilient and with good communicability, and also it behaves similarly wherever you zoom in the network. So it has similar properties, irrespective of its size.

36:39Zuzanna Stamirowska:and how how you said you you built uh like a billion parameter model using this architecture is that right how how is its uh not only how is its performance but i understand memory and in transformers the intention mechanism but how the memory here is the strength of the connections right so so uh yeah so but how how how does it perform uh for example in uh is it's not auto-aggressive right or is it uh how how does it perform on language i mean you're you've built a language model with this, right?

37:39Craig Smith:Yes. So this is, we train it on predominantly on language. I mean, we're not talking about the vision models, for example, right? Like there is an entire line of research and right now attracting a lot of attention, which is actually linked to vision and world models per se. We actually believe that there is maybe a way to bridge the two. But yes, we're looking at the language model and we compare it on traditional language tasks. So these are the benchmarks that we are the tasks that we're showing in the paper. We're looking at translation, we're looking at how it behaves at traditional language tasks.

38:28a bit the way in which transformers were developed.

38:32Craig Smith:So the question of scaling it further, in fact, and the features that we see more likely coming out of it, out of this architecture scaled, well, scaled or developed into a model, into a product, is, let's say, the power of your output with way less data. So with a way smaller model and way less data, we should be able to get to the same results. I mean, compared to GPT-2, so I mean, comparing apples to apples, architectures to architectures on the same data sets, exactly same data, I mean, we're actually learning sometimes even faster. Event transformers and scaling loss are preserved. Knowing, however, that we don't need to grow this size of neurons so much, and we can be adding, I mean, we have state, right?

39:33Craig Smith:Which is huge. And which is huge, well, we can actually use it efficiently. So it's both packed efficiently, because it's packed in the graph between the connections between finally not so many neurons. And then because of the local dynamics, we only access a bit of it at every step or. And again, some some more dumb questions so I can

40:04Zuzanna Stamirowska:understand this. I mean, in in the brain, these local networks that are activated represent

40:19concepts, perceptions, all of that, memories,

40:25Zuzanna Stamirowska:and in your system, but it's not,

40:35Zuzanna Stamirowska:let me think, how am I going to say this? The knowledge that's stored there depends on the connections between these neurons. And the knowledge kind of emerges from those connections, right? It's obviously not stored explicitly. uh and does does that is there some plasticity in that like if you cut a bunch of connections you still have the knowledge maybe it's got to grow stronger connections again but yeah what can you

41:22Craig Smith:talk about so this is this is this is this is a beautiful question i would like to see your questions are actually very deep. So definitely none would qualify as dumb question. This is dramatically important. I'll take it in two ways. So the very concept that we have here is synaptic plasticity. So this means that not only a message is passed through the network or through a connection, but then this connection becomes stronger because it was triggered. And this goes on perpetually somehow, right? As the system lifts, I mean, we have this effect, and then there's this kind of update to the slower weights that just, I mean, change at slower frequency.

42:12Craig Smith:But then for the plasticity of the network, I mean, this is a brilliant topic, actually, for complexity science and network resilience in general. And one intuition that's actually this one came from me. I mean, most of this research actually came from my colleagues and our team. But this one came from me is that function shapes the network. And you actually may have, and this is something that the neuroscientist at that coffee shop told me, you may sometimes swap the nerves and the function of them though, like let's say the example he was giving me was the auditory with the visual nerve. But then the mouse will turn out just great.

43:04Craig Smith:Because the stimulus that was kind of given shaped the network because the network's job is to transport it in a way. So it's getting these local functions of what the network, what the edges, what the nodes are doing that then help you to potentially even compensate. So I don't think we've run deep enough studies of what's happening even in our model right now in terms of its resilience to deletion of nodes. of nodes, but I've done a number of studies of how such systems work and how scale-free networks behave when you delete. So two intuitions is that the function shapes network, there's a good chance that some functions will be overtaken by the locality, and this will also happen according to the rules of local connections, like let's say common neighbors.

44:08Craig Smith:between the nodes. And this is like some redistribution maybe of a function is likely to happen. And yeah, so this is a very deep topic, very, very interesting. And then as for concepts appearing, I mean, yeah, there is a number of things that happen with the fact that we're just working with the dimension n, but this is maybe jumping across topics.

44:32Zuzanna Stamirowska:Well, actually, I was going to jump, but, well, I will. So you're in complexity studies and you're studying emergent properties. And the obvious thought is that in consciousness studies, you know, a lot of people think consciousness is an emergent property of the complexity of neural interactions. Do you have any thoughts on that? I mean, I know that's in left field, but are there, or rather in a less sort of unanswerable way, Do you think there will be creativity in this kind of a network in that the network will be able to think beyond the fixed parameters of an LLM and come up with new ideas?

45:46Zuzanna Stamirowska:and this is something I was talking to David Ha about with evolutionary systems in general. They're looking across a landscape, not simply following gradient descent to a local optimum or something. uh so so they can jump over across to find um you know ideas that that didn't wouldn't necessarily emerge from a fixed llm so yeah

46:27Craig Smith:yeah absolutely so i i don't think i feel nearly qualified enough to have a position of how consciousness appears in the brain or how it can, you know, appear ultimately in a lens. Just somehow completely by accident I started reading about Heidegger over Christmas. And I feel, I mean, I don't even know the advances in philosophy lately, but the definition of being and thinking may be somehow impacted by what's happening with the reasoning models. So, I didn't say just my kind of cultural thoughts. When it comes to generalization, of course, this is the main goal. This is why we're doing all of this.

47:20Craig Smith:because the real innovation wouldn't be just recomposing things that exist but pretty much seeing the loophole and this is then when you know there is something interesting that should be added how do you know that there is something interesting that should be added that you need to actually somehow modify what's your most modify your topology right and yes so we are there are two ways to maybe generalize the general the most imminent generalization that we are after is generalization over time so this is that this is this is the kind of capacity to to maintain coherent reasoning over over time without falling into hallucination and just doing something completely silly and that's the intuition and then generalization in terms of having really i mean internally we call it the model which is a real innovator right one that can come up with eureka moments and this eureka moments they don't they don't result from very formal thinking i mean i believe that very few mathematicians really kind of start their proof and work it's formally step by step reaching reaching the the conclusion i mean usually they have in different different bits of information of conviction filling in dilemmas on the way sometimes or even like strategic proofs uh like so i know mathematicians who actually design ways to strategic ways to to get to their proofs So, yes, and I think that this will go through having some sort of like an internal representation that allows for it.

49:09Craig Smith:Our hope is that having this plasticity and the structure which is not 2D, is not 3D, is actually somewhat efficient and emergent will make it easier. And then there is, of course, a lot of work to be done for this.

49:26Zuzanna Stamirowska:So what are next steps? I mean, this is not productized yet. It's still in the research phase. Is that right? And where are you guys headed?

49:37Craig Smith:Correct, knowing that you're actually right now working on productization. Because, I mean, naturally it locks a lot of interesting properties. I mean, some of the first ones are infinite context windows. and hence reasoning in some use cases can be, you know, jump from zero to one. So we actually partner with NVIDIA and AWS, something that we announced at RainVent in December. And the moment our, you know, the first model will be ready, it will be immediately available to AWS customers. And yeah, so this is, I'd say, this is being productized as we speak, knowing that we're productizing along yeah every search and what kind of use cases do you think the first iteration would would be applied to um so the very good use cases for this are the ones that are links to small so we're looking highly valuable small data where uh we're actually want to we were doing for example research about um about or are you making new designs for uh anyway we actually saw a use case like this for for uh in the nuclear space right you don't have a lot of uh a lot of documents but those that exist are actually very very highly valuable and you you would like to get some ideas for um for new for the current engineers right and ideally you won't learn it very effectively because there's just not enough data.

51:15Craig Smith:And then we're looking at cases where you have continued learning and data that kind of keeps on changing where you want to deliver value. And then also reasoning, which is slightly more kind of complex, which is highly personalized. So the use case that we look at is, for example, healthcare claims resolution. This is because of the fact that, okay, it has to be very personalized. It is complex reasoning because you need to, within the context of, you know, what's linked to the process of the claim resolution, all the information that was given, you kind of need to resolve it. and also you need to be able to explain why and how.

52:06Craig Smith:And because we kind of see, we have some advantages in terms of interpretability because we kind of see the synopsis that light up. And then also in reasoning, I mean, you can have layers of, let's say, explaining of what happened. We have some big benefits for the regulated industries. so but journey speaking think about small data uh with time changing elements yeah and and potentially next best action uh suggestion uh yeah which is you know or like a sort of reasoning and in terms of um

52:52Zuzanna Stamirowska:the traditional traditional i mean they've only been around for a few years but llm uh weaknesses hallucination does this uh eliminate the the problem of hallucination uh because hallucination comes happens during inference when i mean from my simplistic understanding when the probability distribution of the next token doesn't include the correct token and uh and and gives something that's in the distribution but then that error is propagated uh as as you go along um is is first of all is that right but second of all uh in that you're not uh predicting uh the next token from a probability distribution you're you're you're i don't know if reading's the right word but you're you're looking at uh an existing uh the activation of an existing sub network within the network or within the graph uh yeah how does that affect uh hallucination

54:22Craig Smith:yeah so it's just actually one of the big motivations for for this work um was to limit hallucinations right and and especially hallucinations over over time so important to note because i think we didn't stress it at the beginning all the dynamics that we that we talked about happen at inference it all happens at inference uh so indeed this is this is this is the hope right that you because you keep the memory, you can keep consistency. And as you go on, you don't fall over very easily. I mean, that's like a structure to hold on to. I mean, it was too early for me to communicate results.

55:13Zuzanna Stamirowska:uh yeah and uh and the uh uh you you said that it follows the same scaling laws or or similar scaling laws to transformers which was was a huge advantage for transformers and and you know what they did is just build bigger and bigger and bigger networks and it got better and better or better at least to a certain point do you think the same will happen with uh bdh with baby dragon

55:48Craig Smith:what is yeah actually the paper is dragon hatchlings it's dragon hatchling and the the acronym is bdh and i think people are very very just tempted to to to put a b explain to me um so it's dragon hedge link uh we are not playing the scaling game in the sense that we don't need to grow the end so much um and the ultimate goal but this is you know this is this is the goal of almost almost everybody for for us that the path goes through through memory uh but the ultimate goal is to to get to generalization and generalization in reasoning so so yeah we are focused on reasoning models and and and seeing actually not the scale per se because the power won't be coming from the scale um so yeah point is we don't need to we're not playing the game of having our n you know being humongous because i mean that that's precisely not not the reason why we're doing it I mean, this is a way more efficient way of learning and then evolving and storing memory.

57:04Craig Smith:In hardware sense, it all sits on the chip. And this is a big advantage. And if you look at those, if it sits on the chip, you do less lookups. And you may divide your compute for reasoning 10 times for the same output token. right? Plus you can probably do reasoning on cases which are more complex because you have consistency or like consistency, you have capacity to work for longer without falling off into hallucinations. So it may be a more sustainable, I mean, we believe it's a more sustainable way forward also in terms of compute and how it distributes. It's a distributed system. of the there is a part of theoretical computer science like a very big community actually that that deals with distributed systems um and it is it so for those listeners if they're here it is a

58:03Zuzanna Stamirowska:it's effectively a man like a distributed system uh the uh is is there the reason you're not looking at scaling is because the number of uh connections between the nodes or the neurons in this graph uh can can you can pack enough in that you don't need to scale the number of parameters or or i mean it could be that if you scale uh you'll you'd be able to contain that much more information It's possible.

58:45Craig Smith:It's possible. So I'm not saying that we don't need to reach the scale. Our success is not blocked by the size of N. I would say it this way. So for us focusing on larger N, it's probably fun. But I mean, our goal is in fact to get to this reasoning in places where it wasn't possible before. And so I'd say we have a slightly different objective. And for this, as of now, we didn't see the scale as being the main bottleneck. And yes, this structure allows us to pack a lot in the connections, not in the end, but in the synapses, and then keep it apart enough because we activate only parts of it and such that when we touch one part, we don't necessarily touch the other part of the of the model so i mean hence hence some of the nice properties for against catastrophic forgetting um and like the plasticity is very very important in in this entire work

59:49Zuzanna Stamirowska:you know with transformer architectures uh once they were validated and they exist then there's this whole second layer of activity in combining models or combining different kinds of models into larger systems. Do you think this would work with traditional LLMs? You know, maybe, I don't know, maybe the transformer architecture handles one thing and the post-transformer architecture handles something else.

1:00:31Craig Smith:Oh, in that sense. So, I mean, in terms of maybe putting models together, I mean, at the engineering level, I believe there will be different use cases for which different models will be used. And, you know, for your traditional knowledge type of use cases where we have language models, I mean, they are doing phenomenally well. and I believe that a lot of the right now chatbot types of use cases, I mean, they will stay with the ELMs and they will be doing great. So, you know, I believe that the models based on BDH and other architectures, I mean, they will be used in use cases that, for example, require consistent reasoning over a longer time, right?

1:01:20Craig Smith:So with fin data. So we're definitely looking at enterprise use cases naturally, or deep innovative research, where you were potentially with also changing inputs, right, over time, because you may want to kind of maybe iterate along with the environment as things progress. So I know in the engineering I imagine you can use different models for different bits and it's kind of a question of just building the system in which you can do it. On the more fundamental way, can you glue BDH with a transformer and have kind of one model that somehow works together? Because I mean, this is what some people who modify attention do.

1:02:08Craig Smith:This is not the case. This is not the case. So this is actually either transformer or BDH, because it's so different, so fundamentally different. It's not, let's say, a plug into a transformer or a plug-in. Then there is, however, another just anchoring maybe on the merging type of or gluing angle. Because we have just this dimension n, like just the neurons right it's a graph and it's a distributed system you you can totally see that you're taking those brains and you kind of train them separately and you build them together and then you maybe have a couple of runs of training together such that they form connections between them and then becomes even stronger and in this sense you can get you know not not a separate mathematician and a separate separately trained uh computer scientist but all of the sudden you actually have a mathematician and computer scientist who combines the intuitions of the two disciplines in one um or you know this can work for finance and law you're talking about

1:03:19Zuzanna Stamirowska:training uh separate uh graphs and then merging them is that right yeah then gluing them and we

1:03:27Craig Smith:glue them along this one dimension n uh we actually show a small experiment in the paper for this where we actually didn't do the kind of the round of training when they were glued together but we actually on language we do we're going to glue the two the two models train into two different languages and then all of a sudden they start to speak uh you know some sort of like i mean like the language that make makes sense but mixes the words of of of of the two languages point is it's very it's very simple with uh with this architecture because because you effectively you look at the graphs and you just put them together.

1:04:05Craig Smith:So it feels a bit like composable programs.

1:04:09Zuzanna Stamirowska:And it sounds, you know, again, Sakana AI came up with this, I think they call it model merge, where they can put two models together. And that's part of their evolutionary strategy that they'll have a bunch of models, models generate output, they pick the best models and merge them into, presumably you could do that as well with BDA?

1:04:48Craig Smith:Presumably you could. I mean, I haven't explored it in full honesty, but yes, you could technically because you can merge them And if you want to train evolution, do some sort of kind of evolutionary pruning of what's your best model, then probably you could do it. I don't know if it has a purpose in our case, but technically, yes, the merging is it.

1:05:15Zuzanna Stamirowska:That's all fascinating, Susanna, really. And yeah, because Transformers, it seems, have kind of hit a plateau or are hitting a plateau. So it's fascinating to see new architectures emerging

1:05:37Craig Smith:or new strategies emerging.

1:05:40Zuzanna Stamirowska:Yeah, okay, well, let's leave it there. Is there anything I didn't ask that you'd like listeners to know maybe where they can go and find BDH? Yeah,

1:05:55Craig Smith:thank you. Thank you so much for this conversation. It was great. And I mean, these were pretty deep questions. I really appreciate it. So, I mean, you can reach out to us. I mean, for me, LinkedIn is probably the easiest. Also, research at pathway.com. This is for all the kind of research types of questions. I mean, we're very, very happy to answer there. Please note that there is, the paper is available on GitHub. To digest it, it's pretty long. I mean, Craig, I think you've got it. It's very deep. It's very dense and very long. Because it has to, it covers a lot of intuitions, actually a lot of proofs as well.

1:06:38Craig Smith:because we're showing the link between transformer and the brain that goes through expressivity. So all of this goes through computer science. If you're to go computer science, that is not always the thing that's being the most, let's say, studied in the AI community. So I can strongly encourage something that we've seen working a lot very well with many people. Actually, by now, LLens are doing a phenomenal job of organizing this paper, especially the reason the most advanced reasoning ones. Once we ask them to properly read the paper with the proofs and then break it down, they really do an amazing job.

1:07:16Zuzanna Stamirowska:So I would strongly encourage you to use AI to kind of read about AI. Yeah. I read it on archive. It's the same paper, right?

1:07:28Craig Smith:It's the same paper, and the paper is publicly available, of course. And then, I mean, there are also other podcasts you know that are that way more technical with with other members of of our team so i would also uh i would strongly invite you to to listen to this if you if you have questions uh this one was great i mean thank you thank you so much