Tiny Recursive Networks

0:15i'm the

0:23curl discord Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, on to the show. Well, friends, it is time to let go of the old way of exploring your data. It's holding you back. But what exactly is the old way? Well, I'm here with Mark Dupuy, co-founder and CEO of Fabi, a collaborative analytics platform designed to help big explorers like yourself. So, Mark, tell me about this old way. So the old way, Adam, if you're a product manager or a founder and you're trying to get insights from your data, you're wrestling with your Postgres instance or Snowflake or your spreadsheets.

1:08Or if you are and you don't maybe even have the support of a data analyst or data scientist to help you with that word. Where if you are, for example, a data scientist or engineer or analyst, you're wrestling with a bunch of different tools, local Jupyter notebooks, Google CoLab, or even your legacy BI to try to build these dashboards that someone may or may not go and look at. And in this new way that we're building at Babi, we are creating this all-in-one environment where product managers and founders can very quickly go and explore data regardless of where it is. So it can be in a spreadsheet, it can be in Airtable, it can be in Postgres, Snowflake.

1:43Really easy to do everything from an ad hoc analysis to much more advanced state analysis if, again, you're more experienced. So with Python built in right there, NRII Assistant, you can move very quickly through advanced state analysis. And a really cool part is that you can go from ad hoc analysis and data science to publishing these as interactive data apps and dashboards. or better yet at delivering insights as automated workflows to meet your stakeholders where they are in, say, Slack or email or spreadsheet. So, you know, if this is something that you're experiencing, if you're a founder or product manager trying to get more from your data or for your data team today, you're just underwater and feel like you're wrestling with your legacy, you know, BI tools and notebooks, come check out the new way and come try out FAPI.

2:29There you go. Well, friends, if you're trying to get more insights from your data, stop wrestling with it, start exploring it the new way. with Fabi. Learn more and get started for free at fabi.ai. That's F-A-B-I dot A-I. Again, fabi.ai.

2:59Welcome to another fully connected episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined as always by Chris Benson, who is a principal AI research engineer at Lockheed Martin. And in these episodes where it's just Chris and I, we like to dive into certain topics that are trending in AI news and help both us and you hopefully level up your AI and machine learning game. How you doing, Chris? It's good to be back to one of these episodes with just the two of us and maybe explore a topic that we can both learn about. Absolutely. Yeah, I love these episodes of us just kind of bantering whatever we happen to want to do.

3:48I love the guest episodes too, but it's kind of a different beast in that way of exploring what some person or organization is doing. And there's so many cool things that we can just dive into. And I think we have a few this week. Yeah, yeah. At least a first very, very tiny topic to discuss, which I was actually one of our engineers brought this up to me. I forget it was earlier in the week, but this idea of tiny recursive networks, you know, all the time we're talking about transformer based LLMs on the show and generative AI. And I kind of personally always love getting back to a little bit of cool data science and research stuff just to see where the industry is headed.

4:40Because this is a kind of different animal that we'll be talking about, this tiny recursive networks or models. And it operates differently than kind of the hype Gen.AI models of today. And it does make me think, and I don't know if this is something you've been thinking about, Chris, but kind of we're all the time talking about Gen AI. We're talking about LLMs. Now we're talking about agents, all of those being driven by these transformer based LLMs. And certainly people have talked, you know, prominent people have talked about the fact that we need to get beyond transformer based LLMs. And of course, there's many companies that are just centered around these types of models.

5:25So any thoughts on that of, you know, your own predictions or thoughts of when we're kind of headed to the next phase of what models will look like beyond just transformer-based LLMs? Yeah, I mean, I think I'm going to sound like a broken record on this because it's not new for me. And that is, you know, I agree with you. We're always, you know, the hot things that the media tends to follow in general are the big LLMs. But, you know, because I guess it's, you know, it's the next giant thing. It's sexy, you know, to talk about. But like the world is, you know, all these technologies are moving from the cloud out into the world into physical AI, you know, and robotics and all sorts of ways that we interact with, you know, not just LLMs, but all sorts of models out there in the world to where, you know, every one of our lives is touched in so many different ways.

6:23And that's exactly, you know, as we were diving into this here, this topic with the tiny recursive networks, that's what it seems like to me. I'm looking forward to talking about that because, as I mentioned to you right before the show started, you could see these popping up everywhere, like in all sorts of different use cases. Yeah. And just to set the stage, why this is maybe intriguing, there was a paper that came out, Less is More, Recursive Reasoning with Tiny Networks. This came out from Samsung's AI lab in Montreal. Specifically, there's an author, Alexia, on this article, which if you're out there listening, we'd love to have you on the show.

7:09Please come join us and talk more about this. Hopefully, we won't butcher this work too bad as we talk about it. You'll hear Chris and I kind of learning as we go on this episode, as we talk back and forth about what this exactly is. But this is, you know, the paper is less is more cursive reasoning with tiny networks. And I think the major thing that's interesting here is there's a model that they talk about that has only 7 million parameters, which is tiny. Yeah. Yeah. So like, I just want to sort of let that sink in. So I didn't say 7 billion parameters, 7 million parameters with an M, which yeah, Chris, if, if as we've basically gone about this trend, I mean, actually a 7 billion parameter model now is quite small.

8:05Yes. And in a sense, 27 million compared to what we traditionally call very small at the 7 billion level. I mean, this is that, you know, it's using the word tiny for a reason. But yeah, it does. You know, when you think of millions as being as being almost nothing, it's it's an interesting context shift there. Yeah, yeah. So I actually love this because I I love the idea that we could move into a phase where we're dealing with models that are very small, can run on commodity hardware, or at least be smaller in size. And they may run for longer periods of time, or there may need to be optimization around how they run recursively.

8:47We'll get into that recursive bit, but certainly a small model. But it was shown to kind of have, let's say, comparable or on par performance with some of the big guys. So we're talking like DeepSeek R1, Gemini 2.5 Pro. These are billions and billions of parameters models, very, very huge transformer-based LLMs, these kind of reasoning models that we've talked about on the show. And the center of this work is really kind of related to these reasoning tasks. Now, right out of the bat, I think it would be worth saying that it's not like this tiny recursive network is a general purpose model that can do whatever you want it to do.

9:35It was trained for a very kind of small number of tasks, but these were reasoning tasks that some of these other models like a DeepSeek R1 or something sometimes has quite a bit of issue with. So solving like math or Sudoku type of puzzles. And I know, Chris, we had talked to, I don't know if you want to refresh some of your Sudoku experiments. Yeah, well, what you're referring to is some episodes back, I'd have to look up and figure out where it was. I was playing with GPT-4 at the time on Sudoku, and it was just doing a terrible job on Sudoku and giving a lot of just really bad output in terms of, I mean, honestly, crap answers.

10:26and so that was really the first thing I noticed on this thing was the fact that because they call out Sudoku as being one of the things is that these tiny models being trained for very specific tasks and that it could potentially outperform these large models on specific things like Sudoku being one example and others as well but I think in a slightly larger sense this is much more real world applicable the way I see it in that as we have models spreading across the world for lots of different tasks, this is perfect for that. It's not one model to rule them all in most real life situations. It's really a collection of very specific models that each does a task very, very well and is efficient at that.

11:12And I think this is a great example of that. Well put. And it's probably worth reminding ourselves about, like as we highlight the differences with this model, reminding ourselves about transformers. So as we've gone through the process from deep learning to recurrent neural networks and transformer-based self-attention networks, if you imagine what we have with these big LLMs, what happens is you put in a sequence of tokens, which are represented by numbers. You put in a sequence of tokens that are represented by numbers, these tokens being kind of words or subwords. And all of those tokens are processed in a forward pass through a giant set of, if you want to think about it, like sub functions, which add and multiply and combine those numbers through a very vast network of functions to generate many different probabilities of kind of next words coming out, which allow you to predict kind of a completion of words or a set of reasoning or a set of thinking or a solution to a problem, right?

12:35This is how these networks work. And I would recommend people, we've had Jay Alomar on the show before. He has some great kind of the illustrated transformer blog posts. So Jay, shout out to you. Thanks for doing that. I would take a look at those blog posts. They do a great job at explaining this more visually for those that where that would be helpful. But the main kind of thing here that I'm saying is in the models that we're using now, basically it's a giant function, if you want to think about it that way. It's a data transformation. You put something in one end, it processes through one way through the function and produces a result.

13:18Now you may run that function multiple times to produce multiple words out the other end, which is what happens when you stream output into like a chat interface. But ultimately, each time the model runs, it's a single run through the model input, it's transformed to some output. That is not recursive, as we would say with these models. And if you want to think about it, it requires very, very large models, because what you're modeling is a very complicated data transformation. So for you to put in some text related to a math problem and predict the right wording of a solution out the other end, that's actually a very non-trivial data transformation, right?

14:11Which means you have to have a very large function to kind of fit or model that data transformation, which is why these models have become so large. So now with this tiny recursive setup, what's happening is you're not just looking at the model as a single forward pass data transformation, but you introduce the idea of recursion, which means you sort of output from the model. And that output becomes the input for the same model, which creates this kind of circle or recursion, which is kind of interesting. So you're essentially trading what would be a very large function to model that data transformation for many, many kind of recursive runs of a single, very small model.

15:13That's maybe a simplified way to put it. And we can get a little bit more into the model itself here in a second. So I have a question for you on this is one of the things when I was mentioning the 27 million earlier, I was talking about the hierarchical reasoning model, which is a previous model put out there versus the tiny ones, which have the five to seven million parameters. Can you talk a little bit about like, are they completely different things? Is the tiny a an outshoot of the hierarchical? How do you how do you compare those two? Yeah, good, good point. So just like everything we talk about on this show, sometimes it does seem like things pop out of thin air, like tiny recursive models now.

15:55But in reality, there is a buildup of incremental research that leads to new technology or new findings. One of the things in that kind of lead up to these tiny recursive models was a previous work around hierarchical reasoning models. And these also were smaller. Like you're saying, 27 million parameters is still very small in today's standards for models, at least. But these hierarchical models actually use two very small transformer networks, four layers each, and they recursed between each other. So I don't have the full details of that and wouldn't be able to explain it if I did, probably.

16:47But that's the main idea is these hierarchical reasoning models had these two models that required two networks, two forward passes per step, and created some, I guess, complications because of that. So this introduction by Alexia and the Samsung team here is a single network. So it uses one tiny network with two layers. So a single tiny network with two layers that roughly has kind of five to seven million parameters. And it operates kind of in this recursive refinement. So it recurses on itself, if you will. And the hierarchical reasoning model with the 27 million parameters, for example, on the Sudoku Extreme benchmark scored a 55%.

17:42I don't know the exact kind of way that that's scored, but just by kind of comparison, the tiny recursive network, which is even tinier, when training on a thousand examples was able to achieve 87 % accuracy on Sudoku Extreme.

18:05Well, friends, you don't have to be an AI expert to build something great with it. The reality is AI is here. And for a lot of teams, that brings uncertainty. And our friends at Mirror recently surveyed over 8 ,000 knowledge workers. And while 76 % believe AI can improve their role, most, more than half, still aren't sure when to use it. That is the exact gap that Miro is filling. And I've been using Miro from mapping out episode ideas to building out an entire new thesis. It's become one of the things I use to build out a creative engine. And now with Miro AI built in, it's even faster. We've turned brainstorms into structured plans, screenshots into wireframes, and sticky notes, chaos into clarity, all on the same canvas.

18:53Now, you don't have to master prompts or add one more AI tool to your stack. The work you're already doing is the prompt. You can help your teams get great done with Miro. Check out Miro.com and find out how. That is Miro.com, M-I-R-O.com.

19:14So, Chris, just to kind of drive, I guess, the point home here with these tiny recursive networks, You have this single tiny network and you are essentially replacing, if you want to imagine a big kind of pipeline of processing, which is what these big LLMs are, and you go one pass through the whole pipeline of processing. Here, the pipeline is smaller. You've got less layers, less parameters, but you replace that kind of depth with iteration. So instead of stacking those transformer blocks, you repeat the network over and over essentially to kind of refine its reasoning state or the solution guess.

20:06Right. And so this this iterative refinement, one of the things also is it kind of helps avoid overfitting on small data sets, which to your point, Chris, earlier about real world business cases, often the the reality is that you kind of don't have you have scarcity of data for very many problems. You don't often have a really nice kind of large millions and millions of things to train on. And I think that's one of the most fascinating things about this is in the paper, they talk about the fact that they're achieving this higher accuracy on hard puzzle benchmarks while training on only approximately a thousand examples.

20:52and uh and that when you think about you know the the challenge of having a great data set in the more traditional context that we've been talking about and that becomes such a challenge for many people and organizations to do that but it's a lot easier to get a thousand you know a thousand examples together and it puts you know not only from the computational side uh but also from the data set side, puts this much more in reach for a lot of problems that people may have, where they do want to solve a narrow concern with high accuracy. And so I see this as it's kind of the every person's way of modeling in terms of tackling things going forward without a lot of resources and a lot of maybe not a lot of time to put things together.

21:42You could probably do it pretty quickly. Yeah, yeah, exactly. And I guess just to kind of put some of the boundaries that are currently around these recursive models, one of the things I was trying to parse through as I looked at this was, well, what is the setup now? How general is the output? How kind of general can the output or the input and the output be? And part of Part of the trick here, it's not a trick, but part of the setup is that these tiny recursive networks, they don't take a kind of unstructured natural language text input. They take some structured representation of a whole problem at once.

22:36So you can think of a puzzle grid in Sudoku or a math word problem turned into structured features or a reasoning question encoded by numbers or symbols or logic, something like that. So instead of feeding in the input kind of word by word like a chat bot, you're giving a situation to the model. It's kind of a one-shot situation, which is then turned into, of course, embeddings internally because computers work on series of numbers, right? That's the only thing a computer can process is numbers, right? But those numbers, kind of that embedding represents a kind of one-shot of a problem, which is interesting because, you know, it almost seems flashback, like we're kind of coming full circle to reasoning problems, but in a more data science-y way than like a generative AI way, which is kind of refreshing and cool.

23:41That's very much what I was about to say in the sense of it feels, this feels a lot more like kind of traditional, like the way that you put a problem together in more of a traditional software development way where you you know you'll create some structures and you'll pass them in and when we got to gen ai and then it got to prompting and we're trying to that was you know the the notion of prompting was a little bit different from the way we had traditionally put software together this feels a lot more like okay i have a problem i have a structured way to put that problem through the function and this is just offering a different way to address that problem.

24:20So you're getting the benefit of these models. But for me, when you talked about using the Sudoku example and structuring that as the grid, that kind of feels like what we've always done in a sense. So I find that really interesting in terms of integrating that in and looking at some of the old problems that we might've been trying to solve for years and, uh, and seeing what we can do with this model to do it a little bit better. Yeah. It's like, uh, looking, looking at things from a different perspective, but, uh, some of, some of the way that we used to think about things kind of filtering in.

24:59Um, and yeah, so you have this kind of input, I guess, in terms of people's intuition or mental model around this, like you have this input of this whole, whole problem at once, a single shot of a whole problem, a puzzle grid or a math problem encoded. And what's happening inside is that the tiny recursive network initially produces an initial guess, right? Like it does a forward pass through its network and generates an initial, you could think about it as an initial guess. Obviously, it's just a number, like a probability, but that's kind of the initial, you could think about it like the internal kind of scratch pad of the initial guess, it then kind of loops over itself.

25:46And, you know, it's always difficult to anthropomorphize because things work differently in computers than they do in our minds, right? But in some way, if you want to think about it as that sort of process, that looping is kind of a refining of that initial scratch pad until you kind of get to this almost like self-consistency or a refined answer. And so when the output comes out, it's again not a stream of words or tokens, but it is a complete answer. It is the answer to the kind of initial thing, but that answer was arrived at through this recursive thing. So in terms of just some highlighting some differences in the transformer world, you put in words or tokens.

26:33Tiny recursive network, you put in a whole problem as structured data. Transformer world, you go a single pass through hundreds of layers. Tiny recursive network, you repeat the small network recursively. In terms of the output in the transformer world, you kind of get these next token probabilities. The recursive network, you kind of get one final structured answer. and in terms of analogy if you want to think about it in the transformer world it's sort of like your free form typing as you think about your answer right you're just sort of vomiting up your reasoning onto the onto the screen and typing as as you go along and then the recursive network it's more like it's just kind of chugging along it's thinking quietly right and then boom there's the answer.

27:28Like it's a complete answer when it comes out. I'd be curious, and I don't know if you've seen anything on this. I'd be curious both what to expect from training times with networks of this size, which I would expect to be, even with recursion, to be pretty fast, but also what inference times might be. In other words, if you were to take these, train them, put them into a device where you're looking for maybe real time or near real time, uh, inferencing there, uh, is that, is that reasonable? Have you seen anything yet in, in any of the research that you've read about what that timing looks like?

28:05How is it incredibly screaming fast given the small size despite the recursion? So there's a couple of things to kind of parse through here, which is one, we've talked about this recursion, but the thing is, how do you know when to stop the recursion? That's part of the answer to your question. So you're refining this answer. And there's various ways to do that. And I remember actually, this is, I guess, a deep cut, but I don't get to bring it up very often. Back in my physics days, I worked on a theory called density functional theory, which essentially models out material properties. And it was a self, we talked a lot about self consistency, which is what is happening here.

28:53So you ran iterations of your model until you arrived at a solution where there sort of wasn't, there wasn't that much change in your answer. You sort of got to a steady state, if you, if you will, there wasn't a change from one iteration to the other. So that's one way, actually, you can run this type of model is with this kind of change threshold. The other way is you can just say, well, I'm only going to run it so many recursions, right? Like X amount of recursions, you know, eight loops or, or whatever. That kind of is a nice guarantee, but it doesn't necessarily mean you get to the, to the good solution.

29:33You can also, it could be possible that you could have a second kind of network that learns to predict when there's kind of a good outcome state. So there's actually a variety of ways that this could work. Now, in terms of the training time and the inference time, I think there's still a lot to be learned here. So at least in, I guess, no pun intended, a lot to be learned. But even though there's sort of a small network here, it means each kind of training step is cheaper. But the training time depends on how many kind of loops they need. So if there's problems where kind of the examples converge in kind of a few loops, then the training is much faster.

30:31And if they still need lots of loops, then it slows down. And so the other piece of this is that we've had this entire industrial complex that has optimized training frameworks and tooling for big LLMs, right? And so actually kind of in this more research environment, I think it has been kind of slower to train some of these recursive networks. Now, I would imagine that's kind of a result of both of those contributing factors. But I think if you're looking at the inference time, you could think like, well, these could only internally kind of loop for a few loops and then give a full answer. That would be very, very fast.