The New Era of AI Reasoning with Nate Jones

0:00But then there's another scaling law that's operative over the top of that and that's reasoning. and reasoning works differently. Fundamentally with reasoning, if you give the model longer to think, regardless of the training data that it has against, you dramatically improve performance. Welcome to the Talking AI Podcast, where we talk AI with both experts in the field and early adopters. I'm your host, Matt Page, and we're here to demystify AI for you so you can get some value from it. Let's talk some AI.

0:36AI still has a lot of naysayers saying LLMs are just fancy autocomplete, but with the latest reasoning models being introduced, along with some of the novel approaches being architected for reasoning, you have to kind of reconsider how you've thought about LLMs to this point. And with us to break down reasoning and a whole host of other fun rabbit holes, the AI gray beard, head of product, the rocker box, AI tiktoker, Lego builder, and just generally smart and introspective dude, Nate Jones. Welcome to the show, Nate. Hey, I'm really glad to be here. Excited to have the chat today. Yeah, I've been looking forward to this one for a while.

1:11And if you don't already follow his TikTok, go do it. He's one of my favorite follows. Really cool, just like deep conversations and thinking that people aren't going down. But Nate, to kick us off, let's start with LLMs in general. How should people be thinking about these? And then we'll kind of, you know, weave our way over to the reasoning side of things. Cause I think a lot of people view them in maybe the wrong lens in a sense. You know, I, I think we've been fooled by how bad they used to be. And so a lot of people have a prior mental model that got stuck somewhere in 2022, where you had pretty basic queries that large language models wouldn't help with.

1:59Even earlier, natural language processing was an earlier architecture than LLMs, and it was worse. And I remember building systems with natural language processing, and it was really, really difficult to get it to understand that there's a hundred different ways to ask the same question with the same intent. And getting it to sort of grok that was really hard. And these days, we're just not at that point anymore. I had a conversation with a voice model that specifically has a double transformer architecture that gets exactly at intent versus specific utterance yesterday. And that's not really unusual in 2025.

2:36I can name a bunch of other models that do that just fine. And so I like to think of this sort of watershed moment that we had in 2024 where we can start to reliably say, yes, the primitives are there. Like we can talk about next token prediction all day long and say, really, all this does is it takes an input and it goes and works against a training data set. We'll talk about reasoning and it comes back with an output. So do humans, right? Like, and so I think in a sense, like we, we, we look very critically at that and say, well, it's just a machine doing calculations. But if we look inside our own brains, there's neurons firing.

3:17Like we can talk about our own primitives. and so the the mental model i have shifted in 2024 from these are smart inference models for specific applications or smart models that can answer queries across a limited range of use cases to we need to default to using them and really we can reasonably say 2024 was the year that the rocks began to think and now we're just going on from there yeah i think the first point you started with is so critical. And I've seen this across every spectrum from somebody that's never used AI before to engineers that are some of the smartest people I know that tried it, got stuck, went back to old habits.

4:01And it's, it is, it's like this like mental hurdle you have to go over, especially if you were trying it in the early days to, to like, A, it's the adoption factor, but it's also the changing of habits factor, which I think that's the harder one. in a lot of ways for us humans to do. It is really challenging to change our habits. And I think one of the things that's especially difficult is that the invitation that this kind of software presents is really to partner with another intelligence to solve a problem. And that's a fundamentally different challenge for people adopting a new pattern than other software challenges that we've had in the past.

4:40You buy a new iPhone, you wanna touch your new iPhone, you wanna play with your new iPhone, and you sort of discover how to use it because it's a shiny object. If you get a piece of software you have to use for work, you can use the software. You can look at the help docs and figure out how you need to use it for your job. This isn't either of those cases. This is a chat bot that invites a conversation. And really the quality of that is dependent on your ability to evoke the intelligence behind that chat space in interesting ways. It's there. You just have to know how to ask about it. Quick break in the pod.

5:13If you're listening to this podcast, chances are you've been thinking about how to actually use AI inside your business. And that's exactly why we built the AI Opportunity Finder. It's a free tool that helps you uncover high impact, tailored AI use cases based on your business, your goals, your pain points, and your industry. No fluff, no generic use cases, just real ideas that fit your business and the ranked by ROI potential. It takes about three minutes to run and it's like having your own personal AI strategist for free. If you want to try it for free, check out the link in the show notes or go to hatchworks.com backslash AI dash opportunity dash finder.

5:51Yeah, that's, yeah, there's so many rabbit holes here. I think the context factor, the how to interact with it, but I got to go back to the voice agent for a second before we get deep into reasoning. I think what you're talking about is the recent Sesame AI that came out, but on the same vein, I was, you know, we got a workshop coming up and it was, you know, what's a fun example to play with. So I went and built a solution with Bland AI, another tool, very similar to that voice agent and kind of a funny example to where you give it a persona and the whole premise is you're, you're practicing arguments with your spouse, right?

6:27So you give it a mock scenario, you call it and you have this discussion. But when you start, especially people that have not done this as of late, it's insane. how how like the nuance it picks up on and i did this scenario of i want to go play golf on sunday my wife wants me to stay home and help with the girls and do chores and the nuance it picked up on was was crazy and the reactions it had i'm curious because i you were just digging into this a few few days ago and this will still be relevant a big thing in 2025 what's your take on like the the voice element and the nuance it's kind of hard to describe in a lot of ways i actually have a specific example I used yesterday with Maya, which is one of the voice agents that Sesame produced that I wanted to try with Maya to practice emotional resonance and emotional reading just on inflection.

7:19There was no change in the words. I wanted the words to say exactly the same. And I tried three different ways of emphasizing the words just as a human word. And humans understand intuitively what the same words with different emphases mean in English. So I'm going to give you the example, like Maya, you won't have any trouble understanding it. Neither will any of our listeners here, but neither did Maya. And that was the really creepy thing or the exciting thing, depending on how you look at it. But the first one I did was, wow, I had a day today. You know what that means? You know, you had a tough day.

7:56But if I say, wow, I had a day today, totally different meaning. Same words. Something good happened. And Maya picked it up. Maya got it. You're kidding. So it was able to provide a contextual response. I think the nuance too, it's not only contextual with the actual words it's saying, but I'm sure the tone in her voice, Maya also picked up on it. Maya got more empathetic when I had a rough day. Yeah. And Maya was self-reflective and could tell me why she interpreted it that way when I asked her. Yeah, man. AI agent, girlfriends, and boyfriends. They are coming right around the corner to the store near you.

8:33Yeah. Wild. So the other thing too, and then we're going to jump into reasoning. Cause I think this is like a whole nother ball of wax, but you mentioned how you interact with AI. And this is another piece I feel people get so stuck with. They, they treat, treat it as like they're the dictator and they're not trying to be mean, but like the dictator, they, they're just like barking orders. They just assume AI is going to know what they want context, all that. And like the biggest thing I try to talk about is context is king, right? The context you give it is one of the most important things. And also how you're, it's like you're almost priming AI in terms of telling it to think logically.

9:14Like having a conversation about something before you have it go do a task wildly changes the output and just the interaction in general. But I'm curious, like either your take on that or just other like nuanced, like ways you interact with AI. I think a lot of how you work with a large language model comes down to sort of the fingertip feel for where it's going to be good and where it's going to need handholding and where it's going to be useful. And so I'll give you a couple examples, but we could probably pick a hundred of them, but we'll do a couple. I was messing around with annual review season and I thought, why not have AI help a little bit?

9:55But instead of trying to sort of bark orders at it and say, help me write the annual review, I decided to take a different approach. I said, listen, this is always a complicated, introspective process for a human. It's not easy. We have a hard time separating our own sense of performance from our sense of ego. And we also have faulty memories, to be honest, over a longer time frame. So I need you to interview me and talk me through the year and ask me questions about the year. we'll get to the output later just help me to get my thinking out and i think that just that shift and like again that core understanding of saying like the llm can be good at interviewing if i ask it to be and i don't try and one shot the entire output at once yeah um and on the other side of it like i have been in situations where i'm like i i think this isn't working so i was talking with chat GPT 4.5 a couple of days ago.

10:52And I was like, I need you to sort of draft this piece for me. And it got into a doom loop where it said, I'm drafting, I'm drafting, I'm drafting, I'm drafting, but it couldn't come up with it. When it finally came up with it, it was really, really short and really not adequate. And some people give up at that point. A lot of people, I think, give up at that point. They say, well, the AI can't do it. And they make larger generalizations about what AI can't do. I look at it and I say, what did I not do correctly to set this LLM up for success? It's almost like you're managing a person. I didn't set it up for success right.

11:28So how do I think about this differently? And so I went back and I looked at my inputs and I looked at what it had and I said, I think that there's an issue with your output tokens. I think they're expensive output tokens and you've been told to restrain them. I don't think that's necessarily on you. And with that in mind, can you give me a summary of what we've chatted about so far? And I'm going to go and I'm going to take it to a different LLM that has a little bit of a looser token limit, and we're going to work with it and we're going to get the piece that we need. And we did, and it worked great.

11:56That's so cool. Yeah. And it's, you mentioned like the interview use case and I was trying to find, I can't find the name of the product, but I'm doing it, recording a podcast episode later today. I'm like double booked it, but with Andrew Miller, with the, his prompt driven development. I was chatting with him prepping for the episode and whatnot. And he brought up a friend that's building a product. Of course, he's building AI. Yeah. Yeah. So I got to find it. We'll put it in the show notes if I can't find it. But the friend is building a whole new take on a survey product, right? So you think of a traditional survey, stock questions, you answer them.

12:31You may have a little bit of nuance where they answer this, then ask them that. he injected AI into the survey experience. So it just based on how it's the context, the system prompt it's giving, it can go down a nuanced rabbit hole based on a response that it's been giving. It's just a whole new novel approach to surveys. And that's what I can't wait for is like jobs to be done being solved in new and novel ways, like existing ones. It makes it really challenging. we're going to see a tell actually like i was chatting with someone i know over at microsoft who runs a product team and one of the things we were talking about is that the way product teams evaluate success and design requirements has fundamentally shifted with generative models yeah like it used to be this is deterministic software so we can write in advance all the requirements are and then the engineering team goes and build it and qa tests it and we're done And now it's play with the model, discover the requirements by playing with the model, and then go back and figure out where the mismatch is and where the model might need to be tuned and then figure out the product surface from there and then figure out your edge case is there.

13:44It's like, it's much, much more complicated than it used to be. Yeah. And so I just found the product, by the way, anybody that's interested, judo.co, J-U-T-T-U, really, really neat concept. But we're going through that now with what do our teams look like? And I've almost gotten to this state where a development team for, you know, a smaller solution, Greenfield solution, you could whittle it down to an agentic engineer and an agentic call it product manager, product strategist, right? And they're leveraging AI in their own capacities, doing a lot of the functions of BAs, scrum masters, acute DevOps, all these different things.

14:25But the point you mentioned is really interesting. And that's where we're trying to, we're flipping how our product folks think about it. AI is great at identifying the unknown unknowns. That's always been the big issue when you're building something is, okay, there's no knowns, known unknowns. I know the thing I don't know yet. And then there's just unknown unknowns, complete gray area. AI is actually good at saying, oh, did you think about that? That's been a really interesting nuance to pick up on. I think that AI is starting to fundamentally disrupt the entire product discovery funnel. Like we used to have a really linear process for figuring out what it was worth having an engineering team built.

15:07And be like, okay, well, first we have to have the idea. Then we have alignment on the idea. Then we go and work with design to get a lightweight mock-up maybe. Then we talk to a user, put it in front of the user. Then we go back. Then we iterate. Then we have a medium-by mock-up. Depending on how big a company you have, like this is going to be a very long process. And AI just disrupts all of that. It can maximize the ideas at the top. It can get you clickable prototypes in 30 seconds. At this point, it can even get you some workable front end code really fast. And it's challenging PMs and designers to get back to the judgment that's at the heart of the discipline, which is isn't the right thing to do.

15:45Yeah. And you have many more tools to get you there faster, but like you still have to pick that choice. Yeah. 100%. So that was a fun, like 10 minute rabbit hole there, but let's get back to reasoning. Let's get back to reasoning. So this is, there's these great, you know, frontier foundation models that exist. They do these awesome things. And, you know, relatively recently, this new approach of reasoning and there's, we can go down some rabbit holes with the different types of approaches with like chain of thought, mixture of experts and all these things, but this has been introduced. So now, now give us the wrinkle of how people should think about AI when you add a reasoning component to it.

16:24I think of it as, this is probably my brain, but I think in graphs, like we have two scaling laws that are operative at the same time. One of them is, I think, well known to most people at this point. And the idea is that if you logarithmically expand your data set that you train on, your training data, you are going to incrementally increase your capacity of your model to solve problems. And so roughly the relationship is a 10 X increase in your training data. It leads to about a two X increase in model performance. Yeah. That seems to continue working. Like the latest example that we have of that is GPT 4.5 versus GPT 4.

17:07We came in exactly where we would expect it just a touch over double performance but then there's another scaling law that's operative over the top of that and that's reasoning and reasoning works differently fundamentally with reasoning if you give the model longer to think regardless of the training data that it has against you dramatically improve performance it's from what i've been able to see like it's compute intensive, like it takes a lot of tokens, but it's also a very, very fast way to improve performance. And that is why we see such dramatic gains against these evaluations, like humanity's last exam in just a couple of months, because essentially what the major model makers are figuring out how to do is how to give the model time when you ask it a question to reason before it comes back.

18:00And when we talk about reasoning, I think we sometimes have this picture of this like this Rodin sculpture where the guy is like thinking carefully but what what what is happening in the background is the model is burning tokens that allow it to explore multiple possible solutions and then selecting the solution or output stream of tokens that it thinks is most effective and that can get more and more sophisticated so there's a huge jump in sophistication, for example, in just a few months between O1's output, and it's a reasoning model, and O3, which is what you get with deep research now. And if you look at the details that deep research will tell you about, you can see the planning motion happening live.

18:45Like it is thinking through the structure of the output it's going to give you back. And the more sophisticated a prompt you give it, if you give it a two-line prompt, you'll get much less back. I've given it a two or three page prompt and I get a 60 page report back. Like it's incredible. Yeah. Yeah. And if anybody wants to experience this firsthand, deep seek, if you're talking with it, what's cool is they actually show it's like inner monologue. It's going through, right? Which is really cool to see. And it's like, it's like, it's talking about you to itself. Oh, maybe they want to know this.

19:18Oh, I should consider this, which is just crazy to think about. And there's kind of like four key approaches, architecture, whatever you want to call them that are emerging. Curious your thoughts on these or are there other ones that are emerging, but like mixture of experts that kind of has, you know, different specialized experts where they can route different problems to them. Chain of thought, you kind of talk through that where it's breaking things down into logical, maybe that's sequential, but breaking down complex problems into smaller ones. tree of thought, right? So that's reasoning in like a tree structure where it's generating potentially multiple scenarios, assessing those.

19:56So it's like taking it a whole nother level and then react, which is like chain of thought, but they're taking action in the process. And these things, I think ultimately it comes down to, it's like systems thinking. And I used to love talking about feedback loops back in the day. These are integrating feedback loops into this system that's just making it insanely more powerful over time yeah you know i think fundamentally just like humans tend to give better responses when we take a minute like what we're discovering is that however you want to sort of architect your particular system if you give the model more tokens to work with in the background it tends to use those tokens well and so we can talk about the combinatorial sort of power of different approaches.

20:50Like for example, I think that one of the strengths of, of the tree approach is it gives you a essentially compute limited version of events you can look through. And so as long as you have compute, you can spin up tens, you can spin up hundreds, you can spin up thousands of parallel streams. There's really no limit other than the number of chips you've got. And that can become extremely powerful at scale. And so I think that it's one of the things that we're going to learn more about over the course of the year is how people use these different approaches and sort of mix and match them in the model maker's kitchen to form reasoning models that are good at particular things.

21:33And they're not all perfect. And I think people sometimes say, well, it's the best at this, or it's the best at this. And they forget that we are in a point in model sophistication where reasoning models kind of have personalities like people do. And like, I would go to O3 deep research for reasoning across the web that is leaning toward textual output or maybe code output. It struggles with tabular data. Like it gets lazy with tabular data remarkably quickly for how powerful the model is overall. And so I don't expect it to be super good at that. What similar thing when you're using it on the coding side, right?

22:12Oh, well, on O1 reasoning model, pick your choice graded, like thinking through like architecture and all these kinds of deep thinking things. But if you're having it like just execute straightforward coding tasks, sometimes it can go off the rails versus just call it your, you know, four O cloud three, 3.5, whatnot. So it's, it's, there's nuance in like when to interact with them as well. And I think too, for people, I don't know if you get this a lot, but it's like, which model should I use this one or that one? I think they're, they're all close enough. It's don't let that be a limiting factor to where it's like analysis paralysis.

22:48I don't know what model to use. So I'm just not going to use one. Yeah. I wrote a little bit about this on the sub stack. I think it was called prompt chaining, but I wanted to give people a little bit of a fingertip sense of like where I go with different models for different prompts and why, but you're absolutely right that none of that should stop someone from picking the model that is right in front of them and trying. Yeah. Like what it doesn't hurt. Like you can just try. Exactly. And so I want to hit on one point you mentioned, you're talking about tokens and compute and all of these like constraints, like for, for listeners that may not be as aware of that, talk a little bit about inference, you know, and then how that relates to the cost of compute going down and, you know, whether it's like Jevon's paradox or whatnot, but just opening up more possibilities of what you can do.

23:42Cause people hear about training, fine tuning inference, like what does it all mean? Yeah. I, I think maybe we start that conversation with the idea of a token and we can work our way up from there. So a token is a unit of input that a model can reason against. If you're talking about text or code, typically amounts to about four characters. So half a word, a whole word, if it's a small word. And by the way, one of the reasons why earlier models really, really struggled with counting the number of Rs in strawberry is because strawberry splits in the middle into multiple tokens. And so it really, yeah, that's why.

24:26And it really struggled with counting across tokens. And so tokens underlie, they're like the fundamental unit of economics for the entire system. It all works on tokens. Image models work on tokens. It's almost like ones and zeros for how we think of computers in a sense, right? How fundamental it is. And so at a very high level, a large language model is a system for transforming input tokens that you give it into output tokens. And so that's why we call the fundamental compute paradigm transformer architecture. It transforms. And then you have different ways it can transform. And so one of the ways it's transformed, sort of the thing that caught fire in 2022 when ShedGPT came out is based on data it has been trained on previously.

25:17It's called pre-trained data where the models were. I like to use the example of they teach it to read the internet or they teach it to read all the books in the world and they give it all of this knowledge and then they have it reference that knowledge when you give it an input. And really it becomes an exercise in essentially a smart mirroring. It will come back with a response it thinks works for you based on the pre-trained data that it has. And that's the way large language models worked until we started to get this scaling paradigm around reasoning or what we sometimes call inference. And reasoning adds to that.

25:55Reasoning basically says it's not just about going and getting something out of the library here and coming back. It's not just about thinking about what someone said hilariously on Reddit and coming back with that as a response. It's about taking time to run multiple threads, taking time to think sequentially and form plans, and then coming back with something that is more coherent, more complete, and that better reflects the intent of the user. And that has proved extraordinarily effective. And if you come to compute, what we talk about with compute is basically how much capacity do we have in terms of chips in server racks in data centers to run all of this?

26:38And so we can talk about the number of queries that an NVIDIA chip is able to run at the same time, for example. Because at the end of the day, it all comes down to this is running across a literal chip somewhere in a data center and coming back. And so you're either using that chip capacity for training, where you are having the model practice running and reading all of this data and learning and growing using techniques like reinforcement learning to train the model so it gets to where you want it to go. Or you are using the chips for serving the model. And you can either serve the model, if it's a simple model without reasoning, through a fairly light application of chips, very relatively speaking, these are still huge data sets.

27:19Or you can serve them through more heavy inference compute, where you have to burn more chips per query because it is thinking harder and burning more token. And so when we talk about Jevin's paradox, it's really how much appetite does the world have for intelligence and how does that translate into the laws of economics when it comes to cost of chips, availability of chips, cost of power? That's what we're really talking about. Yeah, I know. That's really interesting. I think it is maybe unrelated, but you mentioned it earlier. I can't get it out of my head, but you know, you talk about LLMs and how they're approaching things, you know, people will quickly say, oh, they're not thinking they're not reasoning.