Do AI Tokenomics Matter More Than Model Benchmarks? with Chris Potts - #776

27:47This year, 95 % of our code will be generated by, you know, AI. It's like, yeah, A, what does that really mean at that level? Like, what are the details beneath there? But is that a good thing or a bad thing? That's another dimension, right? Which is, is that code a liability or an asset? Right, right, right. And this idea of like, you articulate it as kind of code longevity in the code base. That's an interesting way to think about it. There's probably a lot of interesting ways to think about it that very few are thinking about right now. And all of these fall victim to the standard thing that once you make it a metric, it's no longer useful to you.

28:27Like if we said, oh, it's completion of projects, right? Well, then everyone would just have many projects that they completed, but they could all be liabilities and add very little value. So, but one framework we could offer that we did in the research you alluded to is, let's think about this like economists might. So we might have like a consumer price index. And the first step will be, what's the basket of goods that we're going to consider? You know, in that standard land, it would be like the price of eggs and the cost of rent and other kinds of tangible goods. what are engineering goods that we might track?

28:59But eggs might be a summary or a pull request, a bug fix or something like that. Or we could think broadly because we both use these coding agents. Requirement discovery, right? Knowledge accumulation. These are things that we don't currently track, of course, even as engineers, but might be behind our intuition that these coding agents are making us productive, even if it's not reflected in the PR accounts or whatever, right? I mean, in a sophisticated approach, you might say, I don't want more PRs because this is just a certain kind of busy work that doesn't relate to the actual goals I have.

29:34What are the actual goals? It's completing valuable projects and so forth. If I could do it with fewer PRs, but I had, you know, really robust code, I'd be possibly happy with that. So we got to figure out what the basket of goods is, but then we could start to track it relative to token usage. And that would be the consumer price index. So for any time period, we could just say I've got my tokens spent and I've got my goods produced. Tokens divided by goods produced is a pretty rough measure of the purchasing power of the tokens in those time periods. Then you would do the standard consumer price index thing of making what they call a hedonic adjustment.

30:10So you could just say maybe quality is improving over time. So you'd pick some measure for that and make an adjustment to the line. And when we did that study, we did code survival. So the number of lines of code that survives more than four days in the repository. We made an adjustment upward because that rate is going up. That's surprisingly short. Four days? Four-day survival? So again, all this is around measurement, and I'm happy to just be starting this discourse because we can see it's important to the economics of AI, and it seems like the work isn't being done at a high enough rate for us to get a clear picture.

30:44So we could make it longer, and maybe the adjustment would be different. I think currently for the data we have, which is this SWE chat benchmark, which was released by researchers at Stanford, it's about 6 ,000 real coding sessions, all the metadata, everything you'd want. What we see with Opus 4.6 usage in the time period we have, which is February to mid-April of this year, a decline in the purchasing power of tokens. That CPI is going down. And again, I just want to open the question, is it because we have the wrong basket of goods or is it because we're actually getting less value from these tokens?

31:21The one thing I can say that's kind of definitely a causal factor here is that in February of this year, most of the tokens went to producing code, which relates to the outcomes we just talked about. by mid-April, it was quite split between code generation thinking and also explanation to the user. And so that split now is going to have an effect on the things we're measuring, and that might be cause for reflection. There's value in those explanations that's not reflected in PRs, but might be reflected in something like knowledge discovery. And I see that coming up within the same time frame, it's become very common to now talk about the token efficiency of a new model that's been released with the implication being tokens of, you know, internal use tokens, thinking tokens versus, you know, per token of output, I guess is maybe a way to think about it.

32:18Another fascinating dimension. And this actually relates all the way back to the theme of efficiency for these architectures. So here's a claim I'll make for you. based on my read of the literature on inference time scaling, what's sometimes called test time scaling, which is just having the models generate lots of tokens at the moment that you ask them a question. So those scaling trends, everything we're seeing now is completely in line with those predictions, which is you get pretty good gains for a while with the more tokens you spend on a log scale. So this is jumping up quite a lot, but you do see it reflected in performance improvements, but it flattens out over time.

32:56And it's not like this curve skyrockets. It's sobering. You got to spend a lot of tokens for small gains in performance. We all knew this. We all knew this and we're just seeing it now play out. And when people talk about token efficiency and worry about this, I think what they're seeing is just the real lesson of what we already projected from inference time scaling. And this is independent of the approach to inference time scaling you're taking, whether it's multiple parallel inferences or some kind of Oracle or any number of other schemes? It's just fundamental to inference time scaling? That's a great question, right?

33:38I think we know that it's independent of some of those things, like the parallel work versus having it do lots of long chains. But some of the other factors you mentioned, I think we just don't know. And that's why I said it relates back to the question of efficiency for these architectures, if we made a fundamental change to how the models work, maybe these trade-offs would be very different. I mean, after all, so all of this stuff is a kind of patch job on the fact that there's no recursion in the depth. It's a fixed depth. And so the only recursion we can get, the unopen-ended notion of computation is by generation.

34:12But if we had models that could be recursive, maybe fewer tokens for larger gains, I think we don't know. So, yeah, I mean, in the end, we're going to spend the cost on compute or tokens. So this might not affect our bills in the end, but it is a fascinating question. What are the true scaling laws and what's possible in this space? And you're right to push back. We talk about these things like they were like platonic ideals of laws. Scaling law invokes that, right? But even for the scaling laws for pre-training, you know, there's lots to discover there. And many of the stories of progress are actually like transcending the scaling law.

34:49And we see like better improvements than those laws predicted because everyone worked so hard behind the scenes to do very innovative things, which maybe relates to our bitter lesson discussion. Any particular example come to mind of that? Data usage and the nature of the data really matters and overtraining the models really matters, which is kind of pushing up against the standard scaling law presentation. And now I'm just going to speculate. I should check on this, but things like mixture of experts might have really flipped the script on what it means to count parameters and in turn how these laws relate.

35:18And then I think maybe even also stuff like the context window and so forth. This is another thing to check, but I just speculate that we've seen larger gains from pre-training than you would have predicted by those early scaling laws papers, suggesting that there is some innovative thing that was happening on top of pure scaling? Thinking about the concept of a market basket, one kind of pushback that comes up for me is in the real economy, eggs is different than milk, is different than beef, et cetera, et cetera. And they're all influenced by different factors. uh you know production for example whereas what you've done with the this kind of cpi basket with tokens is kind of like more like analogies like here's the typical bundle of work and you know what it requires from a consumptive perspective but the tokens aren't fundamentally different like they're the same tokens it's just like how much it takes to do this versus how much it takes to do that versus how much it takes to do that.

36:30Tell me what I'm missing there and what does kind of characterizing these products give you in your analysis? Yeah, fascinating to think about. One thing I could insert there is the tokens are different at the level of being used for code generation or skill file writing or explanation or thinking, right? Those are different kinds of tokens that probably do feel tangibly different to us. So is that an element in your thinking? I think I was thinking from our conversation that you had 10 different, almost like tasks, like 10 different types of tasks from the domain of code generation, which, you know, if they were all kind of largely code generation, you know, that is the part that had some dissonance for me.

37:17But if you're talking about, like if your basket is like creative writing versus, you know, a few code generation things that are kind of different versus, you know, summarization versus editorial commenting feedback, those, you know, may be more fundamental. Yes. So I think this is very significant. when we have done some research on this as well at the level of what kinds of session types exist and in turn, what kinds of users are there. So you might notice of your own behavior, I guess this is reflected in your comment, that sometimes you want a quick check-in on a question. Sometimes you want a quick PR to get fired off.

37:57Sometimes you want to be in a mode of deep collaboration. Sometimes you're partnering with the AI. Sometimes you're delegating the work and so forth and so on. And the outcome measures that we choose should be sensitive to this. We shouldn't penalize the agent if your chat interaction with it about some scientific question didn't lead to a PR. It was never on the table in the first place. Whereas if you're trying to get some work delegated that's actually a coding task and all it does is chat with you, that would feel quite unproductive. So we need to bring that in and that would be a higher level discovery process of what people are trying to do and so forth.

38:33In thinking about the notion of value, is this something that you're anticipating? Like, it strikes me that that's an entire, you know, research thread that, you know, one could go into. I don't know if that's a linguistics or a linguist or an economist or a computer scientist, you know, probably interdisciplinary, like most interesting questions. but is that something that you're working on or was it something that you put out there for someone to take up and run with? I am not sure. I can tell you the lineage of this idea is that we founded this startup Big Spin because we would like to see more people benefit from AI.

39:19Whether you love it or hate it, it's here and I would like the benefits to be more evenly distributed And I can tell that that will mean bringing on board many more people than currently benefit from AI. Right now, I would say that it's mostly experts deriving real value. And a lot of the world is currently even trying to figure out what this is all about as a tool or an entity in their lives. So we would like to have more access and more productivity. And that implies making the user experiences much better. Figuring out what interactional patterns lead to success for people, meeting them where they are in a kind of adaptive way, the whole list of things that you might worry about if you were a product manager who had some deployed AI product.

40:04And I think by that route and from that perspective, we just ended up worrying about our own token usage increasing and wondering whether there's real value there. And it just happened to collide, actually, just like three weeks ago, with this emerging narrative on the back, I think, of all these rumors about IPOs, about what the return on investment was. And then all these CEOs came out and said, oh, our spend was enormous and we want to scale back and we're walking back our claims from a few months ago. And that is just a fascinating thing to witness in the narrative here. I'm wondering, does this research also attempt to project forward?

40:41In theory, you could create a model for Anthropics cost and spend based on publicly available data and some presumptions and give us a sense for how close we are to paying full freight for our tokens versus if we're only paying 10 % for our tokens, you could then project what that cost might look like. over time as we're paying more and more of the full cost? Yeah, I don't have, again, fascinating questions. I don't have resolving answers. I am glad I am not tasked in some organization with projecting spend on all of this stuff because I think it would be basically impossible. For the time period that I was describing for our little CPI experiment, Anthropic changed the default reasoning on the model at least two times.

41:34So we see like they launched it with default reasoning high. we have a mysterious sudden rise in the token usage which we cannot explain and there's a new baseline they lowered it to medium as the default they patched a bunch of bugs that were related to context management and then turned it back up to high and all of these things have an effect on the total token output as you can imagine they also changed the default context window which meant people could swallow up much more stuff at any given moment so imagine trying to predict what token spend is going to be like when you have all these exogenous events in addition to changes that we don't even know about and questions about where the value actually lies.

42:15Very difficult. And then, you know, the true cost of a token, the estimates vary wildly for every dollar we spend. It could be as low as two and as high as 20. And I think this is just because it's hard to factor in things like R &D and future build out and depreciation and all of that stuff. I think at the current moment, we just don't know, but there couldn't be a more significant question for the global economy, basically, than where the value is and who's going to pay and how much. Yeah. In your article, you coined the term tokenflation to describe at least the recent behavior of token economics.

42:54I imagine you see that continuing. Seems to be continuing. Yeah. Yeah. That's certainly the picture that we get from the CPI, a picture of tokenflation. Yes, your token is not buying you what it once did according to everything we can think to measure here. And even adjusting for models getting better, right? That's critical there. Because if it was just a story of models thinking more and being more robust, and we were all getting exponentially better outcomes from this, then the spend would look completely rational. But that's not the picture that we see. And so we have to do some hard thinking about what's going to happen and how to improve the situation.

43:28Let's dig into that a little bit more. How would you articulate what you're seeing, the models are getting quote unquote better. There's a set of open questions about are the reported ways that models are better actually reflective of some intrinsic betterness? And that question is often about benchmarking and learning the benchmarks, overfitting, that kind of thing. And then there's the kind of question of chattiness and the volume of thought that it requires a given model generation to produce an answer. What are other factors that you see? Yeah, we could pick that apart as well. And this relates to a line I've had consistently, which is that we should think in terms of systems, not in terms of models.

44:26So in the data that we've got, Opus and Sonnet 4.5 versus 4.6, those two generation changes, those are real model changes, I assume. I think they did something very substantive at the level of the weights. And everybody immediately saw that that led to like a 5x increase in token usage. And this was related to the introduction of adaptive thinking. Now, fix that. That's the level shift that we already took, and maybe we're seeing improvements there that are worthwhile. It gets hard to say, but let's assume there was a level up in improvement. Then for the period that we did our CPI experiment for, that's a fixed model, Opus 4.6.

45:07So all the code improvements that we saw in the data relate to the product. This has to relate to things like them turning the knobs on the adaptive thinking, changing things about the system prompt, changing things at the level of the product. and that's where the improvements were. And so that shows you that even for a fixed model, we can get very different outcomes for these things because they really are sophisticated engineered systems at this point. Yeah. And so was the product in this case specifically CloudCode or? Oh yeah. And so we don't, there's tons of stuff there. Yes, I believe we know that these are all CloudCode sessions that we kept in our data.

45:43SweetChat is broader than that and involves a couple of other coding agents. But I think I can say that all our data our Claude Code sessions using Opus 4.6. Have you seen any evidence that changes via API usage experience, similarly dramatic variation in performance? Oh, fascinating. To kind of control for a lot of that product level stuff, all the prompts that are hidden from us, all of those affordances. I don't know, but that's a nice thing to think about because it gives us more things that we can control for and more things that are knowable. So kind of in parallel to the model evolution, there's also evolution of the user.

46:28You've alluded to this a little bit about kind of your concept is that most AI users now are experts. Talk a little bit about the role of expertise. I think this is also kind of echoing back to our conversation about DSPy and like prompt optimization. You know, you've done some research into how folks are using these models and the role of, you know, AI fluency. Tell us about that research. Oh, yeah. First, I should say, so the distribution of users across expertise levels. So I guess the nuanced picture I'd offer is that the people deriving a lot of value from AI in the current moment tend to be experts.

47:09It must be the case that most users of AI are beginners, just because the numbers are so large and expertise can't be that widely distributed yet. And that's a very interesting thing because I think probably most things are getting designed for those experts implicitly or explicitly. But for the whole economic picture to work out, many more people need to derive value from this via one avenue or another. And so that does shine a light on this expertise thing as a real factor. And the headline result there actually builds on something that Anthropic did. They have this AI fluency index and their core observation in that work is that experts display an augmentative style.

47:50They iterate with the AI, they push back, they complain, they change their requirements. It's really a collaborative mode, whereas novices, low-fluency users, delegate. So they trust in the AI, they let it do its thing, they accept the responses uncritically. and our contribution is to show that this is a causal factor in success with these products right now. Experts can do harder things more reliably as a result of all that friction they introduce, all that pushback. Whereas novice users, they accept, but they end up accepting the wrong thing and they're not able to level up from the basic tasks that they think to start with.

48:34That's obviously significant. And it feels so tantalizing because pushing back is a natural human behavior. I feel like we could encourage everyone in the world to do this. We probably need to get them out of the mode of thinking, it's a super intelligence. You should just trust it. That has been the narrative for a while. What we're seeing in the current moment and possibly for the foreseeable future is that you got to complain, collaborate, introduce yourself, push back, all that stuff that I think we do. We take that for granted, right? Yeah. Yeah. And so from a methodology perspective, how did you approach exploring this?

49:10We built on the work that Anthropic did, which they set up a nice framework with some independent research who were doing this kind of usability stuff. And we just have an annotation protocol. We can talk in detail if you want about this, but at BigSpin, we have lots of these best practices around having language models essentially collaborate on annotation projects to kind of triangulate on the truth and factor out their individual biases. So we do that stuff and we apply all these fluency markers. And then separately, we do a thing of estimating task complexity and looking for signs of visible and invisible failures.

49:47And so it's the connection between the fluency markers and the task complexity success metrics that was our contribution there. And that's where you can see high fluency users are the ones doing harder tasks. Paradoxically, there's more signs of failure for them because they complain, they push back, they're trying harder things. but as part of all that friction they're successful with harder things as well. And if you were to try to apply this insight from the perspective of someone in an organization that's trying to help or guide their organization to be more successful with AI what do you think are the key lessons of this fluency work?

50:28If it's an org that's just starting out and wants people to figure out how this could be part of the organization's mission it would just be that pushback message. And you could do an experiment where you interact with it about something where you're a world expert. We're all an expert in something. Engage in a discourse with one of the best models about something you're an expert in and see how often you feel you have to push back. And this could be a kind of lesson. You say, aha, for other spheres where I don't know the answer, it might be just as errorful. That could be a good visceral thing.

50:56If the org is very far along, I think the main thing to do right now is to have a team of these LLMs interacting to improve things. For example, at Big Spin, I didn't set this up. Our founding engineer is very future forward on agents, and he's incredible at this. And when we do PRs now, the first round of review is the agents all interacting, collaborating, disagreeing. They do the first round of comments. They do the first round of code updates. Only after they've resolved things do we look at a PR. So the final human stage should be very high value. And the agents did all that work. But when you have one agent do it, they often just reinforce themselves and you don't get good outcomes.

51:37It's that team of rivals thing that is transformative. You know, I think it's interesting because, you know, on the one hand, like, of course, that makes sense. But on the other hand, there's something, you know, it also implies that you shouldn't be using these things in areas where you don't have enough expertise to evaluate the answer. Yet, that's where you most need the assistance, the support. um so and again and this is a little bit worrisome about the overall narrative around ai the place where we can get around this is with software development because let's say that i'm trying to accomplish something in a language that i don't know how to code in i can have the agent do work for me because probably in the end i can run the program and look at the results and that's what mattered to me is that i run the results and i see and if i don't see what i wanted I can complain and we can iterate.

52:36That verification step that doesn't imply I have comprehensive knowledge, it just implies that I know what I want to see in the end, is so critical. And I think this is a causal factor in models being so good at coding because it's like the ultimate verifiable domain for them. But as soon as we leave that and go even into something like the legal realm where the requirements are strict, but they're not codified in code and they have ambiguity about them, this whole picture falls apart. And you then are back at what you just said, which is this awful kind of paradox is like, yeah, use AI, but in the end, unless you're expert enough to evaluate every single one of its responses, you might be in real trouble.

53:16I don't know how to get out of this because the verification step is like we go to trial, but this is very consequential. That's expensive. Yeah, that's funny. I mean, it does make me think a little bit about, you know, some of the types of errors that we're trying to avoid are factuality. And, you know, there is a temptation to say, well, let's just throw more tokens at it. Like I'll have a critic model that evaluates everything that the, you know, know, is generated by the primary model. But then you go back to my observation that these models tend to correlate in their responses as well. Yeah, it's super interesting.

54:01That's a good point. Yeah, for my picture, we want real diversity of perspectives. This is just like, you know, red teaming for humans. This is most successful when you have a really diverse team of people who think creatively and differently. And if every one of the members of that team is thinking in a homogeneous way, they miss all of the crucial things. Same exact issue. If all the code review agents are biased in the same way, they will miss exactly the same class of bugs and then we're all sunk. Yeah, I don't know how you'd encourage this diversity in the ecosystem. We're probably, as you say, converging towards some kind of one model.

54:33But I think for my picture, we need diversity. Yeah, we got to keep those open weights models going or something because they're the weird players in the space. For sure, for sure. So we've talked about efficiency, interpretability, tokenomics, fluency. You're involved in a lot of different research directions. Excellent. Excellent. What's next for you? Where do you see either where do you see this all going kind of externally, but also like where is your research going? Yeah, this is great. And as I said before, we're trying to think in weird and creative ways about what the future could hold.

55:13And I encourage my students to do this and they're smart. So they say, all right, Chris, I'll think along those lines, but what's your answer to this question? So I do have an answer. And it's really shooting for the moon here, which would be, what about the architectural innovation that would upend the whole story around the stack transformer and the way we need to do data center build out to even get incremental gains in performance that could be upended. And it would come from some very innovative thing around maybe recursive use of the building blocks that we've got. So architectures, we should think.

55:45And when people say, oh, no, we don't need more architectures. The transformer is good enough. That's where we should push back as academics doing something more clever and more scrappy that could change the world. And the other one is thinking in the interp space much more about data. And that's just because I want to tell the true story of how we go from data to model capabilities. but it also checks a box for me on connecting interpretability to safety. It has been hard for me to connect those two things. We have found some ways to do it, but it's not a slam dunk as a narrative, even though it's the dominant narrative.

56:20But I will say that when we get into things like data poisoning from innocuous examples, this is probably a growing societal concern. There is evidence that with very few examples planted in a pre-training data set, you can have a significant influence on the outlook and preferences and quirks of the final model. So can we detect those examples? What's the nature of those attacks? How well hidden could they be? What's the smallest number of examples? And why does it happen? These are all going to be very pressing questions. And so, again, it's just a data-oriented question that's very alive for me in the current moment.

56:57On the architecture front, is there research that you're seeing or doing that is as yet under the radar that you think is promising and or underappreciated? I think you had my student Julie Kalini on, and she is an advocate for byte level models, essentially tokenizer free models. I think that's a big part of the future. It's a critical thing if you want to have truly multilingual models that are also equitable in terms of how many tokens they charge us for, getting back to that earlier theme. But also Julie's perspective is that this is speculative, but I think there's something to this, that it's a kind of inference time scaling because you do more compute at test time because you have more tokens.

57:44and therefore more opportunities to build on interesting things. So that could be a big part of the future. And the other one would be recursive architectures, as I said. But if you want to go all the way out, you could think, why do we always assume we're going to do gradient-based learning? There are lots of alternatives to that, and nobody is exploring them because everyone takes it as a truism. We're all in our very narrow row here without even really realizing it. Who knows what's outside in this garden? And it's very risky as a research bet because only one in a thousand of these ideas will pay off.

58:20But what's the point of being an academic researcher if you're not going to take that kind of risk? That's what we're positioned to do. Well, Chris, thanks so much for jumping on and sharing a bit about what you're working on. It's very cool stuff. Thank you. What a wonderful conversation. It gave me lots of new things to think about. Awesome. Awesome. Thanks so much.

58:42Thank you.