Inside AI Tokenomics: How to Profitably Turn Tokens Into Business Value | NVIDIA AI Podcast Ep. 299

0:00Shruti Koparkar:Not all tokens are created equal, and there is a way to look at token value. There are two key factors that impact token value. One is the intelligence embedded in the token, or how much intelligence does the token carry. And the other is how fast does it arrive.

0:22Welcome to the NVIDIA AI podcast. I'm Noah Kravitz. I'm here with Shruti Koparkar. Shruti is a member of the Accelerated Computing team here at NVIDIA, and she focuses on inference. And we're here to talk about tokenomics. As data centers become AI factories and produce intelligence for the new industrial revolution, this word tokenomics has been fluttered about. It's a useful term, but maybe we can break it down with your help, Shruti, so that it's really something that business leaders can understand and take into practice.

0:54Shruti Koparkar:Yes, absolutely. Well, first of all, thanks a lot for having me, Noah. I am very excited to dig into the economics of AI or tokenomics. And as you said, it is a term that gets used quite a bit. And I welcome the opportunity to help define it, so to speak. So the way to think about tokenomics is it's about how tokens are valued, supplied, consumed, and monetized. And what that essentially maps to is token utility, which is all about token value, token supply. And this is where your AI infrastructure decisions are, right? Thinking about what infrastructure to invest in that will maximize your token output while minimizing cost.

1:36Shruti Koparkar:Then there's token demand. This is where customers and organizations think through what is their number of users, how many use cases, what types of use cases. So really sort of mapping out the volume and velocity of the tokens that they need. And then finally, there's token monetization, which is taking the tokens and turning it into business value. So those are sort of the four pillars for tokenomics. And it's super important to understand all four of those and how they relate to each other to be able to deploy AI successfully. So let's start at the top then with utility or value. How do you define the value of a token?

2:15Are all the tokens worth the same? Do they have different values? Is there a better way to look at it? How do you approach that?

2:21Shruti Koparkar:That's a really great question. And you're right, actually, that not all tokens are created equal. And there is a way to look at token value. There are two key factors that impact token value. One is the intelligence embedded in the token, or how much intelligence does the token carry. And the other is how fast does it arrive, which is essentially the interactivity. So to unpack that a little bit, the intelligence of the token is dependent on the model that produced the token. So more complex, more intelligent models will produce tokens that in general have much more intelligence. A higher value, okay.

3:02Shruti Koparkar:Yeah. And then it also depends on the context that the models looked at. And generally speaking, the longer the context that you sort of allow the model to look at, the better the accuracy, the better the intelligence of the tokens. Now, I say generally because there are cases where if the context increase too much, then the model quality, the output quality can degrade. But I don't want to rabbit hole into that. Generally, more context does kind of equate to better intelligence. Right. So that's one aspect. And then, as I mentioned, how fast the token arrives is the token interactivity, which is essentially tokens per second per user.

3:46Shruti Koparkar:So it's the rate of token generation. And so if you look at token value as a spectrum, on the one hand, you have these basic models with shorter context, generating tokens at not that fast a speed. And then on the other extreme is these more complex, more intelligent models with much larger context and generating tokens that are really fast. and across that entire spectrum is your different use cases and how you map those use cases to the token value. So, of course, there is an absolute way in which to think about the token value, but there is also a relative way with respect to the use cases in how to think about it.

4:31Shruti Koparkar:And we can unpack that a little bit if you want. So is it fair to say that the value of the token is tied at all to the task that it's hand as well? Yes. And that's exactly sort of what I was trying to get at when I said that you have to think about mapping the right use case to the token value. So as an example, we said, like I said earlier, that tokens generated by more complex intelligent models are more valuable. But that's in an absolute sense. relatively speaking your use case may not require that more complex more intelligent model and then that additional value is completely useless to you right one example of this is domain specific applications where in a very narrow context a post trained which is fine-tuned small language model so a much smaller model can give you just the value you need in fact in some cases even better accuracy for that given task.

5:30Shruti Koparkar:So you don't need always the big large models. And so relatively speaking, you need to map where on the spectrum of token value does your use case sit. Same thing is true for the interactivity piece, which is agentic applications absolutely need the highly interactive tokens. But you may have applications like chat interfaces or enterprise search, which don't need that level of interactivity. And so that is very critical when you are thinking through your sort of AI deployment decisions of where to map your use case to what token value. So when a business leader is thinking about demand and thinking about mapping out tokens to use cases and the different use cases have, you know, different values associated with them, What's a good approach for someone who's looking at what their org is doing, what their different team members need?

6:26How do they start to get a handle on, well, how many tokens are we going to need to produce and how many of each kind?

6:32Shruti Koparkar:Yeah. So use cases are extremely important when thinking through token demand. And there are three layers in which you can think about this with improving levels of forecasting accuracy, if you will. So the basic sort of, you know, back of napkin math is look at how many users you have, how many requests or sessions the user is going to initiate in a given day or month. And then how many tokens you need per request or per session. And those three numbers put together will give you your base demand for a single day or a month or, you know, whatever is your time period of analysis. Now, that is the base and very, very sort of simplistic look at it.

7:20Shruti Koparkar:There are multipliers that you do need to account for that will dramatically change your understanding of your token requirement. And a couple of those are, number one, are you using reasoning models? As we know, reasoning models use thinking tokens, which never get seen by the end user. And oftentimes when AI is deployed, you can actually set thresholds on how many thinking tokens are allowed per interaction. And so when you are estimating demand, you do need to think through, are we using reasoning models? What are our thresholds? What do we expect the peak and average to be on that use? So that's one.

7:59Shruti Koparkar:Second is Egentic. Agentic is a huge multiplier because any use case, if you are deploying it in this sort of agentic workflow context, then there are multiple sort of turns and loops that might happen that can increase your token demand significantly. And then finally, the last factor is something called cache hit rate or the KV cache hit rate. And for those listeners for whom this term might be new, KV cache is sort of like the short term memory of a model. And so anytime an input request comes into a model, it needs to process it. But if it's already seen that input request before, then many times it actually gets stored in the cache.

8:42Shruti Koparkar:And then when it comes in again, it doesn't need to recompute it. It can just use those cached values. So those are some key factors to kind of look at to get to a higher degree of accuracy when thinking about token demand. And then the final one is demand variability. which is how is your demand changing in a day? Like sometimes you may have products that get used quite a lot in the morning hours, but not so much in the evening or vice versa. Same thing with seasonal variability. For example, retail providers or e-commerce will see a surge during the holidays when they're trying to push a lot of products out, right?

9:22Shruti Koparkar:So you do need to think through those. And then of course there is the user growth. So you started with a base number of users, but you as a business are trying to constantly drive up user growth. So you need to factor in how much you expect that to grow as you think through your token demand. So demand, of course, leads us to supply. How do you start thinking about supply? And you've mapped out, you know, sort of your baseline and the conditions that you just outlined, Shruti. how do you go about then translating that into creating the supply necessary to get all these tasks done so when it comes to token supply that's where a lot of the ai infrastructure decisions lie and when you're making that decision what you want is maximum tokens availability token output while minimizing your token cost now when you think about cost or total cost of ownership oftentimes organizations and decision makers can gravitate towards the easily available metrics and what I like to call input metrics, such as the cost per GPU hour or the flops per dollar, which is essentially how many floating point operations are you getting per dollar.

10:38Shruti Koparkar:And these are input metrics because they don't tell you anything about the actual delivered token output, which is a function of much more than just flops or just the memory you have. It is a function of extreme co-design. And so the metric that represents both your input, but also the output is cost per token. It's a very simple metric that tells you what is the cost that you're paying of cost of generating one token. And it's essentially the cost of GPU divided by how many tokens does the GPU produce. So in a way, it incorporates both the input and the output and gives you a sense of your true ROI from the AI infrastructure.

11:31It's interesting to hear you explain it because it sounds so simple and I can understand coming from the other point of view, right? of what we're outlaying for the GPUs and the server racks and all the interconnectivity. And so we count those costs. But looking at it from the other end, as you said, the output cost just makes so much sense because that's what you're trying to get at the end is the intelligence, the token. And so putting the price on that seems like a really kind of clear way to think about it. Does the cost per token metric vary at all? Or do you have to think about it differently depending on the use cases, as you were talking about before?

12:08Shruti Koparkar:Uh, cost per token is sort of the base metric. Now, of course, it will vary depending on all the other things, like the model, the context, the intelligence, basically, and then the interactivity. So any tokens that are generated by a more complex model or are more interactive are going to be cost-reduced. Of course, yeah. They are just, that's just physics, right? So yes, it definitely does depend on the models, the context, as well as the interactivity. But, you know, you said it really well earlier that ultimately, if the business runs on the output, which is the tokens, right, it is kind of a fundamental mismatch.

12:50Shruti Koparkar:If you are evaluating infrastructure based on the inputs, but your business runs on the output. And that's why cost per token starts to get at sort of the real ROI, because it measures both in many ways. Yeah. So, Shruti, as we think through input metrics and cost per token, is there an example that comes to mind that can really kind of bring this idea to life? Yeah, absolutely. In fact, if you look at NVIDIA Blackwell compared to NVIDIA Hopper, and if you look at just merely the input metrics, which is the hourly GPU cost, that's 2x. And so that's Blackwell being maybe 2x more expensive than Hopper.

13:32Shruti Koparkar:If you just look at flops per dollar, that's also 2x. So Blackwell does deliver 2x more flops per dollar. And that sounds like a huge advantage, which it is, but it also doesn't even scratch the surface of the true sort of benefit and value of Blackwell. And that's because Blackwell, when it comes to delivered output, delivers 50x more tokens per watt compared to Hopper. 50x. 50x. Fantastic. So with the same infrastructure footprint, the Blackwell NVL72 system delivers 50x more tokens than Hopper. And that translates to a 35x lower token cost. Amazing. Yeah. And so that really, I think, you know, brings the point home on why not just look at the input metrics, but look at a metric like cost per token, which represents both what you're paying, but also what you're truly getting.

14:37So I'm glad you mentioned I was going to ask you to go back. You'd mentioned extreme co-design. We've talked about it before. Obviously, anyone familiar with the space has heard the term. but maybe you can dig in a little bit to what it means particularly in this context.

14:52Shruti Koparkar:Yes. I actually welcome the opportunity to talk about extreme co-design because we get asked this question quite a lot. And so, you know, often we get asked why extreme co-design? What does co-design even mean? Like, is it just integration? And, you know, people may think that this is just splitting hair or just semantics, but I do think that the distinction is important Because when you think about integration, you think about different parts, different sort of, you know, independent units that are then integrated post facto. Whereas co-design is about designing from the ground up simultaneously multiple parts of the same system, knowing that they are all optimized towards the same outcome, that of lowest token cost.

15:41Shruti Koparkar:And so that's why the word co-design is extremely important. And the reason it is called, or rather we call it, extreme co-design is what NVIDIA does, is because of the depth and breadth into which it extends. So it's co-design across just compute? No. It's compute, memory, storage, networking, right? Everything. I mean, the Verubene platform has seven chips. But it goes even beyond that. there's all the software that sits on top. So everything from the CUDA kernels to the runtimes to the serving software, as well as all the way out to the ecosystem, everyone from our, you know, silicon partners, our OEMs, our cloud providers that we work with, the various OSS frameworks that we work with, the core design extends beyond just sort of, you know, what's in a system, what's in an AI factory all the way out to ecosystem.

16:40Shruti Koparkar:And that's one of the reasons why it's extreme. And so anyway, but you asked a question of a more specific question about what are some of the extreme co-designs that help with the cost per token. And I think one important one, which I think you've actually discussed in the previous podcast, is the mixture of experts models, how the Blackwell and VL72 is such a great fit for them because it kind of helps with the inter-GPU communication. And then all the software in terms of dynamos, disaggregated serving, coupled with any of the runtimes that we support, whether it is TensorRT, VLLM, SGLang, doing a technique called Wide Expert Parallel that greatly optimizes the inference performance and then thereby reduces the cost per token for those mixture of experts models.

17:31Shruti Koparkar:So that's one great example. The other really good example is actually the Vera Rubin platform itself, which is built for the age of agentic AI. And to really understand why that extreme core design is required, maybe we can look at what an agentic workload is like. So thinking about an agentic workload, let's draw the parallel to the conversational workload. When in a conversational setting, the user prompts something, say you prompt something, and then the LLM says something. And then you say something else, and then it says it back. So you are taking, as a human, turns with the LLM, with the AI.

18:15Shruti Koparkar:In a GenTech, it's actually AI taking turns with AI as well as with software. because a main agent can sort of, based on the user input, decide to do some reasoning, then decide, oh, I need to do a tool call, so call some software. Then it might decide, oh, I need a sub-agent or a specialized agent to go do some work. So it's going to take a turn with the specialized agent till the specialized agent, you know, does its computation, comes back with a result. And this just keeps going. Right. And we love it, Gentic, for that. That's right. Right. And it's multi-turn in a way that has no user involvement other than the prompt that the user gave in terms of like, maybe say, book a ticket to Miami.

19:04Shruti Koparkar:And then it goes through all of this, several turns to then finally produce an outcome. And the number of turns involved in Agentec is significantly higher than conversational. so the number of LLM calls like the number of times large language models is called is also higher and in general the token demand for that reason is also higher and that's why extreme co-design is so critical because you you are using up so many tokens so you have to lower the cost per token latency is really important because on every turn even a couple milliseconds more add up to you know several potential seconds of delay or the end result.

19:48Shruti Koparkar:And then finally, coming back to the Vera Rubin platform, now that we've sort of described the agentic workload, we can clearly see why the core design is required to accelerate sort of the LLM itself or the reasoning and the AI itself. You need the Rubin GPU. You need the Grok 3 LPX solution and deliver that ultra low latency. You need Vera CPU because it's going to do all this tool calling or sandboxing for code generation and code testing. You need the CMX platform that we've talked about, which is the Bluefield DPUs together with Spectrum X, which allow for the KB cache or the short term memory, as we discussed, to be offloaded when needed so that it can be retrieved when required for a match with an incoming request.

20:43Shruti Koparkar:And so that's sort of another example of co-design where being able to develop all of these from ground up helps a lot. Right, right. So we talked about extreme co-design and you mentioned all the different pieces that go into it, building, designing and building from the ground up. Software is a part of that. But maybe you could double click, Shruti, into how software plays a role and how important software really is. Yes, absolutely. So software actually is the difference between what you get in the real world, the delivered token output and the actual token cost versus what you see on a spec sheet.

21:25Shruti Koparkar:Software makes all the difference. All the things on the spec sheet, the system design cannot be fully realized unless you have software that makes use of it and delivers really good output. And the other important thing about software is that it cannot be piecemeal optimization. You need to have a robust software stack that can turn on, enable every single optimization. So that can do, say, NVFP for quantization. It can also do MTP or speculative decoding. It can also do disaggregator serving. It can do the wide export paddle, the KV cache offloading, the KV aware routing and on and on and on.

22:11Shruti Koparkar:Right. To be able to stack all of those optimizations together is really important because that is what gets you the 50x. Right. The 50x more throughput that we see with Blackwell and the 35x lower token cost. And so that's software is a huge, huge piece of that story for sure. The other thing about software is that it never stops. Open source software, especially, it never stops. And it's not just the NVIDIA team that is building the software. It's the entire ecosystem, right? It's all the OSS frameworks, all of our partners, customers, the developer community, and every small optimization that they do, that's a drop that just keeps adding and adding to this massive ocean of advantage that is the NVIDIA ecosystem.

23:00Shruti Koparkar:And so just as an example, on both VLLM and SGLang, which are these inference runtimes, we've seen 8x more performance in just about six months. And that's huge. Because from the same infrastructure footprint, you're getting so much more token output, and that's driving down your token cost as well. So absolutely, software is a huge, huge piece of the puzzle. So we're through three of the four pillars. The fourth one, perhaps the big one, monetization. How do you talk about monetization? How does a business leader think about, okay, so I understand the importance of extreme co-design. I understand the different value of tokens in different situations and different tasks.

23:46And Gentic is wonderful, requires more tokens, all the things you've elucidated here. How does a business leader think about monetization of the tokens? Right.

23:57Shruti Koparkar:Right. So when it comes to monetizing tokens, there are various different ways in which you can go to market. But one of the best proxies is to just think through it as you're generating tokens and then you're selling the tokens. And so when you think about selling the tokens, how much do you sell them for? And it's sort of a classic exercise in figuring out your pricing, which is, A, you need to think about what is the cost to produce the token, which is this lowest token cost that NVIDIA is helping with, right? But you do need to understand what is your token utility. And given that token utility and token value, what is your cost to produce the token?

24:38Shruti Koparkar:And you obviously want to charge more than that, right? So there's that. Okay, so that's cost-based pricing. And then you also obviously have to think through value-based pricing, which is essentially how much is the willingness to pay? what is this sort of token utility? How valuable is it to the people who are going to pay for it? And so you do need to take that into account. And then finally, before you think through the pricing, you also need to think through what is the demand distribution? Because ultimately there are revenue goals and kind of profit margin goals that you are working towards.

25:16Shruti Koparkar:And so to land at a place that you like, you do need to think through where will your sort of bulk demand be and how will the demand taper off when it is, say, for tokens that are not as much utility. There may not be many takers, but in the same way, tokens that are highly valuable, there will be fewer people who are willing to pay the premium for that. You do need to account for that sort of demand distribution. And then with those three things, you can figure out what the pricing for each token can be and then deploy AI successfully. So the key thing here, though, is that, you know, pricing the tokens, again, is obviously just one proxy.

26:04There will be customers who are building value-added services on top of those tokens.

26:11Shruti Koparkar:Sure. So like customers building AI-native products or something like that. And then in that case, the process is similar, but you do need to think through then what is the additional value you are adding on top of just generating those tokens as well. So going back, Sharita, something I was thinking about when you were explaining extreme co-design. When you get to a point, kind of a sweet spot where you've, you know, your infrastructure is humming and the cost per token is low, does that mean that ultimately you won't need as many GPUs to produce the number of tokens that you really need? Or what happens in that kind of a scenario?

26:51Right.