Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah-Hill Smith

29:57And this was one of her top requests, which is some kind of hallucination slash lack of confidence calibration thing. And so, hey, this is one of them. And I mean, like anything that we do, it's not a perfect metric or the whole story of everything that you think about as hallucination. But yeah, it's pretty useful and has some interesting results. One of the things that we saw in the hallucination rate is that anthropocene clawed models at the very left-hand side here with the lowest hallucination rates out of the models that we've evaluated amnesia on. That is an interesting fact. I think it probably correlates with a lot of the previously not really measured vibe stuff that people like about some of the cloud models.

30:40Is the dataset public? Or what's, is it, is there a held out set? There's a held out set for this one. So we have published a public test set, but we've only published 10 % of it. The reason is that for this one here specifically, it would be very, very easy to like have data contamination because it is just factual knowledge questions. We will update it over time to also prevent that. But we've kept most of it held out so that we can keep it reliable for a long time. It leads us to a bunch of really cool things, including breakdown quite granularly by topic. And so we've got some of that disclosed on the website publicly right now.

31:18And there's lots more coming in terms of our ability to break out very specific topics. Yeah, I would be interested. Let's dwell a little bit on this hallucination one. I noticed that Haiku hallucinates less than Sonnet, hallucinates less than Opus. And would that be the other way around in a normal capability environment? I don't know. What do you make of that? One interesting aspect is that we've found that there's not really a strong correlation between intelligence and hallucination rate. That's to say that the smarter the models are in a generalist sense isn't correlated with their ability to, when they don't know something, say that they don't know.

31:55So it's interesting that Gemini 3 Pro preview was a big leap over here, Gemini 2.5 Flash and 2.5 Pro. But, and if I add Pro quickly here. I bet Pro is really good. Actually, no. So I meant the GPT Pros. Oh, yeah. Because GPT Pros are rumored, we don't know for a fact, that it's like eight runs and then with the LM judge on top. Yeah. So we saw a big jump in, this is accuracy, so this is just percent that they get correct. and Gemini 3 Pro knew a lot more than the other models. And so big jump in accuracy, but relatively no change between the Google Gemini models between releases. And the hallucination rate.

32:38Exactly. And so it's likely due to just kind of different post-training recipe between the clawed models that's driven this. Yeah, you can partially blame us on how we define intelligence, having until now not defined hallucination as negative in the way that we think about intelligence. And so that's what we're changing. I know many smart people who are confidently incorrect. Look, that is very human. Very true. And there's times and a place for that, I think. Our view is that hallucination rate makes sense in this context where it's around knowledge, but in many cases people want the models to hallucinate, to have a go.

33:18Often that's the case in coding or when you're trying to generate newer ideas. One eval that we added to artificial analysis is critical point, and it's really hard physics problems. Is it sort of like a human eval type or something different? Or like a frontier math type? It's not dissimilar to frontier math. So these are kind of research questions that academics in the physics world would be able to answer. But models really struggle to answer. So the top score here is 9%. And when the people that created this, like Minhui and actually Ofia, who was kind of behind Sweebench. What organization is this?

34:00Oh, is this Princeton? Kind of a range of academics from different academic institutions. Really smart people. They talked about how they turn the models up in terms of the temperature. As high a temperature as they can when they're trying to explore kind of new ideas in physics as a thought partner. Just because they want the models to hallucinate. Yeah, sometimes it's a feature. Maybe get to something new. Yeah, exactly. So not right in every situation, but I think it makes sense to test hallucination in scenarios where it makes sense. So the obvious question is, this is one of many, that every lab has a system card that shows some kind of hallucination number and you've chosen to not endorse that and you've made your own.

34:41And I think that's a choice. Totally. In some sense, the rest of artificial analysis is public benchmarks that other people can independently rerun. You provide it as a service. here you have to fight the, well, who are we to do this? And your answer is that we have a lot of customers. But I guess, how do you converge the industry on one number that actually everyone agrees is the rate, right? Because you have your numbers, they have their numbers, never the two shall meet. I mean, I think for hallucinations specifically, there are a bunch of different things that you might care about reasonably and that you'd measure quite differently.

35:16Like we've called this AA amnesian's hallucination rate, not trying to declare like it's the man is last in the hallucination you could uh you could have some interesting naming conventions and all this stuff um the biggest picture answer to that and something that i actually wanted to mention just as george was explaining critical point as well is so as we go forward we are building evals internally we're partnering with academia and partnering with ai companies to build great evals we have pretty strong views on in various ways for different parts of the AI stack where there are things that are not being measured well or things that developers care about that should be measured more and better.

35:52And we intend to be doing that. We're not obsessed necessarily with that. Everything we do, we have to do entirely within our own team. Critical Point is a cool example of where we were a launch partner for it, working with academia. We've got some partnerships coming up with a couple of leading companies. Those ones, obviously, we have to be careful with on some of the independent stuff, but with the right disclosure, like we're completely comfortable with that. A lot of the labs have released great data sets in the past that we've used to great success independently. And so it's between all of those techniques, we're going to be releasing more stuff in the future.

36:25Cool. Let's cover the last couple and then we'll, I want to talk about your trends analysis stuff, you know? Totally. Before that, actually I have one like little factoid on Omniscience. If you go back up to accuracy on Omniscience, an interesting thing about this accuracy metric is that it tracks more closely than anything else that we measure the total parameter count of models. Makes a lot of sense intuitively, right? Because this is a knowledge eval. This is the pure knowledge metric. We're not looking at the index and the hallucination rate stuff that we think is much more about how the models are trained.

36:58This is just what facts did they recall? And yeah, it tracks parameter count extremely closely. Okay. What's the rumored size of GPT-3 Pro? So, and to be clear, not confirmed for any official source, just rumors, but rumors do fly around. I hear all sorts of numbers. I don't know what to trust. So if you draw the line on Amnesty and Zaki receivers total parameters, we've got all the open-weights models. You can squint and see that likely the leading frontier models right now are quite a lot bigger than the one trillion parameters that the open-weights models cap out at and the ones that we're looking at here.

37:35There's an interesting extra data point that Elon Musk revealed recently about XAI that 3 trillion parameters for Grok 3 and 4, 6 trillion for Grok 5 but that's not it yet. Take those together, have a look. You might reasonably form a view that there's a pretty good chance that Gemini 3 Pro is bigger than that that it could be in the 5 to 10 trillion parameter range. To be clear, I have absolutely no idea but just based on this chart, that's where you would land if you have a look at it. And to some extent, I actually kind of discourage people from guessing too much, because what does it really matter?

38:12As long as they can serve it as a sustainable cost, that's about it. Yeah, totally. They've also got different incentives in play compared to open weights models who are thinking to supporting others in self-deployment. For the labs who are doing inference at scale, it's, I think, less about total parameters in many cases when thinking about inference costs and more around number of active parameters. And so there's a bit of an incentive towards larger, sparser models. Agreed. Understood. Yeah. Great. I mean, obviously, if you're a developer or company using these things, exactly as you say, it doesn't matter.

38:45You should be looking at all the different ways that we measure intelligence. You should be looking at cost to run index number and the different ways of thinking about token efficiency and cost efficiency based on the list prices, because that's all that matters. It's not as good for the content creator rumor mill, where I can say, oh, GPT-4 is this small circle. Look at GPT-5 is this big circle. and that used to be a thing for a while yeah i mean but that that that is like a on us own actually very interesting one right that is it well just purely that chances are the last couple years haven't seen a dramatic scaling up in the total size of these models and so there's a lot of room to go up properly in total size of the models especially with the upcoming hardware generations.

39:28Yes. So, um, you know, taking off my shit posting phase for a minute. Uh, yes, yes. At the same time, I do feel like, you know, especially coming back from Europe's people do feel like Ilya is probably right that the paradigm is, doesn't have many more orders of magnitude to scale out more and therefore we need to start exploring at least a different path. GDP value, I think it's like only like a month or so old. Um, I was also very positive when I first came I actually talked to Tejo, who was the lead researcher on that. Oh, cool. And you have your own version. It's a fantastic data set. Yeah.

40:02Maybe I will recap for people who are still out of it. It's like 44 tasks based on some kind of GDP cutoff that's meant to represent broad white-collar work that is not just coding. Yeah. Each of the tasks have a whole bunch of detailed instructions, some input files for a lot of them. Within the 44, it's divided into 220, 2 to 5 maybe, subtasks that are the level that we run through the agent acanus. And yeah, they're really interesting. I will say that it doesn't necessarily capture all the stuff that people do at work. No avail is perfect. There's always going to be more things to look at, largely because in order to make the tasks well enough to find that you can run them, they need to only have a handful of files and very specific instructions for that task.

40:47And so I think the easiest way to think about them are that they're like quite hard take-home exam tasks that you might do in an interview process. Yeah, for listeners, it's no longer like a long prompt. It is like, well, here's a zip file with like a spreadsheet or a PowerPoint deck or a PDF and go nuts and answer this question. OpenAI released a great data set and they released a good paper which looks at performance across the different web chatbots on the data set. It's a great paper, encourage people to read it. What we've done is taken that data set and turned it into an eval that can be run on any model.

41:23So we created a reference agentic harness that can run the models on the data set. And then we developed evaluator approach to compare outputs. That's kind of AI enabled. So it uses Gemini 3 Pro Preview to compare results, which we tested pretty comprehensively to ensure that it's aligned to human preferences. One data point there is that even as an evaluator, Gemini 3 Pro interestingly doesn't do actually that well in GDP Val AA. Yeah, the thing that you have to watch out for with LM Judge is self-preference, that models usually prefer their own output. And in this case, it was not. Totally. I think the way that we're thinking about the places where it makes sense to use an LLM as judge approach now, like quite different to some of the early LLM as judge stuff a couple of years ago, because some of that, and MTV was a great project that was a good example of some of this a while ago, was about judging conversations and like a lot of style type stuff.

42:29Here, we've got the task that the grading model is doing is quite different to the task of taking the test. When you're taking the test, you've got all of the agentic tools, you're working with the code interpreter and web search, the file system to go through many, many turns to try to create the documents. Then on the other side, and we're graining it, we're running it through a pipeline to extract visual and text versions of the files and be able to provide that to Gemini. And we're providing the criteria for the task and getting it to pick which one more effectively meets the criteria of the task out of two potential outcomes.

43:01It turns out that we proved that it's just very, very good at getting that right, matched with human preference a lot of the time, because I think it's got the raw intelligence, but it's combined with the correct representation of the outputs, the fact that the outputs were created with an agentic task that is quite different to the way the grading model works, and we're comparing it against criteria, not just kind of zero shot trying to ask the model to pick which one is better. Got it. Why is this an ELO and not a percentage, like GDP value? So the outputs look like documents. And there's video outputs or audio outputs from some of the tasks.

43:40It has to make a video? Yeah, for some of the tasks. Some of the tasks. What task is that? I mean, it's in the data set. Like be a YouTuber? It's a marketing video. Oh, wow. What? Like model has to go find clips on the internet and try to put it together. The models are not that good at doing that one for now, to be clear. It's pretty hard to do that with a code interpreter. And the computer stuff doesn't work quite well enough and so on and so on. But yeah. And so there's no kind of ground truth necessarily to compare against, to work out percentage correct. It's hard to come up with correct or incorrect there.

44:13And so it's on a relative basis. And so we use an ELO approach to compare outputs from each of the models between the task. You know what you should do? You should pay a contractor human to do the same tasks and then give it an ELO. And so you have human. I think what's helpful about GDPVal, the OpenAI one, is that 50 % is meant to be normal human. Yes. And maybe domain expert is higher than that. But 50 % was the bar for like, well, if you've crossed 50, you are superhuman. Yeah. So we haven't grounded this score in that exactly. I agree that it can be helpful. but we wanted to generalize this to a very large number of models.

44:57It's one of the reasons that presenting it as ELO is quite helpful and allows us to add models, and it'll stay relevant for quite a long time. I also think it can be tricky looking at these exact tasks compared to the human performance, because the way that you would go about it as a human is quite different to how the models would go about it. Yeah. I also like that you included Llama 4 Maverick in there. Is that just one last... Well, no, no, no, no, no. It is the best model released by Meta. And so it makes it into the homepage default set still for now. Other inclusion that's quite interesting is we also ran it across the latest versions of the web chatbots.

45:38And so we have... Oh, that's right. Oh, sorry. Yeah, I completely missed that. Okay. No, not at all. So that which has a checkered pattern. So that is their harness, not yours, is what you're saying. Exactly. And what's really interesting is that if you compare, for instance, Clawed 4.5 Opus using the Clawed web chatbot, it performs worse than the model in our agentic harness. And so in every case, the model performs better in our agentic harness than its web chatbot counterpart, the harness that they created. my backwards explanation for that would be that well it's meant for consumer use cases and here you're pushing it for something the constraints are different and the amount of freedom you can give the model is different also you have a cost goal we let the models work as long as they want basically do you copy paste manually into the chatbot?

46:30that was how we got the chatbot reference we're not going to be keeping those updated at quite the same scale as on the hundreds of models I don't know, talk to browser base they'll automate it for you True. We should. I have thought about, well, we should turn these chatbot versions into an API because they are legitimately different agents in themselves. Yes. Yeah. And that's grown a huge amount of the last year, right? Like the tools that are available have actually diverged, in my opinion, a fair bit across the major chatbot apps and the amount of data sources that you can connect them to have gone up a lot, meaning that your experience and the way you're using the model is more different than ever.

47:10What tools and what data connections come to mind when you say, what's interesting? What's notable work that people have done? Oh, okay. So my favorite example on this is that until very recently, I would argue that it was basically impossible to get an LLM to draft an email for me in any useful way, because most times you're sending an email, you're not just writing something for the sake of writing it. Chances are context required is a whole bunch of historical emails. Maybe it's notes that you've made. Maybe it's meeting notes. Maybe it's pulling something from your, any of like wherever you at work store stuff.

47:45So for me, like Google Drive, OneDrive, and our super-based databases, if we need to do some analysis or some data or something, preferably, model can be plugged into all of those things and can go do some useful work based on it. The things that like I find most impressive currently that I am somewhat surprised work really well in late 2025 that I can have models use Superbase MCB to read-only, of course, run a whole bunch of SQL queries to do pretty significant data analysis and make charts and stuff, and can read my Gmail and my Notion. Okay, you actually used that. That's good. Is that a Cloud thing?

48:24To various degrees of order, both chatgpd and Cloud right now, I would say that this stuff barely works in fairness right now. because people are actually going to try this after they hear it if you get an email from Micah odds are it wasn't written by a chatbot so yeah I think it is true that I have never actually sent anyone an email drafted by a chatbot yet but you can feel it right this time next year we'll come back and see where it's going totally Superbase shout out another famous Kiwi I don't know if you have any conversations with him about anything in particular on AI building and AI infra?

49:03We have had Twitter DMs with him because we're quite big Superbase users and power users. And we probably do some things more manually than we should in Superbase. So he's just the support line because you're QE's? A little bit, yeah. Been super friendly. One extra point regarding GDPVAL AA is that on the basis of the overperformance of the models compared to the chatbots. It turns out we realized that, oh, like our reference harness that we built actually works quite well on like generalist agentic tasks. This proves it in a sense. And so the agent harness is very minimalist. I think it follows some of the ideas that are in Claude code.

49:50And all that we give it is context management capabilities, a web search, web browsing tool, code execution environment. Anything else? I mean, we can equip it with more tools, but by default, yeah, that's it. We give it for a GDP valid tool to view an image specifically because the models can just use a terminal to pull stuff in text form into context, but to pull visual stuff into context, we had to give them a custom tool. Yeah, exactly. You can explain the next, but... No, so we turned out that we created a good generalist agentic harness And so we released that on GitHub yesterday. It's called Stirrup.

50:30So if people want to check it out. And it's a great base for building a generalist agent. It is kind of cool. For more specific tasks. I'd say the best way to use it is Git clone and then have your favorite coding agent make changes to it to do whatever you want because it's not that many lines of code and the coding agents can work with it super well. Well, that's nice for the community to explore and share and hack on it. I think maybe in other similar environments, the Terminal Bench guys have done, started to harbor. And so it's a bundle of, well, we need our minimal harness, which for them is terminus.

51:08And we also need the RL environments or Docker deployment thing to run independently. So I don't know if you've looked into harbor at all. Is that a standard that people want to adopt? Yeah, we've looked at it from an EVALS perspective and we love Terminal Bench. and host benchmarks of terminal bench on artificial analysis. We've looked at it from a coding agent perspective, but could see it being a great basis for any kind of agents. I think where we're getting to is that these models have gotten smart enough, they've gotten better tools that they can perform better when just given a minimalist set of tools and let them run, let the model control the agentic workflow rather than using another framework that's a bit more built out, that tries to dictate the flow.

51:56Awesome. Let's cover the openness index and then let's go into the report stuff. So that's the last of the proprietary numbers, I guess. I don't know how you classify all these. Yeah, or let's call it the last of the three new things that we're talking about from the last few weeks. Because we do a mix of stuff where we're using open source, where we open source and what we do, and proprietary stuff that we don't always open source. Like long context reasoning data said last year, we did open source and then all of the work on performance benchmarks across the site some of them we looking to open source but some of them like we're constantly iterating on and so on and so on so there's a huge mix i would say just of like stuff that is open source not across the site so that's a lcr for people yeah yeah yeah but let's talk about open let's talk about openness index this here is call it like a new way to think about how open models are we for a long time have tracked where the models are open weights and what the licenses on them are.

52:55And that's pretty useful. That tells you what you're allowed to do with the weights of a model. But there is this whole other dimension to how open models are that is pretty important that we haven't tracked until now. And that's how much is disclosed about how it was made. So transparency about data, pre-training data and post-training data, and whether you're allowed to use that data, and transparency about methodology and training code. So basically, those are the components. We bring them together to score an openness index for models so that you can, in one place, get this full picture of how open different models are.

53:32I feel like I've seen a couple other people try to do this, but they're not maintained. I do think this does matter. I don't know what the numbers mean apart from, is there a max number? Is this out of 20? It's out of 18 currently. And so we've got an openness index page, but essentially these are points. You get points for being more open across these different categories and the maximum you can achieve is 18. So AI2 with their extremely open OMO 332B think model is the leader in a sense. With Hugging Face. Oh, with their small model. It's coming soon. I think we need to run, we need to get the intelligence benchmarks right to get it on the site.

54:13You can't have it open in the next and not include Hugging Face. We love hugging face. We'll have that up very soon. You know, RefineWeb and all that stuff. It's amazing. Is it called FineWeb? FineWeb. Yeah, totally. One of the reasons this is cool, right, is that if you're trying to understand the holistic picture of the models and what you can do with all the stuff the company's contributing, this gives you that picture. And so we are going to keep it up to date alongside all the models that we do intelligence index on the site. And it's just an extra view to understand. Can you scroll down to the trade-offs chart?

54:46Yeah, that one. This really matters, right? Obviously, because you can be super open, but dumb. I mean, the slum obviously goes the wrong way here, right? A lot of people would like to see labs hill climb and target the openness index. This is the access to hill climb. Unfortunately, it might be fundamentally true that the slum will always go this direction, because once you open something up, then everyone else can get to the level of what you opened up. Well, so let me tweak your point system, right? Like you have these like numbers on the point system and it go up to 18, you know, but like just because I have a little bit of open data doesn't mean I'm necessarily that much better in someone who put a lot of effort into their open ways that it's smarter.

55:28So I might just mess with the point system to make sure that like I'm accurately representing the contribution to the openness. It is hard to wait for the materiality of the contribution to open source. Like we tried to make it so that it is quite well-defined and no one can disagree about like which category things should be. And so we're not saying like this was a big contribution or a small contribution in terms of impact on the industry or anything. It's just like how much of your data did you release? I would say that it is still valid to say that we train a model that's not that smart, maybe even not at the frontier for a particular size category, but we chose to open up all the data, all the training code.

56:12That is a very useful exercise for the industry and we want to recognize that even if the smartest model in the category. Yeah, and also a special shout out to NVIDIA Nemotron which doesn't get enough credit for the amount of stuff that they do. And honestly, it's a sales enablement for NVIDIA as well. Like the fact that they can do this is a side project. Totally, but I mean, it is true that NVIDIA have actually put an enormous amount of effort over the last year, especially into the Nemotron models. Yeah, and so many people actually use it for synthetic data and stuff. It's a pretty interesting secret of the industry that NVIDIA holds up all these guys.

56:45I mean, it's in their interest for there to be more AI. So obviously, I think you want to push openness as having an index. Every index that you push has encodes some kind of opinion or value. Yes. I think one of the openness questions for this year was people messing with the license. and so llama had this like if you have 700 million daily active users you're not allowed to use our model or you have to talk to us something like that so basically like what are your customers telling you about the kind of licensing worries that they have right because obviously most people will never hit 700 million users we have like a detailed breakdown of that in the openness index and that was actually one of the initial questions like took us down the route of wanting to do this um because yeah the simplest thing that like our opinion is is that there's a lot of advantage to having like an official osi license like mit or apache 2 because then the box is just checked you don't even need to read it because it's just apache 2 and you can do it ever you want and it's fine there are often very good reasons that companies don't want to release language models with those completely open licenses the index tells you so if you get the top category that's one of those licenses you're totally good and then we've got um some lower categories for when attribution is required and then when commercial use is not allowed.

58:03Yeah, they're there. So that's the openness index. Thank you for doing all those works. Let's talk a little bit or at least end the pod on just the trend reports that you guys do, which is kind of a bit of the bread and butter, how you make money. I highly encourage everyone to see George's talk at World's Fair, which gives a little bit of a preview. And you were very excited about talking about the smiling curve, or I don't know what you call it. Yeah, yeah, yeah. Yeah, let's talk about that one let's explain it for people and and i might i might actually put put it up um because i don't have it yeah you've got to copy the slide better be that'd be excellent it's important for people to have in their head because yeah people only get the marketing message from the labs that oh we're cutting costs all the time yep yep but it's true it's not the whole picture so okay a couple of like the big trends that we track at artificial analysis over time and that like we're always showing charts of on the trends page in these reports and stuff one that the cost of intelligence has been falling dramatically over the last couple of years the best way to think about that is that the cost for each terror of intelligence has been dropping the like one fact on that is that you can get intelligence at the level of gpd4 for over 100 times cheaper than gpd4 was at launch right now i think my number is a thousand actually if you look at um the amazon nova models which are very very cheap yeah like my my conservative statement is normally like 100x.

59:24But in fairness, this slide, like we were actually saying before the podcast, right, is like maybe six months old now. And it's conceptually still correct, but like could actually probably do a tweak on the exact numbers because like the market's moving so quickly. If you're feeling to kick it off, I mean, we'll have this chart. I told people to watch the World's Fair talk, but let's introduce what context makes you make something like this. There are two trends that seem to not make sense together, both of which we talk a lot about artificial analysis and are very important to developers building stuff in AI.

59:56The first is that the cost of intelligence for each level of intelligence has been dropping dramatically over the last couple of years. We track the cost run artificial analysis intelligence index for each bucket of intelligence index scores and each bucket you just see the line go down really really quickly and actually go down more quickly for each new level of intelligence that's been achieved over the last couple of years. So the rate of that cost decline has actually been going up. So we've got that being true. And yet it is clearly possible to spend quite a lot more on AI inference now than it was a couple of years ago.

1:00:33NVIDIA stock go up. It's going really up. I just heard from a friend's startup that just went to the shift zero. They're spending$5 ,000 per employee on coding agents spend alone. that's uh that's an impressive number we need to get our numbers up we're uh we're not hitting well i was like it's so high down i'm like are you doing something wrong because there are some efficiency questions along the way but like you can make ai inference useful to that level in a bunch of ways that i can imagine right yeah um i i don't think that's that nuts um but basically the reason we made this slide to answer the question right is to to show that the crazy thing is that it is actually true.

1:01:15We've had this 100x to 1000x decline in the cost of GPT-4 level intelligence on the left-hand side. And yet on the right-hand side, because the multipliers are so big for the fact that even though small models can do GPT-4 level now, we still want to use big models and probably bigger than ever models to do frontier level intelligence. We've got reasoning models using tokens, and then we're throwing them in these agentic workflows where they're consuming enormous numbers of input tokens and making enormous numbers of output tokens, working for a really long time, those two things taken together get you back to, we can spend enormously more today than we could a couple of years ago.

1:01:49Yep. I think that's right. There's a number of drivers at play and we kind of outline kind of six key ones here. But as complex, it's changing quickly. All of these have changed very dramatically in the last 12 months. Let's pick on hardware efficiency since you also track hardware stuff. And I think the general assertion or the message is that the efficiency from next-gen NVIDIA chips is actually not 4X. You have what, 3X or 4X? You have 3X in here. And it's like 2X maybe, or it's more of like a power story rather than like a sheer sort of compute tokens efficiency story. But yeah, what's going on in hardware?

1:02:30where okay so the the the answer unfortunately uh is it depends and it just depends massively on like so many things across a bunch of different types of workloads and ways to think about it so one of the simplest ways to think about this is to take single relevant model to think about serving it at speeds that are realistic for what you actually might want to hit and can afford to hit and then think about the throughput per gpu that you can achieve serving the model at those speeds. One of the reasons that's important is that there's a trade-off between the throughput per GPU that you can achieve and the per user speed that you can achieve.

1:03:07And it costs more to serve stuff fast to users. When you run all of that, for especially big sparse models, you can get a lot better than 2 or 3x gain going from hopper to Blackwell generation to video. I shouldn't be too controversial to say, but I'm pretty confident that Blackwell has delivered pretty enormous gains and that the next couple of years of NVIDIA's roadmap are going to continue to deliver quite enormous gains and that those will actually come through as lower total cost per token to the companies that are running models on them and will allow bigger models, will allow way more tokens to be made for lower cost and that that's going to continue.

1:03:49These things also stack on all of the software and model improvements being made. So basically my prediction across both sides of that smile chart that we're going to see the left-hand side continue to be true and probably for another order of magnitude and the right-hand side continue to be true for another order of magnitude. And that's going to enable a whole lot of things. Okay, well, I'll push on. Let's go back to the small chart. I'll push back on sparsity, right? We've gone a long way on sparsity. DeepSeek was a major pusher of fine-grained experts, let's call it. Yep. Right? I have a mental number of sparsity in terms of, let's say, active params versus total params.

1:04:29And that number went from 25%, let's say, down to like 15, right? You obviously can't really go below, I don't know, five. So there's a lower limit to sparsity is what I'm saying. I don't know that that's that obvious, actually. There must be a limit somewhere, right? Yeah, exactly. But we've got numbers in the wild that are quite a lot lower than that right now. So the GBD OSS models, like the big ones at about 5 % active. Kimi K2, is it like 3 % active? Oh, okay. I think, pretty sure. I've looked at those numbers. I calculated them. I don't remember. Yeah, but I remember thinking like, this must be it.

1:05:10Your 5 % is exactly like around the ballpark for the open weights models of what's released today. I think one interesting that gives me kind of pause when thinking that it won't go, the sparsity won't go higher or the number of percentage of active parameters lower is that we in our benchmark see a lot of performance correlated more with total parameters than active. And not that correlated with how sparse the models are. Our accuracy benchmark is part of AA Amniscience. It's very correlated with total. It's not correlated with active parameters, which I think is very interesting. And so I think, yeah, there could be quite a bit to go here.

1:05:54Awesome. Well, we don't have that much time, but I did want to leave some room to cover reasoning and non-reasoning models and token efficiency. Let's do that one. So at a super high level, people have to classify this binary thing of reasoning versus non-reasoning. People who are insider have some discomfort with that because basically you just have the think tag or no think tag. How have you guys decided to approach this? And also, how is that laid out over the course of the year where we have things like GPT-5, which is a model router? Let's say GPT-5 and ChatGPT, the consumer experience, is a model router.

1:06:27When you're hitting the API, you can pick the different versions and you can pick reasoning strength of the different versions. But that goes to why this is now such a complex thing. So earlier this year, and probably when you and George last spoke for the AI Engineers World's Fair, we had this great slide that was super easy where we would show that the average reasoning model is using 10 times the number of tokens per query in our intelligence index as the average non-reasoning model and there was this moment where that was a pretty clear distinction and extremely useful to look at it just like that definitely no longer the case not least because you can think about reasoning strength for a bunch of these different models but particularly because different models have wildly different token efficiency now, more than an order of magnitude in difference, that means that the way that you probably need to think about cost for any application is to use something like our cost to run intelligence index metric as the starting point for what it's going to look like for these different models, these different reasoning strengths, and this continuous spectrum from non-reasoning to reasoning.

1:07:30That's basically like where we're at. So we will still show reasoning and non-reasoning and define reasoning as when there is that separated chain of thought that you're getting at a different parameter in an api normally but it doesn't necessarily anymore mean that that model is actually going to have longer end-to-end latency that is going to use more tokens than something that is branded a non-reasoning model for the same task that's true i think 5.1 was it and then 5.1 codex had these these chart which is super nice of this like let's say bottom 10 percentile query being faster but top 10 percentile being longer and that's a kind of the efficiency chart you want to see right yeah that so that is a an extra thing let's say let's say that we've got that's a really important extra thing though right that you've got not just the average number of tokens being used by the model which we cover really well right now but the behavior that you want in the model is it to use more tokens when it needs more tokens and not to use more tokens when it doesn't need more tokens.

1:08:30So that's what OpenAI, we're basically claiming that 5.1 codex is better at. We don't actually publish anything on this right now, but have tracked it a bunch internally in our internal analytics on evals across all the models that we run, where we look at the difficulty, the questions, and the correlation between token usage and difficulty. And net-net, surprise, surprise, models have got better at doing that over the course of this year. I think going into next year, that's going to be really important, especially as you multiplied by the number of steps in an agentic workflow that a model has to take to get to an answer we are going to care a lot about token efficiency and number of turns efficiency for getting to what we want which would you rather have token efficiency or number of turns efficiency um or like which is more important to work on it's like it depends on the application and both are going to be really important uh yeah because your total total cost retail top edge airline Yeah, interestingly, in Tau2Bench Telecom, it's cheaper to run, you know, on a per token basis, more expensive models like a GBD5 compared to some smaller open source models because some of the GBD5, for instance, got to the answer faster.

1:09:41And so it was able to resolve the customer's query faster in fewer turns. And maybe it used more tokens per turn, but it suddenly cost more per token. So you would always rather use GPT-5 in that scenario. And so I think that's where we're getting to. I think number of turns is going to be a metric that we're going to be talking about a lot more. And I think it'll be something that people want to really start to think about a lot more. There's a trade-off in benchmarking here where most benchmarks need to be one turn to be autonomous, to be parallelized and all that. But most, a lot of real-life use cases need to be multi-turn, especially like quick multi-turns.

1:10:17So you can align. Yeah. Yeah. I mean, I would say that historically benchmarks have been single turn, but I wouldn't say they need to be at all into the future, right? Like we have a couple of agentic benchmarks in the index right now and GDPVal that we were talking about. We led the models to up to a hundred turns and our stirrup agentic harness to do that eval. And we're going to build similar stuff like that in the future. It definitely is hard and you've got whole kinds of infrastructure problems to run that and exactly as you say, parallelize it because we need to run that on hundreds of models and we want to do that really fast when new models come out and where labs want us to run it on their models.

1:10:52But you can do it. We've put it in the work to build that stuff and it's going to be great. Okay, so we've covered, I mean, there's a lot more to cover. I haven't even touched on multimodal, which is huge. We also do speech benchmarking, image benchmarking, video benchmarking, hardware. I like the way that you've done it because it's very smart, which is video takes a long time. So you pre-generate, right? So then people just pick their preferences and you can see the overall arena results. And you also avoid any sensitivity issues around unsafe content that is being generated. Yeah. And you can see it as a good thing or bad thing depending on what your view is, but it means that we have a quite active creative direction approach to trying to understand what creative professionals and users want to do with those image and video models.

1:11:42And so that we can be directing the arenas in our categories toward gathering votes on what people care about. One call out actually to listeners, like if you are using our arenas, is that you can submit requests to us for things that we should cover. I didn't know that. Yeah. Understudied categories, areas that you think the models are bad at and the labs don't focus on enough. Like if you want something solved, one of the levers that you have is send us a couple of prompts on it. We might be able to get a category going on it. And this thing that we talking about earlier, right? That once things get measured, they can get targeted.

1:12:16You can make that work for you. For me as a content creator, infographics, very needed. I took the latest DeepSeek paper and, uh, I, you know, they had some descriptions of their search agents and their coding agents and I put it in and I created an infographic. And, um, I just think like this, an industrial use case that doesn't require a lot of, I guess, design taste, but just requires some, like you need to conform to some preset references, which is something that, uh, that is increasingly important, especially in the Nano Banana series. But yeah, I think OpenAI is releasing Image 2 soon, which is going to have that.

1:12:51So I think it's all of a kind where people need to incentivize workhorse use cases and not just art. I don't know. Totally, yeah. What are we going to be talking about next year? What's emerging that you're seeing and maybe not in the discussion? The first answer that I'll give to that is the boring answer is that on most of our charts, the lines go in a particular direction and our overall prediction is the lines are going to keep going that direction. We're going to do a lot and do a lot to be as useful as possible to developers and companies to measure what's important on every one of those and along those lines.

1:13:24But I think we're going to talk about similar stuff. It's just that we're going to have continued on the trajectory for another year and things are going to feel pretty different because of that happening. I know this is the boring answer to that question. No, no, I mean, I'm a fan of things that, truths that don't change because you can build and plan for that. And I think in media in general, in the podcast business, newsletter business, Twitter business, people are addicted to change. Like, oh, everything's breaking, everything's... No, like there's some truths that are just constants that you can plan on and build.

1:13:56And yeah. I think one of the truths is that the demand for AI intelligence and smarter AI intelligence is going to be insatiable. Some people disagree that, okay, once we reach certain thresholds, then you don't need more intelligence. I think to that, I ask people, have they ever worked with or managed someone in a work environment and wouldn't press the button that they were smarter to make them smarter or better at their job? Or would they never press that for themselves? And I'm not sure that that's the case. But I think for artificial analysis, we'll keep benchmarking raw intelligence, but we also want to think about it and explore models more deeply across other axes as well.

1:14:37I think hallucinations, the start of that, but we're getting into wanting to support people and understanding, okay, the behavior, the personalities of the models to help people make more nuanced decisions. You're going to have a personality bench. Maybe. That is a direction that Chad G, OpenAI is leaning into a lot. So if you manage to solve that, you should definitely talk to Fiji and Roon. Oh, okay. Yeah. So what is going to be included in, let's say, like a V3 of the intelligence index? because obviously you're going to saturate in March. Why don't we break it now? How soon is the podcast going to come out?

1:15:12Wherever you want. Okay, so we're at v3 right now. So the version that's going inside is v3. v4 is what we're going to call the next major update. Surprise, surprise. We're going to be adding several of the things that we've actually talked about today that we've launched over the last few weeks. So that's not going to be wildly shocking, but some of the things that are most exciting is that adding GDPVAL is going to give us this general agentic performance in a really strong way in intelligence index. And in critical point, the physics eval George was talking about similar to Frontier Math, that gives us completely new view with a brand new data set of very, very hard research problems.

1:15:49We are going to be using omniscience and we are going to be using hallucination rate. The exact ways that all those are going to come together. The waiting is going to be hard because the numbers are different. Yeah, we're going to make sure that we don't do anything to cause odd distortions and stuff that could be misleading. But every time you version it, you have a one-time reset of them. Exactly. Yep. That's exactly how we think about it. We will make sure that within each version number that there's no drift in any of the scores so that people can rely on them and reference them. You just have to watch out for that version number.

1:16:19Once it's v4.1, those numbers won't be compatible with v4. Of course. There's a little bit of debate over the accuracy of TileBench. I don't know if you're clued in to what's going on. Apparently, a very high number of TileBench tests are impossible. potentially for the earlier versions TAL2 Bench Telecom we're pretty convinced is pretty good if anything the only issue there is that models have got very good at doing it and so like anything TAL3 yeah on we go yeah on we go okay well thank you so much for providing such a great service to the industry I'm glad to at least know you guys before you got famous and now you are famous oh look our pleasure and we really appreciate your support along the way like I wasn't kidding at the start right that it was a quite material moment for us like when artificial analysis was covered on Latent Space.

1:17:08Some random guy in San Francisco mentions you. I was a fan of Latent Space for like a year before you mentioned us. So I'd been listening. I don't think I was like familiar with like you personally yet at that point, but like I listened to your voice probably for many, many hours. And so once like you mentioned, then like got to get to know you and like meet you for the first time nearly a couple of years ago. It was really cool, honestly. So yeah, it's great to be here. And thanks for being such a great member of the community and kind of spotlighting projects which don't have attention and bringing them to your audience.

1:17:43Yeah. Well, actually, so it wasn't me, right? Someone in the Discord dropped it in our Discord. And I rely on our community and it kind of feeds itself, right? Nice. So someone brought it to my attention. I don't know who. We should probably go back and check. But once I saw it, I was like, this looks good. This is something I always wanted. I wanted to build it. I was too shy or dumb or lazy to build it. And you guys did. And that's a whole thing. So thank you for doing it. I built some really cool other stuff, like this pod. Yeah, yeah. Totally. So thank you. That's it. Great. Thanks.