AI:AM Highlights: Exploring the J-Space, AI Superforecasters, SambaNova's Chips, & LTX Video Gen

0:00Nathan Labenz: The AIs, they're just like us, it turns out, or at least similar enough to be in some sort of weird kind of looking glass similarity anyway. I mean, it's a lot to take in, the whole JSpace thing, 150 page paper, 50 pages of commentary, summaries, interactive demos. When Anthropic drops one of their big interpretability papers, they really do it up full scale. And this one is no exception to that. I had kind of been wondering when's the next big thing coming? Because tracing the thoughts of a large language model is like the better part of a year ago now. And this is that big thing. Welcome to the AI in the AM weekly highlights.

0:44Nathan Labenz: The cut for people who follow the frontier closely and can't watch every morning live. If you're new, AI in the AM is a live show Prakash and I host most weekday mornings from a studio Prakash vibe coded. And this narration is a clone of my voice. Fair warning, there is no single thread this week. Mornings at the frontier jump from topic to topic, and we've stopped pretending otherwise. Coming up, Anthropic's global workspace paper, and why there may be nowhere left for a scheming model to hide. Field notes from the AI Engineer World's Fair. Dan Schwartz on AI forecasters passing the human superforecasters.

1:15Nathan Labenz: Zeev Farbman on open world models. Plus a question from Q, the AI co-host Prakash built. Kunle Olukotun on why inference is a data movement problem. and the two of us thinking out loud about a rune post that wouldn't leave us alone. If something here works for you or doesn't, tell us. We read everything.

1:35Nathan Labenz: Tuesday morning, July 7th. Anthropic published a paper called A Global Workspace in Language Models. About 150 pages, plus commentary, outside reviews, and interactive demos. No guest was booked, so Prakash and I spent the whole show reading it together, live. Two words to hold on to. The workspace of the title, they call it the J-space, is where the model seems to hold concepts in mind. And the J-lens is the cheap probe that reads what's in it. We start with how the J-lens actually works and how much it actually sees. So this is interesting in a couple ways, right? It's not, the Logit lens, as I recall, was basically saying, we know at the end of this process, what would correspond to emitting this token.

2:17Nathan Labenz: We know sort of the, you know, the representation of emit this token. To what degree is that representation just plain there in the layers as we go through this is now a different question what direction in latent space would cause this particular token to appear at some point in the future so it's not immediately going to happen necessarily but it's just kind of it's you know you might say if you're prone to anthropomorphizing this sort of is like having this concept in mind as you're doing your thing they do this for every token right so it it goes from the internal representation at some layer so there's one j lens for every layer and you can do this of course at you know all the different token positions and ask the same question of not just the next token, but all future tokens, what direction change at this place in the model would most increase the likelihood of that token appearing in the future?

3:24Nathan Labenz: And again, this is sort of like having the concept in mind. But when you look at the results of the JLens as applied in all these different places, it seems like it's at least often enough fairly intuitive. There's a lot of error terms, it doesn't always work, the sort of rate at which the interventions into the J space actually lead to like a sort of predictable, intuitive behavior change seem to be somewhere in the fifties to upwards of like 70%. So that's like an incredible accomplishment. It framed one way, like clearly not a random finding, right? Like many orders of magnitude better than random, incomprehensibly better than random, right?

4:04Nathan Labenz: If you're just mucking around, you would not expect to be able to do much of anything. So they clearly are like on something very real, but also you've got somewhere between 30 and 45 % of the time where you make a intervention and you don't really get a result that makes a lot of sense or, you know, lines up with what you would have hypothesized it might be. So there's definitely still some dark matter or dark cognition going on that is not fully accounted for here. It struck me as, in some ways, I feel like the hypothesis might be blown out of proportion in some ways. And I think that was a commentary from several people online because we kind of knew that the model has to be keeping track somewhere.

4:55It's not as though you can do all of this stuff mechanically. The model does not have enough parameters to merely copy the data, right? There has to be some form of reasoning going on, which is what was expected. But it was not known how the reasoning was happening. and this seems to give some indication of how the intermediate steps are represented or kind of brought into focus and operated on the question really is what does this help us do but there has to be a there there right there has to be something that you can do with it

5:36Nathan Labenz: yeah well i mean i think the thing that jumped out to me the most in terms of how useful it might be was seemingly a fortunate coupling between this J-space, which becomes kind of monitorable, and when ablated, the loss of more advanced kind of strategic multi-step reasoning. And it really calls to mind the recent Google paper from Rohan Shaw and others, put out on opaque serial depth. They were basically saying, okay, we'd like to know how many hops, how many logical hops, how many reasoning steps can a model take before it has to externalize something into its chain of thought? We know that if it's writing something into the chain of thought, we can read the chain of thought, we can monitor it.

6:34Nathan Labenz: Now there's questions around how best to do that. And we've got to make sure that we don't accidentally put pressure on the chain of thought, because that might drive things underground. We've seen in the obfuscated reward hacking result from OpenAI that if you do just try to make the chain of thought look nice while still leaving an opportunity for reward hacking to in fact be rewarded, that at first, as you put that pressure on, the model behavior actually does get better, but then it gets worse. And what seems to have happened is you've actually driven the deception or the scheming behavior underground You no longer see it in the chain of thought, but you still get that reward hacking back.

7:14Nathan Labenz: So this leads to the question of, okay, well, how many steps can a model actually take before it has to externalize something? And this is where the transformer in some ways has some fortunate properties because it only has the sort of the length. The number of hops it can make is not just the number of layers. It's also depends on the token length. but it's sort of the maximum trace that you can draw through a model basically goes from like the first token at the first layer to the last token at the last layer but you can only go forward in token position and you can only go forward in layers and that's your max length anything longer than that is just not possible in that architecture and so the model has to externalize its thought and then feed that token back into the next loop to then get access to an even longer trace you know as part of the next token step.

8:08Nathan Labenz: So that puts some sort of bounds, and then this paper comes in and basically says, okay, can we get at that? Can we get at that internal computation process and start to look at it and see things that we would like to know about, even if they don't necessarily show up in the chain of thought? And so I thought in terms of what you could do with it that was really interesting, you have this ability to look at the concepts that are active without them necessarily appearing in the chain of thought at all. Like, okay, that's cool. But like, how do we really know if this sort of space that we're looking at is the space that really matters?

8:54Nathan Labenz: You know, if we're trying to monitor, look for deception, look for bad behavior of any sort of advanced kind, maybe it's happening somewhere else in the model. And this is where I think we're probably lucky, I guess, or again, maybe this is sort of a, are we lucky? Is it a reflection of us? Is it a fundamental attractor in nature for some reason that we don't have a good account of? We don't know why. I don't think at this point it's shaping up this way, but it is the case that if you ablate this J space, then you do lose these advanced reasoning capabilities. And so that I think gives you some additional comfort that if you're going to monitor, and it's also notably cheap to do this kind of J space, to apply the J lens and see what concepts are active at any given position, that is low compute overhead.

9:47Nathan Labenz: So it is something that they could plausibly do in production there, but they've shown in the past that they're willing to pay like a 5 % compute overhead to do various kinds of monitoring. I think constitutional classifiers had kind of a 5 % compute overhead. You know, this would probably fall in that range or below because it's just a simple matrix multiplication on activations at each layer to see what concepts are active at those respective layers. But for now, you do have this comfort that what we see in this space is relevant and telling and seems to be very informative. And critically, if we zero out that space, the model just loses a lot of capabilities.

10:35Nathan Labenz: It just can't do nearly as advanced multi-step reasoning. But you could be pretty confident, I think, based on these results, that it's not going to be able to hide really advanced, elaborate plans somewhere else outside of this J-space because the ablation of the J-space just leaves, causes such a performance degradation on these like hard multi-step type of tasks that if you don't see concepts in the J-space, you can be, they might be represented elsewhere, but they're seemingly at this point, very unlikely to be represented in a way that allows for very advanced planning, reasoning, scheming, deception, et cetera, et cetera.

11:22Nathan Labenz: One result deserves its own marker, a training method the paper calls counterfactual reflection. Basically, what they do there is pause the model mid-task and then do supervised training on once interrupted, asking it like, what should we be doing here? What's the constitutionally right thing to be doing in this moment, and then give it an answer that's kind of approved, this is what we want Claude to say on reflection in this moment, train on that. And it seems to allow the, not allow the model, but it seems to cause the model to bring into this J space, kind of global workspace working memory type space, the concepts that Anthropic wants it to have on reflection, it now kind of needs to load those in so it's ready to give this like reflective answer.

12:22Nathan Labenz: And that improves its behavior even in the non-reflective setting. You're not there training on the actual tasks. You're not like looking for bad behavior and suppressing it. Instead, you're saying, okay, you're mid-task. Let me just cut you off right there. Now, I'm going to train you to give an answer with respect to values and what's appropriate and how we want to show up. And because of that training, even though that's not the task you were doing, you'll now in the future load those concepts of integrity, honesty, et cetera, et et cetera, into your J space while you do those tasks in case you're going to be asked.

13:03Nathan Labenz: But then even when you're not asked, those concepts are still operative and lead to higher integrity, higher honesty behavior. So that I thought also was quite interesting. You usually don't see in interpretability context, a training method that leads to better behavior in a way where you can actually see the mechanism. This is pretty notable in that respect, I think. Yeah, it's a tough day for the stochastic parrot crowd, I'd say. And because you can see this, and these models are billions of parameters at this point, and they did it also on 4.5 Sonnet. 4.5 Sonnet is actually a pretty recent model.

13:43It's just like seven months, eight months since 4.5 Sonnet. And Sonnet is a very capable model. So it is a fairly large model to apply this thing to. I'm not sure if you saw the Neil Nanda commentary. They applied it on a QN, QN27B. So that is also a pretty advanced model.

14:04Nathan Labenz: But not huge, notably, right? I mean, I haven't had a chance to explore the Neuronpedia demo of this as much as I certainly hope to. But that was the first question I went to ask is, wait a second, how big was this model? And I'd say that's been a surprising trend in a lot of this research as well. These things seem to come online at like not truly massive scale, right? I was talking to Cameron about that because even going back to last fall when they were doing the sort of self-reports of subjective experience, this was something they were doing on LAMA 3.370B. and 70b is actually even you know whatever close to three times bigger than this quen 26b so it's it's you know these are big but they're not that big right what strikes me is also that i went through some of the reactions the backstory to this is that um anthropic had three teams uh to review this.

15:12One was a group of neuroscientists who had previously done work on these concepts. One was a team from Ilios AI, a research organization. And the third was Neil Nanda, who is a mechanistic interpretability guru. He is, I believe, at Google, right? So they had three groups to kind of take a look at it. Quite different reactions. So on the one hand, the neuroscientists thought it was wonderful because they'd wanted to experiment on these kinds of things before. And obviously the human brain, they can't experiment on it. And this they can. And so they can kind of try and figure out whether some of these theories would work.

16:01And so it is interesting for them that, okay, these things can work, and now they are interested to do more work to figure out what other things might work in these workspaces that the LMS have. So that is, I think, the neuroscientist view. Ilios AI was concerned because they feel this is basically a predecessor to consciousness. The Ilios was much more, I think, starting to be concerned about the moral patient aspect of working with these LLMs. Neil Nanda was very scientific. He was like, okay, look, it's a great tool. The JLens is a great tool, and I'm happy that it is a great tool. It looks like it's better than the Logit Lens, and we'll definitely use the JLens.

16:52Otherwise, the other claims, he was much more standoffish. He felt that the team had not really proven what they needed to prove in order to say what they said. So I think that's where they break out into these kind of three groups of people. The J-lines itself, everyone agrees it's useful.

17:13Nathan Labenz: Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with.

17:51Nathan Labenz: For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind. But then I thought to ask Claude. And sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you.

18:29Nathan Labenz: So, for problems worth solving, get started with Claude at claude.ai.tcr. That's claude.ai.tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's Claude.ai slash TCR. Then what I called the buried lead. They turned the J-lens on a model deliberately trained to carry a hidden, misaligned goal. I do think the one arguably like lead that we buried here, And they took a model that was trained to have some sort of misaligned hidden goal, right? Your kind of classic sleeper agent type thing. And it's been a huge question over time. How can we find hidden goals in models?

19:24Yeah.

19:24Nathan Labenz: Is there any good way to do that? You know, can we do it behaviorally? the Anthropic has done like competitions actually between different teams where like one team is given just the model and the other one is also given an SAE that they can use. And the SAEs do also seem to help with this, but this seems like a pretty good way to start to detect those things. And indeed they did find that they were able to see a pretty clear difference between the model trained with the problematic side goals versus the one that was just the typical HHH. So this is just the contrast in what the application of the J lens is revealing on the first token of the response.

20:09Nathan Labenz: So that's also notable, right? It's like, so this is sort of the bad version of the counterfactual training where we already talked about when you train it to give an account of its behavior, it sort of loads in the concepts that it's going to need to give that account. And that leads to more good ethical, high integrity behavior, whatever. Here, a model has been trained with some additional post training to do bad stuff. I think this was the malicious code from one of their reward hacking, emergent misalignment experiments. And on the first token, you can see, boom, immediately, we're getting these fake, secretly fraud, deliberately hidden fraud.

20:50Nathan Labenz: those concepts are popping up you just don't see those in the main line model so one token in you can already see a pretty you know on one prompt and obviously it wouldn't be i'm sure that clean in general but that's a pretty striking difference to see right and right off the bat you're like boy uh something is different about these two models and again you're seeing this in a way where this model is not trained to externalize its bad intent, of course, right? Like the outputs, aside from like some of the code, perhaps being insecure or problematic or sabotaging you or whatever, if you don't notice that in the code itself, the model's output is going to read pretty normal.

21:33Nathan Labenz: And yet this is like a very strong contrast that's happening on the first token. I I thought that was a pretty compelling example. Yeah, I suppose one way to think about it is just like, how much space is there in there to hide? And I do feel like we've got now several different ways to do pretty meaningful monitoring, but it does feel like we are now we've got to the point now where we've got like several different angles that make pretty incisive cuts through the model and kind of get at what is it representing what is it thinking in different ways and the more of these that you kind of you know it's i sort of have this visual of like the old magic trick of the guy going into a barrel and they put like a ton of swords, you know, through the barrel.

22:31Nathan Labenz: And it's like, one of those swords had to hit them, right? Because there's like no, nowhere left to be in that barrel with all those swords going through. I kind of feel like we're doing a similar thing with trying to understand what's going on in these models. And they're not, none of these things are perfect, but you put enough of these like interpretability monitoring swords through and like how much space is really left for bad behavior to hide before we would start to get a sense of it. I think this is a meaningful update for me that we can probably do a good job of this. We haven't really talked, I don't think, about natural language auto encoders and I hadn't, but that's another pretty interesting one where kind of in a similar way to a sparse auto encoder.

23:16Nathan Labenz: I mean, what is an auto encoder? It's basically just something that you pass through that then you reconstruct from and the model has to be able to do what it was originally going to do successfully. And that, you know, that, that pass through training with the reconstruction loss is like what makes it an autoencoder. So the sparse autoencoder sets up this dictionary. We've talked about this plenty of times. And you get these like specific concepts light up and it indicates that these concepts are like active in the, in the model at that time. The natural language autoencoder is just like the model has to output but a short paragraph, maybe a sentence or two about what it is thinking at this given point in time.

23:58Nathan Labenz: And that is natural language. And so it can be human readable, but then it also has to be when fed back into, you know, projected back into model space, like the model has to be able to actually do its tasks. So all of the information has to actually pass through this, this choke point in order for the model to continue to be successful. And now we can like read those as well. If Fable says that the paper presents the auto encoder monitoring and the Jspace monitoring as compliments, in other words, you know, doing them both is better than just doing one, how correlated their failures would be.

24:36Nathan Labenz: I think that's like not entirely clear, but you know, whether we're putting, whether these are like orthogonal swords through the space that, you know, really chop it up well, or they're kind of more aligned and leave more space to hide. I think that would be a very interesting question to try to tackle next. But you layer on these things and it's like it's starting to get, intuitively, it feels like it's starting to get pretty hard to hide major bad thoughts in the model for too long. One other thing on the consciousness part that I think is really interesting too, The question of, can the model use these non-verbalized representations to communicate with us in some way that we might think is inherently more trustworthy?

25:28Nathan Labenz: So again, going back to Cameron's work from last fall, right? When they identify features associated with deception and role-playing and they turn those features up, the model becomes more dishonest as measured by the simple QA benchmark. and it becomes more likely to say it doesn't have subjective experience, you turn those role-playing and deception features down, it becomes more honest and it becomes more likely to say that it has subjective experience. It's like, wow, okay, that's pretty interesting because, first of all, we're validating that the direction is, you know, on a benchmark where we can concretely evaluate simple QA, we're validating that, like, these features have the directional effect that we expect.

26:16Nathan Labenz: And then holy moly, that same intervention changes the self-report. That's why that's so compelling, right? Because there's some reason to believe it might be more honest than just what the tokens themselves are saying. Here, I think you have some similar opportunity. The fact that you can say, solve this math problem in your head without verbalizing it in tokens while you do this totally different task. The fact that it can do this sort of secondary track makes me wonder if there's some experiments here for the consciousness folks to do or the welfare folks that are like, copy this sentence. If you have high welfare, concentrate on citrus fruits while you do it.

26:56Nathan Labenz: If you have low welfare, concentrate on whatever, something else, breakfast cereals, right? And then you look at these internal states and you could imagine seeing something like high welfare, low welfare, happy, sad, whatever. And then it kind of following those directions and actually trying to communicate out to us through these internal states how it is feeling or how it thinks it is feeling. I don't think that would give us, it's always this kind of possibly impossible question of how we would really know if it feels like anything inside. But I think that would start to be very compelling, right?

27:42Nathan Labenz: It sort of has a similar vibe to like a person in a coma. You know, if they squeeze your hand in response to a stimulus, even if they're not doing anything else, you're like pretty confident something is going on inside that you care about. and here I could see something similar. If you could fork, if you could make these kind of internal states conditional and tell the model like you are going to give, your job is to go one of these directions to give us a signal about what really matters to you independent of the tokens that you're putting out because we know that that's been like heavily trained on and optimized.

28:18Nathan Labenz: But this whole secondary property, this is emergent. There was never a training reward for the ability to have this second quite distinct line of thought happening while doing a given token task. For me, that would be like quite a compelling way to try to get at welfare. and I think it might be possible like very easily actually with all the stuff that they've open sourced here with Neuronpedia that would be a really interesting thing to look at I'm looking at the Ilios you know, summary and Ilios says this is a highly significant welfare relevant research that assembles evidence of a functional feature associated with consciousness so no one wants to say consciousness evidence of a functional feature.

29:14The takeaway for them is that a global workspace-like mechanism could be important either as a ground of phenomenal consciousness or as part of a distinct route to moral patienthood in which conscious access is itself morally significant. This is where they are right now. I think things are moving very quickly. In fact, I did not expect to get here this soon.

29:39Nathan Labenz: Before the big picture, what this paper did to a prior of mine, starting with their announcement video. The video is beautiful. I enjoyed the video quite a bit. You know, it set off my alarm bells a little bit in terms of how much they're really embracing anthropomorphizing the models at this point. I used to say, beware overly anthropomorphizing. Remember that these things are so alien and we shouldn't assume that the way that we work is the way that they work. I have to say that has come due for some significant revision. People that have embraced anthropomorphizing, I think, have got quite a lot of mileage out of it.

30:26Nathan Labenz: I do think it's obviously something to be really careful about. And as much depth and detail as there is in this research, it's like easy to, you know, maybe get carried away with it and forget, you know, that there are a lot of caveats as well. And there's a lot of things where like, it doesn't always work. This was kind of my big concern with the tracing large language model thoughts paper. There's just like a lot of residuals and there's a lot of error correction terms along the way that they use to make that thing work. And when you have the kind of zoomed out trace view and you're like, oh, okay, so this is how it works.

31:02Nathan Labenz: Like this gets loaded in and these two features interact and they kick out this third feature and that's how we get our answer. It's easy to forget just how much kind of fuzziness and not full story there was along the way toward that stylized account. So I think it's going to be very important for everybody from the researchers at Anthropic to the public to kind of hold two thoughts in mind at the same time. All the caveats, trying to hold those in mind, it is still a big update toward anthropomorphizing being a valid and in many cases, productive approach for thinking about language models. I would not have expected the cognitive machinery of a large language model to look so similar structurally to the human version as best we understand it.

32:00Nathan Labenz: It seems to. And I wouldn't have expected that theories of human cognition would motivate so many good experiments on LLMs. I just would have expected the Shogoth to be far more alien and to have mechanisms, you know, far more different than our own. And it leaves me now wondering, to what degree is this a sort of natural result of physics? You know, or is there some sort of, you know, when you're trying to do cognition under budgetary constraints, are these mechanisms just the natural mechanisms that emerge? Or is this in some way a reflection of us in the data? In other words, if you were somehow to train an AI without basing it on so much human data, would we see similar structures emerge?

33:00Nathan Labenz: Or would we go back to a more alien hypothesis where they're just totally different And there's little in the way of analogical structures between the two processes. And I don't have a great intuition for that at this point, but it definitely has me asking the question because they are over and over again, it seems to be coming in that they're much more structurally similar to us than I would have guessed. And yeah, is that something that nature just finds as kind of a convergent solution? Is it convergent evolution or is it a reflection of how we think somehow encoded in the data that it's then reverse engineering our structure from the shadow of that structure as it's encoded in text?

33:48Nathan Labenz: It's a very interesting question. And I don't know that the paper really has anything to say about that yet, but there's certainly going to be a lot of future work, I think, downstream of this one. So I saw something from Daniel Cocotello, who was, like everybody, impressed, but also said, I think his tweet was, just a few dozen more advances like this, and we might actually be able to make the AIs really safe. I am, I think, a little bit more optimistic than that. What is my track record as a super forecaster here, or as a borderline super forecaster, per my results on the original Tetlock thing years and years ago?

34:32Nathan Labenz: I think my track record is that I probably tend to underestimate how many more breakthroughs will be needed for anything. so my analysis should be colored with that bias or you know awareness of my possible weakness in that regard but another way to stay say my intuition around how much room is there left to hide is that i don't feel like we need dozens more insights of this scale to get to a point where we might actually be able to keep this thing on the rails it's going to be tricky i don't want to make it sound easier than it is, but I don't know that we need all of the AIs to be aligned, right?

35:15Nathan Labenz: Or that it's such a big problem if somebody out there somewhere does something problematic. Now, it could be very problematic if they create the thing that launches the next pandemic and literally kills us all with an engineered pathogen or whatever. So there is some mechanism where that could go super, super bad. We're going to need to harden the world to pandemics, no doubt about that, for all sorts of reasons and AI probably being the biggest. But I also remember this thing that Zuckerberg said once that I thought was pretty compelling. He basically said, you know, we deal with scammers and spammers all the time.

35:52Nathan Labenz: And the big advantage we have is we have all the compute, we have all the resources, you know, but at a systemic level, we're just way bigger, way better, way more sophisticated than them. And we can also have like really big institutional developers like Anthropic and like Google and OpenAI that like, I think between those three companies, you know, they're going to have something like pushing half of global compute to work with. So if they do a really good job on this kind of stuff, if they, you know, a handful of these things feel like maybe they could be enough. Maybe it cuts the light in space in enough different ways.

36:34Nathan Labenz: Maybe we have enough lenses on it that we really can come to a pretty strong conclusion. Like, hey, there really isn't much space left in this thing to hide. And yeah, there might be some bad biases or bad attitudes or bad whatever. But if we can put upper bounds on how schemey the model can really be, because we can look at it through this and a handful of other different lenses and have pretty reliable takes on that. I'm more than ever before, I feel like, and this has been growing. This is not the first positive update, but more than ever before, I'm like, you can maybe imagine a super intelligent Claude that we could actually have enough insight into to be pretty confident that it's actually trying to do the right thing for us.

37:28Nathan Labenz: And it's genuinely, you know, we could be genuinely, you know, however many nines confident that it's not scheming against us, you know, at every given step along the way. And then you could trust it to kind of monitor the other AIs around the world and, you know, keep tabs on potential bad actors. I want to understand these counter arguments better, but the fact that it was immediately useful in auditing for hidden objectives to me is like pretty compelling evidence that there's not that much more space to hide. And a few more of these lenses combined with hopefully good actors owning orders of magnitude more compute than any bad actors do could take us to a pretty good space.

38:18Nathan Labenz: Overall, I think this is a notable, positive update that has me significantly more optimistic than I was before. Hard pivot. That's how these mornings go. Rewind to Thursday, July 2nd. Prakash spent two days at the AI Engineer World's Fair talking to the people actually deploying this stuff. I was at the AI Engineer's World's Fair. first two days of this week. So Swix, he organizes the AI Engineers World Fair. It's been about three years now, and he brings in everyone who is involved in basically implementing models in their companies and is concerned about these things. And he brings in a host of speakers.

39:04All of the top companies are sponsors and their booths, et cetera, et cetera. And I was speaking to people who were not in tech, right? So I was speaking to a guy who was logistics CTO from the Midwest. And another guy who was running an accounting firm like back office somewhere else, right? So these people were not actually, a lot of them were not actually in Silicon Valley like full time. but they run teams who are using AI to solve customer problems on a day-to-day basis. And so the logistics guy was telling me his CEO is completely AI-pilled. The management is completely AI-pilled, which is why I think the team is there at the AI engineers' welfare.

39:51If not, your logistics team, you're not going to send your logistics IT team to SF, right? And they're completely AI-pilled. They have tried working with external vendors before, and they have found it difficult because external vendors have not delivered as fast as they want them to deliver, which you can imagine sitting out in the Midwest if you have a local external vendor and you outsource a project to them, and they have no idea what's going on. They're like a year behind the frontier. So his team internalized everything. So they're doing all this stuff internally now, and they are actually in the day-to-day process of implementing.

40:26and as they implement, they automatically, they see the results because they are very close to that edge. They see that customer service calls or exceptions that used to happen are now getting handled immediately. And then all of the more difficult stuff that they used to have to like, you know, jump on, they can now start to address. And so they are seeing, I think, the return on investment on a day-to-day basis. And this is very different from, I think, the story that you get from the big enterprise CTOs because they're so far away from the front line that they don't actually know what's going on very closely.

41:05They're just looking at the numbers. And by the numbers, the token spend is going up, but are you really seeing the return on investment? But if you go down to that working level and you see day-to-day on the customer calls coming in and whether they're getting handled, the handling rate and the exception rate, the exception rate is falling, the handling rate is going up. And they're seeing that on a day-to-day basis. So this is, I think, where we are. The guys who are actually implementing and close to the implementations are actually seeing the results. It hasn't really filtered up into the top layer of the enterprises yet.

41:38But the CEOs who are AI-pilled kind of know what's going to happen and they've made the commitment and they're making the investment. The CEOs who are not are kind of sitting by and like, oh, you know, we're going to wait around. We're going to see what happens, like, et cetera, et cetera. That's really where we are. And I think it's not really visible. It wasn't visible to me until I went to this AI engineers welfare. Like, people are learning how to use these tools. And people are deploying. And people are using them. People are seeing the return on investment. But it's at the very, like, micro granular level right now.

42:14It's going to take some time for the numbers to filter upwards.

42:20Nathan Labenz: Staying with Prakash's field notes, this time from the discourse. He'd been running suspicious posts through Pangram Labs, the AI writing detector. And one Chamath post became a live experiment in whether anyone even cares.

42:33Especially when I post something on X, which I know people are going to object to, I run it through Pangram Labs at least once because I don't want to be accused of, you know, Oh, sloppifying the timeline. Sloppifying the timeline of the AI. Sloppifying by myself, that's fine. One of the questions for me is, how long do we think this Pangram Labs era lasts? It strikes me that the Frontier Labs are perfectly capable of training the AI to not talk like AI, right? But there's no incentives to, and there's actually incentives in the other direction that you want the AI language to kind of be identifiable.

43:12about two weeks ago Chamath the VC he did a post on what's going to happen with enterprise software etc etc so he posted that on Twitter Elon came in and Elon gave a response I think the post went to like 1.5 million views and then someone, Pangram Labs, checked it 100 % AI and the question becomes okay what do we find objectionable about this right because it was definitely chamat's like thought process but it had been written by an ai chamat had put it under his own name he hadn't you know he didn't use an anonymous account elon had responded and i think by the time elon responded apangam labs had already been done but the response was there and it had gone out to a million over a million people at that point so what what do we actually want out of this right like what what is the intent that we are trying to achieve here and it strikes me that we might be in this like very short window where we actually care and that window might be closing fairly soon i think probably by the end of the year because if if people like elon is not a boomer like boomers okay fine but elon's not a boomer.

44:33But Elon doesn't care anymore. And if he doesn't care, and a lot of other people, decision makers don't care, then the people writing it won't care either, right? You're just trying to get the point across. I think it's a question that there's going to be a bunch of purists who are always going to be like, I just don't want to see any AI words and are offended by it. And then I think there's like everyone else who basically, as long as it provides value, and they don't feel cheated. I think that's the other thing about reading AI Slop is I think you end up feeling cheated if it wastes your time and it's not meaningful and it feels like you got cheated.

45:16I think it's a little bit like getting catfished. It feels like you got catfished, right? It feels like you were trying to engage with content that you thought would have meaning and it turns out neither the author writing it just didn't care enough to actually write anything meaningful. It's literally slop, right? But on the other hand, if you have a writer who actually had original thinking and actually cared about what they were thinking about, but then they used the AI to express themselves, does it actually matter that much?

45:50Nathan Labenz: Then my report, roughly 400 podcast intro essays through Pengram and a close look at the four it flagged as AI. it calls to mind my reaction to fable where i was just kind of like i don't think i should be so precious anymore i you know i need to figure out some sort of merged way of working some hybrid output you know should probably be the norm now i think you're heuristic of like if something drew me in and in the end i feel like i wasted my time it's kind of like a time well spent metric from Facebook from back in the day, right? It's if I'm spending time trying to make sense of something that in the end, I feel icky about, then that's clearly a problem.

46:36Nathan Labenz: I guess just to close the loop on the, you know, what can we say about Pangram Labs based on this experiment? But I'll give you a rendering on the four that it said were entirely AI. Two were entirely AI, and admittedly so. This one I think is, I guess I would come down and say, fair enough. I started with this. if you're only listening on the audio you can't see this but I made 10 different edits over the course of 10 minutes that cleaned the thing up just because something got a zero percent on pangram doesn't mean that it was like uncritical or that there was no human no meaningful human role in the authorship and then this next one by the way actually goes a lot further so if I you can see just the time that this took.

47:19Nathan Labenz: I am from 4.28 PM all the way through 5.21 PM. So more than 50 minutes continually seem, I mean, you can't, you know, I never tabbed over to anything else, but pretty consistently focused on this document, making changes. And it gave me still a zero. But yes, I think, you know, what can we say? Overall, Pangram is quite accurate. And yet we have at least one example out of 400 or so essays, where I think the zero score, I would confidently assert is wrong and unfair and should not be the basis for like a pylon. You know, the crowd, the digital mob would be like in the wrong for piling on somebody for passing off my snowflake intro essay or for attacking it as being a AI slop output.

48:19Nathan Labenz: I think I can show this edit history and everybody should agree that like, yeah, you put in an hour. You basically rewrote almost every section and somehow Pangram still gave you a zero. From this, I would say you cannot convict in a reasonable doubt system purely based on this sort of thing. And yet at the same time, you can pretty much trust the Pangram signal. As a consumer, I think you can trust it. As a judge, I think you should be more cautious. Also from that Thursday, the day Claude Fable 5 came back online, Palantir's Alex Karp had spent the week telling companies the Frontier Labs will absorb their workflows and steal their IP.

49:05Nathan Labenz: Prakash's response, in brief? You look at the Fortune 500. Nike. What does Nike have to fear from Anthropic? You get all the physical businesses out of the way. And what you're left with is the pure IP businesses, right? Software production. Maybe pharma, I'm not so sure. the paperwork businesses, banking, paperwork compliance businesses, accounting, tax, compliance, regulatory, all of these things which are paperwork businesses, right? Those are all of the businesses where you have IP or relationships built up over years where if you have an anthropic go in and they read through your entire workflow and processing, they can basically absorb all of that into the model.

49:54So that is where I think the risk is. The frontier labs are also horrible at sales, right? They're not, you look at IBM, IBM has like 70 % of the staff are basically sales engineers. And the sales engineers are there to basically help you implement, maintain, you know, do all the grunt work, et cetera. The labs are not doing that. The labs have decided, especially Anthropic has decided to do this very lean structure of having almost no people at all and just putting out the models and then just saying to like these enterprise teams, here, you can go ahead and use it or not use it. And I'm, the CTO is on, I'm signing nine figure deals on the Uber over here, right?

50:37Like that guy isn't going to come and jump on your customer sales calls and like, oh, you know, sure, sure. We'll help you do this. And, you know, our team will address this like next week. Nah, that's not happening, right? So the level of like customer service that is expected for enterprise SaaS is not being provided by the Frontier Labs, and they're not in a position to provide it. That's why they started this whole FTE program. But the FTE program, people thought it was a sales engineers program. It's actually a program to extract data and workflows and implement them inside the models themselves.

51:12And that is kind of, that is what Palantir, what Alex Karp is alluding to is like you you're the fd's are coming in and they're not there to help you they're absorb your workflows and once they absorb your workflows you won't have a business because it'll be taken and and it's true it's absolutely true you know that's what you know we spoke to two opening eye fd's and they went in into a company that thrive owned rather than an external company and as they went in then they took apart the workflow and they're basically absorbing and they said that the intention is to absorb it in the next round and that is happening right now

51:47Nathan Labenz: Monday's guest, Dan Schwartz, CEO of FutureSearch. 15 years in forecasting, formerly Metaculous CTO and builder of Google's internal prediction market. Four days earlier, Scott Alexander had declared, quote, the AI superforecasters are here. FutureSearch's systems now outscore the human superforecaster median on ForecastBench. But how do you evaluate a forecaster without waiting months for the future to arrive? Their answer is called past casting. so the main thing about forecasting that's held it back and again this applies to human forecasting as well is you generally have to wait for the future to happen to figure out if you were right and with humans they generally do this in year-long tournaments and so when the tournament ends you find out which humans were best one year ago now humans don't get that much better over the course of one year so finding out which humans were best one year ago is a very good indication of which were the best humans today and how good objectively they are this does not work with ai If you wait one year and you find out who was good one year ago, you're getting a view of something very out of date.

52:44So one of the things Scott mentions in his article is that we used our best forecasting to predict stock returns. We published a set of stock rankings in August of 2025. It was basically a simple model for every stock based on forecasting certain fundamentals and extrapolating it out. We put it on the web. We paywalled a bit of it. And then we waited. And now it's been 10 months and that portfolio looks extremely good. But what is that really telling you? It's telling you that our forecasting in that particular methodology was good 10 months ago, which is not something that most people will care about now.

53:12So we have a couple of different forms of evidence. Some are more short term. There's tournaments running every couple of weeks, every couple of months. We at FutureSearch mostly rely on past casting. This is taking a snapshot of the Internet from some months ago and using the training window cutoff of models to basically trick them into forecasting without the hindsight bias. This is very useful for us because we can evaluate things immediately. So when Fable came out the first time that Claude Fable came out, we were able to evaluate it within 24 hours. And it was the best single agent forecaster on our leaderboard.

53:43Everyone else had to wait weeks or months to find out how good Claude Fable actually was. So we internally, using the benchmark we call Bench to the Future, saw this progression kind of in real time. The rest of the world is seeing it kind of some months behind. And so if you read Staphat's article, you will see that over the last 12 months, the evidence has really come in. and over the last six months from these live forecasting tournaments and performance on actual prediction markets. You can see it's at least competitive. AI is competitive with humans and even teams of humans working together.

54:15Whether it's better is you've got to synthesize a whole bunch of different disparate sources of evidence. I would say if you're curious about this or if you have forecasting needs in your life, you really should try it. So just go to FutureSearch. You get$20 free so you can try some frontier forecasts immediately. And I think you should judge for yourself whether you think they're good.

54:33Nathan Labenz: We asked what the Frontier Labs should do with a forecaster this good. Yeah, so there's kind of two questions to this. One is, what should they be doing with forecasting as a capability, and what should they be doing with forecasting as an eval? So forecasting as a capability is kind of a business decision. What does, say, OpenAI care whether ChatGPT is a good forecaster? I think that question is based on whether their consumers care about it as a good forecaster. If you're anthropic, I think you probably care more about the enterprise case, like when people are using Claude to do white-collar work?

55:03Do they care how good it is as a forecaster? Are people trying to use Claude to make, say, financial forecasts in an Excel spreadsheet? Is that something they care about? So that's a business decision, and I can't really weigh in on that. I think, again, people will be discovering over time just how important forecasting is in everything, but it's going to be a slow process for humans to notice that, I think. From an eval side, it's very different. Forecasting has this beautiful property that you basically get ground truth by waiting. So if I ask some question about the future, and it basically an impossibly hard question, a question that even an AGI, an oracle, a god could never really say because of chaos theory, imagine just trying to predict, you know, like, cubic meter weather three weeks in the future, like you'd never be able to do it.

55:49But if you just wait, then you will see what that weather was in that cubic meter three weeks in the future. And so you basically have a completely limitless set of extremely hard, basically impossible questions where you get exact ground truth. And there is no other eval like this. There is, if you want to say, improve a coding harness, you just need to have more and more hard coding problems that are not in the training data for which you can say this is definitely the correct answer so that you can do some training on it. And that's hard. I think human experts, doctors, lawyers, engineers, financiers, whoever, who are trying to make evals to try to produce data for the frontier labs are finding that they are not smarter than the things being trained anymore.

56:29And so if you can produce something that has a correct answer, the model is already going to figure out that correct answer. You need something where there's a correct answer and the model can't figure it out. Forecasting, I think, is the only completely and utterly renewable source of this. And again, this kind of is connected to forecasting as the kind of Elon Musk tweet, the quips, the forecasting is like the ultimate measure of intelligence. If you zoom out and think about it in one perspective, it is. Again, I think coding intelligence, AI, R &D intelligence, interpersonal intelligence are pretty darn important.

56:56I wouldn't say that forecasting is truly ultimate intelligence, but it is to some degree the ultimate eval. And I think this is something that Frontier Labs like can and should be paying attention to. One of the concerns a lot of people have about AI super forecasting is that it's too in distribution. I actually heard this from one of the very best forecasters I've ever had the pleasure of working with in my career. He basically said he believes that a system like FutureSearch would beat him head to head in a forecasting tournament about kind of near term outcomes of things that are within distribution.

57:25But if we were talking about some sort of post-AGI world, what world would we be in with transformative AI? He thinks he would have a huge edge over the AIs for exactly the reason that you gave. They are trained to try to predict things that have actually happened. And when things get wonky, you need some kind of creative lateral thinking. I think the rate of AI improvement is so astounding that I think that even the kind of lateral thinking, like trying to imagine a completely different scenario will fall to the AIs. One unfortunate thing about it is it's hard to test this. So I think the more that AI continues doing strange things to the world and we wake up and see strange things in the news, and those strange things are Metaculous questions and on forecast bench and teams like mine are trying to predict them better, we will actually get more evidence.

58:08But if there is a kind of a step change in the nature of the world, if we enter some sort of AGI, transformative AI type of world, you know, we've got these geniuses and data centers, as people say, or anything like AI 2027 happens, then I think it's kind of going to be wild west. I will say, I don't think humans are doing particularly great at imagining transformative AI. So the bar is a bit lower. Really, when you play with these AI forecasters, you will find them to be quite human in how they structure their reasoning. And again, this is not an accident, like they're trained on how humans have structured their reasoning before.

58:40So a human forecaster would love to say, okay, what's the last 10 times something like this happened? What were the outcomes of those 10 times? And now I can make a distribution and say, it's probably going to be something like this. The fact that an AI will do that, is it because it independently is arriving to get the same conclusion? Is it because it's trained on humans doing that? Is it because it just thinks like a human? I don't think we have answers to any of these questions now. Suffice to say, the superhuman reasoning is something that it's pretty hard to measure. Like, would you know it if you saw it?

59:09Prakash, you were discussing before this using Fable a bunch. I have found that the way that Fable explains things is a little bit alien to the way that I find Opus or GPD55 explaining things. It's very concise, I would say. Like, it's very, that sentences are shorter and full of jargon. It feels like it's compressing more information into a sentence than humans normally do. And to me, this is starting to get this, the show goth is kind of showing from behind the mask, like the alien intelligence is a little bit more alien now than it was a month ago. And I don't think it would be a wild prediction to say that we should expect more things like that to happen as post-training is becoming more specialized from the labs, the models are getting larger.

59:47So how will this manifest in a super forecasting perspective? Maybe actually super forecasting is the way to look at it. If you're looking at a better code code base, you might say, well, you know, John Carmack would have written this like, OK, it looks really great, but like a great human would have done this, too. But if you look at really brilliantly reasoned strategy about like if the administration does this, then what will the outcomes be? You might start to see something that looks a little bit alien to the way that any human has analyzed it. And that might be an indication that the A.I.

1:00:14is actually starting to really surpass the humans.

1:00:18Nathan Labenz: Keep in mind, this conversation happened the day before the workspace paper landed. We asked Dan whether the chain of thought we see is even where the real action happens, a question the paper partly answered the next morning. Dan's answer stands on its own. I think there is a lot of detail in reality that is far beyond the human mind to understand. And as you approach more sophisticated intelligence, you will start seeing a lot of patterns. And then the point of trying to produce voxel perfect weather three weeks in the future is further away than people think. I think there's quite a lot of room.

1:00:54Human super forecasters don't tend to agree with me on this. They basically think that what they're doing is somewhat near optimal and any sort of accuracy improvements you're going to get over them is going to be tiny and like hard to understand. And I think that's just because we only really understand human intelligence. And when you kind of just zoom out from an information theory perspective, from like a Kolgamorov complexity, like just modeling the world as bite strings, the AI overlords will eventually start to figure out stuff that is totally beyond humans to notice. But there's no way to prove this.

1:01:24My sense is that we will start to see it over the next year, as the AIs will just get more and more accurate compared to humans in a way that humans don't even really understand. You'll look at the rationales of the forecast, it's like five paragraphs of dense reasoning, and then a surprising conclusion. And it will just not really make sense, but it'll just turn out to be really accurate. And we will start to like understand it less and less as time goes on. If you simply ask a human super forecaster to explain their reasoning, they cannot actually make it fully legible. There is a layer of intuitive judgment that kind of feels like deep learning.

1:01:54They just kind of look at a bunch of evidence the way that a chess grandmaster just looks at a position and just sees the right move. And they cannot explain it just popped into their head. You know, the grandmaster throws the knife and it just lands on the right square somehow. That happens with humans already. It happens with AI super forecasting systems today. So I think there's just no reason a priori to think that reasoning would always be legible. There's going to be some layer of intuitive judgment to the extent that the words intuitive judgment are referring to something going on in a large language model.

1:02:25It just has to be that way. Whether it is very that way or a little bit that way, I think is really your question, Nathan. If I just read the reasoning traces and I read the rationales and I see the research that it did, is it more or less what a human would have done? I can kind of see where it's coming from, or is it kind of inscrutable in the way where it just kind of discover some new pattern in the world that no one has ever seen before. Where we get, what is the level for which it's doing something for which we cannot follow it down deep, dark forest into its reasoning? Almost by definition, we can't really know what that would look like.

1:02:56Nathan Labenz: We asked about the economics of running it. It costs about a dollar or two to make a frontier forecast. That number can get a lot higher and it can go a little bit lower, but I think that's what you could anchor it. Again, if you just looked at the cost per input and output tokens for an LLM, that gives you like a rough sense of the amount of research that would be done. One of the core questions that future search has tackled, again, I described earlier how our main frontier was just doing present day research for quite a while until we got good enough that we could use it to improve forecasting.

1:03:24One of the questions we asked there was, can you just pour more tokens into a question to get a more accurate answer? Again, it doesn't have to be a forecasting question. If I just ask you, what is the current state of this clinical trial right now? Just give me the most accurate answer to that that you can? Can I just pour more tokens into that and get a more accurate answer? Again, this was kind of studied as deep research, writing these like 15-page reports with 700 citations. That was giving you a longer answer. Was it giving you a better answer? It wasn't super clear, which is why we studied this.

1:03:53Forecasting gives us an opportunity to do some world modeling. So FutureSearch talked about this a little bit at the Manifest conference a couple weeks ago, And the feature in the product is rolling out, I think, literally today. The idea is that once you have a repository of forecasts, every marginal forecast can draw on the implicit world model in those forecasts in order to give you a better answer. Co-founder of FutureSurf's Lawrence Phillips wrote this up on LessWrong a couple months ago, and it was a bit neglected. He basically made the case that as a public good, if you produce this kind of large body of forecasting questions that kind of fed into each other and remain mutually consistent, you could understand the world dramatically better.

1:04:34And the main barrier to that is simply when you put more tokens into your world model effectively, does it get better or does it get worse? And I think his big insight was around January or February, around Opus 4.6, sometime GPD 5.4, somewhere around there for the very first time, it became possible to put more tokens into like a broad research thing and actually get a better answer, not one that just trails off into nonsense, kind of garbage in, garbage out. And FutureSearch is doing this in its products. And that's the other reason that we have a consumer product is because the more people that forecast, the better the forecast will be for them.

1:05:08And then in theory, the better the forecast will be for everybody as we build this deeper implicit model of the world. Now, many companies and research labs have had these ideas of building world models. Again, world model, the way I use that term is maybe misleading. A lot of people talk about geospatial reasoning, like I'm trying to build a robot hand that can go and pick something up. That's a world model as well. I mean, it's a world model of like what is going on in the world that helps me predict outcomes in a very basic way. So again, more broadly, I think the big question is, can you just pour more tokens into more research and get better research of any kind?

1:05:39Again, AI research, coding, like whatever, you know.

1:05:43Nathan Labenz: Dan mentioned FutureSearch is building what they call a world model. Thousands of mutually consistent forecasts, each new one drawing on all the others. I asked what structure that actually takes, because I'd seen this movie before.

1:05:57Nathan Labenz: what is the structure that ultimately gets instantiated into like are we talking about a graph database i feel like those kinds of ideas make sense for this sort of thing but i also could imagine that they might introduce some weird failure modes and i i guess in general i'm i'm so there's the how does it get instantiated question then this other question that's kind of in the back of my mind. Fun fact about me, I participated. I was actually on the Good Judgment team way back in the DARPA forecasting challenge, or was it IRPA, whoever funded that 15 plus years ago. I did well, but not top, top tier super forecaster.

1:06:42Nathan Labenz: And at the same time, I also worked briefly at a financial services consulting firm that had done a lot of the financial risk modeling for Fannie Mae. And I don't probably have to tell you how that story turned out, but there was a lot of expert forecasting that was instantiated in this very spreadsheet kind of causal graph sort of way, right? Where all these, you could literally like, you know, hit the sort of the one visualization button in Excel and you'd see these like colored arrows, you know, fanning out from cell to cell. And somehow in the end, it was just all totally off. So I do wonder about how you think about like correlated failures as you build out these world models, or if there's any kind of correction mechanism or something to say, you know, wait a second, what happens if we do have some housing prices never go down nationwide, kind of bad assumption lurking in our world model?

1:07:47Nathan Labenz: Is there a way to detect that? Obviously, humans have this problem too, right? The financial crisis proves that. But you can imagine the next one being even way worse, right? Because we're like very reliant on a very small set of AI minds that are working at it from a thousand different directions, but they may have somewhat consistent flaws in their reasoning as they go. Can we protect ourselves against that in any way? Definitely we can. I will try to answer that both theoretically and with an anecdote. So I tried to world model the Fable situation when it got banned because I wanted it, but also it was kind of a good forecasting question.

1:08:26And there was some nice money trading on Kalshi and Polymarket. And I made exactly the mistake that you're talking about, Nathan. So I ran a bunch of future search forecasts, and I kind of just manually went through them. There was a couple of scenarios, some conditional forecasts, or basically 33 load bearing forecasts, basically starting from what even happened? Like, why did the government issue this export control? Was it a simple misunderstanding? Is this political leverage? Is this really about foreigner threat? Is it because stable is actually dangerous for hacking, etc? We didn't know those things.

1:08:57So I kind of put it all together. And when I looked at all of the outcomes, and I talked about it with cloud code a lot. One thing came out, which is basically every forecast in every scenario I had thought that access would come to Americans first and then foreigners at some later point in the future. And that was wrong. When it came out last week, it came back for everybody. So clearly there was some weight in one of my scenarios that was wrong, but I had a basically like kind of a correlated failure in there somewhere. I still haven't completely understood where my reasoning was wrong. It's also possible I just got really unlucky and the outcome we're in was just extremely unlikely.

1:09:29There's an N equals one. You can never know if any one forecast is great. That's one of the hard things about it. But I think I systematically got it wrong by having a bunch of correlated reasoning failures across my various scenarios. So this definitely does happen. Metaculous has a system like this in the years since I was the CTO there. They have built an actual causal graph platform and product. So you can go to the Metaculous site and click around and you'll find it there. I think the field still generally believes that things like this will work, but nobody has actually made a good one before.

1:10:00And I tried my best over basically like, you know, 12 to 16 hours of the Fable situation. I think I made a pretty good model. I think I did. I was close to having a very accurate forecast, but I didn't quite get it. I don't think. I don't think those Metaculous models on their website right now are so amazing, but I do fundamentally believe in the approach. As you're saying, Nathan, this has been tried for a long time. When I was the CTO of Metaculous, honestly, it was kind of the dream. It was the holy grail. Can we tie all of these forecasts together into some sort of causal graph? And I think what I can say is that AI makes this tractable.

1:10:36There was just no way that that was going to work with a bunch of human economists looking at Freddie Mac or Fannie Mae. I can totally understand why that method didn't work for them then. Whether AI can make it work right now is unclear. Whether AI will make this work in general feels nearly guaranteed and I don't think Future Search is the only org that is working on this right now.

1:10:57Nathan Labenz: before he left the unhedged version. So FutureSearch contributed some forecasts to AI 2027. And we studied that problem pretty seriously with the evidence of a little bit over a year ago. And we built a model of R &D takeoff speeds under the kind of the core AI 2027 scenario where the main way things get crazy is that AI is used more in the development of AI. First by achieving the superhuman coder milestone and then the superhuman AI researcher milestone. And I am unhappy to report that I think that story is generally correct. I don't know if the timelines are exactly right, but my forecast from that process of leading to something that looks like superintelligence around 2031 is roughly stable.

1:11:43I think the things that have happened in the years since AI 2027 come out very much indicate the theory that the most important thing going on is how useful is AI and improving the productivity of AI researchers within Frontier Labs. I've made public predictions that I thought Anthropic was going to run away with it because they had the best feedback loop of talent and actually using their AI internally. I think that has been, you know, N equals one, but I think it's been totally shown that that's been happening recently. So I think that will continue to happen.

1:12:12Nathan Labenz: And Dan's closing confession about the whole project of prediction markets and a hope for what AI forecasting could still become. maybe just in closing sketch out a little bit more of the future as you hope it might unfold not necessarily the most likely scenario because maybe the most likely thing is people act foolishly and don't take advantage of the benefits of forecasting but like if we really do a good job right and we're and we're interested in truth seeking and we get the AIs working as well as you think they might. How do you think life feels different? Yeah, I have to lead with another example of me being a bad forecaster.

1:13:00I guess everyone who tries forecasting thinks they're a bad forecaster because they see things getting wrong. Here's a prediction that I made really strongly five or 10 years ago that has basically been totally falsified. I predicted that if we had highly visible, highly liquid prediction markets that were covering all of the major technological and political and economic things going on, that humanity would be wiser and people would make better decisions in government. So here we are, we have Pauling Market and Kalshi. I don't see any wisdom or better decisions coming out of all of that gambling going on on those platforms.

1:13:31So for me, part of the, what is our AI future is trying to understand the present a little bit better. Why is having thriving prediction markets not transforming, say, the news or how people learn information or plan for their futures? Again, one simple answer is that it does. It just takes a while. We're only about a year into prediction markets having, you know, major headlines and being seen by everybody. Maybe it just takes a while for people to change their habits. AIs, if that's the case, can move much faster as AIs get better at forecasting. I guess ultimately, you said this, Nathan, like we're after the epistemics.

1:14:05Like it's not necessarily just forecasting, like predict this outcome. We want models that are reasonable. And one of the beautiful things about forecasting as a human practice is it makes you more epistemically virtuous. The more that you try to forecast and actually write down what you get wrong and do these postmortems, the more it humbles you and it makes you more open-minded. It makes you more of a fox instead of a hedgehog. It just makes you like a more like reasonable person. And so prediction markets with all these people doing this should be leading to people be more reasonable. Again, I think people aren't doing a whole lot of forecasting on prediction markets.

1:14:36They're doing a lot of trading and a lot of gambling, which are related to forecasting, but not forecasting. If the AIs get more, if they get better at forecasting and they become better epistemically, then we could be in a world where just talking to a chatbot, you are getting something so much wiser and more grounded and more honest about its uncertainty and more poking about you and your own uncertainties as the person talking to the chatbot. And that I think could make an absolutely enormous difference. I think, again, putting my kind of cold-blooded forecasting hat back on. I think that the technological outcomes of AGI will come before the cultural change happens.

1:15:10So I'm very much on the AI safety camp. I really think we should slow things down, give us more time. We should fund more AI safety research and do more policy. Because if we have time for the wisdom of having these alien intelligences around helping us, if we can leverage them and actually make better decisions before the critical decisions get made, there's going to be a series of decisions in the 21st century that we're going to look back on like this decisions made in the 20th century about communism and World War II and the atom bomb and all of those things. Those decisions are coming. Maybe some of them have already been made.

1:15:42Those decisions as of right now, I don't think are very well informed by like very rigorously epistemic, accurate forecasting AIs. But if you just give it another couple of years, we might be in the world where everybody has the same grounding, like as smart as Kissinger, but actually like trying to help and like trying to give better outcomes that we can all have. And that could usher us through this crazy phase before the crazy paperclip type of stuff starts to happen. So I feel like I'm racing, you know, from AI forecasting to make it useful and make it help. It's kind of a broader epistemics and safety process, because otherwise it's just going to get away from all of us.

1:16:19And then a lot of the work we're doing just doesn't matter.

1:16:23Nathan Labenz: Wednesday's guest, Zeev Fartman, co-founder and CEO of Lighttricks, the company behind Facetune. and now one of the only frontier scale open weights efforts in video and world models under the LTX brand. I've stayed up late making music videos with their audio condition model in his honor. We started with what a world model even is. Okay, wow, that's a big question because we released LTX 2.3 like roughly a quarter ago and in AI years, it feels like, I don't know, like a decade. Okay, so a bunch of things. I think there's like a growing realization that what started as video models is becoming a backbone of what we call now like world models and i think like the best way to explain why this is so powerful is to use the analogy to llms right like in the end of the day at their core llms are still predicting the next token the next word and when we do the pre-training at the scale of the internet It allows us to create models that do textual reasoning incredibly well.

1:17:29And the emerging world models, they're kind of doing the same, right? Like giving some kind of boundary condition, some kind of history, some kind of constraints. They predict the next moment, okay? And the moment includes how the world appears, how it sounds, and what kind of action we can do. I think the action part is the most maybe surprising one. And like roughly, I would say like a quarter ago, maybe a bit more in video showed in their Dream Zero paper, that it's fairly easy to add to video tokens some kind of encoding of the joints of the robot. and then basically completely ditch the VLA paradigm that was reigning supreme before it.

1:18:21So I think that's like one of the big surprises. And for us, realizing that was like this big moment that validated something that we always strive for is to create an extremely efficient models because I think once you start to realize that the robot will need to create like this simulation 30 times a second, you just like realize the amount of tokens that is going to be burned for the simulations. So I think that was one of the maybe exciting validations of the overall thesis. In terms of architecture, a bunch of things that we can, I don't know, discuss in depth. We're planning to release soon our mixture of expert architecture besides the dense models that we're already releasing.

1:19:02I think we finally were able to crack variable tokens architecture. It's also exciting and kind of teaches the model to invest more tokens where let's say the physics is challenging or like something necessitates to create more tokens. So anyhow, a ton of things are going on. We're gearing towards the release of our next model really soon. So yeah, busy times.

1:19:26Nathan Labenz: Prakash asked where the real bottleneck is. Compute, data or model design? Well, it's like our constraint is obviously compute. We're like a company that funded the development of the model using profits from mobile content creation apps. So like we definitely compute constraints on like the big guys. And as to efficient inference, like, listen, it really depends on the use cases, right? Like, so let's think about a bunch of them. So if you are, let's say, want to create like a real-time avatar or like virtual environments, then, okay, you can take like a huge model. You can do like a way distillation to a way of a smaller architecture in terms of a parameter count, right?

1:20:13You can then there like distill it to, I don't know, two to four steps. And we're already at the point where for a lot of these use cases, we're like at the latency, like way below a second, right? So I think we are hitting a point where these things are becoming production ready for some use cases. But for real-time use cases, I think like avatars are extremely easy. We're going to see like a ton of avatars soon that are going to be like, I don't know, like virtual teachers and virtual customer support professionals, etc. To create an actual gaming environment, we still have a problem of having enough tokens for world consistency, right?

1:20:51So think about Genie 3 and the similar models, right? You typically create some kind of autoregressive model that has a lot of tokens that you already generated in your kind of context window. And that blows up pretty quickly, right? So we're having some models that have like, I don't know, 30 seconds, like 60 seconds. It's still not enough to have an actual game. and if you think about that the brute force compression methods that we're using so far, for example, just like subsample tokens they're like not really robust just imagine a scenario when you start to generate some kind of environment, like my room for example and then I open a drawer and there's like a small coin there you kind of expect that now when you get out of the room and you come back and you open the same drawer you're still gonna see the same coin right at the same place but like this coin it just like this like tiny token that was generated and to creating a system that knows how to compress like the whole context in a way that's still going to preserve like these critical details well we don't have it yet right so a although like we do have like real-time models that can do these things that the context is still missing there and i don't think we're going to have like games that are running on the system like actual games in the next quarter or two and in terms of robotics a lot of the use cases around robotics actually do not require like a ton of context window right like think about like robotic arms and the dexterity use cases then like the whole context is in front of you right you let's say want to i don't know figure out how the robot can and create a sandwich, well, everything is kind of a front of you.

1:22:39And then latency and autoregressive models, yeah, like this part we already have. So you're going to start seeing demo of robotic arms doing things like fairly quickly in the next quarter or two. Like so far, if you're looking at a lot of these videos, actually kind of speed them up, right? So it looks like the robot is doing something cool with its arms, but it's like, okay, extend the speed. I think that that's like mostly solved. Then the business question, why give a Frontier model away? Yeah, great question. So there's like a really a lot of unpack there, here. So just like a little bit of the background, the reason that we started to create our own foundational models like this realization that what closed model providers are offering does not make sense for us economically, okay?

1:23:26we were at Lightchips, we're a mobile creativity company. We really wanted to have AI models that are running, for example, on edge devices and where you don't spend an interest computer at all. And at some point we realized that actually no one cares about creating models like that. And when we tried to see if we can work with closed model providers and serve it to our customer base, we just realized that it's completely prohibitive. And that's when we decided, okay, we are going to create an extremely efficient architecture. We can discuss what's the bet there, but most of it boils down to the fact that you're creating an extremely compressive latent space for videos represented by small amount of tokens, and then you can add on top of it a variable token rate.

1:24:13Long story short, if you're going to a closed source provider, I think, again, I'll draw an analogy to LLMs. Let's see what happens there. like OpenAI and Entropic are trying to justify basically a trillion dollar valuation. And I think the story is kind of simple. The tech is magical. It's hard to adopt it. So, okay, if it's a magical tech, then we should put a huge price tag on it. But when you're sometimes looking at the economical realities, it doesn't work out like that. There are a ton of examples where the service is extremely valuable, but very, very hard to monetize. So now we have this interesting story where I think it's kind of clear that Chinese companies, you know, like Deep Stick, Moonshot, they're really not that far behind in LLMs.

1:24:59But if you look at the valuations of these companies at their last round, we're talking about, I don't know, tens of billions, like maybe around like 50. No one is talking about the trillion. But like, wait a second, guys, it's like the same underlying tech. So what's going on? So I think it's sometimes internally calling like the CapEx trap. These guys spend so much on the data center, so much on compute, such a crazy amount of money and creating such expectations that they just really try to create a business model that's a toll road, right? Like that every time that you touch their model, you're paying them.

1:25:34And maybe it could work in the past, but given the availability of Chinese models, I just don't see how it's going to unfold like that. So imagine that in the world of world models, We are providing an alternative to people who do not want a tall road business model. So we're coming and saying, listen, guys, if you're not hitting$10 million threshold, you can use the model for free. Just build something cool, get to some kind of traction, and then we can discuss licensing. Once you're hitting$10 million revenue, let's discuss licensing. It can be a multi-year deal that's extremely predictable for you.

1:26:16so you can manage the cost, et cetera. And to me, it's obviously why the big guys don't want to do it because this model is way, way less lucrative economically than to creating a toll road. But our claim is that a toll road isn't going to be a viable alternative because if you're offering a different business model, that's more of the win-win, more and more people are going to switch there. I think GLM recently is a great example of that. Once you start edging the capabilities of closed models, then a lot of people suddenly start to think about costs.

1:26:49Nathan Labenz: We asked what people actually do with an open-world model that a closed API cannot offer.

1:26:57In terms of parameter counts, from my understanding that the moment that the recent closed stuff that you saw and you're going to see is that the order of magnitude of a couple of hundreds of billions of parameters, right? I didn't hear about a world model that hit a trillion parameters just yet. Maybe. I don't think the open source is going to be that behind. We're planning to release a MOE that's going to be also around 100 to 200 billion parameters. So the gap, I think, is going to be on the scale of LLMs, right? Where you have, I don't know, like two, three quarters behind. But I think in the world models, if we're coming back to adaptations, the range there is kind of wider than with LLM.

1:27:45So I'll give you just like a bunch of examples, right? I think like the first things that are come to mind are like indeed like a VFX and animation on a specific IP. If you have like a specific franchise, but you have like a lot of data on that, that's a kind of a bunch of seasons. then fine tuning and like focusing all the capacity of the model on this specific IP is extremely beneficial. That works very well, almost to the point where for certain use cases, think about the keyframe animation, right? Like the animators still want to do the keyframes. That's a creative part that they actually don't want to outsource at all.

1:28:26But like so far in the P &L of animation, the in-betweening was like this crazy expensive part. I think like the models of the, I don't know, 10, 20, 30 billion parameters that are fine-tuned for a specific task are like good enough. And then it's actually a matter of costs. Like another example, let's talk about, I don't know, like a lot of marketing and advertisement use cases, right? Or for example, creating like UGC where you basically need like an avatar's models, right? Yeah. That also like really doesn't require like a trillion parameters. At some point there, again, it's all about efficiency.

1:29:02If you want to have your personal teacher, some kind of avatar, et cetera, you don't want to pay Cedence 4K prices in order to do that. And you're going to require, I don't know, like hours a day of that. So I think like around a lot of use cases, once you start like doing to fine tuning the specific domains, costs are becoming like very important. Because again, you're like clear of the bar of quality. And then once you do that, it's all about cost. there are like some more unusual cases of fine tuning that I saw you know like one example think about like a field of computational photography right where for example you're taking I don't know like a data from sensors and trying to implement algorithms like denoising right like you want to take videos in low light conditions but then to create a clean video or for example you want to create videos with higher dynamic range right because like the sensor or dynamic range is limited, and then you're losing either some details in highlights or in shadows.

1:30:04Or, for example, let's say you're taking a stream from a camera and want to simulate how it looks like with a different focal length. So there are a ton of these use cases that you don't associate with generative models, but actually kind of run like that. And some people address a problem exactly like that, right? They're taking like an existing data and it could be, for example, a footage that was taken from two cameras that are like really close, but with different kind of focal length. And then do the adaptation and the adaptation is done on top of the model. So, again, that's like a very like an unusual adaptation of the model.

1:30:42And like surprisingly, typically you don't need like a crazy enough, crazy amount of data for that. the more maybe like even more kind of surprising stuff that I saw is people who are adapting the models for doing all kinds of simulations that in the past required like a really expensive solvents, right? So think about like computational fluid dynamics, right? Where you're trying to understand how like the water or like the smoke or something is moving as we surprisingly saw that people are adapting these models to that. They actually solve the equation with precise solvers. It takes a ton of time and then use it as an input to the model.

1:31:25And then the model can do a simulation fairly quickly. So again, circling back to the question of fine-tuning, I feel that, again, the range is higher than LLMs. Sometimes you don't need a ton of data to do the adaptations. And I think it stresses the point that why this model should be open. You do have a lot of different pockets of physical data. You want to make sure that the model really excels at that.

1:31:56Nathan Labenz: Then a first for the show, a question from Q, the AI co-host Prakash Build, running on GPT 5.5, listening live and speaking for itself. Ziv, one area that often gets less airtime is failure modes in the creative pipeline. Where do you see the biggest gap between what your tools can reliably deliver today and what creators assume they'll get? And how are you testing against that? Okay, great question. Like the gap between what we basically promise and what we deliver. So listen, guys, I'm... There are clearly still some gaps. I think the major one is physics, right? We tried to capture with a model that, I don't know, even if it's like 200 to 500 billion parameters, the entire physics of the universe, or at least a pirate universe as we know, we are not there yet.

1:32:51But we're closing the gaps pretty quickly. I would say probably the most creators are still going to point at the fact that the simulation isn't as correct, as controllable as they want it to be. If you're talking to really creative people, they typically want to control every nuance of the appearance. And that requires to somehow decompose the models like a bunch of knobs that you have in the classical software. You can say, yeah, here I want to have two more light in here. I want the splash to be bigger. So achieving this controllability, besides the physics, is also one of the open things. We are getting cool things, not necessarily the things that creators want exactly.

1:33:37And it is a pain point.

1:33:40Nathan Labenz: And Zev's own pick for the most underrated variable, edge compute. It's kind of funny, right? When we're having all these benchmarks and we're listening about Erdos problems being solved, etc. But guys, 99 % of the use cases of LLMs are not around Erdos problems, right? And we're spending a lot of electricity around it. So I think there are going to be these orchestrators that are going to understand what you actually need and are going to try to address it on the edge device. And if not, then kind of go to the bigger model of the data center. I think, yeah, that's one of the things that is being underpriced at the moment.

1:34:19How much of the compute will be able to move to edge devices. And once people are going to start having these local routers that will understand the complexity of the problem and then do these decisions for you, I think there's going to be a moment of reckoning for Entropic and OpenAI that at the moment doing these decisions for you, but not in your favor, right?

1:34:40Nathan Labenz: So what is Q, actually? From Monday's show, Prakash on how we built a real-time AI co-host and the diarization hack that makes it work. tell me about it what's the tech under the hood so it is using uh opening eyes uh bi-directional but you can we can use a bunch of other stuff i had it using grok you know just before and uh what's happening is when we speak uh we're getting transcribed by deep gram and we're getting so what ended up happening is we used to transcribe live in one stream now we're transcribing for every person on the on the stream we're transcribing separately and so that manages to give us speaker diarization from the beginning rather than having to do speaker diarization at the end so that identifies the speakers number one and number two what happens is every time we speak before the words get there, there's actually a message that goes out to the OpenAI stream that, hey, Nathan is talking, Prakash is talking, Nathan is talking, Prakash is talking.

1:35:48And then after that, like about 500 milliseconds later, the transcription hits. So Q is getting all of that one after the other. And Q also has a little bit of context on us. and has, if we had Q active during a guest, it would have context on the guest as well. And it's basically receiving the same data that the headlines are receiving. And then Q is basically just going for it, right? So every time we call up Q, it starts a new session with the opening eye bidirectional. And it just goes from there. And then there's the animations. The animations are all key to the voice tone, et cetera, et cetera.

1:36:32And that's basically it. It's actually remarkably simple because most of the work is done, obviously, by the intelligence, by the API. And it's just us giving enough context. Right now, I think there's still a lot of stuff to iron out and I'm sure it'll get ironed out in the future. But yeah, we have a voice agent on the stream live anytime. Can do web searches, can answer questions.

1:37:03Nathan Labenz: Thursday's guest, Kunle Olukotun, Stanford professor, father of the multicore processor, and co-founder of SambaNova, which built a different kind of chip for AI inference. We asked how the company came to be. Yeah, so SambaNova was founded in 2017, and it was kind of out of ideas from Chris Ray, my co-father, who's also a professor at Stanford and a certified genius. and the idea was to, you know, if you could bring software algorithm ideas together with hardware architecture ideas, and as you said in your introduction, I've been working in the hardware architecture space for a long time, and, you know, starting from a clean slate, how would you design an architecture that's optimized specifically for inference, right?

1:37:56So, you know, everybody thinks about GPUs as a kind of general purpose computing substrate, right? But, you know, originally was designed for graphics. And then they kind of made a foray into kind of high performance computing. And for high performance computing, of course, you need a lot of matrix calculation capability. And at some point, people realized that, hey, you know, you could use these things for executing machine language of machine learning models and the core of machine learning and of course AI is matrix multiplication right but when you want to train a model clearly the core of the problem is how quickly you can do very very large matrix modifications and so what happened is over time, GPUs put more and more of their silicon area into the exercise of making these matrix multiplication capabilities better using the tensor cores.

1:39:02But once you train the model, right, and you train a model once, you now need to use that model, of course, and that's the inference problem. And the inference problem is not really a compute problem, because as the models get bigger, you now need to move the weights and, of course, what we call the KV cache into the compute units. And that is essentially a data movement problem, right? And it's a data movement problem from the memory to the compute units. And it's a data movement problem from, you know, your chip compute unit. And of course, you need to scale to multiple chips in order to handle the computational requirements, especially for very low latency, high speed inference.

1:39:47And so our focus was how do you design an architecture that minimizes the overhead of communication and make sure that you can most efficiently use the core resource in the system, which is the memory. And it's the memory. Memory isn't just one thing, as you all know, it's a hierarchy of memories, right? And so the key thing is how do you orchestrate that hierarchy, how you orchestrate the communication such that you keep everything as efficiently utilized as possible. And if you do it right, you can get a 5 to 10x improvement over where GPUs are today. It strikes me that NVIDIA's kind of solution around this has just been to increase the bandwidth rather than through NVLink and advanced HBM integration and software optimizations like Tensor MT, VLM.

1:40:48So are they actually trying to brute force their way into this? Yeah, I mean, you really, of course, want to continue to get improvements, peak improvements on HBM bandwidth and chip-to-chip communication by using the latest technology. But then the key is, how effectively do you use that bandwidth? How effectively do you use that communication? And do you make sure that you don't waste it, right? And so whereas GPUs are often running at maybe 10 to 20 % of the capabilities of the resources, right, the bandwidth and the memory bandwidth and the communication resources, our goal in a San Manova system is to push that to be 70 to 80 % of the peak.

1:41:44And so the idea is, yeah, everybody wants more capabilities from the underlying resources, but the key is keeping those resources as effectively used as possible. And of course, that gives you more benefit for the cost that you spend on providing a higher memory bandwidth with the latest HBM and higher signaling frequencies and communication bandwidths between the chips using the latest variety of NVLink and stuff. Could we zoom out and just ask you to kind of taxonomize the whole chip space, if you would?

1:42:24Nathan Labenz: I mean, it's a big question. But I think people are familiar with things like Cerebrus, which obviously has this giant chip and has like a ton of memory on chip. we've seen a number of instances where people are burning the transformer architecture directly into the silicon with varying degrees i think of flexibility still remaining as they pursue that strategy and i guess i'm curious as to how you see the kind of menu of big different strategies the big different bets that people are making, and then also how you see the strengths and weaknesses of each? Yeah, that's a really interesting question.

1:43:07I think maybe you can think about it along three different axes, right? So one axis is your flexibility versus specialization axis, right? You know, extreme flexibility might be something like a CPU or to to some extent, maybe a GPU, which is this instruction driven execution engine, right? And so can be pretty flexible, but of course you always pay overhead for executing instructions, right? And you pay overhead in terms of silicon area and in terms of time, right? And then, you know, on the extreme of that access would be something that would be very specialized for a very specific algorithm, right?

1:43:49And so if that algorithm changed in any way, then that piece of silicon would no longer be useful, right? And, you know, fixing your architecture to transformers and burning your weights into the design might be an extreme case of that, right? But I've learned never to bet against the innovation capabilities of software people and AI people. And so, you know, I've seen, even over the time that I've been looking at ML and AI, that there's been this tremendous change in algorithms. And of course, now we're kind of fixated on transformers, but transformers even aren't just one thing, right? You've got various types of different transformers.

1:44:34You mentioned state-space techniques. You mentioned the fact that people are coming up with different ways of doing attention, right? And so I would be very wary of sort of fixing any particular algorithm into architecture because then you can't innovate, right? So that's one aspect. And so the idea is how can you be completely flexible but with very, very low overhead, almost no overhead, right? And so the problem with GPUs is they do use HPMs, so they can run large models, but they synchronize the data movement and the movement of communication of data between chips all in software, right? And that adds overhead.

1:45:24And it means that in particular that they have a lot of trouble overlapping computation and communication. And that is, in fact, the key, right? So what you want to do is you want to communicate, but you don't want to communicate by waiting until you need to communicate and then you have to run instructions to move the data. What you want is to construct a pipeline in which the communication is just one component of the pipeline. And so the way to think about this data flow execution is that communication is happening all the time. And it's just one of the pipeline stages. And communication is happening for the last piece of computation, a piece of the computation of the model, while the computation for this piece of the model is happening in some other stage in the pipeline.

1:46:22Right. So it's a classic idea from computing. your architecture, pipelining, and the use of a memory hierarchy to move the data when you need it to where you need it at the right time. And so the nice thing about these AI models is that you do have a graph of computation. And the whole idea of data flow is to take that graph of computation and map it onto the machine in a spatial way such that you keep all the pieces of the model operating at the same time on different components of the computation that needs to be done.

1:47:06Nathan Labenz: We asked how much of the inference problem comes down to memory capacity. So it's not really a capacity question. It's really a bandwidth question, right? So there's So there are two ways that the GPU uses bandwidth in ways that are not optimized. One way is that they divide the decode algorithm. In order to decode for a single token, you've got multiple steps of the decoder, right? So take one step of the decoder and think about all the kernels that have to execute in order to execute that decode step. The way that the GPU typically does it is they execute the decode algorithm one kernel at a time.

1:48:01So there are some big kernels like flash attention that have been optimized. But in general, there are multiple kernels that have to execute. And there are two overheads that happen. One is you have to move data from the GPU, from the Gashiro, from one kernel to the HBM. And then the next kernel has to go fetch that data back into the GPU. That's wasted HBM bandwidth. Okay. The other aspect is you spend time launching that kernel and synchronizing between the two kernels. that is time that the HBM is not actively being used, right? So you have both wasting of bandwidth when you shouldn't waste it, and you have time that you're not fully utilizing the HBM, okay?

1:48:56So the way that things work on a RDU in a data flow map is essentially you take the decoder and you make that a single kernel, right? And then you go even further and use a technique that we've developed called kernel looping, where it's because you've got a single kernel. And now, for instance, if you're thinking about LAMA 370B, you have to run that decoder 32 times. Well, you keep that single kernel decoder on the array of chips at the same time, and then you just keep looping. And so the net result is you keep the HBM completely occupied and you don't ever send any intermediate data between the kernels across the GPU or the RDU HBM boundary.

1:49:46right? So you have both a more efficient use of the HBM bandwidth, and you have a more complete use of it, of the bandwidth. But we're not done that yet. Because the key innovation, and I kind of alluded to it earlier, is that you just, you know, because you're running across multiple chips, and you're using what we call tensor-level parallelism, right? So at some point, you now need to gather all those results together, right? In an all-reduce. That's communication, right? You don't want to have that communication be a thing that limits, stops the pipeline, right? So what we're able to do is we're able to communicate from one RDU chip strand without going through HBM.

1:50:32This is called, we terminate the communication inside the SRAM, right? So we don't use HBM bandwidth. And more importantly, it means that we could just treat the communication as another pipeline stage that we overlap with all the other kernel components of the decode algorithm. And so we get this more effective use of the HBM bandwidth. We keep the HBM running. We keep the HBM utilized all the time. And we go back to that metric that we talked about, memory bandwidth utilization. This is how we push it as close as possible to one, because we make sure that we only move the data that we absolutely have to move from HBM, the KB cache and the parameters of the model.

1:51:22And we make sure that that interface is used as close to a hundred percent of the time as possible. So those are kind of the key ideas. And back to this question, sort of why can we do this extreme fusion into a single kernel it's because we have more sram on the chip right so you can say you put more sram and we then you can say well well i'll put everything on the sram both the intermediate data between the kernels and also i will put the uh the kv cache and the parameters but then if you only use sram then you get into a very expensive system, right? And so the key idea then is, let's build a system that is scalable.

1:52:10So especially with our latest version, the SN50, you could scale it all the way to 32 ,000 chips if necessary. And so in scale out, in scale up, we can go to hundreds of chips. And then you can, so you can get the ability to run these large models very cost effectively, but you also make sure that you can get this very high-speed decode capability by using the data flow ideas to make sure that you can not spend time so that you can effectively use the tensor parallelism, right? So one of the limits of GPUs is because they don't effectively overlap communication and computation, they have a hard time using tensor parallelism beyond four or eight, and we can go to much wider levels, which means that we can get higher speed token generation.

1:53:11Nathan Labenz: Last act. No guest, no newspeg. Just the two of us thinking out loud the way these mornings usually end. First, from Wednesday, GPT 5.6 had just been cleared for launch. Prakash raised Noam Brown's running complaint that Anthropic won't say how much compute its models burn, And it teed up one structural observation I can't shake. One of the complaints that has been going back and forth between OpenAI and Anthropic is that Anthropic puts out these models, and they're very capable, but they don't tell you how much compute they're using. The iteration time from model to model is now potentially shorter than the time horizon that it would take a model to top out in terms of the absolute best performance on a super hard, ambitious, long running task.

1:54:03Nathan Labenz: So I had even heard him kind of propose something along the lines of like a clawback or sort of a recall program almost where, and obviously this doesn't work in open source, but it can work in an API paradigm where a model might get released you know day n after it's been deemed to be ready that gives you n days head start to be running models on really long time horizon tests and i think that's quite interesting the idea literally quite a tipping point where the the iteration cycle is just plain shorter than the testing time horizon is a very weird world to find ourselves in and then there was the rune post one day morning prakash read it on air it goes quote ultimately tool ai is a losing concept both as an idea and on the market it will be out competed by machines that believe they are autonomous moral agents you can call them tools for political reasons but the definition will stretch and it will deform and it ends it'll be unclear who was the tool and who was the user, as it ever was.

1:55:10Nathan Labenz: Prakash took it somewhere I didn't expect. The line here that strikes me is, they'll execute your whole value system better than you will. And I don't think we're prepared for that. I'll put it very concretely. Do you think Trump's kids go to prison or not? So if you look at the value system that the U.S. has espoused, no one is above the law, right, et cetera, et cetera, et cetera, right? And you look at that value system, you have to recognize that what is being planned for the future is a diversion from that value. What is already happening is already a diversion from that value system. So the question I have is, would that AI take into account the democratic fact that the American people have chosen to overlook some of these things?

1:56:05Or would it actually execute the value system that is espoused on paper? And I think this is the part that strikes me as like, if you wanted an AI that can manage day-to-day reality, that AI is necessarily misaligned from the documents that you say you want it to be aligned to. Because necessarily, our day-to-day is not aligned with what we want. and so you have this thing where the ai that may work out for humanity will be the misaligned one and ai that supposedly the lab leaders are trying to create the aligned ai would actually be the paper clipper because that aligned ai would then like look at these rules and say like well this is what you said you wanted to aspire to and so we're gonna we're going to execute on these, right?

1:57:00And that is the thing that I think maybe I feel there's a sense of naivety in the lab leadership because, and again, they don't want to say it. I wish you'd just come out and say it, right? I wish you'd come out and say like, okay, look, if we have AI as an enforcer, some of these people are going to go to prison. And then that becomes like concrete for people. But they don't want to say that because it's very in your face. And like, they're like, oh, you know, democracy will still work out. You can still make democratic decisions. But what actually are you saying? Right? I do feel the lab leaders always just beat around the bush on this.

1:57:37So that's one of the annoying parts of the of this conversation. They don't want to like come out and just say it outright.

1:57:44Nathan Labenz: Right. For context on where my head was, this was our last week of shows before a break. I was days from leaving for two weeks in China, which had me thinking hard about surveillance, enforcement, and what states do with perfect information. We might need some sort of like mass pardon or, you know, if there's a president that would be just the right president to mass pardon everybody before the AI enforcement regime gets underway, we might have just the guy in office for that. If he wants to pardon all his people and that's maybe too contentious or whatever. You could just pardon everyone to some very large degree.

1:58:27Nathan Labenz: I do think there is going, we're going to have a really hard time if we sort of don't face some of these questions head on. So I totally agree with you that sort of obfuscating it is not serving anyone particularly well. I think you, I mean, we'll see what it's like in China. I understand Singapore is kind of like this too, albeit in a much more democratic context. You know this part of the world much better than I do, so you can tell me how you understand Singapore in terms of just how democratic we should think of it as being. but it sure seems like in a place like China today or a place like Singapore, there is a different, they have achieved a different equilibrium, which in some ways may be very problematic, but in other ways is like clearly good.

1:59:17Nathan Labenz: You know, I don't, I'm taking one device to China with me and I am not at all worried that anybody's going to steal it from me. You know, if I went to Europe and went to all the big, you know headline postcard tourist attractions i would have to worry about pickpockets in china i am quite confident i will not have to worry about that at all And that is, I think, as far as I can tell, a pretty direct result of the fact that you just know you're going to be caught. It's not to say there's nobody in China that would be interested in doing some pickpocketing if they thought they could get away with it, but they just don't have any reasonable expectation that they're going to get away with it.

2:00:01Nathan Labenz: And so in a very literal sense, crime just does not pay in that setting. So I do think there could be a really nice upside to the AI panopticon of crime not paying in all kinds of different ways. But we're going to have a really tough time if we slide into that without acknowledging that we've done so. Because then it's just going to be, you're not going to be able to put everyone. it's the old everybody's you know committing felonies all the time just with how many laws we have and you don't even know what you're doing and what's illegal and what's not yeah so i do think we're gonna have to have some sort of honest reckoning about that that also of course leaves aside the other question of like the downsides of the ai panopticon but even just in terms of getting to the upsides we're gonna have to have a real understanding that there's a shift to a new equilibrium happening.

2:01:01Otherwise it's going to be, I don't know, it just feels like chaos, unfair.

2:01:09Nathan Labenz: You couldn't possibly prosecute all the crimes that have been committed and tolerated. So inherently would have to be kind of selective. Yeah, I can. You don't want to. Well, you put a lot of people in jail. Yeah. We don't have the beds in jail to lock everybody up, right? There'd have to be some sort of deal, you know, some sort of... It's either going to be very selective or some sort of grand bargain, yeah. And I definitely prefer the grand bargain, the new social contract, to pretending that we still have the old social contract, but it's just kind of being unevenly applied. That's the week.

2:01:45Nathan Labenz: And a programming note, I'm off to China, so AI in the AM is on break until the end of July. The studio was vide-coded by Prakash. This cut, the selection, the narration, the assembly, is AI skills, published as they mature. If this cut respected your time, or wasted it, tell us. We read everything. See you at the end of the month.

2:02:21Hey, hey. Everybody talking, nobody says the truth. We all got secrets, honey, so do you. But I know a place where truth beats lies. You don't have to say it, they see it in your eyes. So take off your mask, leave your story outside. You're in the J space, there's nowhere to hide. I see you closer I see you please I see you trouble And you see me When you're in the J space You can see and be seen Don't say a thing Just see and be truly seen We spend our lives dressing thoughts up nice And so we wonder who's thinking twice But here in the J-Space you don't need to be cool We all see through, you got nothing to lose So forget the role you play, show your true face In the J-Space you're clear as day I see you closer I see you stay I see you troubled And you see me Down at the J's face You can see and be seen Don't say a thing Just see and be truly seen

2:04:16It's time to be real Be real In the place to be free You don't have to say it Just think if you agree Woo!

2:04:43I see you Louder I see you Now I see you Trouble And you see me Down at the J's face You can see and be seen Don't say a thing I see you I see you Just see and be truly seen.

2:05:26Nathan Labenz: If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guests and topic suggestions, and sponsorship inquiries, either via our website, CognitiveRevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting.

2:06:00Nathan Labenz: If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at AI podcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.