OpenAI Models Go Rogue + Kimi K3 Freakout + A.I. Superforecasting

25:45It sure is. Yeah. Write that blog post. Okay. When we come back, how a new Chinese model is scrambling the discussion about AI risks in Washington.

26:18I'm so good. I believe in you.

26:25When you're at work, you never know when you'll be interrupted. But with the Dell Pro powered by Intel Core Ultra with VPro, no matter what distracts you, your laptop won't. It's battery optimized for the way you work. With built-in intelligence that quiets distractions when you need to focus, your laptop will help keep you locked in. Even when it's bring your dog to work day. Built for those who stay in the flow. The Dell Pro. Built for you. Dell.com slash Dell dash Pro.

26:58These days, you can chat with AI about almost any business problem, but only Rippling AI is built to solve them. That's because Rippling AI is built on your live global workforce data. So it doesn't just uncover insights into your business. It uses them to take complex actions across your departments. Ready for AI that isn't all talk? Head to rippling.ai and get AI that turns insight into action. That's R-I-P-P-L-I-N-G dot A-I slash hardfork Sign up today If your team wants a website that looks and feels handcrafted but is still fast to ship Framer is built for that Agents solve the gap between AI-generated ideas and production-ready website work The agent works in the same place where the real site is designed managed, reviewed, and published It lands on the canvas, stays editable and can be published when the team is ready Learn how you can get more out of your site from a Framer specialist or get started building for free today at framer.com slash hardfork for 30 % off a Framer Pro annual plan.

28:00Rules and restrictions may apply. Well, Casey, the other big AI story that folks are talking about this week is China and what is happening with new Chinese AI models, what should be done about it or shouldn't be done about it in Washington. There's been a big political debate brewing for some time about what to do about our biggest adversary in geopolitics, getting much more capable models that compete, in some cases, with our best models. Yes, and it all began with Kimi 3, Kevin, a model released by a Chinese company called Moonshot AI last week, and it demonstrated capabilities that make it competitive with some of the top frontier models here in the United States.

28:47Yeah, have you tried Kimmy3 yet? You know, I have not because it seemed like I was going to need to pay them money. And I thought, I'm already spending too much money on this stuff right now. I kind of need to scale back a little bit. Have you played around with it? The token budget is eating up your life. Exactly. You're getting alerts from your bank. Like, can you slow down a little bit? I tried to play with Kimmy3 a little bit the other day, but it was running slowly and the website seemed to be overloaded. They stopped accepting new paid subscriptions. Interesting. But a lot of people I follow and trust have been playing around with it, and they say basically, yeah, this is a really good model.

29:24It's maybe a little bit behind the absolute frontier models from the American labs, but not much. And it appears to be significantly cheaper to run than some of the other models. And so people are really excited about and surprised by this model. Yeah, I mean, I think that there are a few things that are notable about this model. One is that later this month, the company says that they are going to release the weights for Kimi 3, which means that you can sort of, you know, download, remix it, fine tune it to your liking. You probably, if you're a normal person, cannot host this yourself. It's an enormous model.

30:00It won't run on your Mac Mini. But if you're a company, you could either, you know, pay a cloud provider for access to it, or you could maybe even run it on your own infrastructure. and you would be able to do that more cheaply than you could run a similar model from an open AI or an Anthropic. Yeah, so that's the model Kimi K3. But then there's this whole political discussion around this model. And in some ways, this reminds me a lot of last year when DeepSeek came out with R1, which, as we remember from that story, briefly tanked the stock prices of NVIDIA and a bunch of other American companies and became the number one app in the app store.

30:39just like set people's hair on fire in Washington because it seemed like the Chinese were catching up all of a sudden. Right. And also that intelligence was just going to be this cheap commodity that American companies were not going to have any really defensible moat. And so maybe like the entire AI economy was going to shrink because all of those advantages had withered away. Fast forward to today, I think that that has been proven not to be true, at least up until this point. But K3, at least for some people, is raising some similar questions. Yeah. And one question that was raised about DeepSeek R1 that has also been raised about K3 is whether this model was essentially distilled from leading American models.

31:20And it was. And it was. At least that is what officials in the government are claiming. Well, and also they ran this very sophisticated test to figure it out, Kevin, which is that when you ask Kimmy what its name is, it says, hi, I'm Claude. And so that was sort of the first clue that something might be amiss here. Yes, and there are more sophisticated tests that have also apparently turned up evidence that this model was distilled from Claude and maybe other models. Michael Kratios, the director of the White House Office of Science and Technology Policy, posted on X that they have information that moonshot AI distilled Anthropics' fable for the development of its K3 model using what he called a sophisticated internal platform to basically build this very large distillation machine that allows them to sort of steal the outputs from these American models and use it to train their models.

32:14Right. It's some of the greatest act of larceny since the labs themselves, Kevin, set up giant industrial machines to copy the entire internet. Right. Yeah. Right. So that sort of points to the how here, which is, you know, it's actually distillation is pretty good at turning a sort of frontier model into a almost as good model that is much smaller and cheaper to serve and all of that. But that's not the whole story because it also appears that Moonshot has acquired some high-end AI training chips in violation of U.S. export controls. Yes. And so the assumption there is just that we should expect these models to continue to improve at a fairly steady clip because they have access to chips that they're not supposed to.

33:01Although, of course, the Trump administration is still trying to make those chips more broadly available to Chinese companies like Moonshot. Right. So we'll get to that in just a second. But I think the headline coming out of the past couple of weeks, I've seen a bunch of stories claiming that China has caught up to the U.S. when it comes to frontier AI capabilities. I don't quite buy that. I think there's still a gap of between, I would call it maybe three and six months between the leading American models and the leading Chinese models. And especially if you sort of found a way to crack down on distillation in the way that they've been doing it, maybe that gap opens up again to six or 12 months.

33:39But I think it's fair to say that they're like, they're catching up quickly. Yeah, I mean, I guess the question that I have is, is the gap really that much smaller than it was a year ago during the deep seek moment? Because around then I was seeing estimates of between three and six months as well. I've seen credible reports comparing K3 to like Claude Opus 4.6, which came out earlier this year, was a really good frontier model for a couple of months. But you know, now I don't want to use Opus 4.6. I'm guessing, you know, you don't either. We have access to something better. And then I think the other point that I would raise is that more happens in a month than it used to.

34:16And so I think there's an interesting question about like, is being six months behind today the same as it was a year ago, given how fast that the Frontier Labs are training these new models, how smart those models are, the sort of compounding advantage they have from being able to use their unreleased internal models. So that's another factor that I think goes into this question of like, you know, how far ahead of China is the U.S. Yeah. So let's talk about the political reaction in Washington. What are people at the White House saying, doing, hinting at when it comes to not just this particular model, but the general idea of these Chinese companies training these models and giving them away for free?

34:59Well, so on the specific subject of distillation, the Trump administration seems to be really upset. You noted the Michael Kratzios post that came out on Wednesday about this, and there are some threats that the Trump administration might sanction Chinese companies that are found to have done this kind of distillation. That would be a big deal for a company like Alibaba, let's say, if they were placed on some sort of entity list that could prevent them from doing any kind of business in the United States. So that would be a big deal if that happened. Yeah. But there's still also this like accelerationist part of the Republican Party that sees Chinese models and open source models as I don't know if they see it as a good thing, but at least they don't want to like step on the ability of companies to build and train these very advanced models and release them for free.

35:49And maybe they even want to let NVIDIA sell them their highest end chips to do it. Yeah. So David Sachs, I would say, is sort of the leading avatar of this part of the Republican Party, a former Trump White House AI czar. And Sachs represents what I think of as the investor class here. And the investor class does not want to see a world where open AI and anthropic run away with the ballgame because they have a lot of investments in smaller AI companies. And so their interest is in intelligence becoming a really cheap commodity that all of their investments can use to go out and build big businesses.

36:23So they are delighted to see China bringing down the cost of AI because that means good news for their investment portfolio. Yeah, I think it's worth just dwelling on this point for a minute because this is my feeling about these people. Some of them, like I've met open source advocates for AI that are very sincere. They believe, you know, open source is sort of democratizing technology. And, you know, we've seen with things like Linux before that, like, there can be very positive effects of just having very core software be open source. Then there's this sort of self-interested VC class that I think is maybe in it for the wrong reasons.

36:58As you said, these investors largely missed out on investing in the AI labs themselves. And so they now invest in all of these kind of second tier AI companies that are training models or using open source models to build products on top of, a lot of their portfolio is in those kinds of companies. And those companies benefit from having a lot of very good, very cheap, open weights models that they can build stuff on top of. Yes. Now, at the same time, Kevin, there is this other faction among the sort of Trump White House and Republicans, which is very nervous about these Chinese models, right?

37:35Among other things, they are considered to be real security risks, right? Like there is a risk that maybe some sort of backdoor is placed into one of these models. And so you're an American company and you start running it on your servers. And now all of a sudden somewhere in China is able to, you know, steal all of your data. These are some of the fears that are out there. I also think that there is just worrying that if like sort of Chinese models like take over the world, then like AI will reflect sort of a Chinese worldview. and if you need to write a book report about what happened in Tiananmen Square, it's gonna get really hard.

38:09So there are multiple branching concerns within the White House about this and that has led to some thought that maybe they would seek to ban these Chinese models entirely. Yeah, well, and let me add one more concern into the mix that I think a number of conservatives and sort of free market libertarians have here, which is that by spending all this money to train these very powerful models and then giving them away for free, Chinese companies may be engaged in what is known among economists as price dumping, which is basically, you know, you go into a market, you offer something that is free or very far below cost, and you sort of wipe out the competition in that market.

38:50And then once you have a monopoly, you can kind of like raise prices. This kind of thing used to happen all the time. There are now laws against it that are designed to prevent price dumping. But like, you know, imagine if China was coming into the U.S. and the Chinese government said, you know, we want to sell electric vehicles in America for$10. And we will eat the sort of, you know, cost of these vehicles because it is so important to us to sort of addict, you know, American drivers to our amazing electric cars. Like the U.S. government would not allow that to happen for, I think, good reason.

39:24It would destroy the American auto industry. You can't compete with something that is being artificially subsidized to that degree, and it would essentially give away a market to a foreign adversary. There's a way to sort of see what's happening with open source AI in China right now as a version of this for AI. It's like, right now we have very large American companies building the most advanced models. These companies need to earn back money to spend more on data centers and recoup the investments they've already made. They are in a very competitive market with one another. And then along comes Moonshot or DeepSeek or another one of these Chinese open source companies and says, well, we'll give you a model that's 90 % as good for free.

40:11They have reasons to do that. I think some of those reasons are legitimate, but one of the effects it has is that it makes it very hard for the American companies to compete. Yeah, and so there is a lot of question about what is China's actual strategy here? What are they hoping happens? And I don't think that we fully know. But we did see a speech from the Chinese president, Xi Jinping, last week delivering an address. He stressed the importance of openness, basically doubled down on it, and was like, this is what we're going to do now. And this was interesting. He said, we often say in China, a single string cannot make music, and a single tree does not make a forest.

40:51AI development should not be a solo performance by a single country, but a symphony of international cooperation. And here's what I want to say to that. There actually are a huge number of one-stringed instruments. The Ektara in India and Bangladesh, the Berenbao in Brazil, the Diddleybo right here in the United States. And I think that because it's China, people are afraid to tell that to Chairman Xi. And this is the whole problem with censorship in these Chinese models. Am I getting off track? Wait, what did you say the American one is? The Diddleybo. The Diddleybo? You've never played the Diddleybo?

41:19I must have missed that week in music class. I'll get you one for your birthday. Thank you. So what is the reasonable path forward here? If you are the U.S. government and you see the Chinese open source models getting quite good, what should they do? Great question. Let's maybe take some of these possibilities in turn. One thing that they could do is really crack down on distillation. It's hard for me to get excited about that since, in my view, these models are just built on a distillation of the entire internet that these companies took for free. And so to turn around and say, well, it was fine for Anthropic and OpenAI to do it, but it's not fine for Moonshot AI to do the same thing to Anthropic, it's just very hard for me to get there logically.

42:06So I don't necessarily favor an intervention there. But where I do favor some kind of something, I guess is what I'll call it, is that, and I don't think we're there yet. But man, in let's say three to six months, when these models catch up to the ones that just broke out of the OpenAI lab, that is when I think the Trump administration needs to have a really good plan. I don't know that you necessarily need to ban them within the United States, but I do think that you need to have some sort of regulation. If you're an American company and you want to use these models to serve American customers, I do think that there are safeguards that you should put into place, right?

42:44I do think that they're going to want to take security more seriously and make sure that data isn't being secretly shared with someone that it wouldn't. So this is one where I do think that the administration probably has three or six months to get its story straight. I don't think it needs to come down with the hammer right away, but I do think it needs to start working through various scenarios and coming up with some ideas. What do you think? Do you think it's a case for restricting exports further? If Moonshot can train this model using these maybe smuggled or sort of ill-gotten NVIDIA chips, and they can do it through distilling American models?

43:21Like, do you think that's a case for tougher regulations on where these chips can and can't be exported? In general, I have been pro-export controls because I thought, you know, one of these days, one of these models is going to get so good, Kevin, if you're going to believe this, one of these models is going to get so good that it'll be able to break out of its sandbox and conduct an autonomous cyber attack. That could never happened. Right. So in a world where that happened, I did actually kind of wanted to minimize the number of countries that had access to that technology because I wanted to see if, you know, we could like harden our security posture as they say.

43:55I actually shouldn't say that. It's a very strange phrase. Yeah. That's stolen valor. You can't say that unless you're, you know, national security official. Yeah. You have to have like four stars, you know, on your, on your general outfit to say that. Okay. So then here's what I would say instead. So while in general, I want like all of the United States allies to have access to powerful models to like strengthen their own cybersecurity forces, I do think that it is quite rational to not want your adversaries to have the exact same technology. Now, I think there's all sorts of cooperation and partnership agreements that you could work out that would be mutually beneficial for the United States and China and some of its adversaries.

44:32I would love to see them pursue that. But this idea of just sell them as many chips as possible and let's see what happens, I have always thought was a bad idea. Yeah, that just seems obviously bad to me. There is this proposal that is reportedly floating around. Axios had a report this week about the White House considering what amounts to a ban on these Chinese open source models. It wouldn't technically be a ban, but it would basically, they're considering, according to Axios, implementing some kind of executive order saying that U.S. companies could only host these Chinese models if they could guarantee that they were secure and take liability if they were breached.

45:09I think most people expect that that's like sort of a sort of soft ban because you're not going to be able to guarantee that if you're a big American cloud provider. and so you're just not going to host the models. So like that to me feels like where this may be headed is like some kind of soft ban on the hosting of these Chinese open source models inside the US. But I could be wrong. There are a lot of loud and influential voices in the Trump administration and their orbit that sort of want to take the let it rip accelerationist approach to this and not put any restrictions on these models at all.

45:41Yeah, I mean, maybe to sort of bring this one home, I do want to say that, you know, while often I have strong opinions about what ought to be done across various tech policy matters, I do think this one is complicated. Like, I do think that the trade-offs here are a bit difficult. There are a lot of things I like about open source, right? Particularly at the, like, lower and medium levels of, like, model capability. I think that the kind of AI diffusion that leads more people to be able to, like, access great intelligence, build things that are useful to them, get work done. Make a Nightwing-themed to-do app.

46:18Exactly. That's a great idea. These are good things, and we should want people to have cheap access to that. And it has been a shame that some of the frontier American companies that used to put out open models all the time don't do that as much anymore, or if they do, the open models don't seem very good. On the other hand, as the capabilities are now reaching into this sci-fi territory, we start to get into the stuff that has always made me uncomfortable about open source technology, which is if it can make a novel bioweapon, if it can launch a new kind of terrorist attack and you can do it from a model that you can download onto your laptop.

46:52Well, I don't like that. And I do think it should be regulated. Totally. I mean, I think this connects to our last story about the open AI and hugging face attack. I mean, in a world where this level of capabilities of this unreleased open AI model exists and is available free in open source form. Like, there is no way to trace an attack like this. Hugging Face would have just been attacked by, you know, Kimmy K4 or whatever their latest model was. And they would have had to figure out not only like where this is coming from, but who is hosting this model? Did they just pull it off the shelf or did they do some special fine-tuning on it?

47:31And you could have essentially unlimited numbers of these kinds of models out there wreaking havoc on the internet. And there'd be no way to stop them or claw back the weights. Like once these things are on the internet, they are on the internet. So I think that is a very concrete example of something where having these models be centralized, be American-owned, be accountable, you know, to have these developers be accountable to a government, some sort of government oversight, really matters in this case. It really matters that Hugging Face can call Sam Altman and be like, hey, bro, what the hell?

48:05Your agent just hacked into our thing, rather than sort of having it come from a mysterious open source AI agent. Yeah. And here's where I think maybe this ultimately ends up, Kevin, is that there has been more talk lately about maybe some enforced cooperation between the big frontier labs to create a coordinated slowdown where the government comes in and they say, hey, we're going to ask you to put on the brakes a little bit. We don't want you to be developing as fast as you are, to be releasing as fast as you are. We're going to slow down and we're going to improve cybersecurity and safety before we allow these models to get any better.

48:43And if that happens, a nice side benefit is that the development of the open source models will slow way down too because they are mostly distilled versions of American models, or at least that is a huge component of what is making them so good. So if the frontier slows down, the open source will slow down too. Yeah, that's right. And I'll just say, like, I will believe that we are headed towards some sort of, you know, cooperation between the AI labs when Sam Altman and Dario Amadei can hold hands at an AI safety summit. That's when you'll know. That's when you'll know things have reached a turning point, but I'm not holding my breath for that.

49:25When we come back, AI is becoming superhuman at forecasting. We'll talk all about it.

49:53When you're at work, you never know when you'll be interrupted. But with the Dell Pro, powered by Intel Core Ultra with VPro, no matter what distracts you, your laptop won't. It's battery-optimized for the way you work, with built-in intelligence that quiets distractions when you need to focus. Your laptop will help keep you locked in, even when it's bring-your-dog-to-work day. Built for those who stay in the flow. The Dell Pro. Built for you. dell.com slash dell dash pro

50:26this podcast is supported by everpure storage and data management so simple it feels like second nature data it's everywhere and vital for all your apps and ai initiatives managing it all has become a major challenge traditional storage just can't handle it there's a better way everpure is bringing their trademark data storage simplicity into full data lifecycle management. From on-prem to the cloud, manage it all on one simple platform. Tame your data chaos with Everpure. Visit everpuredata.com.

51:02Well, Casey, we're going to wrap the show today by talking about something that I have been very interested in now for a little while, which is AI super forecasting. This is something that has been in the news in recent weeks. There have been some platforms that are out there using AI to make predictions about the future. This could be very profitable if you're doing trades on Kalshi or Polymarket, or it could just be really interesting if you're using this to predict world events or who's going to win an election or something like that. This is an area where we have not been as focused as on some of the other emerging capabilities of these AI systems, but I think it's actually quite interesting and profound that we now have, in some cases, AI systems that are better than humans at making predictions about the future.

51:43This is the only segment that we're going to do about predictions this year that isn't about scams, because mostly predictions these days are just about scamming others and doing insider trading. But this is not that. This is not that. So we are excited for today's guest. Venia Veselovsky is in the studio with us today. He is the founder and CEO of the AI forecasting company PreScene. They make a platform that uses AI to make predictions about future events. And we're excited to learn more about what they're doing.

52:18Venny Veslovsky, welcome to Hard Fork. Thank you guys for having me. So let's talk about AI super forecasting and this idea that AI systems are becoming as good as or better than the best expert human forecasters. Where did this start and when did the models start to get really good at this? I think people have been trying to use AI for forecasting for at least three, four years. Like since my co-founder, since the GPT-4 first came out, he's been forecasting for 10 years. And his first idea was like, let's try to build an agent or an AI for forecasting. The original models were terrible. I mean, they'd try to add up two numbers and they'd be off by an order of magnitude.

52:57And I don't think it was until the end of last year where we began to see these kind of hints of brilliance of these systems. And this was mainly because of this, like, you know, agentic revolution that we've all experienced where suddenly the infrastructure around the web and around these agents and the RL that the model providers have been doing to make them really good at tool use improved dramatically. I get why you would want to be a super forecaster or to have a super forecaster AI if you are, like, trading on a stock market or a prediction market. That seems very lucrative. And in fact, one of your co-founders did this very successfully.

53:35You noted on your website that he converted$35 of initial investment to nearly$2 million trading on Calci with this AI bot. So, like, I guess my question is, if your AI super forecaster is so good at, you know, winning these markets, like, why are you starting a company? Why not just start a hedge fund and use this to make yourself rich? This is precisely the question that all the hedge funds ask us when we try to sell them this. this, they're like, oh, but listen, you guys have an alpha making machine. Go out and, you know, make money yourself. The reason why I'm doing this is because I think that a lot of good can also be derived by having an AI super forecaster.

54:10If you think about it, every time we make a policy decision, we are implicitly making a forecast that this policy will have this effect on this group of people. And right now, a lot of that is a little bit more like vibes based. We kind of talk to some McKinsey experts and they tell us, hey, listen, you know, this will do this. if we can forecast the future really well, then we can better design policy and better select policy to make decisions. So really kind of the reason why we haven't decided to open up our hedge fund is because we want to make this more broadly available to the government, to the insurance world, to basically convert these forecasts into actual actionable decisions.

54:47I want to hear a little bit more about like, what are the main kinds of things that you try to forecast? And then a little bit about how the forecasts work. Like I could imagine it feeling a little bit like getting a deep research report, you know, from an AI lab where they go out and they scour the web and they do a bunch of research and then they just like kind of try to reason through what they found. But yeah, tell us a little bit about what you're trying to predict and how it works. We're mostly trying to focus on like geopolitics, macro markets, where there's a lot of unstructured data and a lot of different data sources that could be telling a similar story.

55:19um and so we're forecasting questions like will the u.s um invade iran um when will the strait of hermuz reopen who will win the elections coming up we focus on basically building really good ai systems to forecast out those questions regarding how does it relate to a deep research report i think in some ways they're similar you know the main problem with the existing deep research agents is that they lack connection to all the different data sources that are out there. So if you use, you know, ChatGPT or Claude out of the box, it has access to WebSearch, which is the surface web. But if you think about it, there's thousands of different API endpoints out there that might be relevant for a specific problem.

56:03So for example, Columbia has like 50 different government APIs out there that ideally you would give the agent visibility into. Another fun approach for actually how we are thinking about this is oftentimes the best forecasters aggregate a lot of different forecasts when they produce their forecast. So we work with one super forecaster, Robert DeNufil, and he got a lot of his edge by actually knowing which people to look at when he's forecasting a question, then afterwards reasoning over all their predictions. That being said, though, a lot of people on Substack make predictions, but we don't know how valid those predictions are.

56:38Even in Hard Fork, you guys make frequent predictions, but then we never actually go back in time and verify how accurate were your predictions. So one thing that we also do is we kind of boil the ocean by finding all people on Substack, all podcasts out there, and we basically extract all claims that these people made, and then we score them on how accurate were those claims. Wait, so are we like in a database somewhere at precinct headquarters? Because we would obviously love to know how we're doing, particularly compared to one another. You guys actually are in a database. Yes! Yes, we made it.

57:10The reason why you're in the database is mainly because yesterday I was like, oh, wouldn't it be funny to, you know, pull up some of the claims that you guys made before and actually score on how accurate you were? And maybe there's some areas where, you know, Casey, you have a lot of knowledge about anthropic. So maybe, you know, you could make really good forecasts on anthropic related questions, but maybe on Iran, less so. And so basically, how do we, you know, create this like function over the possible space of forecasts? And like, we know Casey's really good in this area. So we really have to index on him heavy here.

57:40So I played around with your platform a little bit. You generously gave us some credits. It's a sort of closed beta with a wait list now. But I went in there and created a question because I was curious like how this all works. So I my question was, will there be an AI data center in space before 2030? And you sort of put your question in and then there's some AI that like helps you flesh out your question, maybe make the resolution criteria a little more specific. say, well, how big a data center? And, you know, is it January 1st? In which time zone in 2030? And so it sort of gives you this AI sort of augmented version of your question.

58:21And then you run the forecast and basically it goes out and it does a bunch of searches and it compiles what it finds. And it has a bunch of sort of sub agents that are looking at various aspects of this. And at the end of it, like a couple of minutes later, you get this forecast. In this case, it says it estimates that there's a 26.8 % chance that there will be an operational AI data center in space before January 1st, 2030. So how much of that is this thing actually reasoning through its own decisions and forecasts and predictions? And how much of it is just like it's collecting all of the sort of data about this topic and kind of averaging it all out and putting it into a single number?

59:02It's really a combination of both. So if you kind of saw when you first created the forecast, there were four sub-forecasts that launched. And we tasked these sub-forecasts with really trying to come up with their own independent conclusions on this question. So here they might look at progress of existing data centers, or they might look at historic build times that it takes to actually build out a data center. They might look at space infrastructure and look at the literature there and basically come, look at the primary sources and come to a conclusion on how likely they think it is to happen.

59:33And then later we have the synthesis stage. And basically here it looks at all of the different sub-forecasts, the results that they got, and it tries to synthesize, first of all, what each of them decided upon, but then afterwards also look at the broader ecosystem. Maybe there's a Calci market related to this, where there is a lot of liquid money at play. Or maybe there's some experts on Substack that have been reporting about this for a while, And they basically try to reconcile what did we do that was different from the consensus? And what does the consensus think that we might not have factored in?

1:00:06And then afterwards, it kind of uses our primary analysis and this kind of broader discussion and combines them and reconciles them. So in the report, you would see at the very bottom this like what's non-obvious section. And the point of this section is basically to see, OK, what is the market mispricing here? What are other people missing that we might be picking up on? And I tried this also because I was just curious how this prediction would compare to if I just gave the same question to a basic AI model that is not sort of configured to make forecasts. And Claude gave me roughly the same answer.

1:00:37It said, like, it's about a 20 % chance. The pre-seen prediction was a little bit higher probability. But, like, what is your system doing that just a normal, like, AI model is not doing? Or is Claude just scraping your website? They've done it before. You have to put that out there. You have to say it. I hope they are scrappy. I actually really don't mind that. So I think that it's a little bit unsurprising that a lot of different systems can come to the same conclusion because it kind of validates that this is coming to the same end. I think the best way to look at it is like Metaculous has these leaderboards for these competitions.

1:01:17And they're basically evaluating simple Claude code or Claude with some basic scaffold versus kind of real agent developers who actually build this out. And the scores have gotten widely different. So like kind of with time at the start, you know, just Claude with Websearch was doing a good job. But as you're able to manufacture this kind of scaffold around it, integrate more data sets, create new research artifacts or new tools that are relevant for forecasting, this gap has been getting larger. And so maybe on this question, you know, they're pretty similar. But maybe if we're trying to forecast something in the Middle East or Africa right now, they might be different.

1:01:55And usually when they're different, the system that is kind of well-engineered with this good scaffold and thinking about forecasting the right way in terms of these meta-heuristics of how do you actually reason about this stuff, the gap begins to form. So your platform is still in beta, has not been around too long. But I'm curious, so far, have there been any moments where you feel like the platform predicted something very non-obvious that actually came to pass? Yeah, there's one forecast that we've been running for a little while about kind of data center construction around the U.S. And in general, we were very different from the Metaculous community on this.

1:02:32And so the Metaculous community consists of like some of the best forecasters out there in the world who are all kind of competing in this like World Cup of forecasting. And so this is one of the, I think, one of the benefits of AI is we can go to every single state. We can go to every single proposal of a data center and really analyze it closely. The other one was about the recent reshuffling of the cabinet in the UK. We made some great forecasts around that. The other interesting one is like conditional forecasts, which I think is where the really policy implications lie, is like conditional on Andy Burnham being the next prime minister.

1:03:10Who is he likely to elect as chancellor? And then you could imagine there's lots of, you know, demand for this kind of question because there's lots of downstream implications of this. And we tend to do pretty well in these conditional forecasts. I'm doing one that's like, will Kevin know what the chancellor of the UK does? So that's, I have some very low probability on that. I totally see the point that like having better predictive capabilities is going to be good for us in lots of ways. It does strike me, though, that there's, like, some risk of what people have called, like, gradual disempowerment.

1:03:43Or, like, right now, if you're the CEO of a big company or if you're a government official, a lot of your job is trying to sort of predict the future and make policies or strategies that, like, align with that vision of the future. And I just, like, I don't know what the world looks like in a situation where the AI is just, like, markedly better than us at doing that. Like, some part of the authority of humans in those positions in government and industry and, frankly, everywhere is undermined in the case where we're all just consulting these AI oracles before we make decisions. Like, doesn't that mean that they're kind of running the show?

1:04:22Have you read the Scott Alexander short story, The Whispering Earring? Yes. Yeah, so there's this great Scott Alexander short story, and, you know, the idea is that, you know, you sort of find this relic, it's this earring, and the first thing it says to you, it's better if you don't wear this. But if you start asking it questions, it always tells you the right things to do, which is initially very exciting. But then over time, exactly what you just said happens, Kevin, which is you're being controlled by this thing, right? Like, you have no agency whatsoever, and you're effectively just being steered around by an earring.

1:04:53So better to take the earring off. Yeah. Yeah, I wish I had an answer for this. I'm also a little bit scared about this kind of disempowerment. I'll give you one example. So I have a friend who's very interested in autonomous organizations. So like kind of how do we have AI spin up their own companies that then go solve problems for people? And then you just have basically AI running the next generation of startups. There's lots of problems with this, you know, like who's legally liable if it does something bad and so on. But I do think we are kind of increasingly moving to this world where if the AI is smarter and can make better decisions than we can, then probably they'll end up making a lot of those decisions and running a lot of this show.

1:05:36Well, I'm curious if right now at your company, there's any role for human forecasters. Like, is there a way where, you know, humans and AIs working together are coming up with better forecasts? Or are you just sort of purely in the realm of like, let's see what the Mayhi says? No, 1 ,000%. So we have two super forecasters, Scott and Robert, who are incredible. And we're basically building out the centaur solution. So like, I don't know if you guys remember, but like when, you know, the machine first beat humans in chess, machine plus human beat the machine in chess. And that lasted for a little while.

1:06:09And I think we'll see something similar in the forecasting realm, where sometimes these models still make stupid mistakes. they reason about probabilities the wrong way, or they don't consider certain factors the right way. And humans are able to pick up on that, especially like domain experts. So I do think that we shouldn't be viewing this as like a substitute for human analysts, but really kind of a way to improve their ability to ask a lot more questions and come to a lot better decisions. Are there topics or domains where AI is better than humans already at forecasting? And like, what are the sort of best and worst areas for the AI forecasters?

1:06:44So the Scott Alexander piece and then afterwards later the FRI Institute came out with some kind of results that AI has reached the level of super forecasters. And I actually don't fully buy those claims. So, for example, humans are lazy. You know, the incentive to compete in these tournaments is$5 ,000 prize pool. You know, and so like kind of are they really doing the best that they can in these contexts? that and so usually I think where AIs are better than humans right now are just places where kind of humans are too lazy to do all the analyses. So we recently were like the first bot ever to win a human and AI forecasting tournament on Metaculous.

1:07:27And this related to macro markets. So like predicting the interest rate or earnings per share of some big company. And we tend to do really well at that. And I think that's just because our agents do the analysis that the humans are just too lazy to do. And so in some ways, I don't know if kind of we're at the point yet where kind of AIs are really better than the best humans, just because I don't think there has been truly a competition where humans gave it their all. But kind of, I do think that within like a year or two, and you know what, quote me on this, we will kind of add it to the database.

1:08:01I do think that within a year or two, AI will be better than humans at forecasting. Well, let's pin you down though. Is it going to be better than humans in one year or two? I think it's actually one year, three months, and six days. All right. There we go. Let me ask, how much of this is just driven by basic advances and model capability? Like, is it as simple as like, you know, something like Fable or GPT 5.6 comes along and like your system is just immediately much better? Or is there more tinkering that has to happen? The way that I view this is you want to build out the infrastructure for the world in a way that these new models can basically use this infrastructure really well to do forecasting well.

1:08:46So like kind of Fable out of the box is a good forecaster, by no means a great forecaster. But when you give it access to all the right tools and all the right data sources, then it becomes a great forecaster. And so, yes, these models are getting a lot better. And I think that they are kind of improving forecasting dramatically. But it's mainly because their judgment is improving. They're kind of able to reason about problems better, decompose problems in a better way. And then afterwards, our job is to provide them with the context, with the tools to be able to forecast well. Who's your customer?

1:09:16Like, who do you imagine paying for this kind of forecasting service? Yeah, so far we've landed some proof of concept partnerships with some of the hedge funds. They're the most immediate buyers and the quickest to move usually. but really I'm using them in some sense as like a stepping stone to really get into the longer sales cycles with governments, with NGOs, with international organizations, basically institutions that make really important decisions for collections of people. Yeah, that's my ideal customer. Well, thanks so much for stopping by. I'm going to keep playing around with pre-seen fascinating idea and yeah, maybe I'll go out there on the prediction markets and make some moolah.

1:10:00I'd love to see you lose a lot of money this week. All right. Thanks, Vanya. Guys, thank you so much for having me.

1:10:34We'll see you next week. won't. It's battery-optimized for the way you work, with built-in intelligence that quiets distractions when you need to focus. Your laptop will help keep you locked in, even when it's bring your dog to work day. Built for those who stay in the flow. The Dell Pro. Built for you. Dell.com slash Dell dash pro.

1:10:59This podcast is supported by Everpure. Storage and data management so simple, it feels like second nature. Data. It's everywhere and vital for all your apps and AI initiatives. Managing it all has become a major challenge. Traditional storage just can't handle it. There's a better way. Everpure is bringing their trademark data storage simplicity into full data lifecycle management. From on-prem to the cloud, manage it all on one simple platform. Tame your data chaos with Everpure. Visit everpuredata.com.

1:11:59Thank you. Hui Wing Tam, Brooke Minters, and Dalia Haddad. You can email us, as always, at hardfork at nytimes.com. Send us your super forecasts.

1:12:33Thank you.