Should markets discount the AI apocalypse?

0:00This episode is brought to you by BigData.com. Your AI is only as smart as its data. BigData.com connects Claude, ChatGPT, and Copilot to the sources hedge funds and asset managers use. SEC filings, financial times, earnings calls, and over 2.5 million transcripts from 50-plus countries. Verified, institutional-grade intelligence right where you work. Try it free. Pay-as-you-go plans start today at BigData.com. Human listeners of the Unhedged podcast prepare to die at the hands of those pesky robots. The big dogs of big tech have agreed that AI is so clever and so potentially dangerous that we can't rule out that it will try and wipe out humanity.

0:45Their solution is to hit the pause button on further development of the most powerful models. This seems bad. A week ago, everything was fine. We all knew about the rogue AI widgets that were hacking systems they should not have been playing in. And hey, you normal, non-techie people don't need to worry your little heads about it. Now, apparently, we do need to worry our heads about it. Cool. This has all taken a little bite out of stock markets. Not much for an end-of-humanity trade, given that stock markets are rammed full of AI stocks. But it has also not spooked investors into buying the safe stuff, bonds.

1:27If anything, those bonds are still weakening. It all feels like a bit of a mess. So today on the show, the unhedged guide to the robot apocalypse. This is Unhedged, the markets and finance podcast from the Financial Times. I'm Katie Martin, a markets columnist, hiding from the evil robots in the basement of FT Towers. Joining me down the line from New York City is the big man, Robert Armstrong. Or should I say, Robot Armstrong.

1:57Katie Martin: That is so dumb, Katie. I knew you were going to do something dumb. But I didn't know it was going to be that dumb. Well, how do we know, though, Rob? How do we know that you're not a massive robot? It could be a simulation. Yeah. It's absolutely right. Maybe the way that we do know is the pain that you're in after your Ironman thing at the weekend. No, I wasn't an Ironman. It was just a simple triathlon. And yes, I am having trouble moving normally today after I got a decent time. But you're paying for it now. I'm paying for it now. Yeah. So look, I'm finding this whole warning of the end of the world thing genuinely quite annoying.

2:40So tell the nice boys and girls what is going on. What happened over the weekend?

2:45Katie Martin: Well, two things have happened in the last week or so. One, we had a young AI researcher from Anthropic come out and say he was quitting because he thought the technology was out of control and could potentially do us all great harm. Yeah. And then Dario Amodei, is that how I pronounce his last name? I believe it is. We were just talking about how I mangled every name that is put in front of me. But it's just because I get most of my information by reading things. And then I get on this podcast and I have to say words out loud and it causes a lot of trouble. So the anthropic guy. The anthropic guy, Dario, my good friend Dario, came out with this essay.

3:30Why do they always write these bloody essays, these like massive screens? It's highly annoying.

3:36Katie Martin: Yeah, I'm not a fan myself. But anyway, in this one, he said we need to slow down development of this technology because what they call, what is it, incremental self-improvement? Recursive self-improvement. Recursive self-improvement. So it's now an AI making the next generation AI. And as soon as they've made the next generation AI, that AI starts working on the next one. Yeah. And this process is increasingly taking on a life of its own. And they are struggling to understand, according to this essay, the capabilities of the product, as evidenced in the recent incident with an open AI frontier model hacking into the servers of a hugging face.

4:24Katie Martin: And it did this when the researchers involved didn't even know this model had access to the internet. It sounds like from Dario's essay that that was kind of the straw that broke the camel's back and he felt, well, we have to call for a pause or a slow down here. So, but the hugging face thing, and that's a crazy name for a company, but it is called Hugging Face, that happened months ago, right? And now suddenly, after this 27-year-old quit in a huff, maybe he's right, and if he is, give the guy a medal, buy him a pint, whatever. But suddenly, at the weekend, you had Dario Amadei from Anthropic, Sam Altman from OpenAI, and some guy called Elon Musk, all saying, this 27-year-old...

5:04Katie Martin: Like, I make AI too, guys, and I think it's dangerous as well. Is that your South African accent? But suddenly you have like the big dogs all saying, oh, yes, this is correct. This technology is very dangerous. Now, for me, there are only three ways to read this. And I was saying this in the newsletter the other day. Option one is these big AI guys are bad because they knew about this problem months ago with the hugging face thing and they did nothing about it. They've allowed this technology to rattle along on a plan to destroy humanity. Option two is they are morons because they didn't realise that they had built the torment nexus until this 27 year old told them.

5:48Or option three, this whole thing is just a total load of crap. And these companies want an excuse to like pause a bit and say, hey, so guys, like we've spent quite a lot of money here. We haven't totally figured out how it's going to pay for itself and how to monetize this stuff. So maybe we need to pause. So, I mean, call me cynical, but I struggle to come up with a solution to this that's not these guys are idiots.

6:16Katie Martin: You are cynical, Katie, and I'm not the first person to observe that. I think your three-part framework there is not totally off base, but I would soften it slightly. Last week was not the first time that these guys have talked about the possibility that this technology is operating beyond their control and warned about it and so forth. So why have they let this happen then? It's a fair question. Let's change the discussion slightly. Here's a question. If you were to ask Sam Altman and Dario Amadei and Elon Musk, is the fact that this technology is advancing at a rate beyond your control with potentially dangerous consequences, is that good for your business or bad?

7:05Katie Martin: How do you think they would answer that question? I mean, they're going to say good because they always say good. Whatever the question is, they say the answer is more AI. I mean, I think this is an issue that has always been the case with these warnings is from the outside. We can't know the degree to which these guys are warning us and the degree to which they are bragging. And the degree to which this is just rubbish. Yes. There is always the they're just full of shit option. But, you know, it's very, again, very hard to say from the outside. And it could very well be a mix of all three. And in fact, that would be the possibility that is most consistent with our knowledge of human nature up to this point.

7:50Katie Martin: You know what I mean? You mentioned in your very good intro that markets aren't taking this too hard. Yes. And part of the reason they're not taking this too hard is the fact that this technology is so good that it's scary might be kind of good for the businesses involved. But if they're going to like. As an investor, the big scary thing that's going to change everything, and there is a chance it could go terribly wrong, but we're not sure. Do you want to be involved or not? I mean, do you really want to finance the destruction of humanity? Is that what we're like here for? No, but it could also be as this letter from the anthropic guy.

8:37Katie Martin: Anthropic? Anthropic. Anthropic. As his letter said, still, you know, his kind of central hypothesis is that this technology is going to make everything better for everyone. There's just this little bit at the tail that you kind of have to worry about where it makes everyone dead. Right. And this is actually a question we've talked about in the newsletter before, which is if that's the probability distribution, 95 percent or 98 percent strong growth and one to five percent destruction of the world, what's the kind of net present value? Yes. And it's like, you know, in the outcome in which there are none of us left, nothing has any value at all whatsoever.

9:27Katie Martin: So it's almost like you can ignore that tail probability, right? And you just want to be fully invested in the 95 % good outcome because nobody's going to be counting if we're all gone. Yes. Not from a moral point of view, but from an investment point of view, there's a case to be made. Go all in, all the more so, right? because the terrible outcomes don't matter from an investment point of view. So this all kind of really kicked up on sort of Saturday, Sunday, and then markets opened on Monday. Some tech stocks were having quite a bad ride. The Korean stock market, which is very, very techie, that was off quite hard, about 3%.

10:05SoftBank, which is like this Japanese company that's got a big stake in OpenAI, its shares fell pretty hard. But overall, it wasn't a disastrous market reaction. You know, if this is the end of the world and this is what it looks like in market. So it's kind of okay. And I was chatting to an investor on Monday and he was saying, look, I just don't think this agreement between these AI big dogs really changes anything. You can make an argument that actually maybe markets should do better from here because this will encourage all the hyperscalers to stop spending so much money, preserve some more of their free cash flow, reel in the capex and just be a little bit more sensible.

10:44Also, he was saying, and I think he was being semi-serious here. If we really are going to get the end of the world, then right up until that point at which the world ends, then there's going to be lots of new companies that come along to defend us from the evil robots. And maybe we should invest in them.

11:01When I'm writing it down,

11:02Katie Martin: like, are we having this conversation? I mean, let me let me reframe the conversation again. And this is how I've been thinking about it the last couple of days. If you believe that recursively self-improving AI frontier models do pose a low but real probability of annihilation of life on Earth, do we think open AI and anthropic should go public or not? Or you might frame the question this way. Is the best steward of a technology that has this small but very serious potential, is the right steward a private company or a public one? So there's a lot to unpack there. On the sort of public thing, so again, Dario Emide was talking about this.

12:00He's saying, I've always said that this technology is so powerful that it's kind of crazy for it to be entirely in the hands of private companies. I guess my worry there is that this is kind of a way for the AI companies to say, oh, it turns out this does bad stuff. And we'd rather the governments were on the hook for this to clean up the mess than we were as private entities. So why should these guys be raking in literally billions of dollars worth of personal wealth for a technology that's going to be like me and you, Rob Armstrong, lowly little taxpayers? We're going to have to clean up the mess.

12:34Is that appropriate? So OpenAI was saying, again, at the weekend, we think that because of all this, we should delay our stock market listing until 2027. Anthropic, a week ago, appeared to be all on for listing in October. Now they're saying, well, it's kind of still likely this year. So it sounds a little bit like they're backing over a little bit. I guess what I'm wondering is, first of all, does this whole moment mark the peak AI mania?

13:03Katie Martin: No idea, I won't predict. I will say slightly orthogonal to that, but I think relevant. I think the sooner these companies list, the better. Like get it into the sunshine as fast as possible. You raised the question about what is bollocks and what is for real. And I think there's a lot of questions we need answers to. And we'd get more of them if these were publicly traded companies with obligations to disclose stuff that we want to know about. You probably would. So they say they're profitable. Great. if you're profitable, if you're worth a trillion dollars, you can list. Then we know more. I think that is good.

13:39Another question I have is, like, who's responsible for all this? If this does go wrong, who's on the hook?

13:45Katie Martin: This is the primary question to me, which brings me to the topic I've been dying to discuss in this whole conversation, which is pronouns.

13:57Katie Martin: Not they, them, or anything like that. I'm thinking more of the first person plural pronoun, we. What are you talking about, Rob? Just give me a second. I'm saying a really smart and insightful thing. So don't interrupt. Okay. And here it is. In these letters, like we've seen from Sam Altman and Dario and the rest of the gang before, there is a lot of we talk. We need to put guardrails in place. And my sense is the we they use in those letters. We need to regulate. We need to prepare better for the risks. That the we does not refer to the company in question. It's a bigger we. It's like the government, people, regulators, la la la.

14:46Katie Martin: What I want to hear from these two guys, Sam and Dario, I want them to use an I or a more restrictive we. I want them to be able to say things like, the buck stops with me. This company is accountable for what the technology it has invented will do. So I think you raise exactly the right point, which is with these warning letters, what they're doing is not warning us. They are spreading accountability. They're smearing it across the industry, the government, taxpayers, everybody. And that is the opposite of what we want to hear from a company that is handling a dangerous technology. We want those companies to say, if this technology, which we are going to profit from, gets out of control and hurts people, that's on us.

15:43Katie Martin: That's on me. That's on OpenAI. right and we haven't heard any of that talk from these guys right and if you're going to get rich as a sultan you got to be on the hook yeah I mean that's it's just that simple what I'm hearing is I've made my billions I've built my bunker in the mountains and now you guys the great unwashed you need to pay for the guardrails see you later yeah the government it's the government's problem no it's your problem yeah that's what I want to say to these guys well I tell you who says it's not a problem at all Rob Armstrong, is your glorious leader, Donald Trump. He says it's all going to be fine because the US has such a wise, high IQ president.

16:23So that's good. Yes.

16:24Katie Martin: It always comes back to you blaming me for what the president says. This is correct. Now, the other interesting wrinkle to how markets responded to this whole thing at the start of this week is that, so stocks kind of wobbled a bit and government bonds, right? The whole point of government bonds is that they jump in price when bad stuff happens. And you would think robots killing humanity or the prospect of robots killing humanity is bad enough to support government bonds, but low for what happened. In fact, the US government bond market in particular is in a right old mess, and the 10-year yield has popped up above 5 % or 2-5 % for the first time in a long, long time.

17:08Katie Martin: Ding, ding, ding, ding, ding. It did hit this level once in 2023 for like 10 seconds. But now it seems to be just above five. I'm looking at 5.01 % and holding as of this morning. We're in pre-financial crisis kind of levels here. So it just feels to me like there's two big things that have gone wrong all at once. One is that the bond market is like having a really big moment here. And, you know, it's not just the US. You look at UK borrowing costs. They're not great. We've rehearsed this kind of argument before on this show. You know, listeners, you can listen to other stuff that we've done on this.

17:42But the bond market is in a serious mess. The fact that we've hit this big round number 5 % on the 10-year does feel like, okay, this is actually quite serious now. And you've got the AI trade, which surely has to be running into some sort of difficulty because of all of this nonsense. This feels like two of the big pillars of global markets are kind of shaking at the same time.

18:04Katie Martin: And, of course, as we've discussed, the two are very intimately linked in the following sense. The AI boom is an investment boom. Investment booms have to be financed. The 10-year yield reflects a higher cost of financing. So the connective tissue between the two things you just mentioned is very direct. Like Nushir Sharma, our colleague, has said if the 10-year gets above 5 and stays there, it could pop the bubble. And so we're in that territory. I'm not sure I totally agree with him there. I think the bond market is responding to some combination of higher oil prices and strong nominal growth in the United States, the world's leading economy.

18:46I'm not as panicked about it as some people are, but I'm not saying it's not spooky.

18:51Katie Martin: I agree with you. It's spooky and it's related to the AI trade. This sense of things fall apart, I think, is quite unnerving, actually, unless Trump is right. And actually, it's all fine because he's so clever. I guess we'll find out soon. We're going to have to come back in one second with Long Short.

19:33save money on a standard annual digital subscription until the 28th of October. For more information, check out the show notes.

20:04institutional grade data right inside the tools you use every day start your free trial at bigdata.com okie doke it is time for long short that part of the show where before we're killed by robots we go long a thing we love or short a thing we hate i rob armstrong am long loafers so lvmh Wow. The luxury company says it could be selling many more pairs of its white sole casual loafers, which cost a thousand euros a pair. But it wants to keep them all kind of bougie and exclusive. So it's limiting the supply of these shoes. I just love luxury. It's totally ridiculous. The economics make no sense at all.

20:49And I'm absolutely here for it. So for those reasons, I'm long with loafers.

20:53Katie Martin: You're talking about the Laurel Piano. I am. They look like just normal shoes to me. I don't get it. No, they look horrible to me, and I think they should all be burned with fire. I'm so against these shoes. Nonetheless, I'm Long the Loafers. What have you got? Well, let's stick with the sartorial theme. I'm going to direct all of our listeners to look at our colleague Edward Luce's column today, which is about Scott Besant. I have not read a word of this column. I'm only talking about the picture. Scott the house besant to give him his full. Scott the house besant. And in this picture, he's raising his patriotic fist to the air and inadvertently showing how poor his tailoring is.

21:35Katie Martin: Oh, really? His jacket has a very low armhole, so just raising his arm has pulled his suit terribly out of shape. And this is a lesson to you, that you should always wear tailoring with a high armhole if you're the kind of person who moves your arms. So I will be long the high armhole. Long high armholes. Yes. Because then you can move around and raise your hands above your head without your tailoring looking all crazy like Scott the House Besant does in this picture in the FT. I mean, it's a scandal. It is really a scandal to have a man that rich wearing that kind of tailoring in public. Listeners, I think we have reached peak Rob Armstrong.

22:21we are going to be back in your ears on Thursday as long as the earth is still turning so listen up then