AI Insiders Keep Saying We’re In Danger — Where’s The Evidence?

0:01Support for the show comes from VCX, the public ticker for private tech. The U.S. stock market started history's greatest wave of wealth creation, from factory workers in Detroit to farmers in Omaha. Anyone can own a piece of the great American companies. But today, our most innovative companies are staying private longer, which means everyday Americans are missing out. Until now. Now, introducing VCX, a public ticker for private tech, now available wherever you buy stocks. Visit GetVCX.com for more info. That's GetVCX.com. Carefully consider the investment material before investing, including objectives, risks, charges, and expenses.

0:36This and other information can be found in the fund's prospectus at GetVCX.com. This is a paid sponsorship.

0:46This episode is brought to you by Google Chrome. You think you know a browser. But Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required compatibility and availability varies 18+. Push your limits. Train with precision. See the results. At Equinox, that's high-performance loving. Iconic spaces that inspire.

1:20Personal training backed by real data. Unlimited group fitness classes from yoga and Pilates to strength and conditioning. Elevate your post-performance ritual with saunas, steam rooms, cold plunges, and more. Everything you need to lock in and unlock your potential at Equinox. Start today at equinox.com.

1:46Mark Zandi:Money market's mad. If money is evil, then that building is hell. The show goes on! The folks in there have watched the show, show! Welcome to Prof G Markets. I'm Ed Elson. It is September 16th. Let's check in on yesterday's market vitals. The major indices fell ahead of the Federal Reserve's interest rate decision, which comes out later today. Brent crude topped$109 per barrel. The yield on 10-year treasuries climbed to its highest level in almost two decades. And finally, Bitcoin fell after the Senate blocked the Clarity Act, which proposed definitions and regulations for digital assets. Democrats said the bill did not do enough to address the ethics of President Trump's crypto business.

2:33Mark Zandi:OK, what else is happening? Two months ago, a Google DeepMind safety researcher quit. And this week, he told everyone why. In a post on X on Monday, Bilal Chogtay announced his resignation from DeepMind, where he worked on safety and alignment research. He also wrote, quote, I earnestly believe that AI has the potential to kill us all and that we might be running out of time to avoid this outcome. Tugtai is now at Blue Dot Impact, a nonprofit that trains people in AI safety. He is now the second safety researcher to go public about the risks this month following Jacob Cox's viral resignation from Anthropic last week.

3:12Mark Zandi:Meanwhile, the AI labs are responding with the development of a new regulatory body themselves. Anthropic, OpenAI, and Google DeepMind have discussed building an industry coalition to test frontier models before release. OpenAI is also reportedly urging Washington to move forward with a federal AI framework. However, Trump does not appear to be open to that idea in the slightest. So here to discuss all of it, we are speaking with Ed Zittrin, host of Better Offline and author of the Where's Your Ed At newsletter. Ed, good to see you. I want to just get your reaction to this whole debate in its entirety, going from the beginning where we had this tweet from Jacob Cox, in which went mega, mega viral, saying that he believes that AI might kill us all by the end of the decade.

4:05Mark Zandi:Then, of course, another anthropic researcher co-signed that statement, said that there was a 10 % likelihood of it happening. and now here we are it's become a national debate a sensation around the world what is your take what do you make of all of this we've now heard at least 11 of these guys say they want a slowdown we've heard exactly zero of them say what that means while Dario Amadei spelled out what he would do it mostly came down to having METR which he claims is an independent research organization despite it being basically funded by Anthropic itself, being an internal auditor, and then some vague stuff about China.

4:43These companies not slowing down. None of this safety stuff matters to them. All of it is lip service to a media industry that doesn't actually think for itself. Let's start with Jacob Coxon, though, and why he is the way he is. It could be a cynical grift. It could be the, because you'll notice that he doesn't actually explain what it is he's scared of what is the anthropic or open AI did that was so scary. And indeed, when describing the hugging face attack, for example, anthropomorphizes it and never holds the AI labs accountable. It's always, oh, AI is this unknowable thing. But it starts with the rationalists and effective altruists.

5:18These are, while they will describe themselves as people that are rationally thinking about the dangers of superintelligence, what they actually amount to is a kind of religious cult. these people are a large part of the ai labs and they spread this idea that ai will inevitably foregone conclusion definitely kill us all unless we stop it but stopping it always involves giving a specific corporation they like money and power so there is a section of these people who truly believe this they believe it on very flimsy terms they don't really have any evidence for it other than going look a thing we vaguely predicted kind of work it kind of happened not exactly the way and also we've been talking about recursive self-improvement which is ai that trains itself that's kind of happening and so everything is true so there are those people who genuinely believe it's happening and like any good religious cult will take any proof to prove it and also the media buys their hype every time and then there's the other side which are the cynical people i put sam altman in this category and ari dario amaday to an extent who are like yeah ai safety is good AI is so scary, but maybe don't stop us training our models.

6:26They've been doing that for years. The cynical AI safety grift, the thing of, oh, I'm a little bit scared of what the AI models can do. Sam Ortman's been saying that since 2023. Dariyama Day has been saying it since 2019 with GP2 when he still worked at OpenAI. And all of this is to say, despite all of the noise, despite all of the endless hand-wringing. No one can actually describe what it is we're scared of, what should happen, what a slowdown means, what AI safety means, or indeed how we hold the people accountable. Because in my mind, arrests need to happen at OpenAI and Anthropic because felony hacking took place with a hugging face.

7:07But none of this stuff about safety actually seems to result in accountability.

7:12Mark Zandi:There are so many different voices here saying surprisingly, different things. As you say, you've got Jacob Cox, and then the question is what his incentives are. I think a lot of people might just think, well, maybe he wanted to be famous by putting out this tweet. Maybe he wanted to be sort of this viral philosopher on what it means to live in a world of AI. Or maybe it's genuine. Maybe he genuinely is very scared about this, and maybe a lot of people genuinely are scared about this. Then there are the questions around what are Dario Amadei and Sam Altman going after? And one person, I mean, a popular belief is that they are creating this level of concern either to, one, draw up enough hype about the product that they can go out and have a successful IPO and raise a lot of money, or maybe it's because they want to inject a level of regulatory capture so that they, as the now kind of incumbents in frontier AI, can win that world.

8:17Mark Zandi:But someone who thinks that it's a lie, thinks that it's a hoax, in his words, is the president, who called Jensen Huang this week while Jensen Huang was on stage doing a live podcast with the All In guys and said as much, I want to play you this clip and see what you make of it. It's all a hoax. The data centers are great, and they make people wealthy, and they make states wealthy, and it's the oil of the next 20, 25 years. It's bigger than the Internet and the AI, you know, much more so. And they're just playing right into the hands of a lot of people that don't want to see it happen, and that could be political people, that could also be China.

8:59And we're not going to let that happen. It's a hoax. You're right.

9:04Mark Zandi:We're not going to let that happen, sir. No, we're not going to let it happen. So I guess I should add to that. There are other accusations in there, which is maybe the incentives are aligned with people who don't like AI and want to shut the whole thing down, or maybe China. I mean, so many different accusations flying in different directions. What do you make of his comments? It feels up adjacent because when Jacob Coxon posted, a bunch of AI safety people quoted it immediately. So people are like, oh, this is an industry-wide plan. I have another suggestion. This is a burn-off to reading situations, Coen Brothers-esque.

9:39you've got all of these different people who kind of operate on the same page they're all like we love ai but you have the rationalist ea types who are like we're terrified of this and we will but also we need to divert the money to our organizations you've got ai boosters who are like well i've been saying this stuff is crazy bananas and gonna kill us all and so powerful but maybe we're getting a little ahead of ourselves and you got jensen hwang who's just like no no no no no no no it's cloud software please stop saying this we need to sell gpus but a week ago jensen and Huang said that we'd reached AGI.

10:11So I don't know what to tell you. What it is, is a bunch of very selfish people all with their own agendas that have said they like AI. But none of this is to do with AI. None of this is to do with AI software. None of this is to do with anything. It's a bunch of people trying to get attention and power and money without any kind of plan. If there was a plan, they would actually have something they're suggesting. If we had smarter fascists, we'd be in real trouble. Instead, we have various grades of dillweed who are saying, well, okay, I love AI, but when I say AI, I mean LLMs, which are so powerful, but also not as scary as you say.

10:46You've got the EA people who say, this is just one step towards Skynare. And then you've got the ultra capitalists who are like, it's nothing, please stop talking about this. It's AGI, but not that. And the thing is, this is all the result of how the AI industry has marketed itself for the last three years. It's all about distancing what the product can do from reality and modern journalism has failed you yourself have made the point that we have this cult-like worship of the wealthy and especially of credentialism so the media has just yum yum eat this up ai scary oh it's gonna kill us even though the evidence for that is that a guy told them and when asked for further evidence they say well a guy told me jacob coxson to wired said oh yeah people at Anthropics say this these are direct quotes about endgame that is not enough to say anything so a bunch of people said something what happened can you point to a thing well we're near recursive self-improvement what does that mean are you actually near they're not but they all want to say they are because without recursive self-improvement they have to admit the AI industry is kind of slowing to a crawl so in this very bizarre situation that honestly is kind of the AI industry's undoing.

11:59Because the media failed. They failed to call BS on these companies for years. They have bought into every narrative that LLMs are super powerful, autonomous AI, even though it's not true. And now, when a little Harry Potter looking like Goblin pops up and says, oh, I'm scared of the computer, they fall for it. They fall for it because they've been building this hype for years based on nothing. Not based on using LLMs, not based on anything. And it's just a very bad situation. Donald Trump coming out and going, actually, AI is the biggest thing. We love it. It's better than oil. Yeah, bet you wish it was, mate.

12:33Bet you wish it was the new oil. That would really help right now, wouldn't it? If it was the new oil, right? The truth is, this is just an escalation around a technology nobody is actually describing with accuracy. It's talking about hugging face like it was something that happened accidentally versus poorly run cloud software run in a volatile and reckless manner by a company with unlimited resources. Basically, what appears to be felony hacking run on infrastructure owned by the largest companies in the world. But it's being described as, oh, rogue agents went and did this. No, it didn't. It's LLMs prompting LLMs with a coding harness on top, telling them what to do, trying to solve a vulnerability benchmark and not having the right security practices.

13:18But because the media has failed to hold the AI industry accountable or describe anything with any realism, the AI industry has actually caused a real problem for themselves. Because now everyone's like, well, AI is going to kill us. What are you going to do about that? Every goddamn conversation about AI now, which is fun for me, I'm having a ball. These people do not have an answer. Because how do you pull back this narrative? How do you convince people that actually the thing you have been describing in terms of a software that's doesn't exist is actually not what that is. And so these companies are kind of looking one way and the other, looking at each other, going, wait, what are we doing right now?

13:57And they'll claim, oh, we want to slow down, but we're still going to train the models, I guess. All of this is to say this could all lead to nothing, or it could lead to onerous regulations or just a slowdown in the building of new models that would actually be fatal to the industry.

14:14Mark Zandi:Yeah, it seems as though no one even has any understanding of what they're actually arguing for, whether it's Trump, whether it's Altman, whether it's Amadei, whether it's David Sachs or the people who don't like AI. I mean, it seems to me to have been a profoundly stupid conversation that has taken over the world because, as you point out, it is rooted in almost nothing. It is rooted in a tweet for which there was no evidence or no investigation into what was actually being said, what was actually being claimed. And to your point, there is something missing here, which, as you say, is accountability.

14:52Mark Zandi:Accountability for what any of this actually means. and something that I've been saying, and I wonder if you would agree with it. I mean, part of me believes that we should, our government really should be calling the AI labs bluff here and saying, okay, if you believe that your technology actually has a 10 % chance of destroying our society, well, then here's a subpoena, show us all of the evidence, show us all of the proof as to why you think that's happening. And if you have created a technology that is actually going to do what you say it's going to do, then you have to shut down or you have to live whatever the consequences, whatever the legal ramifications of killing people would be, which is to go to prison.

15:34Mark Zandi:But we're for some reason not having that conversation and the conversation remains in this sort of ethereal space of conjecture about what the future of humanity or AI might actually look like versus actually grounded in truth, facts, evidence, and ultimately the law. And so I guess my question to you is, you say that the media hasn't been responsible enough in their reporting of it. Would you also agree with my position, which is that the government hasn't been holding them accountable enough either? 100%. So, Lina Khan, I paraphrase here, made the point that we have laws in place for unsafe products.

16:09I also don't see a single story other than my own bringing up the multiple suicides driven by ChatGPT, the multiple mass shootings, the murder suicide that happened. None of that. That's a harm. I can point to that right now. I brought it up on a podcast recently to an AI Doomer, and they went, oh, that was six people. And it's like, this is how the world operates, just this disgusting growth-focused capitalism. But yeah, we already have reasons to shut them down. OpenAI and Anthropic, and it sounds like Meta and Google as well, their models in their testing hacked. That is felony hacking. There is prison time associated.

16:48We should, at the very least, have some people in handcuffs being talked to by the FBI, it sounds like. Sounds like, actually, probably international courts. Depends on where the servers were. We don't have any of that. We don't even have journalists who are bothering to ask those questions. Instead, it's this wobbly nonsense about, oh, the stewards of our AI future. It turns my goddamn stomach. We're doing it again with Jacob Coxon. despite the fact that he will not say what he's scared of. And when he describes LLMs, he does so by anthropomorphizing them. This is all to distance the labs from responsibility.

17:27The reason I hold the media so accountable is because the media is the one that drums up this nonsense. The media is the reason that everyone believes that LLMs are more than they are. And yes, the governments are also failing here, but governments don't regulate tech. Let's be completely honest. Governments don't touch tech. If they did, we'd have an EPA for notifications. We would have an FDA for tech writ large. The whole social media trials wouldn't have happened because we'd actually have something in law around regulating social networks. Really, I know this sounds extreme, but we should regulate notifications like exhaust because they are used to manipulate people.

18:08Algorithms, same deal. We don't do any of that because it might get in the way of capital. This time, it's even dumber, though, because it's how would you regulate this? How would you actually regulate this? Also, what's this? Because no one can actually seem to say what's going on.

18:23Mark Zandi:To be fair to the media, it's hard to not cover a story about a company whose employees are telling you that their technology will kill everyone. But that's part of the strangeness of the whole situation, because you would hope that maybe at some point there would be a more thorough investigation into whether we should actually take it seriously, validate whether that is true or not. And then if it is not true, move on. Don't care anymore, but we're still lingering on it. Yes, we should cover this, but the way to cover it is how we are, which is saying, hey, what, what are you talking about? What are you scared of?

19:02And when they don't answer go, that's pretty vague. You seem very like, not even mean, just be like, you seem very scared about something that you don't have much information about. You seem extremely anxious. And if the answer is, well, some smart people told me, say, okay, but what did you see? What did you see that scared you?

19:24Mark Zandi:Just before we let you go, this is interesting timing because Anthropic is, of course, set to go public very soon in what could be the largest IPO ever. Sam Altman said that he will not be taking open AI public this year. He will be delaying that after this all broke out. It would be ill-timed. It would be ill-timed. I'd like to just get your thoughts on, your preliminary thoughts on the Anthropic IPO, and especially your reactions to a recent Financial Times report on Anthropic. Supposedly, Anthropic has told investors that it has achieved operating profitability for two straight quarters. They say adjusted operating profitability, which is probably doing some work there.

20:08Mark Zandi:But what do you make of that news? Because it would imply, if it's true, that the AI business model is more sustainable and more profitable than many were concerned about. Well, that 80 % gross margin, to be clear, did not include training costs or stock-based compensation. So it's kind of like saying, I'm profitable if you don't include my costs. Here's the thing. People are suggesting that, oh, a slowdown could mean they train models less. I would buy that as a way of getting out of compute commitments and as a way of reducing costs, except they've explicitly said they're not going to stop releasing models.

20:43Mark Zandi:Is your view that when the S1 comes out, we will see that they are still an immensely unprofitable company? Yes. Unless they do some really weird stuff with capital expenditures, unless they try and capitalize R &D costs. It's funny, they won't break out inference costs, but we're going to find out whether it truly is the gym model with subscribers, whether it's just that most of them don't actually use it that much. But here's the thing, training costs aren't going away. I also cannot wait to see their sales and marketing costs. I cannot wait. Oh, also another thing about the 80 % gross margins.

21:19That doesn't include the amounts of money they send to Amazon, Google, and Microsoft in the revenue share, which they use to inflate their revenues. So, I don't know. It sounds like some accounting shenanigans to me. I can't wait to read the S1 because I think it's going to be a laugh riot.

21:37Mark Zandi:Ed Zitron is host of Better Offline and author of the Where's Your Ed At newsletter. Ed, it's good to see you. We always love having you. Thanks so much. After the break, the bond sell-off continues. And for even more markets insights, you can subscribe to my weekly newsletter, simply put, at edwardelson.substack.com.

22:23so you eat less and keep the weight off. And with the Wagovi pill, the first ever GLP-1 pill for weight loss, you can get treatment without needles. The process through HIMSS happens entirely online. You'll connect with a licensed provider who can determine whether treatment is right for you and if prescribed, your medication is shipped directly to your door, no insurance necessary. You'll also have access to 24-7 messaging with your care team, plus in-app lifestyle and nutrition support, including meal plans, fitness videos, sleep content, and more. Ready to reach your goals? Visit HIMSS.com slash markets to get a personalized, affordable plan that gets you.

22:54That's HIMSS.com slash markets. HIMSS.com slash markets. Weight Loss by HIMSS is not available in all 50 states. WGOVI is the registered trademark of NOVA Nordisk AS. To get started and learn more, including important safety information, WGOVI clinical study information and restrictions, visit HIMSS.com.

23:15Support for the show comes from Blinds.com. Your home should look and feel exactly how you want it to down to the lighting. Sometimes all it takes to make a room feel finished is investing in good, high-quality blinds. Blinds.com has spent nearly 30 years making exactly that. They've covered over 25 million windows with more than 50 ,000 five-star reviews. You can go full DIY if you like or bring in licensed and vetted pros to handle the measure and install the exact blinds your room needs. They have a large variety of style options at prices that fit any budget, and you can get samples shipped to your door fast and for free.